Saturday, October 24, 2020

Visualising article coverage in the Biodiversity Heritage Library

It's funny how some images stick in the mind. A few years ago Chris Freeland (@chrisfreeland), then working for Biodiversity Heritage Library (BHL), created a visualisation of BHL content relevant to the African continent. It's a nice example of small multiples.

For more than a decade (gulp) I've been extracting articles from the BHL and storing them in BioStor. My approach is to locate articles based on metadata (e.g., information on title, volume, and pagination) and store the corresponding set of BHL pages in BioStor. BHL in turn regularly harvests this information and displays these articles as "parts" on their web site. Most of this process is semi-automated, but still requires a lot of manual work. One thing I've struggled with is getting a clear sense of how much progress has been made, and how much remains to be done. This has become more pressing given work I'm doing with Nicole Kearney (@nicolekearney) on Australian content. Nicole has a team of volunteers feeding me lists of article metadata for journals relevant to Australian biodiversity, and it would be nice to see where the remaining gaps are.

So, motivated by this, and also a recent comment by Nicky Nicolson (@nickynicolson) about the lack of such visualisations, I've put together a crude tool to try and capture the current state of coverage. You can see the results here:

As an example, here is v.25:pt.2 (1988) of Memoirs of the Queensland Museum. Each contiguous block of colour highlights an article in this volume:

This scanned item-level view is constructed for each item (typically a volume or an issue). I then generate a PNG bitmap thumbnail of this display for each volume, and display them together in a page for the corresponding journal (e.g., Memoirs of the Queensland Museum):

So at a glance we can see the coverage for a journal. Gray represents pages that have not been assigned to an article, so if you want to add articles to BHL those are the volumes to focus on.

There's an argument to be made that it is crazy to spend a decade extracting some 226,000 articles semi-automatically. Ideally we could use a tool like machine learning to identify articles in BHL. It would be a huge time saver if we could simply run a BHL article through a tool that could (a) extract article-level metadata and (b) associate that with the corresponding set of scanned pages. Among the possible approaches would be to develop a classifier that would assign each page in a scanned volume to a category such as "article start", "article end", "article", "plate", etc. In effect, we want a tool that can could segment scans into articles (and hence reproduce the coverage diagrams shown above) simply based on attributes of the page images. This doesn't solve the entire problem, we still need to extract metadata (e.g., article titles), but it would be a start. However, it poses the classic dilemma, do I keep doing this manually, or do I stop adding articles and take the time to learn a new technology in the hope that eventually I will end up adding more articles than if I'd persisted with the manual approach?

Wednesday, October 21, 2020

GBIF Challenge success

Somewhat stunned by the fact that my DNA barcode browser I described earlier was one of the (minor) prizewinners in this year's GBIF Ebbe Nielsen Challenge. For details on the winner and other place getters see ShinyBIOMOD wins 2020 GBIF Ebbe Nielsen Challenge. Obviously I'm biased, but it's nice to see the challenge inspiring creativity in biodiversity informatics. Congratulations to everyone who took part.

My entry is live at I had a difficult time keeping it online over the summer due to meow attacks, but managed to sort that out. Turns out the cloud provider I used to host Elasticsearch switched from securing the server by default to making it completely open to anyone, and I'd missed that change.

Given that the project was a success, I'm tempted to revisit it and explore further ways to combine phylogenetic trees in a biodiversity browser.

Wednesday, September 23, 2020

Using the API to annotate PDFs

With somewhat depressing regularity I keep cycling back to things I was working on earlier but never quite get to work the way I wanted. The last couple of days it's the turn of

One of the things I'd like to have is a database of all taxonomic names such that if you clicked on a name you would get not only the bibliographic record for the publication where that name first appeared (which is what I've bene building for animals in BioNames) but also you could see the actual publication with the name highlighted in the text. This assumes that the publication has been digitised (say, as a PDF) and is accessible, but let's assume that this is the case. Now, we could do this manually, but we have tools to find taxonomic names in text. And in my use case I often know which page the name is on, and what the name is, so all I really want is to be able to highlight it programmatically (because I have millions of names to deal with).

So, time to revisit the API. One of the neat "tricks" have managed is the ability to annotate, say, a web page for an article and have that annotation automagically appear on the PDF version of the same article. As described in How Hypothesis interacts with document metadata this is in part because extracts metadata from the article's web page, such as DOI and link to the PDF, and stores that with the annotation (I say "in part" because the other part of the trick is to be able to locate annotations in different versions of the same text). If you annotate a PDF, stores the URL of the PDF and also a "fingerprint" of the PDF (see PDF Fingerprinting for details). This means that you can also add an annotation to a PDF offline (for example, on a file you have downloaded onto your computer) and - if has already encountered this PDF - that annotation will appear in the PDF online.

What I want to do is have a PDF, highlight the scientific name, upload that annotation to so that the annotation is visible online when anyone opens the PDF (and ideally when they look at the web version of the same article). I want to do this programmatically. Long story short, this seems doable. Here is an example annotation that I created and sent to via their API:

    "uri": "",
    "document": {
        "highwire": {
            "doi": [
        "dc": {
            "identifier": [
        "link": [
                "href": "urn:x-pdf:6124e7bdb33241429158b11a1b2c4ba5"
    "tags": [
    "target": [
            "source": "",
            "selector": [
                    "type": "TextQuoteSelector",
                    "exact": "Alpaida venger sp. nov.",
                    "prefix": "imens preserved in 75% ethanol. ",
                    "suffix": " (Figs 1-9) Type-material. Holot"
                    "type": "TextPositionSelector",
                    "start": 4834,
                    "end": 4857
    "user": "",
    "permissions": {
        "read": [
        "update": [
        "delete": [
        "admin": [

In this example the article and the PDF are linked by including the DOI and PDF fingerprint in the same annotation (thinking about this I should probably also have included the PDF URL in document.highwire.pdf_url[]). I extracted the PDF fingerprint using mutool and added that as the urn:x-pdf identifier.

The actual annotation itself is described twice, once using character position (start and end of the text string relative to the cleaned text extracted from the PDF) and once by including short fragments of text before and after the bit I want to highlight (Alpaida venger sp. nov.). In my limited experience so far this combination seems to provide enough information for to also locate the annotation in the HTML version of the article (if one exists).

You can see the result for yourself using the proxy ( Here is the annotation on the PDF (

and here is the annotation on HTML (

If you download the PDF onto your computer and open the file in Chrome you can also see the annotation in the PDF (to do this you will need to install the extension for Chrome and click the symbol on your Chrome's toolbar).

In summary, we have a pretty straightforward way to automatically annotate papers offline using just the PDF.

Friday, September 11, 2020

Darwin Core Million reminder, and thoughts on bad data

Bob mesibovThe following is a guest post by Bob Mesibov.

No winner yet in the second Darwin Core Million for 2020, but there are another two and a half weeks to go (to 30 September). For details of the contest see this iPhylo blog post. And please don’t submit a million RECORDS, just (roughly) a million DATA ITEMS. That’s about 20,000 records with 50 fields in the table, or about 50,000 records with 20 fields, or something arithmetically similar.

The purpose of the Darwin Core Million is to celebrate high-quality occurrence datasets. These are extraordinarily rare in biodiversity informatics.

I’ll unpick that. I’m not talking about the accuracy of the records. For most records, the “what”, “where”, “when” and “by whom” are probably correct. An occurrence record is a simple fact: Wilma Flintstone collected a flowering specimen of an Arizona Mountain Dandelion 5 miles SSE of Walker, California on 27 June 2019. More technically, she collected Agoseris parviflora at 38.4411 –119.4393, as recorded by her handheld GPS.

What could possibly go wrong in compiling a dataset of simple records like that in a spreadsheet or database? Let me count a few of the ways:

  • data items get misspelled or misnumbered
  • data items get put in the wrong field
  • data items are put in a field for which they are invalid or inappropriate
  • data items that should be entered get left out
  • data items get truncated
  • data items contain information better split into separate fields
  • data items contain line breaks
  • data items get corrupted by copying down in a spreadsheet
  • data items disagree with other data items in the same record
  • data items refer to unexplained entities (“habitat type A”)
  • paired data items don’t get paired (e.g. latitude but no longitude)
  • the same data item appears in different formats in different records
  • missing data items are represented by blanks, spaces, “?”, “na”, “-”, “unknown”, “not recorded” etc, all in the same data table
  • character encoding failures create gibberish, question marks and replacement characters (�)
  • weird control characters appear in data items, and parsing fails
  • dates get messed up (looking at you, Excel)
  • records get duplicated after minor edits

In previous blog posts (here and here) I’ve looked at explanations for poor-quality data at the project, institution and agency level — data sources I referred to collectively as the “PIA”. I don’t think any of those explanations are controversial. Here I’m going to be rude and insulting and say there are three further obstacles to creating good, usable and shareable occurrence data:

Datasets are compiled as though they were family heirlooms.

The PIA says “This database is OUR property. It’s for OUR use and WE understand the data, even if it’s messy and outsiders can’t figure out what we’ve done. Ambiguities? No problem, we’ll just email Old Fred. He retired a few years back but he knows the system back to front.”

Prising data items from these heirlooms, mapping them to new fields and cleaning them are complicated exercises best left to data specialists. That’s not what happens.

Datasets are too often compiled by people with inadequate computer skills. Their last experience of data management was building a spreadsheet in a “digital learning” class. They’re following instructions but they don’t understand them. Both the data enterers and their instructors are hoping for a good result, which is truly courageous optimism.

The (often huge) skills gap between the compilers of digital PIA data and the computer-savvy people who analyse and reformat/repackage the data (users and facilitators-for-users) could be narrowed programmatically, but isn’t. Hands up all those who use a spreadsheet for data entry by volunteers and have comprehensive validation rules for each of the fields? Thought so.

People confuse software with data. This isn’t a problem restricted to biodiversity informatics, and I’ve ranted about this issue elsewhere. The effect is that data compilers blame software for data problems and don’t accept responsibility for stuff-ups.

Sometimes that blaming is justified. As a data auditor I dread getting an Excel file, because I know without looking that the file will have usability and shareability issues on top of the usual spreadsheet errors. Excel isn’t an endpoint in a data-use pipeline, it’s a starting point and a particularly awful one.

Another horror is the export option. Want to convert your database of occurrence records to format X? Just go to the “Save as” or “Export data” menu item and click “OK”. Magic happens and you don’t need to check the exported file in format X to see that all is well. If all is not well, it’s the software’s fault, right? Not your problem.

In view of these and the previously blogged-about explanations for bad data, it’s a wonder that there are any high-quality datasets, but there are. I’ve audited them and it’s a shame that for ethical reasons I can’t enter them myself in the current Darwin Core Million.

Wednesday, August 26, 2020

Personal knowledge graphs: Obsidian, Roam, Wikidata, and Xanadu

I stumbled across this tweet yesterday (no doubt when I should have been doing other things), and disappeared down a rabbit hole. Emerging, I think the trip was worth it.


Markdown wikis

Among the tools listed by @zackfan01 were Obsidian and Roam, neither of which I heard of before. Both Obsidian and Roam are pitched as "note-taking" apps, they are essentially personal wikis where you write text in Markdown and use [[some text goes here]] to create links to other pages (very like a Wiki). Both highlight backlinks, that is, clearly displaying "what links here" on each page, making it easy to navigate around the graph you are creating by linking pages. Users of Obsidian share these graphs in Discord, rather like something from Martin MacInnes' novel "Gathering Evidence". Personal wikis have been around for a long time, but these apps are elegantly designed and seem fun to use. Looking at these apps I'm reminded of my earlier post Notes on collections, knowledge graphs, and Semantic Web browsers where I moaned about the lack of personal knowledge graphs that supported inference from linked data. I'm also reminded of the Blue Planet II, the BBC, and the Semantic Web: a tale of lessons forgotten and opportunities lost where I constructed an interactive tool to navigate BBC data on species and their ecology (you can see this live at, and the fun to be had from simply being able to navigate around a rich set of links. I imagine these Markdown-based wikis could be a great way further explore these ideas.


Personal and global knowledge graphs

Then I began thinking about what if the [[page links]] in these personal knowledge graphs were not just some text but, say, a Wikidata identifier (of the form "Qxxxxx")? Imagine that if you were writing notes on say, a species, you could insert the Wikidata Qid and you would get a pre-populated template that comes with some facts from Wikidata, and you could then use that as a starting point (see for example Toby Hudson Entity Explosion I discussed earlier). Knowing that more and more scholarly papers are being added to Wikidata, this means you could also add bibliographic citations as Qids, fetching all the necessary bibliographic information on the fly from Wikidata. So your personal knowledge graph intersects with the global graph.



Now, I've not used Roam, but anyone who has is likely to balk at my characterisation of it as "just" a Markdown wiki, because there's more going on here. The Roam white paper talks about making inferences from the text using reasoning or belief networks, although these features don't seem to have had much uptake. But what really struck me as I explored Roam was the notion of not just linking to pages using the [[ ]] syntax, but also linking to parts of pages (blocks) using (( )). In the demo of Roam there are various essays, such as Paul Graham's The Refragmentation, and each paragraph is an addressable block that can be cited independently of the entire essay. Likewise, you can see what pages cite that block.

 Now in a sense these are just like fragment identifiers that we can use to link to parts of a web page, but there's something more here because these fragments are not just locations in a bigger document, they are the components of the document.


This strikes me as rather like Ted Nelson's vision of Xanadu, where you could cite any text at any level of granularity, and that text would be incorporated into the document you were creating via transclusion (i.e., you don't include a copy of the text, you include the actual text). In the context of Roam, this means you have the entire text you want to cite included in the system, so you can then show chunks of it and build up a network of ideas around each chunk. This also means that the text being worked on becomes part of the system, rather than remaining isolated, say, as a PDF or other representation. This also got me thinking about the Plazi project, where taxonomic papers are being broken into component chunks (e.g., figures, taxonomic descriptions, etc.) and these are then being stored in various places and reassembled - rather like Frankenstein's monster - in new ways, for example in GBIF (e.g., or Species-ID (see doi:10.3897/zookeys.90.1369 ). One thing I've always found a little jarring about this approach is that you lose the context of the work that each component was taken from. Yes, you can find a link to the original work and go there, but what if you could seamlessly click on the paragraph or figure see them as part of the original article? Imagine we had all the taxonomic literature available in this way, so that we can cite any chunk, remix it (which is a key part of floras and other taxonomic monographs), but still retain the original context?


To come back full circle, in some ways tools like Obsidian and Roam are old hat, we've had wikis for a while, the idea of loading texts into wikis is old (e.g., Wikisource), backlinks are nothing new, etc. But there's something about seeing clean, elegant interpretations of these ideas, free of syntax junk, and accompanied by clear visions of how software can help us think. I'm not sure I will use either app, but they have given me a lot of food for thought.

Tuesday, August 25, 2020

Entity Explosion: bringing Wikidata to every website

A week ago Toby Hudson (@tobyhudson) released a very cool Chrome (and now Firefox) extension called Entity Explosion. If you install the extension, you get a little button you can press to find out what Wikidata knows about the entity on the web page you are looking at. The extension works on web sites that have URLs that match identifiers in Wikidata. For example, here it is showing some details for an article in BioStor ( The extension "knows" that this article is about the Polish arachnologist Wojciech Staręga.

But this is a tame example, see what fun Dario Taraborelli (@ReaderMeter) is having with Toby's extension:

There are some limitations. For instance, it requires that the web site URL matches the identifier, or more precisely the URL formatter for that identifier. In the case of BioStor the URL formatter URL is$1 where $1 is the BioStor identifier stored by Wikidata (e.g., 261148). So, if you visit the extension works as advertised.

However, identifiers that are redirects to other web sites, such as DOIs, aren't so lucky. A Wikidata item with a DOI (such as 10.1371/JOURNAL.PONE.0133602) corresponds to the URL, but if you click on that URL eventually you get taken to, which isn't the original DOI URL (incidently this is exactly how DOIs are supposed to work).

So, it would be nice if Entity Explosion would also read the HTML for the web page and attempt to extract the DOI from that page (for notes on this see<meta>_tag), which means it would work on even more webs sites for academic articles.

Meantime, if you use Chrome of Firefox as your browser, grab a copy and discover just how much information Wikidata has to offer.

Workshop On Open Citations And Open Scholarly Metadata 2020 talk

I'm giving a short talk at the Workshop On Open Citations And Open Scholarly Metadata 2020, which will be held online on September 9th. In the talk I touch on citation patterns in the taxonomic literature, the recent Zootaxa impact factor story, and mention a few projects I'm working on: To create the presentation I played around with mmhmm, which (pretty obviously) I still need to get the hang of...

Anyway, video below:


Friday, August 21, 2020

Taxonomic concepts: a possible way forward

Reading the GitHub issue Define objective rules for taxon concept identity referred to by Markus Döring in a comment on a previous post, I'm once again struck by the unholy mess generated by any discussion of "taxonomic concepts". The sense of déjà vu is overwhelming. What drives me to distraction is how little of this seems to be directed at solving actual problems that biologists have, which are typically things like "what does this random name that I've come across refer to?" and "will you please stop changing the damn names!?".

One thing that's also struck me is the importance of stable identifiers for species, that is, identifiers that are stable even in the face of name changes. If you have that, then you can talk about changes in classification, such as moving a species from one genus to another.

45759416 c9c5ed80 bc1f 11e8 98ca 5f4554ddca42

In the diagram above, the species being moved from Sasia to Verreauxia has the same identifier ("afrpic1") regardless of what genus it belongs to. This enables us to easily determine the differences between classifications (and then link those changes to the evidence supporting the change). I find it interesting that projects that manage large classifications, such as eBird and the Reptile database use stable species identifiers (either externally or internally). If you are going to deal with classifications that change over time you need stable identifiers.

So I'm beginning to think that perhaps the single most useful thing we could do as a taxonomic database community is to mint stable identifiers for each unique, original species name. These could be human readable, for example the species epithet plus author name plus year, suitably cleaned up (e.g., all lower case). So, our species could be "sapiens-linnaeus-1758". This sort of identifier is inspired by the notion of uninomial nomenclature:

If the uninomial system is not accepted, or until it is, I see no hope of ever arriving at a really stable nomenclature. - Hubbs (1930)

For more reading, see for example

  • Cantino, D. P., Bryant, H. N., Queiroz, K. D., Donoghue, M. J., Eriksson, T., Hillis, D. M., & Lee, M. S. Y. (1999). Species Names in Phylogenetic Nomenclature. Systematic Biology, 48(4), 790–807. doi:10.1080/106351599260012
  • Hubbs, C. L. (1930). SCIENTIFIC NAMES IN ZOOLOGY. Science, 71(1838), 317–319. doi:10.1126/science.71.1838.317
  • Lanham, U. (1965). Uninominal Nomenclature. Systematic Zoology, 14(2), 144. doi:10.2307/2411739
  • Michener, C. D. (1963). Some Future Developments in Taxonomy. Systematic Zoology, 12(4), 151. doi:10.2307/2411757
  • Michener, C. D. (1964). The Possible Use of Uninominal Nomenclature to Increase the Stability of Names in Biology. Systematic Zoology, 13(4), 182. doi:10.2307/2411777

Just to be clear I'm NOT advocating replacing binomial names with uninomial names (the references above are just to remind me about the topic), but approaches to developing uninomial names could be used to create simple, human-friendly identifiers. Oh, and hat tip to Geoff Read for the comment on an earlier post of mine that probably planted the seed that started me down this track.

So, imagine going to a web site and with the uninomial identifier being able to get the list of every variation on that name, including species names being in different genera (in other words, all the objective or homotypic synonyms of that name).

OK, nice, but what about taxa? Well the second thing I'd like to get is every (significant) use of that name, coupled with a references (i.e., a "usage"). These would include cases where the name is regarded as a synonym of another name. Given that each usage is dated (by the reference), we then have a timestamped record of the interpretation of taxa referred to by that name. Technically, what I envisage is that we are tracking nomenclatural types, that is, for a given species name we are returning every usage that refers to a taxon that includes the type specimen of that name.

We could imagine doing something trivial such as putting "/n/" before the identifier to retrieve all name variations, and "/t/" to retrieve all usages. One could have a suffix for a timestamp (e.g., "what was the state of play for this name in 1960?")

It seems that something like this would help cut through a lot of the noise around taxa. By itself, a list of names and references doesn't specify everything you might want to know about a taxon, but I suspect that some of the things taxonomists ask for (e.g., every circumscription, every set of "defining" characters, every pairwise relationship between every variation on a taxon's interpretation) are both unrealistic and probably not terribly useful.

For example, circumscriptions (defining a taxon by the set of things it includes) are often mentioned in discussions of taxon concepts, but in reality (at the species level) how many explicit circumscriptions do we have in the taxonomic literature? I'd argue that the circumscriptions that we do have the are the ones being generated by modern databases such as GBIF, iNaturalist, BOLD, and GenBank. These explicitly link specimens, photos, or sequences to a taxon (defined locally within that database, e.g. by an integer number), and in some cases are testable, e.g., BLAST a sequence to see if it falls in the same set of sequences. These databases have their own identifiers and notions of what comprises a taxon (e.g., based on community editing, automated clustering, etc.).

This approach of simple identifiers that link multiple name variations would support the name-based matching that is at the heart of matching records in different databases (despite the wailing that names ≠ taxa, this is fundamentally how we match things across databases). The availability of timestamp usages would enable us to view a classification at given point in time.

This needs to be fleshed out more, and I really want to explore the idea of edit scripts (or patch files) for comparing taxonomic classifications, and how we can use them to document the evidence for taxonomic changes. More to come...