DNA barcode browser

Motivated by the 2020 Ebbe Nielsen Challenge I've put together an interactive DNA barcode browser. The app is live at https://dna-barcode-browser.herokuapp.com.

A naturalist from the 19th century would find little in GBIF that they weren’t familiar with. We have species in a Linnean hierarchy, their distributions plotted on a map. This method of summarising data is appropriate to much of the data in GBIF, but impoverishes the display of sequence data such as barcodes. Given a set of DNA barcodes we can compute a phylogeny for those sequences, and gain evidence for taxonomic groups, intraspecific genetic structure, etc. So I wanted to see if it was possible to make simple tool to interactively explore barcode data. This means we need fast methods for searching for similar sequences, and building phylogenies. I've been experimenting with ways to do this for the last couple of years, but have only now managed to put something together. For more details, see the repository. There is also a quick introductory video.

Taxonomic concepts for dummies

[Work in progress]

The "dummy" in this case is me. I'm trying to make sense of how to model taxa, especially in the context of linked data, and projects such as Wikidata where there is uncertainty over just what a taxon in Wikidata actually represents. There is also ongoing work by the TDWG Taxon Names and Concepts Interest Group. This is all very rough and I'm still working on this, but here goes.

I'm going to assume that names and publications are fairly unproblematic. We have a "controlled thesaurus of scientific names" to use Franck Michel et al.'s phrase, provided by nomenclators, and we have publications. Both names and publications have identifiers.

Then we have "usages" or "instances", which are (name, publication) pairs. These can be of different level of precision, such as "this name appears in this book", or it "ppears on page 5", or in this paragraph, etc. I tend to view "instances" as like annotations, as if you highlighted a name in some text. Indeed, indexing the taxonomic literature and recording the location of each taxonomic name is essentially generating the set of all instances. We can have identifiers for each instance.

Then we can have relationships between instances. For example, in taxonomic publications you will often see a researcher list previous mentions of a name in other works, synonyms, etc. So we can have links between instances (which is one reason why they need identifiers).

OK, now comes the tricky bit. Up to now things are pretty simple, we have strings (names), we have sets of strings (publications), the location of strings in publications (usages/instances/annotations), and relationships between those instances (e.g., there may be a list of them on the same page in a publication). What about taxa?

It seems to me there are at least two different ways people model taxa in databases. The first is the classic Darwin Core spreadsheet style model of one unique name per row. If a name is "accepted" it has a parent, if it isn't "accepted" then it doesn't have a pointer to a parent, but does have a pointer to an accepted name:

If we treat each row as an instance (e.g., name plus original publication of that name), then the identifier for the instance is the same as the identifier for the taxon. A practical consequence of this is that if the name of a taxon changes, so does the identifier, therefore any data attached to that identifier can be orphaned (or, at least, needs to be reconnected) when there is taxonomic change.

In reality, the rows aren't really usages, they are simply names (mostly) without associated publications. We could be generous and model these as instances where we lack the publication, but basically we have rows of names and pointers between them. So taxa = accepted names. This is the model used by ITIS and GBIF.

Now, a different model is to wrap one or more instances into a set and call that a taxon, and the taxon points to an instance that is contains the currently accepted name. The taxon has a parent that is itself a taxon. This separates taxa and names, and has the practical consequence that we can have a taxon identifier that persists even if the taxonomic name changes. Hence data linked to the taxon remains linked even if the name changes. And every name applied to a taxon has the same taxon identifier. This makes it possible to track changes in classifications using the diff approach I've discussed earlier. This is the model used by (as far as I can make out) the Australian NSL (site seems perpetually offline), eBird, and Avibase, for example.

Wikidata has something very like GBIF, there is essentially no distinction between taxa and names. hence the identifiers associated with a Wikidata "taxon" can be identifiers for names (e.g., from IPNI) or for taxa (e.g., Avibase), with no clear distinction made between these rather different things.

So, what to do, and does this matter? Maybe part of the problem is the notion that identifiers for taxa should be unique, that is, only one Wikidata item can have an Avibase ID. This would work if a Wikidata taxon was, in fact, a taxon (and that it matched the Avibase classification). Perhaps we could treat Wikidata "taxa" as something like an instance, and label each Wikidata item that belongs in the same taxon (according to particular identifier being used) with the same taxon identifier?

More to come...

Persistent Identifiers: A demo and a rant

This morning, as part of a webinar on persistent identifiers, I gave a live demo of a little toy to demonstrate linking together museum and herbaria specimens with publications that use those specimens. A video of an earlier run through of the demo appears below, for background on this demo see Diddling with semantic data: linking natural history collections to the scientific literature. The slides I used in this demo are available here: http://pid-demonstrator.herokuapp.com/demo/.

One thing which struck me during this webinar is that discussions about persistent identifiers (PIDs) (also called "GUIDs") seem to endlessly cycle through the same topics, it is as if each community has to reinvent everything and rehash the same debates before they reach a solution. An alternative is to try and learn from systems that work.

In this respect CrossRef is a great example:

1. They have (mostly) brand neutral, actionable identifiers that resolve to something of value (DOIs resolve to an article you can read).
2. They are persistent by virtue of redirection (a DOI is a pointer to a pointer to the thing, bit like a P.O. Box number). The identifiers are managed.
3. They have machine readable identifiers that can be used to support an ecosystem of services (e.g., most references manages just need you to type in a DOI and they do the rest, altimetric “donuts", etc.).
4. They have tools for discoverability, in other words, if you have the metadata for an article they can tell you what the corresponding DOI is.
5. The identifiers deliver something of value to publishers that use them, such as the citation graph of interconnected articles (which means your article is automatically linked to other article) and you can get real time metrics of use (e.g., DOIs being cited in Wikipedia articles).
6. There is a strong incentive for publishers to use other publisher's identifiers (DOIs) in their own content because if that is reciprocated then you get web traffic (and citation counts). If publishers use their own local identifiers for external content they lose out on these benefits.
7. There are services to handle cases when things break. If a DOI doesn’t resolve, you can talk to a human who will attempt to fix it.

My point is that there is an entire ecosystem built around DOIs, and it works. Typically every community considering persistent identifiers attempts to build something themselves, ideally for “free", and ends up with a mess, or a system that doesn’t provide the expected benefits, because the message they got was “we need to get PIDs” rather than “build an infrastructure to enable these things that we need”.

I think we can also learn from systems that failed. In biology, LSIDs failed for a bunch of reasons, mainly because they didn’t resolve to anything useful (they were only machine readable). Also they were free, which seems great, except it means there is no cost to giving them up (which is exactly what people did). Every time someone advocates a PID that is free they are advocating a system that is designed to fail. Minting a UUID for everything costs nothing and is worth nothing. If you think a URL to a website in your organisation's domain is a persistent identifier, just think what happened to all those HTTP URLs stored in databases when post Edward Snowden the web switched to HTTPS.

One issue which came up in the Webinar was the status of ISBNs, which aren't actionable PIDs (in the sense that there's no obvious way to stick one in a web browser and get something back). ISBNs have failed to migrate to the web because, I suspect, they are commercially valuable, which means a fight over who exploits that value. Whoever provides the global resolver for ISBNs then gets to influence where you buy the corresponding book. Book publishers were slow to sell online, so Amazon gobbled up the online market, so in effect the defect global resolver (Amazon) makes all the money. The British Library got into trouble for exactly this reason when they provided links to Amazon (see British Library sparks Amazon row). Furthermore, unlike, DOIs for the scholarly literature, there aren’t really any network effects for ISBNs - publishers don’t benefit from other publishers having their content online. So I think ISBNs are an interesting example of the economics of identifiers, and the challenging problem of identifiers for things that are not specific to one organisation. It's much easier to think about identifiers for your stuff because you control that stuff and how it is represented. But who gets to decide on the PID for, say, Homo sapiens?

So, while we navigate the identifier acronym soup (DOI, LSID, ORCID, URL, URI, IRI, PURL, ARK, UUID, ISBN, Handles) and rehash arguments that multiple communities have been having for decades, maybe it's a good time to pause, take a look at other communities, and see what has worked and what hasn't, and why. It may well be that in many cases the kinds of drivers that make CrossRef a successes (identifiers return something of value, network effects as the citation graph grows, etc.) might not exist in many heritage situations, but that in itself would be useful to know, and might help explain why we have been sluggish in adopting persistent identifiers.

Darwin Core Million now twice a year

The following is a guest post by Bob Mesibov.

The first Darwin Core Million closed on 31 March with no winner. Since I'm seeing better datasets this year in my auditing work for Pensoft, I've decided to run the competition every six months.

Missed the first Darwin Core Million and don't know what it's about? Don't get too excited by the word "million". It refers to the number of data items in a Darwin Core occurrences table, not to the prize!

The rules

• Anyone can enter, but the competition applies only to publicly available Darwin Core occurrence datasets. These might have been uploaded to an aggregator, such as GBIF, ALA or iDigBio, or to an open-data repository.
• Select about one million data items from the dataset. That could be 50000 records in 20 populated Darwin Core fields, or 20000 records in 50 populated Darwin Core fields, or something in between. Email the dataset to me after 1 September and before 30 September as a zipped, plain-text file, together with a DOI or URL for the online version of the dataset.
• I'll audit datasets in the order I receive them. If I can't find serious data quality problems (see below) in your dataset, I'll pay your institution AUD$150 and declare your institution the winner of the Darwin Core Million here on iPhylo. There's only one winner in each competition round; datasets received after the first problem-free dataset won't be checked.
• If I find serious data quality problems, I'll let you know by email. If you want to learn what the problems are, I'll send you a report detailing what should be fixed and charge your institution AUD$150. At 0.3-0.75c/record, that's a bargain compared to commercial data-checking rates. And it would be really good to hear, later on, that those problems had indeed been fixed and that corrected data items had replaced the originals online.

How the data are judged

For a list of data quality problems, see this page in my Data Cleaner's Cookbook. The key problems I look for are:

• duplicate records
• invalid data items
• missing-but-expected items
• data items in the wrong fields
• data items inappropriate for their field
• truncated data items
• records with items in one field disagreeing with items in another
• character encoding errors
• wildly erroneous dates or coordinates
• incorrect or inconsistent formatting of dates, names and other items

This is not just nit-picking. Your digital data items aren't mainly for humans to read and interpret, they're intended in the first place for parsing and managing by computers. "Western Hill" might not be the same as "Western Hill" in processing, for example, because the second placename might have a no-break space between the words instead of a plain space. Another example: humans see these 22 variations on collector names as "the same", but computers don't.

Please also note that data quality isn't the same as data accuracy. Is Western Hill really at those coordinates? Is the specimen ID correct? Is the barely legible collector name on the specimen label correctly interpreted? These are questions about data accuracy. But it's possible to have entirely correct digital data that can't be processed by an application, or moved between applications, because the data suffer from one or more of the problems listed above.

Fine points

I think I'm pretty reasonable about the "serious" in "serious data quality problems". One character encoding error, such as "L'H?rit" repeated in the "scientificNameAuthorship" field, isn't serious, but multiple errors scattered through several fields are grounds for rejection.

For an understanding of "invalid", please refer to the Darwin Core field definitions and recommendations.

"Missing-but-expected" is important. I've seen GBIF mis-match a scientific name because the Darwin Core "kingdom" field was left blank by the data provider, even though all the other higher-taxon fields were filled in.

Please remember, entries received before 1 September won't be audited.

Zootaxa has no impact factor

So this happened:

Stunning misinterpretation by @Clarivate/@WebOfScience in their decision to suppress citation counts for all papers in @Zootaxa. This damages the entire field of zoological taxonomy, and in particular researchers from developing countries.

— Wayne Maddison (@WayneMaddison) July 2, 2020

Zootaxa is a hugely important journal in animal taxonomy:

Also do @clarivate metrics take into account size of the journal? Below is a chart of the top 20 journals ranked by number of papers publishing new names in zoology in 2018. Over half are in @zootaxa. If you are that big, surely self citation is inevitable? pic.twitter.com/Sot8YYlcZB

— Roderic Page (@rdmpage) July 3, 2020

On one hand one could argue that impact factor is a bad way to measure academic impact, so it's tempting to say this simply reflects a poor metric that is controlled by a commercial company using data that is not open. But it quickly became clear on Taxacom that in some countries the impact factor of the journal you publish in is very important (to the point where it has a major effect on your personal income, and whether it is financially possible for you to continue your career). This discussion got rather heated, but it was quite eye opening to someone like me who casually dismisses impact factor as not interesting.

Really interesting thread on @taxacom about @clarivate removing @Zootaxa impact factor and what that means for taxonomists in different countries https://t.co/IWqK2x1aCq pic.twitter.com/bnupuwzeoe

— Roderic Page (@rdmpage) July 8, 2020

Partly in response to this I spent a little time scraping the Zootaxa web site to put together a list of all the literature cited by articles published in Zootaxa. Normally this sort of thing dies a lingering death on my hard drive, but this time I've got myself more organised and created a GitHub project for the code and I've uploaded the data to Figshare doi:10.6084/m9.figshare.c.5054372.v1. Regardless of the impact factor issue, it's potentially a fascinating window into centuries of taxonomic publications.

Lists of species don't matter: thoughts on "Principles for creating a single authoritative list of the world’s species"

Garnett et al. recently published a paper in PLoS Biology that starts with the sentence "Lists of species matter":

Garnett, S. T., Christidis, L., Conix, S., Costello, M. J., Zachos, F. E., Bánki, O. S., … Thiele, K. R. (2020). Principles for creating a single authoritative list of the world’s species. PLOS Biology, 18(7), e3000736. doi:10.1371/journal.pbio.3000736

This paper (one of a forthcoming series) is pretty much the kind of paper I try and avoid reading. It has lots of authors so it is a paper by committee, those authors all have a stake in particular projects, and it is an acronym soup of organisations the paper is pitched at. It's a well-worn strategy: write one or more papers outlining making the case that there is a problem, then get funding based on the notion that clearly there's a problem (you've published papers saying so) and that you and your co-applicants are best placed to solve it (clearly, because you wrote the papers identifying the problem in the first place). I'm not criticising the strategy, it's how you get things done in science. It just makes for a rather uninspiring read.

From my perspective focussing on "lists" is a mistake. Lists don't really matter, it is what is on the list that counts. And I think this is where the real prize is. As I play with Wikidata I'm becoming increasingly aware of the clusterfuck mess the taxonomic database community has created by conflating taxonomic names with taxa, and by having multiple identifiers for the same things. We survive this mess by relying on taxonomic names as somewhat fuzzy identifiers, and the hope that we can communicate successfully with other people despite this ambiguity (I guess this is pretty much the basis of all communication). As Roger Hyam notes:

These taxon names we are dealing with are really just social tags that need to be organised in a central place.

Having lots of names (tags) is fine, and Wikidata is busy harvesting all these taxonomic names and their identifiers (ITIS, IPNI, NCBI, EOL, iNaturalist, eBird, etc., etc., etc.). For most of these names all we have is a mapping to other identifiers for the same name, a link to a parent taxon, and sometimes a link to a reference for the name. But what happens if we want to attach data to a taxon? Take, for example, the African Piculet Verreauxia africana. This bird has at least two scientific names, each with a separate entry in Wikidata: Verreauxia africana Q28123873 and Sasia africana Q1266812. These are the same species yet it has two entries in Wikidata. If I want to add, say, body weight, or population size, or longevity, which Wikidata item do I add that data too?

What we need is an identifier for the species, an identifier that remains unchanged even if the name changes, or if that species moves in the taxonomic hierarchy. Some databases do this already. For example the eBird identifier for Verreauxia africana/Sasia africana is afrpic1. Because the identifier remains unchanged we can do things such as "diffs" between successive classifications showing how the species has moved between different genera (see Taxonomic publications as patch files and the notion of taxonomic concepts):

Ironically it seems that for birds the common name (in this case "African Piculet") is a more stable identifier than the scientific name (although that may well change). By having stable taxon identifiers we can then decide what entity to attach biological data to. Taxonomic names have failed to do this, but are still vital as well known tags. The actual taxon identifiers should be opaque identifiers (like "afrpic1" - not really opaque but close enough - or Avibase's C4DFB5E31495AE94). Make each opaque identifier a DOI, use existing taxonomic names as formalised tags so we aren't disconnected from the literature, use timestamped versions to track changes in species classification over time, and we have something useful.

This, I think, is the real prize. Rather than frame the task as making a list of species so that organisations can have a checklist they can all share, why not frame it as providing a framework that we can hang trait data on? We have vast quantities of data residing in siloed databases, spreadsheets, and centuries of biological literature. The argument shouldn't be about what is on a list, it should be how we group that information together and enable people to do their science. By providing stable identifiers that are resistant to name changes we can confidently associate trait data with taxa. Taxonomy could then actually be what it should be, the organisational framework for biological information (see Taxonomy as Information Science).

Diddling with semantic data: linking natural history collections to the scientific literature

A couple of weeks ago I was grumpy on the Internet (no, really) and complained about museum websites and how their pages often lacked vital metadata tags (such as rel=canonical or Facebook Open Graph tags). This got a response:

If I was bitter I'd launch into a rant about how come people who diddle with semantic data fail to appreciate that the things which support their particular interests have to be balanced with other priorities by people maintaining production systems on a skeleton staff 😉 (1/3)

— vsmithuk (@vsmithuk) June 19, 2020

Vince's lovely line "diddle with semantic data" is the inspiration for the title of this post, in which I describe a tool to display links across datasets, such as museum specimens and scientific publications. This tool builds on ideas I discussed in 2014(!) (Rethinking annotating biodiversity data, see also "Towards a biodiversity knowledge graph" doi:10.3897/rio.2.e8767).

TL;DR;

If you want the quick summary, here it is. If we have persistent identifiers (PIDs) for specimens and publications (or anything other entities of interest), and we have a databases of links between pairs of PIDs (e.g., paper x mentions specimen y), and both entities have web pages, then we can display that relationship on both web pages using a Javascript bookmarklet. We can do this without permission, in the sense that the specimen web page can be hosted by a museum (e.g., The Natural History Museum in London) and the publication hosted by a publisher (e.g., The Public Library of Science), and neither organisation need know about the connection between specimen and publication. But because we do, we can add that connection. (Under the hood this approach relies on a triplestore that stores the relationships between pairs of entities using the Web Annotation Data Model.)

Background

Consider the web page https://data.nhm.ac.uk/object/6e8be646-486e-4193-ac46-e13e23c5daef which is for a specimen of the cestode Gangesia agraensis in the NHM (catalogue number 2012.8.23.3). If you look at this page the information content is pretty minimal, which is typical of many natural history specimens. In particular, we have no idea if anyone has done anything with this specimen. Has it been sequenced, imaged, or mentioned in a publication? Who knows? We have no indication of the scientific importance or otherwise of this specimen.

Now, consider the page https://doi.org/10.1371/journal.pone.0046421 for the PLoS ONE paper Revision of Gangesia (Cestoda: Proteocephalidea) in the Indomalayan Region: Morphology, Molecules and Surface Ultrastructure. This paper has most of bells and whistles of a modern paper, including metrics of attention. However, despite this paper using specimens from the NHM there is no connection between the the paper and the museum's collections.

Making these connections is going to be important for tracking the provenance of knowledge based on those specimens, as well as developing metrics of collection use. Some natural collection web sites have started to show these sort of links, but we need them to be available on a much larger scale, and the links need to be accessible not just on museum sites but everywhere specimens are used.  Nor is this issue restricted to natural history collections. My use of "PIDs" in this blog post (as opposed, say, to GUIDs) is that part of the motivation for this work is my participation in the Towards a National Collection - HeritagePIDs project (@HeritagePIDs), whose scope includes collections and archives from nay different fields.

Magic happens

The screenshots below show the same web pages as before, but now we have a overlay window that displays additional information. For specimen 2012.8.23.3 we see a paper (together with a list of the authors, each sporting an ORCID). This is the PloS ONE paper, which cites this specimen.

Likewise if we go to the PLoS ONE paper, we now see a list of specimens from the NHM that are mentioned in that paper.

What happened?

The overlay is generated by a bookmarklet, a piece of Javascript that displays an overlay on the right hand side of the web page, then does two things:

1. It reads the web page to find out what the page is "about" (the main entity). It does this by looking for tags such as rel="canonical", og:url, or a meta tag with a DOI. It turns out that lots of relevant sites don't include a machine readable way of saying what they are about (which led to my tweet that annoyed Vince Smith, see above). While it may be "obvious" to a human what a site is about, we need to spell that out for computers. The easy way to do this is explicitly include a URL or other persistent identifier for the subject of the web page.
2. Once the script has figured out what the page is about, it then talks to a triple store that I have set up and asks "do you have any annotations for this thing?". If so, they are returned as a DataFeed (basically a JSON-LD variant of an RSS feed) and the results are displayed in the overlay.

Step one hinges on the entity of interest having a persistent identifier, and that identifier being easy to locate in the web page. Academic publishers are pretty good at doing this, mainly because it increases their visibility to search engines such as Google Scholar, and also it helps reference managers such as Zotero automatically extract bibliographic data for a paper. These drivers don't exist for many types of data (such as specimens, or DNA sequences, or people), and so often those sites will need custom code to extract at the corresponding identifier.

Step two requires that we have a database somewhere that knows whether two things are linked. For various reasons I've settled on using a triplestore for this data, and I'm modelling the connection between two things as an annotation. Below is the (simplified) JSON-LD for an annotation linking the NHM specimen 2012.8.23.3 to the paper Revision of Gangesia (Cestoda: Proteocephalidea) in ... .

{
  "type": "Annotation",
  "body": {
 "id": "https://data.nhm.ac.uk/object/6e8be646-486e-4193-ac46-e13e23c5daef",
 "name": "2012.8.23.3"
  },
  "target": {
 "name": "Revision of Gangesia (Cestoda: Proteocephalidea) in ...",
 "canonical": "https://doi.org/10.1371/journal.pone.0046421"
  }
}

Strictly speaking we could have something even more minimal:

{
  "type": "Annotation",
  "body": "https://data.nhm.ac.uk/object/6e8be646-486e-4193-ac46-e13e23c5daef",
  "target": "https://doi.org/10.1371/journal.pone.0046421"
}

But this means we couldn't display the names of the specimen and the paper in the overlay. (The use of canonical in the target is preparation for when annotations will be made on specific representations, such as a PDF of a paper, the same paper in HTML, etc. and I want to be able to group those together.)

Leaving aside these technical details, the key thing is that we have a simple way to link two things together.

Where do the links come from?

Now we hit the $64,000 Question, how do we know that specimen https://data.nhm.ac.uk/object/6e8be646-486e-4193-ac46-e13e23c5daef and paper https://doi.org/10.1371/journal.pone.0046421 are linked? To do that we need to text mine papers looking for specimen codes (like 2012.8.23.3), discover the persistent identifier that corresponds to that code, then combine that with the persistent identifier for the entity that refers to that specimen (such as a paper, a supplementary data file, or a DNA sequence).

For this example I'm spared that work because Ross Mounce (@rmounce) and Aime Rankin (@AimeRankin) did exactly that for some NHM specimens (see doi:10.5281/zenodo.34966 and https://github.com/rossmounce/NHM-specimens). So I just wrote a script to parse a CSV file and output specimen and publication identifiers as annotations. So that I can display more I also grabbed RDF for the specimens, publications, and people. The RDF for the NHM specimens is available by simply appending an extension (such as .jsonld) to the specimen URL, you can get RDF for people and their papers from ORCID (and other sources).

As an aside, I could use Ross and Aime's work "as is" because the persistent identifiers had changed (sigh). The NHM has changed specimen URLs (replacing /specimen/ with /object/) and switched from http to https. Even the DOIs have changed in that the HTTP resolver http://dx.doi.org has now been replaced by https://doi.org. So I had to fix that. If you want this stuff to work DO NOT EVER CHANGE IDENTIFIERS!

How can I get this bookmarklet thingy?

To install the bookmarklet go to https://pid-demonstrator.herokuapp.com and click and hold the "Annotate It!" Link, then drag it to your web browser toolbar (on Safari it's the "Favourites Bar", on Chrome and Firefox it's the "Bookmarks Bar"). When you are looking at a web page click "Annotate It!". At the moment the NHM PLoS example above is the only one that does anything interesting, this will change as I add more data.

My horribly rough code is here: https://github.com/rdmpage/pid-demonstrator.

What's next?

The annotation model doesn't just apply to specimens. For example, I'd love to be able to flag pages in BHL as being of special interest, such as "this page is the original description of this species"). This presents some additional challenges because the user can scroll through BHL and change the page, so I would need the bookmarklet to be aware of that and query the triplestore for each new page. I've got the first part of the code working, in that if you try the bookmarklets on a BHL page it "knows" when you've scrolled to a different page.

I obviously need to populate the triplestore with a lot more annotations. I think the simplest way forward is just to have spreadsheets (e.g., CSV files) with columns of specimen identifiers and DOI and convert those into annotations.

Lastly, another source of annotations are those made by readers using tools such as hypothes.is, which I've explored earlier (see Aggregating annotations on the scientific literature: a followup on the ReCon16 hackday). So we can imagine a mix of annotations made by machine, and annotations made by people, both helping construct a part of the biodiversity knowledge graph. This same graph can then be used to explore the connections between specimens and publications, and perhaps lead to metrics of scientific engagement with natural history collections.