Liberating links between datasets using lightweight data publishing: an example using IPNI and the taxonomic literature

I've written a short paper entitled "Liberating links between datasets using lightweight data publishing: an example using plant names and the taxonomic literature" (phew) and put a preprint on bioRxiv (https://doi.org/10.1101/343996) while I figure out where to publish it. Here's the abstract:

Constructing a biodiversity knowledge graph will require making millions of cross links between diversity entities in different datasets. Researchers trying to bootstrap the growth of the biodiversity knowledge graph by constructing databases of links between these entities lack obvious ways to publish these sets of links. One appealing and lightweight approach is to create a "datasette", a database that is wrapped together with a simple web server that enables users to query the data. Datasettes can be packaged into Docker containers and hosted online with minimal effort. This approach is illustrated using a dataset of links between globally unique identifiers for plant taxonomic names, and identifiers for the taxonomic articles that published those names.

In some ways the paper is simply a record of me trying to figure out how to publish a project that I've been working on for several years, namely linking names from BioNames. The preprint discusses various options, before settling on "datasettes", which is a nice method developed by Simon Willison (@simonw) to wrap up simple databases with their own web server and query API and make them accessible on the web. These can run on a local machine, or be packaged up as a Docker container, which is what I've done. You play with the database here: https://ipni.sloppy.zone. If this link is offline, then you can grab the container here https://hub.docker.com/r/rdmpage/ipni/ and run it yourself. If, like me, you're new to Docker, then I recommend grabbing a copy of Kitematic.

The datasette interface is simple but gives you lots of freedom to explore the data.

For example, you have ability to query the data using SQL, e.g.:

One advantage of this approach is that the data is more accessible. I could just dump the database somewhere but then you'd have to download a large file and figure out how query it. This way, you can play with it straight away. It also means people can make use of it before I make up my mind how best to package it (for example, as part of a larger database of eukaryote names). This is one of the main motivations behind the paper, how to avoid the trap of spending years cleaning and augmenting data and not making it available to others because of the overhead of building a web site around the data. I may look at liberating some other datasets using this approach.

Blue Planet II, the BBC, and the Semantic Web: a tale of lessons forgotten and opportunities lost

David Attenborough’s latest homage to biodiversity, Blue Planet II is, as always, visually magnificent. Much of its impact derives from the new views of life afforded by technological advances in cameras, drones, diving gear, and submersibles. One might hope that the supporting information online reflected the equivalent technological advances made in describing and sharing information. Sadly, this is not the case. Instead the BBC offers a web site with a video clips and a poster... a $%@£ poster.

This is a huge missed opportunity. Where do people go to learn more about the organisms featured in an episode? How do we discover related content on the BBC and elsewhere? How do we discover the science underpinning each episode that has been so exquisitely filmed and edited?

Perhaps the lack of an online resource reflects a lack of resources, or expertise? Yet one look at the series (and the "Into the blue" epilogues) tells us that resources are hardly limiting. Furthermore, the BBC has previously constructed rich, informative web sites to support natural history programming. The now deprecated BBC Nature Wildlife site had an extensive series of web pages for the organisms featured in BBC programmes, with links to individual clips. For each organism the corresponding web page listed key traits such as behaviours, habitats, and geographic distribution, and each of these traits had its own web page list all organisms with those traits (see, for example the page for Steller's Sea Eagle).

Underlying all this information was a simple vocabulary (the Wildlife Ontology), and the entire corpus is also available in RDF: in other words, the BBC used Semantic Web technologies to structure this information. To get this data you simply append ".rdf" to the URL for a web page. For example, below is the RDF for Steller's Sea Eagle. It is not pretty, but it is a great example of machine-readable data which enables all sorts of interesting things to be built.

<?xml version="1.0" encoding="utf-8"?>
<rdf:RDF
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:dc="http://purl.org/dc/terms/"
xmlns:dctypes="http://purl.org/dc/dcmitype/"
xmlns:skos="http://www.w3.org/2004/02/skos/core#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:po="http://purl.org/ontology/po/"
xmlns:wo="http://purl.org/ontology/wo/">
<rdf:Description rdf:about="/nature/species/Steller's_Sea_Eagle">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
<rdfs:seeAlso rdf:resource="/nature/species"/>
</rdf:Description>
<wo:Species rdf:about="/nature/life/Steller's_Sea_Eagle#species">
<rdfs:label>Steller's sea eagle</rdfs:label>
<wo:name rdf:resource="http://www.bbc.co.uk/nature/species/Steller's_Sea_Eagle#name"/>
<foaf:depiction rdf:resource="http://ichef.bbci.co.uk/naturelibrary/images/ic/640x360/s/st/stellers_sea_eagle/stellers_sea_eagle_1.jpg"/>
<dc:description>Steller’s sea eagles are native to eastern Russia, inhabiting coastal cliffs and estuaries where they can easily access good fishing territories. They feed primarily on salmon, which they catch by swooping from perches located by the water's edge. Pairs are monogamous and hatch an average of two chicks each season, although crows and martens commonly take both eggs and young birds from the nest. During winter a small number of birds remain in Russia to tough it out, but the majority fly south to Japan.</dc:description>
<owl:sameAs rdf:resource="http://dbpedia.org/resource/Steller's_Sea_Eagle"/>
<wo:adaptation rdf:resource="/nature/adaptations/Altricial#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Animal_migration#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Carnivore#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Flight#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Hearing_(sense)#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Monogamous_pairing_in_animals#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Oviparity#adaptation"/>
<wo:adaptation rdf:resource="/nature/adaptations/Parental_investment#adaptation"/>
<wo:livesIn rdf:resource="/nature/habitats/Coast#habitat"/>
<wo:livesIn rdf:resource="/nature/habitats/Estuary#habitat"/>
<wo:livesIn rdf:resource="/nature/habitats/Marsh#habitat"/>
<wo:livesIn rdf:resource="/nature/habitats/River#habitat"/>
<wo:livesIn rdf:resource="/nature/habitats/Swamp#habitat"/>
<wo:genus rdf:resource="/nature/life/Sea_eagle#genus"/>
<wo:family rdf:resource="/nature/life/Accipitridae#family"/>
<wo:order rdf:resource="/nature/life/Falconiformes#order"/>
<wo:class rdf:resource="/nature/life/Bird#class"/>
<wo:phylum rdf:resource="/nature/life/Chordate#phylum"/>
<wo:kingdom rdf:resource="/nature/life/Animal#kingdom"/>
</wo:Species>
<wo:TaxonName rdf:about="/nature/species/Steller's_Sea_Eagle#name">
<rdfs:label>Haliaeetus pelagicus</rdfs:label>
<wo:commonName>Steller's sea eagle</wo:commonName>
<wo:scientificName>pelagicuspelagicus</wo:scientificName>
<wo:kingdomName>animalia</wo:kingdomName>
<wo:phylumName>Chordata</wo:phylumName>
<wo:className>Aves</wo:className>
<wo:orderName>Falconiformes</wo:orderName>
<wo:familyName>Accipitridae</wo:familyName>
<wo:genusName>Haliaeetus</wo:genusName>
<wo:speciesName>pelagicus</wo:speciesName>
</wo:TaxonName>
<foaf:Image rdf:about="http://ichef.bbci.co.uk/naturelibrary/images/ic/640x360/s/st/stellers_sea_eagle/stellers_sea_eagle_1.jpg">
<foaf:depicts rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
<foaf:thumbnail rdf:resource="http://ichef.bbci.co.uk/naturelibrary/images/ic/83x104/s/st/stellers_sea_eagle/stellers_sea_eagle_1.jpg"/>
</foaf:Image>
<po:Clip rdf:about="http://www.bbc.co.uk/programmes/p00dhn1t#programme">
<dc:title>Lunch on the wing</dc:title>
<po:subject rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</po:Clip>
<po:Clip rdf:about="http://www.bbc.co.uk/programmes/p00382f5#programme">
<dc:title>Steller's sea eagle</dc:title>
<po:subject rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</po:Clip>
<dctypes:Sound rdf:about="http://downloads.bbc.co.uk/earth/naturelibrary/assets/s/st/stellers_sea_eagle/5015017.mp3">
<dc:title>Calls from Steller's and white-tailed sea eagles</dc:title>
<dc:subject rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</dctypes:Sound>
<foaf:Document rdf:about="http://en.wikipedia.org/wiki/Steller's_Sea_Eagle">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://animaldiversity.ummz.umich.edu/site/accounts/information/Haliaeetus_pelagicus.html">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://www.arkive.org/stellers-sea-eagle/haliaeetus-pelagicus/">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://www.birdlife.org/datazone/species/index.html?action=SpcHTMDetails.asp&sid=3366&m=0">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://www.flickr.com/search/show/?q=steller+sea+eagle&s=int">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://www.iucnredlist.org/details/144342/0">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<foaf:Document rdf:about="http://www.natural-research.org/index.php?cID=169">
<foaf:primaryTopic rdf:resource="/nature/species/Steller's_Sea_Eagle#species"/>
</foaf:Document>
<wo:ReproductionStrategy rdf:about="/nature/adaptations/Altricial#adaptation">
<rdfs:label>Helpless young</rdfs:label>
</wo:ReproductionStrategy>
<wo:SurvivalStrategy rdf:about="/nature/adaptations/Animal_migration#adaptation">
<rdfs:label>Migration</rdfs:label>
</wo:SurvivalStrategy>
<wo:FeedingHabit rdf:about="/nature/adaptations/Carnivore#adaptation">
<rdfs:label>Carnivorous</rdfs:label>
</wo:FeedingHabit>
<wo:LocomotionAdaptation rdf:about="/nature/adaptations/Flight#adaptation">
<rdfs:label>Adapted to flying</rdfs:label>
</wo:LocomotionAdaptation>
<wo:CommunicationAdaptation rdf:about="/nature/adaptations/Hearing_(sense)#adaptation">
<rdfs:label>Acoustic communication</rdfs:label>
</wo:CommunicationAdaptation>
<wo:ReproductionStrategy rdf:about="/nature/adaptations/Monogamous_pairing_in_animals#adaptation">
<rdfs:label>Monogamous</rdfs:label>
</wo:ReproductionStrategy>
<wo:ReproductionStrategy rdf:about="/nature/adaptations/Oviparity#adaptation">
<rdfs:label>Egg layer</rdfs:label>
</wo:ReproductionStrategy>
<wo:LifeCycle rdf:about="/nature/adaptations/Parental_investment#adaptation">
<rdfs:label>Parental investment</rdfs:label>
</wo:LifeCycle>
<wo:TerrestrialHabitat rdf:about="/nature/habitats/Coast#habitat">
<rdfs:label>Coastal</rdfs:label>
</wo:TerrestrialHabitat>
<wo:MarineHabitat rdf:about="/nature/habitats/Estuary#habitat">
<rdfs:label>Estuaries</rdfs:label>
</wo:MarineHabitat>
<wo:FreshwaterHabitat rdf:about="/nature/habitats/Marsh#habitat">
<rdfs:label>Marsh</rdfs:label>
</wo:FreshwaterHabitat>
<wo:FreshwaterHabitat rdf:about="/nature/habitats/River#habitat">
<rdfs:label>Rivers and streams</rdfs:label>
</wo:FreshwaterHabitat>
<wo:FreshwaterHabitat rdf:about="/nature/habitats/Swamp#habitat">
<rdfs:label>Swamp</rdfs:label>
</wo:FreshwaterHabitat>
<wo:Genus rdf:about="/nature/genus/Sea_eagle#genus">
<rdfs:label>Haliaeetus</rdfs:label>
<wo:species rdf:resource="/nature/life/Steller's_Sea_Eagle#species"/>
<wo:species rdf:resource="/nature/life/African_Fish_Eagle#species"/>
<wo:species rdf:resource="/nature/life/White-tailed_Eagle#species"/>
</wo:Genus>
<wo:Family rdf:about="/nature/family/Accipitridae#family">
<rdfs:label>Accipitridae</rdfs:label>
</wo:Family>
<wo:Order rdf:about="/nature/order/Falconiformes#order">
<rdfs:label>Falconiformes</rdfs:label>
</wo:Order>
<wo:Class rdf:about="/nature/class/Bird#class">
<rdfs:label>Aves</rdfs:label>
</wo:Class>
<wo:Phylum rdf:about="/nature/phylum/Chordate#phylum">
<rdfs:label>Chordata</rdfs:label>
</wo:Phylum>
<wo:Kingdom rdf:about="/nature/kingdom/Animal#kingdom">
<rdfs:label>animalia</rdfs:label>
</wo:Kingdom>
</rdf:RDF>

For some reason, this web site is now deprecated. As an exercise I grabbed the RDF from the web site, did a little cleaning, and merged it together resulting in a set of around 94,500 triples (statements of the form “subject”, “predicate”, “object”). For example, this triple says that Steller's Sea Eagle is monogamous.

[/nature/life/Steller's_Sea_Eagle#species,
wo:adaptation,
/nature/adaptations/Monogamous_pairing_in_animals#adaptation]

One reason the Semantic Web has struggled to gain widespread adoption is the long list of things you need to get to the point where it is usable. You need data consistently structured using the same vocabulary. You need identifiers that everyone agrees on (or at least can map their own identifiers too). And you need a triple store, which is essentially a graph database, a technology that is still unfamiliar to many. But in this case the BBC has done a lot of the hard work by cleverly minting identifiers based on Wikipedia URLs (”slugs”), and developing a vocabulary to express relationships between organisms, traits, and habitats. All that’s needed is a way to query this data. Rather than use a triple store (most of which are not much fun to install or maintain) I’ve used the delightfully simple approach of employing a Hexastore. Hexastores provide fast querying of graphs by indexing all six permutations of the subject, predicates, object triple (hence “hexa”). The approach is sufficiently simple that for moderately sized databases we can implement it in Javascript and run it in a web browser.

As a demonstration, I created a very crude hexastore-based version of the BBC pages (https://rdmpage.github.io/bbc-wildlife/www/.

Once you load the page there are no further server requests, other than fetching images. Every query is “live” but takes place in the browser. You can click on the image for a species and get some textural information, as well as images representing traits of that organism. Click on a trait and you discover what organisms share those traits. This example is trivial, but surprisingly rich. I’ve found it fascinating to simply bounce around the images discovering unexpected facts about different species. There’s lots of potential for serendipitous discovery, as well as an enhanced appreciation for just how rich the BBC’s content is. If the Encyclopedia of Life were this engaging I’d be it’s biggest fan.

The question then, is why a similar approach was not taken for Blue Planet II? It can’t be a lack of resources, this series has amazing production values. And yet a wonderful opportunity has been missed. Why not build on the existing work and create an interactive resource that encourages people to explore more deeply and learn more? Much of the existing data could be used, as well as adding all the new species and behaviours we see on our TV screens. Blue Planet also highlights the impacts humans are having on the marine environment, these could be added as categories as well to show wat organisms are susceptible to different impacted (e.g., plastics).

That the BBC thinks a poster is an adequate for of engagement in the digital age speaks of a corporation that, in spite of many triumphs in the digital sphere (e.g., iPlayer) has not fully grasped the role the web can play in making its content more widely useful and relevant, beyond enthralling viewers on a Sunday evening. It also seems oblivious to the fact that it already knows how to deliver rich, informative online content (as evidenced by the now deprecated Wildlife application). So please, BBC, can we have a resource that enables us to learn more about the organisms and habitats that are the subjects of the grandeur and beauty we see on our TV screens?

Follow up

Below is some of the discussion this post generated on Twitter.

Very cool, your hexastore page has now become a great tool, using it for my revision now

— Tiê Caribe Da Rocha (@Whuky) December 6, 2017

Also going to contradict this view slightly (since the /programmes proposition sits in my portfolio). There's a lot more than a poster, for a start: https://t.co/xbHwsaCUHO and there are some assumptions about resource and available content that may not be as simple as suggested

— Ian Davies (@ianjamesdavies) December 5, 2017

“For some reason, this web site is now deprecated.” “That the BBC thinks a poster is an adequate for (sic) of engagement in the digital age” Have a look at this item, especially the comment by George Osborne at the bottom. https://t.co/OmHiYienzB

— John Waterworth (@jwaterworth) December 5, 2017

I took part in some SW pilots several years ago. The problem is the need to reach critical mass, particularly at a time when budgets are dropping. Public-facing providers like BBC and OU will always prioritise reach: anything additional needs to be off-the shelf to get adoption.

— Dr. Chris A Williams (@OU_Williams) December 5, 2017

Maybe I was influenced by your bio. I think you could look at external drivers for reasons some things were withdrawn from. The organisation is not completely free to make its own choices

— Mark Brandon (@icey_mark) December 6, 2017

Meanwhile, in @Wikidata, we're just quietly getting on with building that (and much more besides) / @fantasticlife @kai_en_ong @rdmpage @BBCEarth https://t.co/HKqFqmX3pJ

— Andy Mabbett (@pigsonthewing) December 6, 2017

TDWG 2017: thoughts on day 1

Some random notes on the first day of TDWG 2017. First off, great organisation with the first usable conference calendar app that I've seen (https://tdwg2017.sched.com).

I gave the day's keynote address in the morning (slides below).

Towards a biodiversity knowledge graph from Roderic Page

It was something of a stream of consciousness brain dump, and tried to cover a lot of (maybe too much) stuff. Among the topics I covered were Holly Bik's appeal for better links between genomic and taxonomic data, my iSpecies tool, some snarky comments on the Semantic Web (and an assertion that the reason that GenBank succeeded was due more to network effects than journals requiring authors to submit sequences there), a brief discussion of Wikidata (including using d3sparql to display classifications, see here), and the use of Hexastore to query data from BBC Wildlife. I also talked about Ted Nelson, Xanadu, using hypothes.is to annotate scientific papers (see Aggregating annotations on the scientific literature: a followup on the ReCon16 hackday), social factors in building knowledge graphs (touching on ORCID and some of the work by Nico Franz discussed here), and ended with some cautionary comments on the potential misuse of metrics based on knowledge graphs (using "league tables" of cited specimens, see GBIF specimens in BioStor: who are the top ten museums with citable specimens?).

TDWG is a great opportunity to find out what is going on in biodiversity informatics, and also to get a sense of where the problems are. For example, sitting through the Financial Models for Sustaining Biodiversity Informatics Products session you couldn't help being struck by (a) the number of different projects all essentially managing specimen data, and (b) the struggle they all face to obtain funding. If this was a commercial market there would be some pretty drastic consolidation happening. It also highlights the difficulty of providing services to a community that doesn't have much money.

I was also struck by Andrew Bentley's talk Interoperability, Attribution, and Value in the Web of Natural History Museum Data. In a series of slides Andrew outlined what he felt collections needed from aggregators, researchers, and publishers, e.g.:

What do collections want from aggregators like @GBIF ? #tdwg17 pic.twitter.com/WRmeafSbtv

— Roderic Page (@rdmpage) October 2, 2017

Chatting to Andrew at the evening event at the Canadian Museum of Nature, I think there's a lot of potential for developing tools to provide collections with data on the use and impact of their collections. Text mining the biodiversity literature on a massive scale to extract (a) mentions of collections (e.g., their institutional acronyms) and (b) citations of specimens could generate metrics that would be helpful to collections. There's a great opportunity here for BHL to generate immediate value for natural history collections (many of which are also contributors to BHL).

Also had a chance to talk to Jorrit Poelen who works on Global Biotic Interactions (GloBI). He made some interesting comparisons between Hexastores (which I'd touched on in my keynote) and Linked Data Fragments.

The final session I attended was Towards robust interoperability in multi-omic approaches to biodiversity monitoring. The overwhelming impression was that there is a huge amount of genomic data, much of which does not easily fit into the classic, Linnean view of the world that characterises, say, GBIF. For most of the sequences we don't know what they are, and that might not be the most interesting question anyway (more interesting might be "what do they do?"). The extent to which these data can be shoehorned into GBIF is not clear to me, although doing so may result in some healthy rethinking of the scope of GBIF itself.

Bootstrapping the biodiversity knowledge graph with JSON-LD

@sckottie @rdmpage @christgendreau Three small, early adopting projects should show expected network of synergies if JSON-LD actually useful

— David Shorthouse (@dpsSpiders) February 1, 2016

In a recent Twitter conversation including David Shorthous and myself (and other poor souls who got dragged in) we discussed how to demonstrate that adopting JSON-LD as a simple linked-data friendly format might help bootstrap the long awaited "biodiversity knowledge graph" (see below for some suggestions for keeping JSON-LD simple). David suggests partnering with "Three small, early adopting projects". I disagree.

I think we need to approach this problem, not from the perspective of who would like to try this approach, but what would it take to really be useful? I think if we look at the structure of the biodiversity knowledge graph and look at who has what identifiers, we can gain some insight into what the next steps are.

Sinks

A fundamental problem is that many data providers are essentially dead ends in terms of building a network. They have data, but no connections, and so they are "sinks". A data browser goes there and can't go any further, it has to retrace its steps and go somewhere else. For example, your typical museum might serve up its collection data like this: The museum has its own identifiers (e.g., a URL) and that's the only identifier in the data. Nomenclators are much the same: You get a nomenclator-specific identifier such as an LSID, but no other identifiers. Once again, if you are a data-crawler traversing the web of data, this is a dead end. (This is why I've become obsessed with linking nomenclators to the primary literature, I want to stop them being dead ends).

Connected sources

Then there are sources which have at least one external identifier, that is, an identifier that they themselves don't control. For example, CrossRef manages DOIs for articles, but if you get metadata for a DOI you also get an ISSN (identifying a journal): Now we have a connection to an external source of data, and some we can traverse the data graph further, which means we can start asking questions (such as how many articles in this journal have DOIs?).

Another connected source is ORCID: This gives us identifiers for authors (the ORCID) linked to article identifiers such as DOIs and PubMed ids (PMID). Follow the DOI to CrossRef and we can link people to articles to journals.

Another connected source is the NCBI: NCBI has several internal identifiers (PMIDs, GenBank accession numbers, tax_ids) all of which lead to rich resources, and it's possible to get a lot of information by staying within NCBI's own silo, but there are external links such as DOIs (usually found attached to PubMed articles or to GenBank accessions) and links to external records such as DNA barcodes in BOLD.

Lets join CrossRef, ORCID, and NCBI together: Now we have a bipartite graph linking sources with identifiers. We can imagine playing a game where we try and connect different entities by moving through the graph. For example, we could take an author identified by an ORCID, follow a DOI to a PMID, a PMID to a set of accessions numbers, and then be able to list all the sequences that an author has published. We could also step back and ask questions about which identifiers are the most useful in terms of making connections between different sources, and which sources provide the most cross links.

Filling in the gaps

Now, the graph above is obviously incomplete. I've restricted it to a few of the key services that I'm familiar with, and make use of external identifiers. One of the big obstacles to fleshing out the biodiversity knowledge graph is the frequent lack of reuse of identifiers. It's not enough to pump out data in linked-data form, you need to build the links. And not just "same as" style links connecting multiple identifiers for the same thing, you need to connect identifiers for different things. Until we tackle that, linked data approaches will not deliver much in the way of value. Hence we need sources that provide genuinely linked data by reusing exiting, external identifiers. If we have name in a nomenclator with a citation string, it needs to link that dumb literature string to a DOI. If we have a taxon concept, it needs to link the name of that taxon to a name in a nomenclator. If we have a specimen that has been sequenced, it needs to give the accession number of that sequence.

From this perspective, choosing which kinds of data with which to explore JSON-LD and a linked data graph should be driven by how connected those sources are: the more connected the more interesting the questions that we can ask. Sadly the vast majority of biodiversity data providers don't provide the kind of connected data we need, which means we continually pay lip service to linked data without feeding it the kind of data it needs in order to grow.

* Notes on JSON-LD

It is relatively easy to write horrible JSON-LD, so I think it would be useful to strive to make it as simple and as human-readable as possible (ignoring the @context block which is always going to be awful, this is the price we pay for simplicity elsewhere). To this end I think we should do at least the following:

1. No URLs as identifiers URLs are ugly, and they are not reliable as identifiers. Providers can change URLs, even for persistent identifiers. The DOI prefix has changed from http://dx.doi.org/ to the preferred http://doi.org/, and the rise of HTTPS (prompted by concerns about security, among other issues, see Google's motives, part 2) is going to break a lot of older URLs (see Web Security - "HTTPS Everywhere" harmful). These changes keep happening, so lets try and shield ourselves from these by using standard prefixes for identifiers (such as 'DOI') for a DOI. Put all the alternative URL resolver prefixes in the @context block and use prefixes such as those standardised by http://identifiers.org (which basically formalises what many people in bioinformatics have been doing for a while).
2. No prefixes for keys In other words, no CURIEs. Don't write "dwc:CatalogueNumber", just write "CatalogueNumber". I don't want to be told where the term comes from (that's what @context is for). If you have a namespace clash (i.e., two terms that are the same unless you include the namespace) then IMHO you're doing it wrong. Either you're using more than one vocabulary for the same thing (why?), or you've not modelling the data at the appropriate level of granularity. Either way, let's avoid clutter and keep things simple and readable.

Linked data that isn't: the failings of RDF

OK, a bit of hyperbole in the morning. One of the goals of RDF is to create the Semantic Web, an interwoven network of data seamlessly linked by shared identifiers and shared vocabularies. Everyone uses the same identifiers for the same things, and when they describe these things they use the same terms. Simples.

Of course, the reality is somewhat different. Typically people don't reuse identifiers, and there are usually several competing vocabularies we can chose from. To give a concrete example, consider two RDF documents describing the same article, one provided by CiNii, the other by CrossRef. The article is:

Astuti, D., Azuma, N., Suzuki, H., & Higashi, S. (2006). Phylogenetic Relationships Within Parrots (Psittacidae) Inferred from Mitochondrial Cytochrome-b Gene Sequences(Phylogeny). Zoological science, 23(2), 191-198. doi:10.2108/zsj.23.191

You can get RDF for a CiNii record by appending ".rdf" to the URL for the article, in this case http://ci.nii.ac.jp/naid/130000017049. For CrossRef you need a Linked Data compliant client, or you can do something like this:

curl -D - -L -H "Accept: application/rdf+xml" "http://dx.doi.org/10.2108/zsj.23.191"

You can view the RDF from these two sources here and here.

No shared identifiers
The two RDF documents have no shared identifiers, or at least, any identifiers they do share aren't described in a way that is easily discovered. The CrossRef record knows nothing about the CiNii record, but the CiNii document includes this statement:

<rdfs:seeAlso rdf:resource="http://ci.nii.ac.jp/lognavi?name=crossref
&amp;id=info:doi/10.2108/zsj.23.191" dc:title="CrossRef" />

So, CiNii knows about the DOI, but this doesn't help much as the CrossRef document has the URI "http://dx.doi.org/10.2108/zsj.23.191", so we don't have an explicit statement that the two documents refer to the same article.

The other shared identifier the documents could share is the ISSN for the journal (0289-0003), but CiNii writes this without the "-", and uses the PRISM term "prism:issn", so we have:

<prism:issn>02890003</prism:issn>

whereas CrossRef writes the ISSN like this:

<ns0:issn xmlns:ns0="http://prismstandard.org/namespaces/basic/2.1/">
0289-0003</ns0:issn>

Unless we have a linked data client that normalises ISSNs before it does a SPARQL query we will miss the fact that these two articles are in the same journal.

Inconsistent vocabularies
Both CiNii use the PRISM vocabulary to describe the article, but they use different versions. CrossRef uses "http://prismstandard.org/namespaces/basic/2.1/" whereas CiNii uses "http://prismstandard.org/namespaces/basic/2.0/". Version 2.1 versus version 2.0 is a minor difference, but the URIs are different and hence they are different vocabularies (having version numbers in vocabulary URIs is asking for trouble). Hence, even if CiNii and CrossRef wrote ISSNs in the same way, we'd still not be able to assert that the articles come from the same journal.
Inconsistent use of vocabularies
Both CiNii use FOAF for author names, but they write the names differently:

<foaf:name xml:lang="en">Suzuki Hitoshi</foaf:name>

<ns0:name xmlns:ns0="http://xmlns.com/foaf/0.1/">Hitoshi Suzuki</ns0:name>

So, another missed opportunity to link the documents. One could argue this would be solved if we had consistent identifiers for authors, but we don't. In this case CiNii have their own local identifiers (e.g. http://ci.nii.ac.jp/nrid/1000040179239), and CrossRef has a rather hideous looking Skolemisation: http://id.crossref.org/contributor/hitoshi-suzuki-2gypi8bnqk7yy.

In summary, it's a mess. Both CiNii and CrossRef organisations are whose core business is bibliographic metadata. It's great that both are serving RDF, but if we think this is anything more than providing metadata in a useful format I think we may be deceiving ourselves.

TreeBASE II RDF

One of the potentially powerful features of TreeBASE II is availability of a RDF version of a study. This means that, in principle, one could take the RDF for a TreeBASE study, combine it with RDF from other sources, and generate a richer view of a particular study. For example, if a TreeBASE study has a DOI, then we could link it to bibliographic details for the study, and through them to other information, such as GenBank sequences, specimens, etc. (see my little linked data browser for an example of some of this linking). If we added a phylogeny viewer, then we'd have a great tool for browsing the basic components of a phylogenetic study.

Unfortunately, we're not there yet. I've been trying to make sense of TreeBASE II RDF, and frankly, it's a mess. Here are some of the problems:

TreeBASE URIs aren't linked data compliant
The canonical URI for a study (e.g., http://purl.org/phylo/treebase/phylows/study/TB2:S10423) doesn't conform to the linked data approach. In fact, the URI crashes the linked data validator, so I tried another test.

curl --include 
   --header "Accept: application/rdf+xml" 
   http://purl.org/phylo/treebase/phylows/study/TB2:S10423

To be a valid linked data resource this request should return a 303 HTTP status code. Instead we get a 302 and some HTML. Linked data clients won't be able to extract information from this URI.

SKOS matching
There are some odd things going on in the RDF. It contains statements of the form:

<rdf:Description rdf:ID="otu1789319">
   <skos:closeMatch rdf:resource="http://purl.uniprot.org/taxonomy/76066.rdf">
</rdf:Description>

(I've tidied this up a little from the original, rather verbose RDF). This asserts that the TreeBASE OTU otu1789319 corresponds to the NCBI taxon with the taxonomy id 76066 (represented by the Uniprot URI). Except, it doesn't really. As far as I understand it, SKOS is about matching concepts, not documents. The URI http://purl.uniprot.org/taxonomy/76066.rdf is a document URI (specifically, a RDF document), the URI http://purl.uniprot.org/taxonomy/76066 is the taxon. The match should really be to http://purl.uniprot.org/taxonomy/76066. Then I've come across statements that match TreeBASE OTUs to http://purl.uniprot.org/taxonomy/0.rdf. This URI doesn't exist (we get a 404). This seems an odd way to say that we don't have a match -- if we don't have a match, don't include it in the RDF.

Local URIs for trees don't work
The RDF is full of local URIs such as http://purl.org/phylo/treebase/phylows/#tree1790755, which don't resolve. In fact they generate a rather spectacular Tomcat exception. I don't understand why we need local URIs. Everything in TreeBASE should have a global URI. Then we can avoid unnecessary statements such as:

http://purl.org/phylo/treebase/phylows/#tree1790755 owl:sameAs http://purl.org/phylo/treebase/phylows/tree/TB2:Tr7899

which links a local resource to a global one http://purl.org/phylo/treebase/phylows/tree/TB2:Tr7899. Incidentally, this URI doesn't resolve, despite claims that this bug has been fixed.

No links between tree and study
But the show stopper for me is that there is no link between a study and a tree! There is no triple in the RDF specifying any relationship between these two entities. To me this is just about the most important thing I need. I want to be able to query TreeBASE RDF using a study identifier (either from TreeBASE itself, or from an external identifier such as a DOI or a PubMed number). As it stands the TreeBASE II RDF is almost useless. I can't get it via a linked data client, it's full of URIs that don't resolve, and it lacks key triples that would glue things together.

RDF != XML

I can't help thinking that the RDF output hasn't been designed with end use in mind. I know from my own experience that it's not until you try to do something with the RDF that you realise how poor some design decisions may have been.

It's not enough to pump out RDF and hope for the best. RDF is not XML, which is just a verbose format for moving data around. RDF brings with it all sorts of expectations about how clients will resolve it, how they will interpret URIs, and the kinds of queries that will be performed. We are achingly close to being able to tie everything together, but not with RDF TreeBASE II is currently making available.

Linked data part 2

Continuing the Friday folly theme, below is a screencast of a linked data browser that uses the same ideas as last week's screencast, but uses a custom browser I've written to display the results in a more user-friendly way.

Linking the data together from Roderic Page on Vimeo.

The demo is live, you can view it at http://iphylo.org/~rpage/browser/www/uri/http://bioguid.info/doi:10.1371/journal.pone.0001787. Under the hood the browser uses bioGUID as the primary linked data provider (although it should consume any valid linked data source, for example Dbpedia). The data is stored in a local triple store (ARC), and the web interface is created by transforming SPARQL queries into HTML using XSLT. You can add data to it by editing the URL in the browser location bar and reloading the page, or entering a URL on the page. Linked data URLs be entered next to the Browse button as is, e.g. http://dbpedia.org/resource/Euphausia, or appended to http://iphylo.org/~rpage/browser/www, e.g. http://http://iphylo.org/~rpage/browser/www/http://dbpedia.org/resource/Euphausi. Other identifiers, such as DOIs, PubMed ids, and specimens need to be resolved via bioGUID, e.g. http://iphylo.org/~rpage/browser/www/uri/http://bioguid.info/gi:86161637.

All still very crude, but I hope you get the idea.

Linking biodiversity data

Time for a Friday folly. I've made a clunky screencast showing an example of linking biodiversity data together, using bioGUID as the universal wrapper around various data sources. I started with GenBank sequence EF013683, added another, EF013555, then explored some links (specimen, publication, taxon, journal), using the OpenLink RDF Browser:

Linking biodiversity data from Roderic Page on Vimeo.

You can try the URIs I used in the linked data browser of your choice:

• http://bioguid.info/genbank:EF013683
• http://bioguid.info/genbank:EF013555
• http://bioguid.info/occurrence:antweb:casent0106008
• http://bioguid.info/taxonomy:411739
• http://bioguid.info/doi:10.1073/pnas.0605858103
• http://bioguid.info/issn:0027-8424

The demo is a bit clunky, partly because the linked data browser is generic. What we really need is a browser that is tailored to displaying the kind of data we're interested, and hides the gory details under the hood. But the goal is to show that, once everything we care about has a resolvable URI that provides data in a consistent form, and we re-use identifiers, then we can glue stuff together with relative ease. In principle we can simply crawl this web of data (you can append other DOIs, ISSNs, and Genbank accession numbers to http://bioguid.info and get RDF to your heart's content).

None of this is particularly new, we've had RDF in biodiversity informatics for at least five years, there are various linked data-style projects, such as GeoSpecies and the first iteration of bioGUID, and some people (such as Roger Hyam) have been pushing HTTP URIs + RDF for a while, but we seem remarkably unable to get traction on this. Notably, no major biodiversity provider provides RDF (by major I mean GenBank or GBIF size). We make diagrams like the one I drew for GBIF last year, we make the case that linking is a Good Thing™, and yet nothing much happens. This suggests that the idea is still not be presented in a compelling enough fashion. Certainly, clunky demos like the one above probably won't help much. Linked Data clients are generally pretty awful things to use. I think we're going to need some compelling applications that really grab people's attention.

EOL, the BBC, and Wikipedia

Last month EOL took the brave step of including Wikipedia content in its pages. I say "brave" because early on EOL was pretty reluctant to embrace Wikipedia on this scale (see the report of the Informatics Advisory Group that I chaired back in 2008), and also because not all of EOL's curators have been thrilled with this development. Partly to assuage their fears, EOL displays Wikipedia-derived content on a yellow background to flag its "unreviewed" status, such as this image of the python genus Leiopython:

It's interesting to compare EOL's approach to Wikipedia with that taken by the BBC, as documented in Case Study: Use of Semantic Web Technologies on the BBC Web Sites. The BBC makes extensive use of content from community-driven external sites such as MusicBrainz and Wikipedia. They embed the content in their own pages, stating where the content came from, but not flagging it as any less meaningful or reliable than the BBC's own content (i.e., no garish yellow background).

Furthermore, the BBC does two clever things. Firstly:

To facilitate integration with the resources external to bbc.co.uk the music site reuses MusicBrainz URL slugs and Wildlife Finder Wikipedia URL slugs. This means that it is relatively straight forward to find equivalent concepts on Wikipedia/DBpedia and Wildlife Finder and, MusicBrainz and /music.

This means that if the identifier for the artist Bat for Lashes in Musicbrainz is http://musicbrainz.org/artist/10000730-525f-4ed5-aaa8-92888f060f5f.html, the BBC reuse the "slug" 10000730-525f-4ed5-aaa8-92888f060f5f and create a page at http://www.bbc.co.uk/music/artists/10000730-525f-4ed5-aaa8-92888f060f5f. Likewise, if the Wikipedia page for Varanus komodoensis is http://en.wikipedia.org/wiki/Komodo_dragon, then the BBC Wildlife Finder page becomes http://www.bbc.co.uk/nature/species/Komodo_dragon, reusing the slug Komodo_dragon.

Reusing identifiers like this can greatly facilitate linking between databases. I don't need to do a search, or approximate string matching, I just reuse the slug. Note that this is a two-way thing, it is trivial for Musicbrainz to create links to BBC information, and visa versa. Reusing identifiers isn't new, other examples include Amazon.com's ASIN (which for books are ISBNs), and BHL reuses uBio NameBankIDs -- want literature that mentions the Komodo dragon? Use the uBio NameBankID 2546401 in a BHL URL http://www.biodiversitylibrary.org/name/2546401.

The second clever thing the BBC does is treat the web as a content management system:

BBC Music is underpinned by the Musicbrainz music database and Wikipedia, thereby linking out into the Web as well as improving links within the BBC site. BBC Music takes the approach that the Web itself is its content management system. Our editors directly contribute to Musicbrainz and Wikipedia, and BBC Music will show an aggregated view of this information, put in a BBC context.

Instead of separating BBC and Wikipedia content (and putting the later in quarantine as does EOL), the BBC embraces Wikipedia, editing Wikipedia content if they feel a page need improving. One advantage of this approach is that it avoids the need for the BBC to replicate Wikipedia, either in terms of content (the BBC doesn't need to write its own descriptions of what an organism does) or services (the BBC doesn't need to develop tools for people to edit the BBC pages, people use Wikipedia's infrastructure for this). Wikipedia provides core text and identifiers, BBC provides its own unique content and branding.

EOL is trying something different, and perhaps more challenging (at least to do it properly). Given that both EOL and Wikipedia offer text about organisms, there is likely to be overlap (and possibly conflict) between what EOL and Wikipedia say about the same taxon. Furthermore, there will be duplication of information such as bibliographic references. For example, the Wikipedia content included in the EOL page for Leiopython contains a bibliography, which includes these references:

Hubrecht AAW. 1879. Notes III on a new genus and species of Pythonidae from Salawatti. Notes from the Leyden Museum 14-15.

Boulenger GA. 1898. An account of the reptiles and batrachians collected by Dr. L. Loria in British New Guinea. Annali del Museo Civico de Storia Naturale di Genova (2) 18:694-710

The genus name Leiopython was published by Hubrecht (1879), and Boulenger (1898) is cited in support of a claim that a distribution record is erroneous. Hence, these look like useful papers to read. Neither reference on the Wikipedia page is linked to an online version of the article, but both have been scanned by EOL's partner BHL (you can see the articles in BioStor here, and here, respectively)¹.

Problem is, you'd be hard pressed to discover this from the EOL page. The BHL results do list the journal Notes from the Leyden Museum, but you'd have to visit the links manually to discover whether they include Hubrecht (1879) (they do, as well as various occurences of Leiopython in the indices for the journal). In part this problem is a consequence of the crude way EOL handles bibliographies retrieved from BHL, but it's symptomatic of a broader problem. By simply mashing EOL and Wikipedia content together, EOL is missing an opportunity to make both itself and Wikipedia more useful. Surely it would be helpful to discover what publications cited on Wikipedia pages are in BHL (or in the list of references for hand-curated EOL pages)? This requires genuine integration (for example by reusing existing bibliographic identifiers such as DOIs, and tools such as OpenURL resolvers). If it fails to do this, EOL will resemble crude pre-Web 2.0 mashups where people created web pages that had content from external sites enclosed in <IFRAME> tags.

The contrast between the approaches adopted by EOL and the BBC is pretty stark. The BBC has devolved text content to external, community-driven sites that it thinks will do a better job than the BBC could alone. EOL is trying to integrate Wikipedia into it's own text content, but without addressing the potentially massive duplication (and, indeed, possible contradictions) that are likely to arise. Perhaps it's time for EOL to be as brave as the BBC, as ask itself whether it is sensible for EOL to try and occupy the same space as Wikipedia.

1. Note that the bibliographic details of both papers are wanting, Hubrecht 1879 is in volume 1 of Notes from the Leyden Museum, and Annali del Museo Civico de Storia Naturale di Genova series 2, volume 18 is also treated as volume 38.

GBIF and Linked Data

At the end of day two of the GBIF LSID-GUID Task Group I put together this crude diagram to summarise some of the possible links between biodiversity data and the larger linked data cloud, which I, among others, have argued is where biodiversity informatics should be heading. Here's my hastily put together diagram (created using the wonderful OmniGraffle):

I've put GBIF at the centre since we're at GBIF, and it's them we are trying to convince. Yellow circles are biodiversity data sources (which aren't linked data providers (but some can me made so using, for example, my LSID proxy resolver), white circles are linked data sources.

The "sales pitch"is that if we join the linked data cloud we open up the possibility of some very powerful queries, especially once that are outside the relatively narrow scope of what GBIF and TDWG concern themselves with. Imagine being able to query biodiversity data with respect to population and economic data across countries. These are the sort of things we could realistically aim for.

On a practical level, it also means biodiversity database could devolve a lot of their tasks to other databases (via reusing identifiers). Some taxonomists have DBPedia URIs, and more could be added to Wikipedia (and so will find there way into DBPedia). Geonames provides geographic URIs which we could reuse, and so on. Within our own community we could do a better job of reusing our own identifiers, and reusing external ones (such as taxa in Wikipedia).

It's late, this is a rushed diagram, and I don't know if it's going to end up in whatever report we manage to assemble tomorrow (our final day). But I hope it captures some of the scope of what we're looking at. I know there are some problems (as have been pointed out to me on Twitter), I'll try and deal with these tomorrow.

EOL, Wikipedia, TDWG, LinkedData, and the Vision Thing

Time for more half-baked ideas. There's been a lot of discussion on Twitter about EOL, Linked Data (sometimes abbreviated LOD), and Wikipedia. Pete DeVries (@pjd) is keen on LOD, and has been asking why TDWG isn't playing in this space. I've been muttering dark thoughts about EOL, and singing the praises of Wikipedia. On so it goes on. So, here's one vision of where we could (?should) be going with this.

Let's imagine that we do indeed want to play in the Linked Data space. The concern that tends to raised the most is that biodiversity informatics uses LSIDs as the standard GUID, and this doesn't play nice with Linked Data. This is true, but not life threatening. There are various hacks (like this and this that deal with this).

But, the real concern (I think) is that we need a way to link our stuff to the rest of the Linked Data cloud. That is, wherever possible we need to reuse existing identifiers. In the LOD diagram below (for the latest version see here) DBpedia.org is key to linking much of this together, and major players (such as the BBC) are now using DBpedia.org to make connections.



DBpedia.org is based on Wikipedia, so I think you can see where this is going. There are some 120,000+ taxon pages in Wikipedia, so that's some 120,000+ identifiers in DBpedia.org that others interested in organisms can (and will) use to refer to taxa. Given the centrality of Wikipedia and DBpedia to LOD, why don't we adopt DBpedia.org URIs as the default GUID for our taxa? At present we have numerous, competing identifiers (e.g., NCBI tax ids, ITIS tsn's, Catalogue of Life LSIDs, uBio NameBankID's, plus LSIDs from various nomenclators). For users this is a mess -- which one do I use? Deciding requires dealing with issues (such as the difference between nomenclatural codes, and between taxonomic names and concepts, etc., that frankly, nobody outside our community cares about.

So, if we want to play with LOD, we need to make our identifiers play nice (straightforward), and we should think seriously about adopting DBpedia.org URIs as the default GUID for taxa.

Now, where does this leave EOL? Well, frankly, it should get out of the business of making web pages for taxa, because Wikipedia owns that space already. Their pages are fewer, but often much more detailed than the corresponding EOL page, and Wikipedia reacts faster to new discoveries. Wikipedia supports community editing, versioning, and quite sophisticated tools for handling biblographic references.

There's plenty of scope for userful tools and services for EOL to develop, but I think the real game is elsewhere. Now, Wikipedia is far from perfect. It's basically semi-structured text with a God-awful template language, and it would benefit greatly from more structure (e.g., as could be provided by Semantic Mediawiki), but I think we should think about building upon it. We could build our own (and my experiments over at itaxon.org explore this), but the big challenge is getting a community around a project, and if David Shorthouse's pronouncement that The Community is Dead is correct, then maybe we should get on board with the community that already exists. Perhaps what EOL should be doing is talking to Wikipedia, improving the existing templates for taxon pages, and creating bots to automatically populate Wikipedia with more taxon pages.

Semantic Publishing: towards real integration by linking

PLoS Computational Biolgy has recently published "Adventures in Semantic Publishing: Exemplar Semantic Enhancements of a Research Article" (doi:10.1371/journal.pcbi.1000361) by David Shotton and colleagues. As a proof of concept, they took Reis et al. (doi:10.1371/journal.pntd.0000228) and "semantically enhanced" it:

These semantic enhancements include provision of live DOIs and hyperlinks; semantic markup of textual terms, with links to relevant third-party information resources; interactive figures; a re-orderable reference list; a document summary containing a study summary, a tag cloud, and a citation analysis; and two novel types of semantic enrichment: the first, a Supporting Claims Tooltip to permit “Citations in Context”, and the second, Tag Trees that bring together semantically related terms. In addition, we have published downloadable spreadsheets containing data from within tables and figures, have enriched these with provenance information, and have demonstrated various types of data fusion (mashups) with results from other research articles and with Google Maps.

The enhanced article is here: doi:10.1371/journal.pntd.0000228.x001. For background on these enhancements, see also David's companion article "Semantic publishing: the coming revolution in scientific journal publishing" (doi:10.1087/2009202, PDF preprint available here). The process is summarised in the figure below (Fig. 10 from Shotton et al., doi:10.1371/journal.pcbi.1000361.g010).



While there is lots of cool stuff here (see also Elsevier's Article 2.0 Contest, and the Grand Chalenge, for which David is one of the judges), I have a couple of reservations.

The unique role of the journal article?

Shotton et al. argue for a clear distinction between journal article and database, in contrast to the view articulated by Philip Bourne (doi:10.1371/journal.pcbi.0​010034) that there's really no difference between a database and a journal article and that the two are converging. I tend to favour the later viewpoint. Indeed, as I argued in my Elsevier Challenge entry (doi:10.1038/npre.2008.2579.1), I think we should publish articles (and indeed data) as wikis, so that we can fix the inevitable error. We can always roll back to the original version if we want to see the author's original paper.

Real linking

But my real concern is that the example presented is essentially "integration by linking", that is, the semantically enhanced version gives us lots of links to other information, but these are regular hyperlinks to web pages. So, essentially we've gone from pre-web documents with no links, to documents where the bibliography is hyperlinked (most online journals), to documents where both the bibliography and some terms in the text are hyperlinked (a few journals, plus the Shotton et al. example). I'm a tad underwhelmed.
What bothers me about this is:

1. The links are to web pages, so it will be hard to do computation on these (unless the web page has easily retrievable metadata)
2. There is no reciprocal linking -- the resource being linked to doesn't know it is the target of the link

Web pages are for humans

The first concern is that the marked-up article is largely intended for human readers. Yes, there are associated metadata files in RDF N3, but the core "added value" is really only of use to humans. For it to be of use to a computer, the links would have to go to resource that the computer can understand. A human clicking on many of the links will get a web page and they can interpret that, but computers are thick and they need a little help. For example, one hyperlinked term is Leptospira spirochete, linked to the uBio namebank record (click on the link to see it). The link resolves to a web page, so it's not much use to a computer (unless if has a scrapper for uBio HTML). Ironically, uBio serves LSIDs, so we could retrieve RDF metadata for this name (urn:lsid:ubio.org:namebank:255659), but there's nothing in the uBio web page that tells the computer that.

Of course, Shotton et al. aren't responsible for the fact that most web pages aren't easily interpreted by computers, but simply embedding links to web pages isn't a big leap forward. What could they have done instead? One approach is to link to resources that are computer-readable. For example, instead of linking the term "Oswaldo Cruz Foundation" to that organisation's home page (http://www.fiocruz.br/cgi/cgilua.exe/sys/start.htm?tpl=home), why not use the DBpedia URI http://dbpedia.org/page/Instituto_Oswaldo_Cruz? Now we get both a human-readable page, and extensive RDF that a computer can use. In other words, if we crawl the semantically enhanced PLoS article with a program, I want to be able to have that crawler follow the links and still get useful information, not the dead end of a HTML web page. Quite a few of the institutions listed in the enhanced paper have DBPedia URIs:

• Brazilian National Commission for Ethics in Research
• Brazilian National Research Council
• Centers for Diseases Control and Prevention
• Centro de Pesquisas Gonçalo Moniz, Fundação Oswaldo Cruz, Ministério da Saúde
• Company for Urban Development of the State of Bahia (CONDER)
• Division of International Medicine and Infectious Diseases, Weill Medical College of Cornell University
• Environmental Systems Research Institute
• Escola Nacional da Saúde Pública, Fundação Oswaldo Cruz, Ministério da Saúde
• Image Bioinformatics Research Group, Department of Zoology, University of Oxford
• MRC Biostatistics Unit
• Oswaldo Cruz Foundation
• R Foundation for Statistical Computing
• Royal Tropical Institute, Holland
• Secretária Estadual de Saúde da Bahia
• Universidad de Buenos Aires
• Universidade Federal Rural do Rio de Janeiro
• Urban Health Council of Pau da Lima
• US National Institutes of Health
• Weill Medical College of Cornell University
• WHO Collaborative Laboratory for Leptospirosis

Why does this matter? Well, if you use DBPedia URIs you get RDF, plus you get connections with the Linked Data crowd, who are rapidly linking diverse data sets together:


I think this is where we need to be headed, and with a little extra effort we can get there, once we move on from thinking solely about human readers.

An alternative approach (and one that I played with in my Challenge entry, as well as my ongoing wiki efforts) is to create what Vandervalk et al. term a "semantic warehouse" (doi:10.1093/bib/bbn051). Information about each object of interest is stored locally, so that clicking on a link doesn't take you off-site into the world wide wilderness, but to information about that object. For example, the page for the paper Mitochondrial paraphyly in a polymorphic poison frog species (Dendrobatidae; D. pumilio) lists the papers cited, clicking on one takes you to the page about that paper. There are limitations to this approach as well, but the key thing is that one could imagine doing computations over this (e.g., computing citation counts for DNA sequences, or geospatial queries across papers) that simple HTML hyperlinking won't get you.

Reciprocal links

The other big issue I have with the Shotton et al. "integration by linking" is that it is one-way. The semantically enhanced paper "knows" that it links to, say, the uBio record for Leptospira, but uBio doesn't know this. It would enhance the uBio record if it knew that doi:10.1371/journal.pntd.0​000228.x001 linked to it.

Links are inherently reciprocal, in the sense that if paper 1 cites paper 2, then paper 2 is cited by paper 1.

Publishers understand this, and the web page of an article will often show lists of papers that cite the paper being displayed. How do we do this for data and other objects of interest? If we database everything, then it's straightforward. CrossRef is storing citation metadata and offers a "forward linking" service, some publishers (e.g., Elsevier and Highwire) offer their own versions of this. In the same way, this record for GenBank sequence AY322281 "knows" that it is cited by (at least) two papers because I've stored those links in a database. Knowing that you're being linked to dramatically enhances discoverability. If I'm browsing uBio I gain more from the experience if I know that the PLoS paper cites Leptospira.

Knowing when you're being linked to

If we database everything locally then reciprocal linking is easy. But, realistically, we can't database everything (OK, maybe that's not strictly true, can can think of Google as a database of everything). The enhanced PLoS paper "knows" that it cites the uBio record, how can the uBio record "know" that it has been cited by the PLoS paper? What if the act of linking was reciprocal? How can we achieve this in a distributed world? Some possibilities:

• we have an explicit API embedded in the link so that uBio can extract the source of the link (could be spoofed, need authentication?)
• we use OpenURL-style links that embed the PLoS DOI, so that uBio knows the source of the link (OpenURL is a mess, but potentially very powerful)
• uBio uses the HTTP referrer header to get the source of the link, then parses the PLoS HTML to extract metadata and the DOI (ugly screen scraping, but no work for PLoS)

Obviously this needs a little more thought, but I think that real integration by linking requires that the resources being linked are both computer and human readable, and that both resources know about the link. This would create much more powerful "semantically enhanced" publications.

LSIDs, HTTP URI, Linked Data, and bioGUID

The LSID discussion has flared up (again) on the TDWG mailing lists. This discussion keeps coming around (I've touched on it here and here), this time it was sparked by the LSID SourceForge site being broken (the part where you get the code is OK). Some of the issues being raised include:

• Nobody uses LSIDs except the biodiversity informatics crowd, have we missed something?
• LSIDs don't play nice with the Linked Data/Semantic Web world, which is much bigger than us
• If we adopt HTTP URIs, will this send the wrong message to data providers (LSIDs imply a commitment to persistence, URLs don't)
• The community has invested a lot in LSIDs, it's too late to change course now

There are other issues as well, in many ways much harder, namely how to ensure adoption and long term persistence of whatever identifier technology the techies agree on.

I've been twittering (@rdmpage) about some of this, and Pierre Lindenbaum blogged about my earlier paper on testing LSIDs (doi:10.1186/1751-0473-3-2), so I decided to return to one of the original goals of my bioGUID project, namely providing a tool to resolve existing identifiers in a consistent way (see the now moribund bioGUID blog, I now blog about bioGUID here on iPhylo). One of the goals of bioGUID was to take an identifier and return RDF. I also had an underlying triple store that was populated with this RDF. After a hardware crash I took the opportunity to rebuild bioGUID from scratch, focussing on OpenURL access to literature. Now, I'm looking at LSIDs again.

The standard response to the concern that the rest of the world has gone down the HTTP URI route is to say that we can stick a HTTP proxy on the front of the LSID (e.g., http://lsid.tdwg.org/urn:lsid:indexfungorum.org:names:21364) and play ball with the Linked Data crowd, who are rapidly linking diverse data sets together:

However, sticking a HTTP proxy on an LSID isn't enough. As outlined in the document Cool URIs for the Semantic Web, we need a way of distinguishing between a HTTP URI that identifies real-world objects or concepts (such as a person or a car), and documents describing those things (put another way, if I put a HTTP URI for Angelina Jolie into a web browser, I expect to get a document describing her, not Ms Jolie herself) . One solution (and the one that is gaining traction) is to use 303 redirects to make this explicit:

A client resolving a URI for a thing will get a 303 status code, telling them that the URI identifies an object. They can get the appropriate representation via content negotiation (a web browser wants HTML, a linked data browser wants RDF).

Data URIs. So, in order to get LSIDs to play ball with Linked Data we need a HTTP proxy that supports 303 redirects (as Roger Hyam pointed out). I've implemented a simple one as part of bioGUID. If you append a LSID to http://bioguid.info/ you get a HTTP URI that passes the
Vapour Linked Data validator tests. For example, http://bioguid.info/urn:lsid:indexfungorum.org:names:21364 resolves to a web page in a browser, but clients that ask for RDF will get that. You can see the steps involved in resolving this Cool URI here. Vapour provides a nice graphical overview of the process:



The TDWG LSID proxy doesn't validate, so this is something that should be addressed.

In addition to resolving LSIDs, my service stores the resulting RDF in a triple store using ARC, and you can query this triple store using a SPARQL interface that makes use of Danny Ayers' Javascript SPARQL editor. I've a serious case of déjà vu as I've implemented this feature several times before using 3store3 (usually after much fun getting it to work). I got bored with triple stores as the bigger problem seemed to be the errors in the metadata I was harvesting, which seriously limited my ability to link different objects together (but that's another story).

DBpedia, and integrating taxonomy with the rest of the linked data world

While biodiversity informatics putters along, generating loads of globally unique identifiers that nobody else uses, perhaps it's time to take a look at the bigger picture. DBPedia is an effort to extract data from Wikipedia and make it available as linked data. At the heart of this effort is the use of HTTP URIs to identify resources, and reusing those URIs. Hence, for many concepts DBpedia URIs are the default option.

Interestingly, in addition to taxa, Wikipedia has pages on prominent (and not so prominent) taxonomists, such as Thomas Say and Henri Milne-Edwards. When it comes to assigning GUIDs to people, DBpedia URIs would be an obvious choice. For example, http://dbpedia.org/resource/Henri_Milne-Edwards is the URI for Henri Milne-Edwards.

This approach has several adavantages. For one, it embeds taxonomic authorities in the broader ocean of linked data. It also makes use of Wikipedia to provide biographical details on taxonomic authorities (many of whom are sufficiently notworthy to appear in Wikipedia). Until we start linking to other data sources, taxonomic data will remain in it's own little ghetto.