Thursday, October 20, 2011

Reflections on the TDWG RDF "Challenge"

This is a follow up to my previous post TDWG Challenge - what is RDF good for? where I'm being, frankly, a pain in the arse, and asking why we bother with RDF? In many ways I'm not particularly anti-RDF, but it bothers me that there's a big disconnect between the reasons we are going down this route and how we are actually using RDF. In other words, if you like RDF and buy the promise of large-scale data integration while still being decentralised ("the web as database"), then we're doing it wrong.

As an aside, my own perspective is one of data integration. I want to link all this stuff together so I can follow a path through multiple datasets and extract the information I want. In other words, "linked data" (little "l", little "d"). I'm interested in fairly light weight integration, typically through shared identifiers. There is also integration via ontologies, which strikes me as a different, if related, problem, that in many ways is closer to the original vision of the Semantic Web as a giant inference engine. I think the concerns (and experience) of these two communities are somewhat different. I don't particularly care about ontologies, I want key-value pairs and reusable identifiers so I can link stuff together. If, for example, you're working on something like Phenoscape, then I think you have a rather more circumscribed set of data, with potentially complicated interrelationships that you want to make inferences on, in which case ontologies are your friend.

So, I posted a "challenge". It wasn't a challenge so much as a set of RDF to play with. What I'm interested in is seeing how easily we can string this data together to learn stuff. For example, using the RDF I posted earlier here is a table listing the name, conservation status, publication DOI and date, and (where available) image from Wikipedia for frogs with sequences in GenBank.

SpeciesStatusDOIYear describedImage
Atelopus nanayCR[0229:TNSOAA]2.0.CO;22002
Eleutherodactylus mariposaCR
Phrynopus kauneorumCR
Eleutherodactylus eunasterCR
Eleutherodactylus amadeusCR
Eleutherodactylus lamprotesCR
Churamiti maridadiCR
Eleutherodactylus thorectesCR
Eleutherodactylus apostatesCR
Leptodactylus silvanimbusCR
Eleutherodactylus sciagraphusCR
Bufo chavinCR[0216:NSOBAB]2.0.CO;22001
Eleutherodactylus fowleriCR
Ptychohyla hypomykterCR
Hyla suweonensisDD
Proceratophrys concavitympanumDD
Phrynopus bufoidesDD
Boophis periegetesDD
Phyllomedusa duellmaniDD
Boophis liamiDD
Hyalinobatrachium ignioculusDD[0091:ANSOHA]2.0.CO;22003
Proceratophrys cururuDD
Amolops bellulusDD[0536:ABANSO]2.0.CO;22000
Centrolene bacatumDD
Litoria kumaeDD
Phrynopus pesantesiDD
Gastrotheca galeataDD
Paratelmatobius cardosoiDD
Rhacophorus catamitusDD[0046:NAPKPF]2.0.CO;22002
Huia melasmaDD
Telmatobius vilamensisDD[0253:ANSOTA]2.0.CO;22003
Callulina kisiwamsituEN
Arthroleptis nikeaeEN
Eleutherodactylus amplinymphaEN
Eleutherodactylus glaphycompusEN
Bufo tacanensisEN
Phrynopus brackiEN
Telmatobius sibiricusEN[0127:ANSOTF]2.0.CO;22003
Cochranella macheEN
Eleutherodactylus melacaraEN
Plectrohyla glandulosaEN
Aglyptodactylus laticepsEN
Eleutherodactylus glamyrusEN
Gastrotheca trachycepsEN
Eleutherodactylus grahamiEN
Litoria havinaLC
Crinia ripariaLC
Litoria longirostrisLC
Osteocephalus mutaborLC
Leptobrachium nigropsLC
Pseudis tocantinsLC
Mantidactylus argenteusLC
Aglyptodactylus securiferLC
Pseudis cardosoiLC
Uperoleia inundataLC
Litoria pronimiaLC
Litoria paraewingiLC
Philautus aurifasciatusLC
Proceratophrys avelinoiLC
Osteocephalus deridensLC
Gephyromantis boulengeriLC
Crossodactylus caramaschiiLC
Rana yavapaiensisLC
Boophis lichenoidesLC
Megistolotis lignariusLC
Ansonia endauensisNE[466:ANSOAS]2.0.CO;22006
Ansonia kraensisNE
Arthroleptella landdrosiaNT
Litoria jungguyNT
Phrynobatrachus phyllophilusNT
Philautus ingeriVU
Gastrotheca dendronastesVU
Hyperolius cystocandicansVU
Boophis sambiranoVU
Ansonia torrentisVU
Telmatobufo australisVU
Stefania coxiVU[0327:EDOSAH]2.0.CO;22002
Oreolalax multipunctatusVU
Eleutherodactylus guantanameraVU
Spicospina flammocaeruleaVU
Cycloramphus acangatanVU
Leiopelma pakekaVU
Rana okaloosaeVU
Phrynobatrachus uzungwensisVU

This is a small fraction of the frog species actually in GenBank because I've filtered it down to those that have been linked to Wikipedia (from where we get the conservation status) and which were described in papers with DOIs (from which we get the date of description).

I generated this result using this SPARQL query on a triple store that had the primary data sources (Uniprot, Dbpedia, CrossRef, ION) loaded, together with the all-important "glue" datasets that link ION to CrossRef, and Uniprot to Dbpedia (see previous post for details):

PREFIX rdf: <>
PREFIX rdfs: <>
PREFIX dbpedia-owl: <>
PREFIX uniprot: <>
PREFIX tdwg_tn: <>
PREFIX tdwg_co: <>
PREFIX dcterms: <>

SELECT ?name ?status ?doi ?date ?thumbnail
?ncbi uniprot:scientificName ?name .
?ncbi rdfs:seeAlso ?dbpedia .
?dbpedia dbpedia-owl:conservationStatus ?status .
?ion tdwg_tn:nameComplete ?name .
?ion tdwg_co:publishedInCitation ?doi .
?doi dcterms:date ?date .

?dbpedia dbpedia-owl:thumbnail ?thumbnail
ORDER BY ASC(?status)

This table doesn't tell us a great deal, but we could, for example, graph date of description against conservation status (CR=critical, EN=endangered, VU=vulnerable, NT=not threatened, LC=least concern, DD=data deficient):
In other words, is it the case that more recently described species are more likely to be endangered than taxa we've known about for some time (based on the assumption that we've found all the common species already)? We could imagine extending this query to retrieve sequences for a class of frog (e.g., critically endangered) so we could compute a measure population genetic variation, etc. We shouldn't take the graph above too seriously because it's based on small fraction of the data, but you get the idea. As more frog taxonomy goes online (there's a lot of stuff in BHL and BioStor, for example) we could add more dates and build a dataset worth analysing properly.

It seems to me that these should be fairly simple things to do, yet they are the sort of thing that if we attempt today it's a world of hurt involving scripts, Excel, data cleaning, etc. before we can do the science.

The thing is, without the "glue" files mapping identifiers across different databases even this simple query isn't possible. Obviously we have no say in how many organisations publish RDF, but within the biodiversity informatics community we should make every effort to use external identifiers wherever possible so that we can make these links. This is the core of my complaint. If we are using RDF to foster data integration so we can query across the diverse data sets that speak to biodiversity, then we are doing it wrong.

Here is a nice visualisation of this dataset from @orovellotti (original here), made using ecoRelevé:

AcNbdh2CMAA3ysc png large