The dissection of the colossal squid (Mesonychoteuthis hamiltoni) specimen from Antarctica has been getting a lot of coverage. Pangs of homesickness, especially seeing Steve O'Shea enthusing about the beast. Steve was a contemporary at Auckland Uni when I was a student. I remember him being deeply disappointed in me because I moved away from doing alpha taxonomy of crustaceans (describing taxa such as Pinnotheres atrinicola, right) to fussing with cladistics and computers. Looking at Steve on YouTube, I think it's clear who's having more fun. Maybe I should have stuck to taxonomy after all...Saturday, May 03, 2008
Colossal squid
The dissection of the colossal squid (Mesonychoteuthis hamiltoni) specimen from Antarctica has been getting a lot of coverage. Pangs of homesickness, especially seeing Steve O'Shea enthusing about the beast. Steve was a contemporary at Auckland Uni when I was a student. I remember him being deeply disappointed in me because I moved away from doing alpha taxonomy of crustaceans (describing taxa such as Pinnotheres atrinicola, right) to fussing with cladistics and computers. Looking at Steve on YouTube, I think it's clear who's having more fun. Maybe I should have stuck to taxonomy after all...
Thursday, May 01, 2008
EOL Review
Last month I was at the MBL in Woods Hole, taking part in the review of the Biodiversity Informatics Group. BIG is responsible for the EOL web site. I chair the Informatics Advisory Group, which provides advice to BIG, and it was our task to produce an evaluation of where things stood. I've written a post on the Encyclopaedia of Life blog about some of the big challenges facing EOL as it moves into its second year.
Wednesday, April 30, 2008
Paper published
Bit of a rarity these days. My paper on identifiers in biodiversity informatics, which I mentioned earlier when I deposited the preprint at Nature Precedings, has been published in Briefings in Bioinformatics (doi:10.1093/bib/bbn022).
Here's the abstract:
A major challenge facing biodiversity informatics is integrating data stored in widely distributed databases. Initial efforts have relied on taxonomic names as the shared identifier linking records in different databases. However, taxonomic names have limitations as identifiers, being neither stable nor globally unique, and the pace of molecular taxonomic and phylogenetic research means that a lot of information in public sequence databases is not linked to formal taxonomic names. This review explores the use of other identifiers, such as specimen codes and GenBank accession numbers, to link otherwise disconnected facts in different databases. The structure of these links can also be exploited using the PageRank algorithm to rank the results of searches on biodiversity databases. The key to rich integration is a commitment to deploy and reuse globally unique, shared identifiers [such as Digital Object Identifiers (DOIs) and Life Science Identifiers (LSIDs)], and the implementation of services that link those identifiers.
Monday, April 28, 2008
Google Code wiki using Subversion
For some time now Google Code has been displaying the message:
The web interface for wiki content is currently READ-ONLY for maintenance.
You may still add comments, and members may add, edit, or delete wiki pages via svn. Learn more.
This is a bit of a pain as I've recently put the code for my LSID tester into Google Code (the project is here). Since having a simple wiki is part of the attraction of Google Code, I decided to finally figure out how to add a wiki via Subversion. Turns out it is pretty straight forward. I created a folder called "wiki" and added a file with some wiki markup. I then added it to the repository
(do this from the folder containing "wiki", not within the "wiki" folder itself). This adds the contents of the wiki folder to the Google Code repository. You can then check this out:
This probably seems obvious to many, but I'm used to CVS, having run a CVS repository since the late 1990's when Mike Charleston and I were working on TreeMap. I've been resisting moving to Subversion simply because of the hassle of learning stuff that doesn't actually make my life any easier. That said, Google Code is a nice way to host projects.
The web interface for wiki content is currently READ-ONLY for maintenance.
You may still add comments, and members may add, edit, or delete wiki pages via svn. Learn more.
This is a bit of a pain as I've recently put the code for my LSID tester into Google Code (the project is here). Since having a simple wiki is part of the attraction of Google Code, I decided to finally figure out how to add a wiki via Subversion. Turns out it is pretty straight forward. I created a folder called "wiki" and added a file with some wiki markup. I then added it to the repository
svn import -m "Trying to add wiki"
wiki https://lsid-php.googlecode.com/svn/wiki/
--username USERNAME
(do this from the folder containing "wiki", not within the "wiki" folder itself). This adds the contents of the wiki folder to the Google Code repository. You can then check this out:
svn checkout
https://lsid-php.googlecode.com/svn/wiki/
lsid-php-wiki --username USERNAME
This probably seems obvious to many, but I'm used to CVS, having run a CVS repository since the late 1990's when Mike Charleston and I were working on TreeMap. I've been resisting moving to Subversion simply because of the hassle of learning stuff that doesn't actually make my life any easier. That said, Google Code is a nice way to host projects.
Thursday, April 03, 2008
Biodiversity informatics: the challenge of linking data and the role of shared identifiers
The manuscript for Briefings in Bioinformatics that I alluded to earlier has been accepted for publication. I've put a preprint up at Nature Preceding (hdl:10101/npre.2008.1760.1). The final version will appear in print later this year.
Thursday, March 20, 2008
Phylowidget
Greg Jordan and Bill Piel have released PhyloWidget, a Java applet for viewing phylogenetic trees. It's very slick, with some nice visual effects courtesy of Processing.
PhyloWidget is open source, with code hosted by Google code. I'm a C++ luddite, so it took me a few moments to figure out how to build the applet, but it's simple enough, just type
ant PhyloWidgetat the command prompt. I got a couple of warnings about missing .keystore files (something to do with signing the applet), but otherwise things seemed to work.
The applet has a URL API, which makes it easy to view trees. For example, try this link to view the Frost et al. amphibian tree (doi:10.1206/0003-0090(2006)297[0001:TATOL]2.0.CO;2).
Systematics as Cyberscience
Vince Smith alerted me to "Systematics as Cyberscience", by Christine Hine, whose work I've mentioned earlier. Looks like an interesting read. From the publisher's blurb:
The use of information and communication technology in scientific research has been hailed as the means to a new larger-scale, more efficient, and cost-effective science. But although scientists increasingly use computers in their work and institutions have made massive investments in technology, we still have little idea how computing affects the way scientists work and the kind of knowledge they produce. In Systematics as Cyberscience, Christine Hine explores these questions by examining the developing use of information and communication technology in one discipline, systematics (which focuses on the classification and naming of organisms and exploration of evolutionary relationships). Her sociological study of the ways that biologists working in this field have engaged with new technology is an account of how one of the oldest branches of science transformed itself into one of the newest and became a cyberscience.
Monday, March 10, 2008
Google's Social Graph API
Google's Social Graph API was released earlier this year.
The motivation:
The motivation:
With so many websites to join, users must decide where to invest significant time in adding their same connections over and over. For developers, this means it is difficult to build successful web applications that hinge upon a critical mass of users for content and interaction. With the Social Graph API, developers can now utilize public connections their users have already created in other web services. It makes information about public connections between people easily available and useful.Apart from the obvious application to scientific databases (for example, utilising connections such as co-authorship), imagine the same idea applied to data.
Sunday, March 09, 2008
CrossRef adds more information to OpenURL resolver
Tom Pasley recently drew my attention to CrossRef's addition of a XML format parameter to their OpenURL resolver. Adding &format=xml to the OpenURL request retrieves bibliographic metadata in "unixref" format (for those who like this sort of thing, the XML schema is here). The biggest change is now the metadata lists more than one author for multi-author papers.
I tend to use JSON for my work now, so a common task is to convert XML data streams into JSON. I've modified my bioGUID OpenURL resolver to make use of the unixref format, which meant I had to write a XSLT file to convert unixref to JSON. If you're interested, you can grab a copy here. It's not pretty, but it seems to work OK.
For some years now I've relied on Marc Liyanage's excellent tool TestXSLT to develop XSLT files. If you have a Mac and work with XSLT, then I do yourself a favour and grab a copy of this free tool.
I tend to use JSON for my work now, so a common task is to convert XML data streams into JSON. I've modified my bioGUID OpenURL resolver to make use of the unixref format, which meant I had to write a XSLT file to convert unixref to JSON. If you're interested, you can grab a copy here. It's not pretty, but it seems to work OK.
For some years now I've relied on Marc Liyanage's excellent tool TestXSLT to develop XSLT files. If you have a Mac and work with XSLT, then I do yourself a favour and grab a copy of this free tool.
Thursday, March 06, 2008
PageRank for biodiversity
This will probably tempt fate, but I've an invited manuscript in review for Briefings in Bioinformatics on the topic of identifiers in biodiversity informatics. Readers of this blog will find much of it familiar (DOis, LSIDs, etc.). For fun I constructed a graph for three ant specimens of Probolomyrmex tani, and the images, DNA sequences, and publications that link to these specimens.
Based on this graph I computed the PageRank of each specimen. The motivation for this exercise is that AntWeb lists 43 specimens of this species, in alphabetical order. This is arbitrary. What if we could order them by their "importance"? One way to do this is based on how many times the specimens have been sequenced, photographed, or cited in scientific papers. This gives us a metric for ordering lists of specimens, as well as demonstrating the "value" of a collection (based on people actually using it in their work). I think there is considerable scope for applying PageRank-like ideas to questions in biodiversity informatics. Robert Huber has an intriguing post on TaxonRank that explores this idea further.
Based on this graph I computed the PageRank of each specimen. The motivation for this exercise is that AntWeb lists 43 specimens of this species, in alphabetical order. This is arbitrary. What if we could order them by their "importance"? One way to do this is based on how many times the specimens have been sequenced, photographed, or cited in scientific papers. This gives us a metric for ordering lists of specimens, as well as demonstrating the "value" of a collection (based on people actually using it in their work). I think there is considerable scope for applying PageRank-like ideas to questions in biodiversity informatics. Robert Huber has an intriguing post on TaxonRank that explores this idea further.
Word for the day - "transclusion"
Stumbled across Project Xanadu, Ted Nelson's vision of the way the web should be (e.g., BACK TO THE FUTURE: Hypertext the Way It Used To Be). Nelson coined the term "transclusion", including one document in side another by reference. The screen shot of Xanadu Space may help illustrate the idea:
Nelson envisages a web where instead of just one-way links, documents include parts of other documents, and one can view a document side-by-side with the source documents. Modern web browsers transclude images (the image file is not "physically" in the document, rather it exists elsewhere), but mostly they link to other documents via hyperlinks.
Ted Nelson's writings are a fascinating read, partly because they remind you just how much of the web we take for granted, and how thinks could be different (better?). One thing he objects to is that much of the the web simulates paper
Chris also raises the issue of ranking and relevance -- "What do you do with 19,000 pages containing Hymenoptera?". One possibility is to explore Robert Huber's TaxonRank idea (inspired by Google's PageRank). This would require text mining to build synonomy lists from scanned papers, challenging but not impossible. But I suspect that the network of citations is what will help build a sensible way to rank the 19,000 pages.
A while ago people were speculating what Google could do to help biodiversity informatics. I found much of this discussion to be vague, with no clear notion of what Google could actually do. What I think Google is exceptionally good at are two things we need to tackle -- text mining, and extracting information from links. I think this is where BHL and, by extension, EOL, should be devoting much of their resources.
Nelson envisages a web where instead of just one-way links, documents include parts of other documents, and one can view a document side-by-side with the source documents. Modern web browsers transclude images (the image file is not "physically" in the document, rather it exists elsewhere), but mostly they link to other documents via hyperlinks.
Ted Nelson's writings are a fascinating read, partly because they remind you just how much of the web we take for granted, and how thinks could be different (better?). One thing he objects to is that much of the the web simulates paper
Much of the field has imitated paper: in word processing (Microsoft Word and Adobe Acrobat) and the World Wide Web, whose rectangular page layouts become a focal issue. It should be noted that these systems imitate paper under glass, since you can't annotate it.Nelson also advocates every element of a document having its own unique address, not just at book or article level. This resonates with what is happing with digital libraries. Gregory Crane in his article "What Do You Do with a Million Books?" (doi:10.1045/march2006-crane) notes that:
Most digital libraries still mimic their print predecessors, treating individual objects – commonly chunks of PDF, RTF/Word, or HTML with no standard internal structure – as its constituent units. As digital libraries mature and become better able to extract information (e.g., personal and place names), each word and automatically identifiable chunk of words becomes a discrete object. In a sample 300 volume, 55 million word collection of nineteenth-century American English, automatic named entity identification has added 12,000,000 tags. While this collection focuses on name rich historical materials and includes several reference works, this system already discovers thousands of references to named entities in most book length documents. We thus move from single catalogue entries with a few hundred words to thousands of tagged objects – an increase of at least one order of magnitude with named entities and of at least two orders of magnitude when we consider each individual word as an object.I discovered Crane's paper via Chris Freeland's post On Name Finding in the BHL. Chris summarises BHL's work on scanning biodiversity literature and extracting taxonomic names. BHL's output is at the level of pages, rather than articles. Existing GUIDs for literature (such as DOIs and SICIs) typically identify articles rather than pages (or page elements), so there's a need to extending these to pages.
Chris also raises the issue of ranking and relevance -- "What do you do with 19,000 pages containing Hymenoptera?". One possibility is to explore Robert Huber's TaxonRank idea (inspired by Google's PageRank). This would require text mining to build synonomy lists from scanned papers, challenging but not impossible. But I suspect that the network of citations is what will help build a sensible way to rank the 19,000 pages.
A while ago people were speculating what Google could do to help biodiversity informatics. I found much of this discussion to be vague, with no clear notion of what Google could actually do. What I think Google is exceptionally good at are two things we need to tackle -- text mining, and extracting information from links. I think this is where BHL and, by extension, EOL, should be devoting much of their resources.
Tuesday, February 26, 2008
Encyclopedia of Life - first impressions
Some thoughts on the first release of the Encyclopedia of Life. I am being deliberately critical. This is a high profile project with tens of millions of dollars in funding, lots of people involved, and is accompanied by some of the most overblown hype in organismal biology. In a sense I think EOL has set itself up by over promising and under delivering.
Before continuing, I should point out that I am involved in EOL in an advisory capacity, but not in actually making anything. Some of the tools I've blogged about have made there way into EOL, such as Pygmybrowse and reference parsing (see David Shorthouse's excellent work on this).
Lack of content
I think the first release of EOL should have, at a minimum, provided at least as much information that I can get from iSpecies and Wikipedia. Other projects, such as Freebase, have pre-populated their databases with content from Wikipedia and other sources. Why didn't EOL? If the argument is that they want authenticated content, then this doesn't wash. Their authenticated content is minimal, and waiting for authentication will, in my view, cripple EOL.
Exemplars are incomplete
The first release contains 25 exemplars. Pages for these taxa
However, the GBIF map EOL displays shows lots of dots in Hawaii:
The EOL account is silent on this interesting distribution pattern. It will come as no suprise that the Wikipedia account of the same species tells us that it has been introduced into Hawaii. Wikipedia 1, EOL 0.
Links
If two pages talk about species that are ecologically associated, then surely those pages should be linked? Among the exemplars is Pissodes strobi, the white pine weevil. In the EOL account, among the hosts listed is Pinus strobus, another exemplar taxon. The accounts of these two taxa are not linked. No hyperlink, nothing. The reader has no idea that there is an exemplar account for Pinus strobus. Furthermore, when reading the account for Pinus strobus there is no indication that it is host to the white pine weevil.
Surely the point of having all this information in one place is so that it can be linked together?
BHL
EOL also exposes some limitations of the Biodiversity heritage Library. Consider the exemplar page for Pinus strobus L. The "L." indicates that this species was described by Linnaeus. Among the many references listed by BHL, none are by Linnaeus. What gives?
Well, the IPNI record reveals that this species was described on p. 1001 of Species Plantarum. BHL has digitised Species Plantarum, and page 1001 has Pinus strobus:
Now, BHL relies on uBio's tools to extract names, and Linnaeus didn't make this easy (the specific epithet strobus is in the right hand margin, separate from Pinus), but one would have thought that for the exemplar taxa an effort would have been made to link Linnaean names to BHL content -- what better place to showcase the link between a name and its publication? It's quite easy to do, given that IPNI has page numbers for plant names. Just map page numbers to BHL URLs, and you're done.
Inconsistency
Going down the taxonomic hierarchy weird things happen. When viewing the plant genus Morus if I can see a picture of Morus nigra (presumably this is "authenticated" content). If I drill down to the species Morus nigra, I'm told there is no authenticated content for this species. Either the image is Morus nigra or it isn't. If it is, why not show it, if it isn't, why claim that it is?
Logos
Way too much space is devoted to logos of various contributors, BHL being the worst offender (it doesn't help that the BHL content is incomplete, lacking links for Linnaean names). I don't care about logos. Contributors may care about getting their logos displayed, but users couldn't care less. They get in the way. On some pages, there's more screen space devoted to logos than information (e.g., the page for Apomys datae). This is, frankly, ridiculous, and reflects a warped set of priorities.
What's worse, all these logos are associated with links that take people away from EOL. Hence EOL becomes little more than a collection of web links to other sites.
Search
The search is based on the Catalogue of Life, and inherits the same problems. For example, if I search for "Morus" I get a list in alphabetical order of taxonomic names that contain the string "morus". The two names that are an exact match occur as items three and four on the list -- they should be first and second.
It gets worse if I search on "Tyrannosaurus rex". EOL doesn't do dinosaurs, and so doesn't contain anything on T. rex, but the search results tell me that The following 116 search results contain 'Tyrannosaurus rex'. Nope, none of them do.
The search engine is poorly done, it fails to rank results sensibly, incorrectly reports what it does find, and has no support for spelling mistakes.
Authenticated content
This is probably the thing that, if left as it is, will strangle EOL. The insistence on "authenticated (endorsed)" content places a severe brake on what EOL can offer.
It's a web site
EOL's web site has no mechanism for people to extract data (e.g., RSS feeds, microformats, links to RDF, etc.). It's intended to be read by humans, not machines. This greatly diminishes its utility.
So, I've got that off my chest. The first release was always going to be a disappointment, especially given the hype. What frustrates me, however, is just how far the first release is from what it could have been.
The real question is how much the issues I've raised are things which are easy to fix given time, or whether they reflect underlying problems with the way the project is conceived.
Before continuing, I should point out that I am involved in EOL in an advisory capacity, but not in actually making anything. Some of the tools I've blogged about have made there way into EOL, such as Pygmybrowse and reference parsing (see David Shorthouse's excellent work on this).
Lack of content
I think the first release of EOL should have, at a minimum, provided at least as much information that I can get from iSpecies and Wikipedia. Other projects, such as Freebase, have pre-populated their databases with content from Wikipedia and other sources. Why didn't EOL? If the argument is that they want authenticated content, then this doesn't wash. Their authenticated content is minimal, and waiting for authentication will, in my view, cripple EOL.
Exemplars are incomplete
The first release contains 25 exemplars. Pages for these taxa
...show the kind of rich environment, with extensive information, to which all the species pages will eventually grow. The information on the exemplar pages has been authenticated (endorsed) by the scientists whose names are listed on these pages.Well, I hope this isn't the standard EOL aspires to. The pages are incomplete and not interlinked. One of the 25 chosen exemplars is Anolis carolinensis. EOL lists its distribution as:
Widely-distributed throughout the southeastern United States: North Carolina to Key West, Florida, and west to southest Oklahoma and central Texas.
However, the GBIF map EOL displays shows lots of dots in Hawaii:
The EOL account is silent on this interesting distribution pattern. It will come as no suprise that the Wikipedia account of the same species tells us that it has been introduced into Hawaii. Wikipedia 1, EOL 0.
Links
If two pages talk about species that are ecologically associated, then surely those pages should be linked? Among the exemplars is Pissodes strobi, the white pine weevil. In the EOL account, among the hosts listed is Pinus strobus, another exemplar taxon. The accounts of these two taxa are not linked. No hyperlink, nothing. The reader has no idea that there is an exemplar account for Pinus strobus. Furthermore, when reading the account for Pinus strobus there is no indication that it is host to the white pine weevil.
Surely the point of having all this information in one place is so that it can be linked together?
BHL
EOL also exposes some limitations of the Biodiversity heritage Library. Consider the exemplar page for Pinus strobus L. The "L." indicates that this species was described by Linnaeus. Among the many references listed by BHL, none are by Linnaeus. What gives?
Well, the IPNI record reveals that this species was described on p. 1001 of Species Plantarum. BHL has digitised Species Plantarum, and page 1001 has Pinus strobus:
Now, BHL relies on uBio's tools to extract names, and Linnaeus didn't make this easy (the specific epithet strobus is in the right hand margin, separate from Pinus), but one would have thought that for the exemplar taxa an effort would have been made to link Linnaean names to BHL content -- what better place to showcase the link between a name and its publication? It's quite easy to do, given that IPNI has page numbers for plant names. Just map page numbers to BHL URLs, and you're done.
Inconsistency
Going down the taxonomic hierarchy weird things happen. When viewing the plant genus Morus if I can see a picture of Morus nigra (presumably this is "authenticated" content). If I drill down to the species Morus nigra, I'm told there is no authenticated content for this species. Either the image is Morus nigra or it isn't. If it is, why not show it, if it isn't, why claim that it is?
Logos
Way too much space is devoted to logos of various contributors, BHL being the worst offender (it doesn't help that the BHL content is incomplete, lacking links for Linnaean names). I don't care about logos. Contributors may care about getting their logos displayed, but users couldn't care less. They get in the way. On some pages, there's more screen space devoted to logos than information (e.g., the page for Apomys datae). This is, frankly, ridiculous, and reflects a warped set of priorities.
What's worse, all these logos are associated with links that take people away from EOL. Hence EOL becomes little more than a collection of web links to other sites.
Search
The search is based on the Catalogue of Life, and inherits the same problems. For example, if I search for "Morus" I get a list in alphabetical order of taxonomic names that contain the string "morus". The two names that are an exact match occur as items three and four on the list -- they should be first and second.
It gets worse if I search on "Tyrannosaurus rex". EOL doesn't do dinosaurs, and so doesn't contain anything on T. rex, but the search results tell me that The following 116 search results contain 'Tyrannosaurus rex'. Nope, none of them do.
The search engine is poorly done, it fails to rank results sensibly, incorrectly reports what it does find, and has no support for spelling mistakes.
Authenticated content
This is probably the thing that, if left as it is, will strangle EOL. The insistence on "authenticated (endorsed)" content places a severe brake on what EOL can offer.
It's a web site
EOL's web site has no mechanism for people to extract data (e.g., RSS feeds, microformats, links to RDF, etc.). It's intended to be read by humans, not machines. This greatly diminishes its utility.
So, I've got that off my chest. The first release was always going to be a disappointment, especially given the hype. What frustrates me, however, is just how far the first release is from what it could have been.
The real question is how much the issues I've raised are things which are easy to fix given time, or whether they reflect underlying problems with the way the project is conceived.
EOL live
The first release of the Encyclopedia of Life is officially live today. I have promised to be very good...
Monday, February 18, 2008
LSID Tester, a tool for testing Life Science Identifier resolution services
My short note on the LSID Tester tool has been published in the Open Access journal Source Code for Biology and Medicine. The article has just come out so the DOI (doi:10.1186/1751-0473-3-2) isn't live yet, the direct link is http://www.scfbm.org/content/3/1/2/. Source code for the tester is available from Google Code.
TBMap errors
In the absence of a proper bug reporting system, I'm going to use this post to collect errors in the TBMap project, which maps taxonomic names in TreeBASE onto names in other databases.
| TaxonID | TaxonName | Notes |
| T57654 | Lycorideae | Erroneously agrep matched to the spider family Lycosidae, this is a plant tribe. |
| T56449 | Ficus uncinata | bad agrep to Pinus uncinata |
Sunday, February 17, 2008
CrossRef blogger tool for DOI lookup
CrossRef have released a tool for bloggers to look up DOIs and insert them into blog posts:
So far the tool is only available for WordPress blogs. The idea is that bloggers can use DOIs to uniquely identify papers that they are discussing, while at the same time providing readers with an easy way to go to the site hosting the article, and aggregators such as postgenomic.com can cluster posts about the same paper.
Whilst Googling for reaction, I came across various posts extolling the virtues of OpenURL versus DOIs, or proposing alternative identifiers (DOI or DOH? Proposal for a RESTful unique identifier for papers, and PaperID - An Open Source Identifier for Research Papers). Personally I think much of this discussion focusses on identifiers, when it's the services built on those identifiers that really matter.
The plug-in, which is available for download at: https://sourceforge.net/projects/crossref-cite/, allows the blogger to use a widget-based interface to search CrossRef metadata using citations or partial citations. The results of the search, with multiple hits, are displayed and the author can then either click on a hit to follow the DOI to the publisher’s site, or click on an icon next to the hit to insert the citation into their blog entry (as either a full citation or as a short “op. cit.”).
So far the tool is only available for WordPress blogs. The idea is that bloggers can use DOIs to uniquely identify papers that they are discussing, while at the same time providing readers with an easy way to go to the site hosting the article, and aggregators such as postgenomic.com can cluster posts about the same paper.
Whilst Googling for reaction, I came across various posts extolling the virtues of OpenURL versus DOIs, or proposing alternative identifiers (DOI or DOH? Proposal for a RESTful unique identifier for papers, and PaperID - An Open Source Identifier for Research Papers). Personally I think much of this discussion focusses on identifiers, when it's the services built on those identifiers that really matter.
Tuesday, February 05, 2008
The Data Wars
Wired 16.01 has an article entitled The Data Wars by Josh McHugh. A quote from the printed version:
They call it scraping — when web companies automatically harvest information from the likes of Yahoo, Google, and craigslist. Now the Internet establishment is clamping down.
It's a sobering read for those of us who advocate harvesting data from as many sources as possible, more so in light of Microsoft's bid to buy Yahoo. Yahoo provides free access to many of its tools via an API (such as the image search I use in iSpecies, and in this sense is much more open than Google. Might this change under Microsoft...?
Monday, February 04, 2008
How to visualize a phylogeny with thousands of tips?
Dave Lunt has a nice post on How to visualize a phylogeny with thousands of tips?. Dave lists 12 things that his ideal phylogenetic tree viewing tool should do, and invites comments. It will be interesting to see what comes of this...
Incomplete citation and ranking
Came across the paper "Using incomplete citation data for MEDLINE results ranking" (pmid:16779053, fulltext available in PMC .The authors applied PageRank (the algorithm Google use to rank search results) to papers in MEDLINE and found that PageRank is robust to information loss. In other words, even if a citation database is incomplete it will do a good job of ranking results. This is encouraging, as I'm keen to use this approach to rank both papers and other objects (e.g., sequences and specimens), and will almost certainly never have a complete citation list.
A database of everything
Nothing like a little hubris first thing Monday morning...
After various experiments, such as a triple store for ants (documented on the Semant blog) and bioGUID (documented on the bioGUID blog), I'm starting from scratch and working on a "database of everything". Put another way, I'm working on a database that aggregates metadata about specimens, sequences, literature, images, taxonomic names, etc. But beyond "merely" aggregating data I'm really interested in linking the data. Here are some design decisions I've made so far.
The first is that the database structure used to store metadata uses the Entity–Attribute–Value (EAV) model (for some background see the papers I've bookmarked on Connotea with the tag entity–attribute–value). This tends to send traditional database managers into fits, but for my purposes (storing objects of different kinds with frequently sparse metadata) it seems a natural choice. It's a little bit like a triple store, but I find triple stores frustrating when creating a database. Given the often low quality of the incoming data a lot of post-processing may be needed. Having the data easily accessible makes this task easier -- I view triple stores as "read only" databases. The data I store will be made available as RDF, so users could still use Semantic Web tools to analyse it.
Each object stored in the database gets an identifier based on a md5 hash of one of it's GUIDs (e.g., PubMed identifier, DOI, URL), a technique used by del.icio.us and Connotea (for example, the md5 hash of "http://dx.doi.org/10.2196/jmir.5.4.e27" is d00cf429c001c3c7ae4f2d730718dcc8, hence the Connotea URI for this article is http://www.connotea.org/article/d00cf429c001c3c7ae4f2d730718dcc8). The URIs for the objects in my database will also use these md5 hashes. The original GUIDs (such as DOIs, etc.) are stored within the database.
Much of the programming for this project involves retrieving metadata associated with an identifier. Taking the code I wrote for bioGUID as a starting point, I've added a lot of code to try and clean up the metadata (fixing dates, extracting latitude and longitudes and such like, see Metacrap), but also to extract additional identifiers from text. The database is fundamentally about these links between objects. Typically I write a client for a service that returns metadata for an identifier (typically in XML), transform it to JSON, then post process it.
Another decision is that names of genes and taxa are treated as tags, postponing any decision about what the names actually refer to, as well as whether two or more tags refer to the same thing. The expectation is that much of this can be worked out after the fact (in much the same way as I did for the TbMap project).
The database is being populated in two ways, spidering and bulk upload. Spidering involves taking an identifier, finding any linked identifiers, and resolving those. For example, a PubMed identifier may link to a set of nucleotide sequences, many of which may link to a specimen code. Bulk upload involves harvesting large numbers of records and adding them to the database.
The initial focus is on adding records related to TreeBASE, with a view to creating an annotated version of that database such that a user could ask questions like "find me all phylogenies in TreeBASE that include taxa from Australia."
After various experiments, such as a triple store for ants (documented on the Semant blog) and bioGUID (documented on the bioGUID blog), I'm starting from scratch and working on a "database of everything". Put another way, I'm working on a database that aggregates metadata about specimens, sequences, literature, images, taxonomic names, etc. But beyond "merely" aggregating data I'm really interested in linking the data. Here are some design decisions I've made so far.
The first is that the database structure used to store metadata uses the Entity–Attribute–Value (EAV) model (for some background see the papers I've bookmarked on Connotea with the tag entity–attribute–value). This tends to send traditional database managers into fits, but for my purposes (storing objects of different kinds with frequently sparse metadata) it seems a natural choice. It's a little bit like a triple store, but I find triple stores frustrating when creating a database. Given the often low quality of the incoming data a lot of post-processing may be needed. Having the data easily accessible makes this task easier -- I view triple stores as "read only" databases. The data I store will be made available as RDF, so users could still use Semantic Web tools to analyse it.
Each object stored in the database gets an identifier based on a md5 hash of one of it's GUIDs (e.g., PubMed identifier, DOI, URL), a technique used by del.icio.us and Connotea (for example, the md5 hash of "http://dx.doi.org/10.2196/jmir.5.4.e27" is d00cf429c001c3c7ae4f2d730718dcc8, hence the Connotea URI for this article is http://www.connotea.org/article/d00cf429c001c3c7ae4f2d730718dcc8). The URIs for the objects in my database will also use these md5 hashes. The original GUIDs (such as DOIs, etc.) are stored within the database.
Much of the programming for this project involves retrieving metadata associated with an identifier. Taking the code I wrote for bioGUID as a starting point, I've added a lot of code to try and clean up the metadata (fixing dates, extracting latitude and longitudes and such like, see Metacrap), but also to extract additional identifiers from text. The database is fundamentally about these links between objects. Typically I write a client for a service that returns metadata for an identifier (typically in XML), transform it to JSON, then post process it.
Another decision is that names of genes and taxa are treated as tags, postponing any decision about what the names actually refer to, as well as whether two or more tags refer to the same thing. The expectation is that much of this can be worked out after the fact (in much the same way as I did for the TbMap project).
The database is being populated in two ways, spidering and bulk upload. Spidering involves taking an identifier, finding any linked identifiers, and resolving those. For example, a PubMed identifier may link to a set of nucleotide sequences, many of which may link to a specimen code. Bulk upload involves harvesting large numbers of records and adding them to the database.
The initial focus is on adding records related to TreeBASE, with a view to creating an annotated version of that database such that a user could ask questions like "find me all phylogenies in TreeBASE that include taxa from Australia."
Subscribe to:
Posts (Atom)