Rants, raves (and occasionally considered opinions) on phyloinformatics, taxonomy, and biodiversity informatics. For more ranty and less considered opinions, see my Twitter feed. ISSN 2051-8188. Written content on this site is licensed under a Creative Commons Attribution 4.0 International license.
Hasan Jamil has released PhyQL, a visual system for querying phylogenetic information. To quote from the web site:
Popular phylogenetic databases such as TreeBASE, PhyloFinder, TreeFAM offer complex text-based web forms for structure queries. Still there seems a great need for intelligent visual query formation based on a phylogenetic query language for content exploration. PhyQL offers a visual query design interface where the user can create simple to complex queries based visual query operators. The query language is translated to a list of datalog queries, then executed in XSB, an extension of Prolog. Separating the application layer from the data layer by a logic layer reduces query tools development time. Moreover, PhyQL offers interactive tree visualization which is very convenient for viewing very large trees.
There is also a YouTube screencast:
I haven't had a chance to play with it yet. PhyQL was originally described by Jamil et al. "Querying phylogenies visually", BIBE 2001 (doi:10.1109/BIBE.2001.974405).
I've put the first version of "tvwidget" into Google Code. This is a HTML-only widget to display large evolutionary trees (you can see how my thoughts on how to do this unfolded by following my earlier posts starting with Visualising very big trees, Part V). tvwidget itself is a C++ program that takes a tree and generates the image tiles and Javascript for the viewer. It's poorly documented, I'll deal with tis once I get some time.
I've added a service to bioGUID that takes a PDF and attempts to extract latitude and longitude data from the PDF, returning those co-ordinates in either a Google Earth KML file, or in JSON format. This is one of a bunch of services that I'm adding to bioGUID to support some of the data mining that I'm doing.
Then try this one to get the KML file, and open it in Google Earth. The service uses a bunch of regular expressions to try and extract latitude and longitude pairs from the text (needless to say, there are nearly as many different ways to write a latitude and longitude as there are authors).
The ultimate aim is to assemble a bunch of Open Access PDFs (say, from Zootaxa), run them through this service, then display the result on Google Earth. Think of it as a geography of taxonomy.
Oh, and the irony of me criticising GBIF for displaying poor quality data, then adding to this by providing a service to extract yet more co-ordinates of possibly doubtful validity has not entirely escaped me...
Resurrecting iSpecies after moving it to a new folder on one of my servers, and browsing popular searches, I keep coming across clearly erroneous distributions. FishBase seems a major culprit. For example, the common pandora Pagellus erythrinus is a marine fish, yet GBIF displays numerous occurrences in mainland Africa (dots with black centre on map below).
What gives? Well, after struggling with the somewhat non-intuitive GBIF web site I found that the erroneous records are from FishBase. As for the frog example I blogged about earlier, the actual records have locality information indicating most of the records come from the Mediterranean, but the latitude and longitudes are reversed. Swapping these, the records show a more believable distribution (white dots on SVG map below). If you don't see the map, use a decent web browser such as Safari 3 or Firefox 2. If you must use Internet Explorer, grab the RENESIS player.
I know I've harped on about this before, but surely the time is ripe for some clever data cleaning? Especially if users start to loose their trust in GBIF.
I have an "on again/off again" relationship with treemaps. Lately, I've been taking another look, partly inspired by Björn Engdahl's MSc thesis Ordered and Unordered Treemap Algorithms and Their Applications on Handheld Devices. He describes a simple treemap algorithm which he calls Split Layout. It has the nice properties of having a good aspect ratio (most cells in the treemap are approximately square) and it keeps the cells in roughly the original order. This later property is important as one thing I find distracting with tree diagrams is if the order of the objects in the tree keep changing.
I also have an "on again/off again" relationship with the Catalogue of Life, which is potentially very useful, but seems determined to undermine this with some poor design decisions. But, I finally bit the bullet and extracted a complete classification from the 2008 edition of the Catalogue of Life. I downloaded an ISO image, burnt a CD, installed it on a Windows box (gack), grabbed the MySQL database files, and put those on my MacBook Air. Using some tools I developed for working with the NCBI taxonomy, I wanted to extract the tree from the taxa table, only to discover that this table isn't a tree. Not all the taxa in the table are flagged is_accepted_name, and if you remove those, then the remaining taxa don't form a tree. It's clear that some taxa have been orphaned when the table was created. For example, Enteromorpha flexuosa is not an accepted name, and is flagged as such in the taxa table, yet it is has four child taxa that are accepted (Enteromorpha flexuosa subsp. linziformis, Enteromorpha flexuosa subsp. biflagellata, Enteromorpha flexuosa subsp. pilifera, and Enteromorpha flexuosa forma submarina). These taxa are orphaned in the tree. Eventually I gave up trying to extract the tree using SQL, and had to traverse the entire structure starting at the root node. This extracts a tree, at the cost of the orphans. It appears that Catalogue of Life haven't checked whether there classification is, in fact, a tree (OK, technically it is a forest as it is a set of disjoint trees comprising the eight kingdoms CoL recognises, but I make it a tree by rooting it on a node called "life").
After much anguish, I have a tree. I then coded up Engdahl's algorithm, based on the pseudocode he provides on p. 31 of his thesis (I think there's a bug in his code as he doesn't deal with the case when the cell being partitioned is narrower than it is wide, but this was easy to fix). One thing I was keen to do is just use HTML, no SVG or Flash. Here's an example of the treemap, showing the eight kingdoms. Each taxon is drawn proportional to log10(n + 1), where n is the number of terminal taxa (i.e., species or below) in that taxon (the number of terminals is shown in each cell). The log scale was chosen to avoid mega-diverse groups crowding out the smaller taxa.
Animalia 892,966
Archaea 281
Bacteria 9,588
Chromista 6,855
Fungi 33,017
Plantae 206,843
Protozoa 6,435
Viruses 1,906
The live version is here. It's a bit crude (to go back up the tree just use your browser back button), but it's simple, and it's HTML. The underlying code is PHP, but it would be quite easy to convert this to Javascript to make a simple drop in widget. In addition to Björn Engdahl's algorithm, and the Catalogue of Life data, I should acknowledge Samson's code for generating colour gradients.
There are all sorts of things that could be done to improve this. One approach would be to include exemplar pictures of the taxa in each cell, to help navigate in unfamiliar taxa. Denise Green and Rebecca Shapley's Teaching with a visual tree of life report has some examples of this idea (see their p. 86), and Marcos Weskamp (author of the very cool newsmap) has done a mockup for EOL using Flash.
As to the treemap idea itself, there are some fun things which could be done with it. I'm not convinced that it is great for navigation. However, it is probably very useful for showing changes over time. For example, imagine making the State of Observed Species report dynamic. Take the uBio RSS feed for new names, classify the new names, then colour the treemap cells by the number of new names (in a sense, this is a taxonomic version of newsmap).
Something a little different, my dad's work features in the Winter 2008 issue of Art News New Zealand by Rob Garrett, who has a copy of the article on his web site. Dad designed eight stained glass windows for Westlake Boys High School in Auckland, New Zealand. Rob Garrett describes them thus:
The challenge when designing stained glass windows as a commemoration of 50 years of a high school is tough, no matter how open the brief. There is no unifying faith; no character-driven narrative; and no beginning, middle and end in the dramatic sense. There is only bewildering complexity; layer upon layer of possibility. [Dugald] Page has tackled this complexity head on by establishing unifying strands that weave through all eight panels: prismatic colour, seasonal change and the metaphor of metamorphosis. Stitched into these strands is a palimpsest of iconic symbols of science and technology, sport, the Pacific, discovery, historic events, literature and the arts.
They are gorgeous to look at (the picture below shows them before they were mounted in the school hall).
As much as I like the idea of a globally unique, resolvable identifier, my recent experience with JSTOR is making me wonder.
JSTOR has three identifiers for articles it archives, DOIs, SICIs, and stable URLs (the later being introduced with the new platform released April 4, 2008). Previously JSTOR would publish DOIs for many of its articles. However, not all of these work, and many are now embedded in the HTML (say, in Dublin Core meta elements) but not publicly displayed.
Journals in JSTOR have "moving walls" that define the time lag between the most current issue published and the content available in JSTOR. The majority of journals in the archive have moving walls of between 3 and 5 years, but publishers may elect walls anywhere from zero to 10 years.
Now, imagine that a publisher has an article on its web site, complete with a DOI, and that article is then add to JSTOR, but is still displayed on the publisher's site.
To make this concrete, consider the article by Baum et al. . On the InformaWorld site this is displayed with doi:10.1080/106351598260879. The same article is also in JSTOR, with the URL http://www.jstor.org/pss/2585367. No DOI is displayed on the page, but if you look at the HTML source, we find: <meta name="dc.Identifier" scheme="doi" content="10.2307/2585367">. The DOI prefix 10.2307 is used for all JSTOR DOIs, and some for Systematic Biology still work, e.g. 10.2307/2413524.
Now, what happens when the JSTOR moving wall overlaps with publisher's material? What happens if a publisher digitises back issues, then assigns them DOIs? Do the JSOR DOIs then die (as some of them seem to have already done)? And what happens to the poor sap like me, who has been linking to JSTOR DOIs in the naive belief that DOIs don't die?
Suddenly separating identity from resolution is starting to look very attractive...
Setting all these reservations and biases aside, the total number of living organisms that have received Latin binomial names is currently around 1.5 million or so. Amazingly, there is as yet no centralized computer index of these recorded species. It says a lot about intellectual fashions, and about our values, that we have a computerized catalog entry, along with many details, for each of several million books in the Library of Congress but no such catalog for the living species we share our world with. Such a catalog, with appropriately coded information about the habitat, geographical distribution, and characteristic abundance of the species in question (no matter how rough or impressionistic), would cost orders of magnitude less money than sequencing the human genome; I do not believe such a project is orders of magnitude less important. Without such a factual catalog, it is hard to unravel the pattems and processes that determine the biotic diversity of our planet.
<rant> BioOne sucks. Really, really, sucks. I have lost count of the number of times they break DOIs. These are supposed to be the gold standard globally unique identifier, and BioOne continually buggers them. For example, take this URL:
Note the doi=10.1600/02-14.1 bit at the end. If we go to the web page, we see this DOI displayed at the bottom of the page. Yet, when we resolve the DOI, we get the dreaded DOI Not Found error.
This.should.not.happen.
What is BioOne doing!? Now, it is possible that there's a problem with CrossRef, because Googling this paper I found it also lives on Ingenta with, wait for it, another DOI (doi:10.1600/036364404772973960).
This.should.not.happen.
BioOne and Ingenta are hosting the same paper, with different DOIs, only one of which is working. Will somebody please bang some heads together and sort this out! </rant>
Partly inspired by Pedro Beltra's post Open Science project on domain family expansion about using Google Code as a project management system, I've started to populate the iPhylo project. At this stage I'm uploading some scripts for parsing and extracting bibliographic records, and adding wiki pages describing how this is done, discussing different bibliographic identifiers, etc. The aim is to slowly document the background to all the harvesting and linking that I'm working on. Hence, the Google Code project will have documentation and data, not just code. The code for the web site won't go in for a while yet, it needs massive cleaning and tidying up.
MarkMail is a great tool for searching mail archives. Although focussing on software development projects, they are open to requests, so last week I asked if they could index TAXACOM. My pitch was that TAXACOM is a long running list full of interesting conversations, has been the subject of scholarly study (Christine Hine's book I mentioned earlier), and is topical given interest in biodiversity and the Encyclopedia of Life.
A broader theme might be databases, or the taxonomic impediment. Below is the chart of messages over time for the query "databases". You can "swipe" the mouse across the chart to select messages from a given time span.
I worry that I may end up spending more time playing with this than I should, but it's a neat tool. Hats off to Jason Hunter of MarkMail for adding TAXACOM so promptly.
The JIT is an advanced JavaScript infovis toolkit based on 5 papers about different information visualization techniques. The JIT implements advanced features of information visualization like Treemaps (with the slice and dice and squarified methods), an adapted visualization of trees based on the Spacetree, a focus+context technique to plot Hyperbolic Trees, and a radial layout of trees with advanced animations (RGraph).
Nicolas also links to a talk by Tamara Munzner, which I've embedded below to remind myself to watch it.
Trivial as this may seem, I'm trying to find out who designed this "Open Access" logo, and whether there are some original files for it. I've seen this logo (or variations on it) on the PLoS web site, the open access publisher Hindawi Publishing, and the Mac OS X program Papers uses it.
It's driving me nuts that I can't find the original. Other widely used logos typically have a site where a designer or organisation provides a bunch of versions in different formats, such as the Creative Commons symbols, the ubiquitous RSS feed icon, and other projects such as the Geotag icon. It's sometimes desirable to have different formats of an icon, and ideally have a vector-based version (e.g., in EPS or SVG) format that can be used to create images at different resolutions, and these projects provide these files.
Apart from the interesting fact that there doesn't seem to be a standard logo or symbol for Open Access, does anybody know where this logo came from?
The more I play with GBIF the more I come across some spectacular errors. Here's one small example of what can go wrong, and how easy it would be to fix at least some of the errors in GBIF. This is topical given that the recent review of EOL highlighted the importance of vetting and cleaning data.
Oops, the frog is found in the middle of the South Atlantic(!), and in Brazil(!?). These specimen records are provided by the MCZ, Harvard. Looking at the latitude and longitude co-ordinates, it's clear that there has been a comedy of errors. In the case of MCZ A-119852 the longitude is west instead of east, for MCZ A-119850 and MCZ A-119851 the latitude and longitudes have been swapped, and the longitude is west instead of east (again). If we make these changes, the specimens go back to Madagascar (the rectangle on the SVG map below). If you don't see the map, use a decent web browser such as Safari 3 or Firefox 2. If you must use Internet Explorer, grab the RENESIS player.
Interestingly the DiGIR records all list the country as Madagascar, so for any specimen in GBIF it would be trivial to test whether:
do the co-ordinates for the specimen fall inside the bounding box for the country?
if not, will they if we change sign (i.e., hemispheres) and/or swap latitude and longitude?
These would be trivial things to do, and would probably catch a lot of the more egregious errors in GBIF.
Fixing errors What will be interesting is whether these records will be fixed. I have sent feedback via GBIF's web site, as well as sending an email to the MCZ. I'll let readers know what happens.
Ground truth
Lastly, those interested in the frog itself may find the iSpecies search frustrating as the link returned by Google Scholar leads to a page in Ingenta saying:
This title is now published by Blackwell Publishing and can be found here www.ingentaconnect.com/content/bsc/zoj
Nope, the paper in question is actually at ScienceDirect (doi:10.1006/zjls.1995.0040). This paper describes the species, and gives the latitude and longitude of the collection localities (correctly).
I started this blog with the goal of documenting my own efforts to make a database of evolutionary trees, based on ideas sketched in hdl:10.1038/npre.2007.1028.1. I've felt that the major task is link phylogenies to other information, such as taxon names, specimens, localities, images, publications, etc. That is, to embed trees in a broader context. Discovering how to engage with that broader context led to a bunch of experiments, toys, and diversions:
iSpecies, a toy to aggregate information on a species.
Semant, experiments with RDF and triple stores (AKA the Semantic Web).
bioGUID, an attempt to make identifiers resolvable, with an increasing focus on developing an OpenURL resolver for biodiversity literature.
iSpecies and bioGUID are still operational, but the ant work fell victim to server crashes, and a growing frustration with the limitations of triple stores. Blogs for all three projects document their histories: iSpecies, Semant, and bioGUID. In a sense, these blogs document the steps along the way to iPhylo.
Based on this experience, I started again with what I've previously referred to as a database of everything. The first public demo is online at iphylo.org/~rpage/demo1. It's very crude, but may give a sense of what I'm trying to do.
The goal of iPhylo is to treat biodiversity objects as equal citizens. Each object has a unique identifier, associated metadata, and is linked to other objects (for example, a specimen is linked to sequences, sequences are linked to publications, etc.). By following the links it is possible to generate new views on existing information, such as a map for study that doesn't have any maps. For example, below is a map generated for Brady et al. (doi:10.1073/pnas.0605858103), based on links between sequences and specimens (if you can't see the map you need a SVG-capable web browser, such as Safari 3 or Firefox 2).
Ironically there are no phylogenies yet. At this stage I'm trying to link the bits together.
How does it work? More on this later. Briefly, iPhylo uses a entity-attribute-value model database to store objects and their relationships. Like bioGUID, iPhylo relies on a suite of web services (most external, some I've developed locally) to locate and resolve identifiers. iPhylo resolves identifiers for PubMed records, GenBank sequences, museum specimens, publications, etc. and adds the associated metadata to a local database. Wherever possible it resolves any links in the metadata (e.g., if a GenBank record mentions a specimen, iPhylo will try and retrieve information on that specimen). When you view an object in iPhylo, these links are displayed. iPhylo will also try and convert bibliographic records to identifiers (such as DOIs) if no identifiers are provided, and also extracts georeferences for specimens and sequences, either from original records or by using a georeferencing service. Taxonomic names are resolved using uBio, and are treated as "tags."
At present iPhylo is being populated by various scripts, there is no facility for users to add data. This is something I will add in the future.
Getting the data One of the biggest challenges is getting data (or, to be more precise, figuring out how to harvest available data). iPhylo builds on code for bioGUID. I've also been exploring bulk harvesting data sources. Sometimes this is easy. Many sequences in Genbank are linked to records in PubMed, so if you know the Pubmed id for a paper, you can harvest the sequences. For example, even though the Bulletin of the American Museum of Natural History isn't indexed in PubMed, it is possible to retrieve all the sequences GenBank records as being from papers published in the Bulletin. You can retrieve this list in XML form by clicking here.
Why?
There are all sorts of things which could be done with this. For example, by linking objects together we can also track the provenance of data, and ultimately build "citation networks" of specimens, sequences, etc. For background see my paper on "Biodiversity informatics: the challenge of linking data and the role of shared identifiers" (doi:10.1093/bib/bbn022, preprint at hdl:10101/npre.2008.1760.1).
As mentioned above, we can generate maps even if the original study didn't include one (by following the links). Given that we can geotag studies, this opens up the possibility to query studies spatially. For example, this study on bats and this study on rodents deal with taxa with very similar distributions. A spatial query could find these easily. Imagine interested in, say, Madagascar, and being able to find phylogenetic studies , even if the title and abstract of the paper don't mention Madagascar by name.
There also potential to clean data. One of the first studies I uploaded is Grant et al.'s study of dart-poison frogs. The map for this study shows a outlying point in California:
This point is MVZ 199828, which is a specimen of the salamander Aneides flavipunctatus. In GenBank, MVZ 199828 is listed as the voucher for seven sequences from the frog Mannophryne trinitatis. Oops. A quick iSpecies search, and a click on the GBIF map reveals that there is a specimen MVZ 199838 of Mannophryne trinitatis. i suspect that this is the true voucher for these sequences, and the GenBank records contain a typo.
Future This is all still very, very crude. The demo is slow, the queries aren't particularly clever, and I've probably gone overboard on using Javascript to populate the web pages. The real value isn't in the web pages, but the links between the data objects. This is my main focus -- extracting and adding links. For now the data is displayed but you can't edit it. However, this is coming. Very basic RDF and RSS feeds are available for each object, and fans of microformats will find some goodies, and sociologists of science may find some of the coauthorship graphs intriguing
The dissection of the colossal squid (Mesonychoteuthis hamiltoni) specimen from Antarctica has been getting a lot of coverage. Pangs of homesickness, especially seeing Steve O'Shea enthusing about the beast. Steve was a contemporary at Auckland Uni when I was a student. I remember him being deeply disappointed in me because I moved away from doing alpha taxonomy of crustaceans (describing taxa such as Pinnotheres atrinicola, right) to fussing with cladistics and computers. Looking at Steve on YouTube, I think it's clear who's having more fun. Maybe I should have stuck to taxonomy after all...
Last month I was at the MBL in Woods Hole, taking part in the review of the Biodiversity Informatics Group. BIG is responsible for the EOL web site. I chair the Informatics Advisory Group, which provides advice to BIG, and it was our task to produce an evaluation of where things stood. I've written a post on the Encyclopaedia of Life blog about some of the big challenges facing EOL as it moves into its second year.
Bit of a rarity these days. My paper on identifiers in biodiversity informatics, which I mentioned earlier when I deposited the preprint at Nature Precedings, has been published in Briefings in Bioinformatics (doi:10.1093/bib/bbn022).
Here's the abstract:
A major challenge facing biodiversity informatics is integrating data stored in widely distributed databases. Initial efforts have relied on taxonomic names as the shared identifier linking records in different databases. However, taxonomic names have limitations as identifiers, being neither stable nor globally unique, and the pace of molecular taxonomic and phylogenetic research means that a lot of information in public sequence databases is not linked to formal taxonomic names. This review explores the use of other identifiers, such as specimen codes and GenBank accession numbers, to link otherwise disconnected facts in different databases. The structure of these links can also be exploited using the PageRank algorithm to rank the results of searches on biodiversity databases. The key to rich integration is a commitment to deploy and reuse globally unique, shared identifiers [such as Digital Object Identifiers (DOIs) and Life Science Identifiers (LSIDs)], and the implementation of services that link those identifiers.
For some time now Google Code has been displaying the message:
The web interface for wiki content is currently READ-ONLY for maintenance. You may still add comments, and members may add, edit, or delete wiki pages via svn. Learn more.
This is a bit of a pain as I've recently put the code for my LSID tester into Google Code (the project is here). Since having a simple wiki is part of the attraction of Google Code, I decided to finally figure out how to add a wiki via Subversion. Turns out it is pretty straight forward. I created a folder called "wiki" and added a file with some wiki markup. I then added it to the repository
svn import -m "Trying to add wiki" wiki https://lsid-php.googlecode.com/svn/wiki/ --username USERNAME
(do this from the folder containing "wiki", not within the "wiki" folder itself). This adds the contents of the wiki folder to the Google Code repository. You can then check this out:
This probably seems obvious to many, but I'm used to CVS, having run a CVS repository since the late 1990's when Mike Charleston and I were working on TreeMap. I've been resisting moving to Subversion simply because of the hassle of learning stuff that doesn't actually make my life any easier. That said, Google Code is a nice way to host projects.
The manuscript for Briefings in Bioinformatics that I alluded to earlier has been accepted for publication. I've put a preprint up at Nature Preceding (hdl:10101/npre.2008.1760.1). The final version will appear in print later this year.
Resurrecting iSpecies after 
