Critical Assessment of Function Annotations: Lessons Learned and the Road Ahead





Iddo Friedberg , Wyatt ...
Upcoming SlideShare
Loading in …5

CAFA poster presented at CSHL Genome Informatics 2013


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

CAFA poster presented at CSHL Genome Informatics 2013

  1. 1. Critical Assessment of Function Annotations: Lessons Learned and the Road Ahead 1,* 2,3 4 4 5 Iddo Friedberg , Wyatt T Clark , Alexandra M Schnoes , Patricia C Babbitt , Sean D Mooney and Predrag Radivojac Introduction To understand and improve our ability to computationally annotate proteins, we are holding a series of multi-year challenges to the developers of function annotation programs. The rationale being that having these programs challenged and assessed will lead to understanding and improving predictive ability. The first critical assessment of Function Annotation (CAFA 1) was held over 2010-2011, involved 23 research groups and assessed the performance of 54 algorithms. CAFA 1 was structured as a time-challenge, where proteins which had no experimentally-validated function annotation were presented to the methods, and their function was predicted. Over the course of 10 months, some of these proteins gained experimental validation, and those were used as the final benchmark to assess program performance. Participating Methods Predictions on Human and Mouse Understanding protein function is a key component to understanding life at a molecular level. It is also important for understanding and treating human disease, since many conditions arise as a consequence of the loss or gain of protein function. 2 BPO MFO Database Bias Here we review CAFA 1, and introduce CAFA 2, which is taking place 2013-2014. There is extensive bias in experimentally validated annotations in Uniprot-GOA. The bias is contributed by high throughput experiments. Many HT experimental annotations create redundancies Case Study: hPNPase The CAFA Experiment: Generating Targets A circle represents the sum total of articles annotating each organism. Each colored arch is composed of all the proteins in a single article. A line is drawn between any two points on the circle if the proteins they represent have 100% sequence identity. A black line is drawn if they are annotated with a different ontology (for example, in one article the protein is annotated with the MFO, and in another article with BPO); a red line if they are annotated in the same ontology. Example: S. pombe is described by two articles, one with few protein (light arch on bottom) and one with many (dark arch encompassing most of circle). Many of the same proteins are annotated by both articles. New in CAFA 2 Assessing Method Performance Engaging more communities Human Phenotype Ontology Precision: pr = TP/(TP+FP) Recall: rc = TP/(TP+FN) pr +rc F1 = 2×( ) pr×rc Experimental Biologists Computer Scientists Cellular Component Ontology (a) Domain architecture of human PNPT1 gene according to the Pfam classification. For each domain, the numbers of different leaf terms (associated with any protein in Swiss-Prot database containing this domain are shown. (b) Molecular Function terms (six of which are leaves) associated with the human PNPT1 gene in Swiss-Prot as of December 2011. Colored circles represent the predicted terms for three representative methods as well as two baseline methods. The prediction threshold for each method was selected to correspond to the point in the precision-recall space that provides the maximum F-measure. J (blue), Jones-UCL; O (magenta), Team Orengo; d (navy blue), dcGO; B (green), BLAST; N (brown), Naive. Dashed lines indicate the presence of other terms between the source and destination nodes. Steering Committee Organizing Committee Data Wrangler Iddo Friedberg Michal Linial Mark Wass Sean D Mooney Predrag Radivojac Tal Ronen Oron Algorithms, Assessment methods Download poster Go to our website Computational Biologists CAFA 2 Assessor Patricia Babbitt Steven Brenner Christine Orengo Burkhard Rost Reassessing CAFA 1 methods Targets CAFA Targets & Ontologies Biocurators Anna Tramontano Author Affiliations 1. Miami University, Oxford OH 2. Indiana University, Bloomington, IN 3. Yale University, New Haven, MA 4. University of California San Francisco, CA 5. Buck Institute for Research on Aging, CA * References and more information CAFA: Radivojac et al (2013) Nature Methods doi:10.1038/nmeth.2340 Database Bias: Schnoes et al (2013) PLoS Computational Biology doi:10.1371/journal.pcbi.1003063