What can Community Challenges do for You?
•
0 likes•74 views
Talk on computational community challenges given at the Pacific Symposium of Biocomputing Jan 7, 2021
Recommended
Recommended
More Related Content
What's hot
Similar to What can Community Challenges do for You?
Similar to What can Community Challenges do for You? (20)
More from Iddo
More from Iddo (20)
Recently uploaded
Recently uploaded (20)
What can Community Challenges do for You?
- 1. 1 / 27 What Can Community Challenges do for You? Iddo Friedberg Iowa State University
- 2. 2 / 27 What Can Community Challenges do for You? Iddo Friedberg Iowa State University
- 3. 3 / 27 Science is Symbiotic Discovery Methods
- 4. 4 / 27 Science is Symbiotic
- 5. 5 / 27 There are many methods, which is best?
- 6. 6 / 27 What is “best”?
- 7. 7 / 27 Example: CASP Critical Assessment of Structure Prediction Problem: protein folding, or, predicting structure from sequence Community: Structural biologists & bioinformaticians Challenge set: proteins whose structures were experimentally determined. Varying difficulties Assessment metrics: Root Mean Square Deviation, Q3, etc.
- 8. 8 / 27 Example: CAFA Critical Assessment of Function Annotation Predict sequence from Function Community: function predictors, ontologists, experimental biologists Problem: given a protein, which ontology terms are associated with it? Challenge set: proteins that organizers know their functions, but predictors don’t
- 9. 9 / 27 Biomedical prediction problems ● Protein Sequence to Structure ● Protein Sequence to Function ● Image to Phenotype ● Phenotype to Genotype (disease) ● Genotype to Phenotype ● Symptoms to Disease
- 10. 10 / 27 A History of Critical Assessments TREC: Text Retrieval Conference run by NIST CASP: structure prediction CAPRI: Protein Interaction DREAM: umbrella – many biomedical challenges CAGI: genotype / phenotype CACAO: educational
- 11. 11 / 27 Precision Recall on Ontologies DNA binding True function Predicted function
- 12. 12 / 27 Precision Recall on Ontologies Nucleic acid binding DNA binding Binding True function Predicted function
- 13. 13 / 27 Precision Recall on Ontologies Nucleic acid binding DNA binding Binding rRNA binding RNA binding True Positives : 2 False Positives: 2 False Negatives: 1 True function Predicted function
- 14. 14 / 27 Precision Recall 00 1 1 Recall Precision 2/4 2/3
- 15. 15 / 27 Adding Confidence as Threshold AUTHOR Dr. Natalia Alianovna MODEL 1 KEYWORDS sequence alignment. T96060020120 GO:0008270 0.80 T96060020120 GO:0003700 0.80 T96060020120 GO:0006351 0.80 T96060020119 GO:0005730 0.01 T96060020119 GO:0003676 0.07 T96060020119 GO:0005622 0.07 T96060020119 GO:0046872 0.07 T96060020118 GO:0008270 0.75 T96060020118 GO:0006351 0.68 T96060020118 GO:0003677 0.67 T96060020118 GO:0005634 0.67 T96060020118 GO:0006355 0.55 T96060020118 GO:0003700 0.34 Protein ID GO term Confidence
- 16. 16 / 27 DREAM Digital Mammography Challenge (2016-2017)
- 17. 17 / 27 DREAM Digital Mammography Challenge (2016-2017) 640K mammography images Patient metadata 0 1 Sensitivity: TP/(TP+FN) Specificity: TN/(TN+FP) TP: patient sick, predicted sick FP: patient healthy, predicted sick TN: patient healthy, predicted healthy FN: patient sick, predicted healthy Probability of cancer one year after?
- 18. 18 / 27 Are we improving? https://www.ibm.com/blogs/research/2017/06/dream-challenge-results/ https://predictioncenter.org/ https://biofunctionprediction.org/
- 19. 19 / 27 Are we improving? https://www.ibm.com/blogs/research/2017/06/dream-challenge-results/ https://predictioncenter.org/ https://biofunctionprediction.org/
- 20. 20 / 27 Goodhart’s Law
- 21. 21 / 27 Gaming Metrics
- 22. 22 / 27 Gaming Metrics Metric Claims to measure Used for How to game h-index Individual publication impact Promotion and Tenure Multi-author papers IQ test Intelligence Bragging rights Be born in the right culture SAT Success in college College admissions Study Precision recall CAFA performance Assessing function predictions Not telling you
- 23. 23 / 27 Gaming Metrics Metric Claims to measure Used for How to game h-index Individual publication impact Promotion and Tenure Multi-author papers IQ test Intelligence Bragging rights Be born in the right culture SAT Success in college College admissions Study Precision recall CAFA performance Assessing function predictions Not telling you
- 24. 24 / 27 Gaming Metrics Metric Claims to measure Used for How to game h-index Individual publication impact Promotion and Tenure Multi-author papers IQ test Intelligence Bragging rights Be born in the right culture SAT Success in college College admissions Study Semantic similarity CAFA performance Assessing function predictions Not telling you
- 25. 25 / 27 Gaming Metrics Metric Claims to measure Used for How to game h-index Individual publication impact Promotion and Tenure Multi-author papers IQ test Intelligence Bragging rights Be born in the right culture SAT Success in college College admissions Study Precision recall CAFA performance Assessing function predictions Not telling you
- 26. 26 / 27 Gaming Metrics Metric Claims to measure Used for How to game h-index Individual publication impact Promotion and Tenure Multi-author papers IQ test Intelligence Bragging rights Be born in the right culture SAT Success in college College admissions Study Precision recall CAFA performance Assessing function predictions Not telling you
- 27. 27 / 27 Thank you ● Sage Bionetworks – Lara Mangravite ● CAFA – Wyatt Clark, Indiana University (1,2) – Yuxiang Jiang, Indiana University (2-4) – Naihui Zhou, Iowa State University (3, 3.14) – Tim Bergquist University of Washington (2-4) – Predrag Radivojac (Northeastern University) – Sean Mooney (University of Washington) – Casey Greene (University of Colorado) – Mark Wass (University of Kent) – Kim Reynolds (University of Texas Southwestern) ● Sandra Orchard (EMBL-EBI) ● Maria Martin (EMBL-EBI) ● > 250 co-authors over the years