7. 7 / 27
Example: CASP
Critical Assessment of Structure Prediction
Problem: protein folding, or, predicting structure from sequence
Community: Structural biologists & bioinformaticians
Challenge set: proteins whose structures were experimentally determined. Varying difficulties
Assessment metrics: Root Mean Square Deviation, Q3, etc.
8. 8 / 27
Example: CAFA
Critical Assessment of Function Annotation
Predict sequence from Function
Community: function predictors, ontologists, experimental biologists
Problem: given a protein, which ontology terms are associated with it?
Challenge set: proteins that organizers know their functions, but predictors don’t
9. 9 / 27
Biomedical prediction problems
●
Protein Sequence to Structure
●
Protein Sequence to Function
●
Image to Phenotype
●
Phenotype to Genotype (disease)
●
Genotype to Phenotype
●
Symptoms to Disease
10. 10 / 27
A History of Critical Assessments
TREC: Text Retrieval Conference run by NIST
CASP: structure prediction
CAPRI: Protein Interaction
DREAM: umbrella – many biomedical
challenges
CAGI: genotype / phenotype
CACAO: educational
11. 11 / 27
Precision Recall on Ontologies
DNA binding
True function
Predicted function
12. 12 / 27
Precision Recall on Ontologies
Nucleic acid binding
DNA binding
Binding
True function
Predicted function
13. 13 / 27
Precision Recall on Ontologies
Nucleic acid binding
DNA binding
Binding
rRNA binding
RNA binding
True Positives : 2
False Positives: 2
False Negatives: 1 True function
Predicted function
15. 15 / 27
Adding Confidence as Threshold
AUTHOR Dr. Natalia Alianovna
MODEL 1
KEYWORDS sequence alignment.
T96060020120 GO:0008270 0.80
T96060020120 GO:0003700 0.80
T96060020120 GO:0006351 0.80
T96060020119 GO:0005730 0.01
T96060020119 GO:0003676 0.07
T96060020119 GO:0005622 0.07
T96060020119 GO:0046872 0.07
T96060020118 GO:0008270 0.75
T96060020118 GO:0006351 0.68
T96060020118 GO:0003677 0.67
T96060020118 GO:0005634 0.67
T96060020118 GO:0006355 0.55
T96060020118 GO:0003700 0.34
Protein
ID
GO
term
Confidence
16. 16 / 27
DREAM Digital Mammography
Challenge (2016-2017)
17. 17 / 27
DREAM Digital Mammography
Challenge (2016-2017)
640K
mammography
images
Patient
metadata
0 1
Sensitivity: TP/(TP+FN)
Specificity: TN/(TN+FP)
TP: patient sick, predicted sick
FP: patient healthy, predicted sick
TN: patient healthy, predicted
healthy
FN: patient sick, predicted healthy
Probability of cancer one year after?
18. 18 / 27
Are we improving?
https://www.ibm.com/blogs/research/2017/06/dream-challenge-results/
https://predictioncenter.org/
https://biofunctionprediction.org/
19. 19 / 27
Are we improving?
https://www.ibm.com/blogs/research/2017/06/dream-challenge-results/
https://predictioncenter.org/
https://biofunctionprediction.org/
22. 22 / 27
Gaming Metrics
Metric Claims to
measure
Used for How to game
h-index Individual
publication
impact
Promotion and
Tenure
Multi-author
papers
IQ test Intelligence Bragging rights Be born in the
right culture
SAT Success in
college
College
admissions
Study
Precision recall CAFA
performance
Assessing
function
predictions
Not telling you
23. 23 / 27
Gaming Metrics
Metric Claims to
measure
Used for How to game
h-index Individual
publication
impact
Promotion and
Tenure
Multi-author
papers
IQ test Intelligence Bragging rights Be born in the
right culture
SAT Success in
college
College
admissions
Study
Precision recall CAFA
performance
Assessing
function
predictions
Not telling you
24. 24 / 27
Gaming Metrics
Metric Claims to
measure
Used for How to game
h-index Individual
publication
impact
Promotion and
Tenure
Multi-author
papers
IQ test Intelligence Bragging rights Be born in the
right culture
SAT Success in
college
College
admissions
Study
Semantic
similarity
CAFA
performance
Assessing
function
predictions
Not telling you
25. 25 / 27
Gaming Metrics
Metric Claims to
measure
Used for How to game
h-index Individual
publication
impact
Promotion and
Tenure
Multi-author
papers
IQ test Intelligence Bragging rights Be born in the
right culture
SAT Success in
college
College
admissions
Study
Precision recall CAFA
performance
Assessing
function
predictions
Not telling you
26. 26 / 27
Gaming Metrics
Metric Claims to
measure
Used for How to game
h-index Individual
publication
impact
Promotion and
Tenure
Multi-author
papers
IQ test Intelligence Bragging rights Be born in the
right culture
SAT Success in
college
College
admissions
Study
Precision recall CAFA
performance
Assessing
function
predictions
Not telling you
27. 27 / 27
Thank you
●
Sage Bionetworks
– Lara Mangravite
●
CAFA
– Wyatt Clark, Indiana University (1,2)
– Yuxiang Jiang, Indiana University (2-4)
– Naihui Zhou, Iowa State University (3, 3.14)
– Tim Bergquist University of Washington (2-4)
– Predrag Radivojac (Northeastern University)
– Sean Mooney (University of Washington)
– Casey Greene (University of Colorado)
– Mark Wass (University of Kent)
– Kim Reynolds (University of Texas
Southwestern)
●
Sandra Orchard (EMBL-EBI)
●
Maria Martin (EMBL-EBI)
●
> 250 co-authors over the years