Note: CAGI occurred in Dec 2010, after I left Berkeley. Susanna Repo made the event happen and it would not have occurred without her.

1. ca·gey ˈkā-jē adjective
1: hesitant about committing oneself;
2a: wary of being trapped or deceived;
2b: marked by cleverness
CAGI (ˈkā-jē)
Critical Assessment of Genome Interpretation
A community experiment to evaluate phenotype prediction
Reece Hart (with Steven Brenner and John Moult)
QB3 / Center for Computational Biology
UC Berkeley
reece@berkeley.edu
Human Variome Project Meeting
Paris 2010-05-12

2. The Significance of
“Variants of Uncertain Significance”
“VUS – Variant of uncertain significance. A variation
in a genetic sequence whose association with
disease risk is unknown. Also called variant of
uncertain significance, variant of unknown
significance, and unclassified variant.”
http://www.cancer.gov/cancertopics/genetics-terms-alphalist
2

3. The long tail of rare diseases.
“A rare disease typically affects a patient
population estimated at fewer than 200,000 in
the U.S. There are more than 6,000 rare
diseases known today and they affect an
estimated 25 million persons in the U.S.”
NIH Office of Rare Diseases Research
http://rarediseases.info.nih.gov/
3

4. Interpretation of Unclassified Variants
a sampling of responses from genetic counselors
➢ Routinely used ➢ Selectively used
● dbSNP ● PharmGKB
● OMIM ● LSDBs
● GeneReviews ● Domain prediction
● PolyPhen ● Structure impact
● SIFT analysis
● PubMed ● Homology
● Mailing lists
4

5. Genome Variant Impact Prediction Tools
an incomplete list
Program URL
Align-GVGD http://agvgd.iarc.fr/
AutoMute http://proteins.gmu.edu/automute/
CUPSAT http://cupsat.tu-bs.de/
Dmutant http://sparks.informatics.iupui.edu/hzhou/mutation.html
nsSNPAnalyzer http://snpanalyzer.uthsc.edu/
PantherPSEC http://www.pantherdb.org/tools/csnpScoreForm.jsp
PhD-SNP http://gpcr.biocomp.unibo.it/~emidio/PhD-SNP/PhD-SNP.htm
Pmut http://mmb2.pcb.ub.es:8080/PMut/
PolyPhen http://coot.embl.de/PolyPhen/
SIFT http://sift.jcvi.org/
SNAP http://cubic.bioc.columbia.edu/services/snap/
SNP Function Pred. http://www.ensembl.org/ [N.B. login required]
SNPinfo / FuncPred http://snpinfo.niehs.nih.gov/snpfunc.htm
SNPs3D http://snps3d.org/
UMD-predictor http://www.umd.be/
5

6. Current methods are the tip of the iceberg.
m
C
protein non-protein repeats indels epigenetics
transcripts transcripts
~99%
~1%
6

7. Objectively Assessing Computational Predictions
➢ CASP – Structure prediction
➢ CAPRI – Protein-ligand docking
➢ EGASP – Encode Gene Annotation
➢ RGASP – RNA-Seq mapping
➢ DREAM – network model assessment
Data Acquisition
Publication
The Prediction Window
~1-12 months when unpublished
high-quality data are available
7

8. CAGI – Critical Assessment of Genome Interpretation
A community assessment of the state-of-the-art in phenotype prediction.
➢ Follow the successful critical
assessment framework:
● Solicit pre-publication genotype-
phenotype associations
● Provide genomic data to predictors
and collect their predictions
● Assess predictions against revealed
annotations, mechanisms, and
phenotypes
8

9. Sample Prediction Categories
Molecular Cellular Organismal
A A A
T T T
MTHFR mutants – Breast Cancer – PGP100 –
Yeast growth Segregation of rare Unpublished
rates with various variants among phenotypes from
MTHFR mutations 2500 cases and PGP100 project.
and [folate]. controls.
(Jasper Rine) (Sean Tavtigian) (George Church)
Please contact us if you have pre-publication genotype-phenotype
association data.
9

10. Census of Molecular Mechanisms
possible mechanisms of variant impact for WTCCC SNVs
Wellcome Trust Case Control Consortium Nature. 2007;447(7145):661-78. 10

11. Contributors, Predictors, Assessors
an incomplete list of participants
Gad Getz Sean Tavtigian Rachel Karchin Jasper Rine
Pauline Ng Marc Greenblatt Mauno Vihinen George Church
11

12. Sample CAGI Timeline
Dates are for illustration – exact dates have not been set.
05-24
05-31
06-07
06-14
06-21
06-28
08-23
08-30
09-06
09-13
09-20
09-27
11-22
11-29
12-06
12-13
12-20
12-27
05-03
05-10
05-17
07-05
07-12
07-19
07-26
08-02
08-09
08-16
10-04
10-11
10-18
10-25
11-01
11-08
11-15
01-03
01-10
01-17
01-24
01-31
Data Gathering
Prediction Season
Assessment
Key Dates
▲ finalize data sources ▲ workshop
▲ release prospectus / rules
▲ open participant registration
12

13. CAGI Summary
➢ CAGI will:
● objectively assess phenotype prediction methods
● inform future research directions
● introduce researchers in diverse fields
➢ CAGI is being planned for the end of 2010
or early 2011.
➢ Now seeking data contributors, assessors,
and predictors.
➢ Feedback is sought! reece@berkeley.edu
➢ See http://genomecommons.org/cagi for more
information. 13

14. 14

15. The Genome Commons:
A Flagship Project Within QB3
10 km
15

16. Program in Translational Genomics
Rasmus Nielseno
Michael I. Jordan
Ian Holmes
Kimmen Sjölander
Yun Song
Monty Slatkin
Terry Speed
Steven Hart
Reece Brenner Sandrine Dudoit Robert Nussbaum Mark van der Laan
Plant & Mol. Biology
Chief Scientist Biostatistics Chief, Medical Genetics Richard Karp
UC Berkeley
UC Berkeley UC Berkeley UCSF Bernd Sturmfels
Steven Evans
Elizabeth Purdom
Haiyan Huang
Peter Bickel
Susan Marqusee
Michael Eisen
Lisa Barcellos
Rachel Brem
Tom Alber
Jasper Rine Lior Pachter Bernie Lo
Genetics, Genomics & Dev Mathematics Director, Medical Ethics
Chair, Computational Biology Mol., Cell, Biol Department of Medicine
UC Berkeley UC Berkeley UCSF
16