Downloaded 12 times
Uploaded byGenomeInABottle
160628 giab for festival of genomics
This document summarizes the work of the Genome in a Bottle Consortium to develop reference materials and methods for benchmarking genome sequencing and variant calling. The Consortium has extensively characterized the genome of the NA12878 sample to create a "gold standard" set of variant calls that can be used to evaluate sequencing platforms and bioinformatics pipelines. They are also analyzing additional samples to generate reference materials that will enable standardized testing and validation of clinical genome sequencing. The Consortium aims to develop benchmarking tools and strategies with the Global Alliance for Genomics and Health to facilitate regulatory approval and clinical application of genome sequencing.
More Related Content
Similar to 160628 giab for festival of genomics
![Rheumatic Fever CASE PRESENTATION [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/casepresentationautosaved-251123182512-9d9b0da4-thumbnail.jpg?width=640&height=640&fit=bounds)
160628 giab for festival of genomics
- 1. So you’ve sequencedmy genome. How well did you do? Justin Zook NIST Genome-Scale Measurements Group June 28, 2016
- 2. Sequencing technologies and bioinformaticspipelines disagree O’Rawe et al. Genome Medicine 2013, 5:28
- 3. Sequencing technologies and bioinformaticspipelines disagree O’Rawe et al. Genome Medicine 2013, 5:28
- 4. Genome in aBottle Consortium Whole Genome Variant Calling Sample gDNA isolation Library Prep Sequencing Alignment/Mapping Variant Calling Confidence Estimates Downstream Analysis • gDNA reference materials to evaluate performance – materials certified for their variants against a reference sequence, with confidence estimates • established consortium to develop reference materials, data, methods, performance metrics • Characterized Pilot Genome NA12878 • Ashkenazim Trio, Asian Trio from PGP in process genericmeasurementprocess
- 5. Well-characterized, stable RMs •Obtain metrics for validation, QC, QA, PT • Determine sources and types of bias/error • Learn to resolve difficult structural variants • Improve reference genome assembly • Optimization • Enable regulated applications Comparison of SNP Calls for NA12878 on 2 platforms, 3 analysis methods
- 6. Bringing Principles ofMetrology to the Genome • Reference material – DNA in a tube you can buy from NIST – $45/ug • NA12878 as pilot sample • Extensive state-of-the-art characterization – as good as we can get for small variants – arbitrated “gold standard” calls for SNPs, small indels • “Upgradable” as technology develops • Analysis of PGP trios are ongoing – open project • PGP genomes suitable for commercial derived products • Developing benchmarking tools and software – with GA4GH • Samples being used to develop and demonstrate new technology – for instance, 10X Genomics
- 7.
- 8. Integration Methods toEstablish Reference Variant Calls Candidate variants Concordant variants Find characteristics of bias Arbitrate using evidence of bias Confidence Level Zook et al., Nature Biotechnology, 2014.
- 9. Integration Methods toEstablish Reference Variant Calls Candidate variants Concordant variants Find characteristics of bias Arbitrate using evidence of bias Confidence Level Zook et al., Nature Biotechnology, 2014.
- 10. So, how doesWGS make it into Regulated Clinical Applications? • FDA developing strategy to regulate NGS, which is a novel medical device “...this technology allows broad and indication-blind testing and is capable of generating vast amounts of data, both of which present issues that traditional regulatory approaches are not well-suited to address.” • FDA Workshops Feb ’15, Nov ’15 – strategy to rely on standards-based approaches, including reference materials… “need for reference materials for validation and proficiency testing… there is no substitute for having real samples.” FDA Whitepaper, Dec ‘14 GenomeWeb, Nov ‘15
- 11. Clinical Genome SequencingProcess Preanalytical Sequencing Sequence Bioinformatics Functional Variant Annotation Clinical Variant Knowledgebase Query Clinical Interpretation Reporting EHR Archival
- 12. What is thestandards architecture to demonstrate safety and efficacy? Preanalytical Sequencing Sequence Bioinformatics Functional Variant Annotation Clinical Variant Knowledgebase Query Clinical Interpretation Reporting EHR Archival
- 13. Analytical/Technical Performance Assessment Preanalytical Sequencing Sequence Bioinformatics Functional Variant Annotation ClinicalVariant Knowledgebase Query Clinical Interpretation Reporting EHR Archival
- 14. Global Alliance forGenomics and Health Benchmarking Task Team • Developed standardized definitions for performance metrics like TP, FP, and FN. • Developing sophisticated benchmarking tools • vcfeval – Len Trigg • hap.py – Peter Krusche • vgraph – Kevin Jacobs • Standardized bed files with difficult genome contexts for stratification Credit: GA4GH, Abby Beeler, Ellie Wood Stratification of FP Rates Higher FP rates at Tandem Repeats
- 15. Approaches to BenchmarkingVariant Calling • Well-characterized whole genome Reference Materials • Many samples characterized in clinically relevant regions • Synthetic DNA spike-ins • Cell lines with engineered mutations • Simulated reads • Modified real reads • Modified reference genomes • Confirming results found in real samples over time
- 16. Challenges in BenchmarkingVariant Calling • It is difficult to do robust benchmarking of tests designed to detect many analytes (e.g., many variants) • Easiest to benchmark only within high-confidence bed file, but… • Benchmark calls/regions tend to be biased towards easier variants and regions – Some clinical tests are enriched for difficult sites • Always manually inspect a subset of FPs/FNs • Stratification by variant type and region is important • Always calculate confidence intervals on performance metrics
- 17. How can weextend this approach to structural variants? Similarities to small variants • Collect callsets from multiple technologies • Compare callsets to find calls supported by multiple technologies Differences from small variants • Callsets generally are not sufficiently sensitive to assume that regions without calls are homozygous reference • Variants are often imprecisely characterized – breakpoints, size, type, etc. • Representation of variants is poorly standardized, especially when complex • Comparison tools in infancy
- 18. Callsets Contributed sofar Short reads • Illumina – Spiral Genetics – cortex – Commonlaw – MetaSV – Parliament/assembly – Parliament/assembly-force • Complete Genomics • CG-SV • CG-CNV • CG-vcfBeta Long reads and Linked reads • PacBio • CSHL-assembly • Sniffles • PBHoney-spots and –tails • Parliament/pacbio • Parliament/pacbio-force • MultibreakSV • smrt-sv.dip • Assemblytics-Falcon and-MHAP • Nanopore mapping • Nabsys force calls • optical mapping • BioNano with and without haplotype- aware assembly • 10X Genomics
- 19. Number of CallsSupported by 2 Technologies by Size Range <50bp 50-100bp 100-1000bp 1kb-3kb >3kb pre-filtered 2404 1307 2288 481 600 filtered 2325 1188 1875 379 341
- 20. Sensitivity to DraftBenchmark Calls <50bp 50-100bp 100-1000bp 1kb-3kb >3kb AssemblyticsFalcon 0% 55% 68% 59% 45% AssemblyticsMHAP 0% 51% 66% 56% 52% CGvcf 86% 20% 4% 0% 0% CGCNV 0% 0% 0% 0% 29% CGSV 0% 0% 39% 65% 56% CSHLassembly 0% 47% 62% 49% 42% sniffles 7% 28% 58% 59% 64% BioNano 0% 0% 2% 26% 37% Spiral 85% 44% 57% 38% 40% Cortex 39% 15% 7% 2% 0% CommonLaw 0% 0% 8% 47% 40% PBHoneySpots 0% 39% 63% 9% 0% PBHoneyTails 0% 0% 0% 31% 57% MetaSV 0% 0% 75% 74% 71% ParliamentPacBio 0% 0% 74% 75% 48% ParliamentAssembly 0% 0% 65% 44% 2% MultibreakSV 16% 66% 72% 59% 47% CNVnator 0% 0% 22% 71% 74% ParliamentPacBioForce 1% 45% 72% 31% 18% ParliamentAssemblyForce 0% 42% 63% 11% 2% BionanoHaplo 0% 0% 0% 36% 49% NabsysForce160405 0% 0% 5% 25% 28% smrtsvdip 0% 66% 77% 65% 55% fermikit 94% 86% 83% 59% 56%
- 21.
- 22. Concordance between technologies AllCalls High-confidence Calls
- 23. Acknowledgements • NIST – MarcSalit – Jenny McDaniel – Lindsay Vang – David Catoe • Genome in a Bottle Consortium • GA4GH Benchmarking Team • FDA – Liz Mansfield – Zivana Tevak – David Litwack
- 24. For More Information www.genomeinabottle.org- sign up for general GIAB and Analysis Team google group emails github.com/genome-in-a-bottle – Guide to GIAB data & ftp www.slideshare.net/genomeinabottle www.ncbi.nlm.nih.gov/variation/tools/get-rm/ - Get-RM Browser Data: http://biorxiv.org/content/early/2015/09/15/026468 Global Alliance Benchmarking Team – https://github.com/ga4gh/benchmarking-tools Twice yearly public workshops – Winter at Stanford University, California, USA – Summer at NIST, Maryland, USA NRC postdoc opportunities available! Justin Zook: jzook@nist.gov Marc Salit: salit@nist.gov