Presentation by Justin Zook at GRC/GIAB ASHG 2017 workshop "Getting the most from the reference assembly and reference materials" on benchmarks for indels and structural variants.

1. Genome in a Bottle:
Developing benchmark sets for large indels and
structural variants
Justin Zook, Marc Salit, and the GIAB Consortium
NIST Genome-Scale Measurements Group
Joint Initiative for Metrology in Biology (JIMB)
Oct 16, 2017

2. Take-home Messages
• Genome in a Bottle is authoritatively characterizing human
genomes
• Current characterization enables benchmarking of “easier”
variants/regions in germline genomes
– Clinical validation
– Technology development, optimization, and demonstration
• Now working on difficult variants and regions
– Draft variant calls >=20bp available and feedback requested
– Many challenges remain and collaborations welcome!

3. Why are we doing this?
• Technologies evolving rapidly
• Different sequencing and
bioinformatics methods give
different results
• Now have concordance in easy
regions, but not in difficult
regions
• Challenge:
– How do we characterize 6 billion
bases in the genome with high
confidence?
O’Rawe et al, Genome Medicine, 2013
https://doi.org/10.1186/gm432

4. GIAB is evolving
2012
• No human
benchmark
calls available
• GIAB
Consortium
formed
2014
• Small variant
genotypes
for ~77% of
pilot genome
NA12878
2015
• NIST releases
first human
genome
Reference
Material
2016
• 4 new
genomes
• Small
variants for
90% of 5
genomes for
GRCh37/38
2017+
• Characteriz-
ing difficult
variants

5. Genome in a Bottle Consortium
Authoritative Characterization of Human Genomes
Sample
gDNA isolation
Library Prep
Sequencing
Alignment/Mapping
Variant Calling
Confidence Estimates
Downstream Analysis
• gDNA reference materials to
evaluate performance
• GIAB is developing:
– reference materials
– Reference data
– Methods
– Tools to calculate performance
metrics
genericmeasurementprocess
www.slideshare.net/genomeinabottle

6. Bringing Principles of Metrology
to the Genome
• Reference materials
– DNA in a tube from NIST
• Extensive state-of-the-art
characterization
• “Upgradable” as technology
develops
• Commercial innovation
– PGP genomes suitable for
commercial derived products
• Benchmarking tools and software
– with GA4GH
• Enhance new technologies

7. GIAB has characterized 5 human genome RMs
• Pilot genome
– NA12878
• PGP Human Genomes
– Ashkenazi Jewish son
– Ashkenazi Jewish trio
– Chinese son
• Parents also characterized
National I nstituteof S tandards & Technology
Report of I nvestigation
Reference Material 8391
Human DNA for Whole-Genome Variant Assessment
(Son of Eastern European Ashkenazim Jewish Ancestry)
This Reference Material (RM) is intended for validation, optimization, and process evaluation purposes. It consists
of a male whole human genome sample of Eastern European Ashkenazim Jewish ancestry, and it can be used to assess
performance of variant calling from genome sequencing. A unit of RM 8391 consists of a vial containing human
genomic DNA extracted from a single large growth of human lymphoblastoid cell line GM24385 from the Coriell
Institute for Medical Research (Camden, NJ). The vial contains approximately 10 µg of genomic DNA, with the peak
of the nominal length distribution longer than 48.5 kb, as referenced by Lambda DNA, and the DNA is in TE buffer
(10 mM TRIS, 1 mM EDTA, pH 8.0).
This material is intended for assessing performance of human genome sequencing variant calling by obtaining
estimates of true positives, false positives, true negatives, and false negatives. Sequencing applications could include
whole genome sequencing, whole exome sequencing, and more targeted sequencing such as gene panels. This
genomic DNA is intended to be analyzed in the same way as any other sample a lab would process and analyze
extracted DNA. Because the RM is extracted DNA, it is not useful for assessing pre-analytical steps such as DNA
extraction, but it does challenge sequencing library preparation, sequencing machines, and the bioinformatics steps of
mapping, alignment, and variant calling. This RM is not intended to assess subsequent bioinformatics steps such as
functional or clinical interpretation.
Information Values: Information values are provided for single nucleotide polymorphisms (SNPs), small insertions
and deletions (indels), and homozygous reference genotypes for approximately 88 % of the genome, using methods
similar to described in reference 1. An information value is considered to be a value that will be of interest and use to
the RM user, but insufficient information is available to assess the uncertainty associated with the value. We describe
and disseminate our best, most confident, estimate of the genotypes using the data and methods currently available.
These data and genomic characterizations will be maintained over time as new data accrue and measurement and
informatics methods become available. The information values are given as a variant call file (vcf) that contains the
high-confidence SNPs and small indels, as well as a tab-delimited “bed” file that describes the regions that are called
high-confidence. Information values cannot be used to establish metrological traceability. The files referenced in this
report are available at the Genome in a Bottle ftp site hosted by the National Center for Biotechnology Information
(NCBI). The Genome in a Bottle ftp site for the high-confidence vcf and high confidence regions is:

8. Integration of diverse data types and analyses
• Data publicly available
– Deep short reads
– Linked reads
– Long reads
– Optical/nanopore mapping
• Analyses
– Small variant calling
– SV calling
– Local and global assembly
Discover &
Refine
sequence-
resolved calls
from multiple
datasets &
analyses Compare
variant and
genotype calls
from different
methods
Evaluate/
genotype calls
with other
data
Identify
features
associated
with reliability
of calls from
each method
Form
benchmark
calls using
heuristics &
machine
learning
Compare
benchmarks
to high-
quality
callsets and
examine
differences

9. Paper describing data…
51 authors
14 institutions
12 datasets
7 genomes
Data described in ISA-tab

10. Evolution of high-confidence small variants
Calls
HC
Regions HC Calls
HC
indels
Concordant
with PG
NIST-
only in
beds
PG-only
in beds PG-only
Variants
Phased
v2.19 2.22 Gb 3153247 352937 3030703 87 404 1018795 0.3%
v3.2.2 2.53 Gb 3512990 335594 3391783 57 52 657715 3.9%
v3.3 2.57 Gb 3566076 358753 3441361 40 60 608137 8.8%
v3.3.2 2.58 Gb 3691156 487841 3529641 47 61 469202 99.6%
5-7
errors
in NIST
1-7
errors
in NIST
~2 FPs and ~2 FNs per million NIST variants in PG and NIST bed files

11. Global Alliance for Genomics and Health Benchmarking Task
Team
• Developed standardized
definitions for performance
metrics like TP, FP, and FN.
• Developing sophisticated
benchmarking tools
• Integrated into a single framework
with standardized inputs and
outputs
• Standardized bed files with
difficult genome contexts for
stratification
https://github.com/ga4gh/benchmarking-tools
Variant types can change when decomposing
or recomposing variants:
Complex variant:
chr1 201586350 CTCTCTCTCT CA
DEL + SNP:
chr1 201586350 CTCTCTCTCT C
chr1 201586359 T A
Credit: Peter Krusche, Illumina
GA4GH Benchmarking Team

12. Benchmarking Tools
Standardized comparison, counting, and stratification with
Hap.py + vcfeval
https://precision.fda.gov/https://github.com/ga4gh/benchmarking-tools

13. What are we accessing and what is still
challenging?
Type of variant Genome
context
Fraction
of variants
called*
Number of
variants
missing*
How to improve?
Simple SNPs Not repetitive ~97% >100k Machine learning
Simple indels Not repetitive ~93% >10k Machine learning
All variants Low
mappability
<30% >170k Use linked reads and long
reads
All variants Regions not in
GRCh37/38
0 >>100k??? De novo assembly; long reads
Small indels Tandem repeats
and
homopolymers
<50% >200k STR/homopolymer callers; long
reads; better handle complex
and compound variants
Indels 15-50bp All <25% >30k Assembly-based callers;
integrate larger variants
differently; long reads
Indels >50bp All <1% >20k
* Approximate values based on fraction of variants in GATKHC or FermiKit that are
inside v3.3.2 High-confidence regions

14. How can we extend our approach to structural
variants?
Similarities to small variants
• Collect callsets from multiple
technologies
• Compare callsets to find calls
supported by multiple technologies
Differences from small variants
• Callsets have limited sensitivity
• Variants are often imprecisely
characterized
– breakpoints, size, type, etc.
• Representation of variants is poorly
standardized, especially when complex
• Comparison tools in infancy

15. Our strategy
Collect many candidate calls for AJ Trio
• Gather candidate calls from a variety of
approaches
– Many technologies
• Short, linked, and long reads
• Optical and nanopore mapping
– Many approaches
• Small variant callers
• Structural variant callers
• Local and global de novo assemblies
• Community submitted >1 million calls
from 30+ methods using 5+ technologies
Refine/evaluate/genotype candidates
• Obtain sequence-resolved calls as
often as possible using assembly-based
approaches
• Compare sequence predictions of
candidate calls and merge similar calls
• Determine raw data’s support of each
sequence-resolved call and its
genotype

16. Evaluation/genotyping suite of methods
Current approaches
• svviz – maps reads to REF or ALT alleles
– PacBio
– Illumina paired end and mate-pair
– 10X haplotype-separated
• BioNano – compare size predictions
• Nabsys – evaluates large deletions
Future approaches
• Separate haplotypes on other data
types for svviz using whatshap
• Online manual curation of svviz, IGV,
dotplots, gEVAL, etc.
– Volunteers needed!
• PCR-Sanger targeted sequencing
– Collaborations welcome!

17. Integrating Sequence-resolved Calls >=20bp
>1 million calls from 30+ sequence-resolved callsets from 4 techs for
AJ Trio
>500k unique sequence-resolved calls
30k INS and 32k DEL with 2+ techs or 5+
callers predicting sequences <20%
different or BioNano/Nabsys support
28k INS and 29k DEL
genotyped by svviz in 1+
individuals
v0.4.0

18. Size Distribution of v0.4.0 Calls
Not Tandem Repeat
Tandem Repeat
Deletions Insertions
Alu
LINE
Alu
LINE

19. Sequence-resolved insertion size relative to BioNano

20. Insertion sequence prediction accuracy differs
between methods
Relative Distance from exact match
Illumina local
assembly
PacBio raw
read
PacBio consensus
assembly

21. Developing web-based Manual curation tools
https://github.com/svviz/svviz

22. Outstanding challenges and future work
• Large sequence-resolved insertions
• Many fewer multi-kb insertions
than multi-kb deletions
• Dense calls
• ~1/3 v0.4.0 calls are within 1kb of
another v0.4.0 call
• Sequence-resolved insertion size
doesn’t always match BioNano
• Phasing will be important for
these (e.g., with 10X, whatshap)
• Calls with inaccurate or incomplete
sequence change
• Exploring training a model to
predict sequence accuracy
• Homozygous Reference calls
• Can we definitively state there is
no SV in some regions?
• E.g., using diploid assembly?
• Benchmarking tool development
• How to compare SVs to a
benchmark?
• What performance metrics are
important?

23. New public data planned for late 2017
• PacBio Sequel sequencing of
GIAB Chinese trio
– Collaboration with Mt. Sinai
– 60x/30x/30x coverage planned
– Potentially >15kb N50 read length
• Oxford Nanopore sequencing of
Ashkenazim trio
– Collaboration with Nick Loman and
Matt Loose
– ~50x/25x/25x coverage planned
– Ultralong read sequencing (50-
100kb+ N50 read length)

24. New Samples
Additional ancestries
• Shorter term
– Use existing PGP individual samples
– Use existing integration pipeline
• Data-based selection
– Proportion of potential genomes from
different ancestries
• 3 to 8 new samples
• Longer term
– Recruit large family
– Recruit trios from other ancestry groups
Cancer samples
• Longer term
• Make PGP-consented tumor and
normal cell lines from same individual
• Select tumor with diversity of mutation
types

25. Take-home Messages
• Genome in a Bottle is authoritatively characterizing human
genomes
• Current characterization enables robust benchmarking of “easier”
variants/regions
• Actively working on difficult variants and regions
– Draft variant calls >=20bp available – feedback requested!
• New public long and ultralong read datasets coming!
• What can we help enable?
– Clinical applications – precision medicine
– Research applications – how to know new methods are measuring difficult
regions/variants well

26. Acknowledgements
• NIST/JIMB
– Marc Salit
– Jenny McDaniel
– Lindsay Vang
– David Catoe
– Lesley Chapman
• Genome in a Bottle Consortium
• GA4GH Benchmarking Team
• FDA

27. For More Information
www.genomeinabottle.org - sign up for general GIAB and Analysis Team google group emails
github.com/genome-in-a-bottle – Guide to GIAB data & ftp
www.slideshare.net/genomeinabottle
Data: http://www.nature.com/articles/sdata201625
Global Alliance Benchmarking Team
– https://github.com/ga4gh/benchmarking-tools
– precision.fda.gov – GA4GH benchmarking app
Biweekly Analysis Team calls (open to all)
– https://groups.google.com/forum/#!forum/giab-analysis-team
Public workshops
– Next workshop Jan 25-26, 2018 in Stanford, CA
– http://jimb.stanford.edu/giabworkshops for info and registration
NIST/JIMB postdoc opportunities available!
Justin Zook: jzook@nist.gov
Marc Salit: salit@nist.gov