Giab poster structural variants ashg 2018

1. Introduction
How well can you detect structural variants: Towards a standard
framework to benchmark human structural variation
Justin Zook,1 Lesley Chapman,1 Nancy Hansen,3 Fritz J. Sedlazeck,4 Aaron Wenger,5 Adam English,6 Chunlin Xiao,7 John Oliver,8 Alex Hastie,9 Ian Fiddes,10
Alvaro Barrio,10 Tobias Marschall,11 Mark Chaisson,12 John Farrell,13 Andrew Carroll,14 Marc Salit,2 and the Genome in a Bottle Consortium
(1) National Institute of Standards and Technology; (2) Joint Initiative for Metrology in Biology; (3) NHGRI/NIH; (4) Baylor College of Medicine; (5) Pacific Biosciences; (6) Spiral Genetics;
(7) NCBI/NIH; (8) Nabsys; (9) BioNano Genomics; (10) 10x Genomics; (11) Max Planck Institute; (12) University of Southern California; (13) Boston University Medical School; (15) DNAnexus
• NIST has hosted the Genome in a Bottle Consortium to develop
authoritatively-characterized, human genome Reference Materials
that are an enduring resource for benchmarking variant calls
Integrating data to form benchmark calls
Ongoing and Future GIAB Work
• Using long & linked reads in difficult-to-map regions
• Improved benchmarks for homopolymers and long repeats
• Complex and clustered variants
• New collaborations to characterize difficult regions and
variants in these genomes are welcome! Email jzook@nist.gov
Crowd-sourced manual curation vs. benchmark set
Benchmark calls and Benchmark regions
Zook et al., Scientific Data, 2016.
ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data
Our benchmark sets are useful in evaluating
multiple technologies
2012
• No human
benchmark
calls available
• GIAB
Consortium
formed
2014
• Small variant
genotypes
for ~77% of
pilot genome
NA12878
2015
• NIST releases
first human
genome
Reference
Material
2016
• 4 new
genomes
• Small
variants for
~90% of 5
genomes for
GRCh37/38
2017+
• Characteriz-
ing difficult
variants and
regions
Discover
• Discover sequence-resolved calls with multiple methods
• Using indel and SV callers (assembly-based and split-read-based)
• From short, long, and linked reads
Compare
inputs
• Compare variant and genotype calls from different methods
• Using SVanalyzer to measure similarity of predicted sequence change
• Accounts for differing representations in tandem repeats
Evaluate/
genotype
• svviz – aligns reads to REF or ALT to produce GT and visualization
• BioNano – compare size of discovered calls to our INS/DEL >1kb
• Nabsys – aligns electronic maps to REF or ALT and uses an SVM to classify
deletions >300bp as true or false
Identify
features
• Identify features associated with reliability of calls from each method
• e.g., counts of reads supporting REF & ALT, tandem repeats, segmental
duplications
Form
bench-
mark
• Currently using heuristics; e.g., support from 2+ technologies or 5+ callsets
• Exploring using machine learning with many more features svviz, and using
training data from manual curation
Compare
• Compare high-quality callsets to benchmark set and curate differences
• Receive feedback from the community and iterate to improve calls
• Goal: Trustworthy benchmarking outputs (False Positives & False
Negatives)
Benchmark set and README at tinyurl.com/GIABSV06
• Goal: When comparing any callset
to our Tier 1 vcf within the Tier 1
bed, most putative FPs and FNs
should be errors in the tested callset
• We benchmarked several callsets
from assembly-based and non-
assembly-based methods with short
and long reads.
• Upon manual curation, the majority
of most FPs and FNs were errors in
the tested callset
• One exception is FP insertions
from pbsv, suggesting we may
miss ~5% of true insertions
github.com/nspies/svviz2
50 to 1000 bp
• Short reads
• Illumina paired end
• Illumina 6kb mate-pair
• Complete Genomics
• Linked reads
• 10X Genomics Chromium
• Long reads
• Normal PacBio
• 10kb and 15kb CCS PacBio
• “Ultralong” Oxford Nanopore
• Marker-based
• BioNano Genomics
• Nabsys
Public data for Ashkenazi Trio
New!
svcurator.com
• Tier 1 regions contain 2.68 Gbp with 11,869 isolated SVs >49bp
• Tier 1 calls meet the criteria:
• Discovered by 2+ techs or 5+ callers
• Confirmed and genotyped by long reads
• Not disproven by any technology
• Clusters of calls within 1000bp are excluded
• Regions around calls 20-49bp are excluded
Blue - clustered calls
Red - isolated calls
1kbp to 10kbp
Alu Alu
LINE LINE
• Candidates examined by
11 curators on average
• 627/635 consensus
manual curations agreed
with v0.6 genotype in
benchmark regions
• Most “discordant”
sites related to
inclusion of 20-49bp
indels in curation
github.com/spiralgenetics/truvari
FNs FPs