Establishing validity, reproducibility, and utility of highly scalable genetic tests: Challenges and Solutions - Stephen Lincoln

© 2015 Invitae Corporation. All Rights Reserved. | 1
Improving the accuracy and
reproducibility of clinical
genetic tests
STEPHEN E. LINCOLN
HVP Meeting
May 2016

2 © 2015 Invitae Corporation. All Rights Reserved.
1. Analytic Performance
– Analytic Sensitivity: Can you find the DNA variants
a patient has?
– Analytic Specificity: Of the variants you find, are
they really there?
1. Clinical Performance
– Clinical Sensitivity: Can you find the complete
spectrum of clinically relevant variants for a patient’s
condition?
– Classification: Can you interpret the medical impact
of these variants appropriately?
Questions about any genetic test…

1. Analytical Performance

Lions and Tigers and Bears, Oh My!

Clinical utility of panel testing for HBOC
Desmond et al., JAMA Oncol. 2015
Swisher, JAMA Oncol. 2015

Companion technical paper
A systematic comparison of traditional and
multigene panel testing for hereditary breast and
ovarian cancer in more than 1000 patients
Stephen E. Lincoln1, Yuya Kobayashi1, Michael J. Anderson1, Shan Yang1,
Andrea J. Desmond2, Meredith A. Mills3, Geoffrey B. Nilsen1, Kevin B. Jacobs1,
Federico A. Monzon1, Allison W. Kurian3, James M. Ford3, Leif W. Ellisen2,4
1. Invitae, San Francisco, CA
2. Massachusetts General Hospital Cancer Center, Boston,
MA
3. Stanford University School of Medicine, Palo Alto, CA
4. Harvard Medical School, Boston, MA
Lincoln et al., J Mol Diag 2015

A Significant Fraction of Pathogenic Variants in
Representative Clinical Cases are Technically Challenging
Pathogenic variants among 1062 clinical cases, by type:
Small Indel
i.e. CNVs or “Large
Rearrangements” (Myriad term)
“Hard stuff” (Steve’s term)

Analysis of 30,000 sequential cases with clinical testing
across over 1,000 known disease genes (in total)…
Percentage Pathogenic Variant Type
2.9% Single-exon del/dups (CNVs)
1.8% Large indel or complex sequence change
5.8% In high-GC, low-complexity, or poorly mapabie
regions of the genome
Of the last 5,000 patients with pathogenic findings…
Rebecca Truty, Invitae

 BRCA2: c.9342_9343insAlu
– Novel Alu insertion (at the time we first saw it)
– Not the Portuguese founder mutation (that’s c.156_157insAlu)
 BRCA2: c.9203_9328del126
– Very large indel (or a very small CNV)
 MSH2: c.942+3A>T
– Splice-affecting mutation next to 25 bp homopolymer-A
 MSH2: inv exons 1-7
– Breakpoint detection in intronic regions
 CDKN2A: c.9_32dup24
– Third copy of a wilt-type tandem duplication in 5’ CpG island
 And many other examples...
The spectrum of pathogenic variants in patients includes
cases that the typical NGS workflow does not handle

Analytic validity in N=1105 individuals
NGS vs. Traditional Methods
In 1105 Individuals
Sensitivity 100.0%
Specificity 100.0%
750 Comparable Variants
(Pathogenic or Otherwise)
Sequence Changes 721
Del/dups (CNVs) 29
Single Nucleotide 549
Small Indel 156
Large Indel* 13
Complex** 6
To achieve this, specialized NGS
methods, biochemical and
bioinformatics, are required.
The other challenging classes of
variation tend to be not well
represented in other validation
studies.
* Large Indel is deletion≥10bp, insertion≥5bp
** Complex includes homo-polymer associated
variants, indels in low-complexity sequence, short
range haplotypes, etc.

NGS vs. Traditional Methods in
250 Patients
Sensitivity 100.0%
Specificity 99.99%
3025 Variants Appropriate to
Measure Sensitivity
Sequence Changes 3021
Del/dups (CNVs) 4 ?
Chong et al., PLOS One 2014
Single Nucleotide 3010
Small Indel 11 ?
Large Indel * 0
Complex ** 0
* Large Indel is deletion≥10bp, insertion≥5bp
** Complex includes homo-polymer associated
variants, indels in low-complexity sequence, short
range haplotypes, etc.
Note: These numbers assume that
Table 2 in the paper includes all of
the indels and del/dups, and that
the remaining variants in the are
benign SNPs. This is not entirely
clear to us in the paper text.
A different lab’s study: Analytic validity in 250 individuals

What is one to do?
 Build Awareness
 Develop/Implement Methods
 Improve Validation Resources
 Strengthen Validation Standards

2. Clinical Interpretation

ClinVar top submitters
www.ncbi.nlm.nih.gov/clinvar > Statistics > Submitter List (as of May 3, 2016)

ClinVar entry for BRCA1:c.4410A>T

NM_007294.3(BRCA1):c.4410A>T (p.Glu1470Asp)

This sequence change replaces glutamic acid with aspartic acid at
codon 1470 of the BRCA1 protein (p.Glu1470Asp). The glutamic acid
residue is moderately conserved and there is a small physicochemical
difference between glutamic acid and aspartic acid. This variant has
been published in the literature and is present in population databases
(rs80357075, 0.05%). This variant has been reported in individuals
affected with breast cancer. Due to limited data, whether this variant
segregates with disease remains uncertain (PMID: 20104584,
22682623). This variant has also been reported in a healthy individual
not affected with cancer (PMID: 24728327). ClinVar contains entries for
this variant (RCV000131557, RCV000077572, RCV000074594,
RCV000120257). Algorithms developed to predict the effect of missense
changes on protein structure and function (SIFT, PolyPhen-2, Align-
GVGD) all suggest that this variant is likely to be tolerated, but these
predictions have not been confirmed by published functional studies. In
summary, this is a rare missense change that is not predicted to affect
protein function or cause disease. However the evidence is insufficient
at this time to prove that conclusively. Therefore, it has been classified
as a Variant of Uncertain Significance.

Vail et al., J Community Genet 2015
Finding: 3-14% disagreement rate between databases
(BIC, HGMD, LOVD, UMD, and ClinVar)

The disagreement in public databases
“precludes their wider use in clinical practice”
Finding: 3-14% disagreement rate between databases
(BIC, HGMD, LOVD, UMD, and ClinVar)

The disagreement in public databases
“precludes their wider use in clinical practice”
“Public databases [are] fraught with errors”
“Interpretation accuracy impossible with public
databases”
Analyst Day Presentation, September 2015
investor.myriad.com > Events & Presentations > Financial Events
Accessed 2/1/2016

This issue can be misrepresented to clinicians…
• Experienced and responsible lab directors
never simply copy variant classifications from
any public database
• Instead, they critically evaluate underlying
evidence and report classifications following
established guidelines
Question: What is the clinical impact of
discordance in public databases?

BRCA1/2 variant classification concordance
Positive vs. not positive
result for BRCA1/2
Agree 99.8%
Disagree 0.2%
% of patients with one or
more VUS in BRCA1/2
Our test 4.1%
Traditional test 3.2%
975 patients with both Myriad Genetics and independent test results.
Variants uncovered by the independent test were blindly classified:
• Using only publicly available resources (literature and databases)
• Following a system based on the ACMG/AMP 2015 guidelines
All amended reports provided to clinical sites were incorporated, so the
Myriad variant classifications were up to date

Positive vs. not positive results
Pathogenic
(P)
Likely
Pathogenic
(LP)
VUS
Likely
Benign
(LB)
Benign
(B)
P
Concordant Discordant
LP
VUS
Discordant ConcordantLB
B
Count differences that could substantially
change patient management decisions

 Inclusion Criteria were ClinVar submitters that:
– Are an established and licensed diagnostic laboratory
• or are submitting data from such a lab
– Submitted at least 200 classified variants in BRCA1/2
– Most classifications were from the last 5 years
• Using “last evaluation date”, not “submission date” in ClinVar
 Thus we excluded data from:
– Research projects, software vendors
– Smaller labs, consortia (e.g. ENIGMA)
– Old sources (e.g. BIC)
• Mostly pre-2007 Myriad Genetics data
Examining the larger data set in ClinVar
Shan Yang, Invitae

ClinVar based data set
Shan Yang, Invitae
Name Classified
Variants
Comparable
Variants
Submitter Name in ClinVar
Ambry 2793 1502 AMBRY GENETICS
Myriad
(via SCRP)
2067 1184 SHARING CLINICAL REPORTS PROJECT (SCRP)
Invitae 1479 1082 INVITAE
GeneDx 1214 937 GENEDX
Counsyl 272 256 COUNSYL
CHEO 257 216 MOLECULAR GENETICS DIAGNOSTIC LABORATORY,
CHILDREN'S HOSPITAL OF EASTERN ONTARIO
Emory 203 183 EMORY GENETICS LABORATORY
Total 4725 1800 Non-redundant list from the above
Data from ≥2 labs

BRCA Exchange
Benedict Paten, Melissa Cline, Molly Zhang, David Haussler, et al., UCSC
www.brcaexchange.org
• Replicated our results
• Open-source code to help mine ClinVar
• Integrating many other sources of BRCA1/2 data

Ambry Invitae GeneDx Counsyl CHEO Emory
Myriad
via SCRP
98.7%
939/951
97.9% - 99.3%
99.2%
619/624
98.3% - 99.7%
99.5%
569/572
98.6% - 99.9%
99.4%
171/172
97.3% - 100%
99.5%
139/142
94.5% - 99.4%
97.2%
103/106
92.6% - 99.2%
Ambry
99.2%
860/867
98.4% - 99.6%
99.6%
780/783
99.0% - 99.9%
99.6%
223/224
97.9% - 100%
98.3%
176/179
95.6% - 99.5%
98.8%
161/163
96.1% - 99.7%
Invitae
99.8%
593/594
99.2% - 100%
99.1%
214/216
97.1% - 99.8%
98.2%
161/164
95.2% - 99.5%
99.3%
144/145
96.8% - 100%
GeneDx
99.5%
221/222
97.9% - 100%
97.9%
138/141
94.4% - 99.4%
99.3%
149/150
96.9% - 100%
Counsyl
Concordance
Concordant/All
Conf. Int.
100%
82/82
97.0% - 100%
100%
105/105
97.6% - 100%
CHEO
98.3%
57/58
92.2% - 99.9%

Ambry Invitae GeneDx Counsyl CHEO Emory
Myriad
via SCRP
98.7%
939/951
97.9% - 99.3%
99.2%
619/624
98.3% - 99.7%
99.5%
569/572
98.6% - 99.9%
99.4%
171/172
97.3% - 100%
99.5%
139/142
94.5% - 99.4%
97.2%
103/106
92.6% - 99.2%
Ambry
99.2%
860/867
98.4% - 99.6%
99.6%
780/783
99.0% - 99.9%
99.6%
223/224
97.9% - 100%
98.3%
176/179
95.6% - 99.5%
98.8%
161/163
96.1% - 99.7%
Invitae
99.8%
593/594
99.2% - 100%
99.1%
214/216
97.1% - 99.8%
98.2%
161/164
95.2% - 99.5%
99.3%
144/145
96.8% - 100%
GeneDx
99.5%
221/222
97.9% - 100%
97.9%
138/141
94.4% - 99.4%
99.3%
149/150
96.9% - 100%
Counsyl
100%
82/82
97.0% - 100%
100%
105/105
97.6% - 100%
CHEO
98.3%
57/58
92.2% - 99.9%
On a per-variant basis:
Pairwise concordance: 97.2% - 100.0%
Overall concordance: 98.5%
Only 27/1800 variants have a significant
classification discordance between any
two reporting laboratories

How many individuals does this represent?
050010001500
Patients
UniqueVariants
0 5,000 10,000 15,000 20,000
≈22,000 patients give
≈1800 non-redundant variants
Melissa Cline, UCSC

Did SCRP data bias results from other labs?
Other lab classifications Concordance
Pre-dating SCRP release 99.0%
Post-dating SCRP release 98.9%
Apparently not
Compare ClinVar submission date of Myriad data via SCRP
to evaluation date of same variant from other labs

All of the discordant variants are rare
 All of the 27 variants with classification discordances:
– Have population allele frequencies ≤ 0.05% in all of:
• ExAC
• ESP
• 1000 Genomes
• Invitae patient database
 Only 16.6% of patients carry any such rare variant(s)
 Most (98.4%) of these rare variants are completely
concordant when seen by more than one laboratory

Per-variant and per-patient concordance are
different
16.4%
0.2%
83.4%
Patients with 1 or more rare variants,
all concordant (16.4%)
Patients with 1 or more rare variants
having a discordance (0.2%)
Patients with no rare variants
(83.4%)
Steve Lincoln, Invitae and Benedict Paten UCSC
 Estimated per-patient concordance 2 orthogonal ways:
– From population allele frequencies
– From patient database (n≈20,000)
99.8%
Concordant

Do not over-generalize these results
 Discordance in other genes can be higher
– e.g. some cardiovascular genes
 Discordance from other sources can be higher

Van Driest et al., JAMA 2016
“There was low concordance in designating
SCN5A and KCNH2 variants as pathogenic.
In an unselected population, the putatively
pathogenic genetic variants were not associated
with an abnormal phenotype.”

Discussion
 While substantial disagreements in BRCA1/2 are
infrequent, they are important
 We must resolve differences collaboratively, not
competitively, to deliver the best patient care, as is done
in other areas of medicine
 Labs that maintain data as a proprietary asset can make
arbitrary and unverifiable claims regarding interpretation
accuracy
 Data submission ClinVar facilitates peer review and
interlaboratory quality control on a global scale, as
exemplified by this study

 Desmond et al., 2015 (open access)
 Swisher, 2015 (commentary)
 Lincoln et al., 2015 (open access)
 Jim Ford
 Allison Kurian
 Meredith Mills
 Leif Ellisen
 Andrea Desmond
 Kristen Shannon
 Nadine Tung
 Steve Lincoln
 Shan Yang
 Yuya Kobayashi
 Scott Topper
 Bob Nussbaum
 Melissa Cline
 Molly Zhang
 David Haussler
 Benedict Paten
For copy of slides:
steve.lincoln@invitae.com
steve.lincoln@me.com
or www.invitae.com

Establishing validity, reproducibility, and utility of highly scalable genetic tests: Challenges and Solutions - Stephen Lincoln

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Establishing validity, reproducibility, and utility of highly scalable genetic tests: Challenges and Solutions - Stephen Lincoln

Similar to Establishing validity, reproducibility, and utility of highly scalable genetic tests: Challenges and Solutions - Stephen Lincoln (20)

More from Human Variome Project

More from Human Variome Project (20)

Recently uploaded

Recently uploaded (20)

Establishing validity, reproducibility, and utility of highly scalable genetic tests: Challenges and Solutions - Stephen Lincoln

Editor's Notes