Qiagen Webinar – January 19, 2016
Birgit Funke, PhD, FACMG
Associate Professor of Pathology, Harvard Medical School
Director Clinical R&D; Laboratory for Molecular Medicine
http://www.nature.com/nrg/journal/v11/n1/images/nrg2626-f2.jpg
1
NGS IN THE CLINIC
A review of its impact on test development, operations
and result reporting
WHAT WILL BE COVERED
1. How is NGS used in the clinic today
• Focus on inherited disorders
2. Bringing NGS in house:
• Clinical developmentand validation
• The role of bioinformatics
3. What NGS does not do well
• Technical limitations
1. Time permitting: Real world challenges (case example)
• Clinical NGS is being implemented in an increasing number of labs
• Majorityfocus on gene panels
• Implementationofexome/genome sequencingquicklyincreasing
NGS
(research)
THE DISRUPTIVE NATURE OF NGS
NGS
(clinical)
DETECTION RATE OVER TIME
DCM
2007:~10%
2011:~37%
NGS
WHICH DISORDERS
BENEFIT FROM
NGS TESTING?
• Collective incidence:> 1/500
• Can lead to SCD
• Substantialgenetic component
• Incentiveforpredictivetesting
ARVCDCM
LVNC
HCM
RCM
OTHER(RARE)
1:500 > 1:2,500 1:1,000 - 1:5,000
EXAMPLE: INHERITED CARDIOMYOPATHIES
• Locus heterogeneity - 1 disease / many genes
• Allelic heterogeneity - many disease causing variants/gene
• Spectrum of pathogenic variation - not yet well understood
• Phenotypic overlap - can complicate testing process
DISEASE CHARACTERISTICS
DCM
TTN
LMNA
MYH7
DSP
TNNT2
MYH7
34%
MYBPC3
50%
Ahmed AlfaresMelissa Kelly
HCM
LOCUS HETEROGENEITY
R502WW792fsR663H927-9G>A
MYBPC3MYBPC3 MYH7
#ofVariants
MYBPC3
E258K
63% of variants have been seen only once
Courtesy of Ahmed Alfares and Heidi Rehm
Need to sequence entire coding sequence
of many genes for maximum clinical
sensitivity
# of Probands
Hypertrophic Cardiomyopathy: 10 years – 11 genes tested
ALLELIC HETEROGENEITY
CLINICAL HETEROGENEITY
3
+
3
+
+
+
• Proband with clinical Dx +
family history of DCM
• DCM gene panel detects a
variant of uncertain
significance
• Variant did not segregate
Traditional genetic testing : 1 gene panel for each diagnosis
3
+
3
+
+
+
+
+
+ +
_• Patient was seen again ,
diagnosis was revised to
ARVC
• ARVC panel identified a
likely pathogenic variant
PREVENTING DIAGNOSTIC ODYSSEYS
Traditional testing (disease centric) does not make sense for
disorders with clinical and genetic overlap
HCM DCM ARVC
MULTI-CARDIOMYOPATHY TEST
HCM test DCM test ARVC test
• ~3% of DCM patients carry a pathogenic variant in an ARVC gene
MULTI-DISEASE GENE PANELS
GeneReviews (2012): “80-90% of patients with Costello syndrome carry a
mutation in HRAS.”
MULTI DISEASE GENE PANELS IMPROVE
CLINCAL DIAGNOSIS
(n=23)
LMM broad referral population data:
Mostpatients
would have
received a
NEGATIVE
report if only
HRAS had
been tested
Phenotypic expansion
• Original clinical definition based on most severe cases
• Often too narrow, full range of clinical variability emerges over time
Phenotypic overlap
• Disorders present the same -> diagnostic “error”
• Happens more often as genetic testing is moving out of specialty
clinics to more general (genetics) care
Now widely recognized
• Many physicians are beginning to change workflow
MULTI DISEASE GENE PANELS IMPROVE
CLINCAL DIAGNOSIS
Clinical
Diagnosis
“sequence first”Patient
Clinical
Diagnosis Hypothesis
Genetic
Test
negative
Inter
pretationResult
Sequence
Variants
• Fill infailed
• Confirmvariants
THE CHANGING GENOMIC TESTING WORKFLOW
“SEQUENCE FIRST”
THE CURRENT DEBATE: PANELS OR EXOME?
Complex phenotypes /
diagnostic odysseys
Clinically well
defined cases
GENE PANELS
10s – 100s of
genes
•High coverage
•Completeness
EXOME
22,000
genes
GENOME
Exome +
Intergenic
•Med-Low coverage
•Completeness <100%
?
• A large fraction of panels are NEGATIVE (often >50%)
• Growing appreciation of “phenotypicexpansion” – argument
for hypothesis fee testing
• Additional tests often end up being more expensive than WES
• Always up to date (accelerating pace of gene discovery)
• Easier to maintain for labs than growing # of gene panels
INCENTIVES
BARRIERS
• Cost (though gap is closing)
• Incompletecoverage (suboptimal design)
RISK
• Loss of intimate /a priori knowledge on tested genes
BARRIERS
COMMUNITY EFFORTS TO DEVELOP STANDARDS
ClinGen
Unite medical genetics communityby
developingapproaches to
curate,centralize and share geneticdata
ACMG + AMP (2015)
New guideline forclinical grade variant
classification(Mendelian disorders)
• Traditionally not needed because old tests
contained only well established disease genes
• Many published claims for a gene-disease
relationship do not withstand the rigor of clinical
grade curation
• Most novel variants in genes that are not strongly
linked with disease cannot be interpreted
NEW!
ASSESSMENT OF GENE-DISEASE RELATIONSHIPS
Definitive
evidence
Strong evidence
Moderate
evidence
Limited evidence
Disputed /
no evidence
Exome/Genome
Predictive Tests,
Incidental Findings
Diagnostic Panels
Research
Courtesyof ClinGen(Heidi Rehm)
USING CLINICAL VALIDITY TO CONFIGURE TESTS
CLINICAL TEST
DEVELOPMENT AND
VALIDATION
(NGS)
DEVELOPMENT AND VALIDATION OF AN (NGS) TEST
DESIGNTEST DESIGN
http://www.cdc.gov/genomics/gtesti
ng/ACCE/
LIBRARY
PREP
CAPTURE SEQ ANALYSIS
OPTIMIZATION
• Test each step
• String together
• Optimize
• Lock down protocol
VALIDATION
Test analytical
performance using
locked down
protocol
Content Technology
DESIGN - A THREE DIMENSIONAL EXERCISE
Gene selection
• Clinical validity
Gene characteristics
• Homology,High GC/AT?
• Pathogenicvariant types
• Mosaicism?
Sequencer
• capacity/ln
• Read length
Bioinformatics pipeline
• SNV
• Indel
• CNV
• SV
Operational
• Target size
• # samples/lane
• Sample volume received
• Test cost, TAT
Diagnostic
• % gene unsequenceable
• Difficult variants prevalent
• Abilityto detect low AF
Supplemental assay/s when:
• NGS suboptimalbut gene or variant
type essential
Validation (reference) samples
• Specimen types
• Variant Types
• Reference material sources
Fill-in + confirmation strategy (Sanger)
X =
Techn. Reality Check
Clinical Reality CheckDevelopment + validation plan
Consult domain experts
• Difficult gene critical?
• Alternate assaysoffered
elsewhere
• Required analytical
sensitivity*?
• Required completeness
* TP/TP+FN
THERE IS NO “ONE SIZE FITS ALL”
Renal Panel
(PKD1)
Hearing Loss Panel
(STRC)
Essential gene (detection rate) YES (85%) YES (10%)
Sequenceable via NGS Highly homologous (partial) Highly homologous (entire gene)
Pathogenicvariant spectrum SNVs, indels >>CNVs CNVs >>SNVs, indels
Well sequenced gene YES NO
Clinical specificity HIGH LOW
Keep on panel NO - alternate test for ADPKD YES
Supplemental assay N/A LR-PCR x 2
Operationalfactors N/A High failure LR-PCR #1 – DROP
Impact on performance metric N/A Reduced analytical sensitivity-
OK
Two difficultto sequencegenes– two solutions* (Laboratory for Molecular Medicine)
Two very differentsolutions
Both present similar challenge
TEST DEVELOPMENT/OPTIMIZATION
DESIGN
• Selecttestcontents
• Characterizecontents
• Clinicalneeds
• Technicallimitations
• Supplementalassays?
• Fill-inand variant
confirmation?
• Establish development
+ validationplan
LIBRARY
PREP
CAPTURE SEQ ANALYSIS
OPTIMIZATION
• Test each step
• String together
• Optimize if need be
• Lock down protocol
• Develop Q/C stops
VALIDATION
Test analytical
performance using
locked down
protocol
SECONDARY
ANALYSIS
TERTIARY
ANALYSIS
ANNOTATION
PRIORITIZATION
FILTRATION
CLASSIFICATION
CLINICALREPORT
FASTQ
NGS RUN
BASE CALLING
ALIGNMENT
VARIANTCALLING
BAM
VCF
DEMULTIPLEXING
PRIMARY
ANALYSIS
output
files
Sometimes on the
Sequencer (newer
machines, e.g.
MiSeq)
Trend is to add more
functionalityto the
sequencer
“bioinformaticspipeline”
THE ROLE OF BIOINFORMATICS
(…not just for NGS)
Research
Clinical
Dev
Clinical
Ops
Bioinformatics
IT
BIOINFORMATICS STAFF
(Partners Personalized Medicine)
2008 2009 2010 2011 2013
0
1
2
3
4
5
6
7
1 2 3 4 5 6 7
BioinformaticsFTEs
NGS work
started
20142012
1st NGS
panel
Molecular Dx Lab + Translational Genomics Core
COMMONLY USED DATA ANALYSIS FILTERS
• Secondary Analysis
• Coverage (read depth)
• Strand bias
• Mapping quality (confidence in location of read)
• Allelic fraction (% reads containing variant)
• Quality by depth
• Tertiary Analysis
• Variant frequency in control cohorts
Courtesy of Karl Voelkerding
• In theory, a heterozygous variant is present in 50% of the reads
• In reality, this is not always the case (range can be broad)
• To be confident to identify variants with a range of allelic fractions (e.g. >20%):
Need ~20xcoverageto be 99%confident to detectvariant allele
For details see: Schrijver 2012, J. Mol. Diagn. 14
COVERAGE : HOW MUCH IS ENOUGH?
Red = reads
containinga
variant call
Example LMM (germline) –
46 genes (~250 kb)
• Very high average coverage
(300-400x)
• >97% of bases >20x
Need to sequence at high coverage to maximize completeness
• 3% <20x = 7.5 kb
• Reducing avg. coverage
increases the % of the test with
insufficient coverage!
REALITY CHECK
Need ~20x coverage to be 99% confident to detect heterozygous variant *
* Schrijver 2012, J. Mol. Diagn. 14
TTN_EX155
c.31709T>C
[SB= -10.01]
TTN_EX45A
c.15318T>C
[SB= -146.03]
TTN_EX157
c.31861A>G
[SB= -855.52]
LMNA_EX10
c.*4C>T
[SB= -106.6]
TAZ_EX10
c.718G>C
[SB= -3536.45]
TTN_EX37
c.8887A>C
[SB= 8.04]
TTN_EX37
c.8887A>C
[SB= 14.07]
TTN_EX37
c.8887A>C
[SB= 17.08]
HOM
HET
203 samples; n=1455 variants: NGS + Sanger TRUE POS FALSE POS
ANALYSIS THRESHOLDS – ALLELIC FRACTION
• Need large “truth sets”
• Until recentlythere was none labs could obtain pre-test development
• Data shown here generated from launching NGS pipeline w/o any
threshold (2011)
•  very high overhead for confirmatorytesting till truth set established
via Sanger confirmation
• Now have some communityresources: NIST’s NA12878 high
confidence regions
ALLELIC FRACTION THRESHOLDS SNVs + IN/DELS
See also: Allhoff et al. BMC Bioinformatics 201314 (Suppl 5):S1 doi:10.1186/1471-2105-14-S5-S1
READ 1 READ 2
A T C C G T G G A T T T C G T T A C A C G T T
T
T
T
T
T
STRAND BIAS
ANALYSIS THRESHOLDS – QUALITY BY DEPTH
• OD 260/280DNA Extraction
• Fragment size (each step)
Library
Generation
•Cluster Density,Errorrate
•Total reads,% reads pass filter
Sequencing
• % reads on target, avg coverage
• Min cov/base, librarycomplexity,
• MQ, Strand Bias, GC/AT bias,etc
Alignment,
Variant Calling
Filtration,
Prioritization
• Concordance w/ NGS result
Variant
confirmation
• Double review
Variant + Gene
assessment
• Double review of variants
and interpretation
Interpretation+
Report
QUALITY CONTROL STEPS
• Q/C Approach?
• Individual steps?
• End-to-end?
 Both is best!
• End-to-end criticalmeasures
• Completeness
• Accuracy
IMPORTANCE OF GETTING CLUSTER DENSITY RIGHT
• Too low: Insufficient coverage
• Too high: increased FP
raw clusters #
#of clusters that
pass filter
OK DENSITY DENSITY TOO HIGH
Higher cluster
density canlead to
decreasepass filter
readsdue to lower
base quality
Difficultto prescribestandard – reducedquality at this stepcan be acceptable
dependingon the overalltestdesign (fillin, confirmation)
EXAMPLE Q/C STEPS - CLUSTER DENSITY (HiSeq)
FN % FN SENS 95% CI FP % FP SP
Substitutions * 0/542 0% 100% 99.3-100 1/1321 0.08% 99.9
In/dels 3/70 4.29% 95.70% 89.8-99.2 9/60 15.00% 85.0
LaboratoryforMolecularMedicine2013,unpublisheddata(HiSeq2000,50bPE)
When initial performance assessment reveals high FP rate one needs to
plan for orthogonal confirmation
DETERMINE NEED FOR ORTHOGONAL
CONFIRMATION OF VARIANTS
Today, NGS is deemed of high enough accuracyfor SNVs such that
confirmation may not needed
TEST VALIDATION
DESIGN
• Selecttestcontents
• Characterizecontents
• Clinicalneeds
• Technicallimitations
• Supplementalassays?
• Fill-inand variant
confirmation?
• Establish development
+ validationplan
LIBRARY
PREP
CAPTURE SEQ ANALYSIS
DEVELOPMENT
• Test each step
• String together
• Optimize
• Lock down protocol
VALIDATION
Test analytical
performance using
locked down
protocol
• Genetic testing traditionally synonymous with genotyping (assessing
whether known pathogenic variants are present)
• Sequencing based tests predominantly used to screen the entire coding
sequence  enables novel variant detection
• Core issues
• Cannot validate every theoretically possible variant
• The NGS testing process is highly complex (many steps)
• Increased requirement for bioinformatic processing
• Knowledge required for interpretation does not scale proportionally
WHY IS NGS DIFFERENT?
• These issues are not entirely new – but are an order of magnitude
worse compared to traditional methods (e.g. Sanger sequencing)
• NGS “pushed us over the edge” - catalyzed awareness and a push for
novel approaches and standards
• See also:
• Mattocks 2010: Eur J Hum Genet 18:1276
• CDC MMWR 2009 Vol 58 (www.cdc.gov/mmwr)
• Jennings 2009 Arch Pathol Lab Med.133:743–755
DESCRIPTION EXAMPLE SENS SPEC ACC PREC LOD
Quantitative Test
Result can have anyvalue between 2
limits)
Heteroplasmy of
mitochondrial variants
QualitativeTest
True quantitative signal can onlyhave
one of two possible values
Sequencing(WT or
variant allele)
Different approaches
depending on type of test
Sens: Sensitivity
Spec: Specificity
Acc: Accuracy
Prec: Precision
LOD: Limit of Detection
ANALYTICAL PERFORMANCE METRICS
Base line (“methods-based approach”)
• Maximize # of variants tested to achieve high statistical confidence
• Pathogenicity doesn’t matter
• Determine separately by variant type (SNV, indel, CNV)
• Critical importance of confidence interval!!
VARIANT TYPE # FN SENSITIVITY 95%CI
All variants 415 2 99.52% 98.2-100
Substitutions 350 0 100.00% 98.9-100
Indels (all) 65 2 96.92% 89.8-99.2
>10 bp 14 1 92.86% 70.2-98.8
?10 bp 51 1 98.04% 89.9-99.7
5-10 bp 3 0 100.00% ND
3-5 bp 19 1 94.74% 76.4-99.1
1-2 bp 29 0 100.00% 88.3-100
Disease/gene specific add-ons
• Test samples with common disease causing mutations
• Example: validate the mutations present on the ACMG 23 panel
for CF if NGS test covers CFTR
ANALYTICAL SENSITIVITY
Definition
• Ability to return a negative result when no variant is present
• Reflects frequency of false positives: TN / TN + FP
Common practice – count WT bases correctly identified as WT
• 20 kb sequenced: 5 FP  19995 true negatives
• Specificity = 19995 / 20000 = 99.98%
Not very useful – other metrics are more meaningful
• Technical Positive Predictive Value (likelihood that a variant
call is FP): TP/ TP + FP
• Number of FPs per TEST
• Particularly important if lab doesn’t confirm variants!
ANALYTICAL SPECIFICITY
Well character.
gene + n samples (e.g. frequent pathogenic variants)
In/del rich gene
+ n samples
Mosaicism + n samples
TIER 2: DISEASE/GENE SPECIFICCONSIDERATIONS
CNV richgene
+ n samples
Test 1
(100 genes)
NA12878 Test 2
(70 genes)
Test 3
(30 genes)
Test 4
(200genes)
Variants 600 300 100 800 Analytical sensitivity (+CI)
(=TP/TP+FN)
TIER 1: BASE LINE (METHODS BASED, CUMULATIVE)
VALIDATION LAYERS
EXAMPLE: ANALYTICAL SENSITIVITY
Need formore
reference materials!
• c.3628-41_3628-17del (outside
region interpreted by many labs)
• MAF = 4-8% in SE Asia
• Het: Mild/late onset HCM (high
lifetime penetrance) – OR ~6
• Hom: Severe and early onset
disease, often Heart Failure
Dhandapany et al. Nat Genet. 2009. 41(2):187-91
25 bp deletion (intron 32, MYBPC3)
WHAT NGS
DOES NOT DO WELL
CURRENT NGS
LIMIATIONS
NGS HAS PROBLEMS WITH
• Regions of high homology
• Repeat expansions
• Large in/dels, CNVs and other
structural variants
LESS PROBLEMATIC FOR GENE PANELS
• Testing labs typically have clinical domain expertise
• Contents typically highly curated + well understood
• Add-on tests typically available for difficult genes
KNOWLEDGE DOES NOT SCALE EASILY
• Sequencing ≠ understanding
http://www.trizic.com/the-devil-
is-in-the-details-how-prepared-
are-you-for-a-presence-exam/
THE DEVIL IS IN THE DETAILS
SEQUENCING GENES WITH HOMOLOGY
1
100% identical incl. introns
29
STRC (nonsyndromic hearing loss)
• 99.6% identical across cds (98.9% incl. introns)
• First half of gene 100% identical (incl. introns)
Mandelker 2014, J. Mol. Diagn.
maps to pseudogene
GENE
#
EXONS
#
HOMOL.
EXONS
%
HOMOL.
EXONS
HYDIN 86 78 91
TTN 363 26 7
NEB 183 24 13
STRC 29 23 79
DDX11 27 18 67
RANBP2 29 15 52
DPY19L2 22 13 59
TNXB 44 12 27
NCF1 11 11 100
ADAMTSL2 19 10 53
SMN1 9 9 100
SMN2 9 9 100
ABCC6 31 9 29
CYP21A2 10 8 80
IKBKG 10 8 80
OTOA 28 8 29
CFC1 6 6 100
OPN1MW 6 6 100
CHRNA7 10 6 60
VWF 52 6 12
GENE
#
EXONS
#
HOMOL.
EXONS
%
HOMOL.
EXONS
RPS17 5 5 100
OPN1LW 6 5 83
OCLN 9 5 56
GBA 12 5 42
PMS2 15 5 33
FANCD2 44 5 11
FCGR3B 6 4 67
RHD 10 4 40
CEL 11 4 36
CATSPER2 13 4 31
CHEK2 16 4 25
CLCNKB 20 4 20
NOTCH2 34 4 12
TBX20 8 3 38
KRT81 9 3 33
KRT86 9 3 33
STAT5B 19 3 16
PIK3CA 21 3 14
HS6ST1 2 2 100
HBA1 3 2 67
STRC: 10% of recessive nonsyndromic hearing loss
CYP21A2: Nearly all of congenital adrenal hyperplasia
VWF: Von Willebrandt’s disease (common coagulopathy)
PMS2: Colorectal CA (on ACMG IncidentalFindings gene list)
Diana Mandelker, Himanshu Sharma, Mark Bowser
Exome wide clinical grade resource to be shared via Get-RM (in progress)
NGS IN THE CLINIC
REAL WORLD
CHALLENGES
A WORD OF CAUTION
• THE NGS PROCESS IS HIGHLY COMPLEX
• EVEN THE BEST INFORMATICS AND Q/C MEASURES
DO NOT REPLACE A CRITICAL MIND
• WATCH OUT FOR RESULTS THAT DON’T MAKE SENSE
CASE EXAMPLE
9 year old boy with suspected Noonan syndrome
TEST ORDERED
Noonan Spectrum Panel (12 genes tested via NGS)
NEGATIVE
FAMILY HISTORY REVISITED
some features
• Physician ordered whole exome
• Pathogenic missense variant in SOS1
• Gene WAS covered by our test…
• What went wrong?
• Variant in untested region of SOS1 or
variant type not detectable?
• Technical false negative?
….at this pointthe lab director received
an unpleasantemail
Accession IDPM13-00632A
Index/BarcodeGCAT
Amplicons for Sequencing Follow-up (1)
PTPN11 01
Failed Regions (1 entries, 1 amplicons)--> exons with 1 or more bases that are insufficiently covered (<20x)
PTPN11 PTPN11_Exon_01 44/44
Variant calls
Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias
All entries below are variant calls that were removed by the pipeline
Common variants meeting criteria to be classified as "BENIGN" or "LIKELY BENIGN"(7 entries, 7 amplicons)
Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias
HRAS Exon 02 761 503;258 0.34 57.84 Variant Benign (Het) c.81T>C p.His27His -2494.88
KRAS Exon 05 1000 0;1000 1 59.25 Variant Benign (Hom) c.483G>A p.Arg161Arg -13656.42
MAP2K2 Exon 06 945 501;444 0.47 31.67 Variant Benign (Het) c.660C>A p.Ile220Ile -4434.96
MAP2K2 Intron 07 692 0;692 1 59.38 Variant Benign (Hom) c.919+12A>G -6098.73
SPRED1 Exon 03 942 424;518 0.55 59.54 Variant Benign (Het) c.291G>A p.Lys97Lys -8898.5
SPRED1 Intron 04 992 417;575 0.58 59.24 Variant Benign (Het) c.424-8C>A -6371.93
SPRED1 Exon 07 1000 410;590 0.59 59.22 Variant Benign (Het) c.1044T>C p.Val348Val -9031.77
Common False Positives (3 entries, 3 amplicons): Strand Bias >20 or Allelic fraction <0.2
Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias
HRAS Intron 02 481 424;57 0.12 58.83 Low_AF Benign (Het) c.111+15G>A -222.02
KRAS Exon 06 961 856;105 0.11 59.34 Low_AF Benign (Het) c.519T>C p.Asp173Asp -118.12
SOS1 Exon 14 985 631;354 0.36 59.44 Strand_bias (Het) c.2200A>C p.Thr734Pro 97.48
• SOS1Thr734Pro: strand bias = 97.48
• Highest strand bias observed in technical validation = 1.78
• Used threshold of 20 since
DIGGING IN THE TRASH….
FILTERED (REMOVEDFROM ANALYSIS) SECTION
TEST LIMITATIONS IN CLINICAL PRACTICE
• 100% sensitivefor SNVs
• CI = 99.3-100%
Technical false
negatives will
happen
LMM
• Heidi Rehm
• Matt Lebo
• Sami Amr
• Heather McLaughlin
• Heather Mason-Suares
• Jun Shen
Operations
• Lisa Mahanta
• Robin Hill Cook
• Laura Hutchinson
• Shangtao Liu
• Ashley Frisella
• Timothy Fiore
• Jiyeon Kim
• Maya Miatkowski
• William Camara
• Eli Pasackow
IT Team
• Sandy Aronson
• Natalie Boutin
• All team members
Leadership and Faculty
• Scott Weiss
• Meini Sumbada-Shin
• Robert Green
• Immaculata deVivo
• Mason Freeman
• John Iafrate
• Mira Irons
• Isaac Kohane
• Raju Kucherlapati
• Cynthia Morton
• Soumya Raychaudhuri
• Patrick Sluss
Transl. Genomics Core
Sami Amr
• All team members
Bioinformatics
• Matt Lebo
• Rimma Shakhbatyan
• Jason Evans
• Himanshu Sharma
• Peter Rossetti
• Ellen Tsai
• Chet Graham
Development
• Birgit Funke
• Beth Duffy-Hynes
• Mark Bowser
• Diana Mandelker
Fellows
• Ozge Birsoy
• Yan Meng
• Ahmad A. Tayoun
• Jillian Buchan
• Fahad Hakami
• Hana Zouk
GeneticCounselors
• Amy Lovelette
• MelissaKelly
• Mitch Dillon
• Shana White
• Andrea Muirhead
• Danielle Metterville
ACKNOWLEDGMENTS
Marketing
• Priscilla Cepeda
Office
• Peta-GayeBrooks
• Kim O’Brien
• Lauren Salviati
• Charlotte Mailly
• CassandraNovick
ClinGen leadership and cardiovascular domain working group
THANKS!
Title, Location, Date 63

NGS in Clinical Research: Meet the NGS Experts Series Part 1

  • 1.
    Qiagen Webinar –January 19, 2016 Birgit Funke, PhD, FACMG Associate Professor of Pathology, Harvard Medical School Director Clinical R&D; Laboratory for Molecular Medicine http://www.nature.com/nrg/journal/v11/n1/images/nrg2626-f2.jpg 1 NGS IN THE CLINIC A review of its impact on test development, operations and result reporting
  • 4.
    WHAT WILL BECOVERED 1. How is NGS used in the clinic today • Focus on inherited disorders 2. Bringing NGS in house: • Clinical developmentand validation • The role of bioinformatics 3. What NGS does not do well • Technical limitations 1. Time permitting: Real world challenges (case example)
  • 5.
    • Clinical NGSis being implemented in an increasing number of labs • Majorityfocus on gene panels • Implementationofexome/genome sequencingquicklyincreasing NGS (research) THE DISRUPTIVE NATURE OF NGS NGS (clinical)
  • 6.
    DETECTION RATE OVERTIME DCM 2007:~10% 2011:~37% NGS
  • 7.
  • 8.
    • Collective incidence:>1/500 • Can lead to SCD • Substantialgenetic component • Incentiveforpredictivetesting ARVCDCM LVNC HCM RCM OTHER(RARE) 1:500 > 1:2,500 1:1,000 - 1:5,000 EXAMPLE: INHERITED CARDIOMYOPATHIES
  • 9.
    • Locus heterogeneity- 1 disease / many genes • Allelic heterogeneity - many disease causing variants/gene • Spectrum of pathogenic variation - not yet well understood • Phenotypic overlap - can complicate testing process DISEASE CHARACTERISTICS
  • 10.
  • 11.
    R502WW792fsR663H927-9G>A MYBPC3MYBPC3 MYH7 #ofVariants MYBPC3 E258K 63% ofvariants have been seen only once Courtesy of Ahmed Alfares and Heidi Rehm Need to sequence entire coding sequence of many genes for maximum clinical sensitivity # of Probands Hypertrophic Cardiomyopathy: 10 years – 11 genes tested ALLELIC HETEROGENEITY
  • 12.
    CLINICAL HETEROGENEITY 3 + 3 + + + • Probandwith clinical Dx + family history of DCM • DCM gene panel detects a variant of uncertain significance • Variant did not segregate Traditional genetic testing : 1 gene panel for each diagnosis
  • 13.
    3 + 3 + + + + + + + _• Patientwas seen again , diagnosis was revised to ARVC • ARVC panel identified a likely pathogenic variant PREVENTING DIAGNOSTIC ODYSSEYS Traditional testing (disease centric) does not make sense for disorders with clinical and genetic overlap
  • 14.
    HCM DCM ARVC MULTI-CARDIOMYOPATHYTEST HCM test DCM test ARVC test • ~3% of DCM patients carry a pathogenic variant in an ARVC gene MULTI-DISEASE GENE PANELS
  • 15.
    GeneReviews (2012): “80-90%of patients with Costello syndrome carry a mutation in HRAS.” MULTI DISEASE GENE PANELS IMPROVE CLINCAL DIAGNOSIS (n=23) LMM broad referral population data: Mostpatients would have received a NEGATIVE report if only HRAS had been tested
  • 16.
    Phenotypic expansion • Originalclinical definition based on most severe cases • Often too narrow, full range of clinical variability emerges over time Phenotypic overlap • Disorders present the same -> diagnostic “error” • Happens more often as genetic testing is moving out of specialty clinics to more general (genetics) care Now widely recognized • Many physicians are beginning to change workflow MULTI DISEASE GENE PANELS IMPROVE CLINCAL DIAGNOSIS
  • 17.
  • 18.
    THE CURRENT DEBATE:PANELS OR EXOME? Complex phenotypes / diagnostic odysseys Clinically well defined cases GENE PANELS 10s – 100s of genes •High coverage •Completeness EXOME 22,000 genes GENOME Exome + Intergenic •Med-Low coverage •Completeness <100% ?
  • 19.
    • A largefraction of panels are NEGATIVE (often >50%) • Growing appreciation of “phenotypicexpansion” – argument for hypothesis fee testing • Additional tests often end up being more expensive than WES • Always up to date (accelerating pace of gene discovery) • Easier to maintain for labs than growing # of gene panels INCENTIVES
  • 20.
    BARRIERS • Cost (thoughgap is closing) • Incompletecoverage (suboptimal design) RISK • Loss of intimate /a priori knowledge on tested genes BARRIERS
  • 21.
    COMMUNITY EFFORTS TODEVELOP STANDARDS ClinGen Unite medical genetics communityby developingapproaches to curate,centralize and share geneticdata ACMG + AMP (2015) New guideline forclinical grade variant classification(Mendelian disorders)
  • 22.
    • Traditionally notneeded because old tests contained only well established disease genes • Many published claims for a gene-disease relationship do not withstand the rigor of clinical grade curation • Most novel variants in genes that are not strongly linked with disease cannot be interpreted NEW! ASSESSMENT OF GENE-DISEASE RELATIONSHIPS
  • 23.
    Definitive evidence Strong evidence Moderate evidence Limited evidence Disputed/ no evidence Exome/Genome Predictive Tests, Incidental Findings Diagnostic Panels Research Courtesyof ClinGen(Heidi Rehm) USING CLINICAL VALIDITY TO CONFIGURE TESTS
  • 24.
  • 25.
    DEVELOPMENT AND VALIDATIONOF AN (NGS) TEST DESIGNTEST DESIGN http://www.cdc.gov/genomics/gtesti ng/ACCE/ LIBRARY PREP CAPTURE SEQ ANALYSIS OPTIMIZATION • Test each step • String together • Optimize • Lock down protocol VALIDATION Test analytical performance using locked down protocol
  • 26.
    Content Technology DESIGN -A THREE DIMENSIONAL EXERCISE Gene selection • Clinical validity Gene characteristics • Homology,High GC/AT? • Pathogenicvariant types • Mosaicism? Sequencer • capacity/ln • Read length Bioinformatics pipeline • SNV • Indel • CNV • SV Operational • Target size • # samples/lane • Sample volume received • Test cost, TAT Diagnostic • % gene unsequenceable • Difficult variants prevalent • Abilityto detect low AF Supplemental assay/s when: • NGS suboptimalbut gene or variant type essential Validation (reference) samples • Specimen types • Variant Types • Reference material sources Fill-in + confirmation strategy (Sanger) X = Techn. Reality Check Clinical Reality CheckDevelopment + validation plan Consult domain experts • Difficult gene critical? • Alternate assaysoffered elsewhere • Required analytical sensitivity*? • Required completeness * TP/TP+FN
  • 27.
    THERE IS NO“ONE SIZE FITS ALL” Renal Panel (PKD1) Hearing Loss Panel (STRC) Essential gene (detection rate) YES (85%) YES (10%) Sequenceable via NGS Highly homologous (partial) Highly homologous (entire gene) Pathogenicvariant spectrum SNVs, indels >>CNVs CNVs >>SNVs, indels Well sequenced gene YES NO Clinical specificity HIGH LOW Keep on panel NO - alternate test for ADPKD YES Supplemental assay N/A LR-PCR x 2 Operationalfactors N/A High failure LR-PCR #1 – DROP Impact on performance metric N/A Reduced analytical sensitivity- OK Two difficultto sequencegenes– two solutions* (Laboratory for Molecular Medicine) Two very differentsolutions Both present similar challenge
  • 28.
    TEST DEVELOPMENT/OPTIMIZATION DESIGN • Selecttestcontents •Characterizecontents • Clinicalneeds • Technicallimitations • Supplementalassays? • Fill-inand variant confirmation? • Establish development + validationplan LIBRARY PREP CAPTURE SEQ ANALYSIS OPTIMIZATION • Test each step • String together • Optimize if need be • Lock down protocol • Develop Q/C stops VALIDATION Test analytical performance using locked down protocol
  • 29.
  • 30.
    THE ROLE OFBIOINFORMATICS (…not just for NGS) Research Clinical Dev Clinical Ops Bioinformatics IT
  • 31.
    BIOINFORMATICS STAFF (Partners PersonalizedMedicine) 2008 2009 2010 2011 2013 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 BioinformaticsFTEs NGS work started 20142012 1st NGS panel Molecular Dx Lab + Translational Genomics Core
  • 32.
    COMMONLY USED DATAANALYSIS FILTERS • Secondary Analysis • Coverage (read depth) • Strand bias • Mapping quality (confidence in location of read) • Allelic fraction (% reads containing variant) • Quality by depth • Tertiary Analysis • Variant frequency in control cohorts
  • 33.
    Courtesy of KarlVoelkerding • In theory, a heterozygous variant is present in 50% of the reads • In reality, this is not always the case (range can be broad) • To be confident to identify variants with a range of allelic fractions (e.g. >20%): Need ~20xcoverageto be 99%confident to detectvariant allele For details see: Schrijver 2012, J. Mol. Diagn. 14 COVERAGE : HOW MUCH IS ENOUGH? Red = reads containinga variant call
  • 34.
    Example LMM (germline)– 46 genes (~250 kb) • Very high average coverage (300-400x) • >97% of bases >20x Need to sequence at high coverage to maximize completeness • 3% <20x = 7.5 kb • Reducing avg. coverage increases the % of the test with insufficient coverage! REALITY CHECK Need ~20x coverage to be 99% confident to detect heterozygous variant * * Schrijver 2012, J. Mol. Diagn. 14
  • 35.
    TTN_EX155 c.31709T>C [SB= -10.01] TTN_EX45A c.15318T>C [SB= -146.03] TTN_EX157 c.31861A>G [SB=-855.52] LMNA_EX10 c.*4C>T [SB= -106.6] TAZ_EX10 c.718G>C [SB= -3536.45] TTN_EX37 c.8887A>C [SB= 8.04] TTN_EX37 c.8887A>C [SB= 14.07] TTN_EX37 c.8887A>C [SB= 17.08] HOM HET 203 samples; n=1455 variants: NGS + Sanger TRUE POS FALSE POS ANALYSIS THRESHOLDS – ALLELIC FRACTION • Need large “truth sets” • Until recentlythere was none labs could obtain pre-test development • Data shown here generated from launching NGS pipeline w/o any threshold (2011) •  very high overhead for confirmatorytesting till truth set established via Sanger confirmation • Now have some communityresources: NIST’s NA12878 high confidence regions
  • 36.
  • 37.
    See also: Allhoffet al. BMC Bioinformatics 201314 (Suppl 5):S1 doi:10.1186/1471-2105-14-S5-S1 READ 1 READ 2 A T C C G T G G A T T T C G T T A C A C G T T T T T T T STRAND BIAS
  • 38.
    ANALYSIS THRESHOLDS –QUALITY BY DEPTH
  • 39.
    • OD 260/280DNAExtraction • Fragment size (each step) Library Generation •Cluster Density,Errorrate •Total reads,% reads pass filter Sequencing • % reads on target, avg coverage • Min cov/base, librarycomplexity, • MQ, Strand Bias, GC/AT bias,etc Alignment, Variant Calling Filtration, Prioritization • Concordance w/ NGS result Variant confirmation • Double review Variant + Gene assessment • Double review of variants and interpretation Interpretation+ Report QUALITY CONTROL STEPS • Q/C Approach? • Individual steps? • End-to-end?  Both is best! • End-to-end criticalmeasures • Completeness • Accuracy
  • 40.
    IMPORTANCE OF GETTINGCLUSTER DENSITY RIGHT • Too low: Insufficient coverage • Too high: increased FP raw clusters # #of clusters that pass filter OK DENSITY DENSITY TOO HIGH Higher cluster density canlead to decreasepass filter readsdue to lower base quality Difficultto prescribestandard – reducedquality at this stepcan be acceptable dependingon the overalltestdesign (fillin, confirmation) EXAMPLE Q/C STEPS - CLUSTER DENSITY (HiSeq)
  • 41.
    FN % FNSENS 95% CI FP % FP SP Substitutions * 0/542 0% 100% 99.3-100 1/1321 0.08% 99.9 In/dels 3/70 4.29% 95.70% 89.8-99.2 9/60 15.00% 85.0 LaboratoryforMolecularMedicine2013,unpublisheddata(HiSeq2000,50bPE) When initial performance assessment reveals high FP rate one needs to plan for orthogonal confirmation DETERMINE NEED FOR ORTHOGONAL CONFIRMATION OF VARIANTS Today, NGS is deemed of high enough accuracyfor SNVs such that confirmation may not needed
  • 42.
    TEST VALIDATION DESIGN • Selecttestcontents •Characterizecontents • Clinicalneeds • Technicallimitations • Supplementalassays? • Fill-inand variant confirmation? • Establish development + validationplan LIBRARY PREP CAPTURE SEQ ANALYSIS DEVELOPMENT • Test each step • String together • Optimize • Lock down protocol VALIDATION Test analytical performance using locked down protocol
  • 43.
    • Genetic testingtraditionally synonymous with genotyping (assessing whether known pathogenic variants are present) • Sequencing based tests predominantly used to screen the entire coding sequence  enables novel variant detection • Core issues • Cannot validate every theoretically possible variant • The NGS testing process is highly complex (many steps) • Increased requirement for bioinformatic processing • Knowledge required for interpretation does not scale proportionally WHY IS NGS DIFFERENT? • These issues are not entirely new – but are an order of magnitude worse compared to traditional methods (e.g. Sanger sequencing) • NGS “pushed us over the edge” - catalyzed awareness and a push for novel approaches and standards
  • 44.
    • See also: •Mattocks 2010: Eur J Hum Genet 18:1276 • CDC MMWR 2009 Vol 58 (www.cdc.gov/mmwr) • Jennings 2009 Arch Pathol Lab Med.133:743–755 DESCRIPTION EXAMPLE SENS SPEC ACC PREC LOD Quantitative Test Result can have anyvalue between 2 limits) Heteroplasmy of mitochondrial variants QualitativeTest True quantitative signal can onlyhave one of two possible values Sequencing(WT or variant allele) Different approaches depending on type of test Sens: Sensitivity Spec: Specificity Acc: Accuracy Prec: Precision LOD: Limit of Detection ANALYTICAL PERFORMANCE METRICS
  • 45.
    Base line (“methods-basedapproach”) • Maximize # of variants tested to achieve high statistical confidence • Pathogenicity doesn’t matter • Determine separately by variant type (SNV, indel, CNV) • Critical importance of confidence interval!! VARIANT TYPE # FN SENSITIVITY 95%CI All variants 415 2 99.52% 98.2-100 Substitutions 350 0 100.00% 98.9-100 Indels (all) 65 2 96.92% 89.8-99.2 >10 bp 14 1 92.86% 70.2-98.8 ?10 bp 51 1 98.04% 89.9-99.7 5-10 bp 3 0 100.00% ND 3-5 bp 19 1 94.74% 76.4-99.1 1-2 bp 29 0 100.00% 88.3-100 Disease/gene specific add-ons • Test samples with common disease causing mutations • Example: validate the mutations present on the ACMG 23 panel for CF if NGS test covers CFTR ANALYTICAL SENSITIVITY
  • 46.
    Definition • Ability toreturn a negative result when no variant is present • Reflects frequency of false positives: TN / TN + FP Common practice – count WT bases correctly identified as WT • 20 kb sequenced: 5 FP  19995 true negatives • Specificity = 19995 / 20000 = 99.98% Not very useful – other metrics are more meaningful • Technical Positive Predictive Value (likelihood that a variant call is FP): TP/ TP + FP • Number of FPs per TEST • Particularly important if lab doesn’t confirm variants! ANALYTICAL SPECIFICITY
  • 47.
    Well character. gene +n samples (e.g. frequent pathogenic variants) In/del rich gene + n samples Mosaicism + n samples TIER 2: DISEASE/GENE SPECIFICCONSIDERATIONS CNV richgene + n samples Test 1 (100 genes) NA12878 Test 2 (70 genes) Test 3 (30 genes) Test 4 (200genes) Variants 600 300 100 800 Analytical sensitivity (+CI) (=TP/TP+FN) TIER 1: BASE LINE (METHODS BASED, CUMULATIVE) VALIDATION LAYERS EXAMPLE: ANALYTICAL SENSITIVITY Need formore reference materials!
  • 48.
    • c.3628-41_3628-17del (outside regioninterpreted by many labs) • MAF = 4-8% in SE Asia • Het: Mild/late onset HCM (high lifetime penetrance) – OR ~6 • Hom: Severe and early onset disease, often Heart Failure Dhandapany et al. Nat Genet. 2009. 41(2):187-91 25 bp deletion (intron 32, MYBPC3)
  • 49.
    WHAT NGS DOES NOTDO WELL CURRENT NGS LIMIATIONS
  • 50.
    NGS HAS PROBLEMSWITH • Regions of high homology • Repeat expansions • Large in/dels, CNVs and other structural variants LESS PROBLEMATIC FOR GENE PANELS • Testing labs typically have clinical domain expertise • Contents typically highly curated + well understood • Add-on tests typically available for difficult genes KNOWLEDGE DOES NOT SCALE EASILY • Sequencing ≠ understanding http://www.trizic.com/the-devil- is-in-the-details-how-prepared- are-you-for-a-presence-exam/ THE DEVIL IS IN THE DETAILS
  • 51.
    SEQUENCING GENES WITHHOMOLOGY 1 100% identical incl. introns 29 STRC (nonsyndromic hearing loss) • 99.6% identical across cds (98.9% incl. introns) • First half of gene 100% identical (incl. introns) Mandelker 2014, J. Mol. Diagn. maps to pseudogene
  • 52.
    GENE # EXONS # HOMOL. EXONS % HOMOL. EXONS HYDIN 86 7891 TTN 363 26 7 NEB 183 24 13 STRC 29 23 79 DDX11 27 18 67 RANBP2 29 15 52 DPY19L2 22 13 59 TNXB 44 12 27 NCF1 11 11 100 ADAMTSL2 19 10 53 SMN1 9 9 100 SMN2 9 9 100 ABCC6 31 9 29 CYP21A2 10 8 80 IKBKG 10 8 80 OTOA 28 8 29 CFC1 6 6 100 OPN1MW 6 6 100 CHRNA7 10 6 60 VWF 52 6 12 GENE # EXONS # HOMOL. EXONS % HOMOL. EXONS RPS17 5 5 100 OPN1LW 6 5 83 OCLN 9 5 56 GBA 12 5 42 PMS2 15 5 33 FANCD2 44 5 11 FCGR3B 6 4 67 RHD 10 4 40 CEL 11 4 36 CATSPER2 13 4 31 CHEK2 16 4 25 CLCNKB 20 4 20 NOTCH2 34 4 12 TBX20 8 3 38 KRT81 9 3 33 KRT86 9 3 33 STAT5B 19 3 16 PIK3CA 21 3 14 HS6ST1 2 2 100 HBA1 3 2 67 STRC: 10% of recessive nonsyndromic hearing loss CYP21A2: Nearly all of congenital adrenal hyperplasia VWF: Von Willebrandt’s disease (common coagulopathy) PMS2: Colorectal CA (on ACMG IncidentalFindings gene list) Diana Mandelker, Himanshu Sharma, Mark Bowser Exome wide clinical grade resource to be shared via Get-RM (in progress)
  • 53.
    NGS IN THECLINIC REAL WORLD CHALLENGES
  • 54.
    A WORD OFCAUTION • THE NGS PROCESS IS HIGHLY COMPLEX • EVEN THE BEST INFORMATICS AND Q/C MEASURES DO NOT REPLACE A CRITICAL MIND • WATCH OUT FOR RESULTS THAT DON’T MAKE SENSE
  • 55.
    CASE EXAMPLE 9 yearold boy with suspected Noonan syndrome TEST ORDERED Noonan Spectrum Panel (12 genes tested via NGS) NEGATIVE
  • 56.
    FAMILY HISTORY REVISITED somefeatures • Physician ordered whole exome • Pathogenic missense variant in SOS1 • Gene WAS covered by our test… • What went wrong? • Variant in untested region of SOS1 or variant type not detectable? • Technical false negative? ….at this pointthe lab director received an unpleasantemail
  • 57.
    Accession IDPM13-00632A Index/BarcodeGCAT Amplicons forSequencing Follow-up (1) PTPN11 01 Failed Regions (1 entries, 1 amplicons)--> exons with 1 or more bases that are insufficiently covered (<20x) PTPN11 PTPN11_Exon_01 44/44 Variant calls Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias All entries below are variant calls that were removed by the pipeline Common variants meeting criteria to be classified as "BENIGN" or "LIKELY BENIGN"(7 entries, 7 amplicons) Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias HRAS Exon 02 761 503;258 0.34 57.84 Variant Benign (Het) c.81T>C p.His27His -2494.88 KRAS Exon 05 1000 0;1000 1 59.25 Variant Benign (Hom) c.483G>A p.Arg161Arg -13656.42 MAP2K2 Exon 06 945 501;444 0.47 31.67 Variant Benign (Het) c.660C>A p.Ile220Ile -4434.96 MAP2K2 Intron 07 692 0;692 1 59.38 Variant Benign (Hom) c.919+12A>G -6098.73 SPRED1 Exon 03 942 424;518 0.55 59.54 Variant Benign (Het) c.291G>A p.Lys97Lys -8898.5 SPRED1 Intron 04 992 417;575 0.58 59.24 Variant Benign (Het) c.424-8C>A -6371.93 SPRED1 Exon 07 1000 410;590 0.59 59.22 Variant Benign (Het) c.1044T>C p.Val348Val -9031.77 Common False Positives (3 entries, 3 amplicons): Strand Bias >20 or Allelic fraction <0.2 Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category ClassificationPredicted ZygosityPredicted Variant Predicted AA Strand bias HRAS Intron 02 481 424;57 0.12 58.83 Low_AF Benign (Het) c.111+15G>A -222.02 KRAS Exon 06 961 856;105 0.11 59.34 Low_AF Benign (Het) c.519T>C p.Asp173Asp -118.12 SOS1 Exon 14 985 631;354 0.36 59.44 Strand_bias (Het) c.2200A>C p.Thr734Pro 97.48 • SOS1Thr734Pro: strand bias = 97.48 • Highest strand bias observed in technical validation = 1.78 • Used threshold of 20 since DIGGING IN THE TRASH…. FILTERED (REMOVEDFROM ANALYSIS) SECTION
  • 58.
    TEST LIMITATIONS INCLINICAL PRACTICE • 100% sensitivefor SNVs • CI = 99.3-100% Technical false negatives will happen
  • 59.
    LMM • Heidi Rehm •Matt Lebo • Sami Amr • Heather McLaughlin • Heather Mason-Suares • Jun Shen Operations • Lisa Mahanta • Robin Hill Cook • Laura Hutchinson • Shangtao Liu • Ashley Frisella • Timothy Fiore • Jiyeon Kim • Maya Miatkowski • William Camara • Eli Pasackow IT Team • Sandy Aronson • Natalie Boutin • All team members Leadership and Faculty • Scott Weiss • Meini Sumbada-Shin • Robert Green • Immaculata deVivo • Mason Freeman • John Iafrate • Mira Irons • Isaac Kohane • Raju Kucherlapati • Cynthia Morton • Soumya Raychaudhuri • Patrick Sluss Transl. Genomics Core Sami Amr • All team members Bioinformatics • Matt Lebo • Rimma Shakhbatyan • Jason Evans • Himanshu Sharma • Peter Rossetti • Ellen Tsai • Chet Graham Development • Birgit Funke • Beth Duffy-Hynes • Mark Bowser • Diana Mandelker Fellows • Ozge Birsoy • Yan Meng • Ahmad A. Tayoun • Jillian Buchan • Fahad Hakami • Hana Zouk GeneticCounselors • Amy Lovelette • MelissaKelly • Mitch Dillon • Shana White • Andrea Muirhead • Danielle Metterville ACKNOWLEDGMENTS Marketing • Priscilla Cepeda Office • Peta-GayeBrooks • Kim O’Brien • Lauren Salviati • Charlotte Mailly • CassandraNovick ClinGen leadership and cardiovascular domain working group
  • 60.
  • 63.