Using reference materials can help clinical laboratories meet validation and verification requirements for molecular diagnostic tests. Credit Valley Hospital uses Horizon Diagnostics' pooled DNA reference standards at a 2.5% mutant allele frequency as a low positive control in their EGFR diagnostic assays. Including this control helps eliminate false positive results and provides a qualitative reference to confidently identify true low level positives. This reduces the risks of false negative or false positive reports, improving patient outcomes by ensuring accurate molecular testing results.
2. 5
Bio-Specimens used in Molecular Diagnostics
Most clinical tissue samples are preserved in FFPE
FFPE samples are now being used for molecular diagnostic testing
FFPE based studies: every specimen is different in terms of % tumor contribution to
the specimen and % mutation contribution to the tumor
• Therapeutic choices are made based upon these results
• False positive and false negative results are detrimental to the patient
FFPE
Sample
Cancer
Patient
DNA Extraction Diagnosis
3. 3
External Quality Assessment Proficiency Testing Scheme - 2014
Only 70% of laboratories passed the proficiency test.
False -negatives and false-positives were the main sources of error.
0
5
10
15
20
25
30
35
40
PercentageofIncorrectResults
EGFR Sample Tested
EGFR Genotyping Errors
External Quality Assessment 2014
4. What is the impact of assay failure
in your laboratory and how do
you monitor for it?
7
5. 5
HDx™ Reference Standards - Precise Allelic FrequenciesReference
Slide
FFPE Reference
Standards
IHC Reference Slides
FISH Reference Slides DNA Reference
Standards
Tumor sample
Diagnosis
Therapy
DNA extraction
GenotypingCytogenetics
Histology
8. 8
Verification and Validation Definitions
• Confirmation by examination and provision of
objective evidence that the particular requirements
for a specific intended use are fulfilled.
Verification
(ISO 9000)
• Confirmation by examination and provision of
objective evidence that specified requirements have
been fulfilled.
Validation
(ISO 9000)
• Safe and useful service to clinicians and patients.
• Validation is more than a one time activity and if
used continually can improve the assessment of test
accuracy and performance.
Purpose
9. 9
General Validation Requirements
General Validation Requirements
Test development – protocols and pooling
parameters
Test validation – establish performance parameters
(sensitivity, specificity, reproducibility etc.)
Platform validation – performance and confidence
intervals including software validation
10. 10
Intent Comparison
US - CLIA
• Regulate Laboratories –
limited scope.
• Test used that is not FDA
cleared/approved
establish performance
characteristics before release
of test results.
• No oversight of clinical
validity.
• Oversight occurs through
biennial survey – after
testing has started.
• Test complexity framework.
US - FDA QSR (Potential)
• Regulate manufactures.
• Premarket clearance
evaluates clinical validity.
• Clinical validity – accuracy
with which the test
identifies, measures or
predicts the presence or
absence of a clinical
condition or predisposition in
a patient.
• Risk based framework.
11. 11
Requirements Comparison
• Regulates LDTs made and used within a single facility.
• Focus on accurate reproducible and reliable tests.
• Requirement for analytical validation prior to use.
• Requirements for proficiency testing.
US - CLIA
• Address clinical validity
• Risk based approach to implementation.
• Process validation.
• Design verification and validation.
• Adverse Event Reporting.
• Device Listing and Pre-market Approval (for some devices).
US - FDA
QSR
(Proposed)
12. Canadian Regulations
• Accreditation of medical laboratories in Canada is regulated by
provincial health authorities, five of them having accreditation
bodies.
• Each of the bodies has developed its own standards implementing
ISO documents which generally follows ISO 15189 (balance of
business and technical assurance).
• Risk based scale to validation based on if the test is developed in
house, uses nonstandard methods, originates from scientific
literature or de novo.
• Requirements for proficiency testing.
• Importance of method validation.
Canada
(CAN-P-4E)
13. 13
FDA QSR Verification and Validation (Sec. 820.30 Design controls)
• Validate device design under
standard operating parameters.
• Design input meets design output.
• Part of design history file.
Design
Verification
• Validate device design through
production units.
• Conform to intended use of product.
• Part of design history file.
Design
Validation
14. 14
Cost, Risk and Technical Possibilities…
• More regulation may mean more cost but there are still opportunities for
innovation.
• There is opportunity for clinical laboratories to work with the FDA and
CLIA to develop an appropriate regulatory framework.
Cost
• The FDA’s plans to regulate LDTs are both risk based and to reduce risk.
• Most accreditation programs logically consider risk in the level of
validation required.
• Risks are changing, many LDT’s are moving beyond their traditional
boundaries.
Risk
• Development of new technologies.
• Changes to the standard of care.
• Increasing availability of reference materials, including HDx reference
materials.
Technical
Possibilities
15. 15
Final Thoughts
Final Thoughts
• Use of reference materials can assist in
validation studies but there is still a strong
focus on use of patient specimens, in spite
of increasingly available reference materials.
• Where reference materials are mentioned,
these focus on cell line DNA over plasmids.
• The future for FDA regulation of LDTs is
being developed. The impact of this
regulation remains for debate.
17. Case Study at Credit Valley Hospital
A Discussion About the Value of Including Low Positive DNA
Control in Each Assay
18. 18
Methodology
EGFR mutation status test: Entrogen commercially available kit
Instrument: Roche LC480 Real Time PCR
Allele discriminating assays rely upon the ability of the probes to bind correctly and
the software setting to correctly set a crossing point baseline
Controls: HDx™ Reference Standards to
Validate assay and establish detection
Limits for each mutation
20. Validation Issues: Background Amplification in the G719X Reaction
Mutation Positive Control
Noise band
Patient (Cp 37)
Negative control Cp can lead to a false positive
result for the patient
Manually changing the baseline hides quality
issues such as inefficient probe binding or PCR
contamination in the blank
Mutation Negative Control
Mutation Negative Control
Noise band
20
Inefficient probe binding and background noise is seen most in G719X and T790M
21. Validation Issue: Separation from Background Amplification CP Values
T790M and G719X negative control and 1% Low Positive control values overlap
Crossing point value from 2.5% DNA HDx™ Reference Standard for each mutation
provides best separation from background amplification
2.5% Crossing Point Value used as a cut off
Low Positive Control now included in each assay to remove false positives
20.0
22.0
24.0
26.0
28.0
30.0
32.0
34.0
36.0
38.0
40.0
T790M Exon 19del L858R L861Q G719X
2.5%
1%
Neg Ctl
21
CrossingPoint
EGFR Variant
22. Low Positive Control helps identify and correct false positives from Non-
Specific Probe Binding
EGFR G719X mutations were safely ruled out by use of the 2.5% cut-off value.
G719X Reaction
Kit PC (28.24)
2.5% Control (31.85)
Patient A (34.70)
Patient B
Negative Control
NTC
23. Added Value from Inclusion of a Low Positive Control
Inclusion of a 2.5% low positive control allows background
amplification to be ruled out
Background amplification can lead to false positive diagnostic
results
Big Picture
Cancer patients awaiting treatments from diagnostic results
False positives TKI therapy for true EGFR WT patients
• Proven to be detrimental over first-line chemotherapy (Patton et al., 2014)
24. Diagnostic Test Issue: Low Mutation Content Below Cutoff
Should the 2.5% control crossing point be the absolute cut-off in all
cases?
When mutation is in a low percentage of cells in the specimen, the
crossing point will be high, and possibly above the 2.5% established
cutoff
How should a high crossing point value be evaluated?
Inverse
Relationship
24
Crossing Point (CP) Allelic Frequency %
in Tumor sample
25. Low Positive Control as Absolute Cut Off?
Patient 1: 58 year old female Metastatic lung cancer
• Mutation negative for all mutations except L858R
• L858R crossing point slightly higher than 2.5% control
• Qualitative assessment of amplification curve suggests true positive at about 1-2%
L858R Reaction
Kit PC (29.31)
Patient C (34.41)
2.5% Control (33.58)
Neg Ctl/NTC
26. Low Positive Control as Absolute Cut Off?
Patient 2: 58 Year old female with right lung adenocarcinoma
• Mutation negative except for Exon 19del
• Exon 19del crossing point slightly higher than 2.5% control
• Qualitative assessment suggests this is a true positive, at about 1-2%
Exon 19del Reaction
Kit PC (28.39)
2.5% Control (30.71)
Patient D (33.30)
Neg Ctl/NTC
27. Conclusion – Qualitative and Quantitative QC for each run needed
Patient Outcomes
Qualitative assessment suggests these are true positive results
Patients are receiving anti-EGFR therapy and both report feeling better, less
fatigue, still able to work
Conclusion
Crossing point values alone do not tell the whole story
False negatives can occur from absolute/quantitative use of 2.5% control
• Confidently ruled out by qualitative interpretation of patient amplification curve
False negative chemotherapy for EGFR mutant patient
• Less effective than TKIs for patients with an EGFR variant (Patton et al, 2014)
Horizon Diagnostics is proud to support clinical laboratories with the provision of sustainable reference materials
for research use only; applications include the validation of equipment, consumables and laboratory developed
tests offline from patient testing, supporting the healthcare continuum for the regulators, payors, clinical
laboratory and ultimately the patient.
28. Analytical Utility of a Low Positive Control
Inclusion of a 2.5% low positive control allows elimination of false positives
Low positive controls supply a qualitative reference for true amplification
so that low positive results can be reported with confidence
Reduce the risk of false negative reports
It a QC check for each run
Provide consistency between patient EGFR mutation status experiments
28
Reduction in False Negatives and False Positive Results is needed for
better future PT results and diagnostic testing
29. 2.5% Crossing Point Values Very Consistent Between Assays
T790M Exon19del L858R L861Q G719X
Kit Pos Ctl 28.13 29.17 29.87 28.73 28.36
2.50% 32.64 31.61 34.1 32.12 31.9
26
27
28
29
30
31
32
33
34
35
36
37
CrossingPoint
Control Crossing Points Comparison
29
95% confidence intervals for 17 different assays
Mean Crossing points of 2.5% Low positive control: +/- 0.26 (L858R) to +/- 0.56 (G719X)
30. The Value of a Low Positive Control at Credit Valley Hospital
Case study of how a Canadian clinical laboratory used RUO products to
validate their laboratory developed test
Credit Valley Hospital includes a 2.5% MAF reference standard for all their
EGFR diagnostic tests
• Individual EGFR Base-Seq DNA HDx™ Reference Standards are ordered and
pooled to a 2.5% mutant allelic frequency
• A Low Positive Control has also been implemented for the BRAF, KRAS and
NRAS testing at this laboratory
This gives them the confidence that their assay is working today, and also a
qualitative benchmark for true amplification when patient results are close
to cut off CT values
30
EGFR Low
Positive
Control
KRAS Low
Positive
Control
BRAF Low
Positive
Control
NRAS Low
Positive
Control
31. 7
“The low positive control is critical to my confidence in the
lab’s diagnostic reporting and analytical results.”
Dr. Marsha Speevak, Discipline Lead
Molecular Genetics and Cytogenetics, Credit Valley Hospital,
Trillium Health Partners
What is the impact of assay failure
in your laboratory and how do
you monitor for it?
32. Assay Plate Set Up – Supplemental Data
Credit Valley Uses 4 lanes of controls in their kit
1. Kit PC
2. Blank
3. Kit Negative
4. Horizon Dx pooled 2.5% Control
Editor's Notes
We know that formalin can be challenging to the MDX workflow.
EGFR test, 91 labs participated 70%.
With their best efforts put forward still got it wrong 30% of the time.
In order to address these challenges, horizon diagnostics has created the following reference standards, FFPE to examine DNA extraction, DNA Reference standards for genotyping, and our cell slides for IHC and FISH assays.
Hello everyone. I am Hannah Murfet, Product Quality Manager at Horizon Discovery. Here at Horizon I am responsible for the quality and regulatory management aspects of Horizon’s products including design and new product introduction. Today I am going to provide an overview of the regulatory requirements associated with the clinical application of Next Generation Sequencing with particular focus on validation.
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Validation occurs across multiple areas of the clinical laboratory. Validation can include equipment, reagents, operators, platforms. Two of the key areas are test and platform validation.
Test development includes
Establishing protocol
Optimising performance
Determining pooling parameters
Using synthetic variants to compare tools and facilitate optimisation
Test validation includes
Determine parameters
First tests developed carry highest validation requirements
Changes to tests must follow re-validation required against existing test
Platform validation
Cumulative performance data established
Determine confidence intervals
Track and validate software versions
Changes to platform must follow re-validation required against existing platform
Looking at their validation
Next LOD
Two variants are trickier to report on due to inefficient
Background amp –
Look back at their validation, 1% actually overlaps with background amplification.
Included another dilution in validation, 2.5%
The found that this 2.5% gave them the best separation from background amplification CP values or cut off values.
Patient A amplification rises above the negative control, but is lower than the 1% DNA cut-off giving a false positive Crossing point value
Qualitative and quantitative assessment of curves comparing 2.5%, negative control and patient reactions
Original cutoff for G719X was 36 (at 1% DNA positive Control)
Patient A amplification rises above the negative control, but is lower than the 1% DNA cut-off giving a false positive Crossing point value
In this case, we could safely rule out G719X mutations in both patients by use of the 2.5% cut-off value.
2.5% low positive control also provides a QC check in each run. Qualitative assessment of the amplification curve of the patient sample in comparison with the low positive control provides a higher level of certainty about the result than use of a crossing point cutoff alone.
On the other side of these clinical samples are patients who are waiting on a cancer treatment decision to fight for their life.
Looked into value of low positive control – but should this be an absolute cut off
To explore this further we have two patient runs to look at
Except perhaps L858R
CP value higher than LPC but through qualitative assesment we see it amplifies similarily to our control.
True + the second patient
What is the clinical utility
Low Postive CP value should not be used as an absolute cut off, qualitative inspection of curve should occur tp ID true amplification and to reduce the risk of false negative reporting.