Model Call Girl in Subhash Nagar Delhi reach out to us at 🔝9953056974🔝
Clinical lab-method validation.ppt
1. Method Validation
To understand College of American Pathologists
(CAP) requirements of method validation in a clinical
laboratory
Dr. Sibtain Ahmed
2. What is validation?
• “Validation is the process of testing a measurement
procedure to assess its performance and to determine
whether that performance is acceptable.” [CLSI]
3. Method Validation
Why must we validate?
When should we validate?
What should we validate?
What is method validation?
4. What is Method Validation?
–Method validation and/or verification is the process
by which a method is determined to be fit for
purpose and intended use.
–Although method validation/verification are often
used interchangeably, validation is usually
performed on in-house and/or modified methods,
while verification is taking a marketed/unmodified
assay and verifying its performance.
5. Purposes of Method Validation Studies
To quantifiably characterize system performance
To assess potential for error
To identify method-to-method differences
To meet regulatory guidelines
6. When to Do Method Validation Studies
When considering purchasing a new system
When placing a new system into service
At regular intervals to assess on-going system performance
When troubleshooting questionable system performance
7. Tests to Validate
• Waived tests are approved by the FDA for home use and by definition are
simple to perform (i.e., pregnancy test) – do not require validation.
• Non-waived tests included moderately and highly complex tests – requires
validation
• Degree of validation/verification depends on status of the test method (FDA-
approved/Non-FDA-approved/Modified/Unmodified)
• Unmodified FDA-approved: using as intended by the manufacturer and
licensed for use by the FDA
• Modified or Non-FDA approved: using test kit for indications other than as
intended by the manufacturer; not licensed for use by the FDA.
8. The 6 Elements of Method
Validation:
•If FDA-approved/Unmodified
– only Reportable Range,
Precision, Accuracy and
Reference Intervals need to be
verified.
•If Non-FDA
approved/Modified – all 6
elements must be performed,
including sensitivity and
specificity.
9. samples are analyzed according to any number of schemes depending
upon the analyte availability:
Twice a day in duplicate for 10 days.
Once a day for 20 days.
Twice a day in quadruplicate for 5 days.
Select sufficient material at two levels:
Patient samples
QC material
Reference materials
Standards
Suitable materials that have a known value.
Precision
Multiple measurements are close to each others
i.e. Reproducibility
Gives information related to random error
Introduction
What is
needed
How we
perform the
testing
10. Experiment Design :
Run one sample 20 time once a day for within run precision and for between run
precision run one sample 2-4 times per day.
Data Input Requirements :
20 reading of single specimen,
allowable total error
allowable random error
Statistics:
MEAN
SD
CV
2SD RANGE
SD GOAL
Interpretation :The precision test pass if computed SD below the SD goal.
11. Accuracy
How close to the true value Comparison of methods
Closeness of the agreement between the result
of a measurement and a true value of the
measurand. [CLSI]
Gives information related to systematic error
Introduction
40 different specimens
Cover reportable range of method
What is
needed
Duplicate measurements of each specimen
on each method
Minimum of five days, prefer over 20
(since replicate testing is same)
How we
perform the
testing
12. Accuracy studies
Test results from the new
method against an existing
method which is known to
be accurate
Two or more samples in which the concentration of the
selected analyte concentrations spans but does not exceed
the analytical measuring interval of the assay.
Comparison of Method/ Instrument
Recovery of expected values
from assigned values
13. what is needed for accuracy studies ?
A control sample or certified reference material
Proficiency testing material
Materials provided by the method manufacturer for verification of
trueness or quality control.
14. Method comparison
• WHEN IT IS
PERFORMED?
To validate a new method being brought into
the lab, by demonstrating that it is
statistically identical to the method currently
in use.
HOW IT IS PERFORMED?
Assay about 20-40 patient samples on
both instruments. The specimens should
be somewhat evenly-distributed across
the analytes reportable range.
15. Purpose: To validate a new method being brought into the lab, by demonstrating that it is
statistically identical to the method currently in use.
Experiment Design Requirements:
Run 20-40 specimens that cover reportable range properly on both instruments.
Statistics:
Correlation coefficient(R)
SLOPE
INTERCEPT
Interpretation:
CORELATION COFICIENT SHOULD BE >0.9
95% confidence interval for the Deming Slope includes 1.000
95% confidence interval for the Deming Intercept includes 0.000
Alternate method comparison
16. Purpose:
The Two Instrument Comparison (2IC) procedure will satisfy the CLIA requirement for a lab to compare
instruments performing the same test on a semi-annual basis
Experiment Design:
Select 5-10 Specimens that cover reportable range and run on both instruments one time.
Data Requirements:
5-10 reading of specimens
Total allowable error
Reportable range
Statistics:
Average error index
Slope
Intercept
standard error estimate
Interpretation:
Test passes, when error index of each pair (x-y) is between 1.0 to -1.0
Test fails ,when error index of any pair greater than 1.0 or less than -1.0.
Two instrument comparison
17. Purpose: Multiple Instrument Comparison (MIC) compares 3 or more instruments to determine whether
they are within Total Allowable Error (TEa) of the target.
Experiment Design Requirements:
Select 3-19 instrument and run 3-5 specimens on each instrument that cover reportable range of test.
Select one instrument as a target instrument.
Data Requirements:
3-5 reading of each instrument
Total allowable error
Reportable range
Statistics:
OBSERVE ERROR
ERROR INDEX
ALLOWABLE ERROR
Interpretation:
The experiment passes if every non target instrument is within error index limit of -1 to 1 .
Multi instrument comparison
18. Reportable Range / Linearity
Lowest and highest test results that are reliable
Especially important with two point calibrations
Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR)
Introduction
What is
needed
How we
perform the
testing
Material manufactured specially for linearity checking.
CAP linearity survey or validation materials
Quality control or calibration material.
Single Patient samples.
Pool patient sample.
Spiked patient samples
Run 3-11 (5 is sufficient) specimens in triplicate with known concentration
spread across the reportable range; ideally the lowest and highest
specimen should challenge the limit of range.
19. Verifying reportable range
• AMR must be verified before method introduced, and at least
every 6 months (and after recalibration or major maintenance)
while in use
• CRR is a clinical decision by lab director, and does not require
experiments or re-validation; however, dilution or
concentration protocols must be specified in methods
20. How we evaluate linearity ?
Reportable/Analytical measuring range(linearity) is verified if two
conditions are met:
Assigned values of the lowest and highest specimen should be
within proximity limits of the reportable range limit.
These two specimens are acceptably accurate with respect to total
allowable error.
21. Experiment Design :
Select 3-5 specimens that cover reportable range/AMR and run 3-5 times each specimen
Data In put Requirements :
Low and high reportable range of analyte
Proximity limits
3-5 reading of each specimens
Target value of each specimen(mean)
Allowable error
Statistics
Mean
Slope
Intercept
Error
Percent
Recovery
Interpretation :
The midpoints of target ranges for the lowest and highest specimens respectively are with in
the proximity limit s of the reportable range limits
These two specimens also pass accuracy.
The method is linear if it is possible to draw straight line that passes within a user defined
allowable error of each specimen point.
22. Reference Intervals
Normal range in healthy population
Used for diagnosis/clinical interpretation of results
Introduction
• Test and compare to manufacturer: If all 20 specimens fall within the specified range, the
reference ranges have been verified and you can adopt manufacturer suggestions and
incorporate into your SOP. If not, need to establish reference ranges for your population
(240 specimens: 120 each of male and female).
• Another lab may have established reference ranges that can be used for verification
purposes; must be documented.
What is
needed
How we
perform the
testing
Enter the readings in EP evualtor along with, the
"proposed" reference interval that is to be verified.
23. PURPOSE: Reference interval verification with respect to population
Experiment Design Requirements:
Select 20 specimens of healthy individuals and run one time per specimen
Data Requirements:
20 reading of health individual specimens
Proposed Reference interval range
Statistics:
MEAN
SD
MEDIAN
RANGE
Frequency
Interpretation:
If more than 10% of results are out side the proposed reference interval which mean verification fail,
otherwise passes.
24. Sensitivity
Lowest reliable value; lower limit of
detection, especially of interest in drug testing
and tumor markers
Different terminologies used by different
manufacturers
Introduction
Blank solution: ( often use the zero calibrator)
Low level of positive sample (slightly above
the manufacture's stated lowest detection limit)
What is
needed
Run 20 blanks or low level samples;
How we
perform the
testing
25. Purpose: To verify the vendor claim sensitivity
Experiment Design Requirements :
Replicate response measurements are made for two specimens
blank or zero concentration sample minimum 10 replicate
And 2nd sample of known low concentration minimum 3 replicate
Data Requirements :
10 reading of zero concentration sample
3 reading of low concentration sample
manufacturer’s claim sensitivity
Statistics:
Mean
SD
2 SD 95% confidence interval for Limit of Blank
Interpretation :
Manufacture sensitivity claim is verified when the computed LOB is less than manufacturer ‘s
claim.
26. Specificity
Determination of how well a method measures the
analyte of interest accompanied by potential
interfering materials
Introduction
Standard solutions, participant specimens
or pools
Interferer solutions (standard solutions, if
possible; otherwise, pools or specimens)
added at high concentrations
What is
needed
Duplicate measurements.
The lab can use the manufacturer’s provided list
How we
perform the
testing
27. Specificity: How We Evaluate the Data
Tabulate results for pairs of samples
(dilution and interferent)
Calculate means for each (dilution
and interferent)
Calculate the differences
Calculate the average interference of
all specimens tested at a given
concentration of interference
28. Carry Over
Carryover” in laboratory testing, is defined as “the
contamination of a specimen by the previous one”.
Select specimens for the analytes to be tested.
Select one patient/control specimen with a very low level of
the analyte and one with a very high level.
Select specimens with enough volume so that they can each be
run approximately 10 to 11 times.
WHAT IS NEEDED ?
29. when to perform carry over studies?
Study must be performed, as applicable, as a part of initial
evaluation of an instrument.
Study is repeated after major maintenance or repair of the
pipetting assembly of instrument.
Evaluation for carryover is not required for automatic
pipettes that use disposable tips.
The instrument must be dedicated to the analysis of one
analyte during this experiment and must do this experiment
for one analyte at a time.
Routine performance of tests must be suspended.
30. How to perform carry over?
Run in following order :
3 Low specimens
2 High specimens
1Low specimen
2 High specimens
4 Low specimens
2 High specimens
1 Low Specimen
2 High specimens
1 Low Specimen
2 High specimens
1 Low Specimen
31. Experiment design: Two specimen are prepared ,one with very high concentration and one with very
low concentration .these specimens are aliquot out in to a total of 21 samples :11 with low and 10 with
high concentration.
Run sample in following sequence on analyzer.
LLLHHLHHLLLLHHLHHLHHL
Data Requirements:
21 reading of low and high specimen as above sequence.
Statistics
SD
Mean
Error Limit
Carry over
carry over: The mean of H-L results minus the mean of L-L results
Error limit: Three times the SD of L-L results
Interpretation :Carry over test passes if carry over is less than error limit.
33. Precision for qualitative assay
Runs of specimens with analyte concentrations near the cutoff
point .
Three specimens, one at cutoff, one just below cutoff, and one
just above cutoff (± 20% recommended) .
Replicate measurements of each of three specimens (20 each,
minimum).
34. How to evaluate?
The results obtained from the negative and positive controls
will be used to calculate the CV and compared to the
manufacturer’s claims for reproducibility.
The laboratory CV should be less than or equal to the
manufacturer’s stated CV. In the event that an assay does not
perform as expected
35. What is needed for accuracy studies for
qualitative testing?
Samples with known values, such as :
proficiency testing samples or commercial standards, may be
used as the reference method.
use of EQA samples or other commercially prepared
reference material with known values.
36. accuracy/method comparison for qualitative
testing
What is needed ?
• A minimum of 10 samples for each expected result.
For example, if a test method gives results of
“Positive/Negative”, the accuracy study must include 10
known positives and 10 known negatives.
• Two levels of quality control must be run each day that testing
is performed, not including controls internal to the kit
cartridge/testing device.
37. How it is performed?
• The performance of qualitative tests is most commonly
described in terms of sensitivity and specificity.
• By using a contingency table:
Method being Validated
Diagnostic Sensitivity and Specificity
(Results from Comparison Study) Total
Positive Negative
Positive # true positive (TP) # false positive (FP) TP+FP
Negative # false negative (FN) # true negative (TN) FN+TN
Total TP+FN FP+TN N
38. How it is evaluated?
Compare the results calculated above with the manufacturer’s
stated claims for Sensitivity, Specificity and Agreement found
in the test kit package insert.
Results must be equal to, or greater than, the manufacturer’s
claims for the method to be considered accurate
39. PURPOSE: To compare two qualitative method are correlated to each other
Experiment Design Requirements:
Run 20-40 specimen from both method which consist off positive ,negative and borderline results
equally.
Data Requirements :
20-40 reading of specimens
Statistics:
Total agreement
Positive agreement
Negative agreement
Cohen kappa agreement
Interpretation :
Cohen kappa agreement should be >75% passes.
Total agreement should be >90%
qualitative/semi qualitatine method comparision
What is Method Validation?
Method validation and/or verification is the process by which a method is determined to be fit for purpose and intended use. Although method validation/verification are often used interchangeably, validation is usually performed on in-house and/or modified methods, while verification is taking a marketed/unmodified assay and verifying its performance.
Why is it important?
Different testing environment, need to demonstrate that the test method performs in your lab environment as the manufacturer states it should
Need to prove to yourself that the results reported by the manufacturer are reliable.
When?
Initially, before releasing patient results, and after any manufacturer changes/modifications and/or movement of the equipment.
Measure of the reproducibility of the assay.
Provides information related to random error.
How do you verify precision?
Must repeat testing on 20 samples over one day (short term) and over a period of 20 days (long term).
Within 20 days we hope to see good among of variation in terms of how the test performed and exposure to a variety of environmental conditions.
How does the new method compare to the reference method or standard?
Testing should be performed in duplicate over at least 5 days in order to account for analyzer variations over the specified time period.
Precision gives you information related to random error; Accuracy gives you information related to systematic error.
Linearity may be misnomer; hence the terminology of Analytical Measurement Range and derived Clinical Reportable Range.
For each reagent or kit, need to establish reference intervals to assist clinicians in interpretation of results.
Based on normal “healthy” population.
Test and compare to manufacturer: If all 20 specimens fall within the specified range, the reference ranges have been verified and you can adopt manufacturer suggestions and incorporate into your SOP. If not, need to establish reference ranges for your population (240 specimens: 120 each of male and female).
Another lab may have established reference ranges that can be used for verification purposes; must be documented.
Not required for FDA approved/ unmodified assay.
If FDA approved/Unmodified, information can be used directly and incorporated in the your SOP.
Need to adopt values in your SOP as evidence for sponsor/audit.
Ability of your method to accurately measure an analyte within the presence of potential interfering substances.
Not required for FDA approved/ unmodified assay.
If FDA approved/Unmodified, adopt values in your SOP as evidence for sponsor/audit.
What else can cause interferences? Drugs/medication/hemolysis/lipemia.
Run assay with specimen diluted with blank (saline) and compare to same specimen diluted with potential interfering substance.