Assuring the Quality of Laboratory Testing in Countries Fighting the HIV/AIDS Epidemic CDC November 29-30, 2000

Test Verification & Test Validation Niel T. Constantine, Ph. D. Professor of Pathology Director Clinical Immunology University of Maryland School of Medicine And Institute of Human Virology

Test Verification &Test Validation Considerations when determining the utility of tests A. Protocols for Evaluation of Tests B. Reference Tests C. Algorithms D. Choice and Number of Samples E. Testing Conditions F. Resolution of Discordant Results G. Indicators of test performance

Considerations when determining the utility of tests

A. Protocols for Evaluation of Tests

B. Reference Tests

C. Algorithms

D. Choice and Number of Samples

E. Testing Conditions

F. Resolution of Discordant Results

G. Indicators of test performance

Considerations When Determining the Utility of Tests Protocols for Evaluations of Tests

Protocols for Evaluation of Tests Essential to set guidelines. Must be followed exactly. Must outline all characteristics of samples and procedures. Must describe detailed algorithm to follow for discordant results. Must include QA/QC section.

Essential to set guidelines.

Must be followed exactly.

Must outline all characteristics of samples and procedures.

Must describe detailed algorithm to follow for discordant results.

Must include QA/QC section.

Considerations When Determining the Utility of Tests Reference Tests

Reference Tests Needed to fully characterize samples. Choice depends on purpose of testing. Concordance – against reference screening test. Accuracy – against confirmatory test. Must be careful about “pre-selected samples” to evaluate false positives. Should be tests that are recognized by the scientific community.

Needed to fully characterize samples.

Choice depends on purpose of testing.

Concordance – against reference screening test.

Accuracy – against confirmatory test.

Must be careful about “pre-selected samples” to evaluate false positives.

Should be tests that are recognized by the scientific community.

Considerations When Determining the Utility of Tests Algorithms

UNAIDS and WHO Recommended Alternative Algorithms To maximize accuracy while minimizing cost Depends on objectives of the test and the prevalence of infection

To maximize accuracy while minimizing cost

Depends on objectives of the test and the prevalence of infection

Table 2 UNAIDs and SHO reccommendations for HIV testing strategies Tableau 2 Recommandations de I’onusida et de I’OMS aux strategies according to test objective and prevalence of infection in the de depistage du VIH, en fonction de l’obectif du test et de la sample population prevalence de l’infection dans la population Objective of testing Prevalences of infections Testing strategy Objectif du d é pistage Pr é valences de l’infection Strat é gie de d é pistage Transfustion/transplant safety All Prevalences S écurité des transfusions/transplantations Toutes prévalences  Surveillance >10%   10%  Clinical signs/symptoms of >30%  HIV Infection- Signnes Cliniques/symptôms de  30%  l’infection à VIH a Asymptomatic >10%  Asymptomatique  10%  a World Health Organizaion, Intenm proposal for a WHO staging system for HIV infection and desease (WER no.29, 1990, pp 221-228)- Organisation mondiale de la sante. Echelle provisoire OMS proposee pour la determinationdes strades de l’infecrtiono VIH et de la malodie (REN no 29, 1990. P.221-228)

Considerations When Determining the Utility of Tests Choice and Number of Samples

Choice and Number of Samples Samples: Should represent population where test will be performed. Same matrix of sample (e.g. plasma). Must meet guidelines stated by manufacturer (e.g. not lipemic). Avoid multiple freeze/thaw, etc. Use “clean”samples. Multiple aliquots if possible. Must be well categorized.

Samples:

Should represent population where test will be performed.

Same matrix of sample (e.g. plasma).

Must meet guidelines stated by manufacturer (e.g. not lipemic).

Avoid multiple freeze/thaw, etc.

Use “clean”samples.

Multiple aliquots if possible.

Must be well categorized.

Choice and Number of Samples Samples: Should represent population where test will be performed. Same matrix of sample (e.g. plasma). Must meet guidelines stated by manufacturer (e.g. not lipemic). Avoid multiple freeze/thaw, etc. Use “clean”samples. Multiple aliquots if possible. Numbers of Samples: The more the better (min. 30 positives, 200 negatives). Depends on purpose of testing (e.g. blood donors). Include appropriate percent of variants. Perform precision and reproducibility studies (lg. Volumes).

Samples:

Should represent population where test will be performed.

Same matrix of sample (e.g. plasma).

Must meet guidelines stated by manufacturer (e.g. not lipemic).

Avoid multiple freeze/thaw, etc.

Use “clean”samples.

Multiple aliquots if possible.

Numbers of Samples:

The more the better (min. 30 positives, 200 negatives).

Depends on purpose of testing (e.g. blood donors).

Include appropriate percent of variants.

Perform precision and reproducibility studies (lg. Volumes).

HIV Classification HIV HIV-1 HIV-2 O M A, B, C, D, E, F, G, H, I, J Types Clades ANT 70, MVP5180, VAU ROD NIH2 Groups Guidelines for Classification Types: HIV-1 and HIV-2 50% homology Subtypes/Groups: HIV-1 group M, N and O 60-70% homology Clades: HIV-1 Clades A-J >70% homology N

Considerations When Determining the Utility of Tests Testing Conditions

Testing Conditions Must test under identical conditions. (e.g. same lab, equipment, day, tech). Use non-expired kits that have been properly stored. Follow manufacturer’s recommendations. Sample integrity. Test in a blinded fashion.

Must test under identical conditions.

(e.g. same lab, equipment, day, tech).

Use non-expired kits that have been properly stored.

Follow manufacturer’s recommendations.

Sample integrity.

Test in a blinded fashion.

Considerations When Determining the Utility of Tests Resolution of Discordant Results

Resolution of Discordant Results Check sample integrity, labeling, paperwork, and procedures. Repeat by same technologist. Repeat blindly by another technologist. Repeat reference test blindly. Repeat at different laboratory. Determine true status by other means. What parameters would these investigate?

Check sample integrity, labeling, paperwork, and procedures.

Repeat by same technologist.

Repeat blindly by another technologist.

Repeat reference test blindly.

Repeat at different laboratory.

Determine true status by other means.

What parameters would these investigate?

Resolution of Discordant Results Possible Variants Synthetic peptide tests Specific Western blots Specific IFAs Combination tests Dot blots Immunoconcentration tests Augmented blots and LIA PCR - specific

Synthetic peptide tests

Specific Western blots

Specific IFAs

Combination tests

Dot blots

Immunoconcentration tests

Augmented blots and LIA

PCR - specific

Rapid Assay Evaluation Algorithm Rapid Assay + ELISA - Rapid Assay - ELISA + Discordant Results Repeat Rapid & ELISA Western Blot Assay (FDA Licensed) Negative Indeterminate Positive Resolved IFA (FDA Licensed) Negative Indeterminate Positive Resolved Sample Volume > 1 mL Sample Volume (<1 mL & >0.2 mL) RT-PCR Assay Negative Positive Inconclusive Resolved P24 Ag Assay (FDA Licensed) Negative Positive Inconclusive Ag Neutralization Positive Negative Resolved Resolved

Considerations When Determining the Utility of Tests Indicators of Test Performance

Indicators of the Value of a Diagnostic Assay Sensitivity Specificity Test efficiency Delta values Predictive values

Sensitivity

Specificity

Test efficiency

Delta values

Predictive values

Sensitivity of Tests Sensitivity (epidemiologic) Sensitivity (analytical) Low titer Seroconversion Dilutions

Sensitivity (epidemiologic)

Sensitivity (analytical)

Low titer

Seroconversion

Dilutions

Indicators of the Value of a Diagnostic Assay Sensitivity = True Positives True Positives + False Negatives Specificity = True Negatives True Negatives + False Positives X 100% X 100%

Sensitivity = True Positives

True Positives + False Negatives

Indicators of the Value of a Diagnostic Assay Positive Predictive = True Positives Value True Positives + False Positives X 100% X 100% Negative Predictive = True Negatives Value True Negatives +False Negatives

Positive Predictive = True Positives

Value True Positives + False Positives

Predictive Values Assume: Test Sensitivity = 100% / Specificity = 99.5% Population #1 , where the prevalence of infection is high (5%) Population: 1000 sera tested 50 sera from infected individuals 950 sera from non-infected individuals Test Results: 50 positives: 45 from the infected group 5 false pos from the non-infected group Therefore, the positive predictive value is: PPV = 45 = 90% 45+5 9 out of 10 positive results will be from infected persons

Assume: Test Sensitivity = 100% / Specificity = 99.5%

Population #1 , where the prevalence of infection is high (5%)

Population: 1000 sera tested

50 sera from infected individuals

950 sera from non-infected individuals

Test Results: 50 positives: 45 from the infected group

5 false pos from the non-infected group

Therefore, the positive predictive value is:

PPV = 45 = 90%

45+5

9 out of 10 positive results will be from infected persons

Predictive Values Assume: Test Sensitivity = 100% / Specificity = 99.5% Population #2 , where the prevalence of infection is low (0.7%) Population: 1000 sera tested 7 sera from infected individuals 993 sera from non-infected individuals Test Results: 7 positives: 2 from the infected group 5 false pos from the non-infected group Therefore, the positive predictive value is: PPV = 2 = 28.6% 2+5

Assume: Test Sensitivity = 100% / Specificity = 99.5%

Population #2 , where the prevalence of infection is low (0.7%)

Population: 1000 sera tested

7 sera from infected individuals

993 sera from non-infected individuals

Test Results: 7 positives: 2 from the infected group

5 false pos from the non-infected group

Therefore, the positive predictive value is:

PPV = 2 = 28.6%

2+5

Predictive Values Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations

Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations

Predictive Values Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations. The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives).

Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations.

The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives).

Predictive Values Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations. The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives). This indicates that a positive result by the test will be from an infectd individual in only one of four cases (a guess could yield better chance!).

Therefore, the same test that yields the same number of false-positives produces a different positive predictive value when testing two different populations.

The chance of a positive result being from a truly infected individual in the low prevalence population is only 28.6% (2 true positive detected by the test and 5 false-positives).

This indicates that a positive result by the test will be from an infectd individual in only one of four cases (a guess could yield better chance!).

Test Verification &Test Validation Quality Assurance and Errors A. Common Errors B. Quality Assurance Needs 1. Fundamentals of QA 2. Quality Control 3. Quality Assessment 4. Equipment Issues 5. 10 Key Issues for QA

Quality Assurance and Errors

A. Common Errors

B. Quality Assurance Needs

1. Fundamentals of QA

2. Quality Control

3. Quality Assessment

4. Equipment Issues

5. 10 Key Issues for QA

Most Common Errors Transcription Carelessness Procedures Specimens Environmental conditions Pipettes and pipetting

Transcription

Carelessness

Procedures

Specimens

Environmental conditions

Pipettes and pipetting

Clerical Errors Logging specimens Aliquoting Worksheets Result printouts Translating results Computer entering Reports Supervisory Review

Logging specimens

Aliquoting

Worksheets

Result printouts

Translating results

Computer entering

Reports

Supervisory Review

Specimen Problems Insufficient volume for repeating Hemolysis, lipemia, and bacterial contamination Insufficient and inadequate labeling Misidentified specimens Frozen / Thawed (multiple)

Insufficient volume for repeating

Hemolysis, lipemia, and bacterial contamination

Insufficient and inadequate labeling

Misidentified specimens

Frozen / Thawed (multiple)

Other Types of Errors Kit Dependent Problems. Technologist – dependent errors. Inter-lot variations and Intra-lot variations. Environmental problems. Non repeatable results. Inter-laboratory and Intra-laboratory variations. Equipment problems.

Kit Dependent Problems.

Technologist – dependent errors.

Inter-lot variations and Intra-lot variations.

Environmental problems.

Non repeatable results.

Inter-laboratory and Intra-laboratory variations.

Equipment problems.

Quality Assurance Fundamental for Quality Test Results Record keeping Monitoring laboratory staff Vigilance in the laboratory Verification of true positive and true negatives Parallel testing of resubmitted samples Reporting of results Confidentiality Interaction with physicians Storage of specimens for follow-up testing Laboratory efficiency Total quality management

Record keeping

Monitoring laboratory staff

Vigilance in the laboratory

Verification of true positive and true negatives

Parallel testing of resubmitted samples

Reporting of results

Confidentiality

Interaction with physicians

Storage of specimens for follow-up testing

Laboratory efficiency

Total quality management

Components of Quality Control Record Keeping Kit lot numbers (expiration and open dates). Clearly label reagents with date opened or prepared (include open and expiration date) on each label. Daily temperature monitoring and recording i.e. Incubators water baths, ambient. Performance of controls and action taken when out-of-range. Photograph or clear photocopies of Western blots. Ratios of in-house controls to cut-off values.

Kit lot numbers (expiration and open dates).

Clearly label reagents with date opened or prepared (include open and expiration date) on each label.

Daily temperature monitoring and recording i.e. Incubators water baths, ambient.

Performance of controls and action taken when out-of-range.

Photograph or clear photocopies of Western blots.

Ratios of in-house controls to cut-off values.

Components of Quality Control Controls Kit controls : Use as directed by the manufacturer. In-house controls : preferably three levels to monitor variability between runs and lot numbers of kits. Low positive – absorbance enough above cut-off that it should not be misclassified because of expected run-to-run variability. High positive – well above the cut-off. Negative – well below cut-off. Storage of in-house control sera : Dispense in aliquots sufficient for one week of use. Freeze at -20 °C in a non-self-defrosting freezer. Thaw each aliquot once, store at 4 °C when not in use, do not refreeze and discard after 1 week.

Kit controls : Use as directed by the manufacturer.

In-house controls : preferably three levels to monitor variability between runs and lot numbers of kits.

Low positive – absorbance enough above cut-off that it should not be misclassified because of expected run-to-run variability.

High positive – well above the cut-off.

Negative – well below cut-off.

Storage of in-house control sera :

Dispense in aliquots sufficient for one week of use.

Freeze at -20 °C in a non-self-defrosting freezer.

Thaw each aliquot once, store at 4 °C when not in use, do not refreeze and discard after 1 week.

Trend Monitoring by External Controls Shift

Quality Assessment Internal Quality Assessment Known Reactors Unknown Reactors Blind Testing External Quality Assessment Proficiency Panels Blind Proficiency Panels

Internal Quality Assessment

Known Reactors

Unknown Reactors

Blind Testing

External Quality Assessment

Proficiency Panels

Blind Proficiency Panels

Equipment Issues Pipette Calibrations ESSENTIAL FOR ACCURACY Frequency At least every 6 months Reasoning 1  l inaccuracy = 10% error (total volume of 10  l) Controls – o.k., borderline specimens – loss of sensitivity

Pipette Calibrations

ESSENTIAL FOR ACCURACY

Frequency

At least every 6 months

Reasoning

1  l inaccuracy = 10% error (total volume of 10  l)

Controls – o.k., borderline specimens – loss of sensitivity

Quality Assurance: What Must Be Done? 10 Key Issues Detailed SOP with total compliance. Supervising review of all paperwork. Develop checklists for monitoring all activities. Dev. Organizational schemes for processing, documentation, and assessment. Monitor staff – blind proficiencies. Neat and complete documentation of all results. No deviation from procedures. Maintain confidentiality. Endorse safety measures. Vigilance.

Detailed SOP with total compliance.

Supervising review of all paperwork.

Develop checklists for monitoring all activities.

Dev. Organizational schemes for processing, documentation, and assessment.

Monitor staff – blind proficiencies.

Neat and complete documentation of all results.

No deviation from procedures.

Maintain confidentiality.

Endorse safety measures.

Vigilance.

Test Verification &Test Validation III. Introduction of a New Test A. Selection B. Characteristics C. Approved versus Non-Approved tests D. Continual Monitoring

Selection Availability Appropriateness Cost and bulk purchases Shelf life and robustness Storage Publications and WHO evaluations Regulations

Availability

Appropriateness

Cost and bulk purchases

Shelf life and robustness

Storage

Publications and WHO evaluations

Regulations

Characteristics Laboratory capabilities Testing Purpose Simplicity Cost Concerns Sample type Test limitations Test principles and antigens Test indices

Laboratory capabilities

Testing Purpose

Simplicity

Cost Concerns

Sample type

Test limitations

Test principles and antigens

Test indices

Approved Versus Non-approved Tests Which can be used? When approved tests are unavailable. Validation of non-approved tests. Documentation necessities and qualifications.

Which can be used?

When approved tests are unavailable.

Validation of non-approved tests.

Documentation necessities and qualifications.

Continual Monitoring Necessity to monitor new tests. How long to monitor. Methods of monitoring. Looking for trends. Changing tests – Parallel testing. Documentation.

Necessity to monitor new tests.

How long to monitor.

Methods of monitoring.

Looking for trends.

Changing tests – Parallel testing.

Documentation.

Test Verification &Test Validation IV. Special Considerations for Developing Countries A. Selection of Tests and algorithms B. Testing under non-optimal conditions. C. Best fit Strategies D. When Systems Fail

Special Considerations for Developing Countries Selection of Tests and Algorithms

Selection of Tests Infrastructure Supportability Expertise Accessibility Cost Concerns Algorithms

Infrastructure

Supportability

Expertise

Accessibility

Cost Concerns

Algorithms

Algorithms What’s effective? What can be used? Established and recommended algorithms. Use of additional strategies. Differences due to geographical origins of samples. Cost effectiveness. Sample pooling. Blood donations vs. diagnostic testing. Different algorithms within the same country. Epidemiological testing.

What’s effective?

What can be used?

Established and recommended algorithms.

Use of additional strategies.

Differences due to geographical origins of samples.

Cost effectiveness.

Sample pooling.

Blood donations vs. diagnostic testing.

Different algorithms within the same country.

Epidemiological testing.

Simple, Rapid Test Alternative Algorithm Rapid Test #1 Positive Negative P/N OR P/P Rapid Test #2* Repeat in Duplicate REPORT N/N REPORT Positive Negative REPORT indeterminate Resolve with other tests Report as Positive *Different configuration or antigens

Special Considerations for Developing Countries Testing Under Non-optimal Conditions

Testing Under Non-optimal Conditions Use of expired kits. Unsatisfactory environmental conditions. Limited number of test kits. Limited equipment (e.g. thermometers). Non-calibrated pipettes. Old equipment. Poor integrity of samples. Questionably labeled specimens.

Use of expired kits.

Unsatisfactory environmental conditions.

Limited number of test kits.

Limited equipment (e.g. thermometers).

Non-calibrated pipettes.

Old equipment.

Poor integrity of samples.

Questionably labeled specimens.

Special Considerations for Developing Countries Best Fit Strategies

Best-fit Strategies (to Test or Not to Test?) Consequences and necessities. Cost effective strategies. Pooling of samples. Saving reagents. Parallel testing. Sequential testing. Mixing reagents. Alternate testing areas. Testing when temperatures and conditions fail.

Consequences and necessities.

Cost effective strategies.

Pooling of samples.

Saving reagents.

Parallel testing.

Sequential testing.

Mixing reagents.

Alternate testing areas.

Testing when temperatures and conditions fail.

Pooling of Samples In what situations can pooling be used? How many samples can be pooled? Accuracy. Final sample dilution of pools. Proper sample size for evaluation. Effects of the presence of HIV Antigens.

In what situations can pooling be used?

How many samples can be pooled?

Accuracy.

Final sample dilution of pools.

Proper sample size for evaluation.

Effects of the presence of HIV Antigens.

Non-approved Testing Strategies Re-use of rapid tests. Modification of test kits: Cutting WB strips. Halving reagents. Pooling of samples.

Re-use of rapid tests.

Modification of test kits:

Cutting WB strips.

Halving reagents.

Pooling of samples.

Special Considerations for Developing Countries When Test Systems Fail

When Test Systems Fail Trouble shooting. Repeat testing. Alternative testing. Other personnel, other laboratories. Getting help. Documentation. Reporting. Consequences.

Trouble shooting.

Repeat testing.

Alternative testing.

Other personnel, other laboratories.

Getting help.

Documentation.

Reporting.

Consequences.

Reasons for the Need for Improved Assays Early diagnosis. Resolution of indeterminate results. Differentiation of retroviral infections. Less expensive tests. Simple and foolproof tests. Detection of viral types and variants. Multiple combination tests. Detection of infection in the newborn.

Early diagnosis.

Resolution of indeterminate results.

Differentiation of retroviral infections.

Less expensive tests.

Simple and foolproof tests.

Detection of viral types and variants.

Multiple combination tests.

Detection of infection in the newborn.