Assessing the performance of diagnostic tests
Johanna Lindahl
Laboratory review meeting
Ouagadougou, Burkina Faso
19 December 2019

Screening tests
• To distinguish apparently healthy animals from
infected animals
• For disease surveillance
– To measure disease burden in animal
populations
– To certify that an animal herd or region is free
from a specific disease
– For early detection of infection or sub-clinical
disease in animals
– For making management decisions
Issues
• False positives
• False negatives

Diagnostics properties
• Accuracy
• Sensitivity (SE)
• Specificity (SP)
• Predictive values

Classification of results
Infected Non-infected Total
Test positive a (TP) b (FP) a+b
Test negative c (FN) d (TN) c+d
a+c b+d (a+b+c+d)(N)

True prevalence
Proportion of truly infected animals
Infected / N
(a+c)/(a+b+c+d)

True prevalence
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
10 148 158 (N)

Apparent prevalence
• Proportion of test positive animals
Test positive / N
(a+b)/(a+b+c+d)

Apparent prevalence
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
10 148 158 (N)

True prevalence
• If all you have is your test result and estimates
for Se/Sp
True Prevalence = AP*Se+(1-AP)(1-Sp)

Accuracy
• Proportion of infected and non-infected
animals correctly classified by the test
TP + TN / N
(a+d)/(a+b+c+d)

Sensitivity
• Ability of a test to detect infected animals
• Proportion of infected animals that test
positive
TP / Infected
Infected = TP + FN
a/(a+c)

Sensitivity
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
10 148 158 (N)

Specificity
• Ability of a test to detect non-infected
animals
• Proportion of non-infected animals that
test negative
TN / Not infected
Not Infected = TN + FP
d/(b+d)

Specificity
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
10 148 158 (N)

Picking a test using Se/Sp
If the outcome will be expensive or catastrophic:
Minimize false positives
• Test with high specificity
If the penalty for missing a case is high (e.g., the
disease is fatal, or disease easily spreads):
Maximize true positives
• Test with high sensitivity

Predictive values
• It reflects the way test results are used in
the clinic, hospital, free-living population
• If the test result is negative, what is the
probability that this patient is non-infected
• If the test result is positive, what is the
probability that this patient is infected
• It is used as a method for test selection
• It is affected by the SE and SP of the test,
as well as disease prevalence (I+)

Negative predictive value
• Proportion of non-infected animals among
those that test negative
TN / Test negative
Test negative = TN + FN
c/(c+d)

Negative predictive value
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
100 148 158 (N)

Positive predictive value
• If the test result is positive, what’s the probability
that this patient is infected?
• If we screen a population, what’s the proportion
of animals who have the infection will be
correctly identified?
• Proportion of infected animals among those that
test positive
TP / Test positive
Test positive = TP +
FP
a/(a+b)

Positive predictive value
Infected Non-infected Total
Test positive 7 (TP) 9 (FP) 16
Test negative 3 (FN) 139 (TN) 142
10 148 158 (N)

Selection of diagnostic tests
• High SE and Negative Predictive Value
when it is important to reduce the number
of FN
• Avoid introduction of disease

Selection of diagnostic tests
• High SP and Positive Predictive Value
when it is important to reduce the number
of FP
• To confirm a diagnosis
• Avoid unnecessary elimination of animals

Testing in duplicate

Testing in parallel
• The results of two or more tests must be
negative
• To increase SE and Negative Predictive Value
• The goal is to maximize the probability that
subjects with the disease (true positives) are
identified (increase sensitivity)
• Consequently, more false positives are also
identified (decrease specificity)
• Net sensitivity = Se1 + Se2 –(Se1*Se2)
• Net specificity = Sp1*Sp2

Testing in parallel
Infected Non-infected Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive 8 1 9 SE = 80%
Test negative 2 147 149 SP = 99%
10 148 158
Net sensitivity = 0.7+ 0.8–(0.7*0.8) = 0.94
Net specificity = 0.94*0.99=0.93

Testing in series
• The results of two tests must be positive
• Only use second test when the first test is
positive
• To increase SP and Positive Predictive
Value
• Testing in series leads to a net loss in
sensitivity and a net gain in specificity

Testing in series
Infected Non-
infected
Total Test 1
Test
positive
SE = 70%
Test
negative
SP = 94%
10 148 158

Testing in series
Infected Non-
infected
Total Test 1
Test
positive
7 9 16 SE = 70%
Test
negative
3 139 142 SP = 94%
10 148 158

Testing in series
Infected Non-
infected
Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive SE = 80%
Test negative SP = 99%
7 9 16

Testing in series
Infected Non-infected Total Test 1
Test positive 7 9 16 SE = 70%
Test negative 3 139 142 SP = 94%
10 148 158
Test 2
Test positive 6 0 6 SE = 80%
Test negative 1 9 10 SP = 99%
7 9 16
Overall SE = (10 - 3 - 1 = 6) / 10 = 60%
Overall SP = (139 + 9 = 148) / 148 = 100%
Overall positive predictive value = 6 / 6 = 100%

Testing in series – negative sample
Infected Non-infected Total Test 1
Test positive 70 90 160 SE = 70%
Test negative 30 1390 1420 SP = 94%
100 1480 1580
Test 2
Test positive 24 14 38 SE = 80%
Test negative 6 1376 1382 SP = 99%
30 1390 1420
Overall SE = (70+24 = 94) / 100 = 94%
Overall SP = (1480-90-14 = 1376) / 1480 = 93%

Policy considerations

Policy implications –testing in series
Infected Non-
infected
Total RBPT
Test positive 40 20 60 SE = 100%
Test negative 0 1940 1940 SP = 99%
40 1960 2000
C-ELISA
Test positive 40 0 40 SE = 100%
Test negative 0 20 20 SP = 99.9%
40 20 60
Overall SE = (40 - 0 = 40) / 40 = 100%
Overall SP = (1940 + 20 = 1960) / 1960 = 100%
Overall positive predictive value = 40 / 40 = 100%

Policy implications
Plan A
C-ELISA
2,000 samples x $5 = $10,000
TOTAL: $10,000
Plan B
RBPT
2,000 samples x $1 = $2,000
C-ELISA
60 samples x $5 = $300
TOTAL: $2,300

Relationship between disease prevalence and
positive predictive value
• SE=99% and SP=95%
Disease No
Disease
Total PPV
1% + 99 495 594 17%
- 1 9405 9406
Total 100 9900 10000
10% + 990 450 1440 69%
- 10 8550 8560
Total 1000 9000 10000

Measuring diagnostic test performance
C-ELISA
Positive Negative Total
RBPT Positive 35 6 41
Negative 0 1976 1976
Total 35 1982 2017
Kappa = 0.92

Diagnostic test evaluation

Policy for diagnostic tests

@OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
Factors affecting antimicrobial susceptibility
measurements
• Medium type (Mueller-Hinton, Iso-sensitest, Sensitest
medium)
• Medium manufacturer
• Lot-to-lot variation for both medium and disks
• Effect of additives (e.g. blood)
• Inoculum size and concentration
• Incubation conditions (temperature and duration)
• Human factors (e.g. preparation of dilutions)
Quality control is
essential

How is quality control done?
<Udfyld sidefod-oplysninger her>
• Reference strains of different species should routinely be
included in the testing
• The MIC (or inhibition zone diameter) of the reference strain
has to fall within a given range to validate the test
• If not, the test is not validated and should be repeated
Send also
samples
regularly to
diagnostic labs
to validate
results

@OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
QC Antimicrobial Susceptibility Testing - Module 8 42
AST Methods Interpretation
• agar disk diffusion method provides qualitative interpretive category
results of susceptible, intermediate, and resistant
• microdilution and agar gradient diffusion methods provide a
quantitative result, a minimum inhibitory concentration

QC Antimicrobial Susceptibility Testing -
Module 8
43
AST Methods

QC Antimicrobial Susceptibility Testing -
Module 8
44
44
 Clinical and Laboratory Standards Institute
 French Society of Microbiology
 British Society for Antimicrobial Chemotherapy
References

QC Antimicrobial Susceptibility Testing -
Module 8
45
45
Where errors can occur in susceptibility
testing
•media
•antimicrobials
•inoculum
•incubation
•equipment
•interpretation

QC Antimicrobial Susceptibility Testing -
Module 8
46
46
Reference Strains
E. coli ATCC 25922
S. aureus ATCC 25923
P. aeruginosa ATCC 27853
QC organisms must be obtained from
reputable source
Use specific QC organisms to test different
groups of “drug-bug” combinations

QC Antimicrobial Susceptibility Testing -
Module 8
47
Selection of a Colony to Test
47

QC Antimicrobial Susceptibility Testing -
Module 8
48
48
Disk Susceptibility Testing Problems

QC Antimicrobial Susceptibility Testing -
Module 8
49
Disk Susceptibility Testing Problems

QC Antimicrobial Susceptibility Testing -
Module 8
50
50
Measuring Conditions
RulerCalipers
read with good light, and from the back of the plate
zone size reading is drug specific
magnification may help
millimeters matter

QC Antimicrobial Susceptibility Testing -
Module 8
51
Etest – antimicrobial gradient method
51

@OneHealthHORNHORN@liverpool.ac.ukwww.OneHealthHORN.net
QC Antimicrobial Susceptibility Testing - Module 8 52
Patient results may be incorrect if:
•the organism was misidentified
•a clerical error was made
•inappropriate choice of antimicrobials were tested
and reported
•the wrong patient’s sample was examined
•the wrong test was ordered
•the sample was not preserved properly

This presentation is licensed for use under the Creative Commons Attribution 4.0 International Licence.
better lives through livestock
ilri.org
ILRI thanks all donors and organizations which globally support its work through their contributions
to the CGIAR Trust Fund