evaluating a diagnostic test

1. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
What Is Normal?
Dr. Alaa El Dine Hassan
MB BCh, MSc (PH), MD
Consultant- Community and
Preventive Medicine

2. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Can You Define this “Thing”?
You can define
If Yes
Normality

3. Dr. Alaa Hassan - MB BCh, MSc (PH), MD

4. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
1. The Common
• Normal is synonymous with “Common" .
• The (95% probability limits) constitute
the most widely accepted normal range.
The normal height of young men is the range of
heights that includes 95% of men.

5. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
2. Not Ill
• Clearly a definition of illness would have to
be devised.
• Illness might thus be defined as anything
that causes
– incapacitation
– pain
– swelling
– bleeding, or
– infection
But is sleep then abnormal?
Would child-birth constitute ill-health?
Is an erection a surgical emergency?
Would menstruation be a sickness?
Is colonic fermentation abnormal?

6. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
3. Operating as Intended
• The muscles are designed to contract
• The bones are designed to support
• What is the intention of axillary hair?
• what the appendix is designed to do?
• what is the purpose of our nails?

7. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
4. Conforming to Cultural Norm
• Acceptable Weight standards may vary
according to culture
• Nose piercing is normal for Hindu
• Nudity is the norm in some African
tribes

8. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
5. The Best
• desirable/optimal/favorable/advantageous
• Maximization of the useful life-span so
that, the degenerative diseases of old age
become the commonest causes of death.

9. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
6. The Therapeutic Definition
• The McMaster Group provide this sixth
definition of normality, which is not used in
conversational English, but which has great
clinical utility.
• Here normality is that range of values (of e.g.
arterial blood pressure, or blood sugar)
where therapy is not indicated (does more
harm than good).
• Patients are therefore not labeled abnormal
unless they are to be treated.

10. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
What’s Exactly Your
Point here?

11. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
If the definition of normality is so
controversial ,
how difficult would it be for a TEST to
distinguish normal from abnormal ?

12. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Diagnostic Tool
• Any procedure (X-ray, Lab measurement,
questionnaire, Scale etc..) which aims to
differentiate Normal from abnormal.

13. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Evaluating a Diagnostic Tool

14. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
A diagnostic tool is evaluated in terms
of two distinct features. Namely:
ValidityReliability
(Precision)

15. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
1. Reliability of Diagnostic Tool
• The Percent of agreement or consistency
between multiple observations,
measurements, tests or responses in
regard to the same unit of observation.
• Repeatability
• Giving the same result when applied
several times to the same person.

16. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Types of Reliability
Inter-test reliability
Inter-observer reliability
Intra-observer reliability
Respondent Reliability

17. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Difference between results obtained on the
same person by two or more observers
Inter-observer reliability
Physician 1 Physician 2Same Patient
(Blood Pressure)
140/90 130/80

18. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Difference between the results obtained by
the same observer on the same person
Intra-observer reliability
6:00 pm 6:00 pm
Same Patient
Same Physician
(Blood Pressure)
140/90 130/80

19. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Difference between readings of two tests on
the same case or specimen.
Inter-test reliability
Digital Manometer Mercury Manometer
Same Patient
(Blood Pressure)
140/90 130/80

20. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
A special type of inter-observer reliability where there is a
difference between the response of the same individual
to the same question posed by two different interviewers
Respondent Reliability
Interviewer 1 Interviewer 2
(Same Patient )
Sexual Satisfaction
Question
YES NO

21. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Measurement of reliability
To measure Reliability of blood
Pressure measurement , two
observers were to measure
the systolic blood pressure of
the same 10 patients.

22. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Subjects Observer 1 Observer 2 Agreement
A 90 90 +
B 90 94 -
C 85 84 -
D 88 88 +
E 92 92 +
F 80 82 -
G 85 85 +
H 98 95 -
I 98 100 -
J 90 90 +
Agreement = 5/10 X100 = 50%

23. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
2. Validity
• This is the accuracy of the test
• The ability of the test to measure what it is
intended to measure.

24. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Aspects of Diagnostic tool Validity
1. Face Validity
2. Content Validity
3. Construct Validity
4. Criterion
Based Validity

25. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
1. Face Validity
• Not validity in the technical sense
• It refers, not to what the test actually
measures, but to what it appears
superficially to measure.
• Based on intuitive judgement
• The test "looks valid" to the examinees
who take it, the administrative personnel
who decide on its use, and other
technically untrained observers.
Social &
Behavioral
sciences

26. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
2. Content Validity
• Also known as logical validity
• Refers to the extent to which a measure
represents all aspects of a given Disease .
• For example, a depression scale may lack
content validity if it only assesses the
affective aspect of depression but fails to
take into account the behavioral aspect.
Social &
Behavioral
sciences

27. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
3. Construct Validity
• Evaluation of construct validity requires
examining the correlation of the measure being
evaluated with variables that are known to be
related to the construct (Concept) measured by
the instrument being evaluated or for which
there are theoretical grounds for expecting it to
be related (Campbell & Fiske, 1959)
Social &
Behavioral
sciences

28. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Construct Validity:
A. Convergent Validity
• The scores of a test which evaluates
mathematical skills can be correlated with
scores on other tests that measure basic
mathematical ability,
• High correlations would be evidence of
convergent validity.
Social &
Behavioral
sciences

29. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Construct Validity:
B. Discriminant Validity
• To show the discriminant validity of a test
of math skills, we might correlate the
scores of our test with scores on tests of
verbal ability, where low correlations would
be evidence of discriminant validity.
Social &
Behavioral
sciences

30. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
4. Criterion Based Validity
• A defined population is subjected to
– The test under investigation and
– A standard test (Gold Standard)
• The gold standard will determine
– Those who are diseased and
– Those who are disease free
• The performance of the new test is
compared to the Classification of the gold
standard test

31. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Criterion
Based
Validity
Sensitivity
Specificity
Positive Predictive value
Negative Predictive value

32. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Positive Likelihood Ratios
• likelihood ratios are not yet commonly
reported in peer-reviewed literature or in
marketing information provided by test
manufacturers.
• They can be a valuable tool for comparing
the accuracy of several tests to the gold
standard, and
• They are NOT dependent upon the
prevalence of disease.

33. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
The positive LR represents :
the odds of a positive test in an infected population
compared to
the odds of a positive test a non infected population.
TP/ All Diseased
————--------—
FP/ All Disease free
Or it can also be expressed as :
sensitivity
—————
1-specificity

34. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Diseased
Not
Diseased
Test
Positive
TP+FP
(All Test Positive)
Test
Negative
TN+FN
(All Test Negative)
TP+FN
(All Diseased)
TN+FP
(All Disease Free)
TP
TN
FP
FN
Positive Likelihood Ratio

35. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
• Useful tests have larger positive LR
• Less useful tests will have smaller positive
LR.

36. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Example:
A test with a positive LR of 5.0 :
– For every
• 1 non-infected subjects that test positive
• 5 of the infected subjects will test positive.
– For every
• 1 false positive
• 5 true positives
– True positives are 5 times False Positives

37. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
The Negative LR represents :
the odds of a Negative test in an infected population
compared to
the odds of a Negative test in a non infected population.
FN/ All Diseased
------------—————
TN/ All Disease free
Or it can also be expressed as :
1- Sensitivity
—----------------——
Specificity

38. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
Diseased
Not
Diseased
Test
Positive
TP+FP
(All Test Positive)
Test
Negative
TN+FN
(All Test Negative)
TP+FN
(All Diseased)
TN+FP
(All Disease Free)
TP
TN
FP
FN
Negative Likelihood Ratio

39. Dr. Alaa Hassan - MB BCh, MSc (PH), MD
• Useful tests have negative LR close to 0
• Less useful tests have higher negative LR.