Call Girl Number in Panvel Mumbai📲 9833363713 💞 Full Night Enjoy
Likelihood Ratio, ROC and kappa Statistics
1. Continue Dx Test Evaluation
DR Amita Kashyap
Sr. Prof. PSM
SMS Med College Jaipur
2. Decide whether to order ESR or directly do MRI?
• A 57 yr old man presents with h/o aching low back pain
that persists at rest and is worse by bending and lifting.
• Progressively getting worse in last 6 wks-waking from sleep.
• Within past 10 days he has noticed numbness in Rt buttock
and thigh and weakness in Rt lower limb.
• He had no fever but has lost 10 lb Wt. in last 4 months.
• O/E Temp. is 99.6F, tenderness in the lower lumber spine,
decrease sensation over dorso-lateral aspect of Rt. foot,
weakness in Rt ankle aversion. Deep tendon reflexes Norm.
• You suspect that man has 20% chance of spinal Malignancy
• ESR ≥20 mm/h has 78% sensitivity & 67% specificity
• MRI has 95% sensitivity AND 95% specificity!!
• Suppose we have 1000 patients
3. We can use any of the following methods
• 2 X 2 Table Method
• Likelihood Ratio (gives odds)
• Decision Tree method And
• Bayes Theoram
4. How will the prior probability of 20% Change with +ve ESR
(78% sensitivity & 67% specificity) 2X2 Table method
Disease
Test
(ESR)
D+ D-
T+ (TP)
156
(FP)
264
420
T- (FN)
44
(TN)
536
580
200 800 1000
Predictive Value of a Positive test
PPV (PV+) = 156/420 = 0.37
Increased from 20% to 37%
Predictive Value of a Negative test
NPV = 536/580 = 0.92
A) - What is the probability of FP Rate ?
B)- What is the probability of FN Rate?
A) - 264/420 = 0.63 (hence minimal use as screening test),
B) - 44/580 = 0.08
5. MRI has 95% sensitivity AND 95% specificity
And will have PVs as follows:-
Disease
Test (MRI) D+ D-
T+ (TP) 351.5 (FP) 31.5 383
T- (FN) 18.5 (TN) 598.5 617
370 630 1000
PPV (PV+) = 351.5/383 = 0.918
FP Rate = 31.5/383 = 0.08
NPV (NPV) = 598.5/617= 0.970
FN Rate = 18.5/ 617 = 0.03
6. • Definition: An LR is the probability of a particular test
result for a persons with the disease divided by the
probability of that test result in non-diseased persons
How will the prior probability of 20% change with +ve ESR?
Likelihood Ratio Method
LR+ =
Probability of +ve test result in Diseased persons
(True positive / total diseased i.e. Sensitivity)
Probability of +ve test result in Disease Free persons
(False positive/ total Non-diseased i.e. 1- Specificity)
LR¯ =
Probability of -ve test result in Diseased persons
(False Negative / total diseased i.e. 1- Sensitivity)
Probability of -ve test result in Disease Free persons
(True negatives/ total Non-diseased i.e. Specificity)
8. Pre Test odds = Prior Probability
1- Prior Probability
Post Test odds = LR X Pre test odds
= 2.36 X 0.25 = 0.59
Posterior probability (PPV)
= post test odds/ 1+ post test odds
= 0.59/ 1+ 0.59 = 0.37 = 37%
Likelihood Ratio to
Post Test Probability
= 0.20 /1- 0.2 = 0.25
Pre Test odds (with 20% Prev)
9. 0.3
13
20 40 60 80 100
64
After positive
FNA result
54 yr old women
Before Mammogram
After positive
mammogram
Probability of Breast Cancer (Percent)
palpable lump
Prior to
mammogram
Estimated Probability of Breast Cancer
1) In a 54 yr old women without palpable Breast Mass,
2) After a positive mammogram and
3) Following a positive FNA test result
1% with
H/o Br Ca
In mother
10. How will the prior probability of 13% Change
2X2 Table method
Surgical Biopsy
Cancer No Cancer
FNA
Result
+
-
Total 15 99
22
92
114
Total
Prevalence = 13%
Sensitivity = 93%
Specificity = 92%
PPV = 64%
NPV = 99%
11. LR+ = Sensitivity / 1-specificity
= 0.93 / 1-0.92
= 0.93 / 0.08 = 11.63
LR¯ - 1-Sensitivity) / Specificity
= 1- 0.93 / 0.92
= 0.07/ 0.92 = 0.08
With FNA having 93% sensitivity & 92% specificity
In contrast to PV, LR does not vary as a function of Prevalence
12. Pre Test odds = Prior Probability
1- Prior Probability
Post Test odds = LR X Pre test odds
= 11.63 X 0.15 = 1.74
Posterior probability (PPV)
= post test odds/ 1+ post test odds
= 1.74/ 1+ 1.74 = 0.64 = 64%
Likelihood Ratio to
Post Test Probability
= 0.13 /1- 0.13 = 0.15
Pre Test odds (with 13 % Prev)
13. Receiver Operating Characteristic (ROC) Curve
• Diagnostic tests giving quantitative outcome e.g.
serum levels of enzymes, there are many options
about where to set a cut off point – as the cut off
point rises (from 200 to 250mg/dl for total cholesterol)
the sensitivity will increase with a corresponding
decrease in specificity.
•To get a most suitable cutoff point at each cutoff
point, sensitivity and (1- specificity) are calculated
and plotted on ‘y’ and ‘x’ axis respectively along
the full range of cutoff points
26. Best Test: Worst test:
True
Positive
Rate
0
%
100%
False Positive Rate
0
%
100
%
True
Positive
Rate
0
%
100%
False Positive
Rate
0
%
100
%
The distributions
don’t overlap at all
The distributions
overlap completely
ROC curve extremes
27. Reliability or Repeatability of a Test
• Factors responsible for variation in the results:
1. Intra subject (within the individual) variation
2. Intra observer variation (variation in the
reading of test result by the same observer)-
greater the subjective element in the reading
more is this error
3. Inter observer variation (variation in the
reading of test result between observers)
28. Inter observer Variation
Reading
No. 1
Reading No.
2
Abnormal Suspect Doubtful Normal
Abnormal A B C D
Suspect E F G H
Doubtful I J K L
Normal M N O P
Percent Agreement =
A + F + K + P
Total readings
X 100
29. In general, most persons who are tested have Negative Test Result,
& there is likely to be Considerable agreement on this
Ob 1
Ob 1
30. Kappa Statistics
• The extent to which two observers (physician/
nurse/ radiologist/ Dx Test) agree is an important
Index of good quality of care
• Yet, there is a fraction based ‘solely on chance’
for agreement between two observers
• What we want to know is – to what extent did
the education/ training that the observers
received improve the quality of their observation
(how much increased percent agreement
between them beyond chance! )
31. Rationale of the kappa statistics
• First find out– “how much better is the agreement
between the observers’ readings than would be
expected by chance”
i.e. = (% agreement observed - % agreement
expected by Chance alone)
• We Know that the maximum improvement the
observers can have than expected by chance is:-
100% - % agreement expected by Chance alone
• Kappa Statistics expresses the extent to which
the observed agreement exceeds chance
agreement relative to maximum that the
observer can hope to improve
32. • Kappa =
[Percent Agreement
Observed]
[ Percent Agreement
expected by chance alone ]
-
[ Percent Agreement
expected by chance alone ]
100% -
Landis and Koch suggested that :-
kappa greater than 0.75 = excellent agreement
Kappa of 0.40 to 0.75 = intermediate to good agreement
A decision must be reached on whether to order ESR or proceed directly with lumber MRI. It depends on –
Prior probability of Spinal Malignancy
The accuracy of ESR in
detecting malignancy among who have it (sensitivity)
its ability to label a person not having disease among disease free people (specificity)
Application of probabilistic and statistical principles to individual patient is evidence based medicine
evaluating new diagnostic procedures
determining most cost effective approach
evaluating available Tt options
A physician’s best guess (index of suspicion that pt has the disease – prior probability) depends on knowledge of prev. and is revised upwards or downwards depending up on S/S and other characteristics like race/ age/ sex etc
Decision to do test &/ or Tt depends on the risk of the Diagnostic test, the benefits of Tt to patient, the risk of Tt to patient with and without disease and the accuracy of the Test.
The sizes of the two likelihood ratios indicate the strength of association btw a test result and likelihood of the disease
A diagnostic test with a large LR+ value increases the suspicion of disease for patients with positive results – larger the size better is the diagnostic value of the test…arbitrarily a value of 10 is perceived as an indication of a test of high value for LR+ and 0.1 for LR-
The ROC plot of a given test is obtained by calculating the sensitivity and specificity of every observed value, and then plotting sensitivity (on the Y axis) against 1 - specificity (on the X axis). A test that does not discriminate between normal and abnormal would give a diagonal straight line from the bottom left corner to the top right corner. All points on such a line represent a 1:1 ratio of true to false positives. An ideal test would give a rectangular plot passing from the origin at the bottom left hand corner towards top left hand corner at first and thence to the top right hand corner. In reality the ROC curves of many of the tests in common use fall in between these extremes. The cut-off point for deciding between normal and abnormal is selected arbitrarily where the ROC curve changes direction from being vertical to horizontal. The more the ROC curve arches into the upper left hand corner away from the diagonal, the better the test.
If we ask two roadside persons to mark a set of X-rays as positive and negative indepently there will be some agreement between their readings just by chance
Kappa statistics proposed by Cohen in 1960