Sensitivity specificity
•
7 likes•6,306 views
The document discusses concepts of sensitivity and specificity as they relate to medical testing and how they can also apply to monitoring systems. It provides examples to explain sensitivity as the proportion of actual positives detected, while specificity refers to actual negatives detected. The document uses a word problem about detecting plagiarism to demonstrate calculating sensitivity, specificity, and positive predictive value. It emphasizes how specificity becomes more important as uptime increases to avoid high false positive rates. The document provides advice around focusing on both sensitivity and specificity for monitoring systems.
Recommended
More Related Content
What's hot
What's hot (13)
Viewers also liked
Similar to Sensitivity specificity
Similar to Sensitivity specificity (20)
Recently uploaded
Recently uploaded (20)
Sensitivity specificity
- 2. “Contact”, 1997. Warner Bros.
- 3. Who’s this guy? ● Dan Slimmon ● Senior Platform Engineer at Exosite ● Previously Operations Team Manager at Blue State Digital ● https://twitter.com/danslimmon
- 4. Sensitivity & Specificity ● Medical testing concepts ● “How good is your test?” ● Medical concepts often work great for ops
- 6. A word problem You’ve invented an automated test for plagiarism.
- 7. A word problem ● If a paper contains plagiarism, you have a 90% chance of a positive result. ● If a paper doesn’t contain plagiarism, you still have a 20% chance of a positive result. ● Jerkwad kids plagiarize 30% of the time
- 8. Question 1 Given a random paper, what’s the probability that you’ll get a negative result? ● Plagiarism: 90% chance of positive ● No plagiarism: 20% chance of positive ● 30% chance of plagiarism
- 9. Question 2 If there’s plagiarism, what’s the probability you’ll detect it? ● Plagiarism: 90% chance of positive ● No plagiarism: 20% chance of positive ● 30% chance of plagiarism
- 10. Question 2 If there’s plagiarism, what’s the probability you’ll detect it? ● Plagiarism: 90% chance of positive ● No plagiarism: 20% chance of positive ● 30% chance of plagiarism
- 11. Question 3 If you get a positive result, what’s the probability that the paper is plagiarized? ● Plagiarism: 90% chance of positive ● No plagiarism: 20% chance of positive ● 30% chance of plagiarism
- 15. Question 1 Given a random paper, what’s the probability that you’ll get a negative result?
- 17. Question 2 If the paper is plagiarized, what’s the probability that you’ll get a positive result?
- 19. Question 3 If you get a positive result, what’s the probability that the paper was plagiarized?
- 21. Question 3 If you get a positive result, what’s the probability that the paper was plagiarized? Dark Green -----------------------------------------(Dark Blue) + (Dark Green)
- 22. Question 3 If you get a positive result, what’s the probability that the paper was plagiarized? 27 -----------------------------------------14 + 27
- 23. Question 3 If you get a positive result, what’s the probability that the paper was plagiarized? 65.8%
- 24. Sensitivity & Specificity Sensitivity: Specificity: Proportion of actual positives that are identified as such. Proportion of actual negatives that are identified as such.
- 25. Sensitivity & Specificity Sensitivity: Specificity: High sensitivity High specificity Test is very sensitive to Test works for a problems specific type of problem
- 26. Sensitivity & Specificity Sensitivity: Specificity: High sensitivity High specificity High true positive rate High true negative rate
- 27. Sensitivity & Specificity Sensitivity: Probability that, if a paper is plagiarized, you’ll get a positive. Specificity: Probability that, if a paper isn’t plagiarized, you’ll get a negative. 90% 80%
- 30. Positive Predictive Value The probability that If you get a positive result, Then it’s a true positive.
- 32. When you get paged at 3 AM, Positive Predictive Value is the probability that something is actually wrong.
- 33. Imagine if you will... ● Service S has 99.9% uptime ● Probe P for service S has 99% sensitivity ● Probe P has 99% specificity
- 35. Let’s calculate the PPV.
- 37. The true-positive probability Let’s calculate the probability that any given probe run will produce a true positive. P(TP) = (prob. of service failure) * (sensitivity) P(TP) = 0.1% * 99% P(TP) = 0.099%
- 38. The true-positive probability P(TP) = 0.099% So roughly 1 in every 1000 checks will be a true positive.
- 39. The false-positive probability P(FP) = (prob. working) * (100% - specificity) P(FP) = 99.9% * 1% P(FP) = 0.99% So roughly 1 in every 100 checks will be a false positive.
- 40. “Star Wars Episode V: The Empire Strikes Back”, 1980. Lucasfilm.
- 41. Positive predictive value PPV = P(TP) / [P(TP) + P(FP)] PPV = 0.099% / (0.099% + 0.99%) PPV = 9.1% If you get a positive, there’s only a 1 in 10 chance that something’s actually wrong.
- 42. Why is this terrible?
- 45. You want smoke alarms, not car alarms.
- 47. Semi-Practical Advice As your uptime increases, so must your specificity. It affects your PPV much more than sensitivity.
- 50. Semi-Practical Advice Undetected outages are embarrassing, so we tend to focus on sensitivity. But be careful with thresholds.
- 52. Semi-Practical Advice Use hysteresis (stateful probes, trend analysis, etc.)
- 53. Semi-Practical Advice Separate the concerns of problem detection and problem identification
- 54. A Pony I Want Something like Nagios, but which ● Is SNR-aware ● Helps you separate detection from diagnosis
- 55. Other useful stuff ● Medical paper with a nice visualization: http://tinyurl.com/specsens ● Blog post with some algebra: http://tinyurl.com/carsmoke ● Base rate fallacy: http://tinyurl.com/brfallacy ● Differential diagnosis: http://tinyurl.com/sbddx