1. Sensitivity &
Specificity
& Monitoring

2. “Contact”, 1997. Warner Bros.

3. Who’s this guy?
● Dan Slimmon
● Senior Platform Engineer at Exosite
● Previously Operations Team Manager at
Blue State Digital
● https://twitter.com/danslimmon

4. Sensitivity & Specificity
● Medical testing concepts
● “How good is your test?”
● Medical concepts often work great for ops

5. This talk
contains some
math.

6. A word problem
You’ve invented an automated test for
plagiarism.

7. A word problem
● If a paper contains plagiarism, you have a
90% chance of a positive result.
● If a paper doesn’t contain plagiarism, you
still have a 20% chance of a positive result.
● Jerkwad kids plagiarize 30% of the time

8. Question 1
Given a random paper, what’s the probability
that you’ll get a negative result?
● Plagiarism: 90% chance of positive
● No plagiarism: 20% chance of positive
● 30% chance of plagiarism

9. Question 2
If there’s plagiarism, what’s the probability you’ll
detect it?
● Plagiarism: 90% chance of positive
● No plagiarism: 20% chance of positive
● 30% chance of plagiarism

10. Question 2
If there’s plagiarism, what’s the probability you’ll
detect it?
● Plagiarism: 90% chance of positive
● No plagiarism: 20% chance of positive
● 30% chance of plagiarism

11. Question 3
If you get a positive result, what’s the
probability that the paper is plagiarized?
● Plagiarism: 90% chance of positive
● No plagiarism: 20% chance of positive
● 30% chance of plagiarism

12. No Plagiarism
Plagiarism

13. No Plagiarism
Negative
Positive

14. No Plagiarism
Plagiarism
Negative
Negative
Positive
Positive

15. Question 1
Given a random paper, what’s the probability
that you’ll get a negative result?

16. No Plagiarism
Plagiarism
Negative
Negative
Positive
Positive

17. Question 2
If the paper is plagiarized, what’s the probability
that you’ll get a positive result?

18. No Plagiarism
Plagiarism
Negative
Negative
Positive
Positive

19. Question 3
If you get a positive result, what’s the
probability that the paper was plagiarized?

20. No Plagiarism
Plagiarism
Negative
Negative
Positive
Positive

21. Question 3
If you get a positive result, what’s the
probability that the paper was plagiarized?
Dark Green
-----------------------------------------(Dark Blue) + (Dark Green)

22. Question 3
If you get a positive result, what’s the
probability that the paper was plagiarized?
27
-----------------------------------------14 + 27

23. Question 3
If you get a positive result, what’s the
probability that the paper was plagiarized?
65.8%

24. Sensitivity & Specificity
Sensitivity:
Specificity:
Proportion of actual
positives that are
identified as such.
Proportion of actual
negatives that are
identified as such.

25. Sensitivity & Specificity
Sensitivity:
Specificity:
High sensitivity
High specificity
Test is very sensitive to Test works for a
problems
specific type of
problem

26. Sensitivity & Specificity
Sensitivity:
Specificity:
High sensitivity
High specificity
High true positive rate
High true negative rate

27. Sensitivity & Specificity
Sensitivity:
Probability that, if a
paper is plagiarized,
you’ll get a positive.
Specificity:
Probability that, if a
paper isn’t plagiarized,
you’ll get a negative.
90%
80%

28. Prevalence
Specificity
Sensitivity

29. Questions so
far?

30. Positive Predictive Value
The probability that
If you get a positive result,
Then it’s a true positive.

31. Condition
Absent
Condition
Present
Negative
Result
True
Negative
False
Negative
Positive
Result
False
Positive
True
Positive

32. When you get paged at 3
AM, Positive Predictive
Value is the probability that
something is actually
wrong.

33. Imagine if you will...
● Service S has 99.9% uptime
● Probe P for service S has 99% sensitivity
● Probe P has 99% specificity

34. Pretty decent, right?

35. Let’s calculate the PPV.

36. Condition
Absent
Condition
Present
Negative
Result
True
Negative
False
Negative
Positive
Result
False
Positive
True
Positive

37. The true-positive probability
Let’s calculate the probability that any given
probe run will produce a true positive.
P(TP) = (prob. of service failure) * (sensitivity)
P(TP) = 0.1% * 99%
P(TP) = 0.099%

38. The true-positive probability
P(TP) = 0.099%
So roughly 1 in every 1000 checks will be a
true positive.

39. The false-positive probability
P(FP) = (prob. working) * (100% - specificity)
P(FP) = 99.9% * 1%
P(FP) = 0.99%
So roughly 1 in every 100 checks will be a false
positive.

40. “Star Wars Episode V: The Empire Strikes Back”, 1980. Lucasfilm.

41. Positive predictive value
PPV = P(TP) / [P(TP) + P(FP)]
PPV = 0.099% / (0.099% + 0.99%)
PPV = 9.1%
If you get a positive, there’s only a 1 in 10
chance that something’s actually wrong.

42. Why is this terrible?

43. Car Alarms
http://inserbia.info/news/wp-content/uploads/2013/06/carthief.jpg

44. Smoke Alarms
http://www.props.eric-hart.com/wp-content/uploads/2011/03/nysf_firedrill_2011.jpg

45. You want smoke alarms,
not car alarms.

46. Semi-Practical Advice

47. Semi-Practical Advice
As your uptime increases, so must your
specificity.
It affects your PPV much more than sensitivity.

48. Uptime
Prevalence
False
Negative
Rate
Specificity
Sensitivity
False
Positive
Rate

49. Uptime
Specificity
Sensitivity

50. Semi-Practical Advice
Undetected outages are embarrassing, so we
tend to focus on sensitivity.
But be careful with thresholds.

51. Semi-Practical Advice
Positive
Predictive
Value
Response Time Threshold

52. Semi-Practical Advice
Use hysteresis (stateful probes, trend analysis,
etc.)

53. Semi-Practical Advice
Separate the concerns of problem detection
and problem identification

54. A Pony I Want
Something like Nagios, but which
● Is SNR-aware
● Helps you separate detection from diagnosis

55. Other useful stuff
● Medical paper with a nice visualization:
http://tinyurl.com/specsens
● Blog post with some algebra:
http://tinyurl.com/carsmoke
● Base rate fallacy:
http://tinyurl.com/brfallacy
● Differential diagnosis:
http://tinyurl.com/sbddx