The document discusses improving alert systems by reducing unnecessary alerts. It suggests focusing on alerts that require human action, identifying the business impact of alerts, and documenting how each alert is addressed. Non-actionable alerts should be converted to notices or removed. Organizations should design resilience into their systems to minimize reliance on human intervention during incidents.

1. / Robert Treat
Less Alarming Alerts
Saturday, April 9, 16

2. Hello /@robtreat2
Former
WebDev SysAdmin DBA
I have now been promoted to where I can do the least damage
Saturday, April 9, 16

3. Hello /@robtreat2
Now
CEO @OMNITI
Saturday, April 9, 16

4. Hello /@robtreat2
Who Cares What Some Suite Thinks?
Saturday, April 9, 16

5. Hello /@robtreat2
Phantom Pages
Saturday, April 9, 16

6. Memory Lane /@robtreat2
Benny
Saturday, April 9, 16

7. Memory Lane /@robtreat2
MyFirstPager
Saturday, April 9, 16

8. Memory Lane /@robtreat2
Multiple Rotations
Saturday, April 9, 16

9. Memory Lane /@robtreat2
always available, phone only
no pager for years
Saturday, April 9, 16

10. Hello /@robtreat2
Phantom Pages
Saturday, April 9, 16

11. Hello /@robtreat2
I manage the SRE team at OmniTI
we manage multiple sites
24x7
millions of users
(omniti.com/is/hiring)
Saturday, April 9, 16

12. Why God Why?
paging is useful
“broken systems should not be
just another day at the office”
-- me
Saturday, April 9, 16

13. Why God Why?
paging is useful
Who has ever gotten an alert and ignored it?
(/me looks at alert, says “oh, it’ll probably recover, no need to look further”)
Saturday, April 9, 16

14. Why God Why?
paging is useful
How many alerts were received in the past
week that were not actionable?
(no human action was required)
Saturday, April 9, 16

15. Why God Why?
paging CAN BE useful
Saturday, April 9, 16

16. Can We Fix It?
how to improve?
Saturday, April 9, 16

17. Can We Fix It?
hello@omniti.com
we offer operationally focused services to
help build and manage your infrastructure
:-)
Saturday, April 9, 16

18. Terms
• Metrics
• (anything which can be measured)
Saturday, April 9, 16

19. Terms
• Metrics
• (anything which can be measured)
• Graphs
• (trending systems)
Saturday, April 9, 16

20. Terms
• Metrics
• (anything which can be measured)
• Graphs
• (trending systems)
• Notices
• (notification of event; email)
Saturday, April 9, 16

21. Terms
• Metrics
• (anything which can be measured)
• Graphs
• (trending systems)
• Notices
• (notification of event; email)
• ALERTS
• (wake’n you up; pages)
Saturday, April 9, 16

22. Terms
• Metrics
• (anything which can be measured)
• Graphs
• (trending systems)
• Notices
• (notification of event; email)
• ALERTS
• (wake’n you up; pages)
Saturday, April 9, 16

23. Onward and Upward
If you want to improve
your alerts
use systems thinking to reason about your
“system”
Saturday, April 9, 16

24. Onward and Upward
alerts should be seen as evidence
that your system is behaving in a way
outside of your existing understanding
Saturday, April 9, 16

25. Onward and Upward
If you want to improve
your alerts
think in terms your business can get on
board with
Saturday, April 9, 16

26. Onward and Upward
for every alert you receive
What is the business impact of this alert?
Saturday, April 9, 16

27. Onward and Upward
for every alert you receive
What is the remediation for this alert?
Saturday, April 9, 16

28. Onward and Upward
remediation:
• Summarize the problem
• What was done to solve the problem?
• Who was notified?
• Can this be prevented?
Saturday, April 9, 16

29. Onward and Upward
send the answer to these questions
to everyone on the team
every time
Saturday, April 9, 16

30. Onward and Upward
link to this documentation
from your alerting system
Saturday, April 9, 16

31. Onward and Upward
• Knowledge Transfer
• Gaps Exposed
• Patterns will emerge
Saturday, April 9, 16

32. Onward and Upward
you might be a bad alert
• cannot determine business impact
• no remediation necessary
• no one needs to be told
• work arounds are available
Saturday, April 9, 16

33. Onward and Upward
if you can’t fix it, you don’t
need to wake up for it
Saturday, April 9, 16

34. Onward and Upward
if it can wait until morning,
you don’t need to wake up
for it
Saturday, April 9, 16

35. Onward and Upward
in case of bad alert
• remove the alert
Saturday, April 9, 16

36. Onward and Upward
in case of bad alert
• remove the alert
• convert the alert to a notice
Saturday, April 9, 16

37. Onward and Upward
in case of bad alert
• remove the alert
• convert the alert to a notice
• implement fixes
Saturday, April 9, 16

38. Onward and Upward
pro tip:
never let anyone add an alert
unless they can answer these
questions first
Saturday, April 9, 16

39. Can We Really Do This?
this is partially an organizational issue
Saturday, April 9, 16

40. Can We Really Do This?
thought exercise:
if you launched a new web site today,
you really only need one alarm
Saturday, April 9, 16

41. Can We Really Do This?
“I don’t care if my servers are on fire,
as long as I am still making money”
-- Kevin, actual OmniTI customer
Saturday, April 9, 16

42. This sounds good but...
Most SA/SRE types want to be
pro-active, not re-active.
ie. they want to alert on leading
indicators, not on problems
Saturday, April 9, 16

43. This sounds good but...
Carrie: I-I'm just making sure we don't get hit again.
Saul: Well, I'm glad someone's looking out for us, Carrie.
Carrie: I'm serious. I-I missed something once before, I
won't... I can't let that happen again.
Saul: It was ten years ago. Everyone missed something
that day.
Carrie: Yeah, everyone's not me.
Saturday, April 9, 16

44. Based On A True Story
site down: monitor was checking 200
response code.
failed to notice absence of response
code.
easily fixed, but reactive
Saturday, April 9, 16

45. Based On A True Story
“root cause” ==> OOM
why don’t we alert on OOM?
OOM does not consistently cause outages
Saturday, April 9, 16

46. Based On A True Story
too many false positives leads to
ignoring alarms
Saturday, April 9, 16

47. Digression
Friendman, Naparstek,Taussing-Rubbo,
Alarmingly Useless,The Case For Banning Car Alarms In NYC
http://transalt.org/files/news/reports/caralarms/report.pdf
Blackstone, Buck, Hakim
Evaluation of alternative policies to combat false emergency calls
http://isc.temple.edu/economics/wkpapers/Pubs/FalsePolicy.pdf
Wickens, Rice, Keller, Hutchins, Hughes, Clayton
False Alerts in Air Traffic Control Conflict Alerting System: Is There A Cry Wolf Effect?
http://www.tc.faa.gov/LOGISTICS/grants/pdf/2007/07-G-002.pdf
Görges M, Markewitz BA,Westenskow DR
Improving Alarm Performance In The Medical Intensive Care Unit Using Delays and Clinical Context
http://www.ncbi.nlm.nih.gov/pubmed/19372334
“In an intensive care unit, alarms are used to call attention to a patient, to
alert a change in the patient's physiology, or to warn of a failure in a medical
device; however, up to 94% of the alarms are false.”
Saturday, April 9, 16

48. Digression
AESOP
The Boy Who Cried Wolf
Saturday, April 9, 16

49. Based On A True Story
• send notice of OOM?
• fix the cause of OOM?
• make a useful alert?
Saturday, April 9, 16

50. Based On A True Story
useful alerting
• script that checks for OOM
• restart app server when found
• find offending process; kill it
• spin up new node; kill old node
in the event all of these fail, send an alert?
Saturday, April 9, 16

51. Based On A True Story
thought exercise:
if you launched a new web site today,
you really only need one alert
Saturday, April 9, 16

52. In Conclusion
if we need software that runs 24x7, we should
design resiliency into our software,
not human intervention
Saturday, April 9, 16

53. In Conclusion
thinking doesn’t scale
especially at 2AM
Saturday, April 9, 16

54. In Conclusion
thanks!
more:
Surge 2016
http://surge.omniti.com
@robtreat2
@omniti
Saturday, April 9, 16

55. Saturday, April 9, 16