The signal to noise ratio can be frustratingly high for on-call alerts and downright mystifying when joining a new company or rotation. If you treat PagerDuty like the boy who cried wolf you’ve certainly mired in monitoring debt. Monitoring debit is tragically common and with investment mitigatable! Just how much of a risk are false positives? The data shows that a high number of false alerts trains workers to assume most alerts will be false. Repetition of the same alerts causes even greater alert fatigue. For clinicians the likelihood of acknowledging an alert dropped 30% for each reminder. Paige will share the nuts and bolts of how to start auditing your own alerts to begin your journey towards an on-call oasis.

1. Avoiding Alert
Bankruptcy and
Burnout
Paige Cruz aka @paigerduty

2. Agenda
Scenario I
01

3. Agenda
Scenario I
Alert
Fatigue
01
02

4. Agenda
Scenario I
Alert
Fatigue
Burnout
03
01
02

5. Agenda
Scenario I
Alert
Fatigue
Tactics
Burnout
03
01
02
04

6. Agenda
Scenario I
Alert
Fatigue
Tactics Scenario II
Burnout
03
01
02
04 05

7. Scenario I
01

8. BEEEEEEEP
BEEEEEEEEEEP
BEEEEEEEEEEEP
BEEEEEEEP
BEEEEEEEEEEP
BEEEEEEEEEEEP

9. Why is it
firing?
A. Batteries are running
low

10. Why is it
firing?
A. Batteries are running
low
B. Your roommate burned
toast. Again.

11. Why is it
firing?
A. Batteries are running
low
B. Your roommate burned
toast. Again.
C. You forgot to run the fan
during a steamy shower

12. Why is it
firing?
A. Batteries are running
low
B. Your roommate burned
toast. Again.
C. You forgot to run the fan
during a steamy shower
D. Beatrice, your beloved
cat knocked over a
candle

13. Needs investigation!

14. Alert Fatigue in
Practice
02

15. 🫨 Alert Fatigue 🚨
When the frequency or severity
of alerts causes responder to
ignore important alerts or make
more mistakes

16. Batteries ran low

17. Benny burned toast
(again)

18. Shower Steam

19. Beatrice knocked over
a candle

20. !Emergencies
Warning

21. !Emergencies
Warning
False
Positive

22. !Emergencies
Warning
False
Positive
E_TOO_MANY_
ALERTS

27. Burnout
03

28. 58%
Security and development
professionals report
currently experiencing
burnout

35. Tactics
04

36. AI/ML is the
answer!

37. NO

38. — Dr. David Woods
“The human is not a passive user
of a computer program but is an
active problem-solver in some
world.”

39. Facts

40. Facts
Feels

41. Facts
Feels
Findings

42. “Ugh. None of the last 5
pages I got were
actionable”
- Current primary on-call

43. Primary has been paged
5 times out of business
hours this week
- Pagerduty (the other one)

44. Team Alpaca is getting
frequently paged for
non-actionable alerts
- SRE

45. Gather the
data

46. Survey on-call
engineers

47. Audit your alerts.

48. Audit your alerts.
All
of
them.

49. I know things are
bad, but just
how bad are
they?

51. Leadership is
skeptical of
change

53. No one can
agree how to
craft a “good”
alert

55. Love this…do you
have a project
plan to follow?

57. Where can I
check my risk for
burnout?

59. “Err on the side of removing
noisy alerts – over-monitoring
is a harder problem to solve
than under-monitoring.”
Rob Ewaschuk

60. Scenario II
05

61. Ding!

62. Heads up, there’s
smoke in the
kitchen
Ding!

64. The Other Side
Mass Diameter Gravity
Mercury 0.06 0.38 0.38
Mars 0.11 0.53 0.38
Saturn 95.2 9.4 1.16

66. Upcoming events
10 23 7 26
May May June June
Prometheus
Fundamentals
Workshop SF
Open
Source
Summit
Infobip Shift
(Miami)
Monitorama

67. Resources
How to Pay Down
Monitoring Debt
Sysadvent: A Year In
Review
Increment: On-Call
Issue
On-Call Feels Survey
Yerbo Burnout Index
Quiz
Addressing Burnout Is Key to
Retaining SWE's
My Philosophy On Alerting
Alert Triage Hour of Power
Tackling Alert Fatigue - Catie
McCaffrey Monitorama
When to Alert on What?

68. Questions?