A talk I gave for #sre-office-hours to our Eng team about my proposed way we should run Paging/Alerting/Chaos Eng!

1. Paging/Alerting
Workshop
#sre-office-hours | 04.04.2018

2. What qualifies as an
incident?

3. What should happen in
an incident?

4. Who should be
alerted/paged?

5. What tools would we
use to gather data?

6. How reliable/accurate
would this information be?

7. What’s the acceptable time
to be alerted/respond?

8. An Initial Framework for
Discussing SEVs

9. What is a SEV?
SEV is a term used to refer to an incident, it is derived from the
word severity.

10. Common types of SEV
- Availability Drop
- Product Issue / Feature Broken
- Data Loss
- Security Risk
- etc.

11. SEV Levels

12. Any SEV which involves a
loss of customer data should
be classified as SEV0.

13. Calculate Critical
Uptime in 9’s
https://uptime.is/

14. What uptime do you
think we could safely
publish?
Would we be okay
telling our
customers/vendors that
number?

15. SEV Terminology

16. Lifecycle of a SEV event

17. How to measure a SEV?
% loss * outage duration

18. Visualizing/tracking SEV’s

19. How do we maintain
combat effectiveness
during a SEV?

20. Incident Manager On-Call (IMOC)
- Should be a small rotation of Engineering Leaders
- Only one person is on-call in this role at any point in time
- These people should possess a wide knowledge of services and
engineering teams
- Will be our version of Air-Traffic-Control for the SEV, ensuring
different people working on the SEV are organized and working
coherently as a unit!

21. Tech Lead On-Call (TLOC)
- This would be the engineer driving resolution of the SEV
- Should have deep knowledge of specific domain of knowledge;
be a SME (Subject Matter Expert)
- Should have a deep knowledge of upstream and downstream
dependencies

22. What we need to define to have these roles:
- IMOC runbook/guide
- Designate a Primary and Secondary IMOC at all times
- Escalation should be automatic
- Monthly sync for all IMOC and TLOC
- Way to quickly triage what systems are effected/find root cause
- How would we do this?
- How do we record / document SEV’s?
- Google Form? Git repo? Suggestions??
- SEV naming convention

23. What happens when we
don’t meet our uptime
requirements?!?

24. What causes SEV’s?

25. Technical Issues
● Dependency Failure
● Cloud Provider Region/Zone Failure
● Provider Failure
● Connectivity Issues
● Power issues (our local office power affects AWS RDS!)
● DNS outage/latency
● Misconfiguration of machines/docker images
● Software Bugs
● Corrupt/unavailable backups

26. Cultural Issues
● Lack of knowledge sharing
● Lack of knowledge handover
● Lack of on-call training
● Lack of chaos engineering
● Lack of a high severity incident management program
● Lack of documentation and playbooks
● Lack of alerts and pages
● Lack of effective alerting thresholds
● Lack of backup strategy

27. How do we prevent SEVs from repeating?
● Combination of:
○ Record outages
○ Correlate failures
○ Track SEVs

28. Chaos Engineering!

29. What if we could break
things safely!?
What lessons/data could
we gather?

30. Chaos Engineering...yes, it is a real thing!
● 2010 - Netflix created the Chaos Monkey which can wreak
havoc in AWS at will deleting instances (but fully
customizable/controllable) -- this is OSS as of 2012
● 2011 - Netflix creates the Simian Army--a host of chaos tools to
test failure modes in your infrastructure and applications
● 2014 - the Role of Chaos Engineer is created at Netflix

31. Principles of Chaos Engineering

32. Can we do this?

34. And we all get to “share
the pain” with our new
tool PagerDuty...

35. Credits
- https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-inciden
t-management-program/
- https://www.gremlin.com/community/tutorials/chaos-engineering-the-history-principles
-and-practice/
- https://www.gremlin.com/the-discipline-of-chaos-engineering/
- https://github.com/tammybutow/chaos_engineering_bootcamp
- https://www.usenix.org/conference/srecon17americas/program/presentation/andrus
They did a cool workshop about Chaos Engineering
with hands-on labs at SREcon this year. If you like this
notion of chaos, more of us should go next year!
#notjustforSRE

36. Questions/thoughts?
#sre-office-hours