The document discusses embracing failures in software and systems. It recommends monitoring for failures, conducting blameless postmortem analyses to determine root causes, simulating failures through "gameday" exercises before they occur, and designing systems using a "SafeMachine" approach to make failures safe rather than unsafe. The key ideas are to understand failures will happen, learn from them through root cause analysis and prevention practices, and design systems that can fail safely.

1. Embracing Failure
(not my life story)

4. Setting the Mood
•Understand that they WILL
happen
•Failures are not binary
•Impact determines importance
•deadlines for fixes are variable

5. Terminology
•Website
•Production
•Downtime

6. Monitor Failures

7. What is Monitoring?
•Graphs. Everywhere.
•Alerts on failures
•phone calls
•texts
•Answers: Are we failing?

9. healthcare.gov
•Know when you’re down
before CNN

11. Postmortems
(fool me once. shame on you.
fool me twice. shame on me.)

12. Postmortems
1. Reconstruct the factual
timeline
2. Root cause analysis
3. Remediation items

13. Postmortems
•Why did we fail?
•Blameless
•Moderated

14. Gamedays
(You wouldn’t wing a talk.
Don’t wing a hot fix)

15. Gameday
•Best defense is a good
offense
•Simulate possible failures
•Do it in production

16. kill -9
1. Draw a block
diagram
2. Cut every connection
3. Watch the fireworks

17. SafeMachine
(like a state machine … but safer)

18. Try, Try, Try again
•What if we could just retry
failures?
•Side effects are the root of all
evil
•Safe failures vs Unsafe failures

19. What’s in a SafeMachine
•Actions
•States
START
Computed
File
Uploaded
File
END
compute upload
record
successful

20. initialize_succeeded
initialize_failed
initialize_inprogress
computed_succeeded

21. START
a1
a1
a2
a2
a2
a3
a3
a3
END
The Pipeline

22. The Pipeline
START
Computed
File
Uploaded
File
END
Safe Unsafe Safe

23. Embracing Failure
•Monitor
•Postmortems
•Gamedays - you wouldn’t
wing a talk?
•SafeMachine

24. @chriswu_

25. Additional resources
• Postmortems https://codeascraft.com/2012/05/22/blameless-
postmortems/
• Gamedays - https://stripe.com/blog/game-day-exercises-at-stripe
• links at the bottom of this post are also great
• Error Tracking - https://getsentry.com/welcome/