This document discusses improving incident response procedures through practices such as checklists, documented procedures, realistic incident simulations and postmortems. It recommends extended use of checklists to guide responses while still allowing for experience and independent thought. Regular incident response simulations that test both general processes and specific failures can help refine procedures and build confidence. Postmortems should objectively review incidents, suggest improvements and run through scenarios again over time to prevent complacency.

1. Jorge Salamero Sanz <jsalamero@serverdensity.com>
CfgMgmtCamp 1 Feb 2016
War Games - Flight training for DevOps

2. How to Monitor MySQL

3. The Cost of Uptime
$ 3.55bn 2015 Q4
$ 1.21bn 2015 Q2
$ 4.1bn 2015 Q1

4. How much do you spend?

5. ● Infrastructure automation
● Configuration automation
● Continuous testing
● Continuous deployment / delivery
● Monitoring
● Logs, error handling
● Feedback
● Human Ops
DevOps lifecycle

6. ● Prepare
● Respond
● Postmortem
Expect downtime

8. ● Power failure to half of our servers
● Automated failover unavailable
(known failure condition)
● Manual DNS switch required
● Expected impact: 20 min
● Actual impact: 43min
Incident example

9. ● Unfamiliarity with the process
● Pressure of time sensitive event
(panic effect)
● Escalation introduces delays
The Human Factor

10. ● Extended use of checklists
● Not to follow blindly, use knowledge
and experience
● Independent system
● Searchable
● List of known issues and
documented workarounds/fixes
Documented procedures

11. ● Realistic incident simulation
● Practice general response process
● Practice specific incident response
● Deficiencies: practice and improve
the process
Practiced procedures

12. ● First responder, acknowledge alert
● Load incident response checklist
● Log into #ops-war-room in Slack
● Log incident into JIRA
● Begin investigation
General response process

13. ● The “limits of human memory and
attention”
○ Complexity
○ Stress and fatigue
○ Ego
● Pilots, doctors, divers:
Bruce Willis Ruins All Films
(BCD, weights, releases, air, final)
Pre-flight checklists

15. ● Increase confidence
● Reduce panic
● Better coordination
● Trust relationships
● Improves time to resolution
Humans

16. ● Replica environment
● or mock command line
● Record actions and timing
● Multiple failures
● Unexpected results
Realistic scenarios

17. ● Team and individual test of response
● Run real commands
● Training the people
● Training the procedures
● Training the tools
Simulation goals

18. ● Objective review
● Suggestions for improvements
● Do it again
● Scenario evolves
● People forget
loop(): review and repeat

19. ● Failure sucks
● Fearless, blameless
● Significant learning
● Restores confidence
● Increases credibility
Postmortem

20. ● Short regular updates
● Even “we’re still looking into it”
● ~1 week to publish full version
○ follow-up incidents
○ check with 3rd party providers
○ timeline for required changes
Postmortem Timing

21. ● Root cause
● Turn of event led to failure
● Steps to identify & isolate the cause
● Services affected
● How we fixed it
● What we have learned and changed
Postmortem Content

22. Jorge Salamero Sanz
Chief Developer Advocate
@bencerillo
@serverdensity
our DevOps stories, no product spam
blog.serverdensity.com