War Games - Flight training for DevOps @ TechSummit Amsterdam
•
1 like•641 views
How the human factor affects your infrastructure reliability and what we do to improve it: war games, on call schedules, etc.
Recommended
More Related Content
More from Server Density
More from Server Density (20)
Recently uploaded
Recently uploaded (20)
War Games - Flight training for DevOps @ TechSummit Amsterdam
- 1. Jorge Salamero Sanz <jsalamero@serverdensity.com> TechSummit Amsterdam 2 June 2016 War Games - Flight training for DevOps https://joind.in/talk/2e223
- 3. How to Monitor MySQL
- 4. ● Infrastructure automation ● Configuration automation ● Continuous testing ● Continuous deployment / delivery ● Monitoring ● Logs, error handling ● Feedback ● Human Ops DevOps lifecycle
- 5. ● Humans are part of any system ● Initial design, ongoing improvements ● Maintenance ● Upgrades ● Issues, Incident response Humans in DevOps
- 6. ● System issues = error rates + SLA + ... ● Human issues = alerts out of hours + interruptions + . ● System issues = Human issues Human issues = system issues
- 7. ● Downtime = loss of users, reputation, revenue ● Downtime caused by unreliable systems ● Unhealthy teams reduce reliability ● Unhealthy teams = loss of users, reputation, revenue Humans impact business
- 8. ● Slip ● Lapse ● Mistake ● Violation ● (Always, again, again) Human risk
- 9. What can we do?
- 10. ● Prepare and practice ● Respond ● Postmortem Expect downtime
- 11. Real example (small war story, won’t be long)
- 12. ● Power failure to half of our servers ● Automated failover unavailable (known failure condition) ● Manual DNS switch required ● Expected impact: 20 min ● Actual impact: 43min Incident example
- 14. Lessons learned?
- 15. ● Unfamiliarity with the process ● Pressure of time sensitive event (panic effect) ● Escalation introduces delays The Human Factor
- 16. Handling the Human factor
- 17. ● First responder, acknowledge alert ● Load incident response checklist ● Log into #ops-war-room in Slack ● Log incident into JIRA ● Begin investigation General response process
- 18. 1. Extended use of checklists Documented procedures
- 19. ● The “limits of human memory and attention” ○ Complexity ○ Stress and fatigue ○ Ego ● Pilots, doctors, divers: Bruce Willis Ruins All Films (BCD, weights, releases, air, final) Pre-flight checklists
- 20. 1. Extended use of checklists 2. Not to follow blindly, use knowledge and experience 3. Independent system 4. Searchable 5. List of known issues and documented workarounds/fixes Documented procedures
- 21. ● Realistic replica environment ● or mock command line ● Record actions and timing ● Multiple failures ● Unexpected results War Games
- 22. Results
- 23. ● Team and individual test of response ● Run real commands ● Training the people ● Training the procedures ● Training the tools Results
- 24. ● Increase confidence ● Reduce panic ● Better coordination ● Trust relationships ● Improves time to resolution Humans results
- 25. ● Review ● Suggestions for improvements ● Do it again ● Scenario evolves ● People forget loop(): review and repeat
- 26. What else?
- 27. ● Pressure of just waiting to be paged ● Trouble to sleep: 7.8 days year productivity cost ● Prevent burnout On call
- 28. ● Half self-interruptions ● Avg 23 minutes to resume task ● Only now actionable alerts Alerts notifications