Downloaded 17 times
Uploaded byServer Density
Flight training for DevOps & HumanOps - IncontroDevOps 2016
The document discusses the integration of human factors in DevOps through various methodologies such as monitoring, automation, and incident response. It emphasizes that system and human issues are interconnected, affecting downtime and overall business health. The importance of training, checklists, and iterative reviews to improve team responses during incidents is highlighted, along with principles of human operations.
Internetâ—¦
Flight training for DevOps & HumanOps - IncontroDevOps 2016
- 1. Jorge Salamero Sanz<jsalamero@serverdensity.com> IncontroDevOps 1 April 2016 War Games - Flight training for DevOps
- 2. How to MonitorMySQL
- 3. â—Ź Infrastructure automation â—ŹConfiguration automation â—Ź Continuous testing â—Ź Continuous deployment / delivery â—Ź Monitoring â—Ź Logs, error handling â—Ź Feedback â—Ź Human Ops DevOps lifecycle
- 4. â—Ź Humans arepart of any system â—Ź Initial design, ongoing improvements â—Ź Maintenance â—Ź Upgrades â—Ź Issues, Incident response Humans in DevOps
- 5. â—Ź System issues= error rates + SLA + ... â—Ź Human issues = alerts out of hours + interruptions + . â—Ź System issues = Human issues Human issues = system issues
- 6. â—Ź System healthimpacts human health â—Ź Human health impacts system health Humans impact systems
- 7. â—Ź Downtime =loss of users, reputation, revenue â—Ź Downtime caused by unreliable systems â—Ź Unhealthy teams reduce reliability â—Ź Unhealthy teams = loss of users, reputation, revenue Humans impact business
- 8. â—Ź Slip â—Ź Lapse â—ŹMistake â—Ź Violation â—Ź (Always, again, again) Human risk
- 9. â—Ź Prepare andpractice â—Ź Respond â—Ź Postmortem Expect downtime
- 10. Real example (small warstory, won’t be long)
- 11. â—Ź Power failureto half of our servers â—Ź Automated failover unavailable (known failure condition) â—Ź Manual DNS switch required â—Ź Expected impact: 20 min â—Ź Actual impact: 43min Incident example
- 13.
- 14. â—Ź Unfamiliarity withthe process â—Ź Pressure of time sensitive event (panic effect) â—Ź Escalation introduces delays The Human Factor
- 15. Handling the Humanfactor
- 16. â—Ź First responder,acknowledge alert â—Ź Load incident response checklist â—Ź Log into #ops-war-room in Slack â—Ź Log incident into JIRA â—Ź Begin investigation General response process
- 17. 1. Extended useof checklists Documented procedures
- 18. ● The “limitsof human memory and attention” ○ Complexity ○ Stress and fatigue ○ Ego ● Pilots, doctors, divers: Bruce Willis Ruins All Films (BCD, weights, releases, air, final) Pre-flight checklists
- 19. 1. Extended useof checklists 2. Not to follow blindly, use knowledge and experience 3. Independent system 4. Searchable 5. List of known issues and documented workarounds/fixes Documented procedures
- 20. â—Ź Replica environment â—Źor mock command line â—Ź Record actions and timing â—Ź Multiple failures â—Ź Unexpected results Realistic scenarios: War Games
- 21.
- 22. â—Ź Team andindividual test of response â—Ź Run real commands â—Ź Training the people â—Ź Training the procedures â—Ź Training the tools Results
- 23. â—Ź Increase confidence â—ŹReduce panic â—Ź Better coordination â—Ź Trust relationships â—Ź Improves time to resolution Humans results
- 24. â—Ź Review â—Ź Suggestionsfor improvements â—Ź Do it again â—Ź Scenario evolves â—Ź People forget loop(): review and repeat
- 25. â—Ź On callrotation design â—Ź Alert prioritization â—Ź Notification optimization What else?
- 26.
- 27. 1. Humans arepart of the system 2. Humans impact systems 3. Humans impact business 4. Human issues count as system issues Human Ops principles
- 28.
- 29.
- 30. Jorge Salamero Sanz ChiefDeveloper Advocate @bencerillo @serverdensity our DevOps stories blog.serverdensity.com