HumanOps is a set of principles which focus on the human aspects of running infrastructure. Talk in IncontroDevOps 2016.

1. Jorge Salamero Sanz <jsalamero@serverdensity.com>
IncontroDevOps 1 April 2016
War Games - Flight training for DevOps

2. How to Monitor MySQL

3. ● Infrastructure automation
● Configuration automation
● Continuous testing
● Continuous deployment / delivery
● Monitoring
● Logs, error handling
● Feedback
● Human Ops
DevOps lifecycle

4. ● Humans are part of any system
● Initial design, ongoing improvements
● Maintenance
● Upgrades
● Issues, Incident response
Humans in DevOps

5. ● System issues = error rates + SLA + ...
● Human issues = alerts out of hours + interruptions + .
● System issues = Human issues
Human issues = system issues

6. ● System health impacts human health
● Human health impacts system health
Humans impact systems

7. ● Downtime = loss of users, reputation, revenue
● Downtime caused by unreliable systems
● Unhealthy teams reduce reliability
● Unhealthy teams = loss of users, reputation, revenue
Humans impact business

8. ● Slip
● Lapse
● Mistake
● Violation
● (Always, again, again)
Human risk

9. ● Prepare and practice
● Respond
● Postmortem
Expect downtime

10. Real example
(small war story, won’t be long)

11. ● Power failure to half of our servers
● Automated failover unavailable
(known failure condition)
● Manual DNS switch required
● Expected impact: 20 min
● Actual impact: 43min
Incident example

13. Lesson learned?

14. ● Unfamiliarity with the process
● Pressure of time sensitive event
(panic effect)
● Escalation introduces delays
The Human Factor

15. Handling the Human factor

16. ● First responder, acknowledge alert
● Load incident response checklist
● Log into #ops-war-room in Slack
● Log incident into JIRA
● Begin investigation
General response process

17. 1. Extended use of checklists
Documented procedures

18. ● The “limits of human memory and
attention”
○ Complexity
○ Stress and fatigue
○ Ego
● Pilots, doctors, divers:
Bruce Willis Ruins All Films
(BCD, weights, releases, air, final)
Pre-flight checklists

19. 1. Extended use of checklists
2. Not to follow blindly, use knowledge
and experience
3. Independent system
4. Searchable
5. List of known issues and
documented workarounds/fixes
Documented procedures

20. ● Replica environment
● or mock command line
● Record actions and timing
● Multiple failures
● Unexpected results
Realistic scenarios: War Games

21. Results

22. ● Team and individual test of response
● Run real commands
● Training the people
● Training the procedures
● Training the tools
Results

23. ● Increase confidence
● Reduce panic
● Better coordination
● Trust relationships
● Improves time to resolution
Humans results

24. ● Review
● Suggestions for improvements
● Do it again
● Scenario evolves
● People forget
loop(): review and repeat

25. ● On call rotation design
● Alert prioritization
● Notification optimization
What else?

26. Human Ops

27. 1. Humans are part of the system
2. Humans impact systems
3. Humans impact business
4. Human issues count as system issues
Human Ops principles

28. meetup.com/humanops-london/
Human Ops Meetup

29. www.CloudStatusApp.com

30. Jorge Salamero Sanz
Chief Developer Advocate
@bencerillo
@serverdensity
our DevOps stories
blog.serverdensity.com