David Mytton CEO of Server Density presented this talk to the DevOps Meetup in London. It takes you through how to handle DevOps incidents, outages and downtime -- and more specifically how to make friends, not enemies in the process.

1. How to win friends when
handling outages and downtime
David Mytton
London DevOps - Oct 2014
blog.serverdensity.com

2. David Mytton

3. Server monitoring, cloud management,
dashboards and alerting
serverdensity.com

4. Slides: twitter.com/davidmytton

5. Let’s talk about downtime

6. 2013 Spend: ~$5bn

7. 2013 Spend: ~$6bn

8. 2013 Spend: ~$4bn

9. How much do you spend?
You will have downtime

11. Preparation

12. Preparation - On Call
● Primary?

13. Preparation - On Call
● Primary?
● Secondary?

14. Preparation - On Call
● Primary?
● Secondary?
● Reachability - Tube, 3G/4G
(edge?!), Do Not Disturb mode, at
the gym, family emergency,
system updates

15. Preparation - On Call
● Off call

16. Preparation - On Call
● Off call
● Rotations

17. Preparation - On Call
● Off call
● Rotations
● Illness

18. Preparation - On Call
● Off call
● Rotations
● Illness
● Work the next day?

19. Preparation - Documentation

20. Preparation - Documentation
● Searchable

21. Preparation - Documentation
● Searchable
● Easy to edit

22. Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure

23. Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
● Up to date

25. Preparation - Key Info

26. Preparation - Key Info
● Team contacts

27. Preparation - Key Info
● Team contacts
● Key vendor contacts

28. Preparation - Key Info
● Team contacts
● Key vendor contacts
● Credentials to key systems

29. Unexpected failures

30. Unexpected failures
● Communication systems

31. Unexpected failures
● Communication systems
● Network connectivity

32. Unexpected failures
● Communication systems
● Network connectivity
● Access to support

33. ALERT!

34. ALERT!
1. Load up incident response checklist

35. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA

36. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room

37. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post

38. ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post
5. Initial investigation

39. Key response principles

40. Key response principles
● Log everything

41. Key response principles
● Log everything
● Frequent public status updates

42. Key response principles
● Log everything
● Frequent public status updates
● Gather the team

43. Key response principles
● Log everything
● Frequent public status updates
● Gather the team
● Escalate!

44. Postmortem

45. Postmortem
● Within a few days

46. Postmortem
● Within a few days
● Tell the story

47. Postmortem
● Within a few days
● Tell the story
● Provide technical detail

48. Postmortem
● Within a few days
● Tell the story
● Provide technical detail
● Explain what failed and why

49. Postmortem
● How it’s going to be fixed

50. stspg.io/ZDC

51. Summary
● Preparation
● Documentation
● Communication
● Checklists
● Postmortem

52. どもありがとうございます
＠davidmytton
david@serverdensity.com
blog.serverdensity.com
www.serverdensity.com