The document discusses the importance of resilience engineering and operations, emphasizing preparedness for unexpected failures through practices like 'gameday' exercises. These exercises help improve an organization's ability to anticipate, mitigate, and recover from system failures while fostering confidence and competence among personnel. The text also highlights key definitions related to resilience, the impact of failures, and the need for experience in managing operations effectively.

1. Jesse Robbins
Cofounder, Opscode
@jesserobbins
jesse@opscode.com
1

2. Join Us!!!
2

3. “You don’t choose the moment,
the moment chooses you.
You only choose how prepared
you are when it does.”
-Fire Chief Mike Burtch
3

4. Operations is Work that Matters
4

5. GameDay
5

6. define:
GameDay
An exercise designed to increase
Resilience through large-scale fault
injection across critical systems.
Part of a larger discipline called
Resilience Engineering.
Not new, just new to us ;-)

7. define:
Resilience
Resilience is a the ability
of a System to adapt to
changes, failures, &
disturbances.

8. define:
System
People
Culture
Processes
Applications & Services
Infrastructure
Software
Hardware

9. This will be on the test:
Resilience is a product of
People & Culture
9

10. Copyright © 2010 Opscode, Inc - All Rights Reserved 10

11. This will be on the test:
FAILURE HAPPENS!

13. “multiple & unexpected interactions of
failures are inevitable”
-Charles Perrow

14. Catastrophic Potential
Simple Complexity Complex
Tight
KEEP
OUT!!!
Coupling
Loose
Created by Jesse Robbins
"Catastrophic Potential" adapted from Normal Accidents by Charles Perrow 14

15. define:
The Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds
99.99999% 3 Seconds

16. 99.9% *
99.9% *
99.9%
=
99.7% (oops!)
16

17. MTTR > MTBF
17

18. I ❤ MTTR
18

19. Copyright © 2010 Opscode, Inc - All Rights Reserved 19

20. GameDay
Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
http://www.flickr.com/photos/dnorman/2678090600 20

21. Useful Ops Personality Defects
25% Pyromaniac
75% Paranoid

22. setting a good example

23. GameDay increases Resilience in 3 ways
Preparation
‣ Identification and mitigation of risks and impact from
failure
‣ Reduces frequency of failure (MTBF)
‣ Reduces duration of recovery (MTTR)
Participation
‣ Builds confidence & competence responding to failure
and under stress.
‣ Strengthens individual and cultural ability to anticipate,
mitigate, respond to, and recover from failures of all
types.
Exercises
‣ Trigger and expose “latent defects”
‣ Choose discover them, instead of letting that be
determined by the next real disaster.
23

24. start small...
http://www.flickr.com/photos/oakleyoriginals/5674150237 24

25. increase awareness
http://www.flickr.com/photos/maunzy/5099921731 Copyright © 2010 Opscode, Inc - All Rights Reserved 25

26. build confidence
http://www.flickr.com/photos/skevbo/4864249944

27. full scale, live fire exercises
http://tacomafiredepartment.blogspot.com/2010/05/west-slope-training-burn.html Opscode, Inc - All Rights Reserved
Copyright © 2010 27

28. safety standards &
“building codes”
http://www.flickr.com/photos/peregrinari/3801964067 28

29. GameDay increases Resilience in 3 ways
Preparation
‣ Identification and mitigation of risks and impact from
failure
‣ Reduces frequency of failure (MTBF)
‣ Reduces duration of recovery (MTTR)
Participation
‣ Builds confidence & competence responding to failure
and under stress.
‣ Strengthens individual and cultural ability to anticipate,
mitigate, respond to, and recover from failures of all
types.
Exercises
‣ Trigger and expose “latent defects”
‣ Choose discover them, instead of letting that be
determined by the next real disaster.
29

30. no substitutes for experience...
Failure free operations require
experience with failure.
Ana Grillo © Ana Grillo Photography
30

31. The “OODA” Loop
Observe, Orient, Decide, Act
31

32. OODA: Observe, Orient, Decide, Act
http://en.wikipedia.org/wiki/OODA_loop
32

33. “You don’t choose the moment,
the moment chooses you.
You only choose how prepared
you are when it does.”
-Fire Chief Mike Burtch
33

34. Jesse Robbins
Cofounder, Opscode
@jesserobbins
jesse@opscode.com
34

35. Please See:
John Allspaw
‣ Resilience Engineering:
http://www.kitchensoap.com/2011/04/07/resilience-engineering-part-i/
‣ Advanced Post Mortem Fu:
http://www.slideshare.net/jallspaw/advanced-postmortem-fu-and-human-error-101-velocity-2011
Dr. Richard Cook
‣ How Complex Systems Fail
http://www.ctlab.org/documents/How%20Complex%20Systems%20Fail.pdf
35