Responding to Outages Maturely

Responding to Outages Maturely John Allspaw SVP, Tech Ops Code As Craft, BerlinTuesday, April 24, 12

OPERABILITYTuesday, April 24, 12

PRODUCTIONTuesday, April 24, 12

http://WhoOwnsMyAvailability.comTuesday, April 24, 12

Tuesday, April 24, 12

How important is this?Tuesday, April 24, 12

How Can This Happen?Tuesday, April 24, 12

Complicated? Complex?Tuesday, April 24, 12

Complex Systems • Cascading Failures • Diﬃcult to determine boundaries • Complex systems may be open • Complex systems may have a memory • Complex systems may be nested • Dynamic network of multiplicity • May produce emergent phenomena • Relationships are non-linear • Relationships contain feedback loopsTuesday, April 24, 12

How Can This Happen? It does happen. And it will again.Tuesday, April 24, 12 And again.

Optimization MTBF MTTRTuesday, April 24, 12

http://www.ﬂickr.com/photos/sparktography/75499095/Tuesday, April 24, 12

How does team troubleshooting happen?Tuesday, April 24, 12

Problem Starts Detection Evaluation Response Stable PostMortem Conﬁrmation All Clear TimeTuesday, April 24, 12

Problem Starts Stress Detection Evaluation Response Stable PostMortem Conﬁrmation All Clear TimeTuesday, April 24, 12

Forced beyond learned roles Actions whose consequences are both important and diﬃcult to see Cognitively and perceptively noisy Coordinative load increases exponentiallyTuesday, April 24, 12

So What Can We Do?Tuesday, April 24, 12

We Learn From OthersTuesday, April 24, 12

Characteristics of response to escalating scenariosTuesday, April 24, 12

Characteristics of response to escalating scenarios ...tend to neglect how processes develop within time (awareness of rates) versus assessing how things are in the moment “On the Difﬁculties People Have in Dealing With Complexity” Dietrich Doerner, 1980Tuesday, April 24, 12

Characteristics of response to escalating scenarios ...have diﬃculty in dealing with exponential developments (hard to imagine how fast something can change, or accelerate) “On the Difﬁculties People Have in Dealing With Complexity” Dietrich Doerner, 1980Tuesday, April 24, 12

Characteristics of response to escalating scenarios ...inclined to think in causal series, instead of causal nets. A therefore B, instead of A, therefore B and C (therefore D and E), etc. “On the Difﬁculties People Have in Dealing With Complexity” Dietrich Doerner, 1980Tuesday, April 24, 12

PitfallsThematicVagabondingTuesday, April 24, 12

PitfallsGoal Fixation(encystment)Tuesday, April 24, 12

PitfallsRefusal to makedecisionsTuesday, April 24, 12

Heroism Non-communicating lone wolf-ismsTuesday, April 24, 12

Distraction Irrelevant noise in comm channelsTuesday, April 24, 12

Jens Rasmussen, 1983 Senior Member, IEEE “Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models” IEEE Transactions On Systems, Man, and Cybernetics, May 1983Tuesday, April 24, 12

SKILL - BASED Simple, routine RULE - BASED Knowable, but unfamiliar KNOWLEDGE - BASED (Reason, 1990) WTF IS GOING ON?Tuesday, April 24, 12

Team Troubleshooting • Which causes did you consider ﬁrst? • Which ones did you not consider at all? • How much of what you considered comes from recent history? • How much comes from observations from other team members?Tuesday, April 24, 12

Team Troubleshooting • How eﬀective is the response team in communicating to other groups? Users? • How long does it take to exhaust obvious cause(s)?Tuesday, April 24, 12

Team DynamicsTuesday, April 24, 12

High Reliability Organizations • Air Traﬃc Control • Complex Socio-Technical systems • Naval Air Operations At Sea • Eﬃciency <-> Thoroughness • Electrical Power Systems • Time/Resource Constrained • Etc. • Engineering-drivenTuesday, April 24, 12

“The Self-Designing High-Reliability Organization: Aircraft Carrier Flight Operations at Sea” Rochlin, La Porte, and Roberts. Naval War College Review 1987 http://govleaders.org/reliability.htmTuesday, April 24, 12

Close interdependence between groupsTuesday, April 24, 12

Close reciprocal coordination and information sharing, resulting in overlapping knowledgeTuesday, April 24, 12

High redundancy: multiple people observing the same event and sharing informationTuesday, April 24, 12

Broad deﬁnition of who belongs to the team.Tuesday, April 24, 12

Teammates are included in the communication loops rather than excluded.Tuesday, April 24, 12

Lots of error correction.Tuesday, April 24, 12

High levels of situation comprehension: maintain constant awareness of the possibility of accidents.Tuesday, April 24, 12

High levels of interpersonal skillsTuesday, April 24, 12

Maintenance of detailed records of past incidents that are closely examined with a view to learning from them.Tuesday, April 24, 12

Patterns of authority are changed to meet the demands of the events: organizational ﬂexibility.Tuesday, April 24, 12

The reporting of errors and faults is rewarded, not punished.Tuesday, April 24, 12

So What Else Can We Do?Tuesday, April 24, 12

We DrillTuesday, April 24, 12

We GameDayTuesday, April 24, 12

We Learn To ImproviseTuesday, April 24, 12

IMPROVISATIONTuesday, April 24, 12

We Learn From Our MistakesTuesday, April 24, 12

Postmortems • Full timelines: What happened, when, who involved • Review in public, everyone invited • Search for “second stories” instead of “human error” • Cultivating a blameless environment • Giving requisite authority to individuals to improve thingsTuesday, April 24, 12

Qualifying Response High signal:noise in comm channels? Troubleshooting fatigue? Troubleshooting handoﬀ? All tools on-hand and working? Improvised tooling or solutions? Metrics visibility? Collaborative and skillful communication?Tuesday, April 24, 12

RemediationTuesday, April 24, 12

We Share Near-Miss EventsTuesday, April 24, 12

Near Misses Hey everybody - Don’t be like me. I tried to X, but that wasn’t a good idea. It almost exploded everyone. So, don’t do: (details about X) Love, JoeTuesday, April 24, 12

Near Misses • Can act like “vaccines” - help system safety without actually hurting anything • Happen more often, so provide more data on latent failures • Powerful reminder of hazards, and slows down the process of forgetting to be afraidTuesday, April 24, 12

Practice! • How we troubleshoot in the moment, as a distributed team • How we handle time pressure • How we Observe/Orient/Decide/Act • How we communicate during emergencies • How we trust (or not) each other during emergencies • How we relate to emergencies when things are normal • How we could detect how we are protected during normal times (i.e., why aren’t we going down RIGHT NOW?)Tuesday, April 24, 12

Resilient Response • Can learn from other ﬁelds • Can train for outages • Can learn from mistakes • Can learn from successes as well as failuresTuesday, April 24, 12

http://www.ﬂickr.com/photos/sparktography/75499095/Tuesday, April 24, 12

THE ENDTuesday, April 24, 12

A parting word A parting challengeTuesday, April 24, 12

Two PropositionsTuesday, April 24, 12

100 changes 6 change-related issuesTuesday, April 24, 12

100 > 6Tuesday, April 24, 12

Proposition #1 “Ways in which things go right are special cases of the ways in which things go wrong.”Tuesday, April 24, 12

Proposition #1 Successes = failures gone wrong Study the failures, generalize from that. Potential data sources: 6 out of 100Tuesday, April 24, 12

Proposition #2 “Ways in which things go wrong are special cases of the ways in which things go right.”Tuesday, April 24, 12

Proposition #2 Failures = successes gone wrong Study the successes, generalize from thatTuesday, April 24, 12 Potential data sources: 94 out of 100

94/100 ? ORTuesday, April 24, 12 6/100 ?

What and WHY Do Things Go RIGHT?Tuesday, April 24, 12

Not just: why did we fail? But also: why did we succeed?Tuesday, April 24, 12

Mature Role of Automation “Ironies of Automation” - Lisanne Bainbridge http://www.bainbrdg.demon.co.uk/Papers/Ironies.htmlTuesday, April 24, 12

Mature Role of Automation • Moves humans from manual operator to supervisor • Extends and augments human abilities, doesn’t replace it • Doesn’t remove “human error” • Are brittle • Recognize that there is always discretionary space for humans • Recognizes the Law of Stretched SystemsTuesday, April 24, 12

Law of Stretched Systems “Every system is stretched to operate at its capacity; as soon as there is some improvement, for example, in the form of new technology, it will be exploited to achieve a new intensity and tempo of activity” D.Woods, E. Hollnagel, “Joint Cognitive Systems: Patterns” 2006Tuesday, April 24, 12

Responding to Outages Maturely

by John Allspaw

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Responding to Outages Maturely Presentation Transcript