Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls

Escalating
Scenarios
A Deep Dive Into Outage
Pitfalls

John Allspaw
Velocity
London 2012

Wednesday, October 3, 12

TROUBLESHOOTING

This is NOT about troubleshooting

Or, not just about troubleshooting


LAYOUT
• Criteria
• Situational Awareness
• HROs
• Decision Making
• Communication
• Team Coordination
• A little bit of psychology

How important is this?


Oct 2011

Sept 2012


Where to learn
from?


TMI


Kegworth 1989


Dr. Richard Cook, Velocity US 2012
http://www.youtube.com/watch?v=R_PDc0HFdP0

“The Self-Designing High-Reliability Organization:
Aircraft Carrier Flight Operations at Sea”
Rochlin, La Porte, and Roberts. Naval War College Review 1987

http://govleaders.org/reliability.htm


What Goes On In Our Heads?


Jens Rasmussen, 1983
Senior Member, IEEE

“Skills, Rules, and Knowledge; Signals, Signs,
and Symbols, and Other Distinctions in Human
Performance Models”
IEEE Transactions On Systems, Man, and Cybernetics, May 1983


SKILL - BASED

Simple, routine
RULE - BASED

Knowable, but unfamiliar
KNOWLEDGE - BASED

(Reason, 1990)
WTF IS GOING ON?

Situational Awareness
"the perception of elements in the environment within a volume of
time and space, the comprehension of their meaning, and the
projection of their status in the near future,” - (Endsley, 1995)

"keeping track of what is going on around you in a complex,
dynamic environment" (Moray, 2005, p. 4)

"knowing what is going on so you can ﬁgure out what to
do" (Adam, 1993)


OODA Loop
Observe Orient Decide Act

Metrics Analysis Planning Execution
Monitoring Visualization Resourcing
Alerting Correlation
Alarming
credit: http://blog.b3k.us/ooda.html

Canonical Work

“Towards a Theory of Situational Awareness”
Mica Endsley, Human Factors (1995)

http://www.satechnologies.com/Papers/pdf/Toward%20a%20Theory
%20of%20SA.pdf


Level I
Perception

Level II
Comprehension

Level III
Projection


System capability
Interface design
Stress and workload
Complexity
Automation
Task/System Factors

Feedback


Perception
Performance
State of the of elements
Comprehension Projection Decision of actions
environment in current
situation of current situation of future status
LEVEL I LEVEL II LEVEL III

Information processing
Individual Factors mechanisms
Long term
Goals and memory states Automaticity
objectives

Preconceptions
- Abilities
(expectations)
- Experience
- Training

(Endsley)

Level One: Perception


Context

Can you spot the anomaly?

Context

24 hours

Context

7 days

Context

Normal
But
Noisy

Level
Two

Comprehension

Level
Two


Mental Models

• Categorization & Comprehension
• Mental “map” or “schema”
• Informed by experience, stored in memory

Mental Models


Level Three


Level Three
Common Clues you’re losing SA at this level
• Ambiguity
• Fixation
• Confusion
• Lack of Information
• Failure to maintain
• Failure to meet expected checkpoint or target
• Failure to resolve discrepancies
• A bad gut feeling that things are not quite right


Characteristics of response to
escalating scenarios


...tend to neglect how processes
develop within time (awareness of
rates) versus assessing how things
are in the moment

“On the Difﬁculties People Have in Dealing With Complexity” Dietrich Doerner, 1980


...have difficulty in dealing with
exponential developments (hard to
imagine how fast something can
change, or accelerate)



...inclined to think in causal SERIES,
instead of causal NETS.
A therefore B,
instead of
A, therefore B and C (therefore D and
E), etc.


SA
Pitfalls

Requisite Memory Trap


SA
Pitfalls

Workload, anxiety,
fatigue, other stressors


SA
Pitfalls

Data Overload


SA
Pitfalls

Misplace Salience


http://www.perceptualedge.com/articles/Whitepapers/Dashboard_Design.pdf


SA
Pitfalls

Complexity Creep
“Everything should be as simple as it can be, but not simpler.”
- paraphrased, Einstein


SA
Pitfalls

Poor Mental Models


SA
Pitfalls

Out-Of-The-Loop
Syndrome


SA
Pitfalls

Refusal to make
decisions


Heroism
Non-communicating lone wolf-isms


Distraction
Irrelevant noise in comm channels


TEAMS
• Divide and conquer applied to problem space,
division of labor
• Incident resolution vs. Problem resolution
• Reproducibility
• Fault Tolerance Effects

TEAMS

Shotgun debugging


JOINT
ACTIVITY
• Interpredictability
• Common Ground
• Directability
http://csel.eng.ohio-state.edu/woods/distributed/CG%20ﬁnal.pdf

Interpredictability


Common Ground

Directability


Improvisation


IMPROVISATION


Improvisation

“...you can’t improvise on nothing; you got to
improvise on something.”
Charles Mingus

Diagnose
the problem

Represent
the problem

Detect the Generate a Apply
course of Leverage
Problem/Opportunity action Points

Evaluate


Communication
Recommendations

•Explicitness
•Assertiveness
•Timing


Assertiveness

• Passive
• Assertive
• Aggressive

Exercise


Communication
• IRC?
• Face-To-Face?
• Conference Call?
• Morse Code?

Transmission

Encode Decode
Meaning Meaning
Sender Receiver


Transmission

Encode Decode
Meaning Meaning
Sender Receiver

Meaning Meaning
Receiver Sender
Decode Encode

Transmission


Feedback
Informational


Feedback
Corrective


Feedback
Reinforcing


Decision Making
Naturalistic Decision Making (NDM)
Gary Klein


Decision Making

Step One: What is the problem?


Decision Making

Step Two: What shall I do?


Decision Making

Recognition-Primed Decisions


Decision Making

Rule-Based Decisions


Decision Making

Choice decisions


Decision Making
Creative decisions


Decision Making
Decreasing cognitive effort
Decreasing effects of stress

Creative Choice Rule-Based RPD

Increasing cognitive effort
Increasing effects of stress

Decision Making

PRE-Mortem


Tooling


?
?


Time
Period Metric


Controls


ALERTS
• Meant to boost SA
• Alarm overload
• High false alarm rates
• Routinely disable alerts

Alert Reliability

ALERT DESIGN

• Signal:Noise can be difficult
• Easy to err on more false alarms
• Decay in trust
• Origins: Undetectable conditions

ALERT DESIGN

Conﬁrmation


ALERT DESIGN

Expectancy


ALERT DESIGN


ALERT DESIGN

• Don’t make people singularly reliant on alarms
• Support alarm conﬁrmation activities
• Make alarms unambiguous
• Reduce, reduce, reduce false alerts
• Set missed/false alert trade-offs appropriately

ALERT DESIGN

• Use multiple modalities
• Minimize alarm disruptions to ongoing activities
• Support the assessment/diagnosis of multiple alerts
• Support global SA of systems in an alarm state

Mature Role of Automation

“Ironies of Automation” - Lisanne Bainbridge
http://www.bainbrdg.demon.co.uk/Papers/Ironies.html


Mature Role of Automation
• Moves humans from manual operator to supervisor
• Extends and augments human abilities, doesn’t replace it
• Doesn’t remove “human error”
• Are brittle
• Recognize that there is always discretionary space for humans
• Recognizes the Law of Stretched Systems


SUMMARY

So what can we do?

“In preparing for battle, I have always
found that plans are useless but planning
is indispensable.”
- Eisenhower


So what can we do?
We develop our Non-Technical Skills
• Situational Awareness
• Communication
• Decision Making
• Improvisation
• Crew Resource Management (CRM)

So what can we do?
We tailor our environment to adapt
• Tooling to support SA
• Learning from outages (PostMortem)
• Anticipating problems (PreMortem)
• Gather Meta-Metrics

THE END

Credits
• http://www.flickr.com/photos/28650594@N03/2718027136/

• http://www.flickr.com/photos/telstar/2816887784/

• http://www.flickr.com/photos/soldiersmediacenter/3855375117/



• https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/

• http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/


Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls

More Related Content

Viewers also liked

Similar to Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls

More from John Allspaw

Recently uploaded

Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls