Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
John Allspaw, profile picture
Uploaded byJohn Allspaw
36,490 views

Outages, PostMortems, and Human Error

This document discusses various approaches to analyzing system failures and outages, including root cause analysis. It notes that sequence-of-events analysis can oversimplify issues and ignore surrounding context. Epidemiological models that view failures as resulting from the alignment of multiple factors are preferable. Truly understanding failures requires examining how organizational, technical, and human factors interact as a complex system over time.

Technology◦Business◦
Outages, Post-Mortems, and Human Error 101 John Allspaw SVP, Tech Ops
Science TimeTravel Mythbusting Reading Homework
Here To Challenge You
Resilience Engineering Dr. Erik Hollnagel Dr. David Woods Dr. Sidney Dekker Dr. Richard Cook
COMPLEX + DYNAMIC
COMPLEX != COMPLICATED
Fundamental Surprises
E.T.T.O. Efficiency Thoroughness
Organizations, Policies, Procedures, Regulations BLUNT Resources & Constraints
Organizations, Policies, Procedures, Regulations BLUNT Resources & Constraints Slips Adjustments Operators Compensations Mistakes Lapses Recoveries Violations Improvisations SHARP
FUTURE + PAST
Why Do Them?
Why? Understand the FAILURE
Why? Understand the SYSTEM
Where “System” = Networks Servers Applications Processes People
Where “System” = Networks Servers Applications Processes People
PEOPLE
(Anticipation) Knowing What To Expect
(Anticipation) Knowing Knowing What What To Expect To Look For (Monitoring)
(Anticipation) (Response) Knowing Knowing Knowing What What What To Expect To Look For To Do (Monitoring)
(Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
(Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
Microphones are ON?
Event Awareness Code Deploys
Timeline RICH DATA
Traces of Data Timeline RICH DATA
Status Blog Twitter Feed
Flight Data Recorder
Annotation Traces
Investigation Basics
Start?
TTD How?
TTR
Stable (“all clear”)
Impact Time = TTR - Start
SEVERITY
SEV 1 A. Total loss of service B. Severe degradation, effectively unusable C. Loss of a critical feature
SEV 2 A. Major degradation/feature loss for SUBSET of members B. Minor degradation/feature loss for ALL members
SEV 3 Noticeable non-critical feature loss or degradation
SEV 4 No visible impact, loss of redundancy or capacity headroom
SEV 5 No-impact but unexpected failure
5/11/2011 - Payments/Checkout system issue Start 4:10pm TTD 4:15pm TTR 4:30pm Stable 4:35pm Total Impact 20 min Severity 1
Basic metrics Timeline with details Remediations/Observations
“normal” Incident PostMortem operation Time
How? Why? Prevention?
How
Crisis Patterns Problem Starts PostMortem Time
Crisis Patterns Problem Starts Detection PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Crisis Patterns Problem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Crisis Patterns Problem Starts PostMortem Time
Crisis Patterns Problem Starts Detection PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Crisis Patterns Problem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Crisis Patterns Problem Starts PostMortem Time
Crisis Patterns Problem Starts Detection PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
Crisis Patterns Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Crisis Patterns Problem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time Internal and External Update Communications
Crisis Patterns Problem Starts Detection Evaluation Response Stable Confirmation All Clear Time
Crisis Patterns
Crisis Patterns Forced beyond learned roles
Crisis Patterns Forced beyond learned roles Actions whose consequences are both important and difficult to see
Crisis Patterns Forced beyond learned roles Actions whose consequences are both important and difficult to see Cognitively and perceptively noisy
Crisis Patterns Forced beyond learned roles Actions whose consequences are both important and difficult to see Cognitively and perceptively noisy Coordinative load increases exponentially
Thematic Vagabonding “butterfly minds” NOT STUCK ENOUGH
Goal Fixation (encystment) TOO STUCK
Heroism Non-communicating lone wolf-isms
Distraction Irrelevant noise in comm channels
IMPROVISATION REQUIREMENT for troubleshooting complex systems
IMPROVISATION
IMPROVISATION
Why?
Root Cause Analysis “With the unknown, one is confronted with danger, discomfort, and care; the first instinct is to abolish these painful states. First principle: any explanation is better than none.” Friedrich Nietzsche Twilight of the Idols, or How to Philosophize with a Hammer
Hindsight Bias Knowledge of the outcome influences the analysis of the process Makes steps towards failure appear foreseeable and obvious
After The Fact
After The Fact
Reality: Before and During
Hindsight Bias “Should have known better” “All the signs were there, you just needed to pay attention”
Hindsight Bias BEFORE accident: The future seems implausible AFTER accident: Obviously clear: “how could they not see what mistake they were about to make?!”
Hindsight Bias “...people’s need to be right is stronger than their ability to be objective.” N. Crawford American Psychological Association