Upcoming SlideShare
-
Full NameComment goes here.Delete Reply Block
-
Nancy2222Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @ https://www.ThesisScientist.comReply -
Abiba Faisal yes , Attended Anhanguera EducacionalhelloReply
-
sheikhaghunaimMy dear, How are you today? i will like to be your friend My name is Sheikha Ghunaim , am a 43 years old divorcee. Please write to me in my email ( sheikhaghunaim2@hotmail.com ). im honest and open mind single woman. im going to tell more when i see your response. Regards Sheikha.Reply
-
taniatonni2Hi All, We are planning to start Hadoop online training batch on this week... If any one interested to attend the demo please register in our website... For this batch we are also provide everyday recorded sessions with Materials. For more information feel free to contact us : siva@keylabstraining.com. For Course Content and Recorded Demo Click Here : http://www.keylabstraining.com/hadoop-online-training-hyderabad-bangaloreReply
Advanced PostMortem Fu and Human Error 101 (Velocity 2011)
- AdvancedPostMortem Fu& Human Error 101 John Allspaw VP, Tech Ops Velocity 2011
- We Want YOU etsy.com/careers
- ScienceTimeTravelMythbustingReadingHomework
- Here To Challenge You
- Resilience EngineeringDr. Erik Hollnagel Dr. David Woods Dr. Sidney Dekker Dr. Richard Cook
- Complex, Dynamic
- Fundamental Surprises
- E.T.T.O. Efficiency Thoroughness
- Organizations, Policies, Procedures, Regulations BLUNT Resources & Constraints
- Organizations, Policies, Procedures, Regulations BLUNT Resources & Constraints Slips Adjustments OperatorsCompensations Mistakes Lapses Recoveries Violations Improvisations SHARP
- FUTURE +PAST
- Why Do Them?
- Why?Understand the failure
- Why?Understand the system
- Where “System” = Networks Servers Applications Processes People
- Where “System” = Networks Servers Applications Processes People
- People
- (Anticipation) Knowing What To Expect
- (Anticipation) Knowing Knowing What What To Expect To Look For (Monitoring)
- (Anticipation) (Response) Knowing Knowing Knowing What What What To Expect To Look For To Do (Monitoring)
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- Microphones are ON?
- Event AwarenessCode Deploys
- TIMELINE IRC logging = Rich Data
- TIMELINE Traces of Data IRC logging = Rich Data
- IRC Logs Fed Into Solr
- Status BlogTwitter Feed
- Flight Data Recorder
- Annotation Traces
- Investigation Basics
- Start?
- TTDHow?
- TTR
- Stable(“all clear”)
- Impact Time = TTR - Start
- SEVERITY
- Severity 1A. Total loss of serviceB. Severe degradation, effectively unusableC. Loss of a critical feature
- Severity 2A. Major degradation/feature loss for SUBSET of membersB. Minor degradation/feature loss for ALL members
- Severity 3Noticeable non-critical feature loss or degradation
- Severity 4No visible impact, loss of redundancy or capacityheadroom
- Severity 5No-impact but unexpected failure
- 5/11/2011 - Payments/Checkout system issue Start 4:10pm TTD 4:15pm TTR 4:30pm Stable 4:35pm Total Impact 20 min Severity 1
- • Basic metrics• Timeline with details• Remediations/Observations
- “normal” Incident PostMortemoperation Time
- How?Why?Prevention?
- How
- Crisis PatternsProblem Starts PostMortem Time
- Crisis PatternsProblem Starts Detection PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Crisis PatternsProblem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Crisis PatternsProblem Starts PostMortem Time
- Crisis PatternsProblem Starts Detection PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Crisis PatternsProblem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Crisis PatternsProblem Starts PostMortem Time
- Crisis PatternsProblem Starts Detection PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation Time
- Crisis PatternsProblem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Crisis PatternsProblem Starts Stress Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time
- Problem Starts Detection Evaluation Response Stable PostMortem Confirmation All Clear Time Internal and External Update Communications
- Crisis PatternsProblem Starts Detection Evaluation Response Stable Confirmation All Clear Time
- Crisis Patterns
- Crisis PatternsForced beyond learned roles
- Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to see
- Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to seeCognitively and perceptively noisy
- Crisis PatternsForced beyond learned rolesActions whose consequences are both important anddifficult to seeCognitively and perceptively noisyCoordinative load increases exponentially
- Thematic Vagabonding“butterfly minds” NOT STUCK ENOUGH
- Goal Fixation (encystment)TOO STUCK
- HeroismNon-communicating lone wolf-isms
- DistractionIrrelevant noise in comm channels
- IMPROVISATIONREQUIREMENT for troubleshootingcomplex systems
- IMPROVISATION
- IMPROVISATION
- Why?
- Root Cause Analysis“With the unknown, one is confronted with danger,discomfort, and care; the first instinct is to abolishthese painful states.First principle: any explanation is better thannone.” Friedrich Nietzsche Twilight of the Idols, or How to Philosophize with a Hammer
- Hindsight BiasKnowledge of the outcome influences theanalysis of the processMakes steps towards failure appearforeseeable and obvious
- After The Fact
- After The Fact
- Reality: Before and During
- Hindsight Bias“Should have known better”“All the signs were there, you just needed topay attention”
- Hindsight BiasBEFORE accident: The future seems implausibleAFTER accident: Obviously clear: “how could they not see what mistake they were about to make”
- Hindsight Bias“...people’s need to be rightis stronger than their abilityto be objective.” N. Crawford American Psychological Association
- Outcome BiasJudging a past decisionbased on its outcome.
- ID Fishbone/Ishikawa FMEA Five Whys Fault Tree CED CRT
- why? OUTAGE
- why?because this OUTAGE
- why? why? because this OUTAGE
- why? why?because because this this OUTAGE
- why? why? why? because because this this OUTAGE
- why? why? why?because because because this this this OUTAGE
- why? why? why?because because because this this this OUTAGE but: WHY?
- which causedSome Caused otherAction Some Things things OUTAGE to happen
- Sequence-Of-Events
- • Satisfyingly simple, easy to explain and document
- • Satisfyingly simple, easy to explain and document• Solves for a specific case• Ignorant of surrounding circumstances• Too focused on components• Validates Hindsight and Outcome bias
- NOT HELPFUL• Satisfyingly simple, easy to explain and document• Solves for a specific case• Ignorant of surrounding circumstances• Too focused on components• Validates Hindsight and Outcome bias
- Epidemiological (adapted from Reason, 1990)
- (adapted from Reason, 1990)
- Holes = Active/Latent Failures,Bad Things™ Waiting to Happen (adapted from Reason, 1990)
- Holes = Active/Latent Failures, Bad Things™ Waiting to HappenCheese = Safety Barriers, Layers of Defense (adapted from Reason, 1990)
- (adapted from Reason, 1990)
- Code Servers (adapted from Reason, 1990)
- Code ServersSchedule Training (adapted from Reason, 1990)
- Unmonitored Disk Space (latent condition)
- Capacity Unmonitored Disk Space Miscalculation (latent condition)(latent condition)
- Capacity Unmonitored Disk Space Miscalculation (latent condition)(latent condition) Unit Test In Transition
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Transition
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced Transition (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Violation of known coding Capacity standards Unmonitored Disk Space Miscalculation (latent condition) (latent condition)(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure) FAILURE!
- Capacity Miscalculation(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Capacity Miscalculation(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Capacity Miscalculation(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure)
- Capacity Miscalculation(latent condition) Unit Test In Bug Introduced External API call Transition (active failure) (active failure) NO FAILURE!
- • Better than dominoes, but still linear layers of “defense”• Helps uncover multiple contributors and latent failures (at sharp and blunt ends)
- • Better than dominoes, but still linear layers of “defense”• Helps uncover multiple contributors and latent failures (at sharp and blunt ends)• Doesn’t explain lineups or orientation of holes• Only identifies defects/gaps, nothing more• Still encourages judgements of decisions
- Better, but still NOT HELPFUL• Better than dominoes, but still linear layers of “defense”• Helps uncover multiple contributors and latent failures (at sharp and blunt ends)• Doesn’t explain lineups or orientation of holes• Only identifies defects/gaps, nothing more• Still encourages judgements of decisions
- Multiple Contributors “each necessary but only jointly sufficient”
- Resultant Versus Emergent
- Systemic DatabaseRouter Memcache Webserver
- Systemic DatabaseRouter Memcache Webserver Feature Roadmap
- Systemic DatabaseRouter Memcache Last Round Webserver of Funding Feature Roadmap
- Systemic Dashboard Design DatabaseRouter Memcache Last Round Webserver of Funding Feature Roadmap
- Systemic Dashboard Design Database Router Memcache Last Round Webserver of Funding Feature RoadmapDBA’s car
- Systemic Dashboard Design Database Router Memcache Last Round Webserver of Funding Feature Roadmap No EngDBA’s car Training
- Systemic Dashboard Design Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down No EngDBA’s car Training
- Systemic Dashboard Design Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down Entire Ops Team No Eng Is AtDBA’s car Training Velocity
- Systemic Dashboard Design Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down Entire Ops Team No Eng Techcrunch Is AtDBA’s car Training Article Velocity
- Systemic Hiring Dashboard Design Difficulties Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down Entire Ops Team No Eng Techcrunch Is AtDBA’s car Training Article Velocity
- Systemic Hiring S3 is slow Dashboard Design Difficulties Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down Entire Ops Team No Eng Techcrunch Is AtDBA’s car Training Article Velocity
- Systemic Hiring S3 is slow Dashboard Design Difficulties Database Router Memcache Last Round Webserver of Funding Feature Email is Roadmap down Entire Ops Team No Eng Techcrunch Is AtDBA’s car Training Article Velocity
- Systemic
- Systemic
- Systemic
- Systemic
- Systemic
- Systemic
- Systemic
- Functional ResonanceIn isolation, components act within bounds.Interconnected, they produce emergingbehaviors.
- Causes Are Constructed,Not Found WYLFIWYF Pre-conceived notions on “causes” and behaviors
- Contributors, not causes
- There is no root cause.
- LEARNING
- Quantifying ResponseTime to detect?Time for escalation, internal notification?Time to notify the public?Time to graceful degradation? (feature off)Time to stable/resolve?Time to all clear?
- Qualifying ResponseHigh signal:noise in comm channels?Troubleshooting fatigue?Troubleshooting handoff?All tools on-hand?Metrics visibility?Collaborative and skillful communication?Improvised tooling or solutions?
- All Together Now• Start/TTD/TTR/Stable/etc.• Severity• DATA (graphs, IRC, etc.)• Description (timeline, etc.)• Observations (motivations, latent conditions, etc.)• Actions (remediation tickets, followup)
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- Human Error“...knowledge and error flow from thesame mental sources, only successcan tell one from the other.” Ernest Mach, 1905
- Human ErrorNobody comes to work to do a bad job.
- Human ErrorUseless as a label and ending point.
- Human ErrorHuman error isn’t a cause, it’san effect.
- Why did it make senseto the personat the time?
- Error Categories• Slips• Lapses• Mismatches• Violations
- Error Categories
- First Stories“Human error” seen as root cause.Counterfactuals: saying what they “should”have done.Prevention: be “more careful!”
- Second Stories“Human error” seen as systemic vulnerabilities,deeper inside the organization.Digging into why it made sense for them to do what theydid, at the time they did it.Prevention...
- Why did it make senseto the personat the time?
- Why did it make senseto the personat the time?
- Why did it make senseto the personat the time?
- Substitution TestCould peers have made thesame error under the samecircumstances?
- WHERE to learn from ?
- Two Propositions
- 100 deploys6 deploy-related issues
- 100 > 6
- Proposition #1“Ways in which things go right are special casesof the ways in which things go wrong.”
- Proposition #1Successes = failures gone wrongStudy the failures, generalize from that. data sources: 6 out of 100
- Proposition #2“Ways in which things go wrong are specialcases of the ways in which things go right.”
- Proposition #2Failures = successes gone wrongStudy the successes, generalize from thatdata sources: 94 out of 100
- 94/100 ? OR6/100 ?
- What and WHY Do ThingsGo RIGHT?
- Not just: why did we fail?But also: why did we succeed?
- Near MissesHey everybody -Don’t be like me. I tried to X, butbecause it was no good.It almost exploded everyone.So, don’t do: (details about X) Love, Joe
- Taking the New ViewRecognize that human error isan attribution.
- Taking the New ViewPursue Second Stories.
- Taking the New ViewEscape Hindsight Bias.
- Taking the New ViewUnderstand work as performedat the sharp end.
- Taking the New ViewExamine how changes (at alllayers) will produce newvulnerabilities.
- Taking the New ViewUse technology to support andenhance human expertise.
- Taking the New ViewTame complexity through newforms of feedback.
- Taking the New ViewRealize that your systems arenot inherently safe.
- Taking the New ViewHuman error is an inevitableby-product of strainedcomplex systems.
- Taking the New ViewHuman error isn’t at the root ofyour safety problems.
- Taking the New ViewHuman error isn’t random.
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- Just Culture
- Just CultureBalancing accountability withlearning.
- Intentional Malice
- Negligence Found Severity of the Outage
- NameBlameShame
- NameBlameShame
- Name WHY? Blame#!@%$&#Shame
- “Must set an example!”
- “Has to be some fear that notdoing one’s job correctly couldlead to punishment.”
- “Must set an example!” Punishing Deterrents is a not“Has to be some fear that Losing coulddoing one’s job correctly Propositionlead to punishment.”
- Holding People Accountable != Blaming People
- Accountability and Learning Punishment For Errors
- No Bad ApplesOnly Bad Theories of Error
- Name WHY? Blame#!@%$&#Shame
- Signs Of Old View“Gross Misconduct”“Carelessness”“Negligence”“Egregious Behavior”“Willful Violations”
- Discretionary Spaces
- AcceptableUnacceptable
- AcceptableUnacceptable
- AcceptableUnacceptable
- AcceptableUnacceptable
- Acceptable (who draws this subjective line?)Unacceptable
- Increase Accountability BySupporting Learning
- Empower PeopleLet them own their own stories.Don’t make people pay penalties.Allow them to educate the organization.
- Reduce UncertaintyMake it clear who definesacceptable behavior.
- Organizational RootsAccountability =Responsibility + Requisite Authority
- (Anticipation) (Response) Knowing Knowing Knowing Knowing What What What What To Expect To Look For To Do Has Happened (Monitoring) (Learning)
- (Thanks, Fellas)Dr. Erik Hollnagel Dr. David Woods Dr. Sidney Dekker Dr. Richard Cook
- Homework!
- We Want YOU etsy.com/careers
- • Photo Credits http://www.flickr.com/photos/51035644987@N01/2678090600/• http://www.flickr.com/photos/67196253@N00/2941655917/• http://www.flickr.com/photos/stirwise/417629641• http://www.flickr.com/photos/38383999@N06/3888057995/• http://www.flickr.com/photos/94443490@N00/361543080/• http://www.flickr.com/photos/7729940@N06/4333396494/• http://www.flickr.com/photos/cpstorm/167418602• http://www.flickr.com/photos/63474264@N00/4366221069/• http://www.flickr.com/photos/proimos/4199675334• http://www.flickr.com/photos/30475691@N07/2862060992/• http://www.flickr.com/photos/14663487@N00/797755046/• http://www.flickr.com/photos/25080113@N06/5361445631/
- THE END