• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Escalating complexity: DevOps learnings from Air France 447
 

Escalating complexity: DevOps learnings from Air France 447

on

  • 7,872 views

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of ...

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems.
Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand.
While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated.
In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

Statistics

Views

Total Views
7,872
Views on SlideShare
3,843
Embed Views
4,029

Actions

Likes
5
Downloads
32
Comments
2

29 Embeds 4,029

http://holmwood.id.au 2331
http://fractio.nl 760
https://twitter.com 453
http://infra-talk.org 294
http://www.asymmetrics.net 46
http://supine.newsblur.com 40
http://localhost 21
http://lanyrd.com 15
http://feeds.feedburner.com 13
http://asymmetrics.net 13
http://newsblur.com 9
http://reader.aol.com 7
http://www.newsblur.com 5
http://feedproxy.google.com 3
http://dev.newsblur.com 2
https://sendtoinc.com 2
https://tasks.crowdflower.com 2
http://www.goread.io 2
https://www.google.ca 1
https://www.google.com.au 1
http://feedly.com 1
http://www.inoreader.com 1
https://abs.twimg.com 1
http://webcache.googleusercontent.com 1
http://cloud.feedly.com 1
http://eventifier.co 1
http://tweetedtimes.com 1
http://192.168.3.23 1
http://japhy.soup.io 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Escalating complexity: DevOps learnings from Air France 447 Escalating complexity: DevOps learnings from Air France 447 Presentation Transcript

    • Escalatingcomplexity:DevOps learnings from Air France 447
    • Lindsay Holmwood @auxesis
    • Engineering Manager @Bulletproof Networks
    • cucumber-nagios Visage Flapjack
    • • On 31 May 2009, Air France 447 departed from Rio deJaneiro-Galeão International Airport at 22:29 UTC. It wasscheduled to arrive at Paris-Charles de GaulleInternational Airport 11 hours later.
    • • On 31 May 2009, Air France 447 departed from Rio deJaneiro-Galeão International Airport at 22:29 UTC. It wasscheduled to arrive at Paris-Charles de GaulleInternational Airport 11 hours later.• 3 hours and 45 minutes later, it crashed into the AtlanticOcean, killing 216 passengers, and 12 aircrew.
    • • On 31 May 2009, Air France 447 departed from Rio deJaneiro-Galeão International Airport at 22:29 UTC. It wasscheduled to arrive at Paris-Charles de GaulleInternational Airport 11 hours later.• 3 hours and 45 minutes later, it crashed into the AtlanticOcean, killing 216 passengers, and 12 aircrew.• There were no survivors.
    • • This is AF447’s flight path:
    • • AF447 charted a course through a band of equatorial thunderstorms
    • • This is what happened in the last 15 minutes of the flight.
    • 02:03:44 (Bonin) The inter-tropical convergence... look, were in it, between Salpu and Tasil. And then, look, were right in it...02:05:55 (Robert) Yes, lets call them in the back, to let them know...02:05:59 (FA) Yes? Marilyn.02:06:04 (Bonin) Yes, Marilyn, its Pierre up front... Listen, in 2 minutes, were going to be getting into an area where things are going to be moving around a little bit more than now. Youll want to take care.02:06:13 (FA) Okay, we should sit down then?02:06:15 (Bonin) Well, I think thats not a bad idea. Give your friends a heads-up.02:06:18 (FA) Yeah, okay, Ill tell the others in the back. Thanks a lot.02:06:19 (Bonin) Ill call you back as soon as were out of it.02:06:20 (FA) Okay.
    • 02:03:44 (Bonin) The inter-tropical convergence... look, were in it, between Salpu and Tasil. And then, look, were right in it...02:05:55 (Robert) Yes, lets call them in the back, to let them know...02:05:59 (FA) Yes? Marilyn.02:06:04 (Bonin) Yes, Marilyn, its Pierre up front... Listen, in 2 minutes, were going to be getting into an area where things are going to be moving around a little bit more than now. Youll want to take care.02:06:13 (FA) Okay, we should sit down then?02:06:15 (Bonin) Well, I think thats not a bad idea. Give your friends a heads-up.02:06:18 (FA) Yeah, okay, Ill tell the others in the back. Thanks a lot.02:06:19 (Bonin) Ill call you back as soon as were out of it.02:06:20 (FA) Okay.
    • 02:06:50 (Bonin) Lets go for the anti-icing system. Its better than nothing.02:07:00 (Bonin) We seem to be at the end of the cloud layer, it might be okay.02:08:03 (Robert) You can possibly pull it a little to the left.02:08:05 (Bonin) You can possibly pull it a little to the left. Were agreed that were in manual, yeah?
    • Captain Marc Dubois hands control to Robert + Bonin,•and takes the second mandatory rest break.
    • 02:10:06 (Bonin) I have the controls.02:10:07 (Robert) Okay.02:10:07 (Robert) Whats this?02:10:15 (Bonin) Theres no good... theres no good speed indication.02:10:16 (Robert) Weve lost the, the, the speeds, then?02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed.02:10:28 (Bonin) Okay, okay, Im descending.02:10:30 (Robert) Stabilize.02:10:31 (Bonin) Yeah.02:10:31 (Robert) Descend... It says were going up... It says were going up, so descend.02:10:36 (Robert) Descend!
    • 02:10:06 (Bonin) I have the controls.02:10:07 (Robert) Okay.02:10:07 (Robert) Whats this?02:10:15 (Bonin) Theres no good... theres no good speed indication.02:10:16 (Robert) Weve lost the, the, the speeds, then?02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed.02:10:28 (Bonin) Okay, okay, Im descending.02:10:30 (Robert) Stabilize.02:10:31 (Bonin) Yeah.02:10:31 (Robert) Descend... It says were going up... It says were going up, so descend.02:10:36 (Robert) Descend!
    • 02:10:06 (Bonin) I have the controls.02:10:07 (Robert) Okay.02:10:07 (Robert) Whats this?02:10:15 (Bonin) Theres no good... theres no good speed indication.02:10:16 (Robert) Weve lost the, the, the speeds, then?02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed.02:10:28 (Bonin) Okay, okay, Im descending.02:10:30 (Robert) Stabilize.02:10:31 (Bonin) Yeah.02:10:31 (Robert) Descend... It says were going up... It says were going up, so descend.02:10:36 (Robert) Descend!
    • 02:10:06 (Bonin) I have the controls.02:10:07 (Robert) Okay.02:10:07 (Robert) Whats this?02:10:15 (Bonin) Theres no good... theres no good speed indication.02:10:16 (Robert) Weve lost the, the, the speeds, then?02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed.02:10:28 (Bonin) Okay, okay, Im descending.02:10:30 (Robert) Stabilize.02:10:31 (Bonin) Yeah.02:10:31 (Robert) Descend... It says were going up... It says were going up, so descend.02:10:36 (Robert) Descend!
    • 02:10:37 (Bonin) Here we go, were descending.02:10:38 (Robert) Gently!02:10:41(Bonin) Were... yeah, were in a climb.02:10:49 (Robert) Damn it, where is he?02:10:55 (Robert) Damn it!02:11:03 (Bonin) Im in TOGA, huh?02:11:06 (Robert) Damn it, is he coming or not?02:11:21 (Robert) We still have the engines! What the hell is happening? I dont understand whats happening.02:11:21 (Robert) But we’ve got the engines. Whats happening? Do you understand what’s happening or not?02:11:32 (Bonin) I dont have control of the plane any more now. I dont have control of the plane at all!
    • 02:10:37 (Bonin) Here we go, were descending.02:10:38 (Robert) Gently!02:10:41(Bonin) Were... yeah, were in a climb.02:10:49 (Robert) Damn it, where is he?02:10:55 (Robert) Damn it!02:11:03 (Bonin) Im in TOGA, huh?02:11:06 (Robert) Damn it, is he coming or not?02:11:21 (Robert) We still have the engines! What the hell is happening? I dont understand whats happening.02:11:21 (Robert) But we’ve got the engines. Whats happening? Do you understand what’s happening or not?02:11:32 (Bonin) I dont have control of the plane any more now. I dont have control of the plane at all!
    • 02:10:37 (Bonin) Here we go, were descending.02:10:38 (Robert) Gently!02:10:41(Bonin) Were... yeah, were in a climb.02:10:49 (Robert) Damn it, where is he?02:10:55 (Robert) Damn it!02:11:03 (Bonin) Im in TOGA, huh?02:11:06 (Robert) Damn it, is he coming or not?02:11:21 (Robert) We still have the engines! What the hell is happening? I dont understand whats happening.02:11:21 (Robert) But we’ve got the engines. Whats happening? Do you understand what’s happening or not?02:11:32 (Bonin) I dont have control of the plane any more now. I dont have control of the plane at all!
    • 02:10:37 (Bonin) Here we go, were descending.02:10:38 (Robert) Gently!02:10:41(Bonin) Were... yeah, were in a climb.02:10:49 (Robert) Damn it, where is he?02:10:55 (Robert) Damn it!02:11:03 (Bonin) Im in TOGA, huh?02:11:06 (Robert) Damn it, is he coming or not?02:11:21 (Robert) We still have the engines! What the hell is happening? I dont understand whats happening.02:11:21 (Robert) But we’ve got the engines. Whats happening? Do you understand what’s happening or not?02:11:32 (Bonin) I dont have control of the plane any more now. I dont have control of the plane at all!
    • 02:11:37 (Robert) Controls to the left!02:11:41 (Robert) ...what is that?02:11:41 (Bonin) I have the impression (we have) the speed02:11:42 (Captain) Er, what are you doing?02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening02:11:45 (Bonin) We’re losing control of the aeroplane there02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything.02:11:51 (Captain) So take that, take that02:11:55 (Robert) Take that, take that02:11:57 (Robert) Try to take that
    • 02:11:37 (Robert) Controls to the left!02:11:41 (Robert) ...what is that?02:11:41 (Bonin) I have the impression (we have) the speed02:11:42 (Captain) Er, what are you doing?02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening02:11:45 (Bonin) We’re losing control of the aeroplane there02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything.02:11:51 (Captain) So take that, take that02:11:55 (Robert) Take that, take that02:11:57 (Robert) Try to take that
    • 02:11:37 (Robert) Controls to the left!02:11:41 (Robert) ...what is that?02:11:41 (Bonin) I have the impression (we have) the speed02:11:42 (Captain) Er, what are you doing?02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening02:11:45 (Bonin) We’re losing control of the aeroplane there02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything.02:11:51 (Captain) So take that, take that02:11:55 (Robert) Take that, take that02:11:57 (Robert) Try to take that
    • 02:11:58 (Bonin) I have a problem - it’s that I don’t have vertical speed indication02:12:01 (Captain) Alright02:12:01 (Bonin) I have no more displays02:12:02 (Robert) We have no valid displays02:12:04 (Bonin) I have the impression that we have some crazy speed, no? What do you think?02:12:06 (Robert) No.02:12:07 (Bonin) No?02:12:07 (Robert) No, above all dont extend the02:12:07 (Bonin) Okay02:12:09 (Robert) Dont extend
    • 02:11:58 (Bonin) I have a problem - it’s that I don’t have vertical speed indication02:12:01 (Captain) Alright02:12:01 (Bonin) I have no more displays02:12:02 (Robert) We have no valid displays02:12:04 (Bonin) I have the impression that we have some crazy speed, no? What do you think?02:12:06 (Robert) No.02:12:07 (Bonin) No?02:12:07 (Robert) No, above all dont extend the02:12:07 (Bonin) Okay02:12:09 (Robert) Dont extend
    • 02:12:11 (Bonin) So we’re still going down02:12:12 (Robert) We’re pulling02:12:14 (Robert) What do you think about it? What do you think? What do we need to do?02:12:15 (Captain) There - I don’t know. There - it’s going down.02:12:19 (Bonin) There you are.02:12:20 (Bonin) That’s good we should be wings level, no it won’t02:12:23 (Captain) The wings to flat horizon, the standby horizon02:12:25 (Robert) The horizon!02:12:26 (Bonin) Okay02:12:26 (Robert) Speed?
    • 02:12:11 (Bonin) So we’re still going down02:12:12 (Robert) We’re pulling02:12:14 (Robert) What do you think about it? What do you think? What do we need to do?02:12:15 (Captain) There - I don’t know. There - it’s going down.02:12:19 (Bonin) There you are.02:12:20 (Bonin) That’s good we should be wings level, no it won’t02:12:23 (Captain) The wings to flat horizon, the standby horizon02:12:25 (Robert) The horizon!02:12:26 (Bonin) Okay02:12:26 (Robert) Speed?
    • 02:12:27 (Robert) Youre climbing02:12:28 (Robert) Youre going down down down02:12:28 (Captain) Going down02:12:30 (Bonin) Am I going down now?02:12:31 (Robert) Go down02:12:32 (Captain) No you climb there02:12:32 (Bonin) Im climbing okay so were going down02:12:34 (Captain) Youre climbing02:12:39 (Bonin) Okay, were in TOGA02:12:41 (Bonin) What are we here?02:12:41 (Bonin) On alti what do we have here?
    • 02:12:27 (Robert) Youre climbing02:12:28 (Robert) Youre going down down down02:12:28 (Captain) Going down02:12:30 (Bonin) Am I going down now?02:12:31 (Robert) Go down02:12:32 (Captain) No you climb there02:12:32 (Bonin) Im climbing okay so were going down02:12:34 (Captain) Youre climbing02:12:39 (Bonin) Okay, were in TOGA02:12:41 (Bonin) What are we here?02:12:41 (Bonin) On alti what do we have here?
    • 02:12:27 (Robert) Youre climbing02:12:28 (Robert) Youre going down down down02:12:28 (Captain) Going down02:12:30 (Bonin) Am I going down now?02:12:31 (Robert) Go down02:12:32 (Captain) No you climb there02:12:32 (Bonin) Im climbing okay so were going down02:12:34 (Captain) Youre climbing02:12:39 (Bonin) Okay, were in TOGA02:12:41 (Bonin) What are we here?02:12:41 (Bonin) On alti what do we have here?
    • 02:12:43 (Captain) Its impossible02:12:45 (Bonin) In alti what do we have?02:12:47 (Robert) What do you mean on altitude?02:12:48 (Bonin) Yeah yeah yeah, Im going down, no?02:12:50 (Robert) Youre going down, yes02:12:52 (Captain) Hey you02:12:53 (Captain) Youre in02:12:54 (Captain) Get the wings horizontal02:12:56 (Robert) Get the wings horizontal02:12:56 (Bonin) Thats what Im trying to do02:12:57 (Captain) Get the wings horizontal!
    • 02:12:43 (Captain) Its impossible02:12:45 (Bonin) In alti what do we have?02:12:47 (Robert) What do you mean on altitude?02:12:48 (Bonin) Yeah yeah yeah, Im going down, no?02:12:50 (Robert) Youre going down, yes02:12:52 (Captain) Hey you02:12:53 (Captain) Youre in02:12:54 (Captain) Get the wings horizontal02:12:56 (Robert) Get the wings horizontal02:12:56 (Bonin) Thats what Im trying to do02:12:57 (Captain) Get the wings horizontal!
    • 02:12:43 (Captain) Its impossible02:12:45 (Bonin) In alti what do we have?02:12:47 (Robert) What do you mean on altitude?02:12:48 (Bonin) Yeah yeah yeah, Im going down, no?02:12:50 (Robert) Youre going down, yes02:12:52 (Captain) Hey you02:12:53 (Captain) Youre in02:12:54 (Captain) Get the wings horizontal02:12:56 (Robert) Get the wings horizontal02:12:56 (Bonin) Thats what Im trying to do02:12:57 (Captain) Get the wings horizontal!
    • 02:12:58 (Bonin) Im at the limit... with the roll02:13:00 (Captain) The rudder bar02:13:05 (Captain) Wings horizontal.. go... gently, gently02:13:11 (Captain) Hey er...02:13:11 (Robert) We lost it all at the left02:13:13 (Robert) Ive got nothing there02:13:15 (Captain) What do you have?02:13:17 (Captain) No wait02:13:18 (Bonin) Were there, were there, were passing level one hundred02:13:19 (Robert) Wait, me, I have, I have the controls, eh?02:13:25 (Bonin) What is... how come were continuing to go down right now?
    • 02:12:58 (Bonin) Im at the limit... with the roll02:13:00 (Captain) The rudder bar02:13:05 (Captain) Wings horizontal.. go... gently, gently02:13:11 (Captain) Hey er...02:13:11 (Robert) We lost it all at the left02:13:13 (Robert) Ive got nothing there02:13:15 (Captain) What do you have?02:13:17 (Captain) No wait02:13:18 (Bonin) Were there, were there, were passing level one hundred02:13:19 (Robert) Wait, me, I have, I have the controls, eh?02:13:25 (Bonin) What is... how come were continuing to go down right now?
    • 02:12:58 (Bonin) Im at the limit... with the roll02:13:00 (Captain) The rudder bar02:13:05 (Captain) Wings horizontal.. go... gently, gently02:13:11 (Captain) Hey er...02:13:11 (Robert) We lost it all at the left02:13:13 (Robert) Ive got nothing there02:13:15 (Captain) What do you have?02:13:17 (Captain) No wait02:13:18 (Bonin) Were there, were there, were passing level one hundred02:13:19 (Robert) Wait, me, I have, I have the controls, eh?02:13:25 (Bonin) What is... how come were continuing to go down right now?
    • 02:13:28 (Robert) Try to find what you can do with your controls up there02:13:30 (Robert) The primaries and so on02:13:30 (Captain) It wont do anything02:13:31 (Captain) It wont do anything02:13:31 (Bonin) At level one hundred02:13:36 (Bonin) Nine thousand feet02:13:38 (Captain) Careful with the rudder bar there02:13:39 (Robert) Climb, climb, climb, climb02:13:40 (Bonin) But Ive been at maxi nose-up for a while02:13:42 (Captain) No, no, no... dont climb02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
    • 02:13:28 (Robert) Try to find what you can do with your controls up there02:13:30 (Robert) The primaries and so on02:13:30 (Captain) It wont do anything02:13:31 (Captain) It wont do anything02:13:31 (Bonin) At level one hundred02:13:36 (Bonin) Nine thousand feet02:13:38 (Captain) Careful with the rudder bar there02:13:39 (Robert) Climb, climb, climb, climb02:13:40 (Bonin) But Ive been at maxi nose-up for a while02:13:42 (Captain) No, no, no... dont climb02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
    • 02:13:28 (Robert) Try to find what you can do with your controls up there02:13:30 (Robert) The primaries and so on02:13:30 (Captain) It wont do anything02:13:31 (Captain) It wont do anything02:13:31 (Bonin) At level one hundred02:13:36 (Bonin) Nine thousand feet02:13:38 (Captain) Careful with the rudder bar there02:13:39 (Robert) Climb, climb, climb, climb02:13:40 (Bonin) But Ive been at maxi nose-up for a while02:13:42 (Captain) No, no, no... dont climb02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
    • 02:13:46 (Bonin) Go ahead, you have the controls. We are still in TOGA, eh?02:13:53 (Captain) AP OFF02:13:59 (Bonin) Gentlemen02:14:05 (Captain) Watch out, youre pitching up there02:14:05 (Robert) Im pitching up02:14:06 (Captain) Youre pitching up02:14:07 (Robert) Im pitching up02:14:07 (Bonin) Well, we need to, we are at four thousand feet...
    • 02:13:46 (Bonin) Go ahead, you have the controls. We are still in TOGA, eh?02:13:53 (Captain) AP OFF02:13:59 (Bonin) Gentlemen02:14:05 (Captain) Watch out, youre pitching up there02:14:05 (Robert) Im pitching up02:14:06 (Captain) Youre pitching up02:14:07 (Robert) Im pitching up02:14:07 (Bonin) Well, we need to, we are at four thousand feet...
    • 02:14:10 (Captain) Youre pitching up02:14:18 (Captain) Go on pull02:14:19 (Bonin) Lets go, pull up, pull up, pull up02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!02:14:25 (Bonin) But whats happening?02:14:26 (Captain) Ten degrees of pitch...02:14:28 End of recording
    • 02:14:10 (Captain) Youre pitching up02:14:18 (Captain) Go on pull02:14:19 (Bonin) Lets go, pull up, pull up, pull up02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!02:14:25 (Bonin) But whats happening?02:14:26 (Captain) Ten degrees of pitch...02:14:28 End of recording
    • 02:14:10 (Captain) Youre pitching up02:14:18 (Captain) Go on pull02:14:19 (Bonin) Lets go, pull up, pull up, pull up02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!02:14:25 (Bonin) But whats happening?02:14:26 (Captain) Ten degrees of pitch...02:14:28 End of recording
    • 02:14:10 (Captain) Youre pitching up02:14:18 (Captain) Go on pull02:14:19 (Bonin) Lets go, pull up, pull up, pull up02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!02:14:25 (Bonin) But whats happening?02:14:26 (Captain) Ten degrees of pitch...02:14:28 End of recording
    • 02:14:10 (Captain) Youre pitching up02:14:18 (Captain) Go on pull02:14:19 (Bonin) Lets go, pull up, pull up, pull up02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!02:14:25 (Bonin) But whats happening?02:14:26 (Captain) Ten degrees of pitch...02:14:28 End of recording
    • 02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!
    • 02:14:23 (Bonin) Damn it, were going to crash... This cant be happening!
    • • Final Air France 447 Report: Pilots misunderstood their situation• Poorly-trained pilots to blame for Air France crash that killed 228• Final Air France crash report says pilots failed to react swiftly• Air France 447 downed as crew ignored alarms• Air France 447 crash a result of crew ignoring alarms
    • • The Atlantic• Daily Mail• CNN• New Scientist• Gizmodo
    • Convenient narrative
    • “root cause”
    • “human error”
    • Bad apples
    • “if we weed out thebad apples, the system will equalise”
    • What you call "root cause" issimply the place you stoplooking any further -- Sidney Dekker Professor of Human Factors & Flight Safety, Lund University
    • Dubois 10,988 flying hours, of which 6,258 astotal Captainhours on type 1,747, all as Captainin the previous six months 346 hours, 18 landings, 15 take-offsin the previous three months 168 hours, 8 landings, 6 take-offsin the previous 30 days 57 hours, 3 landings, 2 take-offs
    • Roberttotal 6,547 flying hourshours on type 4,479 flying hoursin the previous six months 204 hours, 9 landings, 11 take-offsin the previous three months 99 hours, 6 landings, 5 take-offsin the previous 30 days 39 hours, 2 landings, 2 take-offs
    • Bonintotal 2,936 flying hourshours on type 807in the previous six months 368 hours, 16 landings, 18 take-offsin the previous three months 191 hours, 7 landings, 8 take-offsin the previous 30 days 61 hours, 1 landings, 2 take-offs
    • Critical flaw:How would other pilots react under the same circumstances?
    • What appears in the crew behavior is that mostprobably, a different crew should have donethe same action. So, we cannot blame thiscrew. What we can say is that most probablythis crew and most crews were not prepared toface such an event. -- Jean-Paul Troadec Bureau dEnquêtes et dAnalyses pour la Sécurité de lAviation Civile
    • What appears in the crew behavior is that mostprobably, a different crew should have donethe same action. So, we cannot blame thiscrew. What we can say is that most probablythis crew and most crews were not prepared toface such an event. -- Jean-Paul Troadec Bureau dEnquêtes et dAnalyses pour la Sécurité de lAviation Civile
    • Actors in acomplex system
    • Systems in acomplex system
    • Systems in aseries of nestedcomplex systems
    • “root cause”
    • Cartesian-Newtonian worldview
    • hindsight != foresight
    • [hindsight] converts a oncevague, unlikely future into an immediate, certain past -- Sidney Dekker Professor of Human Factors & Flight Safety, Lund University
    • We have all the facts
    • Investigation took 3 years
    • Event unfolded in10 minutes
    • Fog of War
    • Limited facts at hand in a rapidly developing situation
    • Local rationality
    • “people make what theythink are best decisionsbased on data at hand”
    • Hindsightaffords us global rationality
    • What systemswere in play?
    • Modes of operation
    • Flight control modes
    • Normal law ground, flight, flare modes Alternate lawalternate law 1, alternate law 2
    • Lawreconfigurationfeedback
    • There are three major categories of message that can be transmitted:• non-vocal (ATC) communication messages with an air traffic control centre• operational communication messages (AOC) with the operator’s operations centre• maintenance messages, exclusively from the aircraft to the maintenance centre
    • Time of Reception Message02:10 WRN/WN0906010210 221002006AUTO FLT AP OFF02:10 WRN/WN0906010210 226201006AUTO FLT REAC W/S DET FAULT02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW02:10 WRN/WN0906010210 228300206FLAG ON CAPT PFD SPD LIMIT02:10 #0210/+2.98-30.5902:10 WRN/WN0906010210 228301206FLAG ON F/O PFD SPD LIMIT02:10 WRN/WN0906010210 223002506AUTO FLT A/THR OFF02:10 WRN/WN0906010210 344300506NAV TCAS FAULT02:11 WRN/WN0906010210 228300106FLAG ON CAPT PFD FD02:11 WRN/WN0906010210 228301106FLAG ON F/O PFD FD02:11 WRN/WN0906010210 272302006F/CTL RUD TRV LIM FAULT02:11 WRN/WN0906010210 279045506MAINTENANCE STATUS EFCS 202:11 WRN/WN0906010210 279045006MAINTENANCE STATUS EFCS 102:11 FLR/FR0906010210 34111506EFCS2 1,EFCS1,AFS,,,,,PROBE-PITOT 1X2 / 2X3 / 1X3 (9DA),HARD02:11 FLR/FR0906010210 27933406EFCS1 X2,EFCS2X,,,,,,FCPC2 (2CE2) /WRG:ADIRU1 BUS ADR1-2 TO FCPC2,HARD02:12 WRN/WN0906010211 341200106FLAG ON CAPT PFD FPV02:12 WRN/WN0906010211 341201106FLAG ON F/O PFD FPV02:12 WRN/WN0906010212 341040006NAV ADR DISAGREE02:13 FLR/FR0906010211 34220006ISIS 1,,,,,,,ISIS(22FN-10FC) SPEED OR MACH FUNCTION,HARD02:13 FLR/FR0906010211 34123406IR2 1,EFCS1X,IR1,IR3,,,,ADIRU2 (1FP2),HARD2:13:16 ~ 2:13:41 Possible "Loss of Signal" with satellite02:13 WRN/WN0906010213 279002506F/CTL PRIM 1 FAULT02:13 WRN/WN0906010213 279004006F/CTL SEC 1 FAULT02:14 WRN/WN0906010214 341036006MAINTENANCE STATUS ADR 202:14 FLR/FR0906010213 22833406AFS 1,,,,,,,FMGEC1(1CA1),INTERMITTENT02:14 WRN/WN0906010214 213100206ADVISORY CABIN VERTICAL SPEED
    • 02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW
    • 02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW
    • small textual warning here
    • Realisation
    • 02:13:40 (Bonin) But Ive been at maxi nose-up for a while02:13:42 (Captain) No, no, no... dont climb02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls! ...02:14:28 End of recording
    • How does your HAprovide feedback?
    • Reconfiguration feedback?
    • How do these modesbehave differently?
    • What about modesyou haven’t seen?
    • Sensory feedback
    • Obvious change incolor, size, font
    • Know howcolour is processedby the brain
    • Familiarise yourself with type
    • Familiarise yourself with type
    • Familiarise yourself with type
    • Optimise for 3am you
    • Input control
    • Co-pilot feedback
    • Averaged input
    • dual input feedback
    • CRM
    • Startling effect
    • Are your inputs averaged?
    • How do yourengineers troubleshoot during incidents?
    • Every manfor himself?
    • How do you co-ordinate change?
    • Does someonehave overview?
    • How is thatresponsibility assigned?
    • How is information disseminated?
    • How doesthe business knowwhat is happening?
    • Do you have a process?
    • Do you practice this?
    • What data doyou rely on?
    • Pair
    • Vocalise
    • Minimise &Compartmentalise
    • Record
    • Timeline
    • maintained byco-ordinator
    • HUD
    • isolated sensors
    • different values
    • discrepancies
    • 02:12:27 (Robert) Youre climbing02:12:28 (Robert) Youre going down down down02:12:28 (Captain) Going down02:12:30 (Bonin) Am I going down now?02:12:31 (Robert) Go down02:12:32 (Captain) No you climb there02:12:32 (Bonin) Im climbing okay so were going down02:12:34 (Captain) Youre climbing02:12:39 (Bonin) Okay, were in TOGA02:12:41 (Bonin) What are we here?02:12:41 (Bonin) On alti what do we have here?
    • CRM
    • Contextualnavigation
    • different navigationrequirements
    • dashboards
    • deep dive on details
    • test a theory
    • scientific method: improvised
    • linkable
    • correlation
    • human pattern recognition
    • human pattern recognition(provided there is enough adaptive capacity)
    • Stream of alerts
    • 70 stall warnings
    • • Final Air France 447 Report: Pilots misunderstood their situation• Poorly-trained pilots to blame for Air France crash that killed 228• Final Air France crash report says pilots failed to react swiftly• Air France 447 downed as crew ignored alarms• Air France 447 crash a result of crew ignoring alarms
    • “They shouldhave reacted!”
    • autopilot disconnect
    • alternate lawreconfiguration
    • alert priority level?
    • overwhelmedby feedback
    • Alert Fatigue
    • startling effect
    • reducedadaptivecapacity
    • dampening
    • Brute force:manual silence
    • limit # of engineers who watch alerts & graphs
    • smarter alertaggregation?
    • PagerDuty
    • Flapjack
    • Systems thinking
    • System capable of failure
    • System capable of success
    • • System that enables communication
    • • System that enables communication• System that exposes secrets
    • • System that enables communication• System that exposes secrets• System that rob us
    • • System that enables communication• System that exposes secrets• System that rob us• System that funds innovation
    • • System that enables communication• System that exposes secrets• System that rob us• System that funds innovation• System that kills us
    • • System that enables communication• System that exposes secrets• System that rob us• System that funds innovation• System that kills us• System that allow us to fly across the world
    • Failure is pervasive
    • Failure is complex
    • Failure is just another mode of operation
    • Your system may not control fate ofpeople’s lives
    • But people may depend on it
    • Anthropocentrism
    • Technocentrism
    • The squishymiddle ground
    • Operable Systemsman + machine
    • No amoral actors
    • We need to look at it from a systemsapproach, a human/technology system thathas to work together. This involves aircraftdesign and certification, training and humanfactors. If you look at the human factorsalone, then youre missing half or two-thirdsof the total system failure -- Chesley Sullenberger Pilot, US 1549, Hudson River Ditching
    • We need to look at it from a systemsapproach, a human/technology system thathas to work together. This involves aircraftdesign and certification, training and humanfactors. If you look at the human factorsalone, then youre missing half or two-thirdsof the total system failure -- Chesley Sullenberger Pilot, US 1549, Hudson River Ditching
    • • Damn it, were going to crash... This cant be happening!
    • • Final Air France 447 Report: Pilots misunderstood their situation• Poorly-trained pilots to blame for Air France crash that killed 228• Final Air France crash report says pilots failed to react swiftly• Air France 447 downed as crew ignored alarms• Air France 447 crash a result of crew ignoring alarms
    • Thank you
    • Liked the talk? Let @auxesis know! Thank you
    • Sidney DekkerField Guide to Understand Human ErrorDrift Into FailureJust CultureJon RauserLook at Your Data @ VelocityRyan TomaykoDistributed Management @ Monktoberfest 2012BEA Report on AF447 incident