Successfully reported this slideshow.
Your SlideShare is downloading. ×

Escalating complexity: DevOps learnings from Air France 447

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 238 Ad

Escalating complexity: DevOps learnings from Air France 447

Download to read offline

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems.
Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand.
While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated.
In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems.
Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand.
While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated.
In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

Advertisement
Advertisement

More Related Content

Advertisement

Recently uploaded (20)

Escalating complexity: DevOps learnings from Air France 447

  1. 1. Escalating complexity: DevOps learnings from Air France 447
  2. 2. Lindsay Holmwood @auxesis
  3. 3. Engineering Manager @ Bulletproof Networks
  4. 4. cucumber-nagios Visage Flapjack
  5. 5. • On 31 May 2009, Air France 447 departed from Rio de Janeiro-Galeão International Airport at 22:29 UTC. It was scheduled to arrive at Paris-Charles de Gaulle International Airport 11 hours later.
  6. 6. • On 31 May 2009, Air France 447 departed from Rio de Janeiro-Galeão International Airport at 22:29 UTC. It was scheduled to arrive at Paris-Charles de Gaulle International Airport 11 hours later. • 3 hours and 45 minutes later, it crashed into the Atlantic Ocean, killing 216 passengers, and 12 aircrew.
  7. 7. • On 31 May 2009, Air France 447 departed from Rio de Janeiro-Galeão International Airport at 22:29 UTC. It was scheduled to arrive at Paris-Charles de Gaulle International Airport 11 hours later. • 3 hours and 45 minutes later, it crashed into the Atlantic Ocean, killing 216 passengers, and 12 aircrew. • There were no survivors.
  8. 8. • This is AF447’s flight path:
  9. 9. • AF447 charted a course through a band of equatorial thunderstorms
  10. 10. • This is what happened in the last 15 minutes of the flight.
  11. 11. 02:03:44 (Bonin) The inter-tropical convergence... look, we're in it, between 'Salpu' and 'Tasil.' And then, look, we're right in it... 02:05:55 (Robert) Yes, let's call them in the back, to let them know... 02:05:59 (FA) Yes? Marilyn. 02:06:04 (Bonin) Yes, Marilyn, it's Pierre up front... Listen, in 2 minutes, we're going to be getting into an area where things are going to be moving around a little bit more than now. You'll want to take care. 02:06:13 (FA) Okay, we should sit down then? 02:06:15 (Bonin) Well, I think that's not a bad idea. Give your friends a heads-up. 02:06:18 (FA) Yeah, okay, I'll tell the others in the back. Thanks a lot. 02:06:19 (Bonin) I'll call you back as soon as we're out of it. 02:06:20 (FA) Okay.
  12. 12. 02:03:44 (Bonin) The inter-tropical convergence... look, we're in it, between 'Salpu' and 'Tasil.' And then, look, we're right in it... 02:05:55 (Robert) Yes, let's call them in the back, to let them know... 02:05:59 (FA) Yes? Marilyn. 02:06:04 (Bonin) Yes, Marilyn, it's Pierre up front... Listen, in 2 minutes, we're going to be getting into an area where things are going to be moving around a little bit more than now. You'll want to take care. 02:06:13 (FA) Okay, we should sit down then? 02:06:15 (Bonin) Well, I think that's not a bad idea. Give your friends a heads-up. 02:06:18 (FA) Yeah, okay, I'll tell the others in the back. Thanks a lot. 02:06:19 (Bonin) I'll call you back as soon as we're out of it. 02:06:20 (FA) Okay.
  13. 13. 02:06:50 (Bonin) Let's go for the anti-icing system. It's better than nothing. 02:07:00 (Bonin) We seem to be at the end of the cloud layer, it might be okay. 02:08:03 (Robert) You can possibly pull it a little to the left. 02:08:05 (Bonin) You can possibly pull it a little to the left. We're agreed that we're in manual, yeah?
  14. 14. Captain Marc Dubois hands control to Robert + Bonin, • and takes the second mandatory rest break.
  15. 15. 02:10:06 (Bonin) I have the controls. 02:10:07 (Robert) Okay. 02:10:07 (Robert) What's this? 02:10:15 (Bonin) There's no good... there's no good speed indication. 02:10:16 (Robert) We've lost the, the, the speeds, then? 02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed. 02:10:28 (Bonin) Okay, okay, I'm descending. 02:10:30 (Robert) Stabilize. 02:10:31 (Bonin) Yeah. 02:10:31 (Robert) Descend... It says we're going up... It says we're going up, so descend. 02:10:36 (Robert) Descend!
  16. 16. 02:10:06 (Bonin) I have the controls. 02:10:07 (Robert) Okay. 02:10:07 (Robert) What's this? 02:10:15 (Bonin) There's no good... there's no good speed indication. 02:10:16 (Robert) We've lost the, the, the speeds, then? 02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed. 02:10:28 (Bonin) Okay, okay, I'm descending. 02:10:30 (Robert) Stabilize. 02:10:31 (Bonin) Yeah. 02:10:31 (Robert) Descend... It says we're going up... It says we're going up, so descend. 02:10:36 (Robert) Descend!
  17. 17. 02:10:06 (Bonin) I have the controls. 02:10:07 (Robert) Okay. 02:10:07 (Robert) What's this? 02:10:15 (Bonin) There's no good... there's no good speed indication. 02:10:16 (Robert) We've lost the, the, the speeds, then? 02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed. 02:10:28 (Bonin) Okay, okay, I'm descending. 02:10:30 (Robert) Stabilize. 02:10:31 (Bonin) Yeah. 02:10:31 (Robert) Descend... It says we're going up... It says we're going up, so descend. 02:10:36 (Robert) Descend!
  18. 18. 02:10:06 (Bonin) I have the controls. 02:10:07 (Robert) Okay. 02:10:07 (Robert) What's this? 02:10:15 (Bonin) There's no good... there's no good speed indication. 02:10:16 (Robert) We've lost the, the, the speeds, then? 02:10:27 (Robert) Pay attention to your speed. Pay attention to your speed. 02:10:28 (Bonin) Okay, okay, I'm descending. 02:10:30 (Robert) Stabilize. 02:10:31 (Bonin) Yeah. 02:10:31 (Robert) Descend... It says we're going up... It says we're going up, so descend. 02:10:36 (Robert) Descend!
  19. 19. 02:10:37 (Bonin) Here we go, we're descending. 02:10:38 (Robert) Gently! 02:10:41(Bonin) We're... yeah, we're in a climb. 02:10:49 (Robert) Damn it, where is he? 02:10:55 (Robert) Damn it! 02:11:03 (Bonin) I'm in TOGA, huh? 02:11:06 (Robert) Damn it, is he coming or not? 02:11:21 (Robert) We still have the engines! What the hell is happening? I don't understand what's happening. 02:11:21 (Robert) But we’ve got the engines. What's happening? Do you understand what’s happening or not? 02:11:32 (Bonin) I don't have control of the plane any more now. I don't have control of the plane at all!
  20. 20. 02:10:37 (Bonin) Here we go, we're descending. 02:10:38 (Robert) Gently! 02:10:41(Bonin) We're... yeah, we're in a climb. 02:10:49 (Robert) Damn it, where is he? 02:10:55 (Robert) Damn it! 02:11:03 (Bonin) I'm in TOGA, huh? 02:11:06 (Robert) Damn it, is he coming or not? 02:11:21 (Robert) We still have the engines! What the hell is happening? I don't understand what's happening. 02:11:21 (Robert) But we’ve got the engines. What's happening? Do you understand what’s happening or not? 02:11:32 (Bonin) I don't have control of the plane any more now. I don't have control of the plane at all!
  21. 21. 02:10:37 (Bonin) Here we go, we're descending. 02:10:38 (Robert) Gently! 02:10:41(Bonin) We're... yeah, we're in a climb. 02:10:49 (Robert) Damn it, where is he? 02:10:55 (Robert) Damn it! 02:11:03 (Bonin) I'm in TOGA, huh? 02:11:06 (Robert) Damn it, is he coming or not? 02:11:21 (Robert) We still have the engines! What the hell is happening? I don't understand what's happening. 02:11:21 (Robert) But we’ve got the engines. What's happening? Do you understand what’s happening or not? 02:11:32 (Bonin) I don't have control of the plane any more now. I don't have control of the plane at all!
  22. 22. 02:10:37 (Bonin) Here we go, we're descending. 02:10:38 (Robert) Gently! 02:10:41(Bonin) We're... yeah, we're in a climb. 02:10:49 (Robert) Damn it, where is he? 02:10:55 (Robert) Damn it! 02:11:03 (Bonin) I'm in TOGA, huh? 02:11:06 (Robert) Damn it, is he coming or not? 02:11:21 (Robert) We still have the engines! What the hell is happening? I don't understand what's happening. 02:11:21 (Robert) But we’ve got the engines. What's happening? Do you understand what’s happening or not? 02:11:32 (Bonin) I don't have control of the plane any more now. I don't have control of the plane at all!
  23. 23. 02:11:37 (Robert) Controls to the left! 02:11:41 (Robert) ...what is that? 02:11:41 (Bonin) I have the impression (we have) the speed 02:11:42 (Captain) Er, what are you doing? 02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening 02:11:45 (Bonin) We’re losing control of the aeroplane there 02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything. 02:11:51 (Captain) So take that, take that 02:11:55 (Robert) Take that, take that 02:11:57 (Robert) Try to take that
  24. 24. 02:11:37 (Robert) Controls to the left! 02:11:41 (Robert) ...what is that? 02:11:41 (Bonin) I have the impression (we have) the speed 02:11:42 (Captain) Er, what are you doing? 02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening 02:11:45 (Bonin) We’re losing control of the aeroplane there 02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything. 02:11:51 (Captain) So take that, take that 02:11:55 (Robert) Take that, take that 02:11:57 (Robert) Try to take that
  25. 25. 02:11:37 (Robert) Controls to the left! 02:11:41 (Robert) ...what is that? 02:11:41 (Bonin) I have the impression (we have) the speed 02:11:42 (Captain) Er, what are you doing? 02:11:43 (Robert) What’s happening? I don’t know, I don’t know what’s happening 02:11:45 (Bonin) We’re losing control of the aeroplane there 02:11:46 (Robert) We lost all control of the aeroplane. We don’t understand anything. We’ve tried everything. 02:11:51 (Captain) So take that, take that 02:11:55 (Robert) Take that, take that 02:11:57 (Robert) Try to take that
  26. 26. 02:11:58 (Bonin) I have a problem - it’s that I don’t have vertical speed indication 02:12:01 (Captain) Alright 02:12:01 (Bonin) I have no more displays 02:12:02 (Robert) We have no valid displays 02:12:04 (Bonin) I have the impression that we have some crazy speed, no? What do you think? 02:12:06 (Robert) No. 02:12:07 (Bonin) No? 02:12:07 (Robert) No, above all don't extend the 02:12:07 (Bonin) Okay 02:12:09 (Robert) Don't extend
  27. 27. 02:11:58 (Bonin) I have a problem - it’s that I don’t have vertical speed indication 02:12:01 (Captain) Alright 02:12:01 (Bonin) I have no more displays 02:12:02 (Robert) We have no valid displays 02:12:04 (Bonin) I have the impression that we have some crazy speed, no? What do you think? 02:12:06 (Robert) No. 02:12:07 (Bonin) No? 02:12:07 (Robert) No, above all don't extend the 02:12:07 (Bonin) Okay 02:12:09 (Robert) Don't extend
  28. 28. 02:12:11 (Bonin) So we’re still going down 02:12:12 (Robert) We’re pulling 02:12:14 (Robert) What do you think about it? What do you think? What do we need to do? 02:12:15 (Captain) There - I don’t know. There - it’s going down. 02:12:19 (Bonin) There you are. 02:12:20 (Bonin) That’s good we should be wings level, no it won’t 02:12:23 (Captain) The wings to flat horizon, the standby horizon 02:12:25 (Robert) The horizon! 02:12:26 (Bonin) Okay 02:12:26 (Robert) Speed?
  29. 29. 02:12:11 (Bonin) So we’re still going down 02:12:12 (Robert) We’re pulling 02:12:14 (Robert) What do you think about it? What do you think? What do we need to do? 02:12:15 (Captain) There - I don’t know. There - it’s going down. 02:12:19 (Bonin) There you are. 02:12:20 (Bonin) That’s good we should be wings level, no it won’t 02:12:23 (Captain) The wings to flat horizon, the standby horizon 02:12:25 (Robert) The horizon! 02:12:26 (Bonin) Okay 02:12:26 (Robert) Speed?
  30. 30. 02:12:27 (Robert) You're climbing 02:12:28 (Robert) You're going down down down 02:12:28 (Captain) Going down 02:12:30 (Bonin) Am I going down now? 02:12:31 (Robert) Go down 02:12:32 (Captain) No you climb there 02:12:32 (Bonin) I'm climbing okay so we're going down 02:12:34 (Captain) You're climbing 02:12:39 (Bonin) Okay, we're in TOGA 02:12:41 (Bonin) What are we here? 02:12:41 (Bonin) On alti what do we have here?
  31. 31. 02:12:27 (Robert) You're climbing 02:12:28 (Robert) You're going down down down 02:12:28 (Captain) Going down 02:12:30 (Bonin) Am I going down now? 02:12:31 (Robert) Go down 02:12:32 (Captain) No you climb there 02:12:32 (Bonin) I'm climbing okay so we're going down 02:12:34 (Captain) You're climbing 02:12:39 (Bonin) Okay, we're in TOGA 02:12:41 (Bonin) What are we here? 02:12:41 (Bonin) On alti what do we have here?
  32. 32. 02:12:27 (Robert) You're climbing 02:12:28 (Robert) You're going down down down 02:12:28 (Captain) Going down 02:12:30 (Bonin) Am I going down now? 02:12:31 (Robert) Go down 02:12:32 (Captain) No you climb there 02:12:32 (Bonin) I'm climbing okay so we're going down 02:12:34 (Captain) You're climbing 02:12:39 (Bonin) Okay, we're in TOGA 02:12:41 (Bonin) What are we here? 02:12:41 (Bonin) On alti what do we have here?
  33. 33. 02:12:43 (Captain) It's impossible 02:12:45 (Bonin) In alti what do we have? 02:12:47 (Robert) What do you mean on altitude? 02:12:48 (Bonin) Yeah yeah yeah, I'm going down, no? 02:12:50 (Robert) You're going down, yes 02:12:52 (Captain) Hey you 02:12:53 (Captain) You're in 02:12:54 (Captain) Get the wings horizontal 02:12:56 (Robert) Get the wings horizontal 02:12:56 (Bonin) That's what I'm trying to do 02:12:57 (Captain) Get the wings horizontal!
  34. 34. 02:12:43 (Captain) It's impossible 02:12:45 (Bonin) In alti what do we have? 02:12:47 (Robert) What do you mean on altitude? 02:12:48 (Bonin) Yeah yeah yeah, I'm going down, no? 02:12:50 (Robert) You're going down, yes 02:12:52 (Captain) Hey you 02:12:53 (Captain) You're in 02:12:54 (Captain) Get the wings horizontal 02:12:56 (Robert) Get the wings horizontal 02:12:56 (Bonin) That's what I'm trying to do 02:12:57 (Captain) Get the wings horizontal!
  35. 35. 02:12:43 (Captain) It's impossible 02:12:45 (Bonin) In alti what do we have? 02:12:47 (Robert) What do you mean on altitude? 02:12:48 (Bonin) Yeah yeah yeah, I'm going down, no? 02:12:50 (Robert) You're going down, yes 02:12:52 (Captain) Hey you 02:12:53 (Captain) You're in 02:12:54 (Captain) Get the wings horizontal 02:12:56 (Robert) Get the wings horizontal 02:12:56 (Bonin) That's what I'm trying to do 02:12:57 (Captain) Get the wings horizontal!
  36. 36. 02:12:58 (Bonin) I'm at the limit... with the roll 02:13:00 (Captain) The rudder bar 02:13:05 (Captain) Wings horizontal.. go... gently, gently 02:13:11 (Captain) Hey er... 02:13:11 (Robert) We lost it all at the left 02:13:13 (Robert) I've got nothing there 02:13:15 (Captain) What do you have? 02:13:17 (Captain) No wait 02:13:18 (Bonin) We're there, we're there, we're passing level one hundred 02:13:19 (Robert) Wait, me, I have, I have the controls, eh? 02:13:25 (Bonin) What is... how come we're continuing to go down right now?
  37. 37. 02:12:58 (Bonin) I'm at the limit... with the roll 02:13:00 (Captain) The rudder bar 02:13:05 (Captain) Wings horizontal.. go... gently, gently 02:13:11 (Captain) Hey er... 02:13:11 (Robert) We lost it all at the left 02:13:13 (Robert) I've got nothing there 02:13:15 (Captain) What do you have? 02:13:17 (Captain) No wait 02:13:18 (Bonin) We're there, we're there, we're passing level one hundred 02:13:19 (Robert) Wait, me, I have, I have the controls, eh? 02:13:25 (Bonin) What is... how come we're continuing to go down right now?
  38. 38. 02:12:58 (Bonin) I'm at the limit... with the roll 02:13:00 (Captain) The rudder bar 02:13:05 (Captain) Wings horizontal.. go... gently, gently 02:13:11 (Captain) Hey er... 02:13:11 (Robert) We lost it all at the left 02:13:13 (Robert) I've got nothing there 02:13:15 (Captain) What do you have? 02:13:17 (Captain) No wait 02:13:18 (Bonin) We're there, we're there, we're passing level one hundred 02:13:19 (Robert) Wait, me, I have, I have the controls, eh? 02:13:25 (Bonin) What is... how come we're continuing to go down right now?
  39. 39. 02:13:28 (Robert) Try to find what you can do with your controls up there 02:13:30 (Robert) The primaries and so on 02:13:30 (Captain) It won't do anything 02:13:31 (Captain) It won't do anything 02:13:31 (Bonin) At level one hundred 02:13:36 (Bonin) Nine thousand feet 02:13:38 (Captain) Careful with the rudder bar there 02:13:39 (Robert) Climb, climb, climb, climb 02:13:40 (Bonin) But I've been at maxi nose-up for a while 02:13:42 (Captain) No, no, no... don't climb 02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
  40. 40. 02:13:28 (Robert) Try to find what you can do with your controls up there 02:13:30 (Robert) The primaries and so on 02:13:30 (Captain) It won't do anything 02:13:31 (Captain) It won't do anything 02:13:31 (Bonin) At level one hundred 02:13:36 (Bonin) Nine thousand feet 02:13:38 (Captain) Careful with the rudder bar there 02:13:39 (Robert) Climb, climb, climb, climb 02:13:40 (Bonin) But I've been at maxi nose-up for a while 02:13:42 (Captain) No, no, no... don't climb 02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
  41. 41. 02:13:28 (Robert) Try to find what you can do with your controls up there 02:13:30 (Robert) The primaries and so on 02:13:30 (Captain) It won't do anything 02:13:31 (Captain) It won't do anything 02:13:31 (Bonin) At level one hundred 02:13:36 (Bonin) Nine thousand feet 02:13:38 (Captain) Careful with the rudder bar there 02:13:39 (Robert) Climb, climb, climb, climb 02:13:40 (Bonin) But I've been at maxi nose-up for a while 02:13:42 (Captain) No, no, no... don't climb 02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls!
  42. 42. 02:13:46 (Bonin) Go ahead, you have the controls. We are still in TOGA, eh? 02:13:53 (Captain) AP OFF 02:13:59 (Bonin) Gentlemen 02:14:05 (Captain) Watch out, you're pitching up there 02:14:05 (Robert) I'm pitching up 02:14:06 (Captain) You're pitching up 02:14:07 (Robert) I'm pitching up 02:14:07 (Bonin) Well, we need to, we are at four thousand feet...
  43. 43. 02:13:46 (Bonin) Go ahead, you have the controls. We are still in TOGA, eh? 02:13:53 (Captain) AP OFF 02:13:59 (Bonin) Gentlemen 02:14:05 (Captain) Watch out, you're pitching up there 02:14:05 (Robert) I'm pitching up 02:14:06 (Captain) You're pitching up 02:14:07 (Robert) I'm pitching up 02:14:07 (Bonin) Well, we need to, we are at four thousand feet...
  44. 44. 02:14:10 (Captain) You're pitching up 02:14:18 (Captain) Go on pull 02:14:19 (Bonin) Let's go, pull up, pull up, pull up 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening! 02:14:25 (Bonin) But what's happening? 02:14:26 (Captain) Ten degrees of pitch... 02:14:28 End of recording
  45. 45. 02:14:10 (Captain) You're pitching up 02:14:18 (Captain) Go on pull 02:14:19 (Bonin) Let's go, pull up, pull up, pull up 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening! 02:14:25 (Bonin) But what's happening? 02:14:26 (Captain) Ten degrees of pitch... 02:14:28 End of recording
  46. 46. 02:14:10 (Captain) You're pitching up 02:14:18 (Captain) Go on pull 02:14:19 (Bonin) Let's go, pull up, pull up, pull up 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening! 02:14:25 (Bonin) But what's happening? 02:14:26 (Captain) Ten degrees of pitch... 02:14:28 End of recording
  47. 47. 02:14:10 (Captain) You're pitching up 02:14:18 (Captain) Go on pull 02:14:19 (Bonin) Let's go, pull up, pull up, pull up 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening! 02:14:25 (Bonin) But what's happening? 02:14:26 (Captain) Ten degrees of pitch... 02:14:28 End of recording
  48. 48. 02:14:10 (Captain) You're pitching up 02:14:18 (Captain) Go on pull 02:14:19 (Bonin) Let's go, pull up, pull up, pull up 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening! 02:14:25 (Bonin) But what's happening? 02:14:26 (Captain) Ten degrees of pitch... 02:14:28 End of recording
  49. 49. 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening!
  50. 50. 02:14:23 (Bonin) Damn it, we're going to crash... This can't be happening!
  51. 51. • Final Air France 447 Report: Pilots misunderstood their situation • Poorly-trained pilots to blame for Air France crash that killed 228 • Final Air France crash report says pilots failed to react swiftly • Air France 447 downed as crew ignored alarms • Air France 447 crash a result of crew ignoring alarms
  52. 52. • The Atlantic • Daily Mail • CNN • New Scientist • Gizmodo
  53. 53. Convenient narrative
  54. 54. “root cause”
  55. 55. “human error”
  56. 56. Bad apples
  57. 57. “if we weed out the bad apples, the system will equalise”
  58. 58. What you call "root cause" is simply the place you stop looking any further -- Sidney Dekker Professor of Human Factors & Flight Safety, Lund University
  59. 59. Dubois 10,988 flying hours, of which 6,258 as total Captain hours on type 1,747, all as Captain in the previous six months 346 hours, 18 landings, 15 take-offs in the previous three months 168 hours, 8 landings, 6 take-offs in the previous 30 days 57 hours, 3 landings, 2 take-offs
  60. 60. Robert total 6,547 flying hours hours on type 4,479 flying hours in the previous six months 204 hours, 9 landings, 11 take-offs in the previous three months 99 hours, 6 landings, 5 take-offs in the previous 30 days 39 hours, 2 landings, 2 take-offs
  61. 61. Bonin total 2,936 flying hours hours on type 807 in the previous six months 368 hours, 16 landings, 18 take-offs in the previous three months 191 hours, 7 landings, 8 take-offs in the previous 30 days 61 hours, 1 landings, 2 take-offs
  62. 62. Critical flaw: How would other pilots react under the same circumstances?
  63. 63. What appears in the crew behavior is that most probably, a different crew should have done the same action. So, we cannot blame this crew. What we can say is that most probably this crew and most crews were not prepared to face such an event. -- Jean-Paul Troadec Bureau d'Enquêtes et d'Analyses pour la Sécurité de l'Aviation Civile
  64. 64. What appears in the crew behavior is that most probably, a different crew should have done the same action. So, we cannot blame this crew. What we can say is that most probably this crew and most crews were not prepared to face such an event. -- Jean-Paul Troadec Bureau d'Enquêtes et d'Analyses pour la Sécurité de l'Aviation Civile
  65. 65. Actors in a complex system
  66. 66. Systems in a complex system
  67. 67. Systems in a series of nested complex systems
  68. 68. “root cause”
  69. 69. Cartesian-Newtonian worldview
  70. 70. hindsight != foresight
  71. 71. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker Professor of Human Factors & Flight Safety, Lund University
  72. 72. We have all the facts
  73. 73. Investigation took 3 years
  74. 74. Event unfolded in 10 minutes
  75. 75. Fog of War
  76. 76. Limited facts at hand in a rapidly developing situation
  77. 77. Local rationality
  78. 78. “people make what they think are best decisions based on data at hand”
  79. 79. Hindsight affords us global rationality
  80. 80. What systems were in play?
  81. 81. Modes of operation
  82. 82. Flight control modes
  83. 83. Normal law ground, flight, flare modes Alternate law alternate law 1, alternate law 2
  84. 84. Law reconfiguration feedback
  85. 85. There are three major categories of message that can be transmitted: • non-vocal (ATC) communication messages with an air traffic control centre • operational communication messages (AOC) with the operator’s operations centre • maintenance messages, exclusively from the aircraft to the maintenance centre
  86. 86. Time of Reception Message 02:10 WRN/WN0906010210 221002006AUTO FLT AP OFF 02:10 WRN/WN0906010210 226201006AUTO FLT REAC W/S DET FAULT 02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW 02:10 WRN/WN0906010210 228300206FLAG ON CAPT PFD SPD LIMIT 02:10 #0210/+2.98-30.59 02:10 WRN/WN0906010210 228301206FLAG ON F/O PFD SPD LIMIT 02:10 WRN/WN0906010210 223002506AUTO FLT A/THR OFF 02:10 WRN/WN0906010210 344300506NAV TCAS FAULT 02:11 WRN/WN0906010210 228300106FLAG ON CAPT PFD FD 02:11 WRN/WN0906010210 228301106FLAG ON F/O PFD FD 02:11 WRN/WN0906010210 272302006F/CTL RUD TRV LIM FAULT 02:11 WRN/WN0906010210 279045506MAINTENANCE STATUS EFCS 2 02:11 WRN/WN0906010210 279045006MAINTENANCE STATUS EFCS 1 02:11 FLR/FR0906010210 34111506EFCS2 1,EFCS1,AFS,,,,,PROBE-PITOT 1X2 / 2X3 / 1X3 (9DA),HARD 02:11 FLR/FR0906010210 27933406EFCS1 X2,EFCS2X,,,,,,FCPC2 (2CE2) /WRG:ADIRU1 BUS ADR1-2 TO FCPC2,HARD 02:12 WRN/WN0906010211 341200106FLAG ON CAPT PFD FPV 02:12 WRN/WN0906010211 341201106FLAG ON F/O PFD FPV 02:12 WRN/WN0906010212 341040006NAV ADR DISAGREE 02:13 FLR/FR0906010211 34220006ISIS 1,,,,,,,ISIS(22FN-10FC) SPEED OR MACH FUNCTION,HARD 02:13 FLR/FR0906010211 34123406IR2 1,EFCS1X,IR1,IR3,,,,ADIRU2 (1FP2),HARD 2:13:16 ~ 2:13:41 Possible "Loss of Signal" with satellite 02:13 WRN/WN0906010213 279002506F/CTL PRIM 1 FAULT 02:13 WRN/WN0906010213 279004006F/CTL SEC 1 FAULT 02:14 WRN/WN0906010214 341036006MAINTENANCE STATUS ADR 2 02:14 FLR/FR0906010213 22833406AFS 1,,,,,,,FMGEC1(1CA1),INTERMITTENT 02:14 WRN/WN0906010214 213100206ADVISORY CABIN VERTICAL SPEED
  87. 87. 02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW
  88. 88. 02:10 WRN/WN0906010210 279100506F/CTL ALTN LAW
  89. 89. small textual warning here
  90. 90. Realisation
  91. 91. 02:13:40 (Bonin) But I've been at maxi nose-up for a while 02:13:42 (Captain) No, no, no... don't climb 02:13:43 (Robert) Descend, then... Give me the controls... Give me the controls! ... 02:14:28 End of recording
  92. 92. How does your HA provide feedback?
  93. 93. Reconfiguration feedback?
  94. 94. How do these modes behave differently?
  95. 95. What about modes you haven’t seen?
  96. 96. Sensory feedback
  97. 97. Obvious change in color, size, font
  98. 98. Know how colour is processed by the brain
  99. 99. Familiarise yourself with type
  100. 100. Familiarise yourself with type
  101. 101. Familiarise yourself with type
  102. 102. Optimise for 3am you
  103. 103. Input control
  104. 104. Co-pilot feedback
  105. 105. Averaged input
  106. 106. dual input feedback
  107. 107. CRM
  108. 108. Startling effect
  109. 109. Are your inputs averaged?
  110. 110. How do your engineers troubleshoot during incidents?
  111. 111. Every man for himself?
  112. 112. How do you co- ordinate change?
  113. 113. Does someone have overview?
  114. 114. How is that responsibility assigned?
  115. 115. How is information disseminated?
  116. 116. How does the business know what is happening?
  117. 117. Do you have a process?
  118. 118. Do you practice this?
  119. 119. What data do you rely on?
  120. 120. Pair
  121. 121. Vocalise
  122. 122. Minimise & Compartmentalise
  123. 123. Record
  124. 124. Timeline
  125. 125. maintained by co-ordinator
  126. 126. HUD
  127. 127. isolated sensors
  128. 128. different values
  129. 129. discrepancies
  130. 130. 02:12:27 (Robert) You're climbing 02:12:28 (Robert) You're going down down down 02:12:28 (Captain) Going down 02:12:30 (Bonin) Am I going down now? 02:12:31 (Robert) Go down 02:12:32 (Captain) No you climb there 02:12:32 (Bonin) I'm climbing okay so we're going down 02:12:34 (Captain) You're climbing 02:12:39 (Bonin) Okay, we're in TOGA 02:12:41 (Bonin) What are we here? 02:12:41 (Bonin) On alti what do we have here?
  131. 131. CRM
  132. 132. Contextual navigation
  133. 133. different navigation requirements
  134. 134. dashboards
  135. 135. deep dive on details
  136. 136. test a theory
  137. 137. scientific method: improvised
  138. 138. linkable
  139. 139. correlation
  140. 140. human pattern recognition
  141. 141. human pattern recognition (provided there is enough adaptive capacity)
  142. 142. Stream of alerts
  143. 143. 70 stall warnings
  144. 144. • Final Air France 447 Report: Pilots misunderstood their situation • Poorly-trained pilots to blame for Air France crash that killed 228 • Final Air France crash report says pilots failed to react swiftly • Air France 447 downed as crew ignored alarms • Air France 447 crash a result of crew ignoring alarms
  145. 145. “They should have reacted!”
  146. 146. autopilot disconnect
  147. 147. alternate law reconfiguration
  148. 148. alert priority level?
  149. 149. overwhelmed by feedback
  150. 150. Alert Fatigue
  151. 151. startling effect
  152. 152. reduced adaptive capacity
  153. 153. dampening
  154. 154. Brute force: manual silence
  155. 155. limit # of engineers who watch alerts & graphs
  156. 156. smarter alert aggregation?
  157. 157. PagerDuty
  158. 158. Flapjack
  159. 159. Systems thinking
  160. 160. System capable of failure
  161. 161. System capable of success
  162. 162. • System that enables communication
  163. 163. • System that enables communication • System that exposes secrets
  164. 164. • System that enables communication • System that exposes secrets • System that rob us
  165. 165. • System that enables communication • System that exposes secrets • System that rob us • System that funds innovation
  166. 166. • System that enables communication • System that exposes secrets • System that rob us • System that funds innovation • System that kills us
  167. 167. • System that enables communication • System that exposes secrets • System that rob us • System that funds innovation • System that kills us • System that allow us to fly across the world
  168. 168. Failure is pervasive
  169. 169. Failure is complex
  170. 170. Failure is just another mode of operation
  171. 171. Your system may not control fate of people’s lives
  172. 172. But people may depend on it
  173. 173. Anthropocentrism
  174. 174. Technocentrism
  175. 175. The squishy middle ground
  176. 176. Operable Systems man + machine
  177. 177. No amoral actors
  178. 178. We need to look at it from a systems approach, a human/technology system that has to work together. This involves aircraft design and certification, training and human factors. If you look at the human factors alone, then you're missing half or two-thirds of the total system failure -- Chesley Sullenberger Pilot, US 1549, Hudson River Ditching
  179. 179. We need to look at it from a systems approach, a human/technology system that has to work together. This involves aircraft design and certification, training and human factors. If you look at the human factors alone, then you're missing half or two-thirds of the total system failure -- Chesley Sullenberger Pilot, US 1549, Hudson River Ditching
  180. 180. • Damn it, we're going to crash... This can't be happening!
  181. 181. • Final Air France 447 Report: Pilots misunderstood their situation • Poorly-trained pilots to blame for Air France crash that killed 228 • Final Air France crash report says pilots failed to react swiftly • Air France 447 downed as crew ignored alarms • Air France 447 crash a result of crew ignoring alarms
  182. 182. Thank you
  183. 183. Liked the talk? Let @auxesis know! Thank you
  184. 184. Sidney Dekker Field Guide to Understand Human Error Drift Into Failure Just Culture Jon Rauser Look at Your Data @ Velocity Ryan Tomayko Distributed Management @ Monktoberfest 2012 BEA Report on AF447 incident

×