AA261: DevOps lessons in collaborative maintenance
•
2 likes•1,163 views
On January 31, 2000, Alaska Airlines Flight 261 plunged into the Pacific ocean in an extreme "nose down" position, killing all 88 crew and passengers on board. The NTSB concluded AA261's horizontal stabiliser trim system's jackscrew was inadequately maintained, causing the pilots to lose all control of the plane. There are striking parallels with the problems we face daily in IT operations & software development, and the 30 years of give and take between the aircraft manufacturer's engineers, airline maintenance staff, and federal regulators that preceded AA261's simple mechanical failure. In this talk, Lindsay looks at the complex interplay between the parties in the AA261 crash through a DevOps lens, investigating the collaborative approach to maintenance and operation of the MD-83 aircraft, and relating the complexities back to the complex IT systems we build and maintain.
Recommended
Recommended
More Related Content
Similar to AA261: DevOps lessons in collaborative maintenance
Similar to AA261: DevOps lessons in collaborative maintenance (11)
More from Lindsay Holmwood
More from Lindsay Holmwood (12)
Recently uploaded
Recently uploaded (20)
AA261: DevOps lessons in collaborative maintenance
- 1. AA261 DevOps lessons in collaborative maintenance
- 2. Lindsay Holmwood @auxesis
- 3. Software Manager @ Bulletproof Networks
- 6. January 31, 2000 Puerto Vallarta
- 7. Seattle
- 8. Departed PVR at 13.37 PST
- 10. 2 hours into flight: Jammed horizontal stabiliser
- 11. No trim control
- 17. 2 pilots 3 crew 83 passengers
- 22. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
- 24. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
- 25. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
- 26. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
- 28. “poorly conceived and woefully executed”
- 32. DC-9 -> MD-80 -> MD-83
- 38. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 39. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 40. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 41. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 42. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 43. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 44. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
- 45. Decrementalism
- 46. Complex system constraints Jens Rasmussen
- 48. wo rkl oad
- 49. wo rkl oad economy
- 50. wo rkl oad economy saf ety
- 52. tim e
- 53. e tim cost
- 54. ty ali qu cost e tim
- 55. wo rkl oad economy saf ety
- 56. wo rkl oad economy saf ety
- 57. wo rkl oad economy saf ety
- 58. wo rkl oad economy saf ety
- 59. wo rkl oad economy saf ety
- 60. wo rkl oad economy saf ety
- 61. wo rkl oad economy saf ety
- 62. wo rkl oad economy saf ety
- 63. wo rkl oad economy saf ety
- 64. outside: failure of foresight oad saf rkl ety wo economy
- 65. outside: failure of foresight oad saf rkl inside: ety trade-offs wo in direction of greater efficiency economy
- 66. trade-offs in direction of greater efficiency
- 67. trade-offs in direction of greater efficiency
- 69. Why would they make bad decisions intentionally?
- 72. “people make what they consider to be the best decision based on available knowledge at the time”
- 73. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
- 74. wo rkl oad economy saf ety
- 75. ty ali qu cost e tim
- 78. “God, our ops team are arseholes. I just want to deploy this change and go home!”
- 79. “God, our ops team are arseholes. I just want to deploy this change and go home!” oad saf rkl ety wo economy
- 80. “God, our ops team are arseholes. I just want to deploy this change and go home!” oad oad saf saf rkl rkl ety ety wo wo economy economy
- 81. What are the circumstances?
- 82. Where are the tensions?
- 83. Have ops been burnt before?
- 84. Is there deployment friction? Why?
- 86. Is deployment time consuming?
- 87. Is deployment important to the business?
- 89. “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?”
- 90. “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?” oad saf rkl ety wo economy
- 91. “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?” oad oad saf saf rkl rkl ety ety wo wo economy economy
- 92. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
- 93. What are the circumstances?
- 94. Where are the tensions?
- 95. Why didn’t the dev know the code would fail like this?
- 96. Why weren’t you involved when the code was written?
- 97. How is code reviewed?
- 98. Is the infrastructure anti-fragile?
- 99. Is the code anti-fragile?
- 101. Hindsight bias
- 102. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
- 104. What are the motivations?
- 105. “amoral actors”
- 106. wo rkl oad economy saf ety
- 107. wo rkl oad economy saf ety
- 108. “root cause” is simply the point you stop looking -- Sidney Dekker
- 109. What are the circumstances?
- 110. Where are the tensions?
- 111. Thank you!
- 112. Thank you! Liked the talk? Let @auxesis know!
- 113. Sidney Dekker [books] Field Guide to Understand Human Error Drift Into Failure Just Culture Dan Manges [blog] How incidents affect infrastructure priorities