J. Paul Reed's DevOpsDays Silicon Valley 2013 presentation "Is Your Team Instrument Rated?" The presentation discusses the operational model similarities between the National Airspace System and a well-run software development shop that employs DevOps methodologies.

1. IS
YOUR
TEAM
INSTRUMENT
RATED?
(OR DEPLOYING 125,000
TIMES A DAY)
J. Paul Reed
Principal Consultant

2. J. PAUL REED
• “Sober Build Engineer”
• @SoberBuildEng
• Fifteen years as a build/release engineer

3. IF MY VOICE SEEMS FAMILIAR...
www.theshipshow.com
@ShipShowPodcast
@eciramella
@buildscientist
@cheeseplus
@sascha_d

4. IN PREPARATION FOR OUR FLIGHT...

5. “CULTURE?”
Set of shared mental assumptions that guide interpretation
and action in organizations by defining appropriate
behavior for various situations.
– Ravisi & Schultz, via Wikipedia
(via Damon’s talk)

6. “CULTURE” FOR TODAY
Incentives
+
Human Factors

7. “CULTURE” FOR TODAY
Incentives
+
Human Factors
(organizational, behavioral, and economic)
(methods for facilitating and fostering those incentives)

8. WHY AVIATION?
Craft
Provided unique requirements,
individuals perfecting their own
methods & techniques
Trade
Groups of “craftspeople” sharing
domain knowledge
Science
Processes consistently repeatable
by others, under different
environments/conditions
Industry
Reduce/combine processes to
optimize for specific business
requirements or outcomes
“Progress”

9. WHY AVIATION?

10. WHY AVIATION?
But... when we’re talking incident response,
the house is already on fire

11. WHY AVIATION?
Dev Ops

12. WHY AVIATION?
Dev Ops

13. WHY AVIATION?
Scale much?

14. VISUAL FLIGHT RULES

15. VISUAL FLIGHT RULES

16. VISUAL FLIGHT RULES

17. INSTRUMENT FLIGHT RULES

18. INSTRUMENT FLIGHT RULES

19. INSTRUMENT FLIGHT RULES

20. soberbuildengineer.com/content/dod-sv2013/
vfr-vs-ifr.mp4
The following video illustrates the contextual differences
betweenVFR and IFR flight.

21. • Standardization
• Communication
• Expectations
• Remediation
WHAT IT IS

22. STANDARDIZATION
A set of
operational
primitives based
on your
organizational
and business
requirements.

23. STANDARDIZATION
Leveraged to
define your
operational
procedures.

24. STANDARDIZATION
Leveraged to
define
operational
procedures.
Leveraged to
define your
operational
procedures.

25. • Standardization
• Communication
• Expectations
• Remediation
WHAT IT IS

26. COMMUNICATION
“We're cleared to NewYork's JFK Airport via the SAN FRANCISCO
EIGHT, radar vectors to Linden, direct JSICA, direct Wilson Creek, Jet
80, Kansas City, Jet 24, Saint Louis, direct Brickyard, direct Rosewood,
Jet 29, Jamestown, Jet 70,Wilkes Barre, to the LENDY FIVE arrival
into JFK; climb and maintain fifteen, one-five-thousand; expect three-
five-zero in ten; squawk six-three-seven-seven.”
– Redwood Flight 34’s Inaugural Clearance

27. COMMUNICATION
SFO SFO8
LIN JSICA ILC J80 MCI
J24 STLVHP ROD
J29 JMS
J70 LVZ LENDY5
– Redwood Flight 34’s Inaugural Clearance

28. COMMUNICATION
SFO SFO8
LIN JSICA ILC J80 MCI
J24 STLVHP ROD
J29 JMS
J70 LVZ LENDY5
– Redwood Flight 34’s Inaugural Clearance

29. COMMUNICATION
In other words, trying to avoid this...
soberbuildengineer.com/content/dod-sv2013/
hold-vs-lineup.mp4

30. • Hesitance to use appropriate terms to communicate the
situation
COMMUNICATION

31. • Hesitance to use appropriate terms to communicate the
situation
• A transcontinental 707 arrives to bad weather & low on
fuel; pilots do not use the single phrase necessary to indicate
their situation, which would’ve activated emergency services
(Avianca Flight 52)
COMMUNICATION

32. • Hesitance to use appropriate terms to communicate the
situation
• A transcontinental 707 arrives to bad weather & low on
fuel; pilots do not use the single phrase necessary to indicate
their situation, which would’ve activated emergency services
(Avianca Flight 52)
• Misuse of defined terminology
COMMUNICATION

33. • Hesitance to use appropriate terms to communicate the
situation
• A transcontinental 707 arrives to bad weather & low on
fuel; pilots do not use the single phrase necessary to indicate
their situation, which would’ve activated emergency services
(Avianca Flight 52)
• Misuse of defined terminology
• In 1995, a controller clears a 757 “directly” to the airport,
setting off an accident chain (American Airlines Flight 965)
COMMUNICATION

34. • Standardization
• Communication
• Expectations
• Remediation
WHAT IT IS

35. EXPECTATIONS
Once standards are
established and
requirements/
intentions
communicated,
expectations and
responsibilities can
be derived.

36. • Standardization
• Communication
• Expectations
• Remediation
WHAT IT IS

37. WHAT IT IS
With expectations and
responsibilities clarified,
remediation processes can
be integrated into processes
& automation, not tacked
on or “invented on the fly.”

38. • Static
• Blind reliance on automation, tooling, or process
• “Fun-Verboten”
WHAT IT IS NOT

39. NOT STATIC

40. NOT STATIC
A perfect example of the
dynamic nature of aviation...
soberbuildengineer.com/content/dod-sv2013/
FedEx-KMEM-wx-diversions.mp4

41. • Static
• Blind reliance on automation, tooling, or process
• “Fun-Verboten”
WHAT IT IS NOT

42. NOT AUTOMATION?!

43. NOT AUTOMATION?!

44. • Misreading/looking at the wrong metrics
NOT AUTOMATION?!

45. • Misreading/looking at the wrong metrics
• A 737 suffers an engine “disturbance”; after not looking at all
the appropriate instruments, pilots shut down the good
engine; the remaining (bad) engine eventually fails fully
(British Midlands Flight 92)
NOT AUTOMATION?!

46. • Misreading/looking at the wrong metrics
• A 737 suffers an engine “disturbance”; after not looking at all
the appropriate instruments, pilots shut down the good
engine; the remaining (bad) engine eventually fails fully
(British Midlands Flight 92)
• Partial automation failure and resulting confusion
NOT AUTOMATION?!

47. • Misreading/looking at the wrong metrics
• A 737 suffers an engine “disturbance”; after not looking at all
the appropriate instruments, pilots shut down the good
engine; the remaining (bad) engine eventually fails fully
(British Midlands Flight 92)
• Partial automation failure and resulting confusion
• After a series of instrument failures in the highly-automated
A-330, the junior pilot pulls the plane into a prolonged stall
(Air France 447)
NOT AUTOMATION?!

48. NOT AUTOMATION?!

49. • Static
• Blind reliance on automation, tooling, or process
• “Fun-Verboten”
WHAT IT IS NOT

50. NOT “FUN VERBOTEN”

51. NOT “FUN VERBOTEN”

52. • Define organizational vocabulary & process primitives
• Formalize roles, responsibilities, and priorities
• Understand (current) limitations
• Investigate outcomes
GETTING “INSTRUMENT RATED”

53. DEFINE YOUR APPROACHES
• Define what you do today,
focusing on the
“operational requirements”
• Derive (or define)
primitives
• Define your operational
dictionary
• Make sure they’re owned!

54. • Define organizational vocabulary & process primitives
• Formalize roles, responsibilities, and priorities
• Understand (current) limitations
• Investigate outcomes
GETTING RATED

55. FORMALIZE “2R+P”
• Be able to answer “Who is responsible for that?”
• Drill/train or delegate
• Determine “priority classes”

56. • Define organizational vocabulary & process primitives
• Formalize roles, responsibilities, and priorities
• Understand (current) limitations
• Investigate outcomes
GETTING RATED

57. LINE UP AND WAIT
For organizational change to even be a possibility,
the current limitations need to be internalized.

58. KNOW WHENTO HOLD ‘EM

59. • Define organizational vocabulary & process primitives
• Formalize roles & responsibilities
• Understand (current) limitations
• Investigate outcomes
GETTING RATED

60. “OOPS” WILL HAPPEN
We’ve all had days like this...
soberbuildengineer.com/content/dod-sv2013/
a380-vs-rj.mp4

61. • NASA Aviation Safety Reporting System
AFTER THE “OOPS”

62. • NASA Aviation Safety Reporting System
• Separation of investigation roles
• NationalTransportation Safety Board
AFTER THE “OOPS”

63. • NASA Aviation Safety Reporting System
• Separation of investigation roles
• NationalTransportation Safety Board
• “No Blame” postmortems
• (Though not for the reason you might think!)
AFTER THE “OOPS”

64. OPERATIONAL MODELS
Incentives
+
Human Factors

65. J. Paul Reed
www.soberbuildengineer.com
@SoberBuildEng
www.releng-approaches.com
Simply Ship. EveryTime.