The document discusses the importance of runbook automation in enhancing operational performance and incident management, proposing that the next major advancements will come from improving operational techniques. It emphasizes the complexity of systems and the need for human operators to effectively monitor, respond, adapt, and learn from incidents rather than relying solely on automation. Key strategies include creating self-service options and improving collaboration between development and operations teams.

1. Damon Edwards
October 13-15, 2020
Runbook Automation:
Old News or a Key to Unlock Performance?

2. Damon Edwards

3. Damon Edwards

4. Damon Edwards

7. Lean/Flow
Fast Feedback Loops
CI/CD
Small Batch Sizes
Shift Left
Cloud Platforms
Microservices
Continuous Testing
Infrastructure as Code
3 Ways / 5 Ideals
Automated Governance
SLOs
Kanban
You build it, you run it

8. Lean/Flow
Fast Feedback Loops
CI/CD
Small Batch Sizes
Shift Left
Cloud Platforms
Microservices
Continuous Testing
Infrastructure as Code
3 Ways / 5 Ideals
Automated Governance
SLOs
Kanban
You build it, you run it
What’s Next?

9. Thesis:
The Next Great Unlocks Will
Come from Ops…

10. Incident Management
Thesis:
The Next Great Unlocks Will
Come from Ops…

11. Incident Management
Service Requests
Thesis:
The Next Great Unlocks Will
Come from Ops…

12. Incident Management

13. Incident Management
What do we all want?

14. Incident Management
What do we all want?
Shorter Incidents.
Fewer Escalations.

15. Incident Management
What do we all want?
Shorter Incidents.
Fewer Escalations.
What always gets in the way?

16. Incident Management
What do we all want?
Shorter Incidents.
Fewer Escalations.
What always gets in the way?
Complexity.

17. Our world is complex,
and not deterministic.
J. Paul Reed
Our world is complex,
and not deterministic!
John Allspaw

18. Development vs Operations

19. (deterministic POV)
Development vs Operations

20. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations

21. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations

22. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)

23. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)
Network traffic
Usage patterns
Config updates
Platform changes
API performance
Library updates
Hardware variation
Tuning
Data changes

24. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)
Network traffic
Usage patterns
Config updates
Platform changes
API performance
Library updates
Hardware variation
Tuning
Tailoring
Data changes

25. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)
Network traffic
Usage patterns
Config updates
Platform changes
API performance
Library updates
Hardware variation
Tuning
Tailoring
Richard Cook, M.D.
Data changes

26. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)
Network traffic
Usage patterns
Config updates
Platform changes
API performance
Library updates
Hardware variation
Tuning
“System as imagined”
Tailoring
Richard Cook, M.D.
Data changes

27. (deterministic POV)
Code → Build → Run? ( 👍 / 👎 )
Development vs Operations
SAIL/cornell.edu
(stochastic POV)
Network traffic
Usage patterns
Config updates
Platform changes
API performance
Library updates
Hardware variation
Tuning
“System as imagined” “System as found”
Tailoring
Richard Cook, M.D.
Data changes

28. What are people doing in “systems as found”:
Richard Cook, M.D.

29. 1. Monitoring
What are people doing in “systems as found”:
Richard Cook, M.D.

30. 1. Monitoring
2. Responding (making sense of what they are seeing)
What are people doing in “systems as found”:
Richard Cook, M.D.

31. 1. Monitoring
2. Responding (making sense of what they are seeing)
3. Adapting (more tailoring of the complex system)
What are people doing in “systems as found”:
Richard Cook, M.D.

32. 1. Monitoring
2. Responding (making sense of what they are seeing)
3. Adapting (more tailoring of the complex system)
4. Learning (feedback loops and understanding)
What are people doing in “systems as found”:
Richard Cook, M.D.

33. 1. Monitoring
2. Responding (making sense of what they are seeing)
3. Adapting (more tailoring of the complex system)
4. Learning (feedback loops and understanding)
What are people doing in “systems as found”:
Automation alone
can’t do this

34. Most important lesson from other high-consequence fields:

35. Most important lesson from other high-consequence fields:
The role of automation is to support the human operator,
not to replace the human operator.

36. “We must develop trust in our operators.”
Richard Cook, M.D.

37. “We must develop trust in our operators.”
“Too much design goes into preventing
people from doing things”
Richard Cook, M.D.

38. “We must develop trust in our operators.”
“Too much design goes into preventing
people from doing things”
“We need to reveal the actual controls
that are available”
Richard Cook, M.D.

39. Abstraction

40. Abstraction
Too high: Black Box. (bad)

41. Abstraction
Too high: Black Box. (bad)
Dr. Lisanne
Bainbridgee
“The Ironies of
Automation”
1982

42. Abstraction
Too high: Black Box. (bad)
Too low: ssh, sudo, and 🙏 (bad)
Dr. Lisanne
Bainbridgee
“The Ironies of
Automation”
1982

43. Instead your
experts become
the “abstraction”

44. Instead your
experts become
the “abstraction”

46. Tickets
Tickets
Tickets

47. Ticket
Ticket
Ticket

48. Self-Service

49. Self-Service
TEAM A
TEAM B
TEAM C
TEAM D

50. An Incident Management Example…

52. 3 Options:

53. 1. Decipher the wiki3 Options:

54. 1. Decipher the wiki 2. Ad-hoc tool/script usage3 Options:

55. 1. Decipher the wiki 2. Ad-hoc tool/script usage 3. ESCALATE!!3 Options:

61. Service Requests Too…

66. Enable New Org Models…

67. I build it.
I run it.
Platform Engineering

68. What’s the magnitude of impact?…

69. (#YMMV)
What’s possible?

70. CFO
?
(#YMMV)
What’s possible?

71. CFO
?
60%
Shorter
Incidents
(#YMMV)
What’s possible?

72. CFO
?
60%
Shorter
Incidents
50%
Fewer
Escalations
(#YMMV)
What’s possible?

73. CFO
?
60%
Shorter
Incidents
50%
Fewer
Escalations
99%Faster
Turnaround
Time
(#YMMV)
What’s possible?

74. 60%
Shorter
Incidents
50%
Fewer
Escalations
99%Faster
Turnaround
Time
CFO
(#YMMV)
What’s possible?

75. 60%
Shorter
Incidents
50%
Fewer
Escalations
99%Faster
Turnaround
Time
CFO
We’ve got
Budget!
(#YMMV)
What’s possible?

76. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
Creating Self-Service

77. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
Creating Self-Service

78. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
Creating Self-Service

79. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
Do
Creating Self-Service

80. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
View Do
Creating Self-Service

81. TEAM A
TEAM B
TEAM C
TEAM D
Runbook
Automation
AIOps
Observability
View Do
Creating Self-Service

82. Self-Service Operations
The next great unlocks?
Runbook Automation
AIOps / Observability

83. Let’s Talk!
@damonedwards
damon@pagerduty.com
Damon Edwards

84. "How Complex Systems Fail” - 1998
Dr Richard Cook, MD
https://how.complexsystems.fail/
“Ironies of Automation” - 1982
Dr. Lisanne Bainbridge
https://www.ise.ncsu.edu/wp-content/uploads/
2017/02/Bainbridge_1983_Automatica.pdf
https://youtu.be/URDBE4q_IgM
The Troubles of Automating “All the Things” - 2019
J. Paul Reed
How Your Systems Keep Running - 2017
John Allspaw
https://youtu.be/xA5U85LSk0M
Slides: rundeck.com/does20

85. https://youtu.be/1zUtBLZ4Lus
Operations: The Last Mile
Damon Edwards
Slides: rundeck.com/does20
Operations:
The Last Mile
Incident Management Meets DevOps
Bhavik Gudka & Surya Avirneni
https://youtu.be/6W-HuG5a8L0