Tickets Make Operations Work Unnecessarily Miserable
•
2 likes•963 views
Presentation by Damon Edwards, co-founder of Rundeck, at Interop ITX 2019 Las Vegas Ticket-driven request queues have become the default way of working in operations for a long time and are the cornerstone of most operations management and ITSM strategies. But what if ticket queues are actually the source of much of the dysfunction, bottlenecks, and capacity issues that have traditionally plagued our organizations? This session will examine the dark side of ticket queues, including hidden costs and how they undermine DevOps and SRE transformations. Then we’ll explore alternative strategies high-performing operations organizations use to minimize dependence on tickets queues.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tickets Make Operations Work Unnecessarily Miserable
Similar to Tickets Make Operations Work Unnecessarily Miserable (20)
More from Rundeck
More from Rundeck (20)
Recently uploaded
Recently uploaded (20)
Tickets Make Operations Work Unnecessarily Miserable
- 1. Tickets Make Operations Unnecessarily Miserable Damon Edwards @damonedwards
- 2. Community Ops Improvement DevOps in Enterprise Ops Tools Damon Edwards
- 3. Please note:
- 4. Please note: No, I’m not against writing things down (nor do I advocate for anarchy)
- 5. Please note: No, I’m not against writing things down (nor do I advocate for anarchy)
- 6. Please note: No, I’m not against writing things down (nor do I advocate for anarchy) Tickets Queues aren’t the only villains (there are many unindicted co-conspirators)
- 7. Please note: No, I’m not against writing things down (nor do I advocate for anarchy) Tickets Queues aren’t the only villains (there are many unindicted co-conspirators)
- 8. Developers have had an unfair advantage.
- 12. OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" “Open up!” “Lock it down!” 2019
- 13. Story time….
- 14. Digital Agile DevOps SRE Cloud Docker Kubernetes Microservices CHANGE Wow That is cool I wish I could work there
- 15. But nobody was talking about what happened after deployment…
- 16. It was just another Tuesday…
- 17. NOC NOC Biz Manager Escalate! NOC NOC NOC (Bob) Open Incident Ticket 9:30am 10:00am NOC (Bob) Biz Manager Ticket Context Wagon Yes, but this looks different Hasn’t there been some intermittent errors this week? v3 ?!
- 18. NOC (Bob) Open Incident Ticket Ticket Biz Manager App-specific SREs “Try this.” “Try that.” SRE SysAdmin with Prod Access (Steve) SRE SRE SRE SRE SRE SRE Bridge Call Biz Manager fixed? fixed? NOC (Bob) Biz Manager NOC (Bob) Biz Manager SysAdmin (Steve) 7 x SRE Ticket Context Wagon Ticket Context Wagon
- 19. SRE “It’s a problem with the Foo service” SRE SRE Foo SRE SRE SRE SRE Bridge Call Biz Manager Foo Service No. NOC (Bob) Update Ticket Ticket Foo Lead Dev + add 12:00pm NOC (Bob) Biz Manager Foo SRE Ticket Context Wagon Can you fix it?
- 20. o Dev Foo Lead Dev (Karen) ding! Ignore. App Manager Hey did you see that ticket? Foo Lead Dev (Karen) sigh. I’ll take a look I’m go mor pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE Scrum Ticket Context Wagon
- 21. k Foo Lead Dev (Karen) I’m going to need more log files Ticket SysAdmin Team + add Update Ticket Chat “Can someone with access to Foo Service in Prod01 help me with ticket #42516?” SysAdmin (Lee) Ticket “logs attached” Foo Lead Dev (Karen) Ticket “no the other ones” Le (K NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Ticket Context Wagon
- 22. Foo Lead Dev (Karen) Logs -Who restarted these services? (and why?) -They didn’t use the correct environment variables! -This entire service pool needs to be restarted! Ticket Update Ticket NOC (Bob) Update Ticket Ticket Middleware Team + add “Middleware, please urgent restart this entire app pool with the correct environment variable” 2:00pm Ticket Context W
- 23. ase s entire e correct able” NOC (Bob) Middleware Manager (Melissa) No way. It’s the middle of the day! You need business approval. NOC (Bob) Update Ticket Ticket SVP for Line of Business + add (S NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager NOC (B Biz Ma App Ma Lead D Foo SR Ticket Context Wagon Ticket Context Wagon 2:30pm
- 24. Update Ticket Ticket SVP for Line of Business + add SVP (Susan) Chief of Staff Tech VP Tech VP Update Ticket Ticket “Restart approved” Customer impact? Ticket Middlewa Manage (Melissa Wh prod 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Ticket Context Wagon
- 25. Share point proved” Ticket Middleware Manager (Melissa) Who knows these production services the best? Ellen! Middleware Middleware (Scott) Ellen to Europe office Middleware (Scott) Trial and error .doc 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Ticket Context Wagon
- 26. Share point Middleware (Scott) Trial and error .doc NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) ket Context Wagon Middleware (Scott) Bar Service 10 min Middleware (Scott) Waiting for Acme Service Acme startup failed Bar Service 6:00pm
- 27. Come on.. no.no.no. What? Why? Middleware (Scott)
- 28. Come on.. no.no.no. What? Why? Middleware (Scott)
- 29. 8888888 Come on.. no.no.no. What? Why? Middleware (Scott)
- 30. -Bar app startup timed out. Error says can’t connect to Acme service. - I looked at Acme but it seems to be running -Is this error message correct? Why can’t Bar connect? Ticket Update Ticket Middleware (Scott) Bar SRE + add Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add 6:45 NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda)Ticket Context Wagon The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked.
- 31. Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add Bar Lead Dev 6:45pm ob) ager nager ev (Karen) E SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Customers are calling. What is going on?The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked. Bar Lead Dev (Liu) Business Managers I can comment out the test… But the CD pipeline only goes to QA ENV!
- 32. Network Dir (Carlos) Middleware (Scott) Carlos, I need a favor. Can you escalate?Middleware Manager (Melissa) Customers are calling. What is going on? Last week.. Net SRE VP VP Priority! Different Incident! Net SRE Net SRE Net SRE Its the network! Business Managers Your network is broken! Business Managers We are already working on it! Network VPs out he ly V!
- 33. Network SRE (Hari) The firewall is blocking the traffic You’ll have to take it up with the Firewall Team -URGENT: Firewall is blocking connection between Bar and Acme Ticket Open Firewall Ticket Firewall Team + add Firewall Engineer (Freddie) Middleware (Scott) Paging on-call… Open bridge… Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. 8:00p NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Ticket Context Wagon
- 34. Firewall Engineer (Freddie) Middleware (Scott) Firewall Engineer (Freddie) Middleware (Scott) Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. There was a rule change last Thursday that would stop Bar from talking to Acme. Can you change it back? Sure we make changes on Thursday… Chief of Staff SVP and VPs are livid… this was supposed to be a safe change!! Freddie, we’ve got customers calling. ES Em pro rul Update Firewall Ticket Firewall Engineer (Freddie) 8:00pm
- 35. d VPs are livid… this was sed to be a safe change!! we’ve got customers calling. ESCALATE: Emergency production firewall rule change review Ticket Update Firewall Ticket NetSec + add Firewall Engineer (Freddie) Paging on-call… NetSec (Nicole) This is production so I’ll have to get others on the Network CAB… Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev 9:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAd Middle SVP Chief o 2 x Tec Ticket Context Wagon
- 36. I’ll have Network Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! APPROVE: Emergency firewall rule change Ticket Update Firewall Ticket NetSec (Nicole) … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev Firewall (Freddie) Net L2 (Bob) Middl (Sc Firewall change Restart Bar 9:30pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole)
- 37. Middleware (Scott) Update Ticket Ticket Customer Engagement Manager + add Policy !! “Ready for API tests” 9:45pm Firewall (Freddie) Net L2 (Bob) Middleware (Scott) Firewall change Restart Bar I think we are good! Middleware Manager VP VP Bar Lead Dev You “think?” pm
- 38. et gement “Ready for API tests” Customer Engagement Manager (Varsha) NOC (Bob) Customer Engagement Manager (Varsha) Update Ticket Ticket “APIs OK” Middleware (Scott) Upda Tick 11:00pm Ticket Co
- 39. e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha)
- 40. e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha) .
- 41. NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz Next Day SVP (Susan) Whose fault is this?! Why are we so bad at change? What additional processes and approvals are you adding to never let this happen again?! VP VP Dir Dir VP Dir VP Scott) da) Carlos) (Bob) die) NetSec (Nicole) Cust. Engmt. (Varsha)
- 42. Later…
- 43. We’ve invested in Cloud, Agile, DevOps, Containers… Why does everything still take too long and cost too much? Executive Team Our transformation has largely ignored Ops
- 44. Most companies chase the symptoms…
- 45. …by following the conventional wisdom:
- 46. “We need better tools” …by following the conventional wisdom:
- 47. “We need better tools” “We need more people” …by following the conventional wisdom:
- 48. “We need better tools” “We need more people” “We need more discipline and attention to detail” …by following the conventional wisdom:
- 49. “We need better tools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals” …by following the conventional wisdom:
- 50. “We need better tools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals” …by following the conventional wisdom:
- 51. Challenge the conventional wisdom about operations work
- 52. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 53. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 54. Where are decisions made? Who can take action? escalate 1° 2° 3° 4° escalate escalateor
- 55. Where are decisions made? Who can take action? escalate 1° 2° 3° 4° escalate escalateor Decisions made here
- 56. All work is contextual John Allspaw
- 57. All work is contextual rm -rf $PATHNAME John Allspaw
- 58. All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw
- 59. All work is contextual rm -rf $PATHNAME John Allspaw
- 60. All work is contextual rm -rf $PATHNAME John Allspaw
- 61. All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw
- 62. All work is contextual rm -rf $PATHNAME John Allspaw
- 63. All work is contextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw
- 64. escalate 1° 2° 3° 4° escalate escalateor Context Where are decisions made? Who can take action?
- 65. Psychological Safety Psychological safety is a shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990
- 66. Psychological Safety Psychological safety is a shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990 Google: most important characteristic to predict team effectiveness? 2016
- 67. Psychological Safety Psychological safety is a shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990 Google: most important characteristic to predict team effectiveness? 2016 Psychological safety!
- 68. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 69. Toil: Name For a Problem We’ve All Felt
- 70. Toil: Name For a Problem We’ve All Felt “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
- 71. Toil vs. Engineering Work Toil Engineering Work Lacks Enduring Value Builds Enduring Value Rote, Repetitive Creative, Iterative Tactical Strategic Increases With Scale Enables Scaling Can Be Automated Requires Human Creativity
- 72. Excessive Toil Prevents Fixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
- 73. Excessive Toil Prevents Fixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
- 74. Excessive Toil Prevents Fixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Downward spiral is inevitable!
- 75. Forces That Undermine Operations Silos Queues Excessive ToilLow Trust
- 77. Backlog Information I need X PrioritiesTools Silos
- 78. Backlog Information I need X PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
- 79. Silos cause disconnects and mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
- 80. 1 2 3 Silos Interfere with feedback loops
- 81. 1 2 3 Silos Interfere with feedback loops Producer Consumer Ops Ops Ops
- 82. Function A Function B Function C Silos create labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
- 83. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 84. How do we cover for our silos’ disconnects and mismatches? Silo A Silo B
- 85. How do we cover for our silos’ disconnects and mismatches? Silo A Silo B Ticket Queue
- 86. ?? Silo A Silo B We all know how well that works Ticket Queue
- 87. Ticket queues are an expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
- 88. What do queues do to value streams?
- 89. What do queues do to value streams? Queue A Queue B
- 90. What do queues do to value streams? Queue A Queue B Queues disintegrate and obfuscate value streams
- 91. Ticket queues are “snowflake makers” ?? Silo A Silo B Ticket Queue
- 92. Ticket queues are “snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes Technically acceptable, but brittle and unreproducible
- 93. Ticket queues are “snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes Technically acceptable, but brittle and unreproducible
- 94. "Queues don’t learn” ?? Silo A Silo B Ticket Queue Scott Prugh CSGi
- 95. "Queues don’t learn” ?? Silo A Silo B Ticket Queue Scott Prugh CSGi
- 96. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 97. So…
- 98. What Can We Do Differently?
- 99. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 100. “Shift Left” the ability to take action escalate 1° 2° 3° 4° escalate escalateor
- 101. “Shift Left” the ability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
- 102. “Shift Left” the ability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor Tools Enablement and tooling
- 103. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 104. Start reducing toil today Toil
- 105. Start reducing toil today 1. Track toil levels for each team Toil
- 106. Start reducing toil today 1. Track toil levels for each team Toil
- 107. Track toil levels for each team
- 108. Track toil levels for each team • Standardize (e.g. meetings and email are “overhead" not “toil”)
- 109. Track toil levels for each team • Standardize (e.g. meetings and email are “overhead" not “toil”) • Track • Self-reporting • Periodic surveys • SM or PM interview/sampling
- 110. Track toil levels for each team • Standardize (e.g. meetings and email are “overhead" not “toil”) • Track • Self-reporting • Periodic surveys • SM or PM interview/sampling • Don’t get lost in time tracking weeds!
- 111. Start reducing toil today 1. Track toil levels for each team Toil
- 112. Start reducing toil today 1. Track toil levels for each team Toil 2. Set toil limit for each team (50% is conventional wisdom)
- 113. Start reducing toil today 1. Track toil levels for each team 2. Set toil limit for each team (50% is conventional wisdom) 3. Fund efforts to reduce toil (with emphasis on teams already over limit) Toil
- 114. Start reducing toil today 1. Track toil levels for each team 2. Set toil limit for each team (50% is conventional wisdom) 3. Fund efforts to reduce toil (with emphasis on teams already over limit) Toil Michael Kehoe Todd Palino (LinkedIn) At SREcon Americas 2019 Example Process “Code Yellow”
- 115. Where to focus? Toil
- 116. Where to focus? Toil Reduce Technical Debt
- 117. Where to focus? Toil Reduce Technical Debt Re-Engineer Processes
- 118. Where to focus? Toil Reduce Technical Debt Re-Engineer Processes Enable Self-Service
- 119. Where to focus? Toil Reduce Technical Debt Re-Engineer Processes Enable Self-Service
- 123. Eliminate Interruptions Eliminate Waiting Self-Service Do X. … and a lot less toil
- 124. How to enable self-service? Empower teams to spot and fix the anti-patterns.
- 125. “Do this for me, do it again, then do it again.” Done.I need you to do X Your other work I need you to do X I need you to do X Ticket Do X Later… Do X Do X Done. Done. Your other work Self-Service Self-Service Self-Service Your other work x2 Your other work x3 Later…Later… Later… Your other work Your other work After Before Wait Interrupt Ticket Wait Interrupt Ticket Wait Interrupt
- 126. “Do this for me, do it again, then do it again.” Done.I need you to do X Your other work I need you to do X I need you to do X Ticket Do X Later… Do X Do X Done. Done. Your other work Self-Service Self-Service Self-Service Your other work x2 Your other work x3 Later…Later… Later… Your other work Your other work After Before Wait Interrupt Ticket Wait Interrupt Ticket Wait Interrupt
- 127. “I could fix it, but I can’t get to it.” Environment I could fix it if I could get to it Before Wait Interrupt
- 128. “I could fix it, but I can’t get to it.” Environment I could fix it if I could get to it Before Wait Interrupt After I’ve got this! Environment Self- Service
- 129. “What’s the BEST way to do this?!” Environment 3:00 AM sar -n Dev,EDEV 1 sar -n TCP,ETCP 1 cat etc/resolv.conf mpstat -P ALL 1 tcpretrans tcpconnect tcpaccept netstat -rnv check firewall config netstat -s Check Network Earlier… After Self-Service
- 130. “Oh no… I’m the Brent.” Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Before
- 131. “Oh no… I’m the Brent.” Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Before
- 132. “The dog-pile.” !! I think its a problem with db07-store2.uswest.acme “$ top” “$ top” db07store2. uswest.acme “$ top” “$ top” “$ top” !! “$ top” !! !! !! healthcheck store2 -all db07store2. uswest.acme Self-Service 1. 2. 3. I think its a problem with db07-store2.uswest.acme
- 133. “I’m an expert, I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before
- 134. “I’m an expert, I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before
- 135. “I’m an expert, I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart” Environment Later… Update Restart Job ✅ I’ve done this before. I’ve got this. Self-Service Self-Service After
- 136. “You have a workaround, deal with it.”
- 137. “You have a workaround, deal with it.”
- 138. Self-Service Operations Design Pattern (in a nutshell) Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 139. Self-Service Operations Design Pattern (in a nutshell) Pull-Based Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 140. Self-Service Operations Design Pattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 141. Self-Service Operations Design Pattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 142. Self-Service Operations Design Pattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Let people who “push buttons” define the buttons Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 143. Self-Service Operations Design Pattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Let people who “push buttons” define the buttons Build in security and compliance Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 144. Self-Service is ultimately about user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 145. Self-Service is ultimately about user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI)
- 146. Self-Service is ultimately about user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI) 2. “Guardrails” (Smart options that helpfully constrain)
- 147. Self-Service is ultimately about user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI) 2. “Guardrails” (Smart options that helpfully constrain) 3.Dynamic resource model (Up-to-date details of your environment)
- 148. Forces That Undermine Operations Silos Ticket Queues Excessive ToilLow Trust
- 149. Obvious: Get rid of as many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
- 150. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible
- 151. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible “Horizontal” shared responsibility, not everyone do everything!
- 152. Shared and dedicated responsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model
- 153. Shared and dedicated responsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model
- 154. Shared and dedicated responsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model Same high-quality, high-velocity results!
- 155. But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
- 156. But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
- 157. But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
- 158. Forces That Undermine Operations Silos Queues Excessive ToilLow Trust
- 159. Self-Service Operations: Turn handoffs into self-service Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Cross-Functional Product Team 2 Ops (embedded)
- 160. Self-Service Operations: Works with any org model Development Team 1 Development Team 2 Development Team n Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
- 161. Development Team 1 Development Team 2 Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Cross-Functional Product Team n Ops (embedded) But, what about security and compliance? Build-in Security Here Build-in Compliance Here
- 162. Are all tickets bad?
- 163. Are all tickets bad? Ticket System No. Just use tickets for what they are good for
- 164. Are all tickets bad? 1.Documenting true problems/issues/exceptionsTicket System No. Just use tickets for what they are good for
- 165. Are all tickets bad? 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System No. Just use tickets for what they are good for
- 166. Are all tickets bad? 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System No. Just use tickets for what they are good for
- 167. Self-Service can also be a foundation for strategic initiatives
- 168. Strategic: Improve incident response times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF
- 169. Strategic: Improve incident response times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower
- 170. Strategic: Improve incident response times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower
- 171. Strategic: Improve incident response times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower • Reduced MTTR by 92% • Reduced escalations by 50% • Reduced overall support costs by 55%
- 172. Strategic: Reduce compliance burden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg
- 173. Strategic: Reduce compliance burden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated
- 174. Strategic: Reduce compliance burden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated LOB #1 LOB #2 LOB #3 LOB …n Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Self-Service ComplianceConsistency
- 175. Strategic: Reduce compliance burden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated LOB #1 LOB #2 LOB #3 LOB …n Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Self-Service ComplianceConsistency 12 months: • Saved 28 person years of time • 13,000+ ops tasks in privileged environments that didn’t require a review • ~200 less customer impacting events
- 176. rundeck.com/self-service Read for free online: Working on documenting the Self- Service Operations design pattern. Where I need your help…
- 177. Recap Don’t forget about Ops. Challenge conventional wisdom. Use self-service to reduce remaining ticket queues “Shift-Left” control and decision making. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silosLearn from SRE: Focus on understanding & reducing toil Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil Toil at manageable percentage of capacity oil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Understand the forces undermining operations work Development Team 1 Development Team 2 Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Cross-Functional Product Team n Ops (embedded)