Downloaded 22 times
Uploaded byRundeck
Tickets Make Operations Work Unnecessarily Miserable
The document discusses the inefficiencies and frustrations caused by ticketing systems in operations, revealing how they contribute to miscommunication and delays in problem resolution. It emphasizes the need to challenge conventional wisdom around operations management, which often leads to excessive bureaucracy and toil, hindering productivity and innovation. The document also highlights the importance of psychological safety in teams to improve decision-making and responsiveness.
More Related Content
Similar to Tickets Make Operations Work Unnecessarily Miserable
![Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]](https://cdn.slidesharecdn.com/ss_thumbnails/does-damonedwards-v2-201015073337-thumbnail.jpg?width=640&height=640&fit=bounds)
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Tickets Make Operations Work Unnecessarily Miserable
- 1. Tickets Make Operations UnnecessarilyMiserable Damon Edwards @damonedwards
- 2. Community Ops Improvement DevOps inEnterprise Ops Tools Damon Edwards
- 3.
- 4. Please note: No, I’mnot against writing things down (nor do I advocate for anarchy)
- 5. Please note: No, I’mnot against writing things down (nor do I advocate for anarchy)
- 6. Please note: No, I’mnot against writing things down (nor do I advocate for anarchy) Tickets Queues aren’t the only villains (there are many unindicted co-conspirators)
- 7. Please note: No, I’mnot against writing things down (nor do I advocate for anarchy) Tickets Queues aren’t the only villains (there are many unindicted co-conspirators)
- 8. Developers have hadan unfair advantage.
- 9.
- 10.
- 11.
- 12. OpsBusiness Idea Shorter Time-to-Market Fast Feedback fromUsers Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" “Open up!” “Lock it down!” 2019
- 13.
- 14. Digital Agile DevOps SRE Cloud Docker Kubernetes Microservices CHANGE Wow That is cool Iwish I could work there
- 15. But nobody wastalking about what happened after deployment…
- 16. It was justanother Tuesday…
- 17. NOC NOC Biz Manager Escalate! NOC NOC NOC (Bob) Open Incident Ticket 9:30am 10:00am NOC(Bob) Biz Manager Ticket Context Wagon Yes, but this looks different Hasn’t there been some intermittent errors this week? v3 ?!
- 18. NOC (Bob) Open Incident Ticket Ticket Biz Manager App-specific SREs “Try this.” “Try that.” SRE SysAdmin withProd Access (Steve) SRE SRE SRE SRE SRE SRE Bridge Call Biz Manager fixed? fixed? NOC (Bob) Biz Manager NOC (Bob) Biz Manager SysAdmin (Steve) 7 x SRE Ticket Context Wagon Ticket Context Wagon
- 19. SRE “It’s a problem withthe Foo service” SRE SRE Foo SRE SRE SRE SRE Bridge Call Biz Manager Foo Service No. NOC (Bob) Update Ticket Ticket Foo Lead Dev + add 12:00pm NOC (Bob) Biz Manager Foo SRE Ticket Context Wagon Can you fix it?
- 20. o Dev Foo Lead Dev (Karen) ding! Ignore. App Manager Hey didyou see that ticket? Foo Lead Dev (Karen) sigh. I’ll take a look I’m go mor pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE Scrum Ticket Context Wagon
- 21. k Foo Lead Dev (Karen) I’m goingto need more log files Ticket SysAdmin Team + add Update Ticket Chat “Can someone with access to Foo Service in Prod01 help me with ticket #42516?” SysAdmin (Lee) Ticket “logs attached” Foo Lead Dev (Karen) Ticket “no the other ones” Le (K NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Ticket Context Wagon
- 22. Foo Lead Dev (Karen) Logs -Who restartedthese services? (and why?) -They didn’t use the correct environment variables! -This entire service pool needs to be restarted! Ticket Update Ticket NOC (Bob) Update Ticket Ticket Middleware Team + add “Middleware, please urgent restart this entire app pool with the correct environment variable” 2:00pm Ticket Context W
- 23. ase s entire e correct able” NOC (Bob) Middleware Manager (Melissa) Noway. It’s the middle of the day! You need business approval. NOC (Bob) Update Ticket Ticket SVP for Line of Business + add (S NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager NOC (B Biz Ma App Ma Lead D Foo SR Ticket Context Wagon Ticket Context Wagon 2:30pm
- 24. Update Ticket Ticket SVP for Lineof Business + add SVP (Susan) Chief of Staff Tech VP Tech VP Update Ticket Ticket “Restart approved” Customer impact? Ticket Middlewa Manage (Melissa Wh prod 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Ticket Context Wagon
- 25. Share point proved” Ticket Middleware Manager (Melissa) Who knows these productionservices the best? Ellen! Middleware Middleware (Scott) Ellen to Europe office Middleware (Scott) Trial and error .doc 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Ticket Context Wagon
- 26. Share point Middleware (Scott) Trial and error .doc NOC(Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) ket Context Wagon Middleware (Scott) Bar Service 10 min Middleware (Scott) Waiting for Acme Service Acme startup failed Bar Service 6:00pm
- 27. Come on.. no.no.no. What?Why? Middleware (Scott)
- 28. Come on.. no.no.no. What?Why? Middleware (Scott)
- 29. 8888888 Come on.. no.no.no. What?Why? Middleware (Scott)
- 30. -Bar app startuptimed out. Error says can’t connect to Acme service. - I looked at Acme but it seems to be running -Is this error message correct? Why can’t Bar connect? Ticket Update Ticket Middleware (Scott) Bar SRE + add Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add 6:45 NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda)Ticket Context Wagon The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked.
- 31. Bar SRE (Linda) Middleware (Scott) -URGENT: Network connectionissue between Bar and Acme Ticket Update Ticket Network SRE Team + add Bar Lead Dev 6:45pm ob) ager nager ev (Karen) E SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Customers are calling. What is going on?The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked. Bar Lead Dev (Liu) Business Managers I can comment out the test… But the CD pipeline only goes to QA ENV!
- 32. Network Dir (Carlos) Middleware (Scott) Carlos, Ineed a favor. Can you escalate?Middleware Manager (Melissa) Customers are calling. What is going on? Last week.. Net SRE VP VP Priority! Different Incident! Net SRE Net SRE Net SRE Its the network! Business Managers Your network is broken! Business Managers We are already working on it! Network VPs out he ly V!
- 33. Network SRE (Hari) The firewall is blockingthe traffic You’ll have to take it up with the Firewall Team -URGENT: Firewall is blocking connection between Bar and Acme Ticket Open Firewall Ticket Firewall Team + add Firewall Engineer (Freddie) Middleware (Scott) Paging on-call… Open bridge… Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. 8:00p NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Ticket Context Wagon
- 34. Firewall Engineer (Freddie) Middleware (Scott) Firewall Engineer (Freddie) Middleware (Scott) Can’tbe the firewall, it hasn’t changed since last Thursday. No its the firewall. There was a rule change last Thursday that would stop Bar from talking to Acme. Can you change it back? Sure we make changes on Thursday… Chief of Staff SVP and VPs are livid… this was supposed to be a safe change!! Freddie, we’ve got customers calling. ES Em pro rul Update Firewall Ticket Firewall Engineer (Freddie) 8:00pm
- 35. d VPs arelivid… this was sed to be a safe change!! we’ve got customers calling. ESCALATE: Emergency production firewall rule change review Ticket Update Firewall Ticket NetSec + add Firewall Engineer (Freddie) Paging on-call… NetSec (Nicole) This is production so I’ll have to get others on the Network CAB… Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev 9:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAd Middle SVP Chief o 2 x Tec Ticket Context Wagon
- 36. I’ll have Network Chief of Staff Firewall (Freddie) Middleware (Scott) Customeroutage! APPROVE: Emergency firewall rule change Ticket Update Firewall Ticket NetSec (Nicole) … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev Firewall (Freddie) Net L2 (Bob) Middl (Sc Firewall change Restart Bar 9:30pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole)
- 37. Middleware (Scott) Update Ticket Ticket Customer Engagement Manager + add Policy !! “Readyfor API tests” 9:45pm Firewall (Freddie) Net L2 (Bob) Middleware (Scott) Firewall change Restart Bar I think we are good! Middleware Manager VP VP Bar Lead Dev You “think?” pm
- 38. et gement “Ready for API tests” Customer Engagement Manager (Varsha) NOC (Bob) CustomerEngagement Manager (Varsha) Update Ticket Ticket “APIs OK” Middleware (Scott) Upda Tick 11:00pm Ticket Co
- 39. e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lightsare green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha)
- 40. e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lightsare green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha) .
- 41. NOC Lights are green… Iguess it is fixed. Close Ticket NOC (Bob) Zzz Next Day SVP (Susan) Whose fault is this?! Why are we so bad at change? What additional processes and approvals are you adding to never let this happen again?! VP VP Dir Dir VP Dir VP Scott) da) Carlos) (Bob) die) NetSec (Nicole) Cust. Engmt. (Varsha)
- 42.
- 43. We’ve invested inCloud, Agile, DevOps, Containers… Why does everything still take too long and cost too much? Executive Team Our transformation has largely ignored Ops
- 44. Most companies chasethe symptoms…
- 45. …by following theconventional wisdom:
- 46. “We need bettertools” …by following the conventional wisdom:
- 47. “We need bettertools” “We need more people” …by following the conventional wisdom:
- 48. “We need bettertools” “We need more people” “We need more discipline and attention to detail” …by following the conventional wisdom:
- 49. “We need bettertools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals” …by following the conventional wisdom:
- 50. “We need bettertools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals” …by following the conventional wisdom:
- 51. Challenge the conventional wisdomabout operations work
- 52. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 53. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 54. Where are decisionsmade? Who can take action? escalate 1° 2° 3° 4° escalate escalateor
- 55. Where are decisionsmade? Who can take action? escalate 1° 2° 3° 4° escalate escalateor Decisions made here
- 56. All work iscontextual John Allspaw
- 57. All work iscontextual rm -rf $PATHNAME John Allspaw
- 58. All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw
- 59. All work iscontextual rm -rf $PATHNAME John Allspaw
- 60. All work iscontextual rm -rf $PATHNAME John Allspaw
- 61. All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw
- 62. All work iscontextual rm -rf $PATHNAME John Allspaw
- 63. All work iscontextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw
- 64. escalate 1° 2° 3°4° escalate escalateor Context Where are decisions made? Who can take action?
- 65. Psychological Safety Psychological safety isa shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990
- 66. Psychological Safety Psychological safety isa shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990 Google: most important characteristic to predict team effectiveness? 2016
- 67. Psychological Safety Psychological safety isa shared belief that the team is safe for interpersonal risk taking. It can be defined as "being able to show and employ one's self without fear of negative consequences of self-image, status or career. - William Kahn Boston University 1990 Google: most important characteristic to predict team effectiveness? 2016 Psychological safety!
- 68. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 69. Toil: Name Fora Problem We’ve All Felt
- 70. Toil: Name Fora Problem We’ve All Felt “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
- 71. Toil vs. EngineeringWork Toil Engineering Work Lacks Enduring Value Builds Enduring Value Rote, Repetitive Creative, Iterative Tactical Strategic Increases With Scale Enables Scaling Can Be Automated Requires Human Creativity
- 72. Excessive Toil PreventsFixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
- 73. Excessive Toil PreventsFixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
- 74. Excessive Toil PreventsFixing the System Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Downward spiral is inevitable!
- 75. Forces That UndermineOperations Silos Queues Excessive ToilLow Trust
- 76.
- 77. Backlog Information I needX PrioritiesTools Silos
- 78. Backlog Information I needX PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
- 79. Silos cause disconnectsand mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
- 80. 1 2 3 Silos Interfere withfeedback loops
- 81. 1 2 3 Silos Interfere withfeedback loops Producer Consumer Ops Ops Ops
- 82. Function A Function B FunctionC Silos create labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
- 83. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 84. How do wecover for our silos’ disconnects and mismatches? Silo A Silo B
- 85. How do wecover for our silos’ disconnects and mismatches? Silo A Silo B Ticket Queue
- 86. ?? Silo A SiloB We all know how well that works Ticket Queue
- 87. Ticket queues arean expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
- 88. What do queuesdo to value streams?
- 89. What do queuesdo to value streams? Queue A Queue B
- 90. What do queuesdo to value streams? Queue A Queue B Queues disintegrate and obfuscate value streams
- 91. Ticket queues are“snowflake makers” ?? Silo A Silo B Ticket Queue
- 92. Ticket queues are“snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes Technically acceptable, but brittle and unreproducible
- 93. Ticket queues are“snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes Technically acceptable, but brittle and unreproducible
- 94. "Queues don’t learn” ?? SiloA Silo B Ticket Queue Scott Prugh CSGi
- 95. "Queues don’t learn” ?? SiloA Silo B Ticket Queue Scott Prugh CSGi
- 96. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 97.
- 98. What Can WeDo Differently?
- 99. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 100. “Shift Left” theability to take action escalate 1° 2° 3° 4° escalate escalateor
- 101. “Shift Left” theability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
- 102. “Shift Left” theability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor Tools Enablement and tooling
- 103. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 104. Start reducing toiltoday Toil
- 105. Start reducing toiltoday 1. Track toil levels for each team Toil
- 106. Start reducing toiltoday 1. Track toil levels for each team Toil
- 107. Track toil levelsfor each team
- 108. Track toil levelsfor each team • Standardize (e.g. meetings and email are “overhead" not “toil”)
- 109. Track toil levelsfor each team • Standardize (e.g. meetings and email are “overhead" not “toil”) • Track • Self-reporting • Periodic surveys • SM or PM interview/sampling
- 110. Track toil levelsfor each team • Standardize (e.g. meetings and email are “overhead" not “toil”) • Track • Self-reporting • Periodic surveys • SM or PM interview/sampling • Don’t get lost in time tracking weeds!
- 111. Start reducing toiltoday 1. Track toil levels for each team Toil
- 112. Start reducing toiltoday 1. Track toil levels for each team Toil 2. Set toil limit for each team (50% is conventional wisdom)
- 113. Start reducing toiltoday 1. Track toil levels for each team 2. Set toil limit for each team (50% is conventional wisdom) 3. Fund efforts to reduce toil (with emphasis on teams already over limit) Toil
- 114. Start reducing toiltoday 1. Track toil levels for each team 2. Set toil limit for each team (50% is conventional wisdom) 3. Fund efforts to reduce toil (with emphasis on teams already over limit) Toil Michael Kehoe Todd Palino (LinkedIn) At SREcon Americas 2019 Example Process “Code Yellow”
- 115.
- 116.
- 117. Where to focus? Toil Reduce TechnicalDebt Re-Engineer Processes
- 118. Where to focus? Toil Reduce TechnicalDebt Re-Engineer Processes Enable Self-Service
- 119. Where to focus? Toil Reduce TechnicalDebt Re-Engineer Processes Enable Self-Service
- 121.
- 122.
- 123. Eliminate Interruptions Eliminate Waiting Self-Service DoX. … and a lot less toil
- 124. How to enableself-service? Empower teams to spot and fix the anti-patterns.
- 125. “Do this forme, do it again, then do it again.” Done.I need you to do X Your other work I need you to do X I need you to do X Ticket Do X Later… Do X Do X Done. Done. Your other work Self-Service Self-Service Self-Service Your other work x2 Your other work x3 Later…Later… Later… Your other work Your other work After Before Wait Interrupt Ticket Wait Interrupt Ticket Wait Interrupt
- 126. “Do this forme, do it again, then do it again.” Done.I need you to do X Your other work I need you to do X I need you to do X Ticket Do X Later… Do X Do X Done. Done. Your other work Self-Service Self-Service Self-Service Your other work x2 Your other work x3 Later…Later… Later… Your other work Your other work After Before Wait Interrupt Ticket Wait Interrupt Ticket Wait Interrupt
- 127. “I could fixit, but I can’t get to it.” Environment I could fix it if I could get to it Before Wait Interrupt
- 128. “I could fixit, but I can’t get to it.” Environment I could fix it if I could get to it Before Wait Interrupt After I’ve got this! Environment Self- Service
- 129. “What’s the BESTway to do this?!” Environment 3:00 AM sar -n Dev,EDEV 1 sar -n TCP,ETCP 1 cat etc/resolv.conf mpstat -P ALL 1 tcpretrans tcpconnect tcpaccept netstat -rnv check firewall config netstat -s Check Network Earlier… After Self-Service
- 130. “Oh no… I’mthe Brent.” Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Before
- 131. “Oh no… I’mthe Brent.” Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Ticket Before
- 132. “The dog-pile.” !! I thinkits a problem with db07-store2.uswest.acme “$ top” “$ top” db07store2. uswest.acme “$ top” “$ top” “$ top” !! “$ top” !! !! !! healthcheck store2 -all db07store2. uswest.acme Self-Service 1. 2. 3. I think its a problem with db07-store2.uswest.acme
- 133. “I’m an expert,I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before
- 134. “I’m an expert,I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before
- 135. “I’m an expert,I don’t read the wiki.” docs Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this… Environment docs Later… Before Service has changed. Use this flag or bad things will happen! Pause monitoring first or we all get woken up! “restart” Environment Later… Update Restart Job ✅ I’ve done this before. I’ve got this. Self-Service Self-Service After
- 136. “You have aworkaround, deal with it.”
- 137. “You have aworkaround, deal with it.”
- 138. Self-Service Operations DesignPattern (in a nutshell) Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 139. Self-Service Operations DesignPattern (in a nutshell) Pull-Based Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 140. Self-Service Operations DesignPattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 141. Self-Service Operations DesignPattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 142. Self-Service Operations DesignPattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Let people who “push buttons” define the buttons Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 143. Self-Service Operations DesignPattern (in a nutshell) Pull-Based Accept tools/languages that teams want to use Let people who “push buttons” define the buttons Build in security and compliance Define “guardrails” to provide work safety Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 144. Self-Service is ultimatelyabout user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge
- 145. Self-Service is ultimatelyabout user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI)
- 146. Self-Service is ultimatelyabout user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI) 2. “Guardrails” (Smart options that helpfully constrain)
- 147. Self-Service is ultimatelyabout user experience Consumer of Ops Capabilities Self-Service On Demand Ops Capability Specialist Knowledge Ops Capability Specialist Knowledge 1.Work how they want to work (GUI, API, CLI) 2. “Guardrails” (Smart options that helpfully constrain) 3.Dynamic resource model (Up-to-date details of your environment)
- 148. Forces That UndermineOperations Silos Ticket Queues Excessive ToilLow Trust
- 149. Obvious: Get ridof as many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
- 150. Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible
- 151. Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible “Horizontal” shared responsibility, not everyone do everything!
- 152. Shared and dedicatedresponsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model
- 153. Shared and dedicatedresponsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model
- 154. Shared and dedicatedresponsibility is key Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget with consequences “Netflix" Model “Google” Model Same high-quality, high-velocity results!
- 155. But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
- 156. But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
- 157. But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
- 158. Forces That UndermineOperations Silos Queues Excessive ToilLow Trust
- 159. Self-Service Operations: Turnhandoffs into self-service Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Cross-Functional Product Team 2 Ops (embedded)
- 160. Self-Service Operations: Workswith any org model Development Team 1 Development Team 2 Development Team n Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
- 161. Development Team 1 DevelopmentTeam 2 Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Cross-Functional Product Team n Ops (embedded) But, what about security and compliance? Build-in Security Here Build-in Compliance Here
- 162. Are all ticketsbad?
- 163. Are all ticketsbad? Ticket System No. Just use tickets for what they are good for
- 164. Are all ticketsbad? 1.Documenting true problems/issues/exceptionsTicket System No. Just use tickets for what they are good for
- 165. Are all ticketsbad? 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System No. Just use tickets for what they are good for
- 166. Are all ticketsbad? 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System No. Just use tickets for what they are good for
- 167. Self-Service can alsobe a foundation for strategic initiatives
- 168. Strategic: Improve incidentresponse times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF
- 169. Strategic: Improve incidentresponse times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower
- 170. Strategic: Improve incidentresponse times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower
- 171. Strategic: Improve incidentresponse times https://youtu.be/USYrDaPEFtM Jody Mulkey at DOES ‘15 SF Services Monitoring Scripts/Tools Services Monitoring Scripts/ToolsServices Monitoring Scripts/Tools DEV STAGE PROD Dev & QA NOC/Ops Dev Promote approved jobs Self-Service Self-Service Empower • Reduced MTTR by 92% • Reduced escalations by 50% • Reduced overall support costs by 55%
- 172. Strategic: Reduce complianceburden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg
- 173. Strategic: Reduce complianceburden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated
- 174. Strategic: Reduce complianceburden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated LOB #1 LOB #2 LOB #3 LOB …n Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Self-Service ComplianceConsistency
- 175. Strategic: Reduce complianceburden & improve consistency Shaun Norris at DOES ‘18 Las Vegas https://youtu.be/d5IMvK0YHTg Optimized for compliance • 86,000+ employees • 60+ countries • Highly regulated LOB #1 LOB #2 LOB #3 LOB …n Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Data Center Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Services Scripts/Tools Cloud Self-Service ComplianceConsistency 12 months: • Saved 28 person years of time • 13,000+ ops tasks in privileged environments that didn’t require a review • ~200 less customer impacting events
- 176. rundeck.com/self-service Read for freeonline: Working on documenting the Self- Service Operations design pattern. Where I need your help…
- 177. Recap Don’t forget aboutOps. Challenge conventional wisdom. Use self-service to reduce remaining ticket queues “Shift-Left” control and decision making. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silosLearn from SRE: Focus on understanding & reducing toil Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil Toil at manageable percentage of capacity oil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Understand the forces undermining operations work Development Team 1 Development Team 2 Ops/SRE Team Self-Service Operations On Demand On Demand On Demand On Demand Ops (operates platform) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Cross-Functional Product Team n Ops (embedded)
- 178.