Rundeck, profile picture
Uploaded byRundeck
PDF, PPTX18,611 views

Ops Happens: Improving Incident Response Using DevOps and SRE Practices

The document discusses the challenges of incident response and operational issues within a DevOps and SRE context, emphasizing that deployment is not the ultimate goal; effective operations are. It illustrates a typical incident management scenario highlighting inefficiencies due to silos, queues, and excessive manual processes, which lead to delays and increased operational risks. The text advocates for transforming operational practices to reduce toil and improve response times by challenging existing processes and breaking down silos.

Technology◦
Related topics:
Incident Management•

Embed presentation

Download as PDF, PPTX
1 / 113
2 / 113
3 / 113
4 / 113
5 / 113
6 / 113
7 / 113
8 / 113
9 / 113
10 / 113
11 / 113
12 / 113
13 / 113
14 / 113
15 / 113
16 / 113
17 / 113
18 / 113
19 / 113
20 / 113
21 / 113
22 / 113
23 / 113
24 / 113
25 / 113
26 / 113
27 / 113
28 / 113
29 / 113
30 / 113
31 / 113
32 / 113
33 / 113
34 / 113
35 / 113
36 / 113
37 / 113
38 / 113
39 / 113
40 / 113
41 / 113
42 / 113
43 / 113
44 / 113
45 / 113
46 / 113
47 / 113
48 / 113
49 / 113
50 / 113
51 / 113
52 / 113
53 / 113
54 / 113
55 / 113
56 / 113
57 / 113
58 / 113
59 / 113
60 / 113
61 / 113
62 / 113
63 / 113
64 / 113
65 / 113
66 / 113
67 / 113
68 / 113
69 / 113
70 / 113
71 / 113
72 / 113
73 / 113
74 / 113
75 / 113
76 / 113
77 / 113
78 / 113
79 / 113
80 / 113
81 / 113
82 / 113
83 / 113
84 / 113
85 / 113
86 / 113
87 / 113
88 / 113
89 / 113
90 / 113
91 / 113
92 / 113
93 / 113
94 / 113
95 / 113
96 / 113
97 / 113
98 / 113
99 / 113
100 / 113
101 / 113
102 / 113
103 / 113
104 / 113
105 / 113
106 / 113
107 / 113
108 / 113
109 / 113
110 / 113
111 / 113
112 / 113
113 / 113
Ops Happens: Improving Incident Response Using DevOps and SRE Practices Damon Edwards @damonedwards 2018
Ops Improvement DevOps Ops Tools Community Damon Edwards
Deployment doesn’t make us money. Operations does.
Deployment doesn’t make us money. Operations does.
Deployment doesn’t make us money. Operations does. Deployment isn’t the goal. But we treat it like it is.
Deployment doesn’t make us money. Operations does. Deployment isn’t the goal. But we treat it like it is.
Deployment doesn’t make us money. Operations does. Deployment isn’t the goal. But we treat it like it is. Operations rarely gets to transform. Time to start.
Deployment doesn’t make us money. Operations does. Deployment isn’t the goal. But we treat it like it is. Operations rarely gets to transform. Time to start.
Let’s start with a true (but all too familiar) story…
Digital Agile DevOps SRE Cloud Docker Kubernetes Microservices CHANGE Bravo!Wow! Impressive
But what happens after deployment?
It was just another Tuesday…
NOC NOC Biz Manager Escalate! NOC NOC NOC (Bob) Open Incident Ticket 9:30am 10:00am NOC (Bob) Biz Manager Ticket Context Wagon Yes, but this looks different Hasn’t there been some intermittent errors this week? v3 ?!
NOC (Bob) Open Incident Ticket Ticket Biz Manager App-specific SREs “Try this.” “Try that.” SRE SysAdmin with Prod Access (Steve) SRE SRE SRE SRE SRE SRE Bridge Call Biz Manager fixed? fixed? NOC (Bob) Biz Manager NOC (Bob) Biz Manager SysAdmin (Steve) 7 x SRE Ticket Context Wagon Ticket Context Wagon
SRE “It’s a problem with the Foo service” SRE SRE Foo SRE SRE SRE SRE Bridge Call Biz Manager Foo Service No. NOC (Bob) Update Ticket Ticket Foo Lead Dev + add 12:00pm NOC (Bob) Biz Manager Foo SRE Ticket Context Wagon Can you fix it?
o Dev Foo Lead Dev (Karen) ding! Ignore. App Manager Hey did you see that ticket? Foo Lead Dev (Karen) sigh. I’ll take a look I’m go mor pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE Scrum Ticket Context Wagon
k Foo Lead Dev (Karen) I’m going to need more log files Ticket SysAdmin Team + add Update Ticket Chat “Can someone with access to Foo Service in Prod01 help me with ticket #42516?” SysAdmin (Lee) Ticket “logs attached” Foo Lead Dev (Karen) Ticket “no the other ones” Le (K NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Ticket Context Wagon
Foo Lead Dev (Karen) Logs -Who restarted these services? (and why?) -They didn’t use the correct environment variables! -This entire service pool needs to be restarted! Ticket Update Ticket NOC (Bob) Update Ticket Ticket Middleware Team + add “Middleware, please urgent restart this entire app pool with the correct environment variable” 2:00pm Ticket Context W
ase s entire e correct iable” NOC (Bob) Middleware Manager (Melissa) No way. It’s the middle of the day! You need business approval. NOC (Bob) Update Ticket Ticket SVP for Line of Business + add (S NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager NOC (B Biz Ma App M Lead D Foo SR Ticket Context Wagon Ticket Context Wagon 2:30pm
Update Ticket Ticket SVP for Line of Business + add SVP (Susan) Chief of Staff Tech VP Tech VP Update Ticket Ticket “Restart approved” Customer impact? Ticket Middlewa Manage (Melissa Wh prod 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Ticket Context Wagon
Share point proved” Ticket Middleware Manager (Melissa) Who knows these production services the best? Ellen! Middleware Middleware (Scott) Ellen to Europe office Middleware (Scott) Trial and error .doc 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Ticket Context Wagon
Share point Middleware (Scott) Trial and error .doc NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) ket Context Wagon Middleware (Scott) Bar Service 10 min Middleware (Scott) Waiting for Acme Service Acme startup failed Bar Service 6:00pm
Come on.. no.no.no. What? Why? Middleware (Scott)
Come on.. no.no.no. What? Why? Middleware (Scott)
8888888 Come on.. no.no.no. What? Why? Middleware (Scott)
-Bar app startup timed out. Error says can’t connect to Acme service. - I looked at Acme but it seems to be running -Is this error message correct? Why can’t Bar connect? Ticket Update Ticket Middleware (Scott) Bar SRE + add Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add 6:45 NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda)Ticket Context Wagon The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked.
Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add Bar Lead Dev 6:45pm ob) ager nager v (Karen) E SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Customers are calling. What is going on?The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked. Bar Lead Dev (Liu) Business Managers I can comment out the test… But the CD pipeline only goes to QA ENV!
Network Dir (Carlos) Middleware (Scott) Carlos, I need a favor. Can you escalate?Middleware Manager (Melissa) Customers are calling. What is going on? Last week.. Net SRE VP VP Priority! Different Incident! Net SRE Net SRE Net SRE Its the network! Business Managers Your network is broken! Business Managers We are already working on it! Network VPs out he ly V!
Network SRE (Hari) The firewall is blocking the traffic You’ll have to take it up with the Firewall Team -URGENT: Firewall is blocking connection between Bar and Acme Ticket Open Firewall Ticket Firewall Team + add Firewall Engineer (Freddie) Middleware (Scott) Paging on-call… Open bridge… Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. 8:00p NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Ticket Context Wagon
Firewall Engineer (Freddie) Middleware (Scott) Firewall Engineer (Freddie) Middleware (Scott) Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. There was a rule change last Thursday that would stop Bar from talking to Acme. Can you change it back? Sure we make changes on Thursday… Chief of Staff SVP and VPs are livid… this was supposed to be a safe change!! Freddie, we’ve got customers calling. ES Em pro rul Update Firewall Ticket Firewall Engineer (Freddie) 8:00pm
d VPs are livid… this was sed to be a safe change!! we’ve got customers calling. ESCALATE: Emergency production firewall rule change review Ticket Update Firewall Ticket NetSec + add Firewall Engineer (Freddie) Paging on-call… NetSec (Nicole) This is production so I’ll have to get others on the Network CAB… Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev 9:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdm Middle SVP Chief o 2 x Tec Ticket Context Wagon
I’ll have Network Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! APPROVE: Emergency firewall rule change Ticket Update Firewall Ticket NetSec (Nicole) … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev Firewall (Freddie) Net L2 (Bob) Middl (Sc Firewall change Restart Bar 9:30pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole)
Middleware (Scott) Update Ticket Ticket Customer Engagement Manager + add Policy !! “Ready for API tests” 9:45pm Firewall (Freddie) Net L2 (Bob) Middleware (Scott) Firewall change Restart Bar I think we are good! Middleware Manager VP VP Bar Lead Dev You “think?” pm
et gement “Ready for API tests” Customer Engagement Manager (Varsha) NOC (Bob) Customer Engagement Manager (Varsha) Update Ticket Ticket “APIs OK” Middleware (Scott) Upda Tick 11:00pm Ticket Co
e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha)
e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha) .
NOC Lights are green… I guess it is fixed. Close Ticket NOC (Bob) Zzz Next Day SVP (Susan) Whose fault is this?! Why are we so bad at change? What additional processes and approvals are you adding to never let this happen again?! VP VP Dir Dir VP Dir VP cott) a) Carlos) (Bob) die) NetSec (Nicole) Cust. Engmt. (Varsha)
Later…
We’ve invested in Cloud, Agile, DevOps, Containers… Why does everything still take too long and cost too much? Executive Team Our transformation has largely ignored Ops
Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting
Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Waiting ! Defects
Manual / Motion Manual / Motion Manual / Motion Manual / Motion Manual / Motion Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Waiting ! Defects ! Motion/Manual
Manual / Motion Manual / Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Waiting ! Defects ! Motion/Manual ! Task Switching
Manual / Motion Manual / Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Partially Done Partially Done Partially Done Partially Done Partially Done Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done
Manual / Motion Manual / Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Partially Done Partially Done Partially Done Partially Done Partially Done Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done ! Extra Process
“We need better tools”
“We need better tools” “We need more people”
“We need better tools” “We need more people” “We need more discipline and attention to detail”
“We need better tools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals”
“We need better tools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals”
Challenge the principles and practices we take for granted
1 2 3 Feedback loops
Where are decisions made? Who can take action? escalate 1° 2° 3° 4° escalate escalateor
All work is contextual John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
All work is contextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw DevOps Enterprise Summit San Francisco 2017
escalate 1° 2° 3° 4° escalate escalateor Context Where are decisions made? Who can take action? They have all of the context But, decisions are made here?
“Shift Left” the ability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
Not Here!
Not Here!
Not Here!
Here?
Even Better!
What gets in the way? Silos Queues Toil Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools ?? Silo A Silo B Ticket Queue
Silos Backlog Information PrioritiesTools
Backlog Information I need X PrioritiesTools Silos
Backlog Information I need X PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
Silos cause disconnects and mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
Function A Function B Function C Becomes siloed labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
How do we cover for our silos disconnects and mismatches? Silo A Silo B
How do we cover for our silos disconnects and mismatches? Silo A Silo B Ticket Queue
?? Silo A Silo B We all know how well that works Ticket Queue
Ticket queues are an expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
Tickets queues become “snowflake makers” ?? Silo A Silo B Ticket Queue
Tickets queues become “snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes (each unique, technically acceptable but unreproducible and brittle)
Silos + Ticket Queues = Locked into a broken system Unreproducible Snowflakes More Outages Longer Lead Times / Unpredictibility Siloed Labor Pools Ticket Queues Diminished Labor Capacity Brittle Environments Errors Error-Prone ManualTasks Interrupt project work Information Mismatches / Miscommunication Silo-Specific Optimization Handoffs With Capacity, Pace, Priority Mismatches “Managing to the Queue” Working out of Context “Align by Functional Capability” Management Strategy
Excessive toil prevents fixing the system
Excessive toil prevents fixing the system “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
Excessive Toil prevents fixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
Excessive Toil prevents fixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
So what can we do differently?
Obvious: Get rid of as many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible “Horizontal” shared responsibility is key feature!
But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
But what about the cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
Operations as a Service: Turn handoffs into self-service Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
Development Team 1 Development Team 2 Development Team n Ops/SRE Team Operations as a Service On Demand On Demand On Demand On Demand Ops (builds & operates) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Operations as a Service: Works with any org model
Use tickets only for what they are good for Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System
Security or compliance “in the way”? Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Build-in Security Here Build-in Compliance Here
Reduce Toil
Reduce Toil 1. Track toil levels for each team
Reduce Toil 1. Track toil levels for each team 2. Set toil limits for each team
Reduce Toil 1. Track toil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits)
Reduce Toil 1. Track toil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits) Bonus: Use Service Level Objectives, Error Budgets, and other lessons from SRE
Recap Don’t forget about Ops. Challenge conventional wisdom. Team A (Dev) Team B (Ops) Ticket System ?? Leverage the Operations as a Service design pattern “Shift-Left” control and decision making. Understand the cost of silos and ticket-driven request queues Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silos and queues Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Learn from SRE: Reduce toil to create capacity to change Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”
Let’s talk… @damonedwards damon@rundeck.com https://www.rundeck.com/oaas Dive Deeper Into Operations as a Service:

More Related Content

PDF
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
PDF
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
PDF
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
PDF
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
PDF
Operations as a Service: Because Failure Still Happens
byRundeck
 
PDF
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
PDF
The "Ops" Side of DevSecOps
byRundeck
 
PDF
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
Operations as a Service: Because Failure Still Happens
byRundeck
 
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
The "Ops" Side of DevSecOps
byRundeck
 
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 

What's hot

PDF
Operations: The Last Mile
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Clearing the Way For SRE In the Enterprise
byRundeck
 
PDF
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
PDF
Operations: The Last Mile
byRundeck
 
PDF
The Last Mile Continued: Incident Management
byRundeck
 
PDF
SRE Lessons for the Enterprise
byRundeck
 
PDF
Self-Service Operations: Because Failure Still Happens (Developer Edition)
byRundeck
 
PDF
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
PDF
Self-Service Operations: Because Ops Still Happens
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Operations: The Last Mile Problem For DevOps
byRundeck
 
PDF
SRE From Scratch
byGrier Johnson
 
PDF
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PDF
PagerDuty + Rundeck = Shorter Incidents, Fewer Escalations
byRundeck
 
PDF
Innovation and Architecture
byAdrian Cockcroft
 
PDF
Agile Infrastructure - Agile 2009
byAndrew Shafer
 
PDF
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 
PDF
My History with Atlassian Tools, and Why I'm Moving to Studio
byAtlassian
 
Operations: The Last Mile
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Clearing the Way For SRE In the Enterprise
byRundeck
 
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
Operations: The Last Mile
byRundeck
 
The Last Mile Continued: Incident Management
byRundeck
 
SRE Lessons for the Enterprise
byRundeck
 
Self-Service Operations: Because Failure Still Happens (Developer Edition)
byRundeck
 
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
Self-Service Operations: Because Ops Still Happens
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Operations: The Last Mile Problem For DevOps
byRundeck
 
SRE From Scratch
byGrier Johnson
 
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PagerDuty + Rundeck = Shorter Incidents, Fewer Escalations
byRundeck
 
Innovation and Architecture
byAdrian Cockcroft
 
Agile Infrastructure - Agile 2009
byAndrew Shafer
 
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 
My History with Atlassian Tools, and Why I'm Moving to Studio
byAtlassian
 

Similar to Ops Happens: Improving Incident Response Using DevOps and SRE Practices

PPTX
Respond to and troubleshoot production incidents like an sa
byTom Cudd
 
PPT
Incident Management
byiicecollege
 
PPTX
Forward Networks - Networking Field Day 13 presentation
byAndrew Wesbecher
 
PDF
DevOps Incident Handling - Making friends not enemies.
byServer Density
 
PDF
Incident Management Report
byCharles Fox
 
PPTX
Ict service desk team meeting
byjuliehall271
 
PDF
A Practical Approach to Incident Management for SaaS/PaaS
byMichael Weber
 
PDF
OSMC 2024 | The story of firefighting: learnings from the incident management...
byNETWAYS
 
PDF
Mission Possible: Future of IT Support
byCompTIA
 
PDF
Forward Networks - Networking Field Day 13 presentation
byForward Networks
 
PPT
PowerPoint Presentation
bywebhostingguy
 
PPT
PowerPoint Presentation
bywebhostingguy
 
PDF
HDI Capital Area Announcements and Presentation August 18 2017
byhdicapitalarea
 
PDF
Rick Clymer - Incident Management.pdf
byQA or the Highway
 
DOC
CV_07.06.16
byArjun Pawar
 
PPTX
ITIL 4 - Practices | Slides UNIVERSIDAD PERUANA DEA CICENCIAS APLICADAS
bycqundaria
 
PDF
Denver Atlassian Community Meeting - April 2019
bydenveraug
 
PPT
6 itil v3 service operation v1.8
byKarthik Arumugham
 
PDF
Incident Management - Obtaining Our #1 Objective
byMark Copeland
 
PDF
9 postproduction
byLen Bass
 
Respond to and troubleshoot production incidents like an sa
byTom Cudd
 
Incident Management
byiicecollege
 
Forward Networks - Networking Field Day 13 presentation
byAndrew Wesbecher
 
DevOps Incident Handling - Making friends not enemies.
byServer Density
 
Incident Management Report
byCharles Fox
 
Ict service desk team meeting
byjuliehall271
 
A Practical Approach to Incident Management for SaaS/PaaS
byMichael Weber
 
OSMC 2024 | The story of firefighting: learnings from the incident management...
byNETWAYS
 
Mission Possible: Future of IT Support
byCompTIA
 
Forward Networks - Networking Field Day 13 presentation
byForward Networks
 
PowerPoint Presentation
bywebhostingguy
 
PowerPoint Presentation
bywebhostingguy
 
HDI Capital Area Announcements and Presentation August 18 2017
byhdicapitalarea
 
Rick Clymer - Incident Management.pdf
byQA or the Highway
 
CV_07.06.16
byArjun Pawar
 
ITIL 4 - Practices | Slides UNIVERSIDAD PERUANA DEA CICENCIAS APLICADAS
bycqundaria
 
Denver Atlassian Community Meeting - April 2019
bydenveraug
 
6 itil v3 service operation v1.8
byKarthik Arumugham
 
Incident Management - Obtaining Our #1 Objective
byMark Copeland
 
9 postproduction
byLen Bass
 

More from Rundeck

PDF
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
PPTX
Introducing PagerDuty Process Automation
byRundeck
 
PDF
How to Build a Custom Plugin in Rundeck
byRundeck
 
PDF
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
PDF
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
PDF
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
PPTX
Mastering Secrets Management in Rundeck
byRundeck
 
PDF
What's New in Rundeck 3.4
byRundeck
 
PDF
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
PDF
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
PPTX
Introduction to Rundeck
byRundeck
 
PPTX
Automated Remediation with Rundeck + Sensu
byRundeck
 
PDF
Modernizing Incident Response
byRundeck
 
PDF
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
PDF
Datadog + Rundeck at DASH 2020
byRundeck
 
PDF
Rundeck Overview
byRundeck
 
PDF
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PPTX
Advanced Cluster Settings
byRundeck
 
PDF
Maximizing Your Rundeck Migration
byRundeck
 
PDF
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
Introducing PagerDuty Process Automation
byRundeck
 
How to Build a Custom Plugin in Rundeck
byRundeck
 
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
Mastering Secrets Management in Rundeck
byRundeck
 
What's New in Rundeck 3.4
byRundeck
 
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
Introduction to Rundeck
byRundeck
 
Automated Remediation with Rundeck + Sensu
byRundeck
 
Modernizing Incident Response
byRundeck
 
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
Datadog + Rundeck at DASH 2020
byRundeck
 
Rundeck Overview
byRundeck
 
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
Advanced Cluster Settings
byRundeck
 
Maximizing Your Rundeck Migration
byRundeck
 
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 

Recently uploaded

PDF
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
PDF
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
PPTX
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
PDF
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
PDF
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
PDF
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PDF
How to Prepare for the RWA Tokenization Trend
byrose mason
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PPTX
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
PDF
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
PDF
How to Install XRDP on CentOS 10 .pdf
byGreen Webpage
 
PDF
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
PDF
final.pdf
byomarbishtawi04
 
PPTX
Workflow and decision Automation with Flowable
bychakib ch
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
How to Prepare for the RWA Tokenization Trend
byrose mason
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
How to Install XRDP on CentOS 10 .pdf
byGreen Webpage
 
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
final.pdf
byomarbishtawi04
 
Workflow and decision Automation with Flowable
bychakib ch
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 

Ops Happens: Improving Incident Response Using DevOps and SRE Practices

  • 1.
    Ops Happens: Improving IncidentResponse Using DevOps and SRE Practices Damon Edwards @damonedwards 2018
  • 2.
  • 4.
    Deployment doesn’t makeus money. Operations does.
  • 5.
    Deployment doesn’t makeus money. Operations does.
  • 6.
    Deployment doesn’t makeus money. Operations does. Deployment isn’t the goal. But we treat it like it is.
  • 7.
    Deployment doesn’t makeus money. Operations does. Deployment isn’t the goal. But we treat it like it is.
  • 8.
    Deployment doesn’t makeus money. Operations does. Deployment isn’t the goal. But we treat it like it is. Operations rarely gets to transform. Time to start.
  • 9.
    Deployment doesn’t makeus money. Operations does. Deployment isn’t the goal. But we treat it like it is. Operations rarely gets to transform. Time to start.
  • 10.
    Let’s start witha true (but all too familiar) story…
  • 11.
  • 12.
    But what happensafter deployment?
  • 13.
    It was justanother Tuesday…
  • 14.
    NOC NOC Biz Manager Escalate! NOC NOC NOC (Bob) Open Incident Ticket 9:30am 10:00am NOC(Bob) Biz Manager Ticket Context Wagon Yes, but this looks different Hasn’t there been some intermittent errors this week? v3 ?!
  • 15.
    NOC (Bob) Open Incident Ticket Ticket Biz Manager App-specific SREs “Try this.” “Try that.” SRE SysAdmin withProd Access (Steve) SRE SRE SRE SRE SRE SRE Bridge Call Biz Manager fixed? fixed? NOC (Bob) Biz Manager NOC (Bob) Biz Manager SysAdmin (Steve) 7 x SRE Ticket Context Wagon Ticket Context Wagon
  • 16.
    SRE “It’s a problem withthe Foo service” SRE SRE Foo SRE SRE SRE SRE Bridge Call Biz Manager Foo Service No. NOC (Bob) Update Ticket Ticket Foo Lead Dev + add 12:00pm NOC (Bob) Biz Manager Foo SRE Ticket Context Wagon Can you fix it?
  • 17.
    o Dev Foo Lead Dev (Karen) ding! Ignore. App Manager Hey didyou see that ticket? Foo Lead Dev (Karen) sigh. I’ll take a look I’m go mor pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE Scrum Ticket Context Wagon
  • 18.
    k Foo Lead Dev (Karen) I’m goingto need more log files Ticket SysAdmin Team + add Update Ticket Chat “Can someone with access to Foo Service in Prod01 help me with ticket #42516?” SysAdmin (Lee) Ticket “logs attached” Foo Lead Dev (Karen) Ticket “no the other ones” Le (K NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Ticket Context Wagon
  • 19.
    Foo Lead Dev (Karen) Logs -Who restartedthese services? (and why?) -They didn’t use the correct environment variables! -This entire service pool needs to be restarted! Ticket Update Ticket NOC (Bob) Update Ticket Ticket Middleware Team + add “Middleware, please urgent restart this entire app pool with the correct environment variable” 2:00pm Ticket Context W
  • 20.
    ase s entire e correct iable” NOC (Bob) Middleware Manager (Melissa) Noway. It’s the middle of the day! You need business approval. NOC (Bob) Update Ticket Ticket SVP for Line of Business + add (S NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager NOC (B Biz Ma App M Lead D Foo SR Ticket Context Wagon Ticket Context Wagon 2:30pm
  • 21.
    Update Ticket Ticket SVP for Lineof Business + add SVP (Susan) Chief of Staff Tech VP Tech VP Update Ticket Ticket “Restart approved” Customer impact? Ticket Middlewa Manage (Melissa Wh prod 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Ticket Context Wagon
  • 22.
    Share point proved” Ticket Middleware Manager (Melissa) Who knows these productionservices the best? Ellen! Middleware Middleware (Scott) Ellen to Europe office Middleware (Scott) Trial and error .doc 5:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Ticket Context Wagon
  • 23.
    Share point Middleware (Scott) Trial and error .doc NOC(Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) ket Context Wagon Middleware (Scott) Bar Service 10 min Middleware (Scott) Waiting for Acme Service Acme startup failed Bar Service 6:00pm
  • 24.
    Come on.. no.no.no. What?Why? Middleware (Scott)
  • 25.
    Come on.. no.no.no. What?Why? Middleware (Scott)
  • 26.
    8888888 Come on.. no.no.no. What?Why? Middleware (Scott)
  • 27.
    -Bar app startuptimed out. Error says can’t connect to Acme service. - I looked at Acme but it seems to be running -Is this error message correct? Why can’t Bar connect? Ticket Update Ticket Middleware (Scott) Bar SRE + add Bar SRE (Linda) Middleware (Scott) -URGENT: Network connection issue between Bar and Acme Ticket Update Ticket Network SRE Team + add 6:45 NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda)Ticket Context Wagon The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked.
  • 28.
    Bar SRE (Linda) Middleware (Scott) -URGENT: Network connectionissue between Bar and Acme Ticket Update Ticket Network SRE Team + add Bar Lead Dev 6:45pm ob) ager nager v (Karen) E SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Customers are calling. What is going on?The new environment pre-flight check is preventing startup. Looks like Bar’s connection to Acme is being blocked. Bar Lead Dev (Liu) Business Managers I can comment out the test… But the CD pipeline only goes to QA ENV!
  • 29.
    Network Dir (Carlos) Middleware (Scott) Carlos, Ineed a favor. Can you escalate?Middleware Manager (Melissa) Customers are calling. What is going on? Last week.. Net SRE VP VP Priority! Different Incident! Net SRE Net SRE Net SRE Its the network! Business Managers Your network is broken! Business Managers We are already working on it! Network VPs out he ly V!
  • 30.
    Network SRE (Hari) The firewall is blockingthe traffic You’ll have to take it up with the Firewall Team -URGENT: Firewall is blocking connection between Bar and Acme Ticket Open Firewall Ticket Firewall Team + add Firewall Engineer (Freddie) Middleware (Scott) Paging on-call… Open bridge… Can’t be the firewall, it hasn’t changed since last Thursday. No its the firewall. 8:00p NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Ticket Context Wagon
  • 31.
    Firewall Engineer (Freddie) Middleware (Scott) Firewall Engineer (Freddie) Middleware (Scott) Can’tbe the firewall, it hasn’t changed since last Thursday. No its the firewall. There was a rule change last Thursday that would stop Bar from talking to Acme. Can you change it back? Sure we make changes on Thursday… Chief of Staff SVP and VPs are livid… this was supposed to be a safe change!! Freddie, we’ve got customers calling. ES Em pro rul Update Firewall Ticket Firewall Engineer (Freddie) 8:00pm
  • 32.
    d VPs arelivid… this was sed to be a safe change!! we’ve got customers calling. ESCALATE: Emergency production firewall rule change review Ticket Update Firewall Ticket NetSec + add Firewall Engineer (Freddie) Paging on-call… NetSec (Nicole) This is production so I’ll have to get others on the Network CAB… Chief of Staff Firewall (Freddie) Middleware (Scott) Customer outage! … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev 9:00pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdm Middle SVP Chief o 2 x Tec Ticket Context Wagon
  • 33.
    I’ll have Network Chief of Staff Firewall (Freddie) Middleware (Scott) Customeroutage! APPROVE: Emergency firewall rule change Ticket Update Firewall Ticket NetSec (Nicole) … I’ll call SVP Susan Middleware Manager VP VP Bar Lead Dev Firewall (Freddie) Net L2 (Bob) Middl (Sc Firewall change Restart Bar 9:30pm NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole)
  • 34.
    Middleware (Scott) Update Ticket Ticket Customer Engagement Manager + add Policy !! “Readyfor API tests” 9:45pm Firewall (Freddie) Net L2 (Bob) Middleware (Scott) Firewall change Restart Bar I think we are good! Middleware Manager VP VP Bar Lead Dev You “think?” pm
  • 35.
    et gement “Ready for API tests” Customer Engagement Manager (Varsha) NOC (Bob) CustomerEngagement Manager (Varsha) Update Ticket Ticket “APIs OK” Middleware (Scott) Upda Tick 11:00pm Ticket Co
  • 36.
    e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lightsare green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha)
  • 37.
    e Ticket “APIs OK” Middleware (Scott) Update Ticket Ticket “Services restarted OK” NOC NOC Lightsare green… I guess it is fixed. Close Ticket NOC (Bob) Zzz 11:30pm N NOC (Bob) Biz Manager App Manager Lead Dev (Karen) Foo SRE SysAdmin (Lee) Middleware Manager SVP Chief of Staff 2 x Tech VP Middleware (Scott) Bar SRE (Linda) Network PM (Carlos) Network SRE (Bob) Firewall (Freddie) Ticket Context Wagon NetSec (Nicole) Cust. Engmt. (Varsha) .
  • 38.
    NOC Lights are green… Iguess it is fixed. Close Ticket NOC (Bob) Zzz Next Day SVP (Susan) Whose fault is this?! Why are we so bad at change? What additional processes and approvals are you adding to never let this happen again?! VP VP Dir Dir VP Dir VP cott) a) Carlos) (Bob) die) NetSec (Nicole) Cust. Engmt. (Varsha)
  • 39.
  • 40.
    We’ve invested inCloud, Agile, DevOps, Containers… Why does everything still take too long and cost too much? Executive Team Our transformation has largely ignored Ops
  • 42.
  • 43.
  • 44.
    Manual / Motion Manual/ Motion Manual / Motion Manual / Motion Manual / Motion Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Waiting ! Defects ! Motion/Manual
  • 45.
    Manual / Motion Manual/ Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Waiting ! Defects ! Motion/Manual ! Task Switching
  • 46.
    Manual / Motion Manual/ Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Partially Done Partially Done Partially Done Partially Done Partially Done Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done
  • 47.
    Manual / Motion Manual/ Motion Manual / Motion Manual / Motion Manual / Motion Task Switching Task Switching Task SwitchingTask Switching Task Switching Task Switching Task Switching Task Switching Task Switching Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Waiting Defects Defects Defects Defects Defects Partially Done Partially Done Partially Done Partially Done Partially Done Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Extra Process Waiting ! Defects ! Motion/Manual ! Task Switching ! Partially Done ! Extra Process
  • 49.
  • 50.
    “We need bettertools” “We need more people”
  • 51.
    “We need bettertools” “We need more people” “We need more discipline and attention to detail”
  • 52.
    “We need bettertools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals”
  • 53.
    “We need bettertools” “We need more people” “We need more discipline and attention to detail” “We need more change reviews/approvals”
  • 54.
    Challenge the principles andpractices we take for granted
  • 55.
  • 56.
    Where are decisionsmade? Who can take action? escalate 1° 2° 3° 4° escalate escalateor
  • 57.
    All work iscontextual John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 58.
    All work iscontextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 59.
    All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 60.
    All work iscontextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 61.
    All work iscontextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 62.
    All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 63.
    All work iscontextual rm -rf $PATHNAME John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 64.
    All work iscontextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw DevOps Enterprise Summit San Francisco 2017
  • 65.
    escalate 1° 2° 3°4° escalate escalateor Context Where are decisions made? Who can take action? They have all of the context But, decisions are made here?
  • 66.
    “Shift Left” theability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
  • 68.
  • 70.
  • 72.
  • 74.
  • 76.
  • 77.
    What gets inthe way? Silos Queues Toil Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools ?? Silo A Silo B Ticket Queue
  • 78.
  • 79.
    Backlog Information I needX PrioritiesTools Silos
  • 80.
    Backlog Information I needX PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
  • 81.
    Silos cause disconnectsand mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
  • 82.
    Function A Function B FunctionC Becomes siloed labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
  • 83.
    How do wecover for our silos disconnects and mismatches? Silo A Silo B
  • 84.
    How do wecover for our silos disconnects and mismatches? Silo A Silo B Ticket Queue
  • 85.
    ?? Silo A SiloB We all know how well that works Ticket Queue
  • 86.
    Ticket queues arean expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
  • 87.
    Tickets queues become“snowflake makers” ?? Silo A Silo B Ticket Queue
  • 88.
    Tickets queues become“snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes (each unique, technically acceptable but unreproducible and brittle)
  • 89.
    Silos + TicketQueues = Locked into a broken system Unreproducible Snowflakes More Outages Longer Lead Times / Unpredictibility Siloed Labor Pools Ticket Queues Diminished Labor Capacity Brittle Environments Errors Error-Prone ManualTasks Interrupt project work Information Mismatches / Miscommunication Silo-Specific Optimization Handoffs With Capacity, Pace, Priority Mismatches “Managing to the Queue” Working out of Context “Align by Functional Capability” Management Strategy
  • 90.
    Excessive toil preventsfixing the system
  • 91.
    Excessive toil preventsfixing the system “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
  • 92.
    Excessive Toil preventsfixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
  • 93.
    Excessive Toil preventsfixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
  • 94.
    So what canwe do differently?
  • 95.
    Obvious: Get ridof as many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
  • 96.
    Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Obvious: Get rid of as many silos as possible “Horizontal” shared responsibility is key feature!
  • 97.
    But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
  • 98.
    But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
  • 99.
    But what aboutthe cross-cutting concerns? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
  • 100.
    Operations as aService: Turn handoffs into self-service Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
  • 101.
    Development Team 1 DevelopmentTeam 2 Development Team n Ops/SRE Team Operations as a Service On Demand On Demand On Demand On Demand Ops (builds & operates) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Operations as a Service: Works with any org model
  • 102.
    Use tickets onlyfor what they are good for Ticket System
  • 103.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions Ticket System
  • 104.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System
  • 105.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System
  • 106.
    Security or compliance“in the way”? Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Build-in Security Here Build-in Compliance Here
  • 107.
  • 108.
    Reduce Toil 1. Tracktoil levels for each team
  • 109.
    Reduce Toil 1. Tracktoil levels for each team 2. Set toil limits for each team
  • 110.
    Reduce Toil 1. Tracktoil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits)
  • 111.
    Reduce Toil 1. Tracktoil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits) Bonus: Use Service Level Objectives, Error Budgets, and other lessons from SRE
  • 112.
    Recap Don’t forget aboutOps. Challenge conventional wisdom. Team A (Dev) Team B (Ops) Ticket System ?? Leverage the Operations as a Service design pattern “Shift-Left” control and decision making. Understand the cost of silos and ticket-driven request queues Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silos and queues Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Learn from SRE: Reduce toil to create capacity to change Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”
  • 113.