Rundeck, profile picture
Uploaded byRundeck
2,906 views

Clearing the Way For SRE In the Enterprise

The document discusses the importance of Site Reliability Engineering (SRE) in addressing operational challenges faced by enterprises, emphasizing the need for service level objectives and error budgets to facilitate shared responsibility. It highlights the detrimental effects of silos, queues, toil, and low trust on operational efficiency and the implementation of SRE principles. The document advocates for systematic improvements and increased capacity for engineering work to enhance operational excellence.

Embed presentation

Downloaded 114 times
1 / 145
2 / 145
3 / 145
4 / 145
5 / 145
6 / 145
7 / 145
8 / 145
9 / 145
10 / 145
11 / 145
12 / 145
13 / 145
14 / 145
15 / 145
16 / 145
17 / 145
18 / 145
19 / 145
20 / 145
21 / 145
22 / 145
23 / 145
24 / 145
25 / 145
26 / 145
27 / 145
28 / 145
29 / 145
30 / 145
31 / 145
32 / 145
33 / 145
34 / 145
35 / 145
36 / 145
37 / 145
38 / 145
39 / 145
40 / 145
41 / 145
42 / 145
43 / 145
44 / 145
45 / 145
46 / 145
47 / 145
48 / 145
49 / 145
50 / 145
51 / 145
52 / 145
53 / 145
54 / 145
55 / 145
56 / 145
57 / 145
58 / 145
59 / 145
60 / 145
61 / 145
62 / 145
63 / 145
64 / 145
65 / 145
66 / 145
67 / 145
68 / 145
69 / 145
70 / 145
71 / 145
72 / 145
73 / 145
74 / 145
75 / 145
76 / 145
77 / 145
78 / 145
79 / 145
80 / 145
81 / 145
82 / 145
83 / 145
84 / 145
85 / 145
86 / 145
87 / 145
88 / 145
89 / 145
90 / 145
91 / 145
92 / 145
93 / 145
94 / 145
95 / 145
96 / 145
97 / 145
98 / 145
99 / 145
100 / 145
101 / 145
102 / 145
103 / 145
104 / 145
105 / 145
106 / 145
107 / 145
108 / 145
109 / 145
110 / 145
111 / 145
112 / 145
113 / 145
114 / 145
115 / 145
116 / 145
117 / 145
118 / 145
119 / 145
120 / 145
121 / 145
122 / 145
123 / 145
124 / 145
125 / 145
126 / 145
127 / 145
128 / 145
129 / 145
130 / 145
131 / 145
132 / 145
133 / 145
134 / 145
135 / 145
136 / 145
137 / 145
138 / 145
139 / 145
140 / 145
141 / 145
142 / 145
143 / 145
144 / 145
145 / 145
Clearing the Way For SRE in the Enterprise Damon Edwards @damonedwards
Community Ops Improvement DevOps Ops Tools Damon Edwards
Digital Agile DevOps CI/CD Cloud Docker Kubernetes Microservices CHANGE Wow That is cool I wish I could work there
OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" “Open up!” “Lock it down!” “Great for Dev, but what about Ops?”
Our transformation has largely ignored Ops. Any ideas? Have you heard of SRE? Google does it.
Jane Doe Systems Administrator
Jane Doe Systems Administrator We have SysAdmins
Jane Doe Systems Administrator They should be SREs!
Jane Doe SRE They should be SREs!
ITIL Book 1 ITIL Book 2 ITIL Book 3 ITIL Book 4 ITIL Book 5 Quality! is job #1 Sys Admin CAB CALENDAR Your new title is SRE. Now write code and be better at ops. PROVISIONING PROCESS Dilbert characters © Scott Adams www.dilbert.com
SysAdmins Overloaded. Constant firefighting. Waiting in ticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. ansformation has largely nored Ops. Any ideas? Have you heard of SRE? Google does it. Everything takes too long, cost too much, and break too often! Executive View
SysAdmins Overloaded. Constant firefighting. Waiting in ticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. ansformation has largely nored Ops. Any ideas? Have you heard of SRE? Google does it. Everything takes too long, cost too much, and break too often! Executive View (False) SRE Overloaded. Constant firefighting. Waiting in ticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. Our transformation has largely ignored Ops. Any ideas? Have you h Google Everything takes too long, cost too much, and break too often! Executive View
Changing job titles or adding individual skills doesn’t make systems administrators SREs.
Principles of SRE are what set SRE apart
Principles of SRE are what set SRE apart 1. SRE needs Service Level Objectives, with consequences
Principles of SRE are what set SRE apart 1. SRE needs Service Level Objectives, with consequences
SLO and Error Budgets: Tools for Shared Responsibility 0 100 Service Level Objective Error Budget* Service Level Indicator (*Use this to improve the service)
SLO and Error Budgets: Tools for Shared Responsibility 0 100 Service Level Objective Error Budget* Service Level Indicator (*Use this to improve the service) DEV BIZ Ops
Principles of SRE are what set SRE apart 1. SRE needs Service Level Objectives, with consequences
Principles of SRE are what set SRE apart 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today
Principles of SRE are what set SRE apart 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today 3. SRE teams have the ability to regulate their workload
Principles of SRE are what set SRE apart Stephen Thorne At DevOps Enterprise Summit London 2018 “Principles of SRE” https://youtu.be/c-w_GYvi0eA 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today 3. SRE teams have the ability to regulate their workload
Forces That Undermine SRE Principles Silos Queues Excessive Toil Low Trust
Forces That Undermine SRE Principles Silos Queues Excessive Toil Low Trust
Silos Backlog Information PrioritiesTools
Backlog Information I need X PrioritiesTools Silos
Backlog Information I need X PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
Silos cause disconnects and mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
1 2 3 Silos Interfere with feedback loops
1 2 3 Silos Interfere with feedback loops Producer Consumer Ops Ops Ops
Function A Function B Function C Silos create labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
Silos Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
Silos Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X
Silos Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X Siloed labor pools, disconnected processes and tools, and slow feedback loops tend to consume all available capacity X
Silos Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X Siloed labor pools, disconnected processes and tools, and slow feedback loops tend to consume all available capacity X Struggling to keep up with demand and unable to protect capacityX
Forces That Undermine SRE Principles Silos Queues Toil Low Trust
How do we cover for our cross-silo disconnects and mismatches? Silo A Silo B
How do we cover for our cross-silo disconnects and mismatches? Silo A Silo B Ticket Queue
?? Silo A Silo B We all know how well that works Ticket Queue
Request queues are an expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
What do queues do to value streams?
What do queues do to value streams? Queue A Queue B
What do queues do to value streams? Queue A Queue B Queues disintegrate and obfuscate value streams
Tickets queues become “snowflake makers” ?? Silo A Silo B Ticket Queue
Tickets queues become “snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes (each unique, technically acceptable but unreproducible and brittle)
Ticket Queues Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
Ticket Queues Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X
Ticket Queues Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X Longer cycle time, more variability, more overhead, lower quality, and more snowflakes consume available capacity X
Ticket Queues Undermine SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X Longer cycle time, more variability, more overhead, lower quality, and more snowflakes consume available capacity X Queues obfuscate the pressure being put on request fulfillersX
Forces That Undermine Operations Silos Queues Toil Low Trust
Toil is the enemy of SRE
Toil is the enemy of SRE “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
Toil vs. Engineering Work Toil Engineering Work Lacks Enduring Value Builds Enduring Value Rote, Repetitive Creative, Iterative Tactical Strategic Increases With Scale Enables Scaling Can Be Automated Requires Human Creativity
Excessive toil prevents fixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
Excessive toil prevents fixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
Excessive Toil Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
Excessive Toil Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X
Excessive Toil Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X Buried in toil… no capacity for engineering work to reduce toil.X
Excessive Toil Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X Buried in toil… no capacity for engineering work to reduce toil.X Buried in toil… no capacity for engineering work to reduce toil.X
Forces That Undermine Operations Silos Queues Toil Low Trust
Where are decisions made? Who can take action? escalate 1° 2° 3° 4° escalate escalateor Decisions made here
All work is contextual John Allspaw
All work is contextual rm -rf $PATHNAME John Allspaw
All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw
All work is contextual rm -rf $PATHNAME John Allspaw
All work is contextual rm -rf $PATHNAME John Allspaw
All work is contextual rm -rf $PATHNAME Is this dangerous? John Allspaw
All work is contextual rm -rf $PATHNAME John Allspaw
All work is contextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw
escalate 1° 2° 3° 4° escalate escalateor Context Where are decisions made? Who can take action?
Low trust + approvals = illusion of control Ticket System
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”)
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”)
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with?
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call?
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call? How many got rejected?
Low trust + approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call? How many got rejected?
Low Trust Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
Low Trust Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X
Low Trust Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X People closest to problems know what to fix but tasking, priorities, and decisions are largely out of their control X
Low Trust Undermines SRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X People closest to problems know what to fix but tasking, priorities, and decisions are largely out of their control X People aren’t trusted to plan or design their own workX
Forces That Undermine Operations Silos Queues Toil Low Trust
So what can we do differently?
Lean on Lean to find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan)
Lean on Lean to find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple
Lean on Lean to find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery
Lean on Lean to find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery Look to Lean for proven improvement techniques (value stream mapping, waste analysis, improvement kata)
Lean on Lean to find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery Look to Lean for proven improvement techniques (value stream mapping, waste analysis, improvement kata) Make it a part of your organization’s discipline
Get rid of as many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible
Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible Key 1: get rid of as many handoffs as possible
Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible Key 2: “Horizontal” shared responsibility, not everyone do everything! Key 1: get rid of as many handoffs as possible
Shared responsibility matters more than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model
Shared responsibility matters more than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model
Shared responsibility matters more than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model Same high-quality, high-velocity results!
Why focus on getting rid of handoffs?
Why focus on getting rid of handoffs? 1. Your people are your most valuable assets
Why focus on getting rid of handoffs? 1. Your people are your most valuable assets 2. The SRE skillset is expensive
Why focus on getting rid of handoffs? 1. Your people are your most valuable assets 2. The SRE skillset is expensive 3. Stay out of their way!
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This:
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Observe Orient Decide Action SRE OODA Loop Reduce friction:
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Observe Orient Decide Action SRE OODA Loop Reduce friction:
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Observe Orient Decide Action SRE OODA Loop Reduce friction:
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Empower them to make decisions! Observe Orient Decide Action SRE OODA Loop Reduce friction:
SREs are expensive, stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Empower them to make decisions! Empower them to take action! Observe Orient Decide Action SRE OODA Loop Reduce friction:
What about the handoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
What about the handoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
What about the handoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
Operations as a Service: Turn handoffs into self-service Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
Development Team 1 Development Team 2 Development Team n Ops/SRE Team Operations as a Service On Demand On Demand On Demand On Demand Ops (builds & operates) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Operations as a Service: Works with any org model
Operations as a Service: Popular Uses for SRE Environment "I could fix it, if I could get to it”
Operations as a Service: Popular Uses for SRE Environment "I could fix it, if I could get to it” Environment O a a S
Operations as a Service: Popular Uses for SRE “Avoiding the dogpile” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “$ top” “$ top” “$ top” “$ top” “$ top”“$ top”
Operations as a Service: Popular Uses for SRE “Avoiding the dogpile” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “$ top” “$ top” “$ top” “$ top” “$ top”“$ top” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “Healthcheck store2 - all” OaaS
“I don’t read wikis. I’m an expert.” docs Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this. Environment docs Later… Operations as a Service: Popular Uses for SRE
“I don’t read wikis. I’m an expert.” docs Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this. Environment docs Later… OaaS Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart” I’ve done this before. I’ve got this. Environment Later… Update Restart Job ✅ OaaS Operations as a Service: Popular Uses for SRE
Operations as a Service: Popular Uses for SRE “Uneven and hidden skills” I don’t know how to do X. I know how to do X. I know how to do Y. I don’t know how to do Y.
Operations as a Service: Popular Uses for SRE “Uneven and hidden skills” I don’t know how to do X. I know how to do X. I know how to do Y. I don’t know how to do Y. OaaS “Do X” “Define Y Procedure” “Define X Procedure” “Do Y” “Do X+Y”
“Let me do that for you again… and again” Done. I need you to do X Later… Ticket Other work Done. I need you to do X Later… Ticket Other work Sigh..Done. I need you to do X Ticket Other work Operations as a Service: Popular Uses for SRE
“Let me do that for you again… and again” Done. I need you to do X Later… Ticket Other work Done. I need you to do X Later… Ticket Other work Sigh..Done. I need you to do X Ticket Other work OaaS Do X Later… Other work 1 Later… Other work 2 Other work 3 Do X Do X OaaS OaaS Operations as a Service: Popular Uses for SRE
Use tickets only for what they are good for Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System
Use tickets only for what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System
But won’t Security or Compliance stop you? Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Build-in Security Here Build-in Compliance Here
But what about ITIL® ?
But what about ITIL® ? • Ask ITIL people and they say SRE is ITIL compatible
But what about ITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?”
But what about ITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature
But what about ITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature • ITIL “Standard Change” is often focus of discussion, but it still implies approval model
But what about ITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature • ITIL “Standard Change” is often focus of discussion, but it still implies approval model • Straight talk: are we doing contortions to defend a sunk cost?
“Shift Left” the ability to take action escalate 1° 2° 3° 4° escalate escalateor
“Shift Left” the ability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
“Shift Left” the ability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor OaaS Enablement and tooling
Reduce Toil
Reduce Toil 1. Track toil levels for each team
Reduce Toil 1. Track toil levels for each team 2. Set toil limits for each team
Reduce Toil 1. Track toil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits)
Start a book club
Recap SRE is more than a title Leverage the Operations as a Service design pattern “Shift-Left” control and decision making. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silos and queues Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Reduce toil to create capacity to change Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Understand the forces undermining SRE ITIL Book 1 ITIL Book 2 ITIL Book 3 ITIL Book 4 ITIL Book 5 Quality! is job #1 Sys Admin CAB CALENDAR Your new title is SRE. Now write code and be better at ops. PROVISIONING PROCESS Dilbert characters © Scott Adams www.dilbert.com
Let’s talk… @damonedwards damon@rundeck.com https://www.rundeck.com/oaas Dive Deeper Into Operations as a Service:

More Related Content

PDF
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
PDF
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
PDF
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
PDF
SRE Lessons for the Enterprise
byRundeck
 
PDF
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 
PDF
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
PDF
Ops Happens: Improving Incident Response Using DevOps and SRE Practices
byRundeck
 
PDF
Operations: The Last Mile
byRundeck
 
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
SRE Lessons for the Enterprise
byRundeck
 
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
Ops Happens: Improving Incident Response Using DevOps and SRE Practices
byRundeck
 
Operations: The Last Mile
byRundeck
 

What's hot

PDF
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Operations: The Last Mile
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
PDF
Operations as a Service: Because Failure Still Happens
byRundeck
 
PDF
The "Ops" Side of DevSecOps
byRundeck
 
PDF
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
PDF
Self-Service Operations: Because Failure Still Happens (Developer Edition)
byRundeck
 
PDF
Self-Service Operations: Because Ops Still Happens
byRundeck
 
PDF
Operations: The Last Mile Problem For DevOps
byRundeck
 
PDF
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
byRundeck
 
PPTX
Mainframe Solutions Introduction
byMicro Focus
 
PDF
Innovation and Architecture
byAdrian Cockcroft
 
PDF
Leveraging Your Company's DevOps Transformation (AppSec USA 2014)
bydev2ops
 
PDF
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 
PDF
My History with Atlassian Tools, and Why I'm Moving to Studio
byAtlassian
 
PDF
examkiller 000-938
byjimenoon
 
PPTX
The 7 Principles of DevOps and Cloud Applications
bySolarWinds
 
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Operations: The Last Mile
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
Operations as a Service: Because Failure Still Happens
byRundeck
 
The "Ops" Side of DevSecOps
byRundeck
 
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
Self-Service Operations: Because Failure Still Happens (Developer Edition)
byRundeck
 
Self-Service Operations: Because Ops Still Happens
byRundeck
 
Operations: The Last Mile Problem For DevOps
byRundeck
 
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
byRundeck
 
Mainframe Solutions Introduction
byMicro Focus
 
Innovation and Architecture
byAdrian Cockcroft
 
Leveraging Your Company's DevOps Transformation (AppSec USA 2014)
bydev2ops
 
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 
My History with Atlassian Tools, and Why I'm Moving to Studio
byAtlassian
 
examkiller 000-938
byjimenoon
 
The 7 Principles of DevOps and Cloud Applications
bySolarWinds
 

Similar to Clearing the Way For SRE In the Enterprise

PDF
2019-11 NewOpsDays Dallas - Sysadmin to SRE _v1.1
byJorn Knuttila
 
PDF
NewOps Days Boston 2019 - SysAdmin to SRE: Creating Capacity to Make Tomorrow...
byJorn Knuttila
 
PDF
S.R.E - create ultra-scalable and highly reliable systems
byRicardo Amaro
 
PPTX
DevOps Torino Meetup - SRE Concepts
byRauno De Pasquale
 
PDF
Kks sre book_ch1,2
byChris Huang
 
PPTX
A Crash Course in Building Site Reliability
byAcquia
 
PDF
stackconf 2022: Want to start with SRE? Start with this talk.
byNETWAYS
 
PPTX
SRE (service reliability engineer) on big DevOps platform running on the clou...
byDevClub_lv
 
PDF
Site-Reliability-Engineering-v2[6241].pdf
byDeepakGupta747774
 
PDF
Site Reliability Engineering: An Enterprise Adoption Story (an ITSM Academy W...
byITSM Academy, Inc.
 
PPTX
Site (Service) Reliability Engineering
byMark Underwood
 
PDF
SRE - drupal day aveiro 2016
byRicardo Amaro
 
PDF
SRE Roundtable with 4 DevOps Ambassadors
byITSM Academy, Inc.
 
PDF
Getting started with Site Reliability Engineering (SRE)
byAbeer R
 
PDF
Sre summary
byYogesh Shah
 
PDF
GCP-pdevops devops engineer exam prepearitaon guide
byskooldevops
 
PPTX
DevOps & Site Reliability Engineering (SRE).pptx
byabiguimeleroy
 
PDF
VS Live Las Vegas - When Down is not good enough - SRE on Azure
byRene Van Osnabrugge
 
PPTX
Site reliability engineering
byJason Loeffler
 
PDF
Site Reliability Engineering slide deck 101
byManikumarKothapalli1
 
2019-11 NewOpsDays Dallas - Sysadmin to SRE _v1.1
byJorn Knuttila
 
NewOps Days Boston 2019 - SysAdmin to SRE: Creating Capacity to Make Tomorrow...
byJorn Knuttila
 
S.R.E - create ultra-scalable and highly reliable systems
byRicardo Amaro
 
DevOps Torino Meetup - SRE Concepts
byRauno De Pasquale
 
Kks sre book_ch1,2
byChris Huang
 
A Crash Course in Building Site Reliability
byAcquia
 
stackconf 2022: Want to start with SRE? Start with this talk.
byNETWAYS
 
SRE (service reliability engineer) on big DevOps platform running on the clou...
byDevClub_lv
 
Site-Reliability-Engineering-v2[6241].pdf
byDeepakGupta747774
 
Site Reliability Engineering: An Enterprise Adoption Story (an ITSM Academy W...
byITSM Academy, Inc.
 
Site (Service) Reliability Engineering
byMark Underwood
 
SRE - drupal day aveiro 2016
byRicardo Amaro
 
SRE Roundtable with 4 DevOps Ambassadors
byITSM Academy, Inc.
 
Getting started with Site Reliability Engineering (SRE)
byAbeer R
 
Sre summary
byYogesh Shah
 
GCP-pdevops devops engineer exam prepearitaon guide
byskooldevops
 
DevOps & Site Reliability Engineering (SRE).pptx
byabiguimeleroy
 
VS Live Las Vegas - When Down is not good enough - SRE on Azure
byRene Van Osnabrugge
 
Site reliability engineering
byJason Loeffler
 
Site Reliability Engineering slide deck 101
byManikumarKothapalli1
 

More from Rundeck

PDF
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
PPTX
Introducing PagerDuty Process Automation
byRundeck
 
PDF
How to Build a Custom Plugin in Rundeck
byRundeck
 
PDF
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
PDF
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
PDF
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
PPTX
Mastering Secrets Management in Rundeck
byRundeck
 
PDF
What's New in Rundeck 3.4
byRundeck
 
PDF
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
PDF
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
PPTX
Introduction to Rundeck
byRundeck
 
PPTX
Automated Remediation with Rundeck + Sensu
byRundeck
 
PDF
Modernizing Incident Response
byRundeck
 
PDF
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
PDF
Datadog + Rundeck at DASH 2020
byRundeck
 
PDF
Rundeck Overview
byRundeck
 
PDF
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PPTX
Advanced Cluster Settings
byRundeck
 
PDF
Maximizing Your Rundeck Migration
byRundeck
 
PDF
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
Introducing PagerDuty Process Automation
byRundeck
 
How to Build a Custom Plugin in Rundeck
byRundeck
 
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
Mastering Secrets Management in Rundeck
byRundeck
 
What's New in Rundeck 3.4
byRundeck
 
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
Introduction to Rundeck
byRundeck
 
Automated Remediation with Rundeck + Sensu
byRundeck
 
Modernizing Incident Response
byRundeck
 
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
Datadog + Rundeck at DASH 2020
byRundeck
 
Rundeck Overview
byRundeck
 
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
Advanced Cluster Settings
byRundeck
 
Maximizing Your Rundeck Migration
byRundeck
 
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 

Recently uploaded

PPTX
The Landscape of Computer Software Development Companies
byYash Singh
 
PDF
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
PDF
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
PDF
8 LLM Surveys https://tinyurl.com/bdz8e6fp
byBob Marcus
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PDF
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
PDF
Innovative AI Solutions for Business Growth.pdf
byVox Works
 
PPTX
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
PDF
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
PDF
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 
PDF
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
PDF
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
PDF
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
PDF
Incident Response Planning with a Foundation Model
byKim Hammar
 
PDF
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
PPTX
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
PDF
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 
The Landscape of Computer Software Development Companies
byYash Singh
 
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
8 LLM Surveys https://tinyurl.com/bdz8e6fp
byBob Marcus
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
Innovative AI Solutions for Business Growth.pdf
byVox Works
 
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
TPPmark2025 Kenta Inoue's answer 12/04/2025
byKenta Inoue
 
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
Incident Response Planning with a Foundation Model
byKim Hammar
 
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 

Clearing the Way For SRE In the Enterprise

  • 1.
    Clearing the Way ForSRE in the Enterprise Damon Edwards @damonedwards
  • 2.
  • 3.
  • 4.
    OpsBusiness Idea Shorter Time-to-Market Fast Feedback fromUsers Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" “Open up!” “Lock it down!” “Great for Dev, but what about Ops?”
  • 5.
    Our transformation haslargely ignored Ops. Any ideas? Have you heard of SRE? Google does it.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
    ITIL Book 1 ITILBook 2 ITIL Book 3 ITIL Book 4 ITIL Book 5 Quality! is job #1 Sys Admin CAB CALENDAR Your new title is SRE. Now write code and be better at ops. PROVISIONING PROCESS Dilbert characters © Scott Adams www.dilbert.com
  • 11.
    SysAdmins Overloaded. Constant firefighting. Waiting inticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. ansformation has largely nored Ops. Any ideas? Have you heard of SRE? Google does it. Everything takes too long, cost too much, and break too often! Executive View
  • 12.
    SysAdmins Overloaded. Constant firefighting. Waiting inticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. ansformation has largely nored Ops. Any ideas? Have you heard of SRE? Google does it. Everything takes too long, cost too much, and break too often! Executive View (False) SRE Overloaded. Constant firefighting. Waiting in ticket queues for everything. Things break. Break again. And again. Everyone is busy, but it doesn’t get any better. Our transformation has largely ignored Ops. Any ideas? Have you h Google Everything takes too long, cost too much, and break too often! Executive View
  • 13.
    Changing job titlesor adding individual skills doesn’t make systems administrators SREs.
  • 14.
    Principles of SREare what set SRE apart
  • 15.
    Principles of SREare what set SRE apart 1. SRE needs Service Level Objectives, with consequences
  • 16.
    Principles of SREare what set SRE apart 1. SRE needs Service Level Objectives, with consequences
  • 17.
    SLO and ErrorBudgets: Tools for Shared Responsibility 0 100 Service Level Objective Error Budget* Service Level Indicator (*Use this to improve the service)
  • 18.
    SLO and ErrorBudgets: Tools for Shared Responsibility 0 100 Service Level Objective Error Budget* Service Level Indicator (*Use this to improve the service) DEV BIZ Ops
  • 19.
    Principles of SREare what set SRE apart 1. SRE needs Service Level Objectives, with consequences
  • 20.
    Principles of SREare what set SRE apart 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today
  • 21.
    Principles of SREare what set SRE apart 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today 3. SRE teams have the ability to regulate their workload
  • 22.
    Principles of SREare what set SRE apart Stephen Thorne At DevOps Enterprise Summit London 2018 “Principles of SRE” https://youtu.be/c-w_GYvi0eA 1. SRE needs Service Level Objectives, with consequences 2. SREs have time to make tomorrow better than today 3. SRE teams have the ability to regulate their workload
  • 23.
    Forces That UndermineSRE Principles Silos Queues Excessive Toil Low Trust
  • 24.
    Forces That UndermineSRE Principles Silos Queues Excessive Toil Low Trust
  • 25.
  • 26.
    Backlog Information I needX PrioritiesTools Silos
  • 27.
    Backlog Information I needX PrioritiesTools Silos Backlog I do X Requests for X Silo A Information Priorities Silo B Tools
  • 28.
    Silos cause disconnectsand mismatches Backlog Information I need X PrioritiesTools Backlog I do X Requests for X Silo A Information Priorities Silo B Tools Context Context Process Process Tooling Tooling Capacity Capacity
  • 29.
  • 30.
    1 2 3 Silos Interfere withfeedback loops Producer Consumer Ops Ops Ops
  • 31.
    Function A Function B FunctionC Silos create labor pools of functional specialists Requests fulfilled by semi- manual or manual effort Primary management focus is on protecting team capacity
  • 32.
    Silos Undermine SREPrinciples 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
  • 33.
    Silos Undermine SREPrinciples 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X
  • 34.
    Silos Undermine SREPrinciples 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X Siloed labor pools, disconnected processes and tools, and slow feedback loops tend to consume all available capacity X
  • 35.
    Silos Undermine SREPrinciples 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Disjointed silos make meaningful SLOs and shared responsibility almost impossible X Siloed labor pools, disconnected processes and tools, and slow feedback loops tend to consume all available capacity X Struggling to keep up with demand and unable to protect capacityX
  • 36.
    Forces That UndermineSRE Principles Silos Queues Toil Low Trust
  • 37.
    How do wecover for our cross-silo disconnects and mismatches? Silo A Silo B
  • 38.
    How do wecover for our cross-silo disconnects and mismatches? Silo A Silo B Ticket Queue
  • 39.
    ?? Silo A SiloB We all know how well that works Ticket Queue
  • 40.
    Request queues arean expensive way to manage work Ticket Queue Queues Create… Longer Cycle Time Increased Risk More Variability More Overhead Lower Quality Less Motivation Adapted from Donald G. Reinertsen, The Principles of Product Development Flow: Second Generation Lean Product Development
  • 41.
    What do queuesdo to value streams?
  • 42.
    What do queuesdo to value streams? Queue A Queue B
  • 43.
    What do queuesdo to value streams? Queue A Queue B Queues disintegrate and obfuscate value streams
  • 44.
    Tickets queues become“snowflake makers” ?? Silo A Silo B Ticket Queue
  • 45.
    Tickets queues become“snowflake makers” ?? Silo A Silo B Ticket Queue Snowflakes (each unique, technically acceptable but unreproducible and brittle)
  • 46.
    Ticket Queues UndermineSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
  • 47.
    Ticket Queues UndermineSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X
  • 48.
    Ticket Queues UndermineSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X Longer cycle time, more variability, more overhead, lower quality, and more snowflakes consume available capacity X
  • 49.
    Ticket Queues UndermineSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Tickets reinforce siloed behaviors and obfuscate the value stream X Longer cycle time, more variability, more overhead, lower quality, and more snowflakes consume available capacity X Queues obfuscate the pressure being put on request fulfillersX
  • 50.
    Forces That UndermineOperations Silos Queues Toil Low Trust
  • 51.
    Toil is theenemy of SRE
  • 52.
    Toil is theenemy of SRE “Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” -Vivek Rau Google
  • 53.
    Toil vs. EngineeringWork Toil Engineering Work Lacks Enduring Value Builds Enduring Value Rote, Repetitive Creative, Iterative Tactical Strategic Increases With Scale Enables Scaling Can Be Automated Requires Human Creativity
  • 54.
    Excessive toil preventsfixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
  • 55.
    Excessive toil preventsfixing the system Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”)
  • 56.
    Excessive Toil UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
  • 57.
    Excessive Toil UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X
  • 58.
    Excessive Toil UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X Buried in toil… no capacity for engineering work to reduce toil.X
  • 59.
    Excessive Toil UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Buried in toil keeps team from contributing engineering work to uphold their end of the shared responsibility deal X Buried in toil… no capacity for engineering work to reduce toil.X Buried in toil… no capacity for engineering work to reduce toil.X
  • 60.
    Forces That UndermineOperations Silos Queues Toil Low Trust
  • 61.
    Where are decisionsmade? Who can take action? escalate 1° 2° 3° 4° escalate escalateor Decisions made here
  • 62.
    All work iscontextual John Allspaw
  • 63.
    All work iscontextual rm -rf $PATHNAME John Allspaw
  • 64.
    All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw
  • 65.
    All work iscontextual rm -rf $PATHNAME John Allspaw
  • 66.
    All work iscontextual rm -rf $PATHNAME John Allspaw
  • 67.
    All work iscontextual rm -rf $PATHNAME Is this dangerous? John Allspaw
  • 68.
    All work iscontextual rm -rf $PATHNAME John Allspaw
  • 69.
    All work iscontextual rm -rf $PATHNAME Answer is always “it depends” John Allspaw
  • 70.
    escalate 1° 2° 3°4° escalate escalateor Context Where are decisions made? Who can take action?
  • 71.
    Low trust +approvals = illusion of control Ticket System
  • 72.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and
  • 73.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”)
  • 74.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”)
  • 75.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
  • 76.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
  • 77.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”)
  • 78.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with?
  • 79.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call?
  • 80.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call? How many got rejected?
  • 81.
    Low trust +approvals = illusion of control Ticket System Add up the total number of approval requests and …subtract the info radiators (“I need to be in the loop”) …subtract the CYAs (“Prove you followed the process”) …subtract the too removed to judge (“mostly guessing”) How many are you left with? How many were the right call? How many got rejected?
  • 82.
    Low Trust UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload?
  • 83.
    Low Trust UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X
  • 84.
    Low Trust UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X People closest to problems know what to fix but tasking, priorities, and decisions are largely out of their control X
  • 85.
    Low Trust UnderminesSRE Principles 1. Org has Service Level Objectives, with consequences? 2. SREs have time to make tomorrow better than today? 3. SRE teams have the ability to regulate their workload? Cultures of low trust have a really difficult time with shared responsibility X People closest to problems know what to fix but tasking, priorities, and decisions are largely out of their control X People aren’t trusted to plan or design their own workX
  • 86.
    Forces That UndermineOperations Silos Queues Toil Low Trust
  • 87.
    So what canwe do differently?
  • 88.
    Lean on Leanto find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan)
  • 89.
    Lean on Leanto find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple
  • 90.
    Lean on Leanto find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery
  • 91.
    Lean on Leanto find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery Look to Lean for proven improvement techniques (value stream mapping, waste analysis, improvement kata)
  • 92.
    Lean on Leanto find what to fix PD TS W EP M M M TS ? PD TS W EP M M M TS ? Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Countermeasure Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Lorem ipsum dolor In aliquet rhoncus urna. Proin eget diam volutpat. Map the end-to-end flow of information and artifacts (using a recent delivery or event) Identify what slows lead times, undermines quality, and impacts flow 1 2 3 Identify countermeasures and create improvement storyboards (justification/plan) All processes should be studied with an improvement disciple Incidents are just as much a “process” as delivery Look to Lean for proven improvement techniques (value stream mapping, waste analysis, improvement kata) Make it a part of your organization’s discipline
  • 93.
    Get rid ofas many silos as possible Old Silo A Old Silo B Old Silo C Old Silo D
  • 94.
    Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible
  • 95.
    Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible Key 1: get rid of as many handoffs as possible
  • 96.
    Old Silo AOld Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Get rid of as many silos as possible Key 2: “Horizontal” shared responsibility, not everyone do everything! Key 1: get rid of as many handoffs as possible
  • 97.
    Shared responsibility mattersmore than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model
  • 98.
    Shared responsibility mattersmore than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model
  • 99.
    Shared responsibility mattersmore than org model Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Development Team 1 Development Team 2 Development Team n SRE Team Clear handoff requirements Error budget consequences “Netflix" Model “Google” Model Same high-quality, high-velocity results!
  • 100.
    Why focus ongetting rid of handoffs?
  • 101.
    Why focus ongetting rid of handoffs? 1. Your people are your most valuable assets
  • 102.
    Why focus ongetting rid of handoffs? 1. Your people are your most valuable assets 2. The SRE skillset is expensive
  • 103.
    Why focus ongetting rid of handoffs? 1. Your people are your most valuable assets 2. The SRE skillset is expensive 3. Stay out of their way!
  • 104.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This:
  • 105.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Observe Orient Decide Action SRE OODA Loop Reduce friction:
  • 106.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Observe Orient Decide Action SRE OODA Loop Reduce friction:
  • 107.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Observe Orient Decide Action SRE OODA Loop Reduce friction:
  • 108.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Empower them to make decisions! Observe Orient Decide Action SRE OODA Loop Reduce friction:
  • 109.
    SREs are expensive,stay out of their way! Ticket Queue ✅Ticket Queue Ticket Queue Ticket Queue Backlog Ticket Queue Ticket Queue ✅ Backlog Not this: This: Invest in the right instrumentation Invest in collaboration, checklists, investigatory tools Empower them to make decisions! Empower them to take action! Observe Orient Decide Action SRE OODA Loop Reduce friction:
  • 110.
    What about thehandoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities
  • 111.
    What about thehandoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue
  • 112.
    What about thehandoffs you can’t get rid of? Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Specialist Capabilities Specialist Capabilities Specialist Capabilities Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue Ticket Queue
  • 113.
    Operations as aService: Turn handoffs into self-service Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist
  • 114.
    Development Team 1 DevelopmentTeam 2 Development Team n Ops/SRE Team Operations as a Service On Demand On Demand On Demand On Demand Ops (builds & operates) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Operations as a Service: Works with any org model
  • 115.
    Operations as aService: Popular Uses for SRE Environment "I could fix it, if I could get to it”
  • 116.
    Operations as aService: Popular Uses for SRE Environment "I could fix it, if I could get to it” Environment O a a S
  • 117.
    Operations as aService: Popular Uses for SRE “Avoiding the dogpile” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “$ top” “$ top” “$ top” “$ top” “$ top”“$ top”
  • 118.
    Operations as aService: Popular Uses for SRE “Avoiding the dogpile” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “$ top” “$ top” “$ top” “$ top” “$ top”“$ top” I think its a problem with dbcluster07-store2.uswest.acme dbcluster07- store2.uswest. acme “$ top” “Healthcheck store2 - all” OaaS
  • 119.
    “I don’t readwikis. I’m an expert.” docs Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this. Environment docs Later… Operations as a Service: Popular Uses for SRE
  • 120.
    “I don’t readwikis. I’m an expert.” docs Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart -doit -now” I’ve done this before. I’ve got this. Environment docs Later… OaaS Service has changed. This flag is now required or bad things will happen! Pause monitoring first or we all get woken up! “restart” I’ve done this before. I’ve got this. Environment Later… Update Restart Job ✅ OaaS Operations as a Service: Popular Uses for SRE
  • 121.
    Operations as aService: Popular Uses for SRE “Uneven and hidden skills” I don’t know how to do X. I know how to do X. I know how to do Y. I don’t know how to do Y.
  • 122.
    Operations as aService: Popular Uses for SRE “Uneven and hidden skills” I don’t know how to do X. I know how to do X. I know how to do Y. I don’t know how to do Y. OaaS “Do X” “Define Y Procedure” “Define X Procedure” “Do Y” “Do X+Y”
  • 123.
    “Let me dothat for you again… and again” Done. I need you to do X Later… Ticket Other work Done. I need you to do X Later… Ticket Other work Sigh..Done. I need you to do X Ticket Other work Operations as a Service: Popular Uses for SRE
  • 124.
    “Let me dothat for you again… and again” Done. I need you to do X Later… Ticket Other work Done. I need you to do X Later… Ticket Other work Sigh..Done. I need you to do X Ticket Other work OaaS Do X Later… Other work 1 Later… Other work 2 Other work 3 Do X Do X OaaS OaaS Operations as a Service: Popular Uses for SRE
  • 125.
    Use tickets onlyfor what they are good for Ticket System
  • 126.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions Ticket System
  • 127.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Ticket System
  • 128.
    Use tickets onlyfor what they are good for 1.Documenting true problems/issues/exceptions 2.Routing for necessary approvals Not as a general purpose work management system! Ticket System
  • 129.
    But won’t Securityor Compliance stop you? Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Build-in Security Here Build-in Compliance Here
  • 130.
  • 131.
    But what aboutITIL® ? • Ask ITIL people and they say SRE is ITIL compatible
  • 132.
    But what aboutITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?”
  • 133.
    But what aboutITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature
  • 134.
    But what aboutITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature • ITIL “Standard Change” is often focus of discussion, but it still implies approval model
  • 135.
    But what aboutITIL® ? • Ask ITIL people and they say SRE is ITIL compatible • Ask people who have seen ITIL implemented and they say “how?” • Agile+DevOps+SRE have self-regulation and shared responsibility features that seem to undermine ITIL command and control nature • ITIL “Standard Change” is often focus of discussion, but it still implies approval model • Straight talk: are we doing contortions to defend a sunk cost?
  • 136.
    “Shift Left” theability to take action escalate 1° 2° 3° 4° escalate escalateor
  • 137.
    “Shift Left” theability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor
  • 138.
    “Shift Left” theability to take action Push the ability to take action this direction escalate 1° 2° 3° 4° escalate escalateor OaaS Enablement and tooling
  • 139.
  • 140.
    Reduce Toil 1. Tracktoil levels for each team
  • 141.
    Reduce Toil 1. Tracktoil levels for each team 2. Set toil limits for each team
  • 142.
    Reduce Toil 1. Tracktoil levels for each team 2. Set toil limits for each team 3. Fund efforts to reduce toil (with emphasis on teams over toil limits)
  • 143.
  • 144.
    Recap SRE is morethan a title Leverage the Operations as a Service design pattern “Shift-Left” control and decision making. Old Silo A Old Silo B Old Silo C Old Silo D Cross-Functional Team 1 Cross-Functional Team 2 Cross-Functional Team n Focus on removing silos and queues Operations as a Service On Demand On Demand On Demand On Demand Ops (embedded)Cross-Functional Product Team 1 Cross-Functional Product Team n Ops (embedded) Ops (builds & operates) Cross-Functional Product Team 2 Ops (embedded) Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Ops Capability SRE, Dev, or Specialist Reduce toil to create capacity to change Toil Engineering Work E.W.Toil Reduce toil Improve the business ǡ No capacity to reduce toil No capacity to improve business Toil at manageable percentage of capacity Toil at unmanageable percentage of capacity (“Engineering Bankruptcy”) Understand the forces undermining SRE ITIL Book 1 ITIL Book 2 ITIL Book 3 ITIL Book 4 ITIL Book 5 Quality! is job #1 Sys Admin CAB CALENDAR Your new title is SRE. Now write code and be better at ops. PROVISIONING PROCESS Dilbert characters © Scott Adams www.dilbert.com
  • 145.