Rundeck, profile picture
Uploaded byRundeck
PDF, PPTX1,545 views

Self-Service Operations: Because Failure Still Happens (Developer Edition)

The document discusses the challenges in operations support within DevOps, highlighting the need for a shift towards self-service operations to reduce escalations and improve resolution times. It emphasizes the importance of integrating development and operations teams, adopting an 'operable first' mindset, and enabling self-service capabilities to streamline processes. Furthermore, it advocates for recognizing the pressures on operations and enhancing collaboration between developers and operations to achieve efficient service operations.

Embed presentation

Download as PDF, PPTX
Self-Service Operations: Because Failure Still Happens Damon Edwards @damonedwards
Damon Edwards Ops Improvement DevOps Consulting Tools Community
Deploy. Deploy. Deploy. Deploy. DevOps Dream
Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens! Escalations! Interruptions! Delays! Context Switching! Bridge Calls! Arguments!
Let’s start with a story…
Mark Maun Jody Mulkey Justin Dean
90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs
How did they do that?
But first…
Let’s look at the principles behind the improvement …
Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service “two-pizza team” Development Team Operations Team Dev Ops Integrated Delivery Team
“You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s the NOC… Talk them through: health checks, reviewing log files, and process of diagnosing and recovering the system. Same as you did for dev teams 2 months ago, QA teams last month, Ops during deploy last week, etc.
“You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm ….
“You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s Ops… “Will your applications be affected if we take down EU-West?” “How do we reconfigure your app if we change these firewall rules?” “We are getting customer complaints about performance. Help us figure out what changed”.
“You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
“You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
“You build it. You run it.” Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
“You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team “two-pizza teams”? Just change how business is structured, funded, and operated.
How do we get the best of both models?
Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models?
Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models? Have the responsiveness/control of “you build it, you run it” without the scaling limitations
“Self-Service Operations” design pattern is the key enabler
“Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor
“Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
What do Developers get out of Self-Service Operations?…
Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
Tighter feedback loops Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
How can Dev help Ops make this a reality…
First understand the pain of Operations today…
Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!"
Ops is Unplanned Work and Planned Work… by design! +
Planned + Unplanned = Expensive Context Switching Gerald Weinberg via Jeff Atwood https://blog.codinghorror.com/the-multi-tasking-myth/
Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
Cross functional teams? ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
Cross functional teams? Never enough to go around. Context Work Task Work Task Work Task Work Task Queue ! Handoffs ! Handoffs Team C ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
Silos + Tool Evolution = Islands of Automation Puppet Chef Shell Scripts Data ETL PowershellScripts Network Management Monitoring Ansible Legacy Datacenter Automation ContainerManagement SQL Tools NewTools New Tools
Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under
Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under = Ops never has enough time or people! +
How developers can help operations…
1. Recognize that great Operations starts in Development
2. Develop with an “Operable First” mindset
Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Develop with an “Operable First” mindset
Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests Develop with an “Operable First” mindset
3. “Shift Left” as much Ops activity as possible
“Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools “Deploy.” Ops Ah-ha! Dev Ka-ching!
“Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.” Ops Ah-ha! Dev Ka-ching!
4. Develop Self-Service Operations capabilities
Step 1: Establish a Secure Ops Hub Custom Mix
Step 2: Establish a SDLC for Ops Procedures Custom Mix
Step 3: Connect with Enterprise Management Systems Custom Mix
Who created the procedure? Who reviewed it? Who? When? Where? Approval trail? Step 4: Make Compliance Really Happy Custom Mix
Again: How can Developers help Operations 1. Recognize that great Operations starts in Development 2. Develop with an “Operable First” mindset 3. “Shift Left” as much Ops activity as possible 4. Develop Self-Service Operations capabilities
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge”
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations
Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response and escalation model
Recap Understand the pressures on Ops Explicit investment in process and tooling OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!" Understand your operating model Operations is the business) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification- driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests “Operable First” mindset Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Apply Self-Service Operations design pattern “Shift left” operational concerns
Let’s talk… @damonedwards damon@rundeck.com

More Related Content

PDF
Self-Service Operations: Because Ops Still Happens
byRundeck
 
PDF
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
PDF
Operations as a Service: Because Failure Still Happens
byRundeck
 
PDF
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
byRundeck
 
PDF
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
PDF
The "Ops" Side of DevSecOps
byRundeck
 
PDF
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
PDF
Ops Happens: Improving Incident Response Using DevOps and SRE Practices
byRundeck
 
Self-Service Operations: Because Ops Still Happens
byRundeck
 
Modern Operations: Solving DevOps’ Last Mile Problem
byRundeck
 
Operations as a Service: Because Failure Still Happens
byRundeck
 
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
byRundeck
 
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
byRundeck
 
The "Ops" Side of DevSecOps
byRundeck
 
Failure Happens: Improving Incident Response In Enterprises
byRundeck
 
Ops Happens: Improving Incident Response Using DevOps and SRE Practices
byRundeck
 

What's hot

PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
The Last Mile Continued: Incident Management
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
Incident Management in the Age of DevOps and SRE
byRundeck
 
PDF
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
PDF
Clearing the Way For SRE In the Enterprise
byRundeck
 
PDF
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
PDF
The History of DevOps (and what you need to do about it)
bydev2ops
 
PDF
SRE From Scratch
byGrier Johnson
 
PDF
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
PDF
Operational Insight: Concepts and Examples (w/o Presenter Notes)
byroyrapoport
 
PDF
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
PDF
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PDF
SRE Lessons for the Enterprise
byRundeck
 
PDF
Operations: The Last Mile
byRundeck
 
PDF
Operations: The Last Mile
byRundeck
 
PDF
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 
PDF
DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...
byMatt Ray
 
PDF
Leveraging Your Company's DevOps Transformation (AppSec USA 2014)
bydev2ops
 
PDF
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
The Last Mile Continued: Incident Management
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
Incident Management in the Age of DevOps and SRE
byRundeck
 
SRE for Everyone: Making Tomorrow Better Than Today
byRundeck
 
Clearing the Way For SRE In the Enterprise
byRundeck
 
SysAdmin to SRE: Solving the Last Mile Problem
byRundeck
 
The History of DevOps (and what you need to do about it)
bydev2ops
 
SRE From Scratch
byGrier Johnson
 
SysAdmin to SRE: Creating Capacity to Make Tomorrow Better Than Today
byRundeck
 
Operational Insight: Concepts and Examples (w/o Presenter Notes)
byroyrapoport
 
Tickets Make Operations Work Unnecessarily Miserable
byRundeck
 
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
SRE Lessons for the Enterprise
byRundeck
 
Operations: The Last Mile
byRundeck
 
Operations: The Last Mile
byRundeck
 
Making Tomorrow Better than Today - Unlocking the Full Potential of Operations
byRundeck
 
DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...
byMatt Ray
 
Leveraging Your Company's DevOps Transformation (AppSec USA 2014)
bydev2ops
 
8 Things That Make Continuous Delivery Go Nuts
byEduards Sizovs
 

Similar to Self-Service Operations: Because Failure Still Happens (Developer Edition)

PPTX
DevOps 101
byErnest Mueller
 
PDF
Introduction to DevOps slides.pdf
byBoreVishnusai
 
PDF
Introduction to DevOps
byRavindu Fernando
 
PDF
What is DevOps All You Need To Know.pdf
byCerebrum Infotech
 
PDF
An Ultimate Guide to DevOps – Principles, How it works, and Real-life Examples
byriyak40
 
PDF
Chris Munns, DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Depl...
byTriNimbus
 
PPTX
Devops
bypenetration Tester
 
PDF
Tickets Make Ops Unnecessarily Miserable: The Journey to Self-Service
byRundeck
 
PDF
Ops Happens: DevOps Beyond Deployment - Damon Edwards
bySeniorStoryteller
 
PPTX
JavaOne 2015 Devops and the Darkside CON6447
bySteve Poole
 
PPTX
DevOps Overview
byOmri Spector
 
PDF
DevOps The Cultural revolution
bySomenath Ghosh
 
PDF
MesosCon_Asia_2016_DevOps_at_GSShop
byVivek Juneja
 
PPTX
DevOps
byDawn Keenan
 
PPTX
Delivering Applications Continuously to Cloud
byIBM UrbanCode Products
 
PPT
IBM Innovate 2013 Session: DevOps 101
bySanjeev Sharma
 
PDF
Accelerate Delivery: Business Case for Agile DevOps, CI/CD and Microservices
byRick Hightower
 
PPT
DevOps 101 for Government
bySanjeev Sharma
 
PDF
Moving from ops to dev ops
byJeffery Smith
 
ODP
Devops, the future is here it's not evenly distributed yet
byKris Buytaert
 
DevOps 101
byErnest Mueller
 
Introduction to DevOps slides.pdf
byBoreVishnusai
 
Introduction to DevOps
byRavindu Fernando
 
What is DevOps All You Need To Know.pdf
byCerebrum Infotech
 
An Ultimate Guide to DevOps – Principles, How it works, and Real-life Examples
byriyak40
 
Chris Munns, DevOps @ Amazon: Microservices, 2 Pizza Teams, & 50 Million Depl...
byTriNimbus
 
Devops
bypenetration Tester
 
Tickets Make Ops Unnecessarily Miserable: The Journey to Self-Service
byRundeck
 
Ops Happens: DevOps Beyond Deployment - Damon Edwards
bySeniorStoryteller
 
JavaOne 2015 Devops and the Darkside CON6447
bySteve Poole
 
DevOps Overview
byOmri Spector
 
DevOps The Cultural revolution
bySomenath Ghosh
 
MesosCon_Asia_2016_DevOps_at_GSShop
byVivek Juneja
 
DevOps
byDawn Keenan
 
Delivering Applications Continuously to Cloud
byIBM UrbanCode Products
 
IBM Innovate 2013 Session: DevOps 101
bySanjeev Sharma
 
Accelerate Delivery: Business Case for Agile DevOps, CI/CD and Microservices
byRick Hightower
 
DevOps 101 for Government
bySanjeev Sharma
 
Moving from ops to dev ops
byJeffery Smith
 
Devops, the future is here it's not evenly distributed yet
byKris Buytaert
 

More from Rundeck

PDF
Rundeck Overview
byRundeck
 
PPTX
Introduction to Rundeck
byRundeck
 
PDF
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
PPTX
Introducing PagerDuty Process Automation
byRundeck
 
PDF
Datadog + Rundeck at DASH 2020
byRundeck
 
PDF
What's New in Rundeck 3.4
byRundeck
 
PDF
How to Build a Custom Plugin in Rundeck
byRundeck
 
PDF
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
PPTX
Automated Remediation with Rundeck + Sensu
byRundeck
 
PDF
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
PPTX
Mastering Secrets Management in Rundeck
byRundeck
 
PDF
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
PPTX
Advanced Cluster Settings
byRundeck
 
PDF
Maximizing Your Rundeck Migration
byRundeck
 
PDF
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
PDF
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
PDF
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
PDF
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
PDF
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 
PDF
Modernizing Incident Response
byRundeck
 
Rundeck Overview
byRundeck
 
Introduction to Rundeck
byRundeck
 
Self Service Cloud Operations: Safely Delegate the Management of your Cloud ...
byRundeck
 
Introducing PagerDuty Process Automation
byRundeck
 
Datadog + Rundeck at DASH 2020
byRundeck
 
What's New in Rundeck 3.4
byRundeck
 
How to Build a Custom Plugin in Rundeck
byRundeck
 
Rundeck Office Hours: Best Practices Access Control Policies
byRundeck
 
Automated Remediation with Rundeck + Sensu
byRundeck
 
Rundeck Community Office Hours: Using Variables with Job Steps
byRundeck
 
Mastering Secrets Management in Rundeck
byRundeck
 
Lunch and learn: Getting started with Rundeck & Ansible
byRundeck
 
Advanced Cluster Settings
byRundeck
 
Maximizing Your Rundeck Migration
byRundeck
 
Automate Yourself Out of a Job: Safely Delegate the Management of your Azure...
byRundeck
 
Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]
byRundeck
 
Super-Charge Your Site Reliability Practices with Runbook Automation
byRundeck
 
Empower Devs, Simplify Ops, and Accelerate your Digital Transformation
byRundeck
 
Business Continuity for Humans: Keeping Your Business Running When Your Peopl...
byRundeck
 
Modernizing Incident Response
byRundeck
 

Recently uploaded

PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
PDF
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
PPTX
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
PPTX
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
PPTX
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
PDF
Smart Cloud Solutions-Smart,Secure & Scalable
byfloydjsmith02
 
PDF
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
PPTX
Migrating-Hadoop-to-Databricks-Ebook-FINAL.pptx
byssuserf37fc8
 
PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PPTX
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
PDF
Introduction to Linux and Basic Commands
byiDev Semarang
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PPTX
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PDF
Ethical AI applied to publishing: Presenting wâsikan kisewâtisiwin, an AI too...
byBookNet Canada
 
PDF
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
PDF
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
PDF
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
PDF
Lightweight, Travel-Ready Freedom for Adults and Seniors
byTopmate
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
Smart Cloud Solutions-Smart,Secure & Scalable
byfloydjsmith02
 
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
Migrating-Hadoop-to-Databricks-Ebook-FINAL.pptx
byssuserf37fc8
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
Introduction to Linux and Basic Commands
byiDev Semarang
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
Ethical AI applied to publishing: Presenting wâsikan kisewâtisiwin, an AI too...
byBookNet Canada
 
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
Lightweight, Travel-Ready Freedom for Adults and Seniors
byTopmate
 

Self-Service Operations: Because Failure Still Happens (Developer Edition)

  • 1.
    Self-Service Operations: Because FailureStill Happens Damon Edwards @damonedwards
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
    Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh…we have a problem DevOps Reality Ops still happens! Escalations! Interruptions! Delays! Context Switching! Bridge Calls! Arguments!
  • 7.
  • 10.
  • 11.
    90% Reduction inMTTR 50% Reduction in escalations 55% Reduction of overall support costs
  • 12.
    How did theydo that?
  • 13.
  • 14.
    Let’s look atthe principles behind the improvement …
  • 15.
    Two schools ofthought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
  • 16.
    Two schools ofthought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
  • 17.
    Two schools ofthought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service “two-pizza team” Development Team Operations Team Dev Ops Integrated Delivery Team
  • 18.
    “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s the NOC… Talk them through: health checks, reviewing log files, and process of diagnosing and recovering the system. Same as you did for dev teams 2 months ago, QA teams last month, Ops during deploy last week, etc.
  • 19.
    “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm ….
  • 20.
    “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s Ops… “Will your applications be affected if we take down EU-West?” “How do we reconfigure your app if we change these firewall rules?” “We are getting customer complaints about performance. Help us figure out what changed”.
  • 21.
    “You build it.They run it.” (aka… the way it always was) Running Service Development Team Operations Team
  • 22.
    “You build it.They run it.” (aka… the way it always was) Running Service Development Team Operations Team
  • 23.
    “You build it.You run it.” Dev Ops Integrated Delivery Team
  • 24.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Dev Ops Integrated Delivery Team
  • 25.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  • 26.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  • 27.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  • 28.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  • 29.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  • 30.
    “You build it.You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team “two-pizza teams”? Just change how business is structured, funded, and operated.
  • 31.
    How do weget the best of both models?
  • 32.
    Have the laborscaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models?
  • 33.
    Have the laborscaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models? Have the responsiveness/control of “you build it, you run it” without the scaling limitations
  • 34.
    “Self-Service Operations” designpattern is the key enabler
  • 35.
    “Self-Service Operations” designpattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor
  • 36.
    “Self-Service Operations” designpattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
  • 37.
    What do Developersget out of Self-Service Operations?…
  • 38.
    Reduced escalations andquicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
  • 39.
    Reduced escalations andquicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
  • 40.
    Tighter feedback loops Self-Service Operations Defineactions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
  • 41.
    Ops managers canfocus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
  • 42.
    Ops managers canfocus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
  • 43.
    How can Devhelp Ops make this a reality…
  • 44.
    First understand thepain of Operations today…
  • 45.
    Operations is gettingsqueezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
  • 46.
    Operations is gettingsqueezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!"
  • 47.
    Ops is UnplannedWork and Planned Work… by design! +
  • 48.
    Planned + Unplanned= Expensive Context Switching Gerald Weinberg via Jeff Atwood https://blog.codinghorror.com/the-multi-tasking-myth/
  • 49.
    Crushing Technical Debt 900Dev 500 QA / App Sup. 200 Ops
  • 50.
    Crushing Technical Debt 900Dev 500 QA / App Sup. 200 Ops
  • 51.
    ContextContext Work Task Work Task WorkTask Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
  • 52.
    ContextContext Work Task Work Task WorkTask Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
  • 53.
    Cross functional teams? ContextContext WorkTask Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
  • 54.
    Cross functional teams?Never enough to go around. Context Work Task Work Task Work Task Work Task Queue ! Handoffs ! Handoffs Team C ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
  • 55.
    Silos + ToolEvolution = Islands of Automation Puppet Chef Shell Scripts Data ETL PowershellScripts Network Management Monitoring Ansible Legacy Datacenter Automation ContainerManagement SQL Tools NewTools New Tools
  • 56.
    Squeezed between competingpressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under
  • 57.
    Squeezed between competingpressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under = Ops never has enough time or people! +
  • 58.
    How developers canhelp operations…
  • 59.
    1. Recognize thatgreat Operations starts in Development
  • 60.
    2. Develop withan “Operable First” mindset
  • 61.
    Develop with an“Operable First” mindset
  • 62.
    Operations is thebusiness (unless you literally sell packaged software) Develop with an “Operable First” mindset
  • 63.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Develop with an “Operable First” mindset
  • 64.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Develop with an “Operable First” mindset
  • 65.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Develop with an “Operable First” mindset
  • 66.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Develop with an “Operable First” mindset
  • 67.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Develop with an “Operable First” mindset
  • 68.
    Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests Develop with an “Operable First” mindset
  • 69.
    3. “Shift Left”as much Ops activity as possible
  • 70.
    “Shift Left” asmuch Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools “Deploy.” Ops Ah-ha! Dev Ka-ching!
  • 71.
    “Shift Left” asmuch Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.” Ops Ah-ha! Dev Ka-ching!
  • 72.
    4. Develop Self-ServiceOperations capabilities
  • 73.
    Step 1: Establisha Secure Ops Hub Custom Mix
  • 74.
    Step 2: Establisha SDLC for Ops Procedures Custom Mix
  • 75.
    Step 3: Connectwith Enterprise Management Systems Custom Mix
  • 76.
    Who created theprocedure? Who reviewed it? Who? When? Where? Approval trail? Step 4: Make Compliance Really Happy Custom Mix
  • 77.
    Again: How canDevelopers help Operations 1. Recognize that great Operations starts in Development 2. Develop with an “Operable First” mindset 3. “Shift Left” as much Ops activity as possible 4. Develop Self-Service Operations capabilities
  • 78.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge”
  • 79.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams
  • 80.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy
  • 81.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks
  • 82.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations
  • 83.
    Back to ourStory… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response and escalation model
  • 84.
    Recap Understand the pressures onOps Explicit investment in process and tooling OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!" Understand your operating model Operations is the business) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification- driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests “Operable First” mindset Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Apply Self-Service Operations design pattern “Shift left” operational concerns
  • 85.