SlideShare a Scribd company logo
1 of 12
Download to read offline
“New hires should be kept off the on-call rotation until they 
know the platform 
inside-and-out.” 
“I just write the code 
and throw it over the wall. 
It’s up to Ops to make it work.” 
“It’s not my job 
to test the code. If there’s a mistake, 
talk to QA.” 
“I’ve never had to 
carry the pager before… 
why should I start now?”
Ideally, companies work out a system in which whoever makes the mistake pays the price. The reason Ops is so often scared of Dev deploying is that Dev doesn't really care how secure their apps are, how hard they are to deploy, how hard they are to keep running or how many times you have to restart it, because Ops pays the price for those mistakes, not Dev. 
In most organizations the mandate of a developer is merely to produce a piece of software that worked on a workstation -- if it worked on your workstation and you can't make it work in production, it's Operations' fault if they can't get that to thousands of machines all around the world. 
http://readwrite.com/2011/08/23/devops-what-it-is-why-it-exist
According to Ernest Mueller, one of 
the authors of the Agile Admin: 
DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support. DevOps means breaking down the traditional silos that have existed between Ops and Devs. 
DevOps Practices-Specific techniques used 
as part of implementing the above concepts 
and processes. 
Continuous integration and 
continuous deployment 
“Give your developers a pager 
and put them on-call” 
Using Configuration management, 
metrics and monitoring schemes 
Using virtualization and cloud 
computing to accelerate change 
http://theagileadmin.com/what-is-devops/
Devs can win instant good will by helping to carry one of Ops’ biggest burdens: the pager. Developers tend to bring more craftsmanship into their work knowing that they, or their pals, might suffer the consequences of half-baked code. Being waist-deep in the immediate resolution and accompanying troubleshooting efforts also provides a valuable perspective on how the application’s architecture interacts with the infrastructure it sits on. 
Clearly define (and yes: document) escalation paths for emergencies and make sure devs have access to the tools and data they’ll need for troubleshooting. Then include the development team in on-call rotation. 
https://www.atlassian.com/devops/culture#!shared-responsibilities
http://www.slideshare.net/mrtazz/goto-amsterdam2014depolyment1024 
Etsy has devs on-call to address the problem of IT Operations asking, 
“Why should I be the only person waking up at 3am?” 
They empower developers with responsibility: 
let them deploy, have them on-call, no passwords, etc. 
Payments 
SupportEscalationNot in another rotation1st time 
On-Call Schedules 
PaymentsSupportEscalationNot in another rotation1st time 
Dev On-Call 
“Make sure that developers stick around to ensure that their deploy worked.”
What if you’re not Etsy and you want to get buy-in from your devs… is there a way to make it easier for devs to be included in the on-call process? 
Read on to see how these 4 companies did it…
Production systems used to be a black box to most people on our team. 
Only a few developers had any insight whatsoever. 
Now that the entire team is on-call (be it in a very flexible way), our front-end devs are really excited when an alert goes off because they want to be the first one to ack it. 
And it is making our devs better devs because they have a holistic view of our system. 
I’ve built on-call systems myself before, and I’m a firm believer in the developers being on-call. A lot of companies aren’t big enough to have dedicated on-call teams of 10 
or 20 people, and devs are smart and very capable of doing the work. 
Bunchball is an agile development shop, so all teams are self-managed. The trick 
was convincing everyone that putting devs on-call was the right idea. 
Our devs now have an understanding of how the system functions on a day-to-day basis. It feels like everyone is more engaged in the ongoing success of our product. 
Nick Goodman, 
Director of Platform Engineering, Bunchball
Before our developers were on-call, I usually had to deal 
with their deploy that they broke on a Friday night. 
Now each of our 13 applications has 2 people in rotation for 
on-call–and one of those is always a dev. It has created a completely different mindset for our dev team. Now that they know they can be woken up at 4 in the morning, they deploy when they know they can be available to fix things. 
There are fewer production issues now that our devs fully understand what it means to push to production. 
Michael D’Auria, 
Infrastructure Lead at CrowdTap
Most of our devs come from writing desktop applications. We wanted to get them into our on-call rotation so they have that shift in mindset to a more holistic view. There was trepidation and a little bit of grumpiness at first because they’ve never done it before. 
Now, we have a paired on-call rotation with our techops and dev team. Each can manage their on-call schedules independently, and each person can manage their notification policies independently. 
If the first tier people don’t respond to alerts in a certain amount of time, it defaults to a secondary. Then the third tier is me, and the fourth tier is management. 
Paul Beltrani, 
TechOps at Onshape
We restructured how on-call works. It used to be just Operations team 
(2-3 people) rotated once per week. 
We thought it would make it easier to include the broader team - 25 
people in rotation. Alerts would go to the engineer on-call first. If they don’t ack within 5 min, it would go on to the next tier of the Operations team. 
Some engineers laughed it off. It was new to them and they felt like they didn’t know how to do anything. 
As a result, we’ve more than quintupled the number of ops people and engineers in rotation at any given time. This has been especially good for engineers as it gave them a more holistic build-to-deploy view of things. 
We even built a wiki with common problems and and how to resolve them so everyone felt like they had more power and could take responsibility. 
Matt Knox, 
SysAdmin at TrackVia
http://victorops.com/blog/onshape-case-study/ 
http://victorops.com/blog/bunchball-case-study/ 
http://victorops.com/blog/trackvia-case-study-victorops-improved-ttr/ 
http://www.confreaks.com/videos/2358-mwrc2013-you-should-be-on-call-too 
http://www.javacodegeeks.com/2014/01/developers-working-in- 
production-of-course-maybe-sometimes-what-are-younuts.html 
http://www.thoughtworks.com/insights/blog/n%C3%A3o-existe-equipe-de-devops 
http://www.paperplanes.de/2013/1/2/on-pager-duty.html

More Related Content

What's hot

JUG Bonn June 2021 - The DevOps disaster
JUG Bonn June 2021 - The DevOps disasterJUG Bonn June 2021 - The DevOps disaster
JUG Bonn June 2021 - The DevOps disasterBert Jan Schrijver
 
Principles of Continuous Delivery and DevOps
Principles of Continuous Delivery and DevOpsPrinciples of Continuous Delivery and DevOps
Principles of Continuous Delivery and DevOpsBert Jan Schrijver
 
DevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsDevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsNitin Bhide
 
Introduction to devops 2016
Introduction to devops 2016Introduction to devops 2016
Introduction to devops 2016gjdevos
 
DevOps – the future of Agile – why, what, how? Agile Israel 2014
DevOps – the future of Agile – why, what, how? Agile Israel 2014DevOps – the future of Agile – why, what, how? Agile Israel 2014
DevOps – the future of Agile – why, what, how? Agile Israel 2014Yuval Yeret
 
The Phoenix Project DevOps Simulation - Paul Wilkinson
The Phoenix Project DevOps Simulation - Paul WilkinsonThe Phoenix Project DevOps Simulation - Paul Wilkinson
The Phoenix Project DevOps Simulation - Paul WilkinsonPink Elephant
 
Devopsguys DevOps 101 for recruiters
Devopsguys   DevOps 101 for recruitersDevopsguys   DevOps 101 for recruiters
Devopsguys DevOps 101 for recruitersDevOpsGroup
 
Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous DeliveryJez Humble
 
DevOps Beyond the Buzzwords: Culture, Tools, & Straight Talk
DevOps Beyond the Buzzwords: Culture, Tools, & Straight TalkDevOps Beyond the Buzzwords: Culture, Tools, & Straight Talk
DevOps Beyond the Buzzwords: Culture, Tools, & Straight TalkMark Heckler
 
DOES15 - Elisabeth Hendrickson - Its All About Feedback
DOES15 - Elisabeth Hendrickson - Its All About FeedbackDOES15 - Elisabeth Hendrickson - Its All About Feedback
DOES15 - Elisabeth Hendrickson - Its All About FeedbackGene Kim
 
Jan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsJan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsAgile Lietuva
 
DevOps: Process, Tool or Mindset?
DevOps: Process, Tool or Mindset?DevOps: Process, Tool or Mindset?
DevOps: Process, Tool or Mindset?Tathagat Varma
 
ReleaseIQ's Next Gen Enterprise Devops Platform Webinar Slides
ReleaseIQ's  Next Gen Enterprise Devops Platform Webinar SlidesReleaseIQ's  Next Gen Enterprise Devops Platform Webinar Slides
ReleaseIQ's Next Gen Enterprise Devops Platform Webinar SlidesWayne Greene
 

What's hot (20)

JUG Bonn June 2021 - The DevOps disaster
JUG Bonn June 2021 - The DevOps disasterJUG Bonn June 2021 - The DevOps disaster
JUG Bonn June 2021 - The DevOps disaster
 
Devops or die!
Devops or die!Devops or die!
Devops or die!
 
Why Even DevOp?
Why Even DevOp?Why Even DevOp?
Why Even DevOp?
 
Principles of Continuous Delivery and DevOps
Principles of Continuous Delivery and DevOpsPrinciples of Continuous Delivery and DevOps
Principles of Continuous Delivery and DevOps
 
DevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsDevOps - Understanding Core Concepts
DevOps - Understanding Core Concepts
 
Intro to DevOps
Intro to DevOpsIntro to DevOps
Intro to DevOps
 
Continuous Delivery e-book
Continuous Delivery e-bookContinuous Delivery e-book
Continuous Delivery e-book
 
Developer Nirvana
Developer NirvanaDeveloper Nirvana
Developer Nirvana
 
Introduction to devops 2016
Introduction to devops 2016Introduction to devops 2016
Introduction to devops 2016
 
DevOps – the future of Agile – why, what, how? Agile Israel 2014
DevOps – the future of Agile – why, what, how? Agile Israel 2014DevOps – the future of Agile – why, what, how? Agile Israel 2014
DevOps – the future of Agile – why, what, how? Agile Israel 2014
 
The Phoenix Project DevOps Simulation - Paul Wilkinson
The Phoenix Project DevOps Simulation - Paul WilkinsonThe Phoenix Project DevOps Simulation - Paul Wilkinson
The Phoenix Project DevOps Simulation - Paul Wilkinson
 
Devopsguys DevOps 101 for recruiters
Devopsguys   DevOps 101 for recruitersDevopsguys   DevOps 101 for recruiters
Devopsguys DevOps 101 for recruiters
 
Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous Delivery
 
DevOps Beyond the Buzzwords: Culture, Tools, & Straight Talk
DevOps Beyond the Buzzwords: Culture, Tools, & Straight TalkDevOps Beyond the Buzzwords: Culture, Tools, & Straight Talk
DevOps Beyond the Buzzwords: Culture, Tools, & Straight Talk
 
DOES15 - Elisabeth Hendrickson - Its All About Feedback
DOES15 - Elisabeth Hendrickson - Its All About FeedbackDOES15 - Elisabeth Hendrickson - Its All About Feedback
DOES15 - Elisabeth Hendrickson - Its All About Feedback
 
DevOps
DevOpsDevOps
DevOps
 
Jan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsJan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wants
 
DevOps 101
DevOps 101DevOps 101
DevOps 101
 
DevOps: Process, Tool or Mindset?
DevOps: Process, Tool or Mindset?DevOps: Process, Tool or Mindset?
DevOps: Process, Tool or Mindset?
 
ReleaseIQ's Next Gen Enterprise Devops Platform Webinar Slides
ReleaseIQ's  Next Gen Enterprise Devops Platform Webinar SlidesReleaseIQ's  Next Gen Enterprise Devops Platform Webinar Slides
ReleaseIQ's Next Gen Enterprise Devops Platform Webinar Slides
 

Similar to Putting Devs On-Call: How to Empower Your Team

DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...
DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...
DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...Rauno De Pasquale
 
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...NETWAYS
 
Deden Fathurahman - Observability Within Your DevOps Environment
Deden Fathurahman - Observability Within Your DevOps EnvironmentDeden Fathurahman - Observability Within Your DevOps Environment
Deden Fathurahman - Observability Within Your DevOps EnvironmentAgile Impact Conference
 
Uklug 2011 administrator development synergy
Uklug 2011 administrator development synergyUklug 2011 administrator development synergy
Uklug 2011 administrator development synergydominion
 
How to Build a Platform Team
How to Build a Platform TeamHow to Build a Platform Team
How to Build a Platform TeamVMware Tanzu
 
Chaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field GuideChaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field Guidematthewbrahms
 
Site-Reliability-Engineering-v2[6241].pdf
Site-Reliability-Engineering-v2[6241].pdfSite-Reliability-Engineering-v2[6241].pdf
Site-Reliability-Engineering-v2[6241].pdfDeepakGupta747774
 
Rails Operations - Lessons Learned
Rails Operations -  Lessons LearnedRails Operations -  Lessons Learned
Rails Operations - Lessons LearnedJosh Nichols
 
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...Dana Gardner
 
ASAS 2015 - Benito de Miranda
ASAS 2015 - Benito de MirandaASAS 2015 - Benito de Miranda
ASAS 2015 - Benito de MirandaAvisi B.V.
 
Reactive Microservice Architecture with Groovy and Grails
Reactive Microservice Architecture with Groovy and GrailsReactive Microservice Architecture with Groovy and Grails
Reactive Microservice Architecture with Groovy and GrailsSteve Pember
 
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOpsWinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOpsWinOps Conf
 
When Things Go Bump in the Night
When Things Go Bump in the NightWhen Things Go Bump in the Night
When Things Go Bump in the Nightahamilton55
 

Similar to Putting Devs On-Call: How to Empower Your Team (20)

Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous Delivery
 
DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...
DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...
DevOps Torino Meetup Group Kickoff Meeting - Why a meetup group on DevOps, wh...
 
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...
stackconf 2023 | Better Living by Changing Less – IncrativeOps by Michael Cot...
 
Deden Fathurahman - Observability Within Your DevOps Environment
Deden Fathurahman - Observability Within Your DevOps EnvironmentDeden Fathurahman - Observability Within Your DevOps Environment
Deden Fathurahman - Observability Within Your DevOps Environment
 
Role of dev ops in it consulting
Role of dev ops in it consultingRole of dev ops in it consulting
Role of dev ops in it consulting
 
Lean / Kanban
Lean / KanbanLean / Kanban
Lean / Kanban
 
Uklug 2011 administrator development synergy
Uklug 2011 administrator development synergyUklug 2011 administrator development synergy
Uklug 2011 administrator development synergy
 
How to Build a Platform Team
How to Build a Platform TeamHow to Build a Platform Team
How to Build a Platform Team
 
Chaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field GuideChaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field Guide
 
Site-Reliability-Engineering-v2[6241].pdf
Site-Reliability-Engineering-v2[6241].pdfSite-Reliability-Engineering-v2[6241].pdf
Site-Reliability-Engineering-v2[6241].pdf
 
Rails Operations - Lessons Learned
Rails Operations -  Lessons LearnedRails Operations -  Lessons Learned
Rails Operations - Lessons Learned
 
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...
Internet of Things Brings On Development Demands That DevOps Manages, Say Exp...
 
What DevOps Isn't
What DevOps Isn'tWhat DevOps Isn't
What DevOps Isn't
 
Tell me how you provision and I'll tell you how you are
Tell me how you provision and I'll tell you how you areTell me how you provision and I'll tell you how you are
Tell me how you provision and I'll tell you how you are
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
Dev ops implementation your go-to guide
Dev ops implementation  your go-to guide Dev ops implementation  your go-to guide
Dev ops implementation your go-to guide
 
ASAS 2015 - Benito de Miranda
ASAS 2015 - Benito de MirandaASAS 2015 - Benito de Miranda
ASAS 2015 - Benito de Miranda
 
Reactive Microservice Architecture with Groovy and Grails
Reactive Microservice Architecture with Groovy and GrailsReactive Microservice Architecture with Groovy and Grails
Reactive Microservice Architecture with Groovy and Grails
 
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOpsWinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
 
When Things Go Bump in the Night
When Things Go Bump in the NightWhen Things Go Bump in the Night
When Things Go Bump in the Night
 

More from VictorOps

Failure as Success Devops Roadtrip Seattle 2016
Failure as Success Devops Roadtrip Seattle 2016Failure as Success Devops Roadtrip Seattle 2016
Failure as Success Devops Roadtrip Seattle 2016VictorOps
 
DevOps Roadtrip Final Speaking Deck
DevOps Roadtrip Final Speaking Deck DevOps Roadtrip Final Speaking Deck
DevOps Roadtrip Final Speaking Deck VictorOps
 
DevOps: A Practical Guide
DevOps: A Practical GuideDevOps: A Practical Guide
DevOps: A Practical GuideVictorOps
 
Crisis Communication Webinar
Crisis Communication WebinarCrisis Communication Webinar
Crisis Communication WebinarVictorOps
 
The Importance of Minimum Viable Runbooks Webinar
The Importance of Minimum Viable Runbooks WebinarThe Importance of Minimum Viable Runbooks Webinar
The Importance of Minimum Viable Runbooks WebinarVictorOps
 
DevOps Roadtrip - Denver
DevOps Roadtrip - DenverDevOps Roadtrip - Denver
DevOps Roadtrip - DenverVictorOps
 
VictorOps & Raygun: A Stunning Integration
VictorOps & Raygun: A Stunning IntegrationVictorOps & Raygun: A Stunning Integration
VictorOps & Raygun: A Stunning IntegrationVictorOps
 
ChatOps: The New Interface of DevOps
ChatOps: The New Interface of DevOpsChatOps: The New Interface of DevOps
ChatOps: The New Interface of DevOpsVictorOps
 
6 Steps to Creating a Minimum Viable Runbook Infographic
6 Steps to Creating a Minimum Viable Runbook Infographic6 Steps to Creating a Minimum Viable Runbook Infographic
6 Steps to Creating a Minimum Viable Runbook InfographicVictorOps
 
Incident Lifecycle Infographic
Incident Lifecycle InfographicIncident Lifecycle Infographic
Incident Lifecycle InfographicVictorOps
 
Crisis Management & Why It's Important Infographic
Crisis Management & Why It's Important InfographicCrisis Management & Why It's Important Infographic
Crisis Management & Why It's Important InfographicVictorOps
 
Real World ChatOps
Real World ChatOpsReal World ChatOps
Real World ChatOpsVictorOps
 
Tips & Tricks To Reducing TTR
Tips & Tricks To Reducing TTRTips & Tricks To Reducing TTR
Tips & Tricks To Reducing TTRVictorOps
 
The Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeThe Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeVictorOps
 
Actors: Not Just for Movies Anymore
Actors: Not Just for Movies AnymoreActors: Not Just for Movies Anymore
Actors: Not Just for Movies AnymoreVictorOps
 
An Introduction to Rearview - Time Series Based Monitoring
An Introduction to Rearview - Time Series Based MonitoringAn Introduction to Rearview - Time Series Based Monitoring
An Introduction to Rearview - Time Series Based MonitoringVictorOps
 
The Art & Zen of Managing Nagios with Puppet
The Art & Zen of Managing Nagios with PuppetThe Art & Zen of Managing Nagios with Puppet
The Art & Zen of Managing Nagios with PuppetVictorOps
 
ChatOps Unplugged
ChatOps UnpluggedChatOps Unplugged
ChatOps UnpluggedVictorOps
 
Post-mortem Fail
Post-mortem FailPost-mortem Fail
Post-mortem FailVictorOps
 
On-call Firefight Survival Guide
On-call Firefight Survival GuideOn-call Firefight Survival Guide
On-call Firefight Survival GuideVictorOps
 

More from VictorOps (20)

Failure as Success Devops Roadtrip Seattle 2016
Failure as Success Devops Roadtrip Seattle 2016Failure as Success Devops Roadtrip Seattle 2016
Failure as Success Devops Roadtrip Seattle 2016
 
DevOps Roadtrip Final Speaking Deck
DevOps Roadtrip Final Speaking Deck DevOps Roadtrip Final Speaking Deck
DevOps Roadtrip Final Speaking Deck
 
DevOps: A Practical Guide
DevOps: A Practical GuideDevOps: A Practical Guide
DevOps: A Practical Guide
 
Crisis Communication Webinar
Crisis Communication WebinarCrisis Communication Webinar
Crisis Communication Webinar
 
The Importance of Minimum Viable Runbooks Webinar
The Importance of Minimum Viable Runbooks WebinarThe Importance of Minimum Viable Runbooks Webinar
The Importance of Minimum Viable Runbooks Webinar
 
DevOps Roadtrip - Denver
DevOps Roadtrip - DenverDevOps Roadtrip - Denver
DevOps Roadtrip - Denver
 
VictorOps & Raygun: A Stunning Integration
VictorOps & Raygun: A Stunning IntegrationVictorOps & Raygun: A Stunning Integration
VictorOps & Raygun: A Stunning Integration
 
ChatOps: The New Interface of DevOps
ChatOps: The New Interface of DevOpsChatOps: The New Interface of DevOps
ChatOps: The New Interface of DevOps
 
6 Steps to Creating a Minimum Viable Runbook Infographic
6 Steps to Creating a Minimum Viable Runbook Infographic6 Steps to Creating a Minimum Viable Runbook Infographic
6 Steps to Creating a Minimum Viable Runbook Infographic
 
Incident Lifecycle Infographic
Incident Lifecycle InfographicIncident Lifecycle Infographic
Incident Lifecycle Infographic
 
Crisis Management & Why It's Important Infographic
Crisis Management & Why It's Important InfographicCrisis Management & Why It's Important Infographic
Crisis Management & Why It's Important Infographic
 
Real World ChatOps
Real World ChatOpsReal World ChatOps
Real World ChatOps
 
Tips & Tricks To Reducing TTR
Tips & Tricks To Reducing TTRTips & Tricks To Reducing TTR
Tips & Tricks To Reducing TTR
 
The Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeThe Open-Source Monitoring Landscape
The Open-Source Monitoring Landscape
 
Actors: Not Just for Movies Anymore
Actors: Not Just for Movies AnymoreActors: Not Just for Movies Anymore
Actors: Not Just for Movies Anymore
 
An Introduction to Rearview - Time Series Based Monitoring
An Introduction to Rearview - Time Series Based MonitoringAn Introduction to Rearview - Time Series Based Monitoring
An Introduction to Rearview - Time Series Based Monitoring
 
The Art & Zen of Managing Nagios with Puppet
The Art & Zen of Managing Nagios with PuppetThe Art & Zen of Managing Nagios with Puppet
The Art & Zen of Managing Nagios with Puppet
 
ChatOps Unplugged
ChatOps UnpluggedChatOps Unplugged
ChatOps Unplugged
 
Post-mortem Fail
Post-mortem FailPost-mortem Fail
Post-mortem Fail
 
On-call Firefight Survival Guide
On-call Firefight Survival GuideOn-call Firefight Survival Guide
On-call Firefight Survival Guide
 

Recently uploaded

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesrafiqahmad00786416
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native ApplicationsWSO2
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Orbitshub
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 

Recently uploaded (20)

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 

Putting Devs On-Call: How to Empower Your Team

  • 1.
  • 2. “New hires should be kept off the on-call rotation until they know the platform inside-and-out.” “I just write the code and throw it over the wall. It’s up to Ops to make it work.” “It’s not my job to test the code. If there’s a mistake, talk to QA.” “I’ve never had to carry the pager before… why should I start now?”
  • 3. Ideally, companies work out a system in which whoever makes the mistake pays the price. The reason Ops is so often scared of Dev deploying is that Dev doesn't really care how secure their apps are, how hard they are to deploy, how hard they are to keep running or how many times you have to restart it, because Ops pays the price for those mistakes, not Dev. In most organizations the mandate of a developer is merely to produce a piece of software that worked on a workstation -- if it worked on your workstation and you can't make it work in production, it's Operations' fault if they can't get that to thousands of machines all around the world. http://readwrite.com/2011/08/23/devops-what-it-is-why-it-exist
  • 4. According to Ernest Mueller, one of the authors of the Agile Admin: DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support. DevOps means breaking down the traditional silos that have existed between Ops and Devs. DevOps Practices-Specific techniques used as part of implementing the above concepts and processes. Continuous integration and continuous deployment “Give your developers a pager and put them on-call” Using Configuration management, metrics and monitoring schemes Using virtualization and cloud computing to accelerate change http://theagileadmin.com/what-is-devops/
  • 5. Devs can win instant good will by helping to carry one of Ops’ biggest burdens: the pager. Developers tend to bring more craftsmanship into their work knowing that they, or their pals, might suffer the consequences of half-baked code. Being waist-deep in the immediate resolution and accompanying troubleshooting efforts also provides a valuable perspective on how the application’s architecture interacts with the infrastructure it sits on. Clearly define (and yes: document) escalation paths for emergencies and make sure devs have access to the tools and data they’ll need for troubleshooting. Then include the development team in on-call rotation. https://www.atlassian.com/devops/culture#!shared-responsibilities
  • 6. http://www.slideshare.net/mrtazz/goto-amsterdam2014depolyment1024 Etsy has devs on-call to address the problem of IT Operations asking, “Why should I be the only person waking up at 3am?” They empower developers with responsibility: let them deploy, have them on-call, no passwords, etc. Payments SupportEscalationNot in another rotation1st time On-Call Schedules PaymentsSupportEscalationNot in another rotation1st time Dev On-Call “Make sure that developers stick around to ensure that their deploy worked.”
  • 7. What if you’re not Etsy and you want to get buy-in from your devs… is there a way to make it easier for devs to be included in the on-call process? Read on to see how these 4 companies did it…
  • 8. Production systems used to be a black box to most people on our team. Only a few developers had any insight whatsoever. Now that the entire team is on-call (be it in a very flexible way), our front-end devs are really excited when an alert goes off because they want to be the first one to ack it. And it is making our devs better devs because they have a holistic view of our system. I’ve built on-call systems myself before, and I’m a firm believer in the developers being on-call. A lot of companies aren’t big enough to have dedicated on-call teams of 10 or 20 people, and devs are smart and very capable of doing the work. Bunchball is an agile development shop, so all teams are self-managed. The trick was convincing everyone that putting devs on-call was the right idea. Our devs now have an understanding of how the system functions on a day-to-day basis. It feels like everyone is more engaged in the ongoing success of our product. Nick Goodman, Director of Platform Engineering, Bunchball
  • 9. Before our developers were on-call, I usually had to deal with their deploy that they broke on a Friday night. Now each of our 13 applications has 2 people in rotation for on-call–and one of those is always a dev. It has created a completely different mindset for our dev team. Now that they know they can be woken up at 4 in the morning, they deploy when they know they can be available to fix things. There are fewer production issues now that our devs fully understand what it means to push to production. Michael D’Auria, Infrastructure Lead at CrowdTap
  • 10. Most of our devs come from writing desktop applications. We wanted to get them into our on-call rotation so they have that shift in mindset to a more holistic view. There was trepidation and a little bit of grumpiness at first because they’ve never done it before. Now, we have a paired on-call rotation with our techops and dev team. Each can manage their on-call schedules independently, and each person can manage their notification policies independently. If the first tier people don’t respond to alerts in a certain amount of time, it defaults to a secondary. Then the third tier is me, and the fourth tier is management. Paul Beltrani, TechOps at Onshape
  • 11. We restructured how on-call works. It used to be just Operations team (2-3 people) rotated once per week. We thought it would make it easier to include the broader team - 25 people in rotation. Alerts would go to the engineer on-call first. If they don’t ack within 5 min, it would go on to the next tier of the Operations team. Some engineers laughed it off. It was new to them and they felt like they didn’t know how to do anything. As a result, we’ve more than quintupled the number of ops people and engineers in rotation at any given time. This has been especially good for engineers as it gave them a more holistic build-to-deploy view of things. We even built a wiki with common problems and and how to resolve them so everyone felt like they had more power and could take responsibility. Matt Knox, SysAdmin at TrackVia
  • 12. http://victorops.com/blog/onshape-case-study/ http://victorops.com/blog/bunchball-case-study/ http://victorops.com/blog/trackvia-case-study-victorops-improved-ttr/ http://www.confreaks.com/videos/2358-mwrc2013-you-should-be-on-call-too http://www.javacodegeeks.com/2014/01/developers-working-in- production-of-course-maybe-sometimes-what-are-younuts.html http://www.thoughtworks.com/insights/blog/n%C3%A3o-existe-equipe-de-devops http://www.paperplanes.de/2013/1/2/on-pager-duty.html