Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Great PagerDuty DevOps Heist of 2016 - Arup Chakrabarti - DevOpsDays Tel Aviv 2017

91 views

Published on

DevOpsDays Tel Aviv 2017

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Great PagerDuty DevOps Heist of 2016 - Arup Chakrabarti - DevOpsDays Tel Aviv 2017

  1. 1. DevOpsDays Tel Aviv 2017 Arup Chakrabarti - @arupchak The Great PagerDuty DevOps Heist of 2016
  2. 2. @arupchak Agenda
  3. 3. @arupchak Who is this guy? • Academic Research • Amazon • Netflix • Currently PagerDuty
  4. 4. @arupchak Disclaimer: I am not a smrt person (I work with smart people)
  5. 5. @arupchak Disclaimer: This will not work for you
  6. 6. @arupchak What is PagerDuty?
  7. 7. @arupchak Incident Management and Resolution Unhappy Customers “Magic” Resolvers
  8. 8. @arupchak January 2016
  9. 9. @arupchak Org Chart VP of Engineering Operations Servers Databases Reliability Backend Services System Reliability Product Frontend Services Customer facing
  10. 10. @arupchak Problems • Centralized Operations Responsibility • Centralized Reliability Responsibility • Centralized Product Responsibility
  11. 11. @arupchak “The Heist”
  12. 12. @arupchak A crime in which valuable things are taken illegally and often violently from a person or people
  13. 13. @arupchak The act of selling something to someone without them knowing what they are going to pay for it
  14. 14. @arupchak …also known as “Fraud”
  15. 15. @arupchak “The Heist” •What we sold to the teams –Business Success depends on Innovation –Innovation depends on Rate of Change –We will increase the Rate of Change by having engineers own more of the stack
  16. 16. @arupchak March 2016
  17. 17. @arupchak New Org Chart VP of Engineering Infrastructure Products SRE Platform Security Product Team Alpha Servers Databases Software Product Team Beta Servers Databases Software
  18. 18. @arupchak Moar Problems • People working with new people – ”Who is that?” • People working on new things – “What is this?” • Lots of changes
  19. 19. @arupchak What Changed?
  20. 20. @arupchak The Job Changed • For previously Dev focused people – The now owned the full vertical stack – Code it. Ship it. Own it. • For previously Ops focused people – Had to empower others to do their previous job – Make the right thing the easy thing
  21. 21. @arupchak ”I need to do what now?” -Every Engineer
  22. 22. @arupchak Self Service Tooling
  23. 23. @arupchak What we Built • Infrastructure Tools – Self Service Server Provisioning – Self Service Metrics and Telemetry – Self Service Deployment – Self Service Common Infrastructure • Documentation – Where we could not automate easily/quickly
  24. 24. @arupchak ”Ok, I think I can handle that” -Most Engineers
  25. 25. @arupchak People over Tooling
  26. 26. @arupchak ”This sucks, I am out of here” -Some Engineers
  27. 27. @arupchak The Hard Part of Leadership • Some changes are not for everyone • Some people who thrived in the old way, will struggle in the new ways • They are not trying to be jerks • Expect attrition
  28. 28. @arupchak Introducing Change is hard, Managing Change is even harder
  29. 29. @arupchak November 2017
  30. 30. @arupchak ”Despite Arup’s efforts, PagerDuty is still in business”
  31. 31. @arupchak What did progress look like?
  32. 32. @arupchak Github Metrics
  33. 33. @arupchak Chef Repo Metrics
  34. 34. @arupchak PagerDuty Metrics
  35. 35. @arupchak Business Metrics
  36. 36. @arupchak Anecdotes • Intern replaced a Kafka cluster • Engineers practicing MySQL recovery • Engineers building out code coverage metrics
  37. 37. @arupchak “The Heist”
  38. 38. @arupchak “The Heist” •What we sold to the teams –Business Success depends on Innovation –Innovation depends on Rate of Change –We will increase the Rate of Change by having engineers own more of the stack
  39. 39. @arupchak “The Heist” •What it cost the teams –All Engineers go on-call –Code it. Ship it. Own it. –People leaving
  40. 40. @arupchak Did we get it completely right?
  41. 41. @arupchak Nope. Not even close.
  42. 42. We are hiring! Arup Chakrabarti @arupchak arup@pagerduty.com

×