Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Self-Service Operations: Because Failure Still Happens (Developer Edition)

602 views

Published on

Damon Edwards (Rundeck) keynote at DevNet Create on May 24, 2017.
Abstract:
Agile and DevOps have provided plenty of lessons for how to speed up the pace of application delivery and the frequency of application deployment. But delivery and deployment only covers one part of the day-to-day life of developers in large enterprises. What about what happens after deployment? In many enterprises, increasing the pace of delivery and frequency of deployment has just increased the operational support load, work interrupts, and context switching that were already cutting deeply into development teams' time.

This talk will focus on the successful design patterns that high-performing, large scale organizations have applied to reduce the operational burden and support costs across their entire organization. Specifically, we’ll look at how they apply DevOps principles to improving the post-deployment lifecycle and how Developers play the key role in reducing the difficultly and cost of operations activity for everyone.

Published in: Technology
  • Be the first to comment

Self-Service Operations: Because Failure Still Happens (Developer Edition)

  1. 1. Self-Service Operations: Because Failure Still Happens Damon Edwards @damonedwards
  2. 2. Damon Edwards Ops Improvement DevOps Consulting Tools Community
  3. 3. Deploy. Deploy. Deploy. Deploy. DevOps Dream
  4. 4. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
  5. 5. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
  6. 6. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens! Escalations! Interruptions! Delays! Context Switching! Bridge Calls! Arguments!
  7. 7. Let’s start with a story…
  8. 8. Mark Maun Jody Mulkey Justin Dean
  9. 9. 90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs
  10. 10. How did they do that?
  11. 11. But first…
  12. 12. Let’s look at the principles behind the improvement …
  13. 13. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
  14. 14. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
  15. 15. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service “two-pizza team” Development Team Operations Team Dev Ops Integrated Delivery Team
  16. 16. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s the NOC… Talk them through: health checks, reviewing log files, and process of diagnosing and recovering the system. Same as you did for dev teams 2 months ago, QA teams last month, Ops during deploy last week, etc.
  17. 17. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm ….
  18. 18. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s Ops… “Will your applications be affected if we take down EU-West?” “How do we reconfigure your app if we change these firewall rules?” “We are getting customer complaints about performance. Help us figure out what changed”.
  19. 19. “You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
  20. 20. “You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
  21. 21. “You build it. You run it.” Dev Ops Integrated Delivery Team
  22. 22. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Dev Ops Integrated Delivery Team
  23. 23. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  24. 24. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  25. 25. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  26. 26. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  27. 27. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
  28. 28. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team “two-pizza teams”? Just change how business is structured, funded, and operated.
  29. 29. How do we get the best of both models?
  30. 30. Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models?
  31. 31. Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models? Have the responsiveness/control of “you build it, you run it” without the scaling limitations
  32. 32. “Self-Service Operations” design pattern is the key enabler
  33. 33. “Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor
  34. 34. “Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
  35. 35. What do Developers get out of Self-Service Operations?…
  36. 36. Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
  37. 37. Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
  38. 38. Tighter feedback loops Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
  39. 39. Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
  40. 40. Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
  41. 41. How can Dev help Ops make this a reality…
  42. 42. First understand the pain of Operations today…
  43. 43. Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
  44. 44. Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!"
  45. 45. Ops is Unplanned Work and Planned Work… by design! +
  46. 46. Planned + Unplanned = Expensive Context Switching Gerald Weinberg via Jeff Atwood https://blog.codinghorror.com/the-multi-tasking-myth/
  47. 47. Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
  48. 48. Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
  49. 49. ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
  50. 50. ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
  51. 51. Cross functional teams? ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
  52. 52. Cross functional teams? Never enough to go around. Context Work Task Work Task Work Task Work Task Queue ! Handoffs ! Handoffs Team C ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
  53. 53. Silos + Tool Evolution = Islands of Automation Puppet Chef Shell Scripts Data ETL PowershellScripts Network Management Monitoring Ansible Legacy Datacenter Automation ContainerManagement SQL Tools NewTools New Tools
  54. 54. Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under
  55. 55. Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under = Ops never has enough time or people! +
  56. 56. How developers can help operations…
  57. 57. 1. Recognize that great Operations starts in Development
  58. 58. 2. Develop with an “Operable First” mindset
  59. 59. Develop with an “Operable First” mindset
  60. 60. Operations is the business (unless you literally sell packaged software) Develop with an “Operable First” mindset
  61. 61. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Develop with an “Operable First” mindset
  62. 62. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Develop with an “Operable First” mindset
  63. 63. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Develop with an “Operable First” mindset
  64. 64. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Develop with an “Operable First” mindset
  65. 65. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Develop with an “Operable First” mindset
  66. 66. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests Develop with an “Operable First” mindset
  67. 67. 3. “Shift Left” as much Ops activity as possible
  68. 68. “Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools “Deploy.” Ops Ah-ha! Dev Ka-ching!
  69. 69. “Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.” Ops Ah-ha! Dev Ka-ching!
  70. 70. 4. Develop Self-Service Operations capabilities
  71. 71. Step 1: Establish a Secure Ops Hub Custom Mix
  72. 72. Step 2: Establish a SDLC for Ops Procedures Custom Mix
  73. 73. Step 3: Connect with Enterprise Management Systems Custom Mix
  74. 74. Who created the procedure? Who reviewed it? Who? When? Where? Approval trail? Step 4: Make Compliance Really Happy Custom Mix
  75. 75. Again: How can Developers help Operations 1. Recognize that great Operations starts in Development 2. Develop with an “Operable First” mindset 3. “Shift Left” as much Ops activity as possible 4. Develop Self-Service Operations capabilities
  76. 76. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge”
  77. 77. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams
  78. 78. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy
  79. 79. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks
  80. 80. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations
  81. 81. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response and escalation model
  82. 82. Recap Understand the pressures on Ops Explicit investment in process and tooling OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!" Understand your operating model Operations is the business) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification- driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests “Operable First” mindset Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Apply Self-Service Operations design pattern “Shift left” operational concerns
  83. 83. Let’s talk… @damonedwards damon@rundeck.com

×