Download as PDF, PPTX
Uploaded byRundeck
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
The document discusses the role of development in facilitating self-service operations, especially in the context of DevOps. It highlights challenges in traditional operational models and advocates for shifting operational responsibilities closer to development teams to enhance efficiency and responsiveness. Strategies such as establishing secure ops portals, automating procedures, and fostering collaboration between teams are recommended to improve operational outcomes.
More Related Content
Similar to Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
![Runbook Automation: Old News or a Key to Unlock Performance? [DOES2020]](https://cdn.slidesharecdn.com/ss_thumbnails/does-damonedwards-v2-201015073337-thumbnail.jpg?width=640&height=640&fit=bounds)
Helping Ops Help You: Development’s Role in Enabling Self-Service Operations
- 1. Helping Ops HelpYou: Development’s Role in Enabling Self-Service Operations Damon Edwards @damonedwards
- 2.
- 3. Some assumptions OpsDev DevOps Problems! OpsDevRemove Shorten Lead Time Feedback 1. Why we are here: We want thisWe don't want this 2. We are going to be talking about enterprise complexity and scale
- 4.
- 5. Deploy. Deploy. Deploy. Deploy. DevOps Promise Deploy. Deploy. Deploy. Depl… uh oh…we have a problem DevOps Reality
- 6. Deploy. Deploy. Deploy. Deploy. DevOps Promise Deploy. Deploy. Deploy. Depl… uh oh…we have a problem DevOps Reality Ops happens!
- 7. Two prevailing modelsof operations support Running Service “You build it. They run it.” “You build it. You run it.” Development Team Operations Team Dev Ops Integrated Delivery Team Running Service
- 8. Two prevailing modelsof operations support Running Service “You build it. They run it.” “You build it. You run it.” Development Team Operations Team Dev Ops Integrated Delivery Team Running Service “two-pizza team”
- 9. “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s the NOC… Talk them through: health checks, reviewing log files, and process of diagnosing and recovering the system. Same as you did for dev teams 2 months ago, QA teams last month, Ops during deploy last week, etc.
- 10. “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm ….
- 11. “You build it.They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s Ops… “Will your applications be affected if we take down EU-West?” “Is it ok if we change these firewall rules?” “We are getting customer complaints about performance. Are you sure you didn’t change something?”.
- 12. “You build it.They run it.” (aka… the way it always was) Running Service Development Team Operations Team
- 13. “You build it.They run it.” (aka… the way it always was) Running Service Development Team Operations Team
- 14. “You build it.You run it.” Dev Ops Integrated Delivery Team
- 15. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!!
- 16. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities!
- 17. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities!
- 18. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities!
- 19. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities!
- 20. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities!
- 21. “You build it.You run it.” Dev Ops Integrated Delivery Team Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! “two-pizza teams”? Just change how business is structured, funded, and operated.
- 22. What if wecould…
- 23. Have the laborscaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs What if we could…
- 24. Have the laborscaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs What if we could… Have the responsiveness/control of “you build it, you run it” without the scaling limitations
- 25. Understand what Opsis dealing with…
- 26. Enterprise Operations isunder tremendous pressure OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
- 27. Organizational complexity compoundedby system complexity Ticket System SysAdmin NetEng CDN SysEng Ticket System SecOps Ticket System DNS PM DBA ? I just need an environment! Something isn’t working!
- 28. Silos + Tools= Islands of Automation Puppet Chef Shell Scripts Data ETL PowershellScripts Network Management Monitoring Ansible Legacy Datacenter Automation ContainerManagement SQL Tools NewTools New Tools
- 29. Great Operations startsin Development.
- 30. Develop with an“Operable First” mindset
- 31. Operations is thebusiness (unless you literally sell packaged software) Develop with an “Operable First” mindset
- 32. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Develop with an “Operable First” mindset
- 33. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Build configuration into the service, don’t externalize it Develop with an “Operable First” mindset
- 34. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Build configuration into the service, don’t externalize it Demand “prod-like” environments everywhere Develop with an “Operable First” mindset
- 35. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Build configuration into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Develop with an “Operable First” mindset
- 36. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Build configuration into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Develop with an “Operable First” mindset
- 37. Operations is thebusiness (unless you literally sell packaged software) Deployability, configurability, monitoring are your responsibility Build configuration into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests Develop with an “Operable First” mindset
- 38. “Shift Left” asmuch Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools “Deploy.”
- 39. “Shift Left” asmuch Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.”
- 40. HOW? “Shift Left” asmuch Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.”
- 41. How do you“shift left” ops procedures and incident response? Those who build something define the procedures to fix it Move the ability to respond closest to the problem 1 2
- 42. But... How do yousafely and securely give out access? How do you enable the experts to contribute remediations? How do you give the experts visibility into operations? How do you do postmortems days/weeks/months later? Those who build something define the procedures to fix it Move the ability to respond closest to the problem 1 2 How do you “shift left” ops procedures and incident response?
- 43. Step 1: Establisha Secure Ops Portal
- 44. Step 2: Establisha SDLC for Ops Procedures
- 45. Step 3: Connectwith Enterprise Management Systems
- 46. Who created theprocedure? Who reviewed it? Who? When? Where? Approval trail? Step 4: Make Compliance Really Happy
- 47. Improve response timeand reduce escalations Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
- 48. Improve response timeand reduce escalations Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
- 49. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model
- 50. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model • Automated Ops procedures written/vetted by the delivery teams
- 51. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model • Automated Ops procedures written/vetted by the delivery teams • Ops remained in full control of what can run and security policy
- 52. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model • Automated Ops procedures written/vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered support teams with self-service ops tasks
- 53. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model • Automated Ops procedures written/vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered support teams with self-service ops tasks • Empowered developers with limited self-service operations
- 54. Example of puttingthese types of principles to work Mark Maun Jody Mulkey Justin Dean Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html Ticketmaster’s “Support at the Edge” model • Automated Ops procedures written/vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response model
- 55. Better for thebusiness and better way to work for all 90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs
- 56. Recap Automated Tests Deployment Automation RunningSecurity Scanning Tools Automated Runbooks Monitoring/Metrics Operational Control “Shift-Left” operational concerns We are all in Operations. Develop like it. Explicit investment in process and tooling OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
- 57.