Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Delivery Patterns for Boring Releases @ DevOps Porto meetup - July 12th 2018

141 views

Published on

In today's world, we need not only to maintain our systems running 24x7 in production but we must also be able to keep them releasable 24x7. We can only do that with modern infrastructure and software delivery practices. Releases should be even more boring than doing your supermarket shopping. Make a code change, see it go through the pipeline, get green or red result. If red, fix or rollback. If green, deploy. If deploy fails, rollback. Monitor forever.

Because everyone wants to go faster but also safer, we're cramming more and more activities in the pipeline, from security controls to database changes, to compliance approvals, soon networking... How can we do this AND still move fast AND avoid burning out teams with all this cognitive load?

I will talk about how to ensure that your delivery system serves its core purpose of quickly and safely progressing our client-facing systems from commit to production. You'll also hear about key practices for software releasability like pipeline-as-code, short and wide pipelines, build and release from zero to production, blue-green deployments, and more!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Continuous Delivery Patterns for Boring Releases @ DevOps Porto meetup - July 12th 2018

  1. 1. Continuous Delivery Patterns for Boring Releases
  2. 2. About me Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net DevOps and Delivery Consultant Focused on teams and flow 2@manupaisable | manuelpais.net
  3. 3. @manupaisable | manuelpais.net 3 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  4. 4. @manupaisable | manuelpais.net 4 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  5. 5. 5@manupaisable | manuelpais.net
  6. 6. 6 App down, refuses to restart @manupaisable | manuelpais.net
  7. 7. 7 App down, refuses to restart Data migration completed… partially @manupaisable | manuelpais.net
  8. 8. 8 App down, refuses to restart… Missing libraries… Data migration completed… partially @manupaisable | manuelpais.net
  9. 9. 9 App down, refuses to restart… The password master… Data migration completed… partially Missing libraries… @manupaisable | manuelpais.net
  10. 10. 10@manupaisable | manuelpais.net
  11. 11. 11@manupaisable | manuelpais.net
  12. 12. @manupaisable | manuelpais.net 12
  13. 13. @manupaisable | manuelpais.net 13
  14. 14. @manupaisable | manuelpais.net 14 Build Test Deploy
  15. 15. @manupaisable | manuelpais.net 15 Build Test Deploy Build Test Deploy
  16. 16. @manupaisable | manuelpais.net 16 Build Test Deploy Build Test Deploy Build Test Deploy
  17. 17. @manupaisable | manuelpais.net 17 Build Test Deploy Build Test Deploy Build Test Deploy Monitor
  18. 18. 18 Outcomes @manupaisable | manuelpais.net
  19. 19. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 19
  20. 20. @manupaisable | manuelpais.net 20 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  21. 21. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 21
  22. 22. STEP ONE map all release activities in a deployment pipeline 22@manupaisable | manuelpais.net
  23. 23. 23@manupaisable | manuelpais.net
  24. 24. Gains from improving visibility to everyone, and increasing repeatability and traceability are enormous 24@manupaisable | manuelpais.net
  25. 25. Value Stream Mapping @manupaisable | manuelpais.net 25
  26. 26. Value Stream Mapping is a pen and paper tool that highlights bottlenecks in a matter of hours and raises awareness of what difficulties other teams face @manupaisable | manuelpais.net 26
  27. 27. STEP TWO measure key metrics on: -speed (e.g. cycle time) -quality (e.g. defect rate) -operability (e.g. MTTR) 27@manupaisable | manuelpais.net
  28. 28. 28@manupaisable | manuelpais.net
  29. 29. STEP THREE PUT IN THE WORK !!! 29@manupaisable | manuelpais.net
  30. 30. Which work? Automated builds in clean environments 30@manupaisable | manuelpais.net
  31. 31. Which work? You can provision a mini-replica of production environnment 31@manupaisable | manuelpais.net
  32. 32. Which work? You have application health checks and fast smoke tests for sanity checking 32@manupaisable | manuelpais.net
  33. 33. Which work? You have acceptance tests with reasonable coverage, including failure scenarios 33@manupaisable | manuelpais.net
  34. 34. Which work? Single source of truth. Single binary. Single path to production. 34@manupaisable | manuelpais.net
  35. 35. STEP THREE PUT IN THE WORK !!! (but in iterative fashion… walking skeletons FTW!) 35@manupaisable | manuelpais.net
  36. 36. 36@manupaisable | manuelpais.net
  37. 37. 37 practices first, tools second @manupaisable | manuelpais.net
  38. 38. Practices for boring releases •Automated build (and unit tests) •Provision prod replica in pipeline •Automated acceptance tests (BDD) •No “invisible” activities •One source of truth •One path to production @manupaisable | manuelpais.net 38
  39. 39. @manupaisable | manuelpais.net 39 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  40. 40. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 40
  41. 41. @manupaisable | manuelpais.net 41 can’t auto- scale people How to cope with ever increasing cognitive load on teams to build and run applications?
  42. 42. @manupaisable | manuelpais.net 42 CI Peer review Infra Security Comply
  43. 43. @manupaisable | manuelpais.net 43 CI Peer review Infra Security Comply Database Accept UX Deploy
  44. 44. @manupaisable | manuelpais.net 44 FixMonitorRun CI Peer review Infra Security Comply Database Accept UX Deploy
  45. 45. Multiple approaches needed 1. Smarter pipelines 2. Team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 45
  46. 46. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 46
  47. 47. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 47
  48. 48. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms @manupaisable | manuelpais.net 49
  49. 49. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 50@manupaisable | manuelpais.net 1. Smarter pipelines
  50. 50. Waiting for pipeline @manupaisable | manuelpais.net 51
  51. 51. Waiting for pipeline @manupaisable | manuelpais.net 52
  52. 52. Reduce wait for pipeline @manupaisable | manuelpais.net 53
  53. 53. Scalable CI/CD Infrastructure •Solved problem in cloud systems •Starts with infrastructure-as-code •Agent farm (auto scaling if possible) •Pipelines need to evolve as number of teams grows@manupaisable | manuelpais.net 54
  54. 54. Waiting for dependencies @manupaisable | manuelpais.net 55
  55. 55. Waiting for dependencies @manupaisable | manuelpais.net 56
  56. 56. Flow efficiency = “teams who aren’t paying attention to this concept generally have flow efficiencies around the 15% mark - that means that work normally spends 85% of its lifecycle waiting on something.” http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using @manupaisable | manuelpais.net 57
  57. 57. issue is not how long it takes to do something, it's how long we're waiting for it to get done @manupaisable | manuelpais.net 58
  58. 58. @manupaisable | manuelpais.net 59
  59. 59. Initial state: serial pipeline @manupaisable | manuelpais.net 60
  60. 60. Intermediate state: CAB in path to prod @manupaisable | manuelpais.net 61
  61. 61. End state: short and wide pipeline @manupaisable | manuelpais.net 62
  62. 62. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 63@manupaisable | manuelpais.net 1. Smarter pipelines
  63. 63. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 64@manupaisable | manuelpais.net 1. Smarter pipelines
  64. 64. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 65@manupaisable | manuelpais.net 1. Smarter pipelines
  65. 65. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 66@manupaisable | manuelpais.net 1. Smarter pipelines
  66. 66. @manupaisable | manuelpais.net 70 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  67. 67. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 71
  68. 68. @manupaisable | manuelpais.net 72
  69. 69. Delivery system itself should be boring, not just the pipeline. 73@manupaisable | manuelpais.net
  70. 70. Resilient delivery system that enables fast(er) feedback loops 74@manupaisable | manuelpais.net
  71. 71. Delivery system •CI tool •Pipeline orchestration tool •Orchestration plugins / 3rd party tools •Pipeline definitions •Source repos •CI + CD infrastructure •And ? @manupaisable | manuelpais.net 75
  72. 72. Delivery system @manupaisable | manuelpais.net 76
  73. 73. Delivery system CI/CD Toolchain @manupaisable | manuelpais.net 77
  74. 74. Delivery system CI/CD Toolchain App 1 App 2 @manupaisable | manuelpais.net 78
  75. 75. Delivery system CI/CD Toolchain Pipelines App 1 App 2 @manupaisable | manuelpais.net 79
  76. 76. Delivery system CI/CD Toolchain Pipelines Infra App 1 App 2 @manupaisable | manuelpais.net 80
  77. 77. How resilient delivery looks like •Tooling and configuration changes do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 81
  78. 78. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 82
  79. 79. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, roll backed if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 83
  80. 80. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button (almost) @manupaisable | manuelpais.net 84
  81. 81. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 85
  82. 82. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 86
  83. 83. @manupaisable | manuelpais.net 87
  84. 84. Bonus Points pipeline-as-code allows designing future states of the value stream and draw evolution @manupaisable | manuelpais.net 88
  85. 85. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 89
  86. 86. Team A Team B Team C Team D @manupaisable | manuelpais.net 90
  87. 87. The glitch is believed to have been caused by a power supply issue and there is no evidence of a cyber-attack, the airline said. @manupaisable | manuelpais.net 91
  88. 88. Team A Team B Team C Team D @manupaisable | manuelpais.net 92
  89. 89. Team D Team BTeam A Team C @manupaisable | manuelpais.net 93
  90. 90. Team A @manupaisable | manuelpais.net 94 Team D Team B Team C
  91. 91. Team A Team C @manupaisable | manuelpais.net 95 Team D Team B
  92. 92. Team A Team B Team C @manupaisable | manuelpais.net 96 Team D
  93. 93. Team A Team B Team C Team D @manupaisable | manuelpais.net 97
  94. 94. Borat’s Law / Caveat @manupaisable | manuelpais.net 98
  95. 95. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments @manupaisable | manuelpais.net 99
  96. 96. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments @manupaisable | manuelpais.net 100
  97. 97. @manupaisable | manuelpais.net 101
  98. 98. @manupaisable | manuelpais.net 102
  99. 99. Practices for resiliency (3/3) •Aggregated logging •Monitoring & alerting •Auto-scaling @manupaisable | manuelpais.net 103
  100. 100. Books 104@manupaisable | manuelpais.net
  101. 101. releasabilitybook.com Book sample out now! Team Guide to Software Releasability by Chris O’Dell & Manuel Pais 105@manupaisable | manuelpais.net
  102. 102. @manupaisable | manuelpais.net 106 manuelpais.net
  103. 103. thank you Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net DevOps and Delivery Consultant Focused on teams and flow 107@manupaisable | manuelpais.net

×