Continuous Delivery Distilled

3,153 views

Published on

Matt Callanan takes the 15 chapters of the famous "Continuous Delivery" book by Jez Humble & Dave Farey and distills it down into 1 hour of convincing arguments, walking through the pieces involved to make it happen including cultural challenges, automated testing, automated deployment & deployment pipelines. Not sure how to get started with DevOps? Finding it hard to convince colleagues & managers that CD is the way forward? Matt has used this presentation to help facilitate enterprise-wide adoption of Continuous Delivery. Slides from a presentation given at DevOps Brisbane March 2014.

Published in: Software, Business, Technology
4 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total views
3,153
On SlideShare
0
From Embeds
0
Number of Embeds
696
Actions
Shares
0
Downloads
135
Comments
4
Likes
10
Embeds 0
No embeds

No notes for slide

Continuous Delivery Distilled

  1. 1. Continuous Delivery Distilled Matt Callanan @mcallana
  2. 2. Why Continuous Delivery? • “The most important problem that we face as software professionals is this: – If somebody thinks of a good idea, how do we deliver it to users as quicklyas possible?” – CD Book • “Our highest priority is: – to satisfy the customer through early and continuousdelivery of valuable software.” – Agile Manifesto
  3. 3. Why Continuous Delivery? • Cycle time should be measured in hours – not months – How long does a single change take to release? • Quality should be built-in to the process – Not an afterthought of manual inspection • Feedback should be close as possible to point of failure – Late feedback is expensive
  4. 4. What is Continuous Delivery? • A set of practices and principles – aimed at, building, testing, and releasing software faster and more frequently Principles of Software Delivery Create a Repeatable, Reliable Process for Releasing Software Automate Almost Everything Version Control Everything If It Hurts, Do It More Often – “Bring the Pain Forward” Build Quality In “Done” Means Released Everybody Is Responsible for the Delivery Process Continuous Improvement
  5. 5. What is Continuous Delivery? • A way of reducing cycle time – How long does a single change take to release? – I.e. how long it takes a simple code change to get to production?
  6. 6. What is Continuous Delivery? • A way of joining development, deployment, testing & release activities – coordinatingthem to make the process as efficient and reliable as possible – reducingfeedback delays by automatingmanual processes Develop Deploy Test Release
  7. 7. Central Concept: Deployment Pipeline • Make process visible • Enable automated testing – Ensure quality is built into process • Enable automated deploys to any environment • Improve feedback cycles Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production
  8. 8. What is not Continuous Delivery: Anti-patterns • Deploying software manually • Deploying to prod-like env only after dev is complete • Manual configuration management of production
  9. 9. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Building Blocks Continuous Integration
  10. 10. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  11. 11. Requirements •Analyst & Customer Design •Architect Development •Developer QA •Tester Release •Operations Months / Years! Waterfall
  12. 12. Agile
  13. 13. Agile -> Continuous Delivery
  14. 14. Agile  Culture Analyst/ Customer TestersDevs Acceptance Criteria Automation
  15. 15. DevOps Culture Analyst/ Customer Testers Ops Devs Acceptance Criteria Automation
  16. 16. Silos • Ultimately, we succeed or fail as a team – not as individuals • Symptoms of Silos – Dev throws work over wall to Test throws over wall to Ops – End up spending as much time blaming each other as fixing defects that arise from siloed approach http://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg
  17. 17. Repairing Silos • How to break down silos – Get everybody involved together from start of project – Ensure they can communicate on a frequent, regular basis – Increase visibilityand self-serviceability
  18. 18. Pipeline Visibility • Let ops see the build pipeline – Build trust! • Information Radiators – Build Status – What changes went into each build? – How many commits outstanding?
  19. 19. Production Visibility • Let dev teams see production dashboards – Build trust! • Information Radiators – Nagios, grafana, splunk – How busy is ops? – Is prod on fire?
  20. 20. Lean • Example Value Stream Map Require- ments Develop- ment Testing Staging Capacity Testing Release Value- Added Time Elapsed Non-Value Added Time Business Customer
  21. 21. Batch Size vs Risk http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change • Reducing Batch Size Reduces Risk
  22. 22. Batch Size Science
  23. 23. High Release Cost = High Batch Size Batch Size Cost
  24. 24. Low Release Cost = Low Batch Size Batch Size Cost Release Cost
  25. 25. Acting on Feedback • The delivery team must receive feedback and then act on it • Involve everybody in feedback process – Work in cross-functional teamsor meet often – Retrospective: discuss how to improve delivery process next iteration • Broadcast the information – Big, visible dashboardsensuresfeedback gets into someone’s head • Feedback must be acted upon – Requires discipline and planning – Stop and decide on a course of action – Only once this is done should the team carry on with their work
  26. 26. Andon Cord http://uvd-production.s3.amazonaws.com/wp-uploads/2015/03/andon-pull.png
  27. 27. Just Culture • In a “Just Culture”: – Managers know that fixing the system is more important than finding scapegoats – Engineers are quick to admit fault • Blameless post-mortems
  28. 28. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  29. 29. Continuous Integration Developer • Commit  to  Version   Control Build  Server • Compile Testing • Unit  Tests • Component  Tests Reporting • Test  Results • Coverage • Static  Analysis
  30. 30. Google CI @ Scale • 20,000+ developers in 40+ offices • 4,000+ projects under development • Single code tree (billions of files) • 30,000 check-ins per day • Everyone develops and releases from head • All builds from source • >100 million test cases executed per day • Anyone can roll back anyone else’s code change if it’s causing problems (e.g. shared libraries) http://www.infoq.com/presentations/google-test-automation (2013)
  31. 31. Move Fast & Don’t Break Things • Ankit Mehta, Google • Innovate – Innovation is now not only the sexy thing to do; it is necessary for survival • Address flaws quickly – Address critical bugs quickly and painlessly. – Chicken and egg: • can’t move fast without this • you can’t do this without moving fast. – Have to take a leap of faith; believe in yourself and your team and take the plunge. • Better productivity – Automation is a key ingredient to this and that in turn helps rapid development. • Better Code Health – Folks don’t try to slide things in to catch a release as there is one going every day! – Focusing on Code Health from the start will help made the 15yr software life smoother https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_82
  32. 32. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  33. 33. Deployment Pipeline Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Increasing confidence in build’s production readiness Environments become more production-like Faster feedback • Tension: – Environmental Confidence vs Feedback Speed
  34. 34. Artifact Repository (e.g. Yum) Version Control (e.g. Git) Deployment Pipeline Source Code Commit stage Compile Static analysis Unit Tests Coverage Build binary artifacts Acceptance stage Configure Env Deploy Compliance Tests Smoke Tests Acceptance Tests artifactsartifacts UAT Deploy Compliance Tests Smoke Tests Capacity Stage Deploy Compliance Tests Smoke Tests Load Tests Production Deploy Compliance Tests Smoke Tests artifacts CM code/data, Orchestration, Puppet, Chef, Fabric, Ansible, etc Testers Self-service deploys Ops Push-button releases Env & App Config Java, Ruby, Groovy, Scala, config, Maven, Gradle, etc
  35. 35. Deployment Pipeline • Every commit is considered a release candidate • Not every release candidate is released • Candidates must first pass: – Compile, Unit Test, Static Analysis, Create/Upload RPMs – Deploy RPM, Start server – Smoke Tests – Acceptance Tests – Performance Tests – UAT / Manual Exploratory – Business decision to go-live – Approval from gatekeepers
  36. 36. Deployment Pipeline http://continuousdelivery.com/2010/02/continuous-delivery/
  37. 37. Deployment Pipeline - Scripts • Deployment pipelines are powered by scripts • Script Ideals – Obey the Unix Philosophy– do one thing well – Environment-agnostic – Sensible exit code • Script Architecture – Build Scripts • Convert build output into deployment input – Deploy Scripts • Environment-independent instructionsto deploy software remotely • Can run on central command-and-control server – Test Scripts • Repeatable instructionsto invoke automated testing
  38. 38. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  39. 39. Disciplines Incremental development Mainline development • Can’t even do CI with multiple branches Feature switches Keep code releasable Small releases Keep automated tests green • TDD: Red before Green Disciplined Development
  40. 40. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  41. 41. Testing http://lisacrispin.com/2011/11/08/using-­‐the-­‐agile-­‐testing-­‐quadrants/
  42. 42. Testing Triangle Unit Manual Automated Other Exploratory System Acceptance Unit
  43. 43. Testing You can not save money if you are more worried about money, than you are about quality. W. Edwards Deming: Costs ↓Focus on Quality ↑ Costs ↑ + Quality ↓Focus on Costs ↓
  44. 44. Testing • “Eliminate the need for mass inspections, as the way of life to achieve quality, by building qualityinto the product in the first place.” • “Quality doesn’t come from inspection, but from improvement of the process. Improve the process so that defects aren’t produced in the first place. This eliminatesthe need for inspection on a mass basis.” • “Routine inspection is the same as planningfor defects, acknowledgingthat the process isn’t correct,or that the specificationsmade no sense in the first place.” • “Inspection is too late as well as ineffective and costly.” http://www.signsculpt.c om.au/wp-content/upl oads /20 13/03/focus -on-quality.j pg http://leanandkanban.wordpress.com/2011/07/15/demings-14-points/
  45. 45. W. Edwards Deming • “Let’s make toast: I’ll burn it, you scrape it!” https://flic.kr/p/8y8Vib
  46. 46. Jez Humble’s Testing Principles 1. Writing good acceptance tests is hard 2. Tests are first-class citizens of your project 3. Always interact with the system under test the same way a user would 4. Continuously curate the structure of your test suites 5. Everyone owns acceptance tests 6. Acceptance tests are responsible for managing their own test data
  47. 47. Value of BDD? • Customer? • Readable English comes at a technical cost • Consider Journey Tests
  48. 48. Prod-Like Environments • Every diff with Prod is a risk • Need to take calculated risks – Trade-offs for efficiency – Need to understand the cost • E.g. – DBs – Configuration Management – Load Balancers – Deployment Process • Spot the Difference = waste http://www.nairaland.com/attachments/1500785_spot_jpge3cbd8220a9ec91ea49adffb6c79aeb2
  49. 49. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  50. 50. Deployment Automation • Releases should be low-ceremony, stress-free events – We’ve alreadypracticed deployment 100s of times using the exact same process in prod-like environments – Don’t have to remember complex manual steps or rely on written instructions – Only testing required is to verify the environment • (not functionality – already tested with every commit) • Testing is automated – Releases take minutes (not hours)
  51. 51. Anti-pattern: Deploying Software Manually Manual Deployment Anti-Pattern Signs Extensive, detailed release documentation Reliance on manual testing to confirm app is correct Explainingwhy deployment is going wrong on release day Frequent correctionsto release process during release Environmentsthat differ in their configuration Releases that take more than a few minutes to perform Releases that are unpredictable in their outcome
  52. 52. Deployment Automation • Deployments should tend towards being fully automated – 1) pick version & environment – 2) press “deploy” button • Automated deployment scripts = up-to-date doco • Don’t depend on the deployment expert • Automated deployment process: – Cheap and easy to test – Fully auditable – Must be used by everybody – Should be the only way in which the software is ever deployed
  53. 53. “CONTINUOUS DEPLOYMENT”?
  54. 54. Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Continuous  Delivery Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Continuous  Deployment Automatic  Trigger Manual  Trigger
  55. 55. Continuous Deployment? • Make final “deploy to prod” step automatic • Intuitive objection: Too risky!? • Forces you to do the right thing • Automated tests have to be fantastic – automated UT/CT/FAT/NFAT cover entire app
  56. 56. Continuous Deployment? • Can’t actually release every set of changes that passes all tests? – Aim to create process that lets you do so • Pipelines: – Repeatable, reliable,automated system for getting changes into prod ASAP – Highest qualitysoftware using highest quality process - massively reducingrisks • Logical conclusion • Paradigm shift
  57. 57. Continuous Deployment? • “Even if you have good reasons for not releasing every change you make—and there are less such reasons than you might think—you should behave as if you were going to do so”
  58. 58. DATA MANAGEMENT
  59. 59. Database Migration DB v10 App v200 compatible with DB v10 & v11 App v210 compatible with DB v11 App v220 compatible with DB v11 & v12 Time App v230 compatible with DB v12 … DB v11 DB v12 … … • Decouple database changesfrom application deployment name firstname surname name firstname surname firstname surname name firstname surname firstname surname namename Update  other readers
  60. 60. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  61. 61. Configuration Management • Provisioning & management should be automated • Keep everything you need to create/maintain infra under version control – E.g. Kickstart,Puppet, DNS zone files, DHCP, SMTP, firewall, scripts – Inputs to deployment pipeline (same as source code)
  62. 62. Deployment Toolchain • Examples – TeamCity -> Yum -> Fabric -> Puppet/Hiera – Bamboo -> Yum -> Rundeck -> Chef – Jenkins -> Nexus -> Ansible – Jenkins -> Yum -> AMInator -> ASGard CI Artifact Repo Orchestra- tion Config Mgmt
  63. 63. DevOps Toolchain http://www.slideshare.net/AnthonyShortland/dto-chefconf2012
  64. 64. Example DevOps Toolchain http://www.slideshare.net/AnthonyShortland/dto-chefconf2012
  65. 65. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  66. 66. Risk Management Confidence↑Repeatability↑
  67. 67. Risk Management http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
  68. 68. Risk Management • What about PCI/SOX/etc? – Lock down who is able to access “privileged” envs – Change management process for making changes to privileged environments – Approvalsfrom management before deployments can be performed – Protect against potential maliciousinterventions – Audit all deployments • Deployment pipeline makes it possible to enforce these strategies while enabling efficient delivery process – Automation over Documentation – Enforce Traceability
  69. 69. Risk Management • Process for managing approvals – Automated CR management system – Adding access control to deployment pipeline is a trivial exercise • How does CAB decide whether change should be executed? – If risks outweigh benefits, change should not be made • Principles: – Keep metrics on the system & make visible: • How long, how many waiting, proportion denied? • MTBF/MTTR • Cycle time – Regular retrospectives& improve system based on feedback http://www.vtexan.com/wp-content/uploads/2014/04/image7.png
  70. 70. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks
  71. 71. ADOPTING CONTINUOUS DELIVERY
  72. 72. CD Adoption 1. Understand: Why? 2. Get Measurable Change Fast – (even if reaching goal takes years) 3. Start with Continuous Integration 4. Focus on Organization & Architecture 5. Culture of Continuous Improvement Jez Humble “Adopting Continuous Delivery”: http://bit.ly/1FWOXNl
  73. 73. How to Address Architecture • Refactor organization – Leverage Conway’s law – Amazon: Steve Yegge’s Platform Rant: bit.ly/shHRff • Branch by abstraction: bit.ly/kAUbEw • Strangler application: bit.ly/R4ZiJZ
  74. 74. Continuous Improvement
  75. 75. CD Maturity Model Figure 15.1 Maturity model
  76. 76. CD Maturity Model https://flic.kr/p/afDTK9 Below this line, we’re slipping backwards Above this line, we’re climbing the ladder of maturity
  77. 77. CASE STUDY
  78. 78. HP LaserJet Firmware http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile
  79. 79. HP LaserJet Improvement • Overall development costs reduced by ~40% • Programs developed increased by ~140% • Development costs per program down 78% • Resources driving innovation increased by 8X
  80. 80. HP LaserJet Improvement http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile
  81. 81. HP Deployment Pipeline • 400 developers • 10m LOC • Individual Git branches – CI build runs 2hrs then auto-merges for Stage 2 then auto-merges to master – (Branches are an imperfect countermeasure due to length of build) • L2 Simulators – acceptance tests • L3 Emulators – not real paper • L4 20,000hrs parallel tests – Which change causes test problem? – Move failing L4 tests up to L1 – Test that always pass at L1 move down
  82. 82. LaserJet Deployment Pipeline http://www.slideshare.net/jezhumble/devops-culture-and-practice s-to-create-flow
  83. 83. SUMMARY
  84. 84. In One Slide Quality Cycle Time
  85. 85. BEYOND THE BOOK
  86. 86. Beyond the Book • How to manage interconnected pipelines? – Independence: Release one service at a time – Be backwards compatible with current consumers – Requires discipline: • Short cycle time & small batch size • Feature Switching • System testing/monitoring – E.g. synthetic transactions
  87. 87. Beyond the Book • Consumer Driven Contracts – Allows independent development and release of dependent services http://martinfowler.com/articles/consumerDrivenContracts.html C B B A A C Code Tests B dev/tester Maintains Calls As ‘C’ changes, its tests for consumers ‘A’ and ‘B’ are executed

×