Continuous Delivery Distilled

1. Continuous Delivery Distilled Matt Callanan @mcallana

2. Why Continuous Delivery? • “The most important problem that we face as software professionals is this: – If somebody thinks of a good idea, how do we deliver it to users as quicklyas possible?” – CD Book • “Our highest priority is: – to satisfy the customer through early and continuousdelivery of valuable software.” – Agile Manifesto

3. Why Continuous Delivery? • Cycle time should be measured in hours – not months – How long does a single change take to release? • Quality should be built-in to the process – Not an afterthought of manual inspection • Feedback should be close as possible to point of failure – Late feedback is expensive

4. What is Continuous Delivery? • A set of practices and principles – aimed at, building, testing, and releasing software faster and more frequently Principles of Software Delivery Create a Repeatable, Reliable Process for Releasing Software Automate Almost Everything Version Control Everything If It Hurts, Do It More Often – “Bring the Pain Forward” Build Quality In “Done” Means Released Everybody Is Responsible for the Delivery Process Continuous Improvement

5. What is Continuous Delivery? • A way of reducing cycle time – How long does a single change take to release? – I.e. how long it takes a simple code change to get to production?

6. What is Continuous Delivery? • A way of joining development, deployment, testing & release activities – coordinatingthem to make the process as efficient and reliable as possible – reducingfeedback delays by automatingmanual processes Develop Deploy Test Release

7. Central Concept: Deployment Pipeline • Make process visible • Enable automated testing – Ensure quality is built into process • Enable automated deploys to any environment • Improve feedback cycles Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production

8. What is not Continuous Delivery: Anti-patterns • Deploying software manually • Deploying to prod-like env only after dev is complete • Manual configuration management of production

9. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Building Blocks Continuous Integration

10. Deployment Pipeline Agile / Lean / DevOps / Culture Configuration Management Testing Deployment Automation Disciplined Development Risk Management Continuous Integration Building Blocks

11. Requirements •Analyst & Customer Design •Architect Development •Developer QA •Tester Release •Operations Months / Years! Waterfall

12. Agile

13. Agile -> Continuous Delivery

14. Agile Culture Analyst/ Customer TestersDevs Acceptance Criteria Automation

15. DevOps Culture Analyst/ Customer Testers Ops Devs Acceptance Criteria Automation

16. Silos • Ultimately, we succeed or fail as a team – not as individuals • Symptoms of Silos – Dev throws work over wall to Test throws over wall to Ops – End up spending as much time blaming each other as fixing defects that arise from siloed approach http://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg

17. Repairing Silos • How to break down silos – Get everybody involved together from start of project – Ensure they can communicate on a frequent, regular basis – Increase visibilityand self-serviceability

18. Pipeline Visibility • Let ops see the build pipeline – Build trust! • Information Radiators – Build Status – What changes went into each build? – How many commits outstanding?

19. Production Visibility • Let dev teams see production dashboards – Build trust! • Information Radiators – Nagios, grafana, splunk – How busy is ops? – Is prod on fire?

20. Lean • Example Value Stream Map Require- ments Develop- ment Testing Staging Capacity Testing Release Value- Added Time Elapsed Non-Value Added Time Business Customer

21. Batch Size vs Risk http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change • Reducing Batch Size Reduces Risk

22. Batch Size Science

23. High Release Cost = High Batch Size Batch Size Cost

24. Low Release Cost = Low Batch Size Batch Size Cost Release Cost

25. Acting on Feedback • The delivery team must receive feedback and then act on it • Involve everybody in feedback process – Work in cross-functional teamsor meet often – Retrospective: discuss how to improve delivery process next iteration • Broadcast the information – Big, visible dashboardsensuresfeedback gets into someone’s head • Feedback must be acted upon – Requires discipline and planning – Stop and decide on a course of action – Only once this is done should the team carry on with their work

26. Andon Cord http://uvd-production.s3.amazonaws.com/wp-uploads/2015/03/andon-pull.png

27. Just Culture • In a “Just Culture”: – Managers know that fixing the system is more important than finding scapegoats – Engineers are quick to admit fault • Blameless post-mortems

29. Continuous Integration Developer • Commit to Version Control Build Server • Compile Testing • Unit Tests • Component Tests Reporting • Test Results • Coverage • Static Analysis

30. Google CI @ Scale • 20,000+ developers in 40+ offices • 4,000+ projects under development • Single code tree (billions of files) • 30,000 check-ins per day • Everyone develops and releases from head • All builds from source • >100 million test cases executed per day • Anyone can roll back anyone else’s code change if it’s causing problems (e.g. shared libraries) http://www.infoq.com/presentations/google-test-automation (2013)

31. Move Fast & Don’t Break Things • Ankit Mehta, Google • Innovate – Innovation is now not only the sexy thing to do; it is necessary for survival • Address flaws quickly – Address critical bugs quickly and painlessly. – Chicken and egg: • can’t move fast without this • you can’t do this without moving fast. – Have to take a leap of faith; believe in yourself and your team and take the plunge. • Better productivity – Automation is a key ingredient to this and that in turn helps rapid development. • Better Code Health – Folks don’t try to slide things in to catch a release as there is one going every day! – Focusing on Code Health from the start will help made the 15yr software life smoother https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_82

33. Deployment Pipeline Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Increasing confidence in build’s production readiness Environments become more production-like Faster feedback • Tension: – Environmental Confidence vs Feedback Speed

34. Artifact Repository (e.g. Yum) Version Control (e.g. Git) Deployment Pipeline Source Code Commit stage Compile Static analysis Unit Tests Coverage Build binary artifacts Acceptance stage Configure Env Deploy Compliance Tests Smoke Tests Acceptance Tests artifactsartifacts UAT Deploy Compliance Tests Smoke Tests Capacity Stage Deploy Compliance Tests Smoke Tests Load Tests Production Deploy Compliance Tests Smoke Tests artifacts CM code/data, Orchestration, Puppet, Chef, Fabric, Ansible, etc Testers Self-service deploys Ops Push-button releases Env & App Config Java, Ruby, Groovy, Scala, config, Maven, Gradle, etc

35. Deployment Pipeline • Every commit is considered a release candidate • Not every release candidate is released • Candidates must first pass: – Compile, Unit Test, Static Analysis, Create/Upload RPMs – Deploy RPM, Start server – Smoke Tests – Acceptance Tests – Performance Tests – UAT / Manual Exploratory – Business decision to go-live – Approval from gatekeepers

36. Deployment Pipeline http://continuousdelivery.com/2010/02/continuous-delivery/

37. Deployment Pipeline - Scripts • Deployment pipelines are powered by scripts • Script Ideals – Obey the Unix Philosophy– do one thing well – Environment-agnostic – Sensible exit code • Script Architecture – Build Scripts • Convert build output into deployment input – Deploy Scripts • Environment-independent instructionsto deploy software remotely • Can run on central command-and-control server – Test Scripts • Repeatable instructionsto invoke automated testing

39. Disciplines Incremental development Mainline development • Can’t even do CI with multiple branches Feature switches Keep code releasable Small releases Keep automated tests green • TDD: Red before Green Disciplined Development

41. Testing http://lisacrispin.com/2011/11/08/using-‐the-‐agile-‐testing-‐quadrants/

42. Testing Triangle Unit Manual Automated Other Exploratory System Acceptance Unit

43. Testing You can not save money if you are more worried about money, than you are about quality. W. Edwards Deming: Costs ↓Focus on Quality ↑ Costs ↑ + Quality ↓Focus on Costs ↓

44. Testing • “Eliminate the need for mass inspections, as the way of life to achieve quality, by building qualityinto the product in the first place.” • “Quality doesn’t come from inspection, but from improvement of the process. Improve the process so that defects aren’t produced in the first place. This eliminatesthe need for inspection on a mass basis.” • “Routine inspection is the same as planningfor defects, acknowledgingthat the process isn’t correct,or that the specificationsmade no sense in the first place.” • “Inspection is too late as well as ineffective and costly.” http://www.signsculpt.c om.au/wp-content/upl oads /20 13/03/focus -on-quality.j pg http://leanandkanban.wordpress.com/2011/07/15/demings-14-points/

45. W. Edwards Deming • “Let’s make toast: I’ll burn it, you scrape it!” https://flic.kr/p/8y8Vib

46. Jez Humble’s Testing Principles 1. Writing good acceptance tests is hard 2. Tests are first-class citizens of your project 3. Always interact with the system under test the same way a user would 4. Continuously curate the structure of your test suites 5. Everyone owns acceptance tests 6. Acceptance tests are responsible for managing their own test data

47. Value of BDD? • Customer? • Readable English comes at a technical cost • Consider Journey Tests

48. Prod-Like Environments • Every diff with Prod is a risk • Need to take calculated risks – Trade-offs for efficiency – Need to understand the cost • E.g. – DBs – Configuration Management – Load Balancers – Deployment Process • Spot the Difference = waste http://www.nairaland.com/attachments/1500785_spot_jpge3cbd8220a9ec91ea49adffb6c79aeb2

50. Deployment Automation • Releases should be low-ceremony, stress-free events – We’ve alreadypracticed deployment 100s of times using the exact same process in prod-like environments – Don’t have to remember complex manual steps or rely on written instructions – Only testing required is to verify the environment • (not functionality – already tested with every commit) • Testing is automated – Releases take minutes (not hours)

51. Anti-pattern: Deploying Software Manually Manual Deployment Anti-Pattern Signs Extensive, detailed release documentation Reliance on manual testing to confirm app is correct Explainingwhy deployment is going wrong on release day Frequent correctionsto release process during release Environmentsthat differ in their configuration Releases that take more than a few minutes to perform Releases that are unpredictable in their outcome

52. Deployment Automation • Deployments should tend towards being fully automated – 1) pick version & environment – 2) press “deploy” button • Automated deployment scripts = up-to-date doco • Don’t depend on the deployment expert • Automated deployment process: – Cheap and easy to test – Fully auditable – Must be used by everybody – Should be the only way in which the software is ever deployed

53. “CONTINUOUS DEPLOYMENT”?

54. Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Continuous Delivery Commit stage Acceptance stage Exploratory testing UAT Capacity Testing Production Continuous Deployment Automatic Trigger Manual Trigger

55. Continuous Deployment? • Make final “deploy to prod” step automatic • Intuitive objection: Too risky!? • Forces you to do the right thing • Automated tests have to be fantastic – automated UT/CT/FAT/NFAT cover entire app

56. Continuous Deployment? • Can’t actually release every set of changes that passes all tests? – Aim to create process that lets you do so • Pipelines: – Repeatable, reliable,automated system for getting changes into prod ASAP – Highest qualitysoftware using highest quality process - massively reducingrisks • Logical conclusion • Paradigm shift

57. Continuous Deployment? • “Even if you have good reasons for not releasing every change you make—and there are less such reasons than you might think—you should behave as if you were going to do so”

58. DATA MANAGEMENT

59. Database Migration DB v10 App v200 compatible with DB v10 & v11 App v210 compatible with DB v11 App v220 compatible with DB v11 & v12 Time App v230 compatible with DB v12 … DB v11 DB v12 … … • Decouple database changesfrom application deployment name firstname surname name firstname surname firstname surname name firstname surname firstname surname namename Update other readers

61. Configuration Management • Provisioning & management should be automated • Keep everything you need to create/maintain infra under version control – E.g. Kickstart,Puppet, DNS zone files, DHCP, SMTP, firewall, scripts – Inputs to deployment pipeline (same as source code)

62. Deployment Toolchain • Examples – TeamCity -> Yum -> Fabric -> Puppet/Hiera – Bamboo -> Yum -> Rundeck -> Chef – Jenkins -> Nexus -> Ansible – Jenkins -> Yum -> AMInator -> ASGard CI Artifact Repo Orchestra- tion Config Mgmt

63. DevOps Toolchain http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

64. Example DevOps Toolchain http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

66. Risk Management Confidence↑Repeatability↑

67. Risk Management http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

68. Risk Management • What about PCI/SOX/etc? – Lock down who is able to access “privileged” envs – Change management process for making changes to privileged environments – Approvalsfrom management before deployments can be performed – Protect against potential maliciousinterventions – Audit all deployments • Deployment pipeline makes it possible to enforce these strategies while enabling efficient delivery process – Automation over Documentation – Enforce Traceability

69. Risk Management • Process for managing approvals – Automated CR management system – Adding access control to deployment pipeline is a trivial exercise • How does CAB decide whether change should be executed? – If risks outweigh benefits, change should not be made • Principles: – Keep metrics on the system & make visible: • How long, how many waiting, proportion denied? • MTBF/MTTR • Cycle time – Regular retrospectives& improve system based on feedback http://www.vtexan.com/wp-content/uploads/2014/04/image7.png

71. ADOPTING CONTINUOUS DELIVERY

72. CD Adoption 1. Understand: Why? 2. Get Measurable Change Fast – (even if reaching goal takes years) 3. Start with Continuous Integration 4. Focus on Organization & Architecture 5. Culture of Continuous Improvement Jez Humble “Adopting Continuous Delivery”: http://bit.ly/1FWOXNl

73. How to Address Architecture • Refactor organization – Leverage Conway’s law – Amazon: Steve Yegge’s Platform Rant: bit.ly/shHRff • Branch by abstraction: bit.ly/kAUbEw • Strangler application: bit.ly/R4ZiJZ

74. Continuous Improvement

75. CD Maturity Model Figure 15.1 Maturity model

76. CD Maturity Model https://flic.kr/p/afDTK9 Below this line, we’re slipping backwards Above this line, we’re climbing the ladder of maturity

77. CASE STUDY

78. HP LaserJet Firmware http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile

79. HP LaserJet Improvement • Overall development costs reduced by ~40% • Programs developed increased by ~140% • Development costs per program down 78% • Resources driving innovation increased by 8X

80. HP LaserJet Improvement http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile

81. HP Deployment Pipeline • 400 developers • 10m LOC • Individual Git branches – CI build runs 2hrs then auto-merges for Stage 2 then auto-merges to master – (Branches are an imperfect countermeasure due to length of build) • L2 Simulators – acceptance tests • L3 Emulators – not real paper • L4 20,000hrs parallel tests – Which change causes test problem? – Move failing L4 tests up to L1 – Test that always pass at L1 move down

82. LaserJet Deployment Pipeline http://www.slideshare.net/jezhumble/devops-culture-and-practice s-to-create-flow

83. SUMMARY

84. In One Slide Quality Cycle Time

85. BEYOND THE BOOK

86. Beyond the Book • How to manage interconnected pipelines? – Independence: Release one service at a time – Be backwards compatible with current consumers – Requires discipline: • Short cycle time & small batch size • Feature Switching • System testing/monitoring – E.g. synthetic transactions

87. Beyond the Book • Consumer Driven Contracts – Allows independent development and release of dependent services http://martinfowler.com/articles/consumerDrivenContracts.html C B B A A C Code Tests B dev/tester Maintains Calls As ‘C’ changes, its tests for consumers ‘A’ and ‘B’ are executed