Matt Callanan takes the 15 chapters of the famous "Continuous Delivery" book by Jez Humble & Dave Farey and distills it down into 1 hour of convincing arguments, walking through the pieces involved to make it happen including cultural challenges, automated testing, automated deployment & deployment pipelines. Not sure how to get started with DevOps? Finding it hard to convince colleagues & managers that CD is the way forward? Matt has used this presentation to help facilitate enterprise-wide adoption of Continuous Delivery. Slides from a presentation given at DevOps Brisbane March 2014.
2. Why Continuous Delivery?
• “The most important problem that we face as software
professionals is this:
– If somebody thinks of a good idea, how do we deliver it to users
as quicklyas possible?” – CD Book
• “Our highest priority is:
– to satisfy the customer through early and continuousdelivery of
valuable software.” – Agile Manifesto
3. Why Continuous Delivery?
• Cycle time should be measured in hours – not months
– How long does a single change take to release?
• Quality should be built-in to the process
– Not an afterthought of manual inspection
• Feedback should be close as possible to point of failure
– Late feedback is expensive
4. What is Continuous Delivery?
• A set of practices and principles
– aimed at, building, testing, and releasing software faster and
more frequently
Principles of Software Delivery
Create a Repeatable, Reliable Process for Releasing Software
Automate Almost Everything
Version Control Everything
If It Hurts, Do It More Often – “Bring the Pain Forward”
Build Quality In
“Done” Means Released
Everybody Is Responsible for the Delivery Process
Continuous Improvement
5. What is Continuous Delivery?
• A way of reducing cycle time
– How long does a single change take to release?
– I.e. how long it takes a simple code change to get to
production?
6. What is Continuous Delivery?
• A way of joining development, deployment, testing &
release activities
– coordinatingthem to make the process as efficient and reliable
as possible
– reducingfeedback delays by automatingmanual processes
Develop Deploy Test Release
7. Central Concept: Deployment Pipeline
• Make process visible
• Enable automated testing
– Ensure quality is built into process
• Enable automated deploys to any environment
• Improve feedback cycles
Commit
stage
Acceptance
stage
Exploratory
testing
UAT
Capacity
Testing
Production
8. What is not Continuous Delivery:
Anti-patterns
• Deploying software manually
• Deploying to prod-like env only after dev is complete
• Manual configuration management of production
16. Silos
• Ultimately, we succeed or fail as a team
– not as individuals
• Symptoms of Silos
– Dev throws work over wall to Test throws over wall to Ops
– End up spending as much time
blaming each other as fixing defects
that arise from siloed approach
http://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg
17. Repairing Silos
• How to break down silos
– Get everybody involved together from start of project
– Ensure they can communicate on a frequent, regular basis
– Increase visibilityand self-serviceability
18. Pipeline Visibility
• Let ops see the build pipeline
– Build trust!
• Information Radiators
– Build Status
– What changes went into each build?
– How many commits outstanding?
19. Production Visibility
• Let dev teams see production dashboards
– Build trust!
• Information Radiators
– Nagios, grafana, splunk
– How busy is ops?
– Is prod on fire?
20. Lean
• Example Value Stream Map
Require-
ments
Develop-
ment
Testing Staging
Capacity
Testing
Release
Value-
Added
Time
Elapsed
Non-Value Added
Time
Business Customer
25. Acting on Feedback
• The delivery team must receive feedback and
then act on it
• Involve everybody in feedback process
– Work in cross-functional teamsor meet often
– Retrospective: discuss how to improve delivery
process next iteration
• Broadcast the information
– Big, visible dashboardsensuresfeedback gets into
someone’s head
• Feedback must be acted upon
– Requires discipline and planning
– Stop and decide on a course of action
– Only once this is done should the team carry on with
their work
27. Just Culture
• In a “Just Culture”:
– Managers know that fixing the system is more
important than finding scapegoats
– Engineers are quick to admit fault
• Blameless post-mortems
29. Continuous Integration
Developer
• Commit
to
Version
Control
Build
Server
• Compile
Testing
• Unit
Tests
• Component
Tests
Reporting
• Test
Results
• Coverage
• Static
Analysis
30. Google CI @ Scale
• 20,000+ developers in 40+ offices
• 4,000+ projects under development
• Single code tree (billions of files)
• 30,000 check-ins per day
• Everyone develops and releases from head
• All builds from source
• >100 million test cases executed per day
• Anyone can roll back anyone else’s code change if it’s
causing problems (e.g. shared libraries)
http://www.infoq.com/presentations/google-test-automation (2013)
31. Move Fast & Don’t Break Things
• Ankit Mehta, Google
• Innovate
– Innovation is now not only the sexy thing to do; it is necessary for survival
• Address flaws quickly
– Address critical bugs quickly and painlessly.
– Chicken and egg:
• can’t move fast without this
• you can’t do this without moving fast.
– Have to take a leap of faith; believe in yourself and your team and take the plunge.
• Better productivity
– Automation is a key ingredient to this and that in turn helps rapid development.
• Better Code Health
– Folks don’t try to slide things in to catch a release as there is one going every day!
– Focusing on Code Health from the start will help made the 15yr software life
smoother
https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_82
35. Deployment Pipeline
• Every commit is considered a release candidate
• Not every release candidate is released
• Candidates must first pass:
– Compile, Unit Test, Static Analysis, Create/Upload RPMs
– Deploy RPM, Start server
– Smoke Tests
– Acceptance Tests
– Performance Tests
– UAT / Manual Exploratory
– Business decision to go-live
– Approval from gatekeepers
43. Testing
You can not save money if you are more worried about money, than
you are about quality.
W. Edwards Deming:
Costs ↓Focus on Quality ↑
Costs ↑ + Quality ↓Focus on Costs ↓
44. Testing
• “Eliminate the need for mass inspections, as the way of life to
achieve quality, by building qualityinto the product in the first
place.”
• “Quality doesn’t come from inspection, but from improvement of
the process. Improve the process so that defects aren’t produced in
the first place. This eliminatesthe need for inspection on a mass
basis.”
• “Routine inspection is the same as planningfor defects,
acknowledgingthat the process isn’t correct,or that the
specificationsmade no sense in the first place.”
• “Inspection is too late as well as ineffective and costly.”
http://www.signsculpt.c om.au/wp-content/upl oads /20 13/03/focus -on-quality.j pg
http://leanandkanban.wordpress.com/2011/07/15/demings-14-points/
45. W. Edwards Deming
• “Let’s make toast: I’ll burn it, you scrape it!”
https://flic.kr/p/8y8Vib
46. Jez Humble’s Testing Principles
1. Writing good acceptance tests is hard
2. Tests are first-class citizens of your project
3. Always interact with the system under test
the same way a user would
4. Continuously curate the structure of your
test suites
5. Everyone owns acceptance tests
6. Acceptance tests are responsible for
managing their own test data
47. Value of BDD?
• Customer?
• Readable English comes at a technical
cost
• Consider Journey Tests
48. Prod-Like Environments
• Every diff with Prod is a risk
• Need to take calculated risks
– Trade-offs for efficiency
– Need to understand the cost
• E.g.
– DBs
– Configuration Management
– Load Balancers
– Deployment Process
• Spot the Difference = waste
http://www.nairaland.com/attachments/1500785_spot_jpge3cbd8220a9ec91ea49adffb6c79aeb2
50. Deployment Automation
• Releases should be low-ceremony, stress-free events
– We’ve alreadypracticed deployment 100s of times using the
exact same process in prod-like environments
– Don’t have to remember complex manual steps or rely on
written instructions
– Only testing required is to verify the environment
• (not functionality – already tested with every commit)
• Testing is automated
– Releases take minutes (not hours)
51. Anti-pattern:
Deploying Software Manually
Manual Deployment Anti-Pattern Signs
Extensive, detailed release documentation
Reliance on manual testing to confirm app is correct
Explainingwhy deployment is going wrong on release day
Frequent correctionsto release process during release
Environmentsthat differ in their configuration
Releases that take more than a few minutes to perform
Releases that are unpredictable in their outcome
52. Deployment Automation
• Deployments should tend towards being fully
automated
– 1) pick version & environment
– 2) press “deploy” button
• Automated deployment scripts = up-to-date doco
• Don’t depend on the deployment expert
• Automated deployment process:
– Cheap and easy to test
– Fully auditable
– Must be used by everybody
– Should be the only way in which the software is ever deployed
55. Continuous Deployment?
• Make final “deploy to prod” step automatic
• Intuitive objection: Too risky!?
• Forces you to do the right thing
• Automated tests have to be fantastic
– automated UT/CT/FAT/NFAT cover entire app
56. Continuous Deployment?
• Can’t actually release every set of changes that
passes all tests?
– Aim to create process that lets you do so
• Pipelines:
– Repeatable, reliable,automated system for getting changes into
prod ASAP
– Highest qualitysoftware using highest quality process -
massively reducingrisks
• Logical conclusion
• Paradigm shift
57. Continuous Deployment?
• “Even if you have good reasons for not releasing
every change you make—and there are less such
reasons than you might think—you should
behave as if you were going to do so”
59. Database Migration
DB
v10
App v200
compatible with
DB v10 & v11
App v210
compatible with
DB v11
App v220
compatible with
DB v11 & v12
Time
App v230
compatible with
DB v12
…
DB
v11
DB
v12
… …
• Decouple database changesfrom application deployment
name
firstname
surname
name
firstname
surname
firstname
surname
name
firstname
surname
firstname
surname
namename
Update
other
readers
61. Configuration Management
• Provisioning & management should be automated
• Keep everything you need to create/maintain infra
under version control
– E.g. Kickstart,Puppet, DNS zone files, DHCP, SMTP, firewall,
scripts
– Inputs to deployment pipeline (same as source code)
68. Risk Management
• What about PCI/SOX/etc?
– Lock down who is able to access “privileged” envs
– Change management process for making changes
to privileged environments
– Approvalsfrom management before deployments
can be performed
– Protect against potential maliciousinterventions
– Audit all deployments
• Deployment pipeline makes it possible to
enforce these strategies while enabling
efficient delivery process
– Automation over Documentation
– Enforce Traceability
69. Risk Management
• Process for managing approvals
– Automated CR management system
– Adding access control to deployment pipeline
is a trivial exercise
• How does CAB decide whether change
should be executed?
– If risks outweigh benefits, change should not be made
• Principles:
– Keep metrics on the system & make visible:
• How long, how many waiting, proportion denied?
• MTBF/MTTR
• Cycle time
– Regular retrospectives& improve system based on feedback
http://www.vtexan.com/wp-content/uploads/2014/04/image7.png
72. CD Adoption
1. Understand: Why?
2. Get Measurable Change Fast
– (even if reaching goal takes years)
3. Start with Continuous Integration
4. Focus on Organization & Architecture
5. Culture of Continuous Improvement
Jez Humble “Adopting Continuous Delivery”: http://bit.ly/1FWOXNl
73. How to Address Architecture
• Refactor organization
– Leverage Conway’s law
– Amazon: Steve Yegge’s Platform Rant: bit.ly/shHRff
• Branch by abstraction: bit.ly/kAUbEw
• Strangler application: bit.ly/R4ZiJZ
79. HP LaserJet Improvement
• Overall development costs reduced by ~40%
• Programs developed increased by ~140%
• Development costs per program down 78%
• Resources driving innovation increased by 8X
81. HP Deployment Pipeline
• 400 developers
• 10m LOC
• Individual Git branches
– CI build runs 2hrs then auto-merges for Stage 2 then
auto-merges to master
– (Branches are an imperfect countermeasure due to length
of build)
• L2 Simulators – acceptance tests
• L3 Emulators – not real paper
• L4 20,000hrs parallel tests
– Which change causes test problem?
– Move failing L4 tests up to L1
– Test that always pass at L1 move down
86. Beyond the Book
• How to manage interconnected pipelines?
– Independence: Release one service at a time
– Be backwards compatible with current consumers
– Requires discipline:
• Short cycle time & small batch size
• Feature Switching
• System testing/monitoring
– E.g. synthetic transactions
87. Beyond the Book
• Consumer Driven Contracts
– Allows independent development and release of
dependent services
http://martinfowler.com/articles/consumerDrivenContracts.html
C
B
B
A
A
C
Code
Tests
B dev/tester
Maintains
Calls
As ‘C’ changes,
its tests for
consumers ‘A’ and
‘B’ are executed