Continuous Delivery Distilled

Continuous
Delivery
Distilled
Matt Callanan
@mcallana

Why Continuous Delivery?
• “The most important problem that we face as software
professionals is this:
– If somebody thinks of a good idea, how do we deliver it to users
as quicklyas possible?” – CD Book
• “Our highest priority is:
– to satisfy the customer through early and continuousdelivery of
valuable software.” – Agile Manifesto

Why Continuous Delivery?
• Cycle time should be measured in hours – not months
– How long does a single change take to release?
• Quality should be built-in to the process
– Not an afterthought of manual inspection
• Feedback should be close as possible to point of failure
– Late feedback is expensive

What is Continuous Delivery?
• A set of practices and principles
– aimed at, building, testing, and releasing software faster and
more frequently
Principles of Software Delivery
Create a Repeatable, Reliable Process for Releasing Software
Automate Almost Everything
Version Control Everything
If It Hurts, Do It More Often – “Bring the Pain Forward”
Build Quality In
“Done” Means Released
Everybody Is Responsible for the Delivery Process
Continuous Improvement

• A way of reducing cycle time
– How long does a single change take to release?
– I.e. how long it takes a simple code change to get to
production?

• A way of joining development, deployment, testing &
release activities
– coordinatingthem to make the process as efficient and reliable
as possible
– reducingfeedback delays by automatingmanual processes
Develop Deploy Test Release

Central Concept: Deployment Pipeline
• Make process visible
• Enable automated testing
– Ensure quality is built into process
• Enable automated deploys to any environment
• Improve feedback cycles
Commit
stage
Acceptance
stage
Exploratory
testing
UAT
Capacity
Testing
Production

What is not Continuous Delivery:
Anti-patterns
• Deploying software manually
• Deploying to prod-like env only after dev is complete
• Manual configuration management of production

Deployment Pipeline
Agile / Lean / DevOps / Culture
Configuration
Management
Testing
Deployment
Automation
Disciplined
Development
Risk Management
Building Blocks
Continuous Integration

Deployment Pipeline
Agile / Lean / DevOps / Culture
Configuration
Management
Testing
Deployment
Automation
Disciplined
Development
Risk Management
Building Blocks

Requirements
•Analyst &
Customer
Design •Architect
Development •Developer
QA •Tester
Release •Operations
Months / Years!
Waterfall

Agile Culture
Analyst/
Customer
TestersDevs
Acceptance
Criteria
Automation

DevOps Culture
Analyst/
Customer
Testers
Ops
Devs
Acceptance
Criteria
Automation

Silos
• Ultimately, we succeed or fail as a team
– not as individuals
• Symptoms of Silos
– Dev throws work over wall to Test throws over wall to Ops
– End up spending as much time
blaming each other as fixing defects
that arise from siloed approach
http://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg

Repairing Silos
• How to break down silos
– Get everybody involved together from start of project
– Ensure they can communicate on a frequent, regular basis
– Increase visibilityand self-serviceability

Pipeline Visibility
• Let ops see the build pipeline
– Build trust!
• Information Radiators
– Build Status
– What changes went into each build?
– How many commits outstanding?

Production Visibility
• Let dev teams see production dashboards
– Build trust!
• Information Radiators
– Nagios, grafana, splunk
– How busy is ops?
– Is prod on fire?

Lean
• Example Value Stream Map
Require-
ments
Develop-
ment
Testing Staging
Capacity
Testing
Release
Value-
Added
Time
Elapsed
Non-Value Added
Time
Business Customer

Batch Size vs Risk
http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
• Reducing Batch Size Reduces Risk

High Release Cost = High Batch Size
Batch Size
Cost

Low Release Cost = Low Batch Size
Batch Size
Cost
Release Cost

Acting on Feedback
• The delivery team must receive feedback and
then act on it
• Involve everybody in feedback process
– Work in cross-functional teamsor meet often
– Retrospective: discuss how to improve delivery
process next iteration
• Broadcast the information
– Big, visible dashboardsensuresfeedback gets into
someone’s head
• Feedback must be acted upon
– Requires discipline and planning
– Stop and decide on a course of action
– Only once this is done should the team carry on with
their work

Andon Cord
http://uvd-production.s3.amazonaws.com/wp-uploads/2015/03/andon-pull.png

Just Culture
• In a “Just Culture”:
– Managers know that fixing the system is more
important than finding scapegoats
– Engineers are quick to admit fault
• Blameless post-mortems

Developer
• Commit
to
Version

Control
Build
Server
• Compile
Testing
• Unit
Tests
• Component
Tests
Reporting
• Test
Results
• Coverage
• Static
Analysis

Google CI @ Scale
• 20,000+ developers in 40+ offices
• 4,000+ projects under development
• Single code tree (billions of files)
• 30,000 check-ins per day
• Everyone develops and releases from head
• All builds from source
• >100 million test cases executed per day
• Anyone can roll back anyone else’s code change if it’s
causing problems (e.g. shared libraries)
http://www.infoq.com/presentations/google-test-automation (2013)

Move Fast & Don’t Break Things
• Ankit Mehta, Google
• Innovate
– Innovation is now not only the sexy thing to do; it is necessary for survival
• Address flaws quickly
– Address critical bugs quickly and painlessly.
– Chicken and egg:
• can’t move fast without this
• you can’t do this without moving fast.
– Have to take a leap of faith; believe in yourself and your team and take the plunge.
• Better productivity
– Automation is a key ingredient to this and that in turn helps rapid development.
• Better Code Health
– Folks don’t try to slide things in to catch a release as there is one going every day!
– Focusing on Code Health from the start will help made the 15yr software life
smoother
https://docs.google.com/presentation/d/15gNk21rjer3xo-b1ZqyQVGebOp_aPvHU3YH7YnOMxtE/edit#slide=id.g437663ce1_53_82

Deployment Pipeline
Commit
stage
Acceptance
stage
Exploratory
testing
UAT
Capacity
Testing
Production
Increasing confidence in build’s production readiness
Environments become more production-like
Faster feedback
• Tension:
– Environmental Confidence vs Feedback Speed

Artifact Repository (e.g. Yum)
Version Control (e.g. Git)
Deployment Pipeline
Source
Code
Commit stage
Compile
Static analysis
Unit Tests
Coverage
Build binary artifacts
Acceptance stage
Configure Env
Deploy
Compliance Tests
Smoke Tests
Acceptance Tests
artifactsartifacts
UAT
Deploy
Compliance Tests
Smoke Tests
Capacity Stage
Deploy
Compliance Tests
Smoke Tests
Load Tests
Production
Deploy
Compliance Tests
Smoke Tests
artifacts
CM code/data,
Orchestration, Puppet,
Chef, Fabric, Ansible,
etc
Testers
Self-service
deploys
Ops
Push-button
releases
Env &
App
Config
Java, Ruby, Groovy,
Scala, config, Maven,
Gradle, etc

Deployment Pipeline
• Every commit is considered a release candidate
• Not every release candidate is released
• Candidates must first pass:
– Compile, Unit Test, Static Analysis, Create/Upload RPMs
– Deploy RPM, Start server
– Smoke Tests
– Acceptance Tests
– Performance Tests
– UAT / Manual Exploratory
– Business decision to go-live
– Approval from gatekeepers

Deployment Pipeline
http://continuousdelivery.com/2010/02/continuous-delivery/

Deployment Pipeline - Scripts
• Deployment pipelines are powered by scripts
• Script Ideals
– Obey the Unix Philosophy– do one thing well
– Environment-agnostic
– Sensible exit code
• Script Architecture
– Build Scripts
• Convert build output into deployment input
– Deploy Scripts
• Environment-independent instructionsto deploy software remotely
• Can run on central command-and-control server
– Test Scripts
• Repeatable instructionsto invoke automated testing

Disciplines
Incremental development
Mainline development
• Can’t even do CI with multiple branches
Feature switches
Keep code releasable
Small releases
Keep automated tests green
• TDD: Red before Green
Disciplined Development

Testing
http://lisacrispin.com/2011/11/08/using-‐the-‐agile-‐testing-‐quadrants/

Testing Triangle
Unit
Manual
Automated
Other
Exploratory
System
Acceptance
Unit

Testing
You can not save money if you are more worried about money, than
you are about quality.
W. Edwards Deming:
Costs ↓Focus on Quality ↑
Costs ↑ + Quality ↓Focus on Costs ↓

Testing
• “Eliminate the need for mass inspections, as the way of life to
achieve quality, by building qualityinto the product in the first
place.”
• “Quality doesn’t come from inspection, but from improvement of
the process. Improve the process so that defects aren’t produced in
the first place. This eliminatesthe need for inspection on a mass
basis.”
• “Routine inspection is the same as planningfor defects,
acknowledgingthat the process isn’t correct,or that the
specificationsmade no sense in the first place.”
• “Inspection is too late as well as ineffective and costly.”
http://www.signsculpt.c om.au/wp-content/upl oads /20 13/03/focus -on-quality.j pg
http://leanandkanban.wordpress.com/2011/07/15/demings-14-points/

W. Edwards Deming
• “Let’s make toast: I’ll burn it, you scrape it!”
https://flic.kr/p/8y8Vib

Jez Humble’s Testing Principles
1. Writing good acceptance tests is hard
2. Tests are first-class citizens of your project
3. Always interact with the system under test
the same way a user would
4. Continuously curate the structure of your
test suites
5. Everyone owns acceptance tests
6. Acceptance tests are responsible for
managing their own test data

Value of BDD?
• Customer?
• Readable English comes at a technical
cost
• Consider Journey Tests

Prod-Like Environments
• Every diff with Prod is a risk
• Need to take calculated risks
– Trade-offs for efficiency
– Need to understand the cost
• E.g.
– DBs
– Configuration Management
– Load Balancers
– Deployment Process
• Spot the Difference = waste
http://www.nairaland.com/attachments/1500785_spot_jpge3cbd8220a9ec91ea49adffb6c79aeb2

Deployment Automation
• Releases should be low-ceremony, stress-free events
– We’ve alreadypracticed deployment 100s of times using the
exact same process in prod-like environments
– Don’t have to remember complex manual steps or rely on
written instructions
– Only testing required is to verify the environment
• (not functionality – already tested with every commit)
• Testing is automated
– Releases take minutes (not hours)

Anti-pattern:
Deploying Software Manually
Manual Deployment Anti-Pattern Signs
Extensive, detailed release documentation
Reliance on manual testing to confirm app is correct
Explainingwhy deployment is going wrong on release day
Frequent correctionsto release process during release
Environmentsthat differ in their configuration
Releases that take more than a few minutes to perform
Releases that are unpredictable in their outcome

Deployment Automation
• Deployments should tend towards being fully
automated
– 1) pick version & environment
– 2) press “deploy” button
• Automated deployment scripts = up-to-date doco
• Don’t depend on the deployment expert
• Automated deployment process:
– Cheap and easy to test
– Fully auditable
– Must be used by everybody
– Should be the only way in which the software is ever deployed

Commit
stage
Acceptance
stage
Exploratory
testing
UAT
Capacity
Testing
Production
Continuous
Delivery
Commit
stage
Acceptance
stage
Exploratory
testing
UAT
Capacity
Testing
Production
Continuous
Deployment
Automatic
Trigger Manual
Trigger

Continuous Deployment?
• Make final “deploy to prod” step automatic
• Intuitive objection: Too risky!?
• Forces you to do the right thing
• Automated tests have to be fantastic
– automated UT/CT/FAT/NFAT cover entire app

• Can’t actually release every set of changes that
passes all tests?
– Aim to create process that lets you do so
• Pipelines:
– Repeatable, reliable,automated system for getting changes into
prod ASAP
– Highest qualitysoftware using highest quality process -
massively reducingrisks
• Logical conclusion
• Paradigm shift

• “Even if you have good reasons for not releasing
every change you make—and there are less such
reasons than you might think—you should
behave as if you were going to do so”

Database Migration
DB
v10
App v200
compatible with
DB v10 & v11
App v210
compatible with
DB v11
App v220
compatible with
DB v11 & v12
Time
App v230
compatible with
DB v12
…
DB
v11
DB
v12
… …
• Decouple database changesfrom application deployment
name
firstname
surname
name
firstname
surname
firstname
surname
name
firstname
surname
firstname
surname
namename
Update
other
readers

Configuration Management
• Provisioning & management should be automated
• Keep everything you need to create/maintain infra
under version control
– E.g. Kickstart,Puppet, DNS zone files, DHCP, SMTP, firewall,
scripts
– Inputs to deployment pipeline (same as source code)

Deployment Toolchain
• Examples
– TeamCity -> Yum -> Fabric -> Puppet/Hiera
– Bamboo -> Yum -> Rundeck -> Chef
– Jenkins -> Nexus -> Ansible
– Jenkins -> Yum -> AMInator -> ASGard
CI
Artifact
Repo
Orchestra-
tion
Config
Mgmt

DevOps Toolchain
http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

Example DevOps Toolchain
http://www.slideshare.net/AnthonyShortland/dto-chefconf2012

Risk Management
Confidence↑Repeatability↑

Risk Management
http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Risk Management
• What about PCI/SOX/etc?
– Lock down who is able to access “privileged” envs
– Change management process for making changes
to privileged environments
– Approvalsfrom management before deployments
can be performed
– Protect against potential maliciousinterventions
– Audit all deployments
• Deployment pipeline makes it possible to
enforce these strategies while enabling
efficient delivery process
– Automation over Documentation
– Enforce Traceability

Risk Management
• Process for managing approvals
– Automated CR management system
– Adding access control to deployment pipeline
is a trivial exercise
• How does CAB decide whether change
should be executed?
– If risks outweigh benefits, change should not be made
• Principles:
– Keep metrics on the system & make visible:
• How long, how many waiting, proportion denied?
• MTBF/MTTR
• Cycle time
– Regular retrospectives& improve system based on feedback
http://www.vtexan.com/wp-content/uploads/2014/04/image7.png

CD Adoption
1. Understand: Why?
2. Get Measurable Change Fast
– (even if reaching goal takes years)
3. Start with Continuous Integration
4. Focus on Organization & Architecture
5. Culture of Continuous Improvement
Jez Humble “Adopting Continuous Delivery”: http://bit.ly/1FWOXNl

How to Address Architecture
• Refactor organization
– Leverage Conway’s law
– Amazon: Steve Yegge’s Platform Rant: bit.ly/shHRff
• Branch by abstraction: bit.ly/kAUbEw
• Strangler application: bit.ly/R4ZiJZ

CD Maturity Model
Figure 15.1 Maturity model

CD Maturity Model
https://flic.kr/p/afDTK9
Below this line, we’re slipping backwards
Above this line, we’re climbing the ladder of maturity

HP LaserJet Firmware
http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile

HP LaserJet Improvement
• Overall development costs reduced by ~40%
• Programs developed increased by ~140%
• Development costs per program down 78%
• Resources driving innovation increased by 8X

HP LaserJet Improvement
http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile

HP Deployment Pipeline
• 400 developers
• 10m LOC
• Individual Git branches
– CI build runs 2hrs then auto-merges for Stage 2 then
auto-merges to master
– (Branches are an imperfect countermeasure due to length
of build)
• L2 Simulators – acceptance tests
• L3 Emulators – not real paper
• L4 20,000hrs parallel tests
– Which change causes test problem?
– Move failing L4 tests up to L1
– Test that always pass at L1 move down

LaserJet Deployment Pipeline
http://www.slideshare.net/jezhumble/devops-culture-and-practice s-to-create-flow

In One Slide
Quality Cycle Time

Beyond the Book
• How to manage interconnected pipelines?
– Independence: Release one service at a time
– Be backwards compatible with current consumers
– Requires discipline:
• Short cycle time & small batch size
• Feature Switching
• System testing/monitoring
– E.g. synthetic transactions

Beyond the Book
• Consumer Driven Contracts
– Allows independent development and release of
dependent services
http://martinfowler.com/articles/consumerDrivenContracts.html
C
B
B
A
A
C
Code
Tests
B dev/tester
Maintains
Calls
As ‘C’ changes,
its tests for
consumers ‘A’ and
‘B’ are executed

Continuous Delivery Distilled

More Related Content

What's hot

Viewers also liked

Similar to Continuous Delivery Distilled

Recently uploaded

In this document

Continuous Delivery Distilled