Releasing Frequently
Reduces Risk
Owen Rogers
Twitter: @exortech
http://exortech.com/blog

1 year

48 releases

~ 1 release / week

10+/day

50+/day

continuous
deployment

crazy

sea change

competitive advantage

evolve software

resolve problems

faster than the
competition

capability

respond to change

Agile Manifesto
▪
Individuals and interactions over processes and tools
▪
Working software over comprehensive documentation
▪
Customer collaboration over contract negotiation
▪
Responding to change over following a plan

proposition

frequent releases
increase risk

frequent releases
increase risk
reduce

1) expose

2) mitigate

?

can you deploy
daily?

“we can’t do that...”
(3 minutes, 3 reasons)

3 common excuses

1. not enough time to test

1. not enough time to test
2. disruptive to users

1. not enough time to test
2. disruptive to users
3. too much overhead

1. not enough time to test
2. disruptive to users
3. too much overhead

1. not enough time to test

1. not enough time to test
Risk: cost of bugs getting into production

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time
• all bugs can be found in test

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time
• all bugs can be found in test

1. not enough time to test
Assume: we know what our customers want
testing: did we
build it right?

1. not enough time to test
Assume: we know what our customers want
risk: did we build
the right thing?

1. not enough time to test
Assume: we know what our customers want
$O

1. not enough time to test
Assume: we know what our customers want
$O
(well, negative $ actually)

1. not enough time to test
Assume: we know what our customers want

1. not enough time to test
Assume: we know what our customers want
• we understand our customer’s needs

1. not enough time to test
Assume: we know what our customers want
• we understand our customer’s needs
• our customers know what they want

1. not enough time to test
Assume: we know what our customers want
• we understand our customer’s needs
• our customers know what they want
• our customers will still want what we
have when we give to them

1. not enough time to test
Assume: we know what our customers want
validated learning
about customers

1. not enough time to test
Assume: we know what our customers want
simplest solution
to validate
hypothesis

1. not enough time to test
Assume: we know what our customers want
split test

1. not enough time to test
Assume: we know what our customers want
“deliver fast enough that the
customer doesn’t have time
to change their mind”

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time
• all bugs can be found in test

1. not enough time to test
Assumption: bugs are expensive
$ million bug

1. not enough time to test
Assumption: bugs are expensive
= $10 billion

1. not enough time to test
Assumption: bugs are expensive
= 120M users/day

1. not enough time to test
Assumption: bugs are expensive
bugs are inevitable

1. not enough time to test
Assumption: bugs are expensive
speed of response

1. not enough time to test
Assumption: bugs are expensive
continuous
monitoring

1. not enough time to test
Assumption: bugs are expensive

1. not enough time to test
Assumption: bugs are expensive

1. not enough time to test
Assumption: bugs are expensive

1. not enough time to test
Assumption: bugs are expensive

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time
• all bugs can be found in test

1. not enough time to test
Assumption: testing takes a long time
small changes

1. not enough time to test
Assumption: testing takes a long time
continuous
integration

1. not enough time to test
Assumption: testing takes a long time
continuous
testing

1. not enough time to test
Assumption: testing takes a long time
test automation

1. not enough time to test
Assumption: testing takes a long time
always releaseable

1. not enough time to test
Risk: cost of bugs getting into production
Assumptions:
• we know what our customers want
• bugs are expensive
• testing takes a long time
• all bugs can be found in test

1. not enough time to test
Assumption: all bugs can be found in test
= 40,000 photos/sec

1. not enough time to test
Assumption: all bugs can be found in test
= 6 Pb of storage

1. not enough time to test
Assumption: all bugs can be found in test

1. not enough time to test
2. disruptive to users
3. too much overhead

2. disruptive to users

2. disruptive to users
Risk: new releases will annoy users

2. disruptive to users
Risk: new releases will annoy users
Assumptions:

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime
• users will notice changes

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime
• users will notice changes
• changes are visible to all users

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime
• users will notice changes
• changes are visible to all users

2. disruptive to users
Assumption: releases incur downtime
patch releases?

2. disruptive to users
Assumption: releases incur downtime
good will

2. disruptive to users
Assumption: releases incur downtime
zero-downtime
deployment

2. disruptive to users
Assumption: releases incur downtime
redundancy

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime
• users will notice changes
• changes are visible to all users

2. disruptive to users
Assumption: users will notice changes

2. disruptive to users
Assumption: users will notice changes

2. disruptive to users
Assumption: users will notice changes
version?

2. disruptive to users
Risk: new releases will annoy users
Assumptions:
• releases incur downtime
• users will notice changes
• changes are visible to all users

2. disruptive to users
Assumption: changes are visible to all
users
big-bang release

2. disruptive to users
Assumption: changes are visible to all
users
options

2. disruptive to users
Assumption: changes are visible to all
users

2. disruptive to users
Assumption: changes are visible to all
users
Options:

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)
• rollout to some servers

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)
• rollout to some servers
• downgrade feature

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)
• rollout to some servers
• downgrade feature
• disable feature (feature darkmode)

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)
• rollout to some servers
• downgrade feature
• disable feature (feature darkmode)
• defer commit

2. disruptive to users
Assumption: changes are visible to all
users
Options:
• release to user subset (private beta)
• rollout to some servers
• downgrade feature
• disable feature (feature darkmode)
• defer commit
• defer release

2. disruptive to users
Assumption: changes are visible to all
users
options = $$$

2. disruptive to users
Assumption: changes are visible to all
users
“release is a
marketing decision”

2. disruptive to users
Assumption: changes are visible to all
users
bonus:

1. not enough time to test
2. disruptive to users
3. too much overhead

3. too much overhead

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead
• releases are risky

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead
• releases are risky
• deployment takes a long time

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead
• releases are risky
• deployment takes a long time

3. too much overhead
Assumption: high coordination overhead
functional silos

3. too much overhead
Assumption: high coordination overhead
dev vs. ops

3. too much overhead
Assumption: high coordination overhead
devs don’t know
prod

3. too much overhead
Assumption: high coordination overhead
ops don’t know
code

3. too much overhead
Assumption: high coordination overhead
shared
accountability

3. too much overhead
Assumption: high coordination overhead

3. too much overhead
Assumption: high coordination overhead
shared
accountability

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead
• releases are risky
• deployment takes a long time

3. too much overhead
Assumption: releases are risky
“if it hurts, do it
more often”

3. too much overhead
Assumption: releases are risky
incremental
change

3. too much overhead
Assumption: releases are risky
more or less same
system

3. too much overhead
Assumption: releases are risky
practice makes
perfect

3. too much overhead
Risk: cost of a release is greater than the
benefit of its changes
Assumptions:
• high coordination overhead
• releases are risky
• deployment takes a long time

3. too much overhead
Assumption: deployment takes a long
time
risk: manual steps

3. too much overhead
Assumption: deployment takes a long
time
automated
deployment

3. too much overhead
Assumption: deployment takes a long
time
one-step process

Summary

Frequent Releases?
?

concerns

1. not enough time to test
2. disruptive to users
3. too much overhead

risks

1. bugs getting into production
2. new releases will annoy users
3. cost of a release is greater
than the benefit of its changes

localized risk

1) expose

underlying risks
Risks:
• do we know what customers want?
• can we respond fast enough to
problems?
• can we detect problems when they
happen?
• can we limit the impact of changes?
• can we deploy without downtime?
• can we build production-ready
software?

2) mitigate

mitigation strategies
Strategies:
• validated learning about customers
• split testing
• continuous monitoring
• continuous integration
• test automation
• zero-down time deployment
• deployment options
• operation levers
• automated deployment

Thank You!

Shameless Plug

Nov 2 - 5
• Eric Evans
• Michael Feathers
• Martin Fowler
• Jeff Patton
• Mary Poppendieck
• Michael Nygard
• Linda Rising
• Johanna Rothmann

Releasing Frequently
Reduces Risk
Owen Rogers
Twitter: @exortech
http://exortech.com/blog

1. not enough time to test
2. disruptive to users
3. too much overhead

1. bugs getting into production
2. new releases will annoy users
3. cost of a release is greater
than the benefit of its changes