This document discusses several anti-patterns related to manual software deployments including extensive documentation, reliance on manual testing, unpredictable releases, and lack of collaboration between development and operations teams. It advocates for automating deployments to make them repeatable and frequent in order to reduce risk and provide quick feedback. Continuous delivery of software through practices like blue-green deployments and canary releases is recommended to satisfy customers.

1. Continuous Delivery
@masasdani

2. Problem

3. Anti Pattern
Deploying Software Manually
• The production of extensive, detailed documentation that describes the steps to be taken and
the ways in which the steps may go wrong
• Reliance on manual testing to confirm that the application is running correctly
• Frequent calls to the development team to explain why a deployment is going wrong on a
release day
• Frequent corrections to the release process during the course of a release
• Environments in a cluster that differ in their configuration, for example application servers
with different connection pool settings, filesystems with different layouts, etc.
• Releases that take more than a few minutes to perform
• Releases that are unpredictable in their outcome, that often have to be rolled back or run into
unforeseen problems
• Sitting bleary-eyed in front of a monitor at 2 A.M. the day after the release day, trying to
figure out how to make it work

4. Anti Pattern
Deploying to a Production-like Environment Only after Development Is
Complete
• If testers have been involved in the process up to this point, they have tested the system on
development machines.
• Releasing into staging is the first time that operations people interact with the new release.
In some organizations, separate operations teams are used to deploy the software into
staging and production. In this case, the first time an operations person sees the software is
the day it is released into production.
• Either a production-like environment is expensive enough that access to it is strictly
controlled, or it is not in place on time, or nobody bothered to create one.
• The development team assembles the correct installers, configuration files, database
migrations, and deployment documentation to pass to the people who perform the actual
deployment—all of it untested in an environment that looks like production or staging.
• There is little, if any, collaboration between the development team and the people who
actually perform deployments to create this collateral.

5. Anti Pattern
Manual Configuration Management of Production Environments
• Having deployed successfully many times to staging, the deployment into production fails.
• Different members of a cluster behave differently—for example, one node sustaining less
load or taking longer to process requests than another.
• The operations team take a long time to prepare an environment for a release.
• You cannot step back to an earlier configuration of your system, which may include
operating system, application server, web server, RDBMS, or other infrastructural settings.
• Servers in clusters have, unintentionally, different versions of operating systems, third-party
infrastructure, libraries, or patch levels.
• Configuration of the system is carried out by modifying the configuration directly on
production systems.

6. Manual deployments depend on the deployment expert. If he or she is on
vacation or quits work, you are in trouble.
Performing manual deployments is boring and repetitive
With a manual process, there is no guarantee that the documentation has been
followed

7. How Do We Achieve Our Goal?
• Automated.
If the build, deploy, test, and release process is not automated, it is not repeatable.
• Frequent.
If releases are frequent, the delta between releases will be small.
Every Change Should Trigger the Feedback Process
The Feedback Must Be Received as Soon as Possible
The Delivery Team Must Receive Feedback and Then Act on It

9. agile 101
Iteration 0 1 2 3 4
Analysis + Design
Development
Testing + Showcase
Integration + QA Release and operation
Customer
Centralized QA IT Operations
"Agile" team
The "last mile"

10. web 2.0
disrupting traditional businesses
http://code.flickr.com/

11. releasing frequently
1. build the right thing
Eric Ries, “The Lean Startup” http://bit.ly/8ZoX5F
Customer
development
Agile product
development

12. scientific method
create hypothesis
deliver minimum
viable product
get feedback
(repeat)
Eric Ries, The Lean Startup
Ideas
CodeData
Build
Measure
Learn

13. ask this question
“How long would it take your
organization to deploy a change
that involved just one single line
of code? Do you do this on a
repeatable, reliable basis?”
Mary and Tom Poppendieck, Implementing Lean Software Development, p59.

14. releasing frequently
1. build the right thing
2. reduce risk of release
John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

15. releasing frequently
1. build the right thing
2. reduce risk of release
3. real project progress

16. agile manifesto
Our highest priority is to satisfy
the customer through early
and continuous delivery of
valuable software

17. Fast, automated feedback on
the production readiness of
your applications every time
there is a change - to code,
infrastructure, or configuration
production-ready software

18. Software always production ready
Releases tied to business needs, not
operational constraints
Customer
Delivery team
Constant flow of new features into production
continuous delivery

19. automation
patterns and practices
collaboration
continuous delivery

20. configuration management
continuous integration
automated testing
ingredients

21. Mainline Server
Develop
Build
Build
pull
Local
Workstation
Build
push
✔
Done!

22. Mainline Server
Develop
Build
Build
pull
Local
Workstation
Build
push
✔
Done!
Everyone Commits
To the Mainline
Every Day

23. build quality in
“Cease dependence on
mass inspection to achieve
quality. Improve the
process and build quality
into the product in the first
place”
W. Edwards Deming

24. Functional acceptance
tests
Showcases
Usability testing
Exploratory testing
Unit tests
Integration tests
System tests
Non-functional
acceptance tests
(performance, scaling, ...)
Business facing
Technology facing
Critiqueproject
Supportprogramming
AUTOMATED
AUTOMATED
MANUAL
MANUAL / AUTOMATED
Diagram invented by Brian Marick
different kinds of testing

25. visibility
an automated implementation of your
system’s build, deploy, test, release process
control
feedback
deployment pipeline

26. deployment pipeline
Delivery team Version control Build & unit
tests
Automated
acceptance tests
User acceptance
tests
Release
Check in
Feedback
Trigger
Check in
Feedback
Trigger
Trigger
Check in
Trigger
Trigger
Approval
Approval
Feedback
Feedback
Feedback
Feedback

27. deployment pipeline

28. deployment pipeline

29. incrementalism
devops
decoupling deployment and release
reducing release risk

30. culture
automation
measurement
sharing
devops

31. low risk releases
are incremental
blue-green deployments
canary releases
dark launching
production immune system
feature toggles

32. STATIC CONTENT
/static/1.1
/static/1.0
DEPENDENT SERVICE
1.0 1.1
Abstraction layer Abstraction layer
APPLICATION
Database
Router /
Load balancer
Interwebs

33. blue-green
deployments

34. router
web
server
app
server
DB
server

35. router
v1.1 v1.1 v1.1
web
server
app
server
DB
server

36. router
v1.1 v1.1 v1.1
web
server
app
server
DB
server
v1.2 v1.2 v1.2

37. router
v1.1 v1.1 v1.1
web
server
app
server
DB
server
v1.2 v1.2 v1.2

38. release !=
deployment

39. segregation of duties
risk management
SOX, ITIL, COBIT
auditing and compliance
change management
enterprise governance

40. big, visible displays
inceptions, showcases, retrospectives
rotate people, share the pain
continuous improvement (kaizen)
people are the key

41. innovate
You can't just ask
customers what
they want and then
try to give that to
them.
By the time you get
it built, they'll want
something new.
Steve Jobs

42. Thanks