2. Introduction
• Releasing software into production and deploying to testing
environment
• Can be done at the same deployment process but
configuration files and ability of roll back are different
• How to create and follow a deployment strategy
• Blue-green deployments and canary releasing are introduced
for zero-downtime releases and rollbacks
• Ae a part of deployment pipeline implementation
• which version in which environment, who authorized the
deployment and what changes have be made to application
• Emergency fixes, upgrade plan...
3. Create a release strategy
• How?
• All stakeholders meet up > get common understanding
concerning the deployment and operations > create the
release strategy > update and maintain
• Followings should be considered at the first version of
release strategy at the beginning of project
• Parties in charge of deployments to each environment, as
well as in charge of the release
• An asset and configuration management strategy
• A description of the technology used for deployment. This
should be agreed upon by both the operations and
development teams
• A plan for implementing the deployment pipeline
4. Create a release strategy
• An enumeration of the environments available for
acceptance, capacity, integration, and user acceptance testing,
and the process by which builds will be moved through these
environments
• A description of the processes to be followed for deployment
into testing and production environments, such as change
requests to be opened and approvals that need to be granted.
• Requirements for monitoring the application, including any
APIs or services the application should use to notify the
operations team of its state.
• A discussion of the method by which the application’s deploy-
time and runtime configuration will be managed, and how this
relates to the automated deployment process.
5. Create a release strategy
• Description of the integration with any external systems. At
what stage and how are they tested as part of a release?
How do the operations personnel communicate with the
provider in the event of a problem?
• Details of logging so that operations personnel can
determine the application’s state and identify any error
conditions.
• A disaster recovery plan so that the application’s state can be
recovered following a disaster.
• The service-level agreements for the software, which will
determine whether the application will require techniques
like failover and other high-availability strategies.
6. Create a release strategy
• Production sizing and capacity planning: How much data will
your live application create? How many log files or databases
will you need? How much bandwidth and disk space will you
need? What latency are clients expecting?
• An archiving strategy so that production data that is no
longer needed can be kept for auditing or support purposes.
• How the initial deployment to production works.
• How fixing defects and applying patches to the production
environment will be handled.
• How upgrades to the production environment will be
handled, including data migration.
7. Create a release strategy
• Why?
• Be a source of both functional and nonfunctional
requirements for both software development and for
design, configuration, and commissioning of hardware
environment
• Others
• the release strategy should be added and changed during
the project progresses
• vital component is the release plan describing how releases
are performed
8. The release plan
• Release plan: automated scripts, documents or other
procedures for reliable and repeatable deployment:
• Comparing to deployment strategy
• The steps required to deploy the application for the first
time
• How to smoke-test the application and any services it uses
as part of the deployment process
• The steps required to back out the deployment should it
go wrong
• The steps required to back up and restore the application’s
state application’s state
9. The release plan
• The steps required to upgrade the application without
destroying the application’s state
• The steps to restart or redeploy the application should it
fail
• The location of the logs and a description of the
information they contain
• The methods of monitoring the application
• The steps to perform any data migrations that are
necessary as part of the release
• An issue log of problems from previous deployments, and
their solutions
10. Releasing products
• More should be considered for software product
• Pricing model
• Licensing strategy
• Copyright issues around third-party technologies used
• Packaging
• Marketing materials-print, web-based, podcasts, blogs, press
releases, conferences
• Production documentation
• Installers
• Preparing sales and support teams
11. The first deployment
• The first deployment should be in the first iteration
• Principal goals of the first iteration is to get the early stages
of our deployment pipeline running and be able to deploy and
demonstrate somethings
• deployment pipeline’s commit stage
• production-like environment to deploy to
• An automated process that takes the binaries created by
your commit stage and deploys them into the environment
• A simple smoke test that verifies that the deployment
worked and the application is running
12. Production like environment
• Development and production environment
• Try to build the same or similar environment, use
virtualization and chicken-counting
• chicken-counting: if 250 web servers, 2 should be enough to
represent the significant process boundaries
• Production like environment
• same operating system
• same software installed (none development toolchain like IDE)
• be managed the same way as production environment
• for client-installed software: UAT environment should be
representative of clients’ hardware statistics
13. Modeling your release process and promoting builds
• With the grow of application, deployment pipeline
implementation becomes complex and test and release process
should be modeled considering the following:
• What stage a build has to go through in order to be released
• What the required gates or approval are
• For each gate, who has the authority to approve a build
passing through that gate
14. ur release process. We discussed value stream mapping as a way to
Example test and release process diagram
ur release process in Chapter 5, “Anatomy of the Deployment
Integration integration
Acceptance QA sign-off
testing sign-off
testing gate testing gate
User customer Operations
acceptance sign-off Staging sign off-gate Production
testing gate
Figure 10.1 An example test and release process diagram
’ve created this diagram, you can create placeholders for each part
ase process in the tool you use for managing deployments. Go and
15. Managing the test and release process
• Model and manage deployment process
• Create placeholder for each part
• Used for approval of people responsible for approvals
• Manage information about which builds have passed all previous
stages and whether are ready for next stage.
• Choose one build and press a deploy button (promotion)
• Get everybody involved in the delivery process to manage their
own works after promotion, pull system
16. Tools
• GO
• Agile Release management
• http://www.thoughtworks-studios.com/go-agile-release-
management
• AntHill Pro (DevOps Platform)
• http://www.urbancode.com/html/default.html
• Other Agile management tools
• IceScrum: http://www.icescrum.org/
• Jira: http://www.atlassian.com/ja/software/jira/overview
17. Test and deploy workflow
• Can select which version to deploy to test environment
• Prepare environment and associated infrastructure
• Deploy the application’s binaries.
• Configure the application
• Prepare or migrate any data managed by the application
• Smoke-test the deployment
• Perform the testing
• Approve promotion if test passed. if not, record why
18. Promoting configuration
• Configuration of environment and application should be promoted
• Managing promotion of configuration associated with an
application is complex
• Don’t want to promote all the configuration after test at SIT
database or a test double of external service
• One way is use smoke test to verify that you are pointing at the
right things
• In case of service-oriented architectures and componentized
applications, all the services and components should be promoted
together
19. Orchestration
• Environment shared between several applications
• Extra care trying not to disturb the operation of any other
applications in the environment
• If applications depend on each other, integration testing
environment is ensure the new versions of each application
cooperate with each other well
20. Deployments to staging environments
• Ensure your production, capacity testing, and staging environments
are commissioned. In particular, on a green field project, have your
production environment ready some time before the release, and
deploy to it as part of your pipeline.
• Have an automated process for configuring your environment,
including networks, external services, and infrastructure.
• Ensure the deployment process is adequately smoke-tested
• Measure the warm-up period for your application. This is especially
applicable if your application uses caching. Incorporate this into
your deployment plan.
• Test integration with external systems.You don’t want your
application’s release to be the first time you run against the real
external systems
21. Deployments to staging environments
• If possible, get your application into its production environment
well before release. If “release” can be as simple as reconfiguring
some router to direct traffic from a holding page to your
production environment, so much the better. This technique,
known as blue-green deployment, is described a little later in this
chapter.
• If possible, try rolling your system out to a small group of users
before you roll it out to everybody. This technique is known as
canary releasing, and is also described later in this chapter
• Deploy every change that passes acceptance tests to your staging
environment (although not necessarily to production).
22. Rollback by redeploying previous good version
• Good:
• low risk if you don’t have automated rollback but deploy
process
• deployment process is much more well tested than rollback
• Demerit
• Has downtime as time of redeploy the old version is nonzero
• Make it hared to debug what went wrong
• May lose data created since last deployment
23. Zero-Downtime Releases
• Zero-downtime release, hot deployment
• Key to zero-downtime release is decoupling the various parts of
release so they can happen independently as far as possible
• It is easy for static resources and web-based services, but it is
harder for resources like DB
24. This is one of the most powerful techniques we know for managing releases. The
Blue-Green Deployments
idea is to have two identical versions of your production environment, which
we’ll call blue and green.
Web server Application server Database server
Green slice Green slice Green database
Users Router
Blue slice Blue slice Blue database
Figure 10.2 Blue-green deployments
In the example in Figure 10.2, users of the system are routed to the green envi-
•
ronment,Deploy to blue environment, let it warm up -> run to release a new
which is the currently designated production. We want smoke tests to
version of the application. So we deploy it toversion by changing the router
check it works -> move the new the blue environment, and let the
configuration
application warm up (you can do this as much as you like). This does not in any
way affect the operation of the green environment. We can run smoke tests against
•
the blue environment to goes wrong, change router When to green
If somethings check it is working properly. back we’re ready, moving
environment
to the new version is as simple as changing the router configuration to point to
the blue environment instead of the green environment. The blue environment
thus becomes production. This switchover can typically be performed in much
less than a second.
25. Database switch at Blue-Green deployment
• Data migration may be needed if schema is changed
• One: Put application into read-only mode before switchover,
copy green db, restore into blue db, perform migration, switch,
change to read-write mode
• Two: design the application so that you can migrate the db
independently of the upgrade process
26. Blue-Green deployment and budget
• Low budget:
• Have two copies of application runs side by side on the same
environment
• use virtualization
• Sufficient budget
• Blue and green environment can be completely separate replicas
• Using staging and production environment as blue and green
environment
27. Canary releasing
• Why canary releasing?
• Only have one version of software in production at a time
makes easier to manage bug fixes and indeed infrastructure in
general
• However, it presents and impediment to test as defects pop up
in production
• To create a meaningful capacity testing environment is
impossible if we have extremely large production environment
28. jority of users. This is a great way to reduce the risk of releasing a new version.
Canary releasing
Users
Router
Most users Small set of users
Version x Version x+1
Figure 10.3 Canary releasing
29. Canary releasing
• Benefits:
• Make rolling back easy
• A/B testing can be done by routing some users to new version
and some to the old
• You can check if the application meets capacity requirement
gradually
• Others
• not only way for A/B test; runtime configuration setting to
change behavior
• Hard to use for user install soft on own computer
• Impose further constraints on DB upgrades
• Limit few versions of application in production
30. Emergency fixes
• Do not, under any circumstances, subvert your process, even when
critical defect is discovered and has to be fixed
• Have to go through the same build, deploy, test and release process
as any other change. In other works, run every emergency fix
through standard deployment pipeline
• If not,
• The change will lead to regression and exacerbate the problem
• Change is often not recorded and environment ends up in an
unknown state
• Check the emergency fix is wether or not is needed, roll back is
another choice
31. Emergency fixes
• Considerations about dealing with defect in production
• Never do them late at night, and always pair with somebody
else.
• Make sure you have tested your emergency fix process.
• Only under extreme circumstances circumvent the usual
process for making changes to your application.
• Make sure you have tested making an emergency fix using your
staging environment.
• Sometimes it’s better to roll back to the previous version than
to deploy a fix. Do some analysis to work out what the best
solution is. Consider what happens if you lose data or face
integration or orchestration problems.
32. Continuous deployment
• If it hurts, do it more often => Take the pipeline and make
deployment to production automatic
• automated unit tests, component tests, acceptance tests
covering entire application
• Continuous deployment can be combined with canary release
• Continuous deployment isn’t for everyone, as some new features
are not wanted to be released to production
• By release every change, amount of risk is limited to the risk
inherent in one change
• Continuous deployment force we to do the right thing: build,
deploy, test and release process
33. Continuously releasing user-installed software
• Issues to consider for client-installed software
• Managing the upgrade experience
• Migrating binaries, data, and configuration
• Testing the upgrade process
• Getting crash reports from users
34. Upgrade plan
• Support different versions of software is difficult, so we need to
make upgrade as painless as possible
• Upgrade plan
1. Have your software check for new versions and prompt the user
to download and upgrade to the latest version
2. Download in the background and prompt for installation
3. Download in the background and silently upgrade the next time
the application is restarted
• 1,2 seems attractive but wrong as users can choose not to upgrade
35. Upgrade plan
• Correct solution:
• Make upgrade process bullet proof and upgrade silently
• Prompt to user if corrective action needs to be taken
• Sometimes, you don’y want your software upgrade silently
because of poor network and so on...
• User can turn off automatic upgrades
• Migrating binaries, data and configuration: keep old ones until
upgrade successful
• Be able to upgrade from any version to any other version
• Test upgrade process as part of deployment pipeline
• Be able to report crashes back to development team
36. Tips and tricks
• People who do the deployment should be involved in creating
the deployment process
• Log deployment activities: log all files for debug
• Don't delete the old files, move them
• Deployment is the whole team's responsibility: any member
should do
• Server application should not have GUIs: the machine must have
had a user logged in and the UI showing. Reboot, accidental or
due to upgrades will log the user out, and the server will stop.
• Have a warm-up period for a new deployment (canary release)
• Fail fast
• Don't make changes directly on the production environment
