Deploying and Releasing Applications Continuous Delivery, Chap. 10 Xuebin MA
Introduction• Releasing software into production and deploying to testing environment • Can be done at the same deployment process but conﬁguration ﬁles and ability of roll back are different• How to create and follow a deployment strategy• Blue-green deployments and canary releasing are introduced for zero-downtime releases and rollbacks• Ae a part of deployment pipeline implementation • which version in which environment, who authorized the deployment and what changes have be made to application• Emergency ﬁxes, upgrade plan...
Create a release strategy• How? • All stakeholders meet up > get common understanding concerning the deployment and operations > create the release strategy > update and maintain• Followings should be considered at the ﬁrst version of release strategy at the beginning of project • Parties in charge of deployments to each environment, as well as in charge of the release • An asset and conﬁguration management strategy • A description of the technology used for deployment. This should be agreed upon by both the operations and development teams • A plan for implementing the deployment pipeline
Create a release strategy• An enumeration of the environments available for acceptance, capacity, integration, and user acceptance testing, and the process by which builds will be moved through these environments• A description of the processes to be followed for deployment into testing and production environments, such as change requests to be opened and approvals that need to be granted.• Requirements for monitoring the application, including any APIs or services the application should use to notify the operations team of its state.• A discussion of the method by which the application’s deploy- time and runtime conﬁguration will be managed, and how this relates to the automated deployment process.
Create a release strategy• Description of the integration with any external systems. At what stage and how are they tested as part of a release? How do the operations personnel communicate with the provider in the event of a problem?• Details of logging so that operations personnel can determine the application’s state and identify any error conditions.• A disaster recovery plan so that the application’s state can be recovered following a disaster.• The service-level agreements for the software, which will determine whether the application will require techniques like failover and other high-availability strategies.
Create a release strategy• Production sizing and capacity planning: How much data will your live application create? How many log ﬁles or databases will you need? How much bandwidth and disk space will you need? What latency are clients expecting?• An archiving strategy so that production data that is no longer needed can be kept for auditing or support purposes.• How the initial deployment to production works.• How ﬁxing defects and applying patches to the production environment will be handled.• How upgrades to the production environment will be handled, including data migration.
Create a release strategy• Why? • Be a source of both functional and nonfunctional requirements for both software development and for design, conﬁguration, and commissioning of hardware environment• Others • the release strategy should be added and changed during the project progresses • vital component is the release plan describing how releases are performed
The release plan• Release plan: automated scripts, documents or other procedures for reliable and repeatable deployment:• Comparing to deployment strategy • The steps required to deploy the application for the ﬁrst time • How to smoke-test the application and any services it uses as part of the deployment process • The steps required to back out the deployment should it go wrong • The steps required to back up and restore the application’s state application’s state
The release plan• The steps required to upgrade the application without destroying the application’s state• The steps to restart or redeploy the application should it fail• The location of the logs and a description of the information they contain• The methods of monitoring the application• The steps to perform any data migrations that are necessary as part of the release• An issue log of problems from previous deployments, and their solutions
Releasing products• More should be considered for software product • Pricing model • Licensing strategy • Copyright issues around third-party technologies used • Packaging • Marketing materials-print, web-based, podcasts, blogs, press releases, conferences • Production documentation • Installers • Preparing sales and support teams
The ﬁrst deployment• The ﬁrst deployment should be in the ﬁrst iteration• Principal goals of the ﬁrst iteration is to get the early stages of our deployment pipeline running and be able to deploy and demonstrate somethings • deployment pipeline’s commit stage • production-like environment to deploy to • An automated process that takes the binaries created by your commit stage and deploys them into the environment • A simple smoke test that veriﬁes that the deployment worked and the application is running
Production like environment• Development and production environment • Try to build the same or similar environment, use virtualization and chicken-counting • chicken-counting: if 250 web servers, 2 should be enough to represent the signiﬁcant process boundaries• Production like environment • same operating system • same software installed (none development toolchain like IDE) • be managed the same way as production environment • for client-installed software: UAT environment should be representative of clients’ hardware statistics
Modeling your release process and promoting builds• With the grow of application, deployment pipeline implementation becomes complex and test and release process should be modeled considering the following: • What stage a build has to go through in order to be released • What the required gates or approval are • For each gate, who has the authority to approve a build passing through that gate
ur release process. We discussed value stream mapping as a way to Example test and release process diagram ur release process in Chapter 5, “Anatomy of the Deployment Integration integration Acceptance QA sign-off testing sign-off testing gate testing gate User customer Operations acceptance sign-off Staging sign off-gate Production testing gate Figure 10.1 An example test and release process diagram’ve created this diagram, you can create placeholders for each partase process in the tool you use for managing deployments. Go and
Managing the test and release process• Model and manage deployment process• Create placeholder for each part• Used for approval of people responsible for approvals• Manage information about which builds have passed all previous stages and whether are ready for next stage. • Choose one build and press a deploy button (promotion)• Get everybody involved in the delivery process to manage their own works after promotion, pull system
Tools• GO • Agile Release management • http://www.thoughtworks-studios.com/go-agile-release- management• AntHill Pro (DevOps Platform) • http://www.urbancode.com/html/default.html• Other Agile management tools • IceScrum: http://www.icescrum.org/ • Jira: http://www.atlassian.com/ja/software/jira/overview
Test and deploy workﬂow• Can select which version to deploy to test environment• Prepare environment and associated infrastructure• Deploy the application’s binaries.• Conﬁgure the application• Prepare or migrate any data managed by the application• Smoke-test the deployment• Perform the testing• Approve promotion if test passed. if not, record why
Promoting conﬁguration• Conﬁguration of environment and application should be promoted• Managing promotion of conﬁguration associated with an application is complex • Don’t want to promote all the conﬁguration after test at SIT database or a test double of external service• One way is use smoke test to verify that you are pointing at the right things• In case of service-oriented architectures and componentized applications, all the services and components should be promoted together
Orchestration• Environment shared between several applications • Extra care trying not to disturb the operation of any other applications in the environment • If applications depend on each other, integration testing environment is ensure the new versions of each application cooperate with each other well
Deployments to staging environments• Ensure your production, capacity testing, and staging environments are commissioned. In particular, on a green ﬁeld project, have your production environment ready some time before the release, and deploy to it as part of your pipeline.• Have an automated process for conﬁguring your environment, including networks, external services, and infrastructure.• Ensure the deployment process is adequately smoke-tested• Measure the warm-up period for your application. This is especially applicable if your application uses caching. Incorporate this into your deployment plan.• Test integration with external systems.You don’t want your application’s release to be the ﬁrst time you run against the real external systems
Deployments to staging environments• If possible, get your application into its production environment well before release. If “release” can be as simple as reconﬁguring some router to direct trafﬁc from a holding page to your production environment, so much the better. This technique, known as blue-green deployment, is described a little later in this chapter.• If possible, try rolling your system out to a small group of users before you roll it out to everybody. This technique is known as canary releasing, and is also described later in this chapter• Deploy every change that passes acceptance tests to your staging environment (although not necessarily to production).
Rollback by redeploying previous good version• Good: • low risk if you don’t have automated rollback but deploy process • deployment process is much more well tested than rollback• Demerit • Has downtime as time of redeploy the old version is nonzero • Make it hared to debug what went wrong • May lose data created since last deployment
Zero-Downtime Releases• Zero-downtime release, hot deployment• Key to zero-downtime release is decoupling the various parts of release so they can happen independently as far as possible• It is easy for static resources and web-based services, but it is harder for resources like DB
This is one of the most powerful techniques we know for managing releases. The Blue-Green Deploymentsidea is to have two identical versions of your production environment, whichwe’ll call blue and green. Web server Application server Database server Green slice Green slice Green database Users Router Blue slice Blue slice Blue database Figure 10.2 Blue-green deployments In the example in Figure 10.2, users of the system are routed to the green envi- •ronment,Deploy to blue environment, let it warm up -> run to release a new which is the currently designated production. We want smoke tests toversion of the application. So we deploy it toversion by changing the router check it works -> move the new the blue environment, and let the conﬁgurationapplication warm up (you can do this as much as you like). This does not in anyway affect the operation of the green environment. We can run smoke tests against •the blue environment to goes wrong, change router When to green If somethings check it is working properly. back we’re ready, moving environmentto the new version is as simple as changing the router conﬁguration to point tothe blue environment instead of the green environment. The blue environmentthus becomes production. This switchover can typically be performed in muchless than a second.
Database switch at Blue-Green deployment• Data migration may be needed if schema is changed • One: Put application into read-only mode before switchover, copy green db, restore into blue db, perform migration, switch, change to read-write mode • Two: design the application so that you can migrate the db independently of the upgrade process
Blue-Green deployment and budget• Low budget: • Have two copies of application runs side by side on the same environment • use virtualization• Sufﬁcient budget • Blue and green environment can be completely separate replicas • Using staging and production environment as blue and green environment
Canary releasing• Why canary releasing? • Only have one version of software in production at a time makes easier to manage bug ﬁxes and indeed infrastructure in general • However, it presents and impediment to test as defects pop up in production • To create a meaningful capacity testing environment is impossible if we have extremely large production environment
jority of users. This is a great way to reduce the risk of releasing a new version. Canary releasing Users Router Most users Small set of users Version x Version x+1 Figure 10.3 Canary releasing
Canary releasing• Beneﬁts: • Make rolling back easy • A/B testing can be done by routing some users to new version and some to the old • You can check if the application meets capacity requirement gradually• Others • not only way for A/B test; runtime conﬁguration setting to change behavior • Hard to use for user install soft on own computer • Impose further constraints on DB upgrades • Limit few versions of application in production
Emergency ﬁxes• Do not, under any circumstances, subvert your process, even when critical defect is discovered and has to be ﬁxed• Have to go through the same build, deploy, test and release process as any other change. In other works, run every emergency ﬁx through standard deployment pipeline• If not, • The change will lead to regression and exacerbate the problem • Change is often not recorded and environment ends up in an unknown state• Check the emergency ﬁx is wether or not is needed, roll back is another choice
Emergency ﬁxes• Considerations about dealing with defect in production • Never do them late at night, and always pair with somebody else. • Make sure you have tested your emergency ﬁx process. • Only under extreme circumstances circumvent the usual process for making changes to your application. • Make sure you have tested making an emergency ﬁx using your staging environment. • Sometimes it’s better to roll back to the previous version than to deploy a ﬁx. Do some analysis to work out what the best solution is. Consider what happens if you lose data or face integration or orchestration problems.
Continuous deployment• If it hurts, do it more often => Take the pipeline and make deployment to production automatic • automated unit tests, component tests, acceptance tests covering entire application• Continuous deployment can be combined with canary release• Continuous deployment isn’t for everyone, as some new features are not wanted to be released to production• By release every change, amount of risk is limited to the risk inherent in one change• Continuous deployment force we to do the right thing: build, deploy, test and release process
Continuously releasing user-installed software• Issues to consider for client-installed software • Managing the upgrade experience • Migrating binaries, data, and conﬁguration • Testing the upgrade process • Getting crash reports from users
Upgrade plan• Support different versions of software is difﬁcult, so we need to make upgrade as painless as possible• Upgrade plan 1. Have your software check for new versions and prompt the user to download and upgrade to the latest version 2. Download in the background and prompt for installation 3. Download in the background and silently upgrade the next time the application is restarted• 1,2 seems attractive but wrong as users can choose not to upgrade
Upgrade plan• Correct solution: • Make upgrade process bullet proof and upgrade silently • Prompt to user if corrective action needs to be taken• Sometimes, you don’y want your software upgrade silently because of poor network and so on... • User can turn off automatic upgrades• Migrating binaries, data and conﬁguration: keep old ones until upgrade successful• Be able to upgrade from any version to any other version• Test upgrade process as part of deployment pipeline• Be able to report crashes back to development team
Tips and tricks• People who do the deployment should be involved in creating the deployment process• Log deployment activities: log all ﬁles for debug• Dont delete the old ﬁles, move them• Deployment is the whole teams responsibility: any member should do• Server application should not have GUIs: the machine must have had a user logged in and the UI showing. Reboot, accidental or due to upgrades will log the user out, and the server will stop.• Have a warm-up period for a new deployment (canary release)• Fail fast• Dont make changes directly on the production environment