This document discusses deployment pipelines and best practices for continuous delivery. It covers topics like the basic components of a deployment pipeline including different stages like commit, testing, and release. It also discusses practices for deployment pipeline like deploying the same way to every environment, automating deployments, and making the deployment process idempotent. Scripting tools for automating deployments and metrics for monitoring pipelines are also covered at a high level.

1. RLMCA202
Continuous Delivery
Module3
Dr. Sudheer Sankara Marar MCA, MBA, MTech, MA.JMC, PhD
HOD-MCA,
Nehru College of Engineering and Research Centre

2. Introduction to Deployment Pipeline
• A deployment pipeline is an automated manifestation
of your process for getting software from version
control into the hands of your users.
• Every change to your software goes through a complex
process on its way. That process involves building the
software, followed by the progress of these builds
through multiple stages of testing and deployment.
• This, requires collaboration between many individuals,
and perhaps several teams.
• The deployment pipeline models this process.

3. Deployment Pipeline
• The process modeled by the deployment pipeline, the
process of getting software from check-in to release,
forms a part of the process of getting a feature from
the mind of a customer into their hands.
• The entire process—from concept to cash—can be
modeled as a value stream map as below

4. Creating a value stream map
• With a pencil and pad in hand, go to the place where a
customer request comes into your organization.
• You goal is to draw a chart of the average customer
request, from arrival to completion. Working with the
people involved in each activity, you sketch all the
process steps necessary to fill the request, as well as
the average amount of time that a request spends in
each step.
• At the bottom of the map, draw a timeline that shows
how much time the request spends in value-adding
activities and how much in waiting states and non-
value-adding activities.

5. Sequence Diagram visualization

6. • As the build passes each test of its fitness, confidence in it
increases.
• Eliminate unfit release candidates as early in the process
• Any build that fails a stage in the process will not generally be
promoted to the next. These trade-offs are shown below

7. A Basic Deployment Pipeline

8. A Basic Deployment Pipeline
• Developers committ changes into their version control
system.
• CI management system responds to the commit by
triggering a new instance of pipeline.
• The first stage of the pipeline compiles the code, runs unit
tests, performs code analysis, and creates installers.
• The second stage is composed automated acceptance tests.
CI server split these tests into suites which can be executed
in parallel to increase their speed and give you feedback
• A feedback cycle enables to see which build is deployed
into which environment, and which stages in your pipeline
each build has passed.
• Demo screenshot from GoCD is shown in next slide

9. A Basic Deployment Pipeline
demo screen of ‘GoCD'

10. Deployment Pipeline Practices
• To get the benefits of Deployment Pipeline approach, there
are some good practices
• Only Build Your Binaries Once
– Collections of executable code are binaries,
– include source files. Jars, .NET assemblies etc., are considered as
the canonical source for many steps.
– The version of the compiler installed in the later stages may be
different from the version that you used for your commit tests.
– only build your binaries once, during the commit stage of the
build.
– Till then, these binaries should be stored on another file-system
somewhere where it is easy to retrieve them for later stages in
the pipeline.

11. Deployment Pipeline Practices
• Deploy the Same Way to Every Environment
– Use the same process to every environment
– Every environment is different in some way.
– Operating system and middleware settings, database location etc..
needs to be set at deployment time.
– Keep the settings that are unique for each environment separate.
– Use separate properties files to hold configuration information for
each environment.
• Smoke-Test Your Deployments
– have an automated script that does a smoke test, to make sure that
the general features are up and running.
– The smoke test, or deployment test, is probably the most important
test to write once you have a unit test suite up and running
– If it doesn’t run, the smoke test should be able to give you some basic
diagnostics as to whether your application is down

12. Deployment Pipeline Practices
• Deploy into a Copy of Production
– At times, production environment is significantly different from testing
and development environments.
– To get a good level of confidence, you need to do your testing and CI
on environments that are as similar production environment.
– Infrastructure details (Network topology and firewall configuration)
and Operating system configuration should be the same.
• Each Change Should Propagate through the Pipeline Instantly
– Refer to the Sequence diagram we discussed y’day..
– The first stage should be triggered upon every check-in, and each
stage should trigger the next one immediately on successful
completion
• If Any Part of the Pipeline Fails, Stop the Line
– The most important step of CD —rapid, repeatable, reliable releases
– If a deployment to an environment fails, the whole team owns that
failure.
– They should stop and fix it before doing anything else.

13. Different Stages
of Deployment Pipeline
The Commit Stage
• A new instance of deployment pipeline is created upon
every check-in
• As the first stage passes, it results in the creation of a
release candidate.
• When a developer commits a change to the version control
system, we must evaluate the latest version of the
application quickly.
• The commit stage typically includes the following steps:
– Compile the code
– Run a set of commit tests.
– Create binaries for use by later stages.
– Perform analysis of the code to check its health.
– Prepare artifacts, such as test databases, for use by later stages.

14. Commit Stage Best Practices
• Most of the practices described in “Continuous Integration,” apply to the
commit stage.
• Developers are expected to wait until the commit stage
of the deployment pipeline succeeds.
• If it fails, they should either quickly fix the problem, or
back their changes out from version control.
• Get feedback as quickly as possible, when problems are
cheap to fix
• Passing the commit stage is an important milestone in
the journey of a release candidate.
• be ready to pay technical debts once your gamble fail.

15. Automated Acceptance Test Gate
• A comprehensive commit test suite is an excellent
litmus test for many classes of errors, but there is much
that it won’t catch.
• Unit tests only test a developer’s perspective of the
solution to a problem; not what it is supposed to from
a users perspective.
• Extend the range of CI process to test modes
• The goal of the acceptance test gate is to assert that
the system delivers the value the customer is expecting
and that it meets the acceptance criteria.
• It also verify that no bugs are introduced into existing
behavior by new changes

16. Subsequent Test Stages
• The acceptance test stage is a significant milestone in the
lifecycle of a release candidate
• Even if you have comprehensive set of automated tests,
some form of
– Manual testing &
– NonFunctional testing, is desirable before release..
• Release management systems such as AntHill Pro and Go
provide the ability to see what is currently deployed into
each environment

17. Preparing to Release
• In order to reduce the business risk associated with
release of a production system, consider it as a natural
outcome of deployment pipeline..
– Have a release plan that is created and maintained by
everybody involved
– Minimize the effect of people making mistakes by
automating as much as possible
– Rehearse the procedure often in production-like
environments
– Have the ability to back out, if things don’t go as per plan
– Have a strategy for migrating configuration and production
data as part of the upgrade or rollback processes

18. Preparing to Release
• In order to reduce the business risk associated with
release of a production system, consider it as a natural
outcome of deployment pipeline..
– Have a release plan that is created and maintained by
everybody involved
– Minimize the effect of people making mistakes by
automating as much as possible
– Rehearse the procedure often in production-like
environments
– Have the ability to back out, if things don’t go as per plan
• Automating Deployment and Release
• Backing out Changes
• Building on Success

19. Implementing a Deployment Pipeline
• You’re starting a new project from scratch or trying to
create an automated pipeline for an existing system,
always take an incremental approach to implementing
a deployment pipeline.
• Deployment pipeline is done through the steps below:
– Model your value stream and create a walking skeleton.
– Automate the build and deployment process.
– Automate unit tests and code analysis.
– Automate acceptance tests.
– Automate releases.

20. Metrics
• Feedback is at the heart of any software delivery process.
– Improve feedback by maintaining the feedback cycles short and
the results visible.
– You should measure continually and broadcast the results of the
measurements, on a visible wall-poster or on a computer
display. (known as information radiators)
• the most important global metric is cycle time.
– the time between deciding that a feature needs to be
implemented and having that feature released to users.
• Use the Theory of Constraints to reduce cycle time by
applying the following process.
1. Identify the limiting constraint on your system.
2. Exploit the constraint.
3. Subordinate all other processes to the constraint.
4. Elevate the constraint.
5. Rinse and repeat.

21. Metrics
• Though cycle time is the most important metric in
software delivery, there are a number of other
diagnostics that can warn you of problems, like..
– Automated test coverage
– Codebase Properties like amount of duplication,
cyclomatic complexity
– Number of defects
– Velocity, the rate at which your team delivers reliable code
– Number of commits to the version control system per day
– Number of builds per day
– Duration of build

22. Metrics
• How are these metrics are presented.?
– Reports produce a huge amount of data, and
interpreting this data is an art.
– Program managers, might expect to see this data
analyzed and aggregated into a single “health” metric
that is represented in the form of a traffic light that
shows red, amber, or green.
– Team/Technical lead will want much more detail, but
do not want to wade through pages of reports.
• Julias Shaw, created Panopticode which runs a
series of these reports and produces rich
visualizations .
• It gives a quick glance to whether there is a
problem and where it lies.

23. Scripting for Deployment Stages
• Build is the construction of something that has an observable and
tangible result. It is the process of converting source code files into
standalone software artifact(s) that can be run on a computer.
• Build automation is the process of automating the creation of a
software build and the associated processes including: compiling
computer source code into binary code, packaging binary code, and
running automated tests.
There are two general categories of tools:
• Build-automation utility : includes utilities like Make, Rake, Cake,
MSBuild, Ant, Maven or Gradle (Java) etc.
– Their primary purpose is to generate build artifacts through activities
like compiling and linking source code.
• Build-automation servers : are general web based tools that
execute build-automation utilities on a scheduled or triggered basis
– a continuous integration server is a type of build-automation server.

24. Scripting for Deployment Stages
• On very simple projects, building and testing the
software can be accomplished using the capabilities of
your IDE (Integrated Development Environment).
• But when the project extends beyond a single person,
spans more than a few days, or produces more than a
single executable file as its output, it demands more
control if it is not to become complex and unwieldy.
• It is also vital to script building, testing, and packaging
applications when working on large or distributed
teams

25. Scripting for Deployment Stages
• Deploying software into
testing and production
environments is not as
simple as dropping a
single binary file into the
production environment
• It requires a series of
steps such as configuring
your application,
initializing data,
configuring the
infrastructure, operating
systems, and
middleware, setting up
mock external systems,
and so on.

26. An Overview of Build Tools
• Automated build tools have been a part of software development
• Various tools and their variants are used in the industry for many
years.
• All build tools have a common core:
• They allow you to model a dependency network.
• It will calculate how to reach the goal you specify by executing tasks
in the correct order, running each task that your goal depends on
exactly once.
For example, To run your tests.
it’s necessary to initialize of your environment,
then compile your code and your tests,
and set up your test data.

27. Build Tools
• Make
– Make is a powerful product-oriented build tool capable of tracking
dependencies within a build and building only those components that
are affected by a particular change.
– GOOD in optimizing the performance of a development team when
compile time is a significant cost in the development cycle.
– BAD: for complex applications, when the number of dependencies
increases, Make become hard to debug.
• Ant
– With the emergence of Java and cross-platform development,
limitations in Make became more painful.
– Java community tried to port Make to Java.
– Meanwhile XML was coming to prominence as a convenient way to
build structured documents.
– These two approaches converged and resulted in the Apache Ant build
tool.

28. Build Tools
• NAnt and MSBuild
– As Microsoft introduced the .NET framework, it had many features in
common with the Java language and environment.
– Java developers then ported some of their favorite open source Java
tools to .Net
– JUnit and JMock were ported as NUnit and Nmock
– And then came the buid tool Nant, which uses Ant like syntax
– Microsoft later introduced their own minor variation on NAnt and
called it MSBuild.
• Maven
– As innovation continued Java Community designed Maven which could
perform better than Ant files by having a more complex domain that
makes many assumptions about the way your Java project is laid out.
– Maven, will perform almost any build, deploy, test, and release task
with a single command,

29. Build Tools
• Rake
– Ant falls as an external domain-specific languages (DSLs) uses XML to
represent
– Rake, is a Ruby build tool, which could easily reproduce Make’s functionality
by creating an internal DSL in Ruby.
– Rake is a product-oriented tool similar to Make, but it can also be used as a
task-oriented tool.
– Rake scripts use Ruby’s API to carry all the tasks
• Buildr
– The simplicity and power of Rake sets a standard that build scripts should be
written in a real programming language. The new generation of build tools,
such as Buildr, Gradle, and Gantt, have taken this approach.
– Buildr is built on top of Rake, Buildr uses the same conventions that Maven
does, Buildr lets you use Ant tasks with zero configuration.
• Psake
– Windows users absorbed the new wave of internal DSL build tools.
– Psake is an internal DSL written in PowerShell, which provides task-oriented
dependency networks.

30. Principles and Practices
of Build and Deployment Scripting
Some general principles and practices of build and deployment scripting
• Create a Script for Each Stage in Your Deployment Pipeline
– once your script gets sufficiently long, you can divide it up into separate scripts for
each stage in your pipeline.
• Use an Appropriate Technology to Deploy Your Application
– It is vital that deployment is automated, one should use the right tool for the job
when automating deployment, not a general-purpose scripting language
• Use the Same Scripts to Deploy to Every Environment
– It is essential to use the same process to deploy to every environment in which your
application runs, and do mention the differences between environments
• Use Your Operating System’s Packaging Tools
– Whenever your deployments involve sprinkling files across the filesystem use a
packaging system to do it. Ubuntu use the Debian package system; flavors of Linux
use the RedHat package system; Windows users can use the Microsoft Installer
system, and so forth..
• Ensure the Deployment Process Is Idempotent
– Your deployment process should always leave the target environment in the same
state, regardless of the state it finds it in when starting a deployment.
• Evolve Your Deployment System Incrementally
– You don’t have to have completed all of the steps to get value from your work

31. Deployment Scripting
• Environment management insists that the changes to testing
and production environments should only be made through
an automated process.
• One should not log into such systems remotely to perform
deployments; they should be entirely scripted.
• We have three options for deploying onto remote machines.
– Write a script that logs into each box and runs the appropriate
commands.
– Write a script that runs locally, and have agents that run the script
on each of the remote machines.
– Package your application, using your platform’s appropriate
packaging technology and have an infrastructure management or
deployment tool push out new versions.
– The third option is the most powerful

32. • Deploying and Testing Layers
– Always strive to build on foundations
that are known to be good.
– Before we bother to copy our binary
deliverables to the correct place in
the filesystem, we want to know that
our environment is ready for us.
– To achieve this, think of deployment
as depositing a series of layers

33. • Testing Your Environment’s Configuration
– Test each layer as it is applied, so that you may fail the
environment configuration process quickly if a
problem occurs.
– Test should give a clear indication of where the
problem lies.
– These tests need not be exhaustive, but should be
very simple “smoke tests” that assert the presence or
absence of some key resources

34. Tips and Tricks
Some solutions and strategies to solve common build and deployment problems.
• Always Use Relative Paths
– ensures that everything is in the right place and works as it should.
• Eliminate Manual Steps
– Every deployment is unique, and, the deployment procedure must be revised for each release
avoiding the reuse of Knowledge and artifacts from previous deployments
• Build In Traceability from Binaries to Version Control
– Be able to determine from any given binary which revision in version control was used to
generate it.
• Don’t Check Binaries into Version Control as Part of Your Build
– To avoid recipe for confusion, put binaries and reports onto a shared filesystem.
• Test Targets Should Not Fail the Build
– record the fact that the activity has failed, and continue with the rest of the build process.
• Constrain Your Application with Integrated Smoke Tests
– Design a constrain for your applications so that they do not work if they find themselves in an
unfamiliar situation.
• .NET Tips and Tricks
– turn on the Show Hidden Files feature and have “clean” call Devenv’s clean solution
command.

35. Details of Commit Stage
• The commit stage begins with a change to the
state of the project in version control
• It ends with either
– a report of failure or,
– if successful, a collection of binary artifacts and
deployable assemblies and reports on the state of
the application.

36. Commit Stage Principles and Practices
• Commit stage is a bouncer at the door of the
deployment pipeline so as to eliminate builds
that are not fit to pass into production.
• The principal goal of the commit stage is
– To either create deployable artifacts, or
– To fail fast and notify the team of the reason for the
failure.
• Some principles and practices can make an
effective commit ., are discussed in next 2 slides

37. • Provide Fast, Useful Feedback
• Failures in the commit tests can be of three causes.
– a syntax error has been introduced into the code, caught by compilation in
compiled languages;
– a semantic error has been introduced into the application, causing one or
more tests to fail;
– a problem with the configuration of the application or its environment
• Whatever the problem, in case of failures, the commit stage should notify
the developers as soon as the commit tests are complete and provide a
concise summary of the reasons for the failures
• What Should Break the Commit Stage?
• At times a green commit stage can even be a false positive, suggesting
that the quality of the application is acceptable when in fact it isn’t.
– Hence, it should be possible to provide richer information, like graphs
representing code coverage and other metrics.
– This information could be aggregated using a series of thresholds into a
3colored traffic lights display

38. • Tend the Commit Stage Carefully
• The commit stage will include both build scripts and scripts to run various tests.
• These scripts need to treated with the same level of importance as you would
treat any other part of your application.
– A poor build system adversely affects the expensive development effort
– Ensure that your scripts are modular, and avoid environment-specific scripts
• Give Developers Ownership
• At some organizations, there are teams of specialists who are experts at the
creation of effective, modular build pipelines and the management of the
environments in which they run.
– But it’s quite bad if only those specialists can maintain the CI system.
• Rather let the delivery team have a sense of ownership for the entire commit stage
• Use a Build Master
• If the teamsize is larger or widely spread in nature, it isn’t always easy to spot the
locate a person who made an error.
– have someone to play the role of a “build master.” to encourage and enforce build discipline.
– If a build breaks, the build master notices and gently (or not gently) reminds the culprit of
their responsibility to fix the build quickly or back out their changes.

39. The Results of the Commit Stage
• The commit stage, has both inputs and outputs.
– The inputs are source code, and the outputs are binaries and reports.
• The outputs of the commit stage, need to be stored somewhere for
your team to be able to reuse them.
• The obvious place might appear to be your version control system,
but is not a good method
– It may misuse the disk space and
– some version control systems won’t support it
• Modern CI servers provide an artifact repository, which allow
unwanted artifacts to be eliminated after some length of time.
– Another way is using a dedicated artifact repository like Nexus,
Maven-style repository manager, which would access the binaries
from development machines without having to directly integrate with
CI server.

40. The role of the Artifact Repository
• Process detailed in next slide

41. The role of the Artifact Repository
1. Somebody commits a change.
2. CI server runs the commit stage.
3. The binary, reports and metadata are saved to the artifact repository.
4. CI server retrieves the binaries and deploys to a production-like test environment.
5. CI server runs the acceptance tests, reusing the binaries created by the commit stage.
6. On success, the release candidate is marked as having passed the acceptance tests.
7. Testers obtain a list of such builds, and press a button to run the automated process to
deploy them into a manual testing environment.
8. Testers perform manual testing.
9. On success, testers update the status of the release candidate
10. CI server retrieves the latest candidate that has passed, from the artifact repository and
deploys the application to the production test environment.
11. The capacity tests are run against release candidate.
12. On success, the status of the candidate is updated to “capacity-tested.”
13. This pattern is repeated for as many stages as the pipeline requires.
14. Once the RC has passed through all of the relevant stages, it is “ready for release,” by
QA and operations people.
15. At the conclusion of the release process, the RC is marked as “released.”

42. Commit Test Suite
Test automation pyramid
• Mike Cohn proposed a visualizing to structure the
automated test suite.
• Test automation pyramid, has the unit tests form
the vast majority of the tests.
• But since they execute so fast, the unit test suite
should complete in just a few minutes.
• There are fewer acceptance tests
• these will typically take far longer to execute
because they run against the full running system.
• All levels are essential to ensure that the
application is working and delivering the expected
business value.
• This test pyramid covers the left-hand side of the
testing quadrant diagram we discussed in
Module1

43. Commit Test Suite Principles and Practices
• There are some important principles and practices governing the design of
a good commit test suite
• Avoid the User Interface
– don’t test via the UI at all. because it tends to involve a lot of components or
levels of the software under test.
– UIs are designed to work at human timescales which are desperately slow.
• Use Dependency Injection
– Dependency injection, or inversion of control, is a design pattern that
describes how the relationships between objects should be established from
outside the objects rather than from within. (only for object-oriented
language)
– a great route to flexible, modular software
• Avoid the Database
– eliminate the database from your tests, to enhance layering and separation of
concerns in your code
– separate the code under test from its storage and use of techniques like
dependency injection

44. Commit Test Suite Principles and Practices
• Using Test Doubles
– In a well-designed system, each class is relatively small in size and achieves its goals through
interactions with other classes.
– in such a modular system, testing an object may require lengthy setup in all the surrounding
classes.
– The solution is to mock the interactions with a class’ dependents using Stub that could
simulate the operation of the live system.
– There are several mocking toolsets, such as Mockito, Rhino, EasyMock, JMock, Nmock etc.
• Faking Time
– Time can be a problem in automated testing
– Eg: system needs to trigger an end-of-day process at 8 P.M. or it needs to do something
different on February 29th of a leap year.
– Problem: It would be disastrous for your unit-testing strategy if you try to tie them to the real
system clock.
– Solution: stub or mock the behavior of our Clock class / Calendar class, which is fully under
our control
• Brute Force
– Its good to have the fastest commit cycle.
– But at times, it is better to accept a slower commit stage than to spend too much time
optimizing the tests for speed or reducing the proportion of bugs they catch.
– This is an optimization process that can only work through trial and error.

45. Thank You
• End of M#3
© dr. sudheer s marar
DEPARTMENT OF MCA
NEHRU COLLEGE OF ENGINEERING AND RRESEARCH CENTRE