Continuous Integration

Whitepaper
/
Continuous integration: stability,
reliability and speed in software
development

Table of Contents
Introduction......................................................................................................................................................................................................3
Why continuous integration?...........................................................................................................................................................................4
Benefits of continuous integration....................................................................................................................................................................6
Challenges and Solutions................................................................................................................................................................................9
Achieving full test automation-gradually.........................................................................................................................................................10
Managing complexity......................................................................................................................................................................................11

Introduction
Stability, reliability and speed are just
some of the words used to describe the
advantages of a software development
practice known as Continuous Integration
(CI). It has transformed systems
development. Martin Fowler, a thought
leader on CI describes1 it as the practice
of integrating code into a shared repository
regularly with automated check-ins
authenticating the build to identify issues.
This procedure allows teams to detect and
resolve problems early and quickly –
before they become major downstream
issues.
CI’s key benefit could be summed in one
word. Quality. Developer teams at
Computaris use CI for one aim - to achieve
excellence. Some projects are more
demanding than others. Larger projects
require change often during the
development lifecycle.
The CI process allows Computaris to
deliver outstanding results – faster.
Computaris has an excellent track record
of delivering successful projects.
Customers come to Computaris for both
new projects and to seek counsel on
existing projects expecting swift
turnaround times combined with excellent
quality. This whitepaper explains best
practice methodology for CI and the
benefits it delivers.
/
development
page 3

Why continuous
integration?
In today’s dynamic programming
environment, software development is
getting increasingly complex. For example,
systems being built on established
languages2 today could easily be outdated3
in a matter of a few years. As such, IT
teams need to stay ahead of the curve in
software development best practice to
create systems and programs that operate
effectively not just in the short term, but
also continue to deliver results in the
future. Teams that adopt CI can ensure
continuity.
By using CI, development teams can
continuously integrate and modify code
within a shared repository several times a
day. An automated build verifies each
check-in and this allows development
teams to detect problems early4. This is an
important step to minimize the cost of
fixing anomalies and to maximize build
quality.
The technique that underpins CI is the
integration of code being constantly
developed and added to a code base or a
source code control system5. In the past,
most developer teams relied on a daily
build. Now it is at least once a day with
each significant change. This enables the
developer team to have constant feedback
on the status of the software and
anomalies are easier to manage.
These are the basic tenets of CI:
> Use a single repository for source code
control: This is the developer team’s fail-
safe and hub for automation6.
> Self-testing the builds: Have a suite of
automated tests to examine the code base
for bugs regularly7.
> Validate each commit: Select and stick
to a consistent coding standard. Regularly
monitor the repository for oversights and
check each commit8.
> Commit often: Team members need to
commit regularly within an agreed
timeframe to reduce the likelihood of
conflict changes9.
> Test accurately: A realistic test
environment that reflects the production
environment will contribute to obtaining a
higher-quality of the final build10.
> Efficient and transparent processes:
Having good API documentation with
robust processes can ensure that
everyone, including customers, are in-sync
with the build to avoid needless and costly
rebuilds11.
> Deployment strategy: When the
business is ready, use a script to deploy
the application into production via an
automated live test server that's
accessible to key stakeholders12.
page 4
2Different Programming Languages
3Fabien Potencier blog
4Mike Jackson, Software Sustainability Institute
5C. Aaron Cois, Software Engineering Institute
6Martin Fowler blog
7Martin Fowler blog
8Paul Jones, Why Coding Standards Matter
9Seth Robertson, Blog
10John Overbaugh, TechTarget
11David Flanders and Malcolm Ramsey, Joint Information
Systems Committee
12Eric Reiss, Radar
/
development

Benefits of continuous
integration
Faster time-to-market
In the past, development teams were
bogged down with paperwork and had to
write deployment codes for other teams to
follow. This was a silo approach. Such
time-consuming process can mean that
time-to-market for new products and
services could take months – even
years13. CI has transformed and
accelerated time-to-market.
An automated pipeline of continuous
delivery ensures that only good quality
code makes it all the way through14. As
such, automation plays a fundamental role
in CI and helps to build a robust
development work flow where fixing bugs
and improving quality is at the heart of the
system. This level of quality control speeds
up time-to-market and gives businesses a
competitive edge15.
Faster development lifecycles
The usual developer process, without CI,
can often comprise of the following steps:
1. Develop new code OR perform bug
fixing a. This includes Unit tests or
manual smoke tests
2. Commit code in version control
3. Wait for QA teams to raise issues or
verify fixes
4. Receive issues from QA teams and
return to Step 1
Some developers write code, run a few
tests and pass the code to someone else
to verify it16. That’s inefficient, it damages
quality and hampers knowledge sharing
within the team17. Steps 3 and 4 usually
take a long time, measured in hours if not
days. By the time a developer received
feedback, he could have moved on to
different tasks and prioritized something
else. It could take more time for the
developer to familiarize himself with the
code again and retrieve that context. If the
duration for steps 3-4 took even longer
than just a few days, it is very likely that
another developer could have taken over
the fixes and the original developer might
not even be available. All this adds
unnecessary delays and is a setback for a
business hoping to deploy a solution to the
market.
page 5
13Sven Malvik blog
14Florian Motlik, Codeship
15Jason Tee, TheSeverSide
16Scott W Ambler
17Jason Cohen, SmartBear Software
/
development

integration
CI provides a new lease of life for the
developer process:
1. Develop new code OR perform bug
fixing
a. This includes adjusting the automated
tests to verify the new code
b. For large test suites, nominal bugs and
glitches should be addressed at this step
2. Commit code in version control
3. CI server (e.g. Jenkins) detects the new
commit and runs a new build
4. Team members are notified via e-mail if
the CI build found issues Some developers
write code, run a few tests and pass the
code to someone else to verify it16.
That’s inefficient, it damages quality and
hampers knowledge sharing within the
team17. Steps 3 and 4 usually take a long
time, measured in hours if not days. By the
time a developer received feedback, he
could have moved on to different tasks
and prioritized something else. It could
take more time for the developer to
familiarize himself with the code again and
retrieve that context. If the duration for
steps 3-4 took even longer than just a few
days, it is very likely that another
developer could have taken over the fixes
and the original developer might not even
be available. All this adds unnecessary
delays and is a setback for a business
hoping to deploy a solution to the market.
CI provides a new lease of life for the
developer process.
The main difference is that steps 3 and 4
are automated and happens in a window
of just a few minutes – not days - from the
time the developer finished coding. The
developer will not have time to lose
context, any issue detected by the CI build
will be fixed much faster.
Other CI applications referenced by this
White Paper aim to have 1 to a few builds
per day, a build lasting one to several
hours. We try to design our builds to be
faster, in the range of 10 to 15 minutes
with a test suite having over a thousand
full end-to-end automated tests (see “4.4
Parallelization and configuration”).
Quality: from good to great
CI makes the integration process efficient.
Without CI, Quality Assurance (QA)
engineers could spend more than 50% of
total project time running nominal tests for
integration. This will shorten the time the
QA team spends looking for bugs and
other subtle issues18.
page 6
18Gautham Pallapa, blog
/
development

integration
The CI process ensures that the QA team
receives code with integration done and
the nominal test cases are successful.
This frees the QA team to dive deeper and
focus on problems, test more scenarios
and yield better overall quality and
results19 .
Stable codes and demo ready
The fast development lifecycle described
in section 3.2 will allow a large part of the
code to be stable even when under active
development. Agile development
methodologies20, which rely on
incremental development and getting
constant feedback from stakeholders,
ensures that the project is on the right
track. Regardless of the project
management techniques used, by using CI
the team can be ready at any time to demo
the system. This allows for quick feedback
during customer meetings and ensures
requirements are well understood and the
project is moving in the right direction. As
such, customer-led improvements can be
incorporated within a matter of a few hours
– and not days21.
Smarter risk mitigation
During a build's lifecycle, there is always
the possibility of probable mistakes22. If
there are suitable parameters and metrics
in place, the status of the build can be
constantly monitored accurately. CI
enables developer teams to discover and
resolve flaws promptly. Furthermore CI
provides intelligence for teams to eliminate
potential vulnerabilities too. This is
especially the case when they have
access to historical data which allows QA
teams to build test cases to ensure that
historic mistakes are not repeated.
CI mitigates risk because check-ups and
trials are run frequently - in some cases
every few hours - so there's a higher
probably of detecting issues and fixing
them before launch. This ensures that
businesses have robust solutions they can
deploy with confidence. Joe Green, a
developer and champion of CI wrote:
"Having a good CI system is akin to having
a fire alarm in your house. It won't fix your
error for you but you can be rest assured
that it will flag up the error as soon as it is
integrated and that the build containing the
error will not make it into your staging or
production environments.23
Flexibility
CI underpins flexibility and can foster
creativity in the development team. Thanks
to the automated nature of CI,
development teams have the flexibility to
change code quickly and without fear. Fast
CI builds give developers the opportunity
and freedom to innovate and try new
techniques to enhance the system. This is
one of the key benefits of CI.
page 7
19Gautham Pallapa, blog
20Agile Alliance
21Duval, Matyas et al. Tech Target
22Francis Adanza
23Joe Green, Smrtr
/
development

integration
Faster onboarding for new team
members
Sometimes onboarding new team
members, especially within a large project,
can take up valuable time. If key members
of the developer team are not available,
bringing someone up to speed on a build
can be challenging. CI makes onboarding
more efficient. It generates API-based
documentation to make the software
architecture easily accessible to new
members on the developer or clients’
teams. You only need to browse the test
cases to have a quick overview of the
functionality of the system.
Having a CI process allows you to have
the tools to automatically build, deploy,
and test the system. These are the same
tools needed for a new developer/QA
engineer to hit the ground fast25.
24Manuel Weiss, Codeship
25Darragh Curran, Inside Intercom
/
development

Challenges and solutions
While there are many benefits to CI,
developer teams and customers can face
a number of challenges from the build
phase to deployment.
Achieving full test automation
gradually
Automating an entire build at the start is
not easy, so start simple26. Manage the
process continually by keeping
deployments consistent and have a well-
defined process that can be constantly
automated and developed over time. This
continuous automation methodology can
help developers automate the entire build.
Even with the 80/20 principle27, achieving
100% automation is obviously more
difficult than 80% automation. Some
developers have found that fixing 20% of
the bottlenecks could give you 80% gain in
speed28. Yet coding automated tests is still
coding that requires careful handling.
Some teams fail to build a reliable CI
process because they consider it is easy
and do not pay enough attention to detail.
Here are some common mistakes:
> Poorly-coded automated tests will cause
team members to spend more time
investigating failures related to the test
suite itself, rather than investigate real
errors of the product that’s being tested.
> After a while, frustration grows, and the
entire test suite is abandoned or rewritten
from scratch.
> Poorly-managed infrastructure will cause
environmental problems and will probably
impact reliable CI builds. These problems
can include: malfunctioning machines,
patchy networks, disks with limited space,
and sluggish server and CPU
performance, which can cause CI builds to
run slowly.
26Eric Reiss, Radar
27Pareto Principle
28Naresh Jain, Key Principles for Reducing
Continuous Integration Build Time
/
development

Managing complexity
Software development best practices must
be carefully applied to test code not only to
production code. Especially for
large/complex test suites with
hundreds/thousands tests, poorly
implementing automated tests will greatly
increase cost of maintenance and may
even lead to completely abandoning the CI
test suite.
It is important for companies to ensure that
their developer/quality assurance teams
adhere to best practices, including Don’t
Repeat Yourself (DRY) and Keep it Simple
Stupid (KISS). In their book, the Pragmatic
Programmer, Andy Hunt and Dave
Thomas coined the phrase DRY29. DRY is
about reducing complexity to manageable
units by dividing a system into pieces30. As
systems are fragmented into components
and subcomponents, complexity could be
reduced to a single responsibility.
Keep It Simple Stupid or Keep it Stupid
Simple (KISS) is another best practice
concept in software development which
means – the simplest explanation is likely
to be the best one31. In KISS, developers
aim to find a resolution as easily as
possible and produce code to solve
complex problems in fewer lines of code –
faster and with remarkably better quality.
Best practice guidelines for complex
coding
> Conduct tests on asynchronous I/O
(Input/Output) for faster and accurate
results instead of relying on “sleep” mode32
> To manage test data efficiently, select an
accessible tool or language so team
members can retrieve data efficiently33
> Test scenarios must produce log files
easy to read and follow, this is mandatory
for easy tracking of issues when the CI
build fails34.
29Andy Hunt and Dave Thomas, The Pragmatic
Programmer
30Chris Peters, Code
31The Apache Software Foundation
32Mixu, blog
33Cohen, Hunter et al., Association of
Computational Linguistics
34Tamir Korem, Cloudify
/
development

Managing complexity
Maintaining build speed
For improved productivity, builds need to
be fast. Along with best practice concepts,
the developer teams also need to consider
the following:
> Ideally a build should be fast enough to
be able to run and validate each commit a
developer makes in the version control
system early and regularly35. Of course
this is hard to achieve especially for large
projects with teams of many developers
where a commit might happen every few
minutes
> It is difficult to find an issue with 10
commits and 200 failures out of 1000
tests. Investigating each commit takes
time. A fast build exposing problems in
each individual commit would minimize
costs of debugging/investigating
> Black box testing, where testers are only
aware of what the software is supposed to
do and not how it does it, runs much
slower than unit tests. Unit tests verify the
functionality of a specific section of code.
All this can be complex36.
> For black box test, all components of the
system under test must be built, deployed
and configured fast. Assertions on system
behavior, where tests can be run on your
assumptions, are black box. They verify
the system’s output so they must wait for
disk I/O, network I/O, caches to fill up,
messages to pass through queues and all
other type of events that must happen until
a noticeable output from the system can
be tested. All these operations take time
and making the build fast becomes a
challenge.
Parallelization and configuration
The solution to make the test suite faster
is to use parallelization, which means
running tests on multiple components or
subcomponents simultaneously. One of
Computaris’ test suites can run
approximately 1000 tests in parallel with
corresponding sub-tests concurrently. A
rough estimation would be 40 processes
each with an average of 100 threads
running in parallel. Of course this requires
a fair amount of hardware.
Most systems have system-wide
configurations which are changed very
rarely. A CI build must test various
configuration options and this creates a
new problem, how can you run parallel
tests with different configurations? The
solution is yet again parallelization.
Computaris has the capability to carry out
multiple system tests that can run
simultaneously, each instance with its own
configuration.
35Wikipedia, Revision Control
36Black Box Testing
/
development

Managing complexity
This approach requires the following:
> The need for robust hardware: multiple
instances of parallel system runs tend to
require a great deal of hardware
resources.
> The need to automate the infrastructure:
Computaris has a regression suite with
over 1000 tests running on 16 parallel
systems being tested and uses tools such
as Chef to manage infrastructure
automatically37. This also helps to solve
the need for many hardware resources.
Having automation allows the team to run
CI builds on commodity hardware that may
fail. Recovery from these failures is
extremely fast using “infrastructure as
code” which eliminates the need for
expensive server hardware38.
Make tracking down real problems easy
and fast
Ultimately, finding and resolving problems
faster is why CI has been used for the
build. For large test suites (e.g. 1000 tests)
if you have 200 tests that fail, the simple
action of checking the failure message of
each of the 200 failures can be time
consuming.
In this instance, a “Root Cause Analysis
Report” which looks at the origin of the
problem and the corrective measures
required can be helpful for the developer
team. The Root Cause Analysis Report
can group tests by cause of failure and
provide an overview of the problems that
caused the build to fail in the first place39.
Decide on your tests
For the CI build to be successful, it is
important for the developer team to settle
on a testing regime. Part of Extreme
Programming (XP), Test-Driven
Development (TDD) uses automated unit
tests to “get something working now and
perfect it later40.” After each test has been
evaluated, similar tests are performed
again until it reaches a satisfactory result.
A collaborative practice, Acceptance Test
Driven Development (ATDD) involves
customer being part of the test design
process to define criteria41. ATDD ensures
that all stakeholders are project members
and they understand precisely what needs
to be done and implemented42.
37 JT Gray, Safari Blog
38Hummer, Rosenberg et al, Distributed Systems
Group
39Alon Linetzki, Root Cause Analysis for Software
Testers
40Narayana Maruvada, Agile Record
41Scott W Ambler 42Gabo Esquivel, blog
42Gabo Esquivel, blog
/
development

Managing complexity
Behavior-Driven Development (BDD) is, as
the name suggests, behavior-based build
which incorporates the practices of TDD
and focuses on language and
interactions43. One benefit of BDD is that it
provides the developer team and customer
a clearer indication as to what the system
should do.
TDD can be more technical and complex
that requires developer teams to become
conversant with the detailed object
model44.
Continuous deployment
Continuous Deployment or continuous
delivery (CD) is described as the practice
of releasing only good build to the final
user45. This concept is based on CI and is
a continuation of this principal to the 'last
mile' for customer delivery46. With CD,
teams can quickly adapt to business
requirements and user needs which in turn
enabled greater co-operation between
operations and delivery, championing best
practice and driving real change47.
Software developers deploy their software
into production each night and this
placement provided a number of benefits:
written software was not wasted,
developers responded quickly to problems
and fast turn-around times led to valuable
partnerships between teams and their
customers48. Similar to CI, CD mitigates
risk and detects bugs fast. Crucially, it also
enables the developer team to progress on
to fully functioning software even faster.
Fine-tuning
As we've seen, CI is one of the most
effective concepts for software
development. However for it to work
perfectly, it requires discipline in the team.
However, developer teams are human
after all. Sometimes, the bigger the team
and the code, the more likely it can get
broken and the lengthier the process will
be to fix it because other business
priorities take precedence49. Team
members can ignore the red flags from
Jenkins and overtime flawed software can
get passed on to the end customer. The
solution? Forbid team members from
including anything into the main repository
and make a script that anyone can access
easily. The script should merge, test, and
commit without exceptions. The developer
of each code should be responsible for
broken tests and the team should raise red
flags before a code gets into the master
repository.
43Agile Alliance
44Gabo Esquivel, blog
45Carl Caum, Puppet Labs
46Martin Fowler, blog
47Florian Motlik, Blog
48Sarah Goff-Dupon, Atlassian Blog
49Yegor Bugayenko, Dev Ops
/
development

/
Thank
you!
/
[+44]20.7193.9189
www.computaris.com

Continuous Integration

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Viewers also liked

Viewers also liked (11)

Similar to Continuous Integration

Similar to Continuous Integration (20)

More from Computaris

More from Computaris (20)

Recently uploaded

Recently uploaded (20)

Continuous Integration