Continuous test automation

Continuous Test Automation
Test Automation Framework using
Jenkins CI to drive the execution and
present the results
vireshdoshi@time2test.co.uk 1Feb 2015

About
• This presentation details the thinking behind
how I managed to integrate Test Automation
with Continuous Integration for a client in the
Broadcasting technology sector.

Take away message
“Test automation alone is useful but test
automation hooked up with a Continuous
Integration Platform is damn powerful”
Viresh Doshi – February 2015
Feb 2015 vireshdoshi@time2test.co.uk 3

About Viresh Doshi
• Test Specialist with over 15 years industry
experience.
• Started out testing Air Traffic Management
Systems for Lockheed Martin.
• Gained Test Consultancy working for one of
the first pure play test consultancies.
• Now help businesses/projects mature the QA,
Agile and Test process.

The problems
• The client was only manually testing with
some adhoc automated unit testing.
• The team was transforming to Agile.
• Team was using Jenkins CI for building
software.
• The test environments were not controlled.
• Multiple teams from the UK, USA and Canada
updating the main code base and difficult to
pinpoint regressions.

The biggest problem
• The software gets to the client and…
– Software does not install.
– The software is missing a database change.
– The client can not login
– Basic functionality is broken

Pressure to Deliver

Hold on!
• Jenkins is a build tool and not a testing tool.

The system
Made up of
three key
componentsTest
automation
tool
Jenkins CI
Jenkins
Plugins
GIT –
source code
Control

Test automation tool
• This is any test automation tool that best
compliments the system under test e.g.
– Test Complete
– Selenium
– jMeter
– Quick Test Professional
– Home Grown Code and Tools

GIT source control
• GIT is open source
• Easy to use and manage
• Client tools allow for easy usage
• Integrates with Jenkins CI
• Web management system
• Ideal choice for management of all test
configurable items and test code.

What does it look like?

In more detail?
 The radiator view is what we
stare at daily!
 Each block is a labeled “job”
that does something.
 Green indicates success and
Red/Yellow indicates failure or
issues.
 Running progress is shown.
 Job run time and version is
shown.
 Red blocks need investigating
immediately.

What “things” does it do?
• Smoke tests – functional test coverage of the
key functions of the system
• End to End business scenarios
• New features
• Software installation
• Configuration
• Monitoring of servers , database server and
log files

Test Starting point
• A new production development build is the
initial trigger point
• That build is copied across to the test
automation system and then installed and
then the suite of automation tests are run and
reported.

System Vision and Values
• Continuous installation
• Continuous testing
• Continuous reporting
• Easy to manage
• Source Controlled
• Open Source Technology
• Scalable

JENKINS
Continuous Integration Platform

About
• This section contains information on the
Jenkins plugins used that achieve Continuous
Test Automation.

Jenkins
What is Jenkins?
• A CI Build Platform that allows the user to
create build process jobs that do “things”.
• Jenkins Plugins allow the integration and
ability to enrich the “things” being done.
• When those “things” are not successful then a
red fail is reported and it’s possible to drill
down further and investigate problems.

Concepts
• Automate Repetitive tasks
• Report Pass and Fails
• Provide easy access to Log files
• Allow for Parameters
• Integration to existing systems
• Establish a Workspace

Plugin: Build Monitor
What does it do?
• Provides a visual “radiator” view of configured
jobs.
Why?
• The view shows the jobs key features like: how
long it took to execute, running state, pass/fail
and build number.

Plugin: Build Name Setter
What does it do?
• Allows for a user set a build name
Why?
• The standard build name is just an
incremented number.

Plugin: Build Pipeline
What does it do?
• Provides a visual view of upstream and
downstream connected jobs.
Why?
• Builds and jobs can get complicated, so it is
useful to have a visual view of connected jobs.

Plugin: Config slicing
What does it do?
• Allows multiple projects settings to be
adjusted conveniently.
Why?
• A great way to manage configuration settings
when you have 100s of similar Jenkins jobs.

Plugin: Copy Artifact
What does it do?
• Allows artifacts to be copied from other build
projects
Why?
• Useful for taking successful executables from
other projects for usage.

Plugin: Green Balls
What does it do?
• Simply shows a green for Pass and Red for Fail
Why?
• Jenkins automatically uses a colour blind
scheme – orange and yellow.

Plugin: GIT Plugin
What does it do?
• Integrates into GIT source code control.
Why?
• Needed to pull the latest code changes from
source control.

Plugin: HTML Publisher
What does it do?
• A HTML report is published per run
Why?
• It was used to publish the HTML execution
results as created by Test Complete
Automation Tool

Plugin: PowerShell
What does it do?
• Integrates the ability to run PowerShell scripts.
Why?
• PowerShell is a very powerful scripting
language that adds endless possibilities for
windows based servers.

Plugin: jUnit
What does it do?
• A nice graph with a pass/fail report in jUnit
style is presented per run.
Why?
• A test is not a test without a pass/fail result.
Any failures can also display stack traces or
error screenshots.

Plugin: MSBuild
What does it do?
• Allows MSBuild to build .NET projects
Why?
• The executable isbuilt consistently using
Jenkins instead of adhocly on a developer’s
machine.

Plugin: Sidebar Link
What does it do?
• Simply adds a HTML anchor link to the Jenkins
Sidebar
Why?
• To simplify our life!

Plugin: Workspace Cleanup
What does it do?
• Cleans up the workspace before a job is
executed.
Why?
• Only that run’s current artifacts are stored.

CONCLUSION
Lessons learned and tips

Tips
• Try and limit the full suite of tests to 2 hours
execution time.
• Too many builds provided by the developers –
Limit the execution to twice a day.
• Failures could be with problems with the tests
and not the code.
• The system needs to be tested. Watch the test
execution and double check that it is doing
what you expect.

Next Steps
• Introduce Maven or equivalent to manage
dependencies.
• Introduce a first time install test on virtualized
servers.
• Introduce Continuous Delivery

Thank You
I am looking for a new challenge!
vireshdoshi@time2test.co.uk
07970 291828
UK

Continuous test automation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Continuous test automation

Similar to Continuous test automation (20)

Recently uploaded

Recently uploaded (20)

Continuous test automation