This presentation and discussion was given at the April 2015 TMF UK by Viresh Doshi

1. Discussion at the TMF UK - London
Test Automation Framework using
Jenkins CI to drive the execution and
present the results
1April 2015
Presented at the TMF UK
See http://uktmf.com/

2. About
• This presentation details the thinking behind
how I integrated Test Automation with
Continuous Integration.
2April 2015
Feel free to interrupt at any time!

3. Today’s aim
Sell the vision and principles of continuous
test automation to help you make the change
on your project.
3April 2015

4. Today’s agenda
1st Half – presentation
• Walk through the
concepts and case
study.
4April 2015
2nd Half – Discussion
• How to get started?
• What needs
automating?
• What skills are
needed?
• Advantages and
Disadvantages

5. Take away message
“Test automation alone is useful but test
automation hooked up with a Continuous
Integration Platform is damn powerful”
Viresh Doshi – April 2015
April 2015 5

6. 6April 2015
Ultimate Aim
“Continuous Deployment” – Sit back and Enjoy
the Ride

7. Alternative view point
7April 2015
“Continuous Deployment” – Sit back and Relax
And get sacked!

8. About Viresh Doshi
• Test Specialist with over 15 years industry
experience.
• Started out testing Air Traffic Management
Systems for Lockheed Martin.
• Gained Test Consultancy working for one of
the first pure play test consultancies.
8April 2015

9. CASE STUDY
Real life example
9April 2015

10. The problems
• Manually testing with some adhoc automated
unit testing.
• The team was transforming to Agile.
• Team was using Jenkins CI for building
software.
• The test environments were not controlled.
• Multiple teams from the UK, USA and Canada
updating the main code base and difficult to
pinpoint regressions.
10April 2015

11. The biggest problem
11April 2015
• The software gets to the client and…
– Software does not install.
– The software is missing a database change.
– The client can not login
– Basic functionality is broken

12. Pressure to Deliver
April 2015 12

13. Hold on!
• Jenkins is a build tool and not a testing tool.
13April 2015

14. The system
Made up of
five key
componentsTest
automation
tool
Jenkins CI
Jenkins
Plugins
GIT –
source code
Control
14April 2015
Glue Tools

15. Test automation tool
• This is any test automation tool that best
compliments the system under test e.g.
– Test Complete
– Selenium
– jMeter
– Quick Test Professional
– Home Grown Code and Tools
15April 2015

16. GIT source control
• GIT is open source
• Easy to use and manage
• Client tools allow for easy usage
• Integrates with Jenkins CI
• Web management system
• Ideal choice for management of all test
configurable items and test code.
16April 2015

17. Build Tools and Glue
• ANT
• PHING
• NANT
• Bash Shell Scripts
• PowerShell Scripts
• Other
April 2015 17

18. What does it look like?
18April 2015

19. In more detail?
 The radiator view is what we
stare at daily!
 Each block is a labeled “job”
that does something.
 Green indicates success and
Red/Yellow indicates failure or
issues.
 Running progress is shown.
 Job run time and version is
shown.
 Red blocks need investigating
immediately.
19April 2015

20. What “things” does it do?
• Smoke tests – functional test coverage of the
key functions of the system
• End to End business scenarios
• New features
• Software installation
• Configuration
• Monitoring of servers , database server and
log files
20April 2015

21. Test Starting point
• A new production development build is the
initial trigger point
• That build is copied across to the test
automation system and then installed and
then the suite of automation tests are run and
reported.
21April 2015

22. System Vision and Values
• Continuous installation
• Continuous testing
• Continuous reporting
• Easy to manage
• Source Controlled
• Open Source Technology
• Scalable
22April 2015

23. JENKINS
Continuous Integration Platform
23April 2015

24. About
• This section contains information on the
Jenkins plugins used that achieve Continuous
Test Automation.
April 2015 24

25. Jenkins
What is Jenkins?
• A CI Build Platform that allows the user to
create build process jobs that do “things”.
• Jenkins Plugins allow the integration and
ability to enrich the “things” being done.
• When those “things” are not successful then a
red fail is reported and it’s possible to drill
down further and investigate problems.
25April 2015

26. Concepts
• Automate Repetitive tasks
• Report Pass and Fails
• Provide easy access to Log files
• Allow for Parameters
• Integration to existing systems
• Establish a Workspace
• Reduce School Boy Errrors.
26April 2015

27. Plugin: Build Monitor
What does it do?
• Provides a visual “radiator” view of configured
jobs.
Why?
• The view shows the jobs key features like: how
long it took to execute, running state, pass/fail
and build number.
27April 2015

28. Plugin: Build Name Setter
What does it do?
• Allows for a user set a build name
Why?
• The standard build name is just an
incremented number.
28April 2015

29. Plugin: Build Pipeline
What does it do?
• Provides a visual view of upstream and
downstream connected jobs.
Why?
• Builds and jobs can get complicated, so it is
useful to have a visual view of connected jobs.
29April 2015

30. Plugin: Config slicing
What does it do?
• Allows multiple projects settings to be
adjusted conveniently.
Why?
• A great way to manage configuration settings
when you have 100s of similar Jenkins jobs.
30April 2015

31. Plugin: Copy Artifact
What does it do?
• Allows artifacts to be copied from other build
projects
Why?
• Useful for taking successful executables from
other projects for usage.
31April 2015

32. Plugin: Green Balls
What does it do?
• Simply shows a green for Pass and Red for Fail
Why?
• Jenkins automatically uses a colour blind
scheme – orange and yellow.
32April 2015

33. Plugin: GIT Plugin
What does it do?
• Integrates into GIT source code control.
Why?
• Needed to pull the latest code changes from
source control.
33April 2015

34. Plugin: HTML Publisher
What does it do?
• A HTML report is published per run
Why?
• It was used to publish the HTML execution
results as created by Test Complete
Automation Tool
34April 2015

35. Plugin: PowerShell
What does it do?
• Integrates the ability to run PowerShell scripts.
Why?
• PowerShell is a very powerful scripting
language that adds endless possibilities for
windows based servers.
35April 2015

36. Plugin: jUnit
What does it do?
• A nice graph with a pass/fail report in jUnit
style is presented per run.
Why?
• A test is not a test without a pass/fail result.
Any failures can also display stack traces or
error screenshots.
36April 2015

37. Plugin: MSBuild
What does it do?
• Allows MSBuild to build .NET projects
Why?
• The executable isbuilt consistently using
Jenkins instead of adhoc on a developer’s
machine.
37April 2015

38. Plugin: Sidebar Link
What does it do?
• Simply adds a HTML anchor link to the Jenkins
Sidebar
Why?
• To simplify our life!
38April 2015

39. Plugin: Workspace Cleanup
What does it do?
• Cleans up the workspace before a job is
executed.
Why?
• Only that run’s current artifacts are stored.
39April 2015

40. CONCLUSION
Lessons learned and tips
40April 2015

41. Tips
• Try and limit the full suite of tests to 2 hours
execution time.
• Too many builds provided by the developers –
Limit the execution to twice a day.
• Failures could be with problems with the tests
and not the code.
• The system needs to be tested. Watch the test
execution and double check that it is doing
what you expect.
41April 2015

42. Next Steps
• Introduce Maven or equivalent to manage
dependencies.
• Introduce a first time install test on virtualized
servers.
• Introduce Continuous Delivery
42April 2015

43. DISCUSSION
Experiences and Feedback
43April 2015

44. Test Automation Challenge
• Manual testing only
• Manual Builds
• Manual deployments
• Manual Release notes
• Manual dependencies
• Semi Automated Development Integration
• Some Automated Test Data Generation
• Monthly release to production
44April 2015

45. 45April 2015
• The company is doubled every year for the
past three years
• One dedicated QA Body
• Developers are the kings
• Not everybody is sold
• More clients
• More staff
• More Room for…
My Current Challenge

46. Let’s talk
46April 2015
Wake up!

47. Your challenges
• Tell me about your manual challenges?
• What is stopping you from automating?
• What is the starting point?
• Do you have the skills?
• Who is responsible?
• What can go wrong?
• What are the advantages and disadvantages?
• What is the overall goal?
47April 2015

48. Thank You
48April 2015