Running a Test Harness in Parallel using jenkins. Presented at SiteSpect for meetup.com automated testing boston group, this presentation covers the fundamentals of how sitespect took their 350+ test system and parallelized it over 75 virtual machines, reducing run time from 8 hours to 30 minutes.

1. Thank you for attending
Automated Testing Boston Meetup
Sponsored by
Follow us:
@SiteSpect

2. Running Regression
Tests in Parallel
using Jenkins
Presented by Josh Shapiro & Dustin Masterson
SiteSpect Inc.

3. Outline
Background
Description of Parallelized System
Jenkins Performance
Conclusions
Q&A

4. Part I : Introduction

5. What is SiteSpect?
SiteSpect is website testing and optimization
It is a reverse proxy
Uses regex to apply changes to page
Control panel for creating tests, analyzing data

6. SiteSpect Example
vs

8. What is the Test Harness?
Sets up SiteSpect and a website
Verifies:
Using the control panel
Visitor assignment
Data collection

9. Diagram of the Test Harness
Harness VM
Headless Browser
Harness
Test Web Site
Control Panel
SiteSpect
Traffic Processing
1. Configure Test
Objects/Features
2. Stream Traffic
a. User
b. SiteSpect
c. Website
d. SiteSpect
e. User

10. Why change what works?
350 tests
550,000 subtests
Full run takes 7.5 hours
Requires a specific system configuration

11. Part II : Jenkins and the Harness Lab

12. What is Jenkins?
Distributes and manages external jobs
Works across multiple machines
1000+ plugins, good community
Runs in JRE, uses Groovy for internal scripting

13. Jenkins Components/Terms
● Jobs
o Series of steps, each of which can fail or succeed
o shell scripts, groovy scripts, email notifications,
start other jobs, etc.
● Build
o A single invocation of a given job
● Downstream/Upstream
o Jobs & builds that are called by another jobs or that call
another are downstream or upstream respectively

14. Summary of Harness Lab
1 master VM, 24+ worker nodes (~75 VMs)
Master hosts:
Jenkins’ Control Panel
MySQL
Custom Test Results Viewers
Worker hosts:
SiteSpect, Harness, Jenkins worker Agent

15. Diagram of Jenkins
Jenkins Workers
24 Harness Clusters
SVN
Are there new commits?
Configure Machines, Distribute Tests
Test Results to MySQL, Test Result TAP

16. General Outline of How it All Works
1. Find SiteSpect version and Harness version
2. Configures a single worker
3. Configures the rest of the workers
a. Cannot configure multiple boxes in parallel without
some customizations
b. Matrix Project Plugin—It’s a TRAP!
c. NodeLabel Parameter Plugin

17. General Outline of How it All Works
5. Finds list of all tests in Harness branch
a. Automatically creates Jobs for new tests
6. Queues all the tests
7. Adds results to the database
8. Creates a result page
9. Emails the responsible developer

18. Other Uses for Jenkins
● Building and testing our GeoIP and
Mobile Device databases (WURFL)
● iOS Testing
o OS X worker node
o appium testing with iOS simulator
● VM setup tool

19. Part III: Performance
At first, performance was a nightmare!
We could not:
Load Control Panel
Display Test Results
Go a single day without a reboot

20. Performance Issues and Solutions
Culprits
Heap size/Machine Power
A runaway restart worker call
Front-end issues
Using the file system

21. Views

22. Build History

23. Aggregating Downstream Results

24. Test Results Searching
Harness API Viewer - Hackathon Project
Searching test results DB
See relationships between runs that are not
tracked by Jenkins
Much faster than Jenkins

25. Avoid the Jenkins UI
For us:
The Jenkins UI is slow
It doesn’t look good
It hurts performance
Still popular, and they are redesigning UI

26. Part IV : Conclusions

27. Takeaways
Parallelizing your tests makes everyone happy
Now takes 30 minutes instead of 8 hours
Jenkins is powerful and flexible
It is a very quirky program
Always check the plugins

28. List of Useful Plugins
Parameterized Trigger
Groovy/Groovy Post-Build
Purge Build Queue
JobConfigHistory
Priority Sorter
Build Timeout
Build User Vars

29. Conclusions
Thank you!
Questions?
Answers?