Trainee project about Jenkins and visual regression testing.

1. Jenkins
and Visual
Regression
Mikko Nirkko, 2019

2. Table of Contents
● What is Jenkins
● What is visual regression testing
● Case study - Screenshot comparison software
in Jenkins

3. ● Continuous integration
Server
● The main goal is to
automate all kinds of
software builds
● Can be used for
○ Continuous Integration (CI)
○ Continuous Delivery
○ Continuous Deployment
○ Automation
○ Testing
○ … Among many other things!
What is
Jenkins?

4. What Can Jenkins
Do?

5. ● Automate software builds
and delivery
○ Timed builds
○ Triggered builds
○ Build across different
platforms
● Flexible notifications
when builds fail
○ Email and Slack mostly used
● Extendable with optional
plug-ins to further increase
the number of supported
technologies

6. What is
Visual
Regression
Testing?

7. Visual
regression
● Search for differences between
two or more images of a
common target
● In this case, compare screenshots
of a website between builds
● When new updates or other
changes cause unwanted visual
effects, this means image
regression has happened!

8. Case Study:
Visual Regression
Testing in Jenkins

9. ● The main concept was to
build a proof-of-concept
version of a screenshot
comparison software,
running entirely within
Jenkins
● It was designed to be used
by Exove staff, as a simple
internal tool
● Since the application
operates on (confidential)
client sites, the application
use is restricted to Exove
employees only.
Visual
Regression
Testing Tool

10. How it works
1. Collect screenshots from a website of choice
2. Save and index the screenshots to a gallery
3. Run the same test again, after a site update
(or whenever)
4. This collects a second, identically formed
gallery
5. Compare the galleries against each other
6. If notable differences are found, create a
detailed report of the findings

11. Workflow
1. Choose
“Build with
parameters”
from the project page
2. Enter Site Url and
press “Build now”
(Other parameters are
optional)
3. Wait until testing
is
done.
Results appear at the
project front page
That’s It!

12. ● Robot Framework
○ for crawling websites and capturing the screenshots
○ Extended with my own Python library and BeautifulSoup4 for
processing HTML reports
○ With ImageMagick for screenshot processing and comparison
● Jenkins for:
○ Hosting
○ Screenshot storage
○ Test execution
○ Reporting the test results
Technologies Used

13. PROS & CONS
● Very easy to use, even
without technical knowledge
● In most cases, the only
required input from the user
is the URL of the site they
want to test
● No local installations or files
needed
● Multiple ways of viewing and
distributing test results
● Current proof-of-concept
contains primitive methods
of difference recognition
● This results in many false
positives with things like
updated text and image
content
● Somewhat slow to execute