slides for the VodQA talk on how to create a repeatable robust setup for visual testing of web applications using Docker, Xvfb inside a CI/CD setup.

1. VISUAL TESTING OF
WEB APPS IN A
HEADLESS
ENVIRONMENT
Manish Katoch, Software Developer
1

2. THE PREMISES
Team Alpha wants to build a perfect web
application.
“ We want to it be robust, bug free, stellar! The best of
the best. We will provide an unparalleled user
experience!”
2

3. THE STRATEGY
3
all is good until..
“How do we ensure we don’t break
the UI of our product?”
“We have GUI tests, it will ensure all
our functionality is flawless!”
“But what about aesthetics, the look
and feel?”
“Err…”
UI Tests
API/Integration
Tests
Unit Tests

4. VISUAL TESTS SAVES THE DAY
4
1
2
Go to
screen
Take a
snapshot
While there
are more
screens
Save images
1
2
Go to
screen
Take a
snapshot
While there
are more
screens
Compare
images
Identical Differences
First Pass Successive Passes

5. THE SETUP
5
Code
committed
Build
Created
Unit/Integration
tests
Git
CI-CD system
Functional
Tests
Visual
Tests
No display Agents (virtualized or physical)
(Build and Unit tests)
Test agents with displays
(Functional and visual tests)

6. BUT…
6
“The tests are flaky they thought, the tools insufficient!
the identical images won’t match. ”

7. WHY MY TESTS ARE FLAKY?
Pixel Differences
■ pixel per inch (PPI).
■ PPI is defined as number of pixel that a display can project.
● PPI is calculated as
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙 𝑜𝑛 𝑎 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙
𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑠𝑖𝑧𝑒 𝑜𝑓 𝑑𝑖𝑠𝑝𝑙𝑎𝑦 𝑖𝑛 𝑖𝑛𝑐ℎ
7

8. WHY MY TESTS ARE FLAKY? - II
8
For 1024x768 resolution
98.4 PPI
85.33 PPI
75.29 PPI
The floating point arithmetic is
hardware dependent.
It can differ for same
manufacturer too.

9. WHY MY TESTS ARE FLAKY? - III
What if we could make PPI a constant whole
number?
Good idea. Problem: PPI is determined by hardware
display.
Can we program hardware displays?
9

10. WHY MY TESTS ARE FLAKY? - IV
10
Graphics
Processing Unit
• Processes
data
• Determines
what to
display
• Puts on its
RAM called
Frame Buffer.
Frame Buffer
• Part of
modern
GPUs.
• Is essentially
a bitmap of
image to be
displayed.
• Evolved to
also denote
colors via
lookup
tables.
• Creates video
signals using
DACs
Display
• Displays the
video signal
recieved

11. WHY MY TESTS ARE FLAKY? - V
11
The virtual frame buffer: Xvfb
Graphics
Processing Unit
• Processes
data
• Determines
what to
display
• Puts on its
RAM called
Frame Buffer.
Virtual Frame
Buffer
• Lives in
RAM.
• Implements
entire
framebuffer
protocol
• End point to
GPU
Display

12. THE IMPROVED SETUP FOR VISUAL TESTS
12
Start Xvfb with
1024x768
resolution and
100 PPI
Launch
application
(Browser)
Execute your
visual
tests
Report
“Fixed PPI and resolution guarantees that the image comparison will be truthful“

13. A good test suite must be
repeatable and should give the
same result every single time
13

14. TAKING IT A STEP FURTHER
 Snowflake agents.
 repeatability vs flakiness.
“Containerize the test suite!”
14

15. CONTAINERS: DOCKER
Docker: OS level virtualization ( no virtual machines )
 Think of it as layers of different files, unified together into a single
file system view
 This OS like abstraction gives you ability to run your tests in a
sandbox environment.
15
Image courtesy devopscube.com

16. USING DOCKER
 Create your Docker image with all the dependency packages
added to the image
 Share this image on a location accessible to your CI/CD system.
 Run this image, mount your test files into a folder and
execute tests from there.
16
Image courtesy devopscube.com

17. THE FINAL SETUP
17
Fetch Docker
image if not
already present
Checkout code/
Download Build
Start Docker
image
Run Visual TestsReport To CI
Stop Docker
image

18. WHAT DID
TEAM ALPHA
ACHIEVE?
A consistent and headless environment to
run builds and all tests.
They moved to totally virtualized
environment
No more flaky visual tests, very high
repeatability
Full reusability of infrastructure
Savings on specialized test tools
Early feedbacks even on UI specifications
Rave reviews of their product!
18

19. WHO IS TEAM ALPHA?
19
Image courtesy devopscube.com
Team Alpha is not a team of developers or
a team of QAs.

20. THANK YOU