Slides presented at Selenium Conference India 2016
Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.
If you don’t know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!
5. WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical
User Interface appears correctly to users
6.
7.
8.
9.
10. WHY AUTOMATE?
The test matrix is too big to cover manually
Web browsers
Devices
Operating systems
Screen resolutions
Responsive design
L10n
3rd Party upgrades
11. WHY AUTOMATE?
Mobile apps quality is critical
Harder to roll back changes
Can’t push daily
Updates take battery and data
Higher quality bar
27. ImageMagick
A powerful command line tool for image processing.
Fuzzing is used to eliminate slight color differences
An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
28. THE JAVASCRIPT ENGINES
Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
Blink-Diff
Pixel by pixel + error ratio + anti-aliasing + perceptual color distance
computation
https://github.com/yahoo/blink-diff
Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same
29. APPLITOOLS EYES
Handles anti-aliasing, pixel offsets, color similarity, and image scaling
Dynamic and moving content
Compare images of different sizes
No error ratio configuration required
Validates full UI pages
Fast!
Supports layout / structure matching
31. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
32. QUICK FEEDBACK TOOLS
MOTIVATION
Get fast feedback on code changes
Run tests locally in the background
SETUP
Render screenshots with a headless
browser
PhatomJS, SlimerJS
Configuration file driven tests
CONS
Partial coverage
Chrome, IE?
Old browser layout engine versions
Limited navigation
33. VISUAL COVERAGE TOOLS
MOTIVATION
Verify that the app / site looks right in
all real execution environments
Test as many UI states as possible
Execute many tests in parallel
SETUP
Render screenshots using real browsers
on a variety of operating systems and
devices in parallel
WebDriver, Grid
Code driven tests
WebDriver, DSLCONS
Requires test infrastructure
Implement and maintain test code
42. AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
43. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
Visual testing of frontend components by
frontend developers
44. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
Full or partial validation of application screens
A collaboration tool
45. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
Validate your staging deployment using your
production deployment as a baseline
46. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
No missing resources in production
No corruption due to 3rd party data
No corruption due to browser / OS upgrades