This document discusses automating visual software testing. It begins by explaining that visual testing is used to verify a graphical user interface appears correctly. It then discusses why visual testing should be automated, such as the large test matrix of browsers, devices, resolutions etc. that is too big to manually test. It also notes that mobile app quality is critical given limitations on rolling back changes. The document outlines how automated visual testing works by taking screenshots, comparing them to baselines, and reporting differences. It discusses various tools and techniques used for image comparison, visual testing frameworks, and integrating tests into development and deployment pipelines. In general, it promotes automating visual testing for regression testing, localization, accessibility, and more throughout the development and deployment process
4. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
5. Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users
9. Automating visual software testing
WHY AUTOMATE?
The test matrix is too big to cover manually
• Web browsers
• Devices
• Operating systems
• Screen resolutions
• Responsive design
• L10n
• 3rd Party upgrades
10. Automating visual software testing
WHY AUTOMATE?
Mobile apps quality is critical
• Harder to roll back changes
• Can’t push daily
• Updates take battery and data
• Updates are not mandatory
• Higher quality bar
12. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
13. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
15. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
21. Automating visual software testing
AND MORE…
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance
24. Automating visual software testing
IMAGE MAGICK
• A command line tool for bitmap processing
• Fuzzing is used to eliminate slight color differences
• An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
25. Automating visual software testing
THE JAVASCRIPT ENGINES
• Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
• Blink-Diff
Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation
https://github.com/yahoo/blink-diff
• Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same
26. Automating visual software testing
APPLITOOLS EYES
• Handles anti-aliasing, pixel offsets, color similarity, and image scaling
• Dynamic and moving content
• Compare images of different sizes
• No error ratio configuration required
• Validates full UI pages
• Fast!
• Supports layout / structure matching
28. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
29. Automating visual software testing
QUICK FEEDBACK TOOLS
MOTIVATION
• Get fast feedback on code changes
• Run tests locally in the background
CONS
• Partial coverage
• Chrome, IE?
• Limited navigation
• Stale browser versions
SETUP
• Render screenshots with a headless
browser
• PhatomJS, SlimerJS
• Configuration file driven tests
30. Automating visual software testing
VISUAL COVERAGE TOOLS
MOTIVATION
• Verify that the app / site looks right in
all real execution environments
• Test as many UI states as possible
• Execute many tests in parallel
CONS
• Requires test infrastructure
• Implement and maintain test code
SETUP
• Render screenshots using real
browsers on a variety of operating
systems and devices in parallel
• WebDriver, Grid
• Code driven tests
• WebDriver, DSL
32. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
36. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
41. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
42. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Before committing / merging code:
• See how UI changes appear in other execution environments
• Make sure there are no unintended changes
43. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Visual testing of frontend components by frontend developers
44. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Regression, Localization, Accessibility, Cross browser/device, responsive
design testing and more…
All team members can review and maintain
A collaboration tool
45. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Validate your staging deployment using your production
deployment as a baseline
46. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
No missing resources in production
No corruption due to 3rd party data, upgrades, hackers, etc.