This document discusses automating visual software testing. It begins by explaining that visual testing is used to verify a graphical user interface appears correctly. It then discusses why visual testing should be automated, such as the large test matrix of browsers, devices, resolutions etc. that is too big to manually test. It also notes that mobile app quality is critical given limitations on rolling back changes. The document outlines how automated visual testing works by taking screenshots, comparing them to baselines, and reporting differences. It discusses various tools and techniques used for image comparison, visual testing frameworks, and integrating tests into development and deployment pipelines. In general, it promotes automating visual testing for regression testing, localization, accessibility, and more throughout the development and deployment process

1. Automating visual software testing
Adam Carmi
Co-Founder and CTO
Applitools
INTRODUCTION TO
AUTOMATED VISUAL
TESTING

2. Automating visual software testing
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

3. Automating visual software testing

4. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A

5. Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users

6. Automating visual software testing

7. Automating visual software testing

8. Automating visual software testing

9. Automating visual software testing
WHY AUTOMATE?
The test matrix is too big to cover manually
• Web browsers
• Devices
• Operating systems
• Screen resolutions
• Responsive design
• L10n
• 3rd Party upgrades

10. Automating visual software testing
WHY AUTOMATE?
Mobile apps quality is critical
• Harder to roll back changes
• Can’t push daily
• Updates take battery and data
• Updates are not mandatory
• Higher quality bar

11. Automating visual software testing
WHY AUTOMATE?
Release cycles keep getting shorter

12. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A

13. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

14. Automating visual software testing
DEMOHELLO WORLD

15. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

16. Automating visual software testing

17. Automating visual software testing

18. Automating visual software testing

19. Automating visual software testing

20. Automating visual software testing

21. Automating visual software testing
AND MORE…
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance

22. Automating visual software testing

23. Automating visual software testing
IMAGE COMPARISON ENGINES

24. Automating visual software testing
IMAGE MAGICK
• A command line tool for bitmap processing
• Fuzzing is used to eliminate slight color differences
• An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

25. Automating visual software testing
THE JAVASCRIPT ENGINES
• Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
• Blink-Diff
Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation
https://github.com/yahoo/blink-diff
• Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same

26. Automating visual software testing
APPLITOOLS EYES
• Handles anti-aliasing, pixel offsets, color similarity, and image scaling
• Dynamic and moving content
• Compare images of different sizes
• No error ratio configuration required
• Validates full UI pages
• Fast!
• Supports layout / structure matching

27. Automating visual software testing
DEMOVISUAL LAYOUT MATCHING

28. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

29. Automating visual software testing
QUICK FEEDBACK TOOLS
MOTIVATION
• Get fast feedback on code changes
• Run tests locally in the background
CONS
• Partial coverage
• Chrome, IE?
• Limited navigation
• Stale browser versions
SETUP
• Render screenshots with a headless
browser
• PhatomJS, SlimerJS
• Configuration file driven tests

30. Automating visual software testing
VISUAL COVERAGE TOOLS
MOTIVATION
• Verify that the app / site looks right in
all real execution environments
• Test as many UI states as possible
• Execute many tests in parallel
CONS
• Requires test infrastructure
• Implement and maintain test code
SETUP
• Render screenshots using real
browsers on a variety of operating
systems and devices in parallel
• WebDriver, Grid
• Code driven tests
• WebDriver, DSL

31. Automating visual software testing
SAMPLE TOOLS
CODE / SCRIPT
● Needle (Python WD)
● Gemini (JS DSL)
● Selenium Visual Diff (Java WD)
● VisualCeption (PHP CodeCeption)
● Shoov (JS WebDriverIO)
● Vizregress (.NET WD)
● Rspec Page Regression (Capibara)
● Applitools Eyes (Appium, XCUI, Espresso)
● PhantomCSS (JS CasperJS)
● Specter (JS DSL)
● FBSnapshotTestCase (XCTest)
CONFIGURATION
● Viff
● CSS Visual Test
● Wraith
● Dpdxt
● Grunt PhotoBox
● Snap And Compare
● BackstopJS
● CSSCritic
● Kobold
● DiffCop
● SUCCSS

32. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

33. Automating visual software testing
REPORT DIFFERENCES
As files on the file system

34. Automating visual software testing
REPORT DIFFERENCES
As a gallery

35. Automating visual software testing
REPORT DIFFERENCES
In your CI build report

36. Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

37. Automating visual software testing
UPDATE THE BASELINE
Gemini

38. Automating visual software testing
UI VERSION CONTROL
• Keep CI tests green
• Multiple app versions
• Revert to previous baselines
• A visual change log (documentation)
1 1 1
2
1
2
3 3
2
4
4

39. Automating visual software testing

40. Automating visual software testing
DEMOAUTOMATED MAINTENANCE

41. Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A

42. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Before committing / merging code:
• See how UI changes appear in other execution environments
• Make sure there are no unintended changes

43. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Visual testing of frontend components by frontend developers

44. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Regression, Localization, Accessibility, Cross browser/device, responsive
design testing and more…
All team members can review and maintain
A collaboration tool

45. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Validate your staging deployment using your production
deployment as a baseline

46. Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
No missing resources in production
No corruption due to 3rd party data, upgrades, hackers, etc.

47. Automating visual software testing
Adam Carmi (@carmiadam)
Co-Founder and CTO at Applitools
QUESTIONS?