This document discusses automated visual testing for mobile apps. It explains that visual testing is a quality assurance activity to verify a graphical user interface appears correctly. Visual bugs are shown as examples. Automated visual testing is recommended because the test matrix is too large to test manually across different devices, browsers, resolutions and more. It also discusses tools for automated visual testing like Applitools Eyes, and where in the development process automated visual testing can be useful such as for components, code reviews, staging vs production, and monitoring.

1. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
ADVANCED AUTOMATED
VISUAL TESTING FOR
MOBILE APPS

2. YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

4. AGENDA
 Why automatedvisualtesting?
 Tools& Technology
 Where does it fit?

5. WHAT IS VISUAL TESTING?
A qualityassurance activity aimed to verify that a
Graphical User Interface appears correctly to users

6. A VISUAL BUG

7. AND ANOTHER…

8. WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO
APPROXIMATE MANUALLY
 Webbrowsers
 Devices
 Operating systems
 Screen resolutions
 Responsive designs
 L10n

9. WHY SHOULD IT BE AUTOMATED?

10. MANUAL
VISUAL
TESTING
IS
ERROR
PRONE
WHY SHOULD IT BE AUTOMATED?

11. WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS
 Harder to roll back changes
 Can’t pushdaily
 Updates take battery and data
 Higherquality bar

12. WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSS Critc
Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
Eyes
Huxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare

13. AGENDA
 Why automatedvisualtesting?
 Tools & Technology
 Where does it fit?

14. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

15. DEMOhttps://github.com/webdriverio/webdrivercss

16. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

17. DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java) CSS Critic (URLs)Wraith (URLs)
Needle (Python)
Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCase
(XCTest)
Selenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
dpdxt (URLs)
Gemini (JS)
WebdriverCSS
(Appium – coming soon)

18. SCREENSHOTS
 Real browsers?
 Full page?
 Frames?
 Regions?
 Viewport size?
 Page stabilization?
 Page preparation?

19. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

21. WHY NOT?
FALSE POSITIVES

22. ANTI-ALIASING?

23. ANTI ALIASING 1/2

24. ANTI ALIASING 2/2

25. BRIGHTNESS 1/2

26. BRIGHTNESS 2/2

27. SUB PIXEL SHIFTS 1/2

28. SUB PIXEL SHIFTS 2/2

29. SUB PIXEL SHIFTS 1/2

30. SUB PIXEL SHIFTS 2/2

31. ANDMORE…
1 pixeloffsets in elementpositioning
Dynamic content
Movingelements
Images ofdifferentsize

33. Image Comparison APIs

34. ImageMagick
 A powerful command linetool for image processing.
 APIs are available for most programming languages.
 Fuzzing is used to eliminateslight color differences
 An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

35. Resemble.js
 An imagecomparison tool implementedinJavascript
 Used by PhantomCSSand other tools.
 Good antialiasingsupport
 An error ratio is usually used to determinea match
 http://huddle.github.io/Resemble.js/

36. Applitools Eyes
 A specialized imageprocessing stack designed to compare
computergenerated UIimages
 Anti-aliasing
 Partial and full pixel offsets
 Images of different size
 Dynamic content
 Moving elements
 Layout matching
 Available as a cloud service

37. DEMO

38. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

39. REPORT DIFFERENCES
As files on the file system (combined with source control)

40. REPORT DIFFERENCES
As a Gallery (example from Selenium Visual Diff)

41. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

42. UPDATE THE BASELINE
 Rename or commit individualimage files
 Overwrite mode

43. UPDATE THE BASELINE
 GUI (GeminiGUI)
 Automatic maintenance (demo)

44. AGENDA
 Why automatedvisualtesting?
 Tools& Technology
 Where does it fit?

45. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA

46. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others

47. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA

48. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA

49. QUESTIONS?

50. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
THANK YOU