In agile software development world, we are dealing with many test tasks such as user story testing, exploratory testing, check-list based testing, regression testing, performance testing, security testing in each sprint. Besides these testing activities, one of the test types which is considerably getting crucial is visual regression testing. Visual regression testing focuses on to check visual contents and animations, page layout, and responsive design of a website/app. Because of the limits of human vision, human-based visual regression testing is generally error-prone and cumbersome. Hence, automation is inevitable. It enables us to run the tests much more precisely in a short time period. Also, it saves us a significant amount of time to deal with other manual test activities in each sprint. In this talk, we will walk through well-known open-source and commercial solutions for visual test automation. We will learn which technologies they use, what type of visual tests they are suitable for, and their major differences between each other. Besides this overview, we will also make a real-life visual test automation demo by using Selenium, ImageMagick, and AShot.

2. COLORFUL WORLD
of
VISUAL TEST AUTOMATION
Onur Başkırt

3. Who am I?
▪ I am Onur Başkırt
▪ Head of Software Testing at
▪ Co-owner of
▪ Formerly: Teknosa, Ericsson, Huawei, STMicroelectronics, BAU & USCS

4. Online Employment and Recruitment Market Leader
in TURKEY since 1999
5.6 million
Visitors
180 million
Page views
25 million
Resumes
75.000+
Employers

5. Technical Software Testing Blog
Automation Performance Security DevOps

6. Outline
• What is Visual Test Automation?
• Why Visual Testing?
• Why Automation?
• Challenges
• Visual Test Automation Tools
• Brief Overviews
• Detailed Reviews (Applitools Eyes, Percy.io, Galen Framework)
• Selenium & ImageMagick & AShot Example
• Q&A

7. What is Visual Test Automation?
• Verifies the GUI is displayed as expected to the end users.
• Focuses on visual contents:
• Colors
• Shapes
• Sizes
• Locations of elements
UI Test vs Visual Test

8. Why do We do Visual Testing?
WordPress
Template
CSS
animations
Widgets Sliders etc.
Mobile App
Generation
Engine
.ipa
.apk
GUI MUST APPEAR CORRECTLY
USER INTEARCTIONS
COMPANY BRAND
CRITICAL PRODUCTS and SERVICES

9. Why do We Automate Visual Testing?
Variety of Test Environments
• Browsers – Mobile Devices – Resolutions etc.
Time to Market Pressure
• Release cycles getting shorter.
Manual Visual Testing is Cumbersome
• Human eyes error-prone and getting blind.
In Mobile World, you will live with visual bugs longer
• Can not update mobile app daily.

10. What are the Challenges?
• Anti-aliasing

11. What are the Challenges?
• Dynamic Content (CSS Animations, GIFs)

12. What are the Challenges?
Dynamic content
Dynamic Content
Auto-Kerning Offsets

13. BRIEF OVERVIEW of
Well-Known
VISUAL AUTOMATION TOOLS

14. PhantomCSS (Open-Source by Huddle, 4197+ Stars on GitHub)
• PhantomJS (Webkit) and SlimerJS (Gecko)Browsers
• CasperJSNavigation
• ResembleJSComparison
• Ignores Colors, Antialiasing, Moving
Elements
Features

15. WebdriverCSS (Open-Source, 447+ Stars on GitHub)
• Chrome, Firefox, PhantomJSBrowsers
• WebdriverIO v2.0
Selenium Bindings for NodeJS
Navigation
• GraphicsMagickComparison
• Ignores Color, AntialiasingFeatures

16. Wraith (Open-Source by BBC, 3883+ Stars on GitHub)
• PhantomJS, SlimerJSBrowsers
• Configuration Based .yaml file
• CasperJS
Navigation
• ImageMagickComparison
• Fuzz Option
• Antialiasing and small changes
Features

17. • and many tools exists in the market both open-source and commercial.
• FBSnapshotTestCase
• Needle
• Rspec Page Regression
• Pix-Diff
• Selenium Visual Diff
• Vizregress
• VisualCeption
• Specter
• Fighting Layout Bugs
• dpdxt
• Huxley
• SiteEffect
• Shoov
• Screener.io
• BaskstopJS
• CSSCritic
• Grunt Photobox
• VIFF
• GreenOnion
• Kobold
• CSS Visual TestAsd
• Snap And Compare
• Grunt-Vigo
• Hardy
• Browsershots
• Screenbeacon
• VisualReview
• Gemini

18. APPLITOOLS EYES

19. What is Applitools Eyes?
• Automated Visual Testing Tool for Web sites and Mobile apps
• Finds Visual Differences with Several Match Levels
• Eyes SDK for Selenium & Appium
• Cloud-Based Clean and Informative Dashboard

20. How Does it Work?
Visually notifies the differences
At 2nd run and more it compares actual screenshots with the baselines.
At 1st run it saves all screenshots as baseline
Takes screenshots with Eyes SDK and sends to the Applitools cloud

21. Match Levels and Their Meanings?
• Pixel by Pixel ComparisonEXACT
• Mimics the human eyes.
• Ignores trivial mismatches (antialiasing, small pixel movements, etc.)
STRICT
• Similar with STRICT plus it ignores color changes.CONTENT
• Compares the layouts.
• Checks the elements locations.
LAYOUT

22. 4 Main Steps of Eyes SDK
Close the Test
eyes.close();
Add UI validation Check Points
eyes.checkWindow("Validation Window Name");
Start a Test
eyes.open(driver, APP_NAME, TEST_NAME);
Create and Configure an Eyes Object
Eyes eyes = new Eyes();

23. Mobile Visual Testing Example
Open BitbarSampleApp.apk
* Visually check that start screen as expected?
Click second option “Use Testdroid Cloud”
*Visually check that Second Option is clicked?
Write “SW Test Academy” to text bar
Click Answer button
Wait until the second page appear
*Visually check that the answer is correct?
Close the driver.

24. Mobile Visual Testing Example
Step-1: Install Eyes SDK with Maven Dependency
<dependency>
<groupId>com.applitools</groupId>
<artifactId>eyes-selenium-java</artifactId>
<version>RELEASE</version>
</dependency>
Step-2: Do Appium, Android SDK and GenyMotion Settings

25. Mobile Visual Testing Example
Step-3: Configure Eyes SDK in your Setup Method

26. Mobile Visual Testing Example
Step-4: Open Eyes  Do Visual Check  Close Eyes
Open Eyes!
Perform a Visual Check!
Close Eyes!
Abort if Eyes not closed!

27. Mobile Visual Testing Example
Step-5: Run and Check Results – First RUN (BASELINES)

28. Mobile Visual Testing Example
Step-5: Run and Check Results – PASSED RESULTS

29. Mobile Visual Testing Example
Step-5: Run and Check Results – FAILED RESULTS

30. Applitools Settings

31. GALEN FRAMEWORK

32. What is Galen Framework?
Automated LAYOUT and RESPONSIVE testing framework.
• Runs in Selenium Grid (SauceLabs & BrowserStack)
• Supports Parallel Testing

33. Programming Languages
• Basic Syntax
• JavaScript
• JAVA & Selenium Webdriver & TestNG/Junit 

34. Galen Reporting
• Error Reporting
Generates HTML reports where you can see
all your test objects
• Screenshots
Highlights the misaligned elements
• Image Comparison
Compares images and shows differences

35. Galen Example with JAVA & Webdriver & TestNG
Logo: 31 pixel
below from the
header
Header: 0px to
the right, left, top
Navigation Links:
Horizontally aligned.
Logo should displayed
correctly.

36. Galen Example with JAVA & Webdriver & TestNG
Step-1: Installation with Maven Dependency
<dependency>
<groupId>com.galenframework</groupId>
<artifactId>galen-java-support</artifactId>
<version>2.3.2</version>
</dependency>

37. Galen Example with JAVA & Webdriver & TestNG
Step-2: Write a Galen Specifications (.gspec)
web elementsfor desktop
Declaring locations
tagging
Image Comparison
loops
5

38. Galen Example with JAVA & Webdriver & TestNG
Step-3: Write your Test Code
Get Layout Report
Get Galen Test
Information from
Layout Report
Generate HTML Report based on
Galen Test Information

39. Galen Example with JAVA & Webdriver & TestNG
Step-4: Folder Structure

40. Galen Example with JAVA & Webdriver & TestNG
Step-5: Run the Test and Check The Results
Galen Test Report

41. Galen Example with JAVA & Webdriver & TestNG
Homepage Layout Report

42. Galen Example with JAVA & Webdriver & TestNG
LAYOUT ERRORS

43. Galen Example with JAVA & Webdriver & TestNG
HEATMAP

44. Galen Example with JAVA & Webdriver & TestNG
IMAGE COMPARISON ERRORS

46. What is Percy?
Visual regression testing and review platform.
• COMPARES images, FINDS differences.
• We can REVIEW and APPROVE changes.
• Supports RESPONSIVE Design testing.
• Prevents FALSE-POSITIVES.
• PARALLEL TESTING support.
• Integrated with GitHub and many CI tools.

47. Which Technologies Does it Support?
CI Tool Integrations Languages/Clients
Travis CI Ember
CircleCI Ruby – Capybara
Semaphore Python – Selenium (BETA)
Codeship Static websites
Buildkite
Drone
Jenkins

48. How Does it Work?
When commit any code changes.

49. Installation Ember Percy
Requirements for Ember Percy with Jenkins and GitHub
git node.js ember.js
ember-
percy
Phantom.js bower Jenkins GitHub

50. Percy Installation Steps (Percy Side)
• Go to Percy.io and click Sign In WITH GITHUB.
• Go to Percy Dashboard and Create an Organization
• Install GitHub Integration
• Create a project and get your CI integration variables:
PERCY_TOKEN & PERCY_PROJECT

51. Percy Installation Steps (Jenkins Side)
• Set your PERCY_TOKEN and PERCY_PROJECT environment variables
• Do GitHub Integration settings and Create a Jenkins Job.
• Do your Ember Project’s Repository settings.
• Add build and test Windows Batch commands:
• For BUILD:
• For TEST:
npm install && bower install
npm test

52. How to Write Test with Ember-Percy?
percySnapshot(name, options);
scope  For element to snapshots. Default is full page.
percySnapshot('homepage', {scope: '#header'});
Takes ScreenShots

53. Sample Test Code Official TODO Example of Percy
Navigate the homepage
Take ScreenShot
Write sth to the bar
and
Press Enter Assertions
Take Final ScreenShot

54. Responsive Design Test Code
Define Resolutions
by Screen Widths

55. First Run  No Comparison (BASELINES)

56. Second Run  PASSED!

57. Third Run  FAILED!

58. Visual Test Automation by Using
ImageMagick & Selenium & AShot

59. What is ImageMagick?
• Open-source software suite that manipulates
images in a variety of formats (over 200) including
PNG, JPEG, GIF, etc.
• Edits, Converts, Resizes, Compares, Rotates,
Adjusts Colors etc.
• For JAVA it has Im4java which is a pure-java
interface.

60. What is AShot?
• Yandex's screenshot utility which takes, crops, prettifies and
compares screenshots.
• Entire Page
• Web Element

61. Example Test Scenario (Basic CSS Animation Check)
• Open www.kariyer.net
• Unhide text area of Uzman Photo
• Hover on the “UZMAN” weblement on main page.
• Wait 2 seconds to finish CSS animation
• Take screenshot of an element and save as Baseline
for the 1st Run.
• For 2nd and more runs COMPARE baseline image
with actual image.
• Put all differences in "Differences folder"
HOVER
BASELINE

62. Example Test Scenario (Basic CSS Animation Check)
• Step-1: Install ImageMagick with all options and add to system path.

63. Example Test Scenario (Basic CSS Animation Check)
• Step-2: Create a Selenium & TestNG project and add dependencies

64. Example Test Scenario (Basic CSS Animation Check)
Test Method’s Code
AShot
ImageMagick
JavascriptExecutor

65. Example Test Scenario (Basic CSS Animation Check)
How to Take Screenshot with AShot

66. Example Test Scenario (Basic CSS Animation Check)
If Baseline Exists, Do Comparison!

67. ImageMagick Comparison
ImageMagick
Configurations
Add images to ImageMagick
Operation object for comparison
Do Comparison!

68. First Run
Test ScreenShot Folder is created
by using Test Name

69. Second Run

70. Change The Baseline
Test Folder

71. Third Run!
Arithmetic Error
Number
DifferencesFolder
TestFolder
TestFailed!

72. THANKS  QUESTIONS?
onurb@kariyer.net
@swtestacademy
@onurbaskirt