Session presented by Shweta Sharma of Axelerant on Advanced Automated Visual testing at DrupalCon Seattle 2019

1. 1axelerant.com@shwetasharma84
Advanced Automated
Visual Testing
— Shweta Sharma
@shwetasharma84

2. 2axelerant.com@shwetasharma84
About Me
Shweta Sharma
Quality Assurance Lead
India, Pune
@shwetasharma84
@shwetaneelsharma
@shwetaneelsharma
@shweta_ns

3. 3axelerant.com@shwetasharma84
Axelerant from above
Fully-distributed, 6 Time Zones
13 Years Of Growth
60+ Enthusiastic, Kind,
Open Professionals
100+ Partner Engagements
1,000+ Open Source
Contributions
@shwetasharma84

4. 4axelerant.com@shwetasharma84
Agenda
● What is Automated Visual testing?
● Why do we need it?
● How does it work?
● When to use what?
● Challenges with Automated Visual testing
● How WebdriverIO / Applitools addresses these challenges
● Reports in WebdriverIO
● Demo

5. 5axelerant.com@shwetasharma84
What is Automated Visual testing?
● Validate the following:
○ Visual content
○ Page layout
○ Responsive design of Web App

6. 6axelerant.com@shwetasharma84
Spot the difference!

7. 7axelerant.com@shwetasharma84
Spot the difference by Tool

8. 8axelerant.com@shwetasharma84
Why do we need it?
Human Factor
Larger Device/OS Matrix
Larger Release Cycles
Invest time to QA new
features

9. 9axelerant.com@shwetasharma84
How does it work?
Test Runner
Capture screenshots at run time
Report differences
Update baselineBug
Compare
Capture baseline images
First run
Subsequent runs

10. 10axelerant.com@shwetasharma84
Challenges
Anti Aliasing Dynamic Content

11. 11axelerant.com@shwetasharma84
WebdriverIO
● Open source
● Support for BDD
● Good community support
● Support for multiple test framework (Mocha, Jasmine, Cucumber)
● Corresponding support for assertion libraries (Chai for Mocha)
● Wide CLI options
● Page Object framework
● Multiple reporting options
● Multiple ready to use services (Visual regression service)
● Cloud service support (Browserstack, Sauce Labs or TestingBot)

12. 12axelerant.com@shwetasharma84
How WebdriverIO handles few challenges
Visual regression service by WebdriverIO
Hide
● Hide all elements queried by
WebdriverIO selector strategies
● Element’s space is filled and
considered in the layout of the
screenshot captured
● Good candidates - Ad blocks, Images
that change
Remove
● Remove all elements queried by
WebdriverIO selector strategies
● Element is removed from the DOM -
affects the design of the page
● Good candidates - Sticky
headers/footers, popover help/chat
windows
NOTE: Visual regression service is available till version 4.*. It is replaced by Applitools service from 5.*

13. 13axelerant.com@shwetasharma84
How WebdriverIO handles few challenges

14. 14axelerant.com@shwetasharma84
Organize Test Suites
● Feature wise
● Browser wise
● Device wise

15. 15axelerant.com@shwetasharma84
WebdriverIO Reports
● Dot (default)
● Spec
● Junit (Sophisticated reports for CI server)
● Allure
● Teamcity
● Concise
● Tap ()
● Mochawesome (creates a json output consumed by marge)
● Slick (local development)

16. 16axelerant.com@shwetasharma84
Demonstration
● Speed up tests - Parallel execution on multiple browsers
● Tests using below frameworks:
○ Mocha
○ Jasmine
● Organize Test Suites
● Run tests on browserstack
○ WebdriverIO has out of the box support
○ Creates a tunnel between local machine and browserstack (browserstack.Local: true)
○ Supports multiple browsers with different browser/OS versions
○ Supports multiple real devices (needs Mobile App plan activation)

17. 17axelerant.com@shwetasharma84
Applitools
● 40+ SDKs available free to use
○ Different programming languages
○ Test Runners
○ Web/Mobile apps
○ Quick to extend your current tests to include Visual Validation
● Capture screenshot commands
○ checkWindow - Entire browser/window
○ checkElement - Particular element
○ checkFrame - Particular frame
○ checkRegion - To handle dynamic and shifting content
● Handles dynamic content which is also floating/shifting
● In-built support for anti-aliasing

18. 18axelerant.com@shwetasharma84
Match Levels in Applitools
Strict
● Recommended
● Content
● Font
● Layout
● Color
● Position of elements
Layout
● Tests the layout
● Validates the alignment
and relative position of
all elements on the page,
such as: buttons, menus,
text areas, paragraphs,
images, and columns
● Ignores the content,
colour and other style
changes between the
pages
Content
● Similar to Strict
● Compares content but
ignores colors

19. 19axelerant.com@shwetasharma84
Applitools
● Cool dashboard with AI support
● Applitools SIDE extension (Chrome and Firefox)
● Integrations
● Free Support for open source projects

20. 20axelerant.com@shwetasharma84
Applitools - Demonstration

21. 21axelerant.com@shwetasharma84
How we integrated in our development workflow?
BUILD
CI PIPELINE
COMMIT
CODE
ACCEPTANCE
TESTING
VISUAL
REGRESSION
DATA
SEEDING
CD PIPELINE
REVIEW
STAGING
PRODUCTION

22. 22axelerant.com@shwetasharma84
Advantages of Gold Master testing
● Test data
○ Time saved to handle dynamic content scenarios because of data consistency
○ Minimal efforts for Visual tests
● Seamless integration of Automation tools involved
● Visual tests validated for the site which is a replica of the wireframe
● Less number of assertions required for the Acceptance test Automation suite

23. 23axelerant.com@shwetasharma84
Facts
● Efforts needed to maintain baseline images
● Respect the Test Pyramid
○ Include Unit tests, Service tests, Acceptance tests and Visual tests
● Best practices in Automating Visual tests:
○ Ensure full page validation
○ Do not have too many element specific tests
■ Avoid element locators maintenance
○ General rules for automating Acceptance tests apply here too:
■ Pick up the right candidates for Automation
■ Logical division of tests
■ Usage of Seams
● Do not expect overnight success

24. 24axelerant.com@shwetasharma84
When to use what?
Static content
● Wraith
● PhantomCSS
● Spectre
Shifting Content
● Applitools (SDKs and
Selenium IDE extension)
Dynamic content
● WebdriverIO Visual
regression service
● Shoov
● BackstopJS
● WebdriverCSS
● Gemini

25. 25axelerant.com@shwetasharma84
References
● http://v4.webdriver.io/guide.html
● http://v4.webdriver.io/guide/services/visual-regression.html
● https://applitools.com/tutorials/selenium-ide.html
● https://www.axelerant.com/resources/team-blog/visual-regression-testing-u
sing-wraith
● https://www.browserstack.com/automate/webdriverio

26. 26axelerant.com@shwetasharma84
Join us for contribution opportunities
Mentored Contribution
9:00-18:00
Room: 602
General Contribution
9:00-18:00
Room: 6A
First time Contributor
Workshop
9:00-12:00
Room: 606
Friday, April 12, 2019

27. 27axelerant.com@shwetasharma84
What did you think?
Locate this session at the DrupalCon Seattle website:
https://events.drupal.org/seattle2019/schedule/builder/2019-04-11
Take the Survey!
https://www.surveymonkey.com/r/DrupalConSeattle

28. 28axelerant.com@shwetasharma84
Q & A
● Concept of Automated Visual testing
● Algorithm around all tools
● Challenges in Automated Visual
testing
● Challenges handled by WebdriverIO
and Applitools
● Gold Master Testing
Twitter - @shwetasharma84
LinkedIn - @shwetaneelsharma
GitHub - @shwetaneelsharma
Skype - @shweta_ns