Recommended
More Related Content
What's hot
What's hot (20)
Similar to Advanced automated visual testing at DrupalCon Europe 2020
Similar to Advanced automated visual testing at DrupalCon Europe 2020 (20)
Recently uploaded
Recently uploaded (20)
Advanced automated visual testing at DrupalCon Europe 2020
- 1. 1@shwetasharma84 Advanced Automated Visual Regression Testing — Shweta Sharma
- 2. 2@shwetasharma84 @shwetasharma84 @shwetaneelsharma @shwetaneelsharma @shweta_ns Connect with me Shweta Sharma Director of QA Services India, Pune
- 3. 3@shwetasharma84 Fully-distributed, 6 Time Zones 10 Years Of Growth 120+ Enthusiastic, Kind, Open Professionals 100+ Enterprise Engagements Qualitative Drupal Contributor Axelerant from above
- 4. 4@shwetasharma84 01 What is Automated Visual testing? 02 How does it work? 03 Level - 1 (Objectives, Tools, demo, challenges-resolutions) 04 Level - 2 (Objectives, Tools, demo, challenges-resolutions) 05 Level - 3 (Objectives, Tools, demo, challenges-resolutions) Agenda 06 Comparison between tools 07 Testing Strategy 08 Good practices 09 Key results achieved 10 Facts 11 Where to go from here? 12 References
- 5. 5@shwetasharma84 What is Automated Visual testing? Validate the following: ● Visual content ● Page layout ● Responsive design of Web App
- 6. 6@shwetasharma84 How does it work? Test Runner Capture screenshots at run time Test passes Update baselineBug Compare Capture baseline images First run Subsequent runs unintended intended Fails and report differences
- 7. 7@shwetasharma84 ● Site development in progress/maintenance ● Optimise testing efforts as an individual ● Resolve challenges faced at individual level LEVEL 1 - Beginner
- 8. 8@shwetasharma84 BackstopJS - viewports and scenario defined (element)
- 9. 9@shwetasharma84 BackstopJS - viewports and scenario defined (full page)
- 10. 10@shwetasharma84 BackstopJS Demo - Create reference images
- 12. 12@shwetasharma84 BackstopJS Demo - Execute Tests
- 13. 13@shwetasharma84 Challenge 1 - Dynamic content
- 16. 16@shwetasharma84 How few tools handle some challenges
- 18. 18@shwetasharma84 New reference image using hide strategy
- 20. 20@shwetasharma84 GET STARTED - TOOLS AVAILABLE Open-source BackstopJS, Wraith, Spectre Commercial Applitools SIDE (Selenium IDE) extension, Percy, Diffy
- 21. 21@shwetasharma84 LEVEL 2 - OBJECTIVES and CHALLENGES OBJECTIVES CHALLENGES Involve more than 1 individual so that more people benefit from automation Baseline images created from one machine fail on others due to anti-aliasing Simulating user actions like filling up forms, clicking, hovering etc. Seamless integration with existing testing frameworks Optimise testing efforts for a small development/testing team Harmonious integration with your CI setup
- 22. 22@shwetasharma84 Resolution for Level 2 challenges ● Anti-aliasing ○ Use of Docker so that the environment is the same every time ○ Run visual tests on cloud ■ BrowserStack ■ SauceLabs ■ AWS Device Farms ■ Cross Browser testing ■ Select a tool which handles it implicitly ● Integration with existing testing frameworks ○ Open source and commercial tools available ● Integration with the CI setup ○ Commercial tools available
- 23. 23@shwetasharma84 TOOLS AVAILABLE Open-source Several visual testing plugins by Cypress Commercial Applitools, Percy
- 24. 24@shwetasharma84 Page to be tested
- 25. 25@shwetasharma84 Cypress & Percy - The actual test
- 26. 26@shwetasharma84 Cypress + Percy - Create baseline images
- 27. 27@shwetasharma84 Cypress + Percy - Approve baseline images
- 28. 28@shwetasharma84 Cypress + Percy - Bug introduced in feature branch
- 29. 29@shwetasharma84 LEVEL 3 - OBJECTIVES and CHALLENGES CHALLENGES CAUSE OBJECTIVES Running into false positives ● Small vertical/horizontal shifts ● Browser updates Avoid pixel-to-pixel matching Increased time test on several browsers & devices Execution happens on all configured browsers/devices Better Test coverage for visual tests at a faster pace Frequently/Major change in elements Change requests Better baseline management Scattered tests, poor readability Inability to group Better Test Suite management
- 30. 30@shwetasharma84 TOOLS AVAILABLE Open-source None Commercial Applitools, Percy (to some extent)
- 31. 31@shwetasharma84 FALSE POSITIVES - Resolution PERCY APPLITOOLS It ignores 1px difference It uses AI-based comparison algorithm
- 32. 32@shwetasharma84 BROWSERS/DEVICES & SCATTERED TESTS - Resolution ● APPLITOOLS ○ Supports all browsers & devices ○ Executes once and captures everywhere in background in parallel ○ Create relevant Batches and group tests ● PERCY ○ Supports only Chrome & FF ○ Supports only viewports
- 33. 33@shwetasharma84 Comparison between tools (Must have) PARAMETER PERCY APPLITOOLS Pixel-to-pixel matching ✅ ✅ AI-based matching ❌ ✅ Browsers Chrome and Firefox All Devices Only Viewports All Dynamic content handling ✅ ✅ Baseline management ❌ ✅ Team management ✅ ✅
- 34. 34@shwetasharma84 Comparison between tools (Need basis) PARAMETER PERCY APPLITOOLS Root cause analysis ❌ ✅ Integration tools Source control, build servers, collaboration Source control, build servers, collaboration SDK support Good Excellent On-premise support ❌ ✅ Collaborative dashboard ❌ ✅ Desktop, mobile app support ❌ ✅
- 36. 36@shwetasharma84 Testing Strategy ● Capturing screenshots ○ Element level ■ Component based libraries ● Storybook ● Pattern Lab ■ Development in progress ● Group the components logically (For e.g. Header, Footer etc.) ○ Full page ■ Comparing two similar environments ■ Entire page is developed
- 37. 37@shwetasharma84 Testing Strategy contd... ● Plan the level of visual coverage needed ○ Both for browsers and devices ■ Every project is different as our customers are different ■ Identify patterns from previously learned lessons ○ Application wide ■ Don’t capture all pages ■ Identify appropriate samples
- 38. 38@shwetasharma84 Good practices to follow ● Organize Test Suites ○ Browser wise ○ Feature wise ○ Device wise ● Speed up test execution by running tests in parallel ● Identify the frequency to run visual tests
- 39. 39@shwetasharma84 Organizing tests as per browser
- 41. 41@shwetasharma84 Key results achieved at Axelerant ● Test data ○ Time saved to handle dynamic content scenarios because of data consistency ○ Minimal efforts for Visual tests ● Seamless integration of Automation tools involved ● Visual tests validated for the site which is a replica of the wireframe ● Less number of assertions required for the Acceptance test Automation suite
- 42. 42@shwetasharma84 Facts ● Efforts needed to maintain baseline images ● Respect the Test Pyramid ○ Include Unit tests, Service tests, Acceptance tests and Visual tests ● Best practices in Automating Visual tests: ○ Ensure full page validation ○ Do not have too many element specific tests ■ Avoid element locators maintenance ○ General rules for automating Acceptance tests apply here too: ■ Pick up the right candidates for Automation ■ Logical division of tests ■ Usage of Seams ● Do not expect overnight success
- 43. 43@shwetasharma84 Where to go from here? ● No coding knowledge? ○ Use configuration based tools like BackstopJS, Wraith, Applitools SIDE browser extension ○ Pair up with developers, SDETs to have the structure ready ● Good with coding? ○ Look for harmonious integration with your automated functional suite ■ Reduce the unnecessary functional assertions ● Maintainable suite ● Better execution time ○ Be brave and bring the tests in your CI pipeline ■ Let the whole team benefit
- 44. 44@shwetasharma84 References ● BackstopJS ● WebdriverIO - Visual Regression Service ● Applitools for Selenium IDE Tutorial ● Visual Regression Testing Using Wraith ● Run Selenium tests with WebDriverIO ● Cypress visual testing plugins ● Diffy ● Spectre ● Applitools Eyes ● Applitools Ultrafast Test Cloud ● Percy - Getting started