Slides from hands-on workshop at Agile India 2021. Machine setup instructions are available in the proposal - https://confengine.com/conferences/agile-india-2021/proposal/15552/next-generation-functional-amp-visual-testing-powered-by-ai In this workshop, using numerous examples, we will explore: Why Automated Visual Validation is essential to be part of your Test Strategy How Visual AI increases the coverage of your functional testing, while reducing the code, and increasing stability of your automated tests Potential solutions / options for Automated Visual Testing, with pros & cons of each How an AI-powered tool, Applitools Eyes, can solve this problem Hands-on look at Applitools Visual AI and how to get started using it

1. Next Generation Functional &
Visual Testing Powered by AI
@BagmarAnand
Anand Bagmar
Software Quality Evangelist, Essence Of Testing

2. About Me
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand

3. • Clone this repo:
• https://github.com/anandbagmar/getting-started-with-visualtesting
• Navigate to: https://github.com/anandbagmar
• Search for “getting-started-with-visualtesting”
• Open project in IDE and refresh Maven dependencies
• Signup for free Applitools license:
https://auth.applitools.com/users/register
• Save the API key received in email as an environment variable –
APPLITOOLS_API_KEY
@BagmarAnand
Before we start – Machine setup

4. Game time!
@BagmarAnand

5. How do you test ….
@BagmarAnand

6. @BagmarAnand

7. http://littlecoffeefox.com/wp-content/uploads/2017/05/DSCN1738.jpg
@BagmarAnand

8. @BagmarAnand

9. @BagmarAnand
http://www.rfwireless-world.com/images/telecom-billing-system-architecture.jpg

10. What does this mean?
@BagmarAnand

11. What is missing here?
@BagmarAnand

12. How do you ensure what
was working before, is
working now as well!
@BagmarAnand

13. Spot the difference!
@BagmarAnand

14. @BagmarAnand

15. @BagmarAnand

16. @BagmarAnand

17. @BagmarAnand

18. It happens in Software too!
@BagmarAnand
https://cdn.pixabay.com/photo/2013/07/12/15/58/smiley-150663_1280.png
Surprise

19. @BagmarAnand

20. @BagmarAnand

21. @BagmarAnand

22. @BagmarAnand

23. @BagmarAnand

24. @BagmarAnand

25. @BagmarAnand

26. @BagmarAnand

27. UX / Visual Testing
The Missing Piece
@BagmarAnand

28. Test Automation Pyramid
Unit (xUnit / JavaScript)
Component
Integration
View
API Workflow
UI
Manual / Exploratory
@BagmarAnand

29. Visual Testing
How do you do it?
@BagmarAnand

30. • Mostly done manually done
• Functional Automation can help
@BagmarAnand
Visual Testing

31. • Business / Revenue loss
• Brand / Credibility loss
• You loose your users!
@BagmarAnand
If Visual Testing is NOT done …..

32. • Mostly done manually done
• tedious
• very error prone
• impossible to scale and repeat – especially in days of CD!
• Functional Automation may help – a little
@BagmarAnand
Challenges of Visual Testing

33. Solution
Visual Test Automation
@BagmarAnand

34. • Create baselines by taking screenshots of expected UI
• Compare screenshots
• Whole page
• Snippets
• Update baselines as product evolves / changes
@BagmarAnand
How does it work?

35. Have you used any
Visual Testing tools?
@BagmarAnand

36. Challenges of Automating
Visual Testing
@BagmarAnand

37. • False positives / negatives
• Baseline maintenance
• Browsers
• Devices
• Resolutions / Viewports
• Result analysis & maintenance
@BagmarAnand
Challenges of Automated Visual Validation

38. AI in Visual Test Automation
@BagmarAnand

39. A Simple
One Page,
Single Browser
Example
41

40. First, Let’s Provide Initial Coverage
18 Lines of
Code
21 Locators
and Labels
60 Minutes
42
Navigation
Button
Click
Validation

41. NEW
43
Real Bug Escapes
Text Changes
Caught
New Feature
Missed
Visual Bug
Escapes
Broken Locators
Next, We Test a New Build
2 Late
Stage Bugs
2 Changes
Caught
2 False
Positives
Baseline New Candidate
New Feature
Missed

42. Test Results Maintenance Action Time
Functional Bug Write Another Validation 15 mins
Visual Bug Try Writing New Validations, Then Give Up 15 mins
Placeholder Text Differences Update Existing Validations 10 mins
New Terms & Conditions
Feature
Write Another Validation 5 mins
Changed Locators Update Locators 15 mins
Total Time Spent 60 mins
Finally, We Have to Maintain our Tests
NEW

43. Plus Those 21 Locators and Labels That Might
Change Without any Notice!
Build and Maintenance Summary
45
Build & Maintain:
120 Minutes
Bug Detection:
40%
False Positive Rate:
40%

44. Login Page to Test
Remember Our
Simple Example?
Let’s Try Again.
46

45. 47
Let’s Provide Initial Coverage Using Visual AI
2 Minutes to Create
Vs. 60 Minutes
1 Locator
Vs. 21 Locators
and Labels

46. 48
Visual AI Handles the New Build
NEW
Functional Bug
Found
Text Changes
Found
Visual Bug Found
New Feature
Found
Original New Build
No Broken
Locators
All Bugs Found
Text Changes and
New Feature
Detected
0 Broken Locators

47. Test Results Visual AI Maintenance Action Legacy Visual AI
Functional Bug None; Found Automatically 15 mins 0 mins
Visual Bug None; Found Automatically 15 mins 0 mins
Placeholder Text Differences “Thumbs Up” for Each Difference 10 mins 40 secs
New Terms & Conditions Feature “Thumbs Up” for Difference 5 mins 20 secs
Changed Locators Locators No Longer Used 15 mins 0 mins
Total Time Spent 60 mins 1 min
Visual AI Streamlines Test Maintenance
NEW

48. Legacy Test Automation
Visual AI Delivers Results
50
Visual AI
40x Faster
21x More Stable
2.5x Increase in
Bug Detection
Eliminate False Positives!
Build & Maintain:
3 Minutes
Locators:
1
Build & Maintain:
120 Minutes
Locators and Labels:
21
Bug Detection:
100%
Bug Detection:
40%
False Positive Rate:
0%
False Positive Rate:
40%

49. 51
Visual Assertions Advantages
A single assertion for complete functional
coverage
Bonus: validates the UI
Does not break when the UI changes
No coding skills required to maintain baselines
Root cause analysis
60+ SDKs to integrate with your Automated
Functional Tests

50. 53
Applitools Visual AI

51. 54
Next Generation Test Automation Through Visual AI
Help teams release perfect apps much faster at a reduced
cost.
Inventers of Visual AI that replicates the human eye and
brain, looks at every screen and page in seconds, and spots
app regressions that truly matter.

52. 55
Visual AI
Industry’s only AI powered computer vision
replicates the human eye & brain.
SDKs & Integrations
SDKs to augment all modern test frameworks, and
integrations with popular source control, CI, and
defect tracking systems.
Perfection Across All
Screens
Ultrafast Grid ensures visual perfection across all
browsers, screens, and viewports.
Ultrafast Grid Architecture >>
Speed of CI/CD
Execute tests in seconds vs minutes for faster builds
and on-time delivery.
Ultimate Efficiency
Create tests faster & with fewer brittle locators and
labels.
Increase Test Coverage
Open-ended, AI powered assertions, reduce code,
maintenance while increasing test coverage.
Applitools Cloud
Visual AI Platform
One Intelligent Platform
Adding Applitools Visual AI to your existing test automation framework allows quality teams to release perfect apps
faster and as a reduced cost.

53. 56
Eyes
Industry’s only AI powered computer vision
replicates the human eye & brain.
Contrast Advisor
Reduce contrast violations, save time, & ensure
your app is accessible to everyone.
Ultrafast Grid
Execute tests across all browsers, devices and
viewports in seconds vs minutes for faster builds
and on-time delivery.
Root Cause Analysis
Zero-in on differences with all the noise.
Ultrafast Test Cloud
Applitools - Ultrafast Test Cloud
The Applitools Ultrafast Test Cloud platform combines the Visual AI power of Eyes with the next generation of
cross browser testing.
Optional Add-On Component
Optional Add-On Component

54. 57
Hundreds of customers have already adopted the next generation of cross-
browser testing including:
The team at Applitools has taken a fresh approach to cross browser testing with the Ultrafast Grid. While traditional cloud
testing platforms are subject to false positives and slow execution, Applitools’ unique ability to run Visual AI in parallel
containers can give your team the unfair advantage of stability, speed, and improved coverage. This modern approach to
testing is something that all DevOps professionals should strongly consider.
Igor Draskovic - VP Developer Specialist, BNY Mellon
Perfection Across All Screens, Browsers and Viewports

55. 58
Applitools Ultrafast Grid for Native Apps

56. Visual Test Automation
needs a Strategy!
@BagmarAnand

57. Demo + Activity
@BagmarAnand

58. Test Pyramid … includes UX & NFRs
Unit (xUnit / JavaScript)
Component
Integration
View
Web Service
UI
P
e
r
f
o
r
m
a
n
c
e
S
e
c
u
r
i
t
y
Accessibility
UX
@BagmarAnand
Manual / Exploratory

59. • The Practical Test Pyramid - https://martinfowler.com/articles/practical-test-
pyramid.html
• Testing Pyramids & Ice-Cream Cones - https://watirmelon.blog/testing-
pyramids/
• Applitools – www.applitools.com
• Sample code to get started: https://github.com/anandbagmar/getting-
started-with-visualtesting
@BagmarAnand
References

60. Thank you!
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand