The document discusses advancements in automating visual software testing, emphasizing the limitations of traditional UI testing and the advantages of using VisualAI for improved functional coverage. It outlines the benefits of a new Ultrafast Grid for cross-device testing, which accelerates test implementation, reduces code complexity, and enhances stability while addressing security concerns. The solution aims to significantly improve test coverage and efficiency, ultimately leading to faster release cycles and reduced costs.

1. Automating visual software testing
CUTTING-EDGE FUNCTIONAL
UI TESTING TECHNIQUES:
TAKING AUTOMATION TO
THE NEXT LEVEL
Adam Carmi
CTO

2. Automating visual software testing
AGENDA
• Challenges with traditional UI testing
• UI testing with VisualAI
• About VisualAI
• Cross device / browser testing with the Ultrafast Grid

3. Automating visual software testing
THE #1
AUTOMATION
PRIORITY
Increase functional
test automation
coverage to safely
speed up release
cycles

4. Automating visual software testing
TYPES OF FUNCTIONAL TESTING
UI Testing (E2E)
API Testing
Unit Testing

5. Automating visual software testing
E2E TESTING IS IMPORTANT!

6. Automating visual software testing
UI TESTING
• The most widespread type of functional testing
• Automated with Selenium, Appium, Cypress, …
• Tests the end-to-end functionality of a system through its UI
• A test is a sequence of steps in which you
• Simulate user inputs: navigation, clicks, swipes, keystrokes, etc.
• Validate the system output on the page using assertions

7. Automating visual software testing
TEST REQUIRED LOGIN DETAILS

8. Automating visual software testing
A CYPRESS UI TEST

9. Automating visual software testing
WHAT IF THE CSS CHANGES?
A visual bug!

10. Automating visual software testing
WHAT IF THE CSS CHANGES?
A functional bug!

11. Automating visual software testing
OUR TEST STILL
PASSES…

12. Automating visual software testing
HOW MANY
FUNCTIONAL
ASSERTIONS
ARE NEEDED
HERE?

13. Automating visual software testing
HOW MANY
FUNCTIONAL
ASSERTIONS
ARE NEEDED
HERE?

14. Automating visual software testing
WHEN THE UI CHANGES…

15. Automating visual software testing
THE PAIN
• Most teams struggle to succeed with UI testing
• Every assertion increases the maintenance overhead - Is it really worth automating?
• Coding skills required to create and maintain tests
• Can’t assert on non-dom elements: image, canvas, css graphics, WebGL, etc.
• Checks system functionality via the UI – Not the UI!
• Can only catch expected bugs
• Manual testing is still needed!

16. Automating visual software testing
THE SOLUTION…
UI Testing with Visual AI!

17. Automating visual software testing
ADDING A VISUAL CHECKPOINT

18. Automating visual software testing

19. Automating visual software testing
VISUAL ASSERTIONS ADVANTAGES
• A single assertion for complete
functional coverage
• Does not break when the UI
changes
• No coding skills required to
maintain baselines
• Finds unexpected issues
• Bonus: validates the UI

20. Automating visual software testing
Empirical evidence based on 288 completions (10 grand prize winners)
• Time to implement test suite: 5.8x faster (7.8x for winners)
• Total lines of code: 49% less code (72% for winners)
• Number of locators: 4.6x more stable (5.9x for winners)
• Bugs found: 95% vs 65% (100% vs 65% for winners)

21. Automating visual software testing
Visual
Testing
UI
Testing
Better covered with
API or unit tests
Much more coverage for
a fraction of the effort
UI VS VISUAL TESTING

22. Automating visual software testing
99.9999%
accuracy
…
and
improving!

23. Automating visual software testing

24. Automating visual software testing

25. Automating visual software testing
IGNORE DISPLACEMENT DIFFS

26. Automating visual software testing
TEST RESULTS DE-DUPLICATION

27. Automating visual software testing
ROOT CAUSE ANALYSIS
CSS rule removed
CSS rules changed

28. Automating visual software testing
UNIVERSAL FRAMEWORK SUPPORT

29. Automating visual software testing
CROSS
DEVICE /
BROWSER
TESTING
Everyone wants to
verify that their apps
work on their
customer’s devices

30. Automating visual software testing
RUNNING A UI TEST LOCALLY
Dev / Test Machine
Corporate Firewall
Application under test
+ Fast
+ Stable
+ Secure
- Machine busy
- Scale
- Platforms

31. Automating visual software testing
UI TESTING USING A LOCAL LAB
Application under test
Dev / Test / Build machine
Corporate Firewall
Lab VM / device
Lab VM / device
+ Fast
+ Stable
+ Secure
+ Scale
+ Platforms
- Maintenance

32. Automating visual software testing
UI TESTING USING A REMOTE LAB
Application under test
Dev / Test / Build machine
Corporate Firewall
Lab VM / device
Lab VM / device
Lab provider
+ Scale
+ Platforms
+ Maintenance
- Slow
- Unstable
- Security

33. Automating visual software testing
REMOTE LAB SECURITY CONCERNS
• Require inbound network access
• Tunnels (e.g., sauce connect etc.) are just as insecure
• The remote VM is an unmanaged asset
• Can run any software
• Can access the internet
• Has access to customer’s internal network
• Compliance

34. Automating visual software testing
BUT WHERE ARE THE BUGS?
• Server bugs are device / browser independent
• A broken DB query, logical error, backend performance issues, etc.
• Functional app bugs are 99% device / browser agnostic
• Modern browsers conform to the W3C standard
• Logical bugs occur in all client environments.
• Examples: not validating an input field, reversed sorting order, etc.
• The real value is in finding visual bugs!

35. Automating visual software testing
THE PAIN
• Most teams are very unhappy with their remote test lab setup
• Tests take much longer to execute
• Tests are much less stable due to delays, timeouts and network failures
• Security concerns
• Need to run each test multiple times
• More failures, more time to analyze test failures, more time to wait for test
completion, high concurrency costs
• Very low ROI (does not test the UI)

36. Automating visual software testing
THE SOLUTION…
The Ultrafast Grid!

37. Automating visual software testing
UI TESTING USING THE UFG
Corporate Firewall
Ultrafast Grid
Application under test
Dev / Test / Build machine
new page resources
checkpoint images

38. Automating visual software testing
A TURNKEY SOLUTION
38

39. Automating visual software testing
ULTRAFAST GRID ADVANTAGES
• Covers all environments at the speed of running a single local test
• Much more robust and stable
• Full functional coverage and visual coverage
• No security vulnerabilities
• Reduce test data preparation and consumption
• Does not load the corporate network
• Costs much less

40. Automating visual software testing
DEMO

41. Automating visual software testing
FUNCTIONAL
UI TESTING
WITH
VISUAL AI
• Drastically increase test coverage
and find unexpected issues
• Drastically reduce test creation and
maintenance time
• Drastically reduce test execution
time
• Supports any framework you are
using today or will use tomorrow
• Reduce test lab costs

42. Automating visual software testing
QUESTIONS?