Slides from my talk on how to Eradicate Flaky Tests from AppiumConf 2021
https://confengine.com/conferences/appium-conf-2021/proposal/15581/eradicate-flaky-tests
4. @BagmarAnand
Internet Internal
B2C
App
Product DB
DBMS
B2B
App
Caching System
Elastic Search
Authentication
Gateway
Service 3
Service 4
Service 7
Service 5
Service 6
Admin
User DB
Outgoing
Notification
Onboarding
Event
Consumer
Order DB
Reporting System
Admin
Portal
Support
Msg Q
Direct API
Service 2
Notification
Service 1
Event
Processor
Service 8
Payment Gateway
Credit System
Master Product
Warehouse
Royalty system
Offers
….
….
….
6. API
workflow
Tests
E2E
Functional
UI Tests
@BagmarAnand
Internet Internal
B2C
App
Product DB
DBMS
B2B
App
Caching System
Elastic Search
Authentication
Gateway
Service 3
Service 4
Service 7
Service 5
Service 6
Admin
User DB
Outgoing
Notification
Onboarding
Event
Consumer
Order DB
Reporting System
Admin
Portal
Support
Msg Q
Direct API
Service 2
Notification
Service 1
Event
Processor
Service 8
Payment Gateway
Credit System
Master Product
Warehouse
Royalty system
Offers
….
….
….
7. • Long running scenarios
• Slow feedback
Challenges of E2E Test Automation
@BagmarAnand
8. • Limitations of automatable scenarios
• Error case / Edge case handling from dependent systems
• Triggers from external systems
Challenges of E2E Test Automation
@BagmarAnand
9. • Pain of Cross Browser execution
• Slow, special handling for browsers, infrastructure cost /
maintenance
Challenges of E2E Test Automation
@BagmarAnand
14. • Reduce number of UI tests
• Remove external dependencies via Intelligent Virtualization
• Use Visual Assertions instead of Functional Assertions
How to Reduce Flakiness in Tests?
@BagmarAnand
19. @BagmarAnand
Internet Internal
B2C
App
Product DB
DBMS
B2B
App
Caching System
Elastic Search
Authentication
Gateway
Service 3
Service 4
Service 7
Service 5
Service 6
Admin
User DB
Outgoing
Notification
Onboarding
Event
Consumer
Order DB
Reporting System
Admin
Portal
Support
Msg Q
Direct API
Service 2
Notification
Service 1
Event
Processor
Service 8
Payment Gateway
Credit System
Master Product
Warehouse
Royalty system
Offers
….
….
….
21. API
workflow
Tests
E2E
Functional
UI Tests
@BagmarAnand
Internet Internal
B2C
App
Product DB
DBMS
B2B
App
Caching System
Elastic Search
Authentication
Gateway
Service 3
Service 4
Service 7
Service 5
Service 6
Admin
User DB
Outgoing
Notification
Onboarding
Event
Consumer
Order DB
Reporting System
Admin
Portal
Support
Msg Q
Direct API
Service 2
Notification
Service 1
Event
Processor
Service 8
Payment Gateway
Credit System
Master Product
Warehouse
Royalty system
Offers
….
….
….
23. @BagmarAnand
Step #1:
• Test calls Service 2
Step #2:
• Service 2 calls the external
service, which may
(unexpectedly) take long
time to respond
Step #3:
• Service 2 gets an error back
from external service
• Returns an error to the test
Step #4:
• Test fails because of
external service
• We may get
incorrect error as
well!
Test Scenario
27. • Inculcates Contract Driven Development
• Supports OpenAPI Spec
• Executable Contract spec
• Defines http method, path, header format, payload format, query parameter format, form field payload format
• Supports JSON, XML data formats
• Supports Kafka
• Dynamically generates and executes contract tests
• Apt for service virtualization
• Compares contracts to ascertain backward compatibility
• Manages contracts through versioning
• Integrates with CI/CD
Specmatic – Contract Driven Development
@BagmarAnand
• Decouples the ”producers” and “consumers”, with confidence!
28. Use Specmatic as part of Test execution
@BagmarAnand
Step #3:
• Test calls Service 2
Step #4:
• Service 2 calls the
stubbed external service
Step #2:
• Test sets expectations on Specmatic for
specific stubbed endpoint
• Http POST call on Specmatic server with
exact request & response as json payload
Step #5:
• If request matches set expectation, corresponding set
response is returned in response
• Service 2 processes response and responds to test
appropriately
Step #6:
• Test asserts on the
received response
• Test passes
Step #1. Specmatic as a Stub Server in your Test
Environment
• Specmatic Server is started in the Stub mode in
the Environment
• External services are stubbed out
• Internal services point to Specmatic stubbed-
endpoints instead of real external endpoints
29. Use Specmatic as part of Test execution
@BagmarAnand
Step #3:
• Test calls Service 2
Step #4:
• Service 2 calls the
stubbed external service
Step #2:
• Test sets expectations on Specmatic for
specific stubbed endpoint
• Http POST call on Specmatic server with
exact request & response as json payload
Step #5:
• If request matches set expectation, corresponding set
response is returned in response
• Service 2 processes response and responds to test
appropriately
Step #6:
• Test asserts on the
received response
• Test passes
Step #1. Specmatic as a Stub Server
• Specmatic Server is started in the Stub mode in
the Environment
• External services are stubbed out
• Internal services point to Specmatic stubbed-
endpoints instead of real external endpoints
Allows to simulate positive,
negative, edge-case responses –
allowing you to test your
implementation logic
51. Visual AI should work across all Platforms
PDF, Web, Native Mobile
@BagmarAnand
52. Perfection Across All Screens, Browsers and
Viewports
Tests run local - Applitools
SDKs available for all major
testing frameworks DOM & CSS sent to
Applitools Ultrafast Grid
Applitools Eyes
Visual AI analysis
Ultrafast Grid renders
DOM snapshots in parallel
Login to Eyes to
view test results
53
.
Firewall
54. @BagmarAnand
Application under test
Login
Get list of users
Edit User
Test Scenario
Login 1
Get list of users 2
Internet
Return dynamic
list of users
3
4
Edit User 5
Verify User
Details
6
Return dynamic
list of users
Test Scenario
Example Scenario
55. Website with rest services in
backend hosted on local machine
Login
Get list of users
Edit User
Test Scenario
Functional
& Visual
Assertions
using
Applitools
Login 1
Get list of users
3
Verify User
Details
7
Edit User
6
4
5
Test Environment
Return dynamic
list of users
Stubbed
using
Specmatic
Static
Random
Dynamic
Expectations
@BagmarAnand
2. Test sets “dynamic
expectation” in context
of the scenario
Example Scenario using Specmatic
I
n
t
e
r
n
e
t
56. @BagmarAnand
Example Scenario using Specmatic
Website with rest services in
backend hosted on local machine
Login
Get list of users
Edit User
Test Scenario
Functional
& Visual
Assertions
using
Applitools
Login 1
Get list of users
3
Verify User
Details
7
Edit User
6
4
5
Test Environment
Return dynamic
list of users
Stubbed
using
Specmatic
Static
Random
Dynamic
Expectations
2. Test sets “dynamic
expectation” in context
of the scenario
I
n
t
e
r
n
e
t
Return dynamic
list of users
Test Scenario
59. • Reduce number of UI tests – Test Pyramid
• Use Visual Assertions instead of Functional Assertions
• Remove external dependencies via Intelligent Virtualization
Make your Automation Intelligent!
@BagmarAnand
60. • Move granular tests to lower layers of pyramid
• Get faster feedback
• Reduce flakiness / brittleness due to UI changes
Reduce UI tests
@BagmarAnand
61. • A single assertion for complete
functional coverage
• Bonus: validates the UI
• Does not break when the UI changes
• No coding skills required to maintain
baselines
• Validate UX at scale for all supported
browsers
Use Visual Assertions
@BagmarAnand
64. • Covers all environments at the speed of running a single local test
• Much more robust and stable
• Full functional and visual coverage
• No security vulnerabilities
• Reduce test data preparation and consumption
• Does not load the corporate network
• Costs much less
Run once, validate everywhere
@BagmarAnand
65. • Test environment independent of external dependencies
• Tests set dynamic expectations based on context of the
test execution
• Stubbed service should be able to give static responses /
random responses
Virtualize Dependencies at Runtime
@BagmarAnand
66. • Visual AI using Applitools
• Getting Started with Visual Testing - https://testautomationu.applitools.com/automated-
visual-testing-a-fast-path-to-test-automation-success/
• Modern Functional Test Automation Through Visual AI -
https://testautomationu.applitools.com/modern-functional-testing/
• Intelligent Virtualization using Specmatic
• https://specmatic.in/
• https://github.com/znsio/specmatic
• Test Automation University - https://testautomationu.applitools.com/
• Test Pyramid - https://martinfowler.com/articles/practical-test-pyramid.html
Resources
@BagmarAnand