Download as PDF, PPTX
Uploaded byAnand Bagmar
Eradicate Flaky Tests - AppiumConf 2021
The document discusses the challenges of flaky tests in end-to-end (e2e) test automation, highlighting issues such as external dependencies, slow feedback, and browser execution difficulties. It proposes solutions including reducing UI tests, utilizing intelligent virtualization to eliminate dependencies, and employing visual assertions instead of functional assertions to improve test reliability. The use of Specmatic for contract-driven development and Applitools for visual AI-based testing are recommended to enhance automation effectiveness.
More Related Content
Similar to Eradicate Flaky Tests - AppiumConf 2021
Eradicate Flaky Tests - AppiumConf 2021
- 1. Eradicate Flaky Tests @BagmarAnand AnandBagmar Software Quality Evangelist Essence Of Testing
- 2.
- 3.
- 4. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App CachingSystem Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 5. Approach for API Workflow& E2E Test Execution @BagmarAnand
- 6. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App ProductDB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 7. • Long runningscenarios • Slow feedback Challenges of E2E Test Automation @BagmarAnand
- 8. • Limitations ofautomatable scenarios • Error case / Edge case handling from dependent systems • Triggers from external systems Challenges of E2E Test Automation @BagmarAnand
- 9. • Pain ofCross Browser execution • Slow, special handling for browsers, infrastructure cost / maintenance Challenges of E2E Test Automation @BagmarAnand
- 10. • Flaky Tests Challengesof E2E Test Automation @BagmarAnand
- 11. Why are TestsFlaky? @BagmarAnand
- 12. • Brittle /Flaky tests • UI changes • Downstream dependencies • Data dependencies • Network (speed) fluctuations Challenges of E2E Test Automation @BagmarAnand
- 13. Solutions / Wayforward @BagmarAnand
- 14. • Reduce numberof UI tests • Remove external dependencies via Intelligent Virtualization • Use Visual Assertions instead of Functional Assertions How to Reduce Flakiness in Tests? @BagmarAnand
- 15. #1 Reduce the numberof UI Tests @BagmarAnand
- 16.
- 17. #2 Intelligent Virtualization to EradicateDependencies @BagmarAnand
- 18.
- 19. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App CachingSystem Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 20. Approach for APIWorkflow & E2E Test Automation @BagmarAnand
- 21. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App ProductDB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 22.
- 23. @BagmarAnand Step #1: • Testcalls Service 2 Step #2: • Service 2 calls the external service, which may (unexpectedly) take long time to respond Step #3: • Service 2 gets an error back from external service • Returns an error to the test Step #4: • Test fails because of external service • We may get incorrect error as well! Test Scenario
- 24. Challenge! Scenarios having external dependenciesare flaky @BagmarAnand
- 25. Solution @BagmarAnand Stub it outusing Specmatic!
- 26.
- 27. • Inculcates ContractDriven Development • Supports OpenAPI Spec • Executable Contract spec • Defines http method, path, header format, payload format, query parameter format, form field payload format • Supports JSON, XML data formats • Supports Kafka • Dynamically generates and executes contract tests • Apt for service virtualization • Compares contracts to ascertain backward compatibility • Manages contracts through versioning • Integrates with CI/CD Specmatic – Contract Driven Development @BagmarAnand • Decouples the ”producers” and “consumers”, with confidence!
- 28. Use Specmatic aspart of Test execution @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server in your Test Environment • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints
- 29. Use Specmatic aspart of Test execution @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints Allows to simulate positive, negative, edge-case responses – allowing you to test your implementation logic
- 30. #3 Visual Assertions insteadof Functional Assertions @BagmarAnand
- 31.
- 32. Bugs still escape @BagmarAnand Traditionalautomated testing frameworks are not built for modern apps
- 33.
- 34.
- 35.
- 36.
- 37.
- 38. Bugs escape becauseour approach to testing is incorrect @BagmarAnand
- 39. @BagmarAnand Mundane Error prone Tedious Running againstTime Approach to Testing is Incorrect
- 40.
- 41.
- 42. But, we aretesting real software products @BagmarAnand
- 43. 43 @BagmarAnand Text, Images, Responsive-pages, Form-factors,User-experience Product has Context!
- 44.
- 45.
- 46.
- 47. @BagmarAnand Computer Vision that replicates thehuman eye and brain
- 48.
- 49. @BagmarAnand Visual AI detectsbugs, not browser render diffs
- 50. @BagmarAnand Pixel comparisons wastetime with false positives
- 51. Visual AI shouldwork across all Platforms PDF, Web, Native Mobile @BagmarAnand
- 52. Perfection Across AllScreens, Browsers and Viewports Tests run local - Applitools SDKs available for all major testing frameworks DOM & CSS sent to Applitools Ultrafast Grid Applitools Eyes Visual AI analysis Ultrafast Grid renders DOM snapshots in parallel Login to Eyes to view test results 53 . Firewall
- 53.
- 54. @BagmarAnand Application under test Login Getlist of users Edit User Test Scenario Login 1 Get list of users 2 Internet Return dynamic list of users 3 4 Edit User 5 Verify User Details 6 Return dynamic list of users Test Scenario Example Scenario
- 55. Website with restservices in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations @BagmarAnand 2. Test sets “dynamic expectation” in context of the scenario Example Scenario using Specmatic I n t e r n e t
- 56. @BagmarAnand Example Scenario usingSpecmatic Website with rest services in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations 2. Test sets “dynamic expectation” in context of the scenario I n t e r n e t Return dynamic list of users Test Scenario
- 57.
- 58.
- 59. • Reduce numberof UI tests – Test Pyramid • Use Visual Assertions instead of Functional Assertions • Remove external dependencies via Intelligent Virtualization Make your Automation Intelligent! @BagmarAnand
- 60. • Move granulartests to lower layers of pyramid • Get faster feedback • Reduce flakiness / brittleness due to UI changes Reduce UI tests @BagmarAnand
- 61. • A singleassertion for complete functional coverage • Bonus: validates the UI • Does not break when the UI changes • No coding skills required to maintain baselines • Validate UX at scale for all supported browsers Use Visual Assertions @BagmarAnand
- 62. Run once, validateeverywhere @BagmarAnand
- 63. Ultrafast Grid @BagmarAnand Corporate Firewall UltrafastGrid new page resources checkpoint images visual assertions Application under test Test / Build machine Lab VM / containers
- 64. • Covers allenvironments at the speed of running a single local test • Much more robust and stable • Full functional and visual coverage • No security vulnerabilities • Reduce test data preparation and consumption • Does not load the corporate network • Costs much less Run once, validate everywhere @BagmarAnand
- 65. • Test environmentindependent of external dependencies • Tests set dynamic expectations based on context of the test execution • Stubbed service should be able to give static responses / random responses Virtualize Dependencies at Runtime @BagmarAnand
- 66. • Visual AIusing Applitools • Getting Started with Visual Testing - https://testautomationu.applitools.com/automated- visual-testing-a-fast-path-to-test-automation-success/ • Modern Functional Test Automation Through Visual AI - https://testautomationu.applitools.com/modern-functional-testing/ • Intelligent Virtualization using Specmatic • https://specmatic.in/ • https://github.com/znsio/specmatic • Test Automation University - https://testautomationu.applitools.com/ • Test Pyramid - https://martinfowler.com/articles/practical-test-pyramid.html Resources @BagmarAnand
- 67.