The document discusses the shift from traditional quality assurance to quality intelligence in the context of end-to-end test automation. It emphasizes the need for modular, data-driven testing within a continuous integration and deployment framework to enhance efficiency and reduce long lead times associated with legacy testing tools. Additionally, it highlights the importance of collaborative workflows and comprehensive diagnostics in the development process to ensure faster delivery of quality software.

1. End-to-end Test Automation
at Scale

2. Anton Hristov
Product Manager @ mabl
@AntonHristov

3. Agenda
● Quality assurance ➡ quality intelligence
● End-to-end test creation & execution
● Reusable modular tests
● Root-cause analysis
● Data-driven (parameterized) testing

4. Continuous integration and deployment (CI/CD) tools enable developers
to ship code much more frequently, often multiple times a day
software delivery has changed...
...but traditional testing tools haven’t changed much. 4

5. Outsourced QA
- Waiting for test results
takes too much time
- Long lead times on test
creation and updates
- Limited by legacy and
open source tools
Open/custom source
- Flaky, brittle and hard to
maintain tests
- Little diagnostics data, if any
- Highly specialized technical
skills, hard to onboard new
team members
5
Legacy tools
- Complex setup
and maintenance
- Costly infrastructure
- Limited DevOps workflow
support
Challenges with traditional testing tools and processes.

6. Waterfall → Agile → DevTestOps
1990s 2000s Today
DevTestOps is here..
Continuous delivery creates the need for continuous testing, and that
requires quality intelligence to operate effectively at scale.
6

7. Exploratory testing
Unit testing
Monitoring
Integration testing
Peer review
End-to-end testing
Quality intelligence
redefining quality assurance.

8. 8
Dev Unit Integration End-to-end Production
Catch bugs sooner, before users do!
…and bring customer intelligence closer to the development team.
User
Shorten the feedback loop and increase the velocity of delivering value to users.
We love shift-left in testing….

9. Customers demand fixes within 8 hours
DevTestOps Landscape Survey 2019, n=500, mabl.com/results

10. Intelligent testing principles
1. Easy and fun to create, and auto-create
2. Adapt seamlessly to change
3. Run in the cloud
4. Produce insightful output
5. Part of continuous delivery pipeline (DevTestOps)
6. Support collaborative workflow
7. New definition of quality

11. Agenda
● Quality assurance ➡ quality intelligence
● End-to-end test creation & execution
● Reusable modular tests
● Root-cause analysis
● Data-driven (parameterized) testing

12. Demo
End-to-end test creation and
execution

13. Agenda
● Quality assurance ➡ quality intelligence
● End-to-end test creation & execution
● Reusable modular tests
● Root-cause analysis
● Data-driven (parameterized) testing

14. User flows
● A flow is a common set of steps across multiple user journeys
● Typical testing use cases
○ Navigation to a page from different locations within the app under test
○ Login with different users - client portal vs admin portal
○ Application entity creation - user, product, ticket, etc.
○ Component testing - carousels, tables, etc.
● Often tested with variables and data-driven testing with parameters

15. Demo
Reusable modular testing and
root-cause analysis

16. Modular, data-driven testing at scale.
1616
Parameter values driven by DataTables
Multiple journeys running same flow in
multiple browsers & environments
Flows
Code snippets
Journey

17. Adapt tests when your app changes.
17

18. Customize with JavaScript.
18
+ Create a library of
shared JavaScript code
snippets that everyone
can use, including
business users

19. Agenda
● Quality assurance ➡ quality intelligence
● End-to-end test creation & execution
● Reusable modular tests
● Root-cause analysis
● Data-driven (parameterized) testing

20. 20
Comprehensive diagnostics data.

21. + Capture screenshots and
detected visual changes
without any scripting or
configuration.
+ Dynamic content regions are
automatically detected to
reduce noise and help you
focus on what matters
Built-in visual testing and monitoring.
2121

22. Identify application load time issues.
+ mabl detected that
application performance
improved after a code
deployment
22
+ Speed index measures
perceived load speeds, similar
to Google Lighthouse tools

23. Agenda
● Quality assurance ➡ quality intelligence
● End-to-end test creation & execution
● Reusable modular tests
● Root-cause analysis
● Data-driven (parameterized) testing

24. Data-driven testing
A software testing methodology where you repeatedly execute the same set
of steps using a data source to supply inputs and expected outputs for
verification.

25. Functional Test
Data-driven testing
Data
source
Execute test steps
App or website
under test
Input values
Compare values
(actual vs expected)
Expected output Actual output

26. Calculator example
Scenario Input A Input B Operator Output
Addition 7 4 + 11
Subtraction 20 6 - 14
Multiplication 3 5 * 15
Division 21 3 / 7

27. Calculator example
Data
source
Execute test steps
Calculator app
7 + 4 = ?
Input: 7, 4, +
Compare values
(actual vs expected)
Expected output: 11 Actual output: 11

28. Calculator example
Data
source
Execute test steps
Calculator app
7 + 4 = ?
Input: 7, 4, +
Compare values
(actual vs expected)
Expected output: 11 Actual output: 15

29. Data-driven testing benefits
● Separation of concerns
○ functionality vs data inputs (parameters)
● Increased test coverage
● Eliminated redundancy
● Easier maintenance
● Expensive to manually test but cheap when automated
○ Takes advantage of parallel test execution
● Easy to add new testing scenarios as the application evolves

30. Demo
Data-driven testing

31. mabl DataTables summary
● Parameters are first specified in the mabl Trainer
● Test inputs and expected outputs are retrieved from a DataTable
● Support for copy + paste from a spreadsheet such as Microsoft Excel
● Tight association between a journey and a DataTable
● Each DataTable scenario triggers a separate journey run, in parallel
● Ability to overwrite Journey and DataTable association from plan settings

32. And one more thing… Security
● Encryption of all data in transit and at rest with a key for each customer
● Data isolation from other customers
● Secure tunneling to test internal/local environments
● Proactive approach to security
● Compliance with data privacy standards - GDPR, US/EU Privacy Shield
● Trusted partner

33. Thank you!
Try mabl for free at app.mabl.com/signup