The document discusses the integration of AI in software testing, emphasizing areas such as automated asset and test case creation, as well as the analysis of results to improve efficiency and coverage. It highlights the advantages of AI over traditional testing methods, particularly in generating insights and focusing on high-risk areas. Additionally, it outlines the systematic approach needed to leverage AI throughout the testing process, including input and output data capture.

1. HOW DO I GET THE MOST FROM AI-DRIVEN TESTING
JUNE 2019

2. OVERVIEW
1. Where should we apply AI in testing?
2. What do we need to understand about AI?
3. Automation of…
1. Asset creation
2. Test case creation
3. Understanding of results
4. Summary

3. WHERE SHOULD WE APPLY AI IN TESTING?
Automatic & manual
model building
Defined Regression
Tests if required
Automatically Generate
Test Cases (AI powered
Exploratory)
Business, technical
and predictive insights
of the results
Execute all types of
Tests using a Fusion
Engine
Import existing assets
(e.g. gherkin)
Analyze test results and
overall release; highlighting
key development risks
Focus testing on
important areas (e.g.
have changed)
Automatically Enter
defects and insights
Model
generation from
real customer
behavior
1. Auto Generation +
Maintenance of
Assets
2. Auto Generation +
Optimization of
Test Cases
3. Understanding Results
+ Analysis of Change

4. WHAT IS AI?
?

5. WHAT IS AI?
• Learning algorithms
• Neural networks

6. WHAT IS AI? LEARNING ALGORITHMS
• Learning algorithms
• Neural networks
f (developer) => is_risky
developer is_risky
Antony No
Jonathan No
Michel Yes
Andre Definitely
Simon Yes
Change developer defects
1 Antony No
2 Andre Yes
3 Andre Yes
4 Antony No
5 Simon Yes
6 Antony No
7 Andre Yes
8 Antony Yes
9 Andre No
10 Simon No

7. WHAT IS AI? LEARNING ALGORITHMS
• Learning algorithms
• Neural networks2.1s
1.2s
1.6s

8. HOW DO WE APPLY THIS TO TESTING?
• …understand inputs
• Metadata about UI elements
• Type
• Origin
• Location
• Relation to code and issue tracking
• Real user behaviors
• …understand outputs
• Locations and patterns of failures
• Learn from past outcomes both in dev and production
• Differences between defects and changes
• Data beyond pass/fail – coverage, performance, UX…
• Difficult to manage through conventional scripting
• Creation of a Model/Digital Twin of the application is key
We need to…

9. WHERE SHOULD WE APPLY AI IN TESTING?
Automatic & manual
model building
Defined Regression
Tests if required
Automatically Generate
Test Cases (AI powered
Exploratory)
Business, technical
and predictive insights
of the results
Execute all types of
Tests using a Fusion
Engine
Import existing assets
(e.g. gherkin)
Analyze test results and
overall release; highlighting
key development risks
Focus testing on
important areas (e.g.
have changed)
Automatically Enter
defects and insights
Model
generation from
real customer
behavior
1. Auto Generation +
Maintenance of
Assets
2. Auto Generation +
Optimization of
Test Cases
3. Understanding Results
+ Analysis of Change

10. AUTOGENERATION OF ASSETS
tdfsf
eggplant Bank

11. WHERE SHOULD WE APPLY AI IN TESTING?
Automatic & manual
model building
Defined Regression
Tests if required
Automatically Generate
Test Cases (AI powered
Exploratory)
Business, technical
and predictive insights
of the results
Execute all types of
Tests using a Fusion
Engine
Import existing assets
(e.g. gherkin)
Analyze test results and
overall release; highlighting
key development risks
Focus testing on
important areas (e.g.
have changed)
Automatically Enter
defects and insights
Model
generation from
real customer
behavior
1. Auto Generation +
Maintenance of
Assets
2. Auto Generation +
Optimization of
Test Cases
3. Understanding Results
+ Analysis of Change

12. AUTOGENERATION OF TEST CASES
?9.2B

13. AUTOGENERATION OF TEST CASES
Auto-Generate Tests to Optimize Coverage in Time Available
Next Action &
Values

14. SHORTCOMINGS WITH TRADITIONAL, STRUCTURED TESTING
• Test cases are manually defined
⇢ Time-consuming to produce
⇢ Includes assumptions & happy paths
• Test cases have fixed structures
⇢ Same tests run every time
⇢ No use of results to improve coverage
Shortcomings With Traditional, Structured Testing
Structured testing
Tests
Identified defects
Missed defects
Like playing Battleship game without feedback:
• Using the same grid locations for every game
• Not using a hit to focus on sinking a ship
• Only have a limited supply of pegs
• Test cases are manually defined
⇢ Time-consuming to produce
⇢ Includes assumptions & happy paths
• Test cases have fixed structures
⇢ Same tests run every time
⇢ No use of results to improve coverage
Structured testing
Tests
Identified defects
Missed defects
Like playing Battleship game without feedback:
• Using the same grid locations for every game
• Not using a hit to focus on sinking a ship
• Only have a limited supply of pegs
Tests
Identified defects
Missed defects
Structured testing
Test cases are manually defined
• Time-consuming to produce
• Includes assumptions & happy paths
Test cases have fixed structures
• Same tests run every time
• No use of results to improve coverage
Like playing a Battleship game without feedback:
- Using the same grid locations for every game.
- Not using a hit to focus on sinking a ship.
- Only have a limited supply of pegs.

15. TEST OPTIMIZATION – RECOMMENDATION ENGINES
Author Sylvia Bishop
Genre Mystery
Length 240 pages
Year Published 2018

16. TEST OPTIMIZATION – RECOMMENDATION ENGINES
✓
✓
✓
✘

17. WHERE SHOULD WE APPLY AI IN TESTING?
Automatic & manual
model building
Defined Regression
Tests if required
Automatically Generate
Test Cases (AI powered
Exploratory)
Business, technical
and predictive insights
of the results
Execute all types of
Tests using a Fusion
Engine
Import existing assets
(e.g. gherkin)
Analyze test results and
overall release; highlighting
key development risks
Focus testing on
important areas (e.g.
have changed)
Automatically Enter
defects and insights
Model
generation from
real customer
behavior
1. Auto Generation +
Maintenance of
Assets
2. Auto Generation +
Optimization of
Test Cases
3. Understanding Results
+ Analysis of Change

18. UNDERSTANDING RESULTS

19. SUMMARY
Need an approach that systematically captures
- Input data
- Output data
Leverage these to increase automation across each stage of the testing
process
- Asset generation
- Test Case generation
- Execution
- Understanding results

20. Q&A TIME
JUNE 2019