This document discusses using real customer data and artificial intelligence to improve software testing. Traditional testing is time-consuming and misses defects because test cases are manually defined and do not adapt. A new approach called Digital Experience Optimization uses AI to automatically generate test cases based on real user journeys and behaviors. This continuous and adaptive testing approach finds more issues while requiring less effort over time compared to traditional structured testing.

1. USING REAL CUSTOMER DATA TO DRIVE TESTING
JUNE 2019

2. HOW A SOFTWARE BUG COSTS $45 MILLION AND A MARRIAGE
Software is so heavily embedded in our
day to day lives that a software failure can
have severe implications
Implications can be loss of reputation,
revenue and in some cases lives

3. WHO IS TO BLAME FOR BUGS CREEPING THROUGH?

4. SHORTCOMINGS WITH TRADITIONAL, STRUCTURED TESTING
• Test cases are manually defined
⇢ Time-consuming to produce
⇢ Includes assumptions & happy paths
• Test cases have fixed structures
⇢ Same tests run every time
⇢ No use of results to improve coverage
Shortcomings With Traditional, Structured Testing
Structured testing
Tests
Identified defects
Missed defects
Like playing Battleship game without feedback:
• Using the same grid locations for every game
• Not using a hit to focus on sinking a ship
• Only have a limited supply of pegs
• Test cases are manually defined
⇢ Time-consuming to produce
⇢ Includes assumptions & happy paths
• Test cases have fixed structures
⇢ Same tests run every time
⇢ No use of results to improve coverage
Structured testing
Tests
Identified defects
Missed defects
Like playing Battleship game without feedback:
• Using the same grid locations for every game
• Not using a hit to focus on sinking a ship
• Only have a limited supply of pegs
Tests
Identified defects
Missed defects
Structured testing
Test cases are manually defined
• Time-consuming to produce
• Includes assumptions & happy paths
Test cases have fixed structures
• Same tests run every time
• No use of results to improve coverage
Like playing a Battleship game without feedback:
- Using the same grid locations for every game.
- Not using a hit to focus on sinking a ship.
- Only have a limited supply of pegs.

5. THE COST OF TEST-CASE PRODUCTION
Manual test case development Eggplant AI
The Cost of Test-Case Production
Linear growth of limited coverage
with constant effort
Time
Test cases
Test cases
Time
Effort
Hundreds
Millions
Effort
App1
App
2
W
ebsite1
App
3
W
ebsite2
System
1
System
2
Full coverage from only
small initial effort
Hugely
reduced
maintenance

6. A NEW APPROACH: DIGITAL EXPERIENCE OPTIMIZATION
continuous quality
Continuous
Quality
Operational
Excellence
DIGITAL
EXPERIENCE
TESTING
DIGITAL
EXPERIENCE
MONITORING
DEV OPS

7. AI EXPLORATION BASED ON REAL USER JOURNEYS
Restaurants
Location Search
AI Exploration Based on Real User Journeys
Homepage
Hotels
Flights
Near Me Now
Restaurants
Near Me Now
Hotels
Restaurants
Things to Do
Rentals
Hotels
Location
Dates
Type
Search
Flights
From
Class
To
Search
Passengers
Back Back
Back
Back
Rotate Screen
Exit
Exit
AI model favors
sequences of actions that
are part of defined test
cases.

8. CXO IN ACTION

9. USING REAL CUSTOMER DATA TO DRIVE TESTING
JUNE 2019