Analytics-driven testing uses data on user behavior and software usage to prioritize testing efforts on the most critical areas. This allows testing teams to significantly reduce regression testing time by focusing on the most popular features and environments rather than exhaustively testing all possible combinations. Analytics can provide insights into actual usage patterns to determine the optimal risk-based testing strategy and achieve high test coverage within constrained timelines.

1. Why Adopt
Analytics Driven Testing
Ori Bendet | Inbound Product Manager, HPE Software |
@bendet_ori
#SrijanWW | @srijan

2. ↓80%Reduction in regression time

3. 7y in HPE Software
in various managerial positions
Started in testing
Today:
Inbound Product
Manager
StormRunner Functional
ABOUT ME

4. AGENDA • Applications Development Overview
• Today’s Testing Challenges
• What are the risks?
• Real World Examples
• How can you empower your testing by
using Analytics

5. The World of Software is changing
Mega-trends create demands on modern applications
MODERN APPLICATION DEVELOPMENT
Reduce costs
Increase customer attraction/retention
Increase the value of your brand
Get to market faster
300,000
tweets
200 million+
emails sent
220,000
new photos posted
50,000 apps
downloaded
$80,000
in online sales
72 hours
of new video content uploaded
2.5 million
pieces of content shared
Agile
Every minute*…
* Source: “The Data Explosion in 2014 Minute-by-Minute”by ACI InformationGroup.
http://aci.info/2014/07/12/the-data-explosion-in-2014-minute-by-minute-infographic/
Mobile
Cloud
Digital
DevOps

6. 8

7. What we need to
test
TestingTime

8. =Quality

9. $59.5BAnnual cost of software defects

10. How to
mitigate the
risk?
• Have more testers
• Crowdsource
• Don’t test
• Let your users test for you
• Or… use Analytics!

11. Analytics

12. Example

13. Example • My product: 30 features
• Supported Matrix: 10 environments
(OS/Browser)
• My testing team: 10 testers
• Testing a feature on an environment = 1 day
• Target regression cycle: <3 days!

14. 30 features X 10 combinations = 300 days
300 days / 10 testers = 30 days

15. Google Analytics • A freemium web analytics service offered by Google
that tracks and reports website traffic.
• Google launched the service in November 2005
after acquiring Urchin.
• Google Analytics is now the most widely used web
analytics service on the Internet.
• Note: I’m not affiliated with Google Analytics in
anyway
* Source: https://w3techs.com/technologies/overview/traffic_analysis/all

16. Environment
Optimization • Focus on the majority of combinations (OS x
Browser)
• Reduce your regression risk to a minimum
• Wait for customer feedback on the other areas

17. Let’s see it live

18. 30 features X 10 2 combinations* = 60 days
60 days / 10 testers = 6 days
(*) with 89%
confidence

19. Example

20. User Behavior • Focus on the majority of the functionality
• Target the most active areas
• Wait for customer feedback on other areas (% of escaped
defects)
• Calculated risk(!)

21. Let’s see it live - again

22. 0 12 features X 10 2 combinations* = 24 day
24 days / 10 testers = 2.4 days
(*) with 76%
confidence

23. Example – part
2

24. Building a risk
calculator • Test everything (100%) = 300 days
• All functionality, top combinations = 60 days
• Main features, top combinations = 24 days
• Add additional levels based on your needs and
time

25. Building a risk
calculator
Top flows
1 OS +
Browser
Main
Functionalitie
s
top
combinations
All
Functionalitie
s
top
combinations
All
Functionalitie
s
Additional
combinations
Test
Everything
50%
75%
90%
100%
95%

26. BewareZombie tests!

28. Don’t have
analytics? • Use market analysis and statistics
• https://netmarketshare.com/
• https://www.w3schools.com/browsers/
• Many more!
• Conduct annual user survey
• CustomersValidations – before & after

29. A word about Test Automation

30. Re-use calculator for
Automation
Top flows
1 OS +
Browser
Main
Functionalitie
s
top
combinations
All
Functionalitie
s
top
combinations
All
Functionalitie
s
Additional
combinations
Test
Everything
50%
75%
90%
100%
95%
CI
Nightly
Full
regression

31. Summary

32. Summary • Don’t test everything!
• Get to know your users
• Calculate your risk
• Measure your un-tested areas
• % of Escaped defects
• Customer Satisfaction