Sabermetrics turned the baseball world upside down by challenging decades-old measures of individual performance and their perceived linkage to team success. After cementing their legacy as the Lovable Losers for 108 years, the Chicago Cubs were able to leverage a data-driven approach to finally win a World Series. A high-school football coach, devoted to statistical analysis, has won three state championships—by never punting. Formula 1 racing teams collect staggering amounts of telemetry data from their race cars for the purpose of eking out seconds during the course of a race. Geoff Meyer shares how Dell leverages data from sources as varied as engineering, sales, and the factory to augment test operations. From the identification of high-value test cases and test configurations to determining which test scripts should be retired or re-factored, Geoff explains how you can take advantage of big data to optimize the planning, execution, and communication of your organization’s test efforts.

1.
T1
Test
Management
10/5/17
9:45
Leverage
Big
Data
and
Analytics
for
Testing
Presented
by:
Geoff
Meyer
Dell
EMC
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Geoff
Meyer
Dell
EMC
A
test
architect
in
the
Dell
EMC
infrastructure
solutions
group,
Geoff
Meyer
helps
craft
the
test
strategies
for
Dell
EMC
server,
firmware,
and
software
solutions
and
supports
more
than
400
test
engineers
across
India,
Taiwan,
and
the
United
States.
With
more
than
thirty
years
in
the
software
industry,
Geoff
was
a
developer,
lead
engineer,
manager,
and
test
director
before
finding
his
passion
in
test
architecture.
He
leads
initiatives
in
agile
testing,
Infrastructure
as
a
Service,
continuous
testing,
and
most
recently,
predictive
analytics.
A
member
of
the
Agile
Austin
community,
Geoff
is
an
instructor
and
mentor
to
military
veterans
participating
in
the
Austin-­‐based
Vets4Quality.Org
program.

3. Leverage Big Data and
Analytics for Testing:
Lessons from the Lovable
Losers on Optimizing your
Test Engine
5 October 2017
Geoff Meyer, Test Architect
geoff_meyer@dell.com

4. 2 of 27
Agenda
• The analytics-driven sports world
• What is Data Analytics?
• Analytics applied across Dell
• Tuning Test using Analytics
• Lessons From the Lovable Losers

5. 3 of 27
The Lovable Losers
“I think we have a
pretty good chance to
win if we get more runs
than they do.”
--- Willie Wilson
1908 1992 2004 20161918
Businessinsider.com
? ?
"A modern front office is
trying to convert dollars
into wins"
--- Bill James

6. 4 of 27
Formula One: Data Center in a Garage
Goal: Win races (20 in 2017)
Constraints: New regulations annually
Engineering: Car is an evolving
prototype throughout the season
Massive Data Gathering:
• Simulations
• On-track telemetry

7. 5 of 27
Who Needs a Punter?
Kevin Kelley, the coach that never punts
• Head coach, Pulaski Academy HS
• Derived Contrarian insights
• Actionable decisions
1. His team never punts
2. Always on-side kick
3. Big-play offense
• 3 state championships
By Calvin22
“The most precious commodity in an era of intense
disruption is imagination coupled with courage.” ---
Paul Roehrig, When Machines Do Everything

8. 6 of 27
Predictive Policing & Justice
Not sports, but really cool
Police direct limited
resources to patrol
the areas predicted to
spring up as crime
hot spots
https://www.ted.com/talks/anne_milgram_why_smart_statistics_are_the_key_to_fighting_crime

9. 7 of 27
Report Correlate Predict Recommend
The Analytics Continuum
Descriptive
Diagnostic
Predictive
Prescriptive
What
Happened?
Why did it
Happen?
What will
Happen?
How can we
make it Happen?
Source: Gartner
BusinessValue
Data Analytics Sophistication

10. 8 of 27
Analytics
Engine
Domain
Knowledge
(i.e. Rules)
Data
Sources
Data
Cleanse
Insights
Predictions
Recommendations
Big Data and Analytics Modeling

11. 9 of 27
Where is Analytics being applied at Dell?
Performance Analytics Group
Sales, Pricing and
Product Analytics
Global Audit
Financial Services
Marketing Analytics
Sales Pipeline
Global Services &
Support
Digital Commerce
Services & Analytics
Operations
“As AI is progressively applied, organizations will be able to fix
problems and offer new services at speed, representing a
continuous cycle of improvement.” --- Human-Machine Partnerships

12. 10 of 27
Dell PAG
Hard Drive Failure Prediction

13. 11 of 27
Dell EMC | XtremIO
Duplicate Defect Prediction

14. 12 of 27
Analytics Landscape for Product Engineering
“Pinpoint your Painpoint”
Design Plan Develop Launch/Sustain
• Test Automation Optimization
• Changed-based Regression
• Risk-based Testing
• Duplicate Defect Prediction
• SUT Re-planning
• SUT Planning
• Test Data Planning
• Automation Planning
• Test Case
Planning
• Development
patterns
• Field Issues
• Customer logs

15. 13 of 27
Product Engineering Data Sources

16. 14 of 27
Context at Dell ISG Server Test
Products:
• Remote Management
• Converged Platforms
• Servers
• SW Consoles
Characteristics:
• Large hardware support test matrix
• Software is installed in the Data Center
• Enterprise update cycles ~6 months

17. 15 of 27
How is Analytics applicable to Test Operations?
What are the
high-value SUT
configurations?
What Test Cases
have generated
the most critical
defects?
What is the
release risk
given the testing
that’s been
completed?
What test scripts
should be retired
rather than be
re-factored?
How many
defects can
we expect to
find?
How do we “Moneyball” Test?
What are our
most under-
utilized
systems

18. 16 of 27
Why is Server Test well-suited for Analytics?
• Structured data from our Test operations
• Our Test Automation is generating a reservoir of both
structured and unstructured data
• Continuous Testing promises to generate even more
“As data piles up, we have ourselves a genuine gold rush.”
Eric Siegel,“Predictive Analytics:The Power to Predict Who Will Click, Buy, Lie, or Die”
• User Stories(Jira)
• Test Cases(QMetry)
• Test Suites(QMetry)
• Defects (Jira)
• Test Configurations(QMetry)
• Sales Configuration History (Athena)

19. 17
Analytics
Engine
SUT
Configurations
Methodology
• High-Value is defined by 3 primary
metrics:
• System Quantity Sold
• Requirements
• Restrictions/Slot Config/etc.
• Reference Data Creation:
• Generation mapping
• Attribution standardization
Objective
• Ability to accurately and quickly
prioritize SUT configurations during
planning and test execution
High-value SUT Configurations
High Value SUT Configurations
Platform & FW Test

20. 18 of 27
Methodology
• High-Value is defined by following Rules:
• Config/Test Fail Frequency Score
• Most Recent Config/Test Failures
• Weightage by Release Phases
• Failures resulted a code change
• Most Failed Test& Config Combinations.
• Identify Data Quality Challenges /Gaps
Objective
• Learn from the historical data and
predict the high priority tests and
configurations that will fail
Prioritized list of
high risk SUT
Configs
Prioritized
list of tests.
Prioritized list of
test and config
combinations .

21. 19 of 27
The Dell EMC | Server Test approach
2017
• Proof-of-Concepts
• Close Process/Data
Gaps
• Operationalize:
• SUT Configs
• Test Case
Analytics
2018
• Extend project
coverage
• Unstructured data
• Develop Predictive
Model
• Extend to Dev
systems
2019
• Deploy Predictive
Model
• Extend to Customer
Ops

22. 20 of 27
• What’s your most important
outcome?
• Establish your KPI’s
• Ingrain analytics into your culture
Lesson #1
Lovable Losers & Sabermetrics

23. 21 of 27
• F1 season ~ SDLC
• Capture your test data
• Deploy analytics to make
informed data-driven
decisions
Lesson #2
Formula One

24. 22 of 27
• Analytics aren’t perfect,
but they’re necessary
• Measurement improves
performance
• Going against conventional
wisdom isn’t easy
Lesson #3
Kevin Kelly (Football)

25. 23 of 27
• Analytics to support, not
eliminate, Test Engineers
• Enables more informed
planning decisions
• Provides deeper insights
during test execution
Lesson #4
Predictive Policing & Justice

26. 24 of 27
•Pinpoint your painpoint
•Capture your data
•Find the right Analytics partner
•Be willing to go against
convention
Lessons from the Lovable Losers

27. 25 of 27
Questions?
T1 – Leverage Big Data and Analytics for Testing
Thursday, October 5, 9:45AM – 10:45AM
Geoff Meyer

28. 26 of 27
Resources
• Moneyball: The Art of Winning an Unfair Game (Michael Lewis): https://www.amazon.com/dp/B000RH0C8G/ref=dp-kindle-
redirect?_encoding=UTF8&btkr=1
• Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die (Eric Siegel):
https://www.amazon.com/dp/B019HR9X4U/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1
• Predictive Analytics World: http://www.predictiveanalyticsworld.com/
• Capgemini TestOps: https://www.in.capgemini.com/resource-file-access/resource/pdf/testops_in_product_engineering.pdf
• The Curious Have Won: https://theringer.com/2016-world-series-chicago-cubs-theo-epstein-analytics-war-9f1248c44eb7#.hwa66mzyb
• Why Smart Statistics are the Key to Fighting Crime (Anne Milgram):
http://www.ted.com/talks/anne_milgram_why_smart_statistics_are_the_key_to_fighting_crime?share=17f8c6678a#t-220415
• Red Bull Racing and AT&T share the need for speed: https://www.iothub.com.au/news/red-bull-racing-and-att-share-the-need-for-
speed-417131
• World Quality Report 2016-17 (Capgemini): https://www.capgemini.com/thought-leadership/world-quality-report-2016-17
• What To Do When Machines Do Everything: http://www.whenmachinesdoeverything.com/
• The next era of Human|Machine Partnerships: https://www.delltechnologies.com/en-us/perspectives/realizing-2030.htm