In the right hands, combinatorial testing can be a powerful testing strategy to reduce the number of tests. Unfortunately, when used improperly, it can hurt more than it helps. Ingo Philipp explains that there is not one single combinatorial testing technique that guarantees success. Rather, success lies in creating a tight interaction among all of the techniques, since each is intended to serve a different purpose. In nearly every test project, even small test sets are overrun with countless business-related dependencies. The number and nature of these dependencies do not necessarily bother test experts—until the maintenance of a test set becomes a problem. Join Ingo to find a way out of this dilemma, and learn how you can customize combinatorial testing techniques —without coding—in a business readable fashion. Discover how to achieve the highest possible risk coverage as early as possible. See how eliminating redundant test cases reduces maintenance and prevents costly manual cleanup of test data sets. Learn how to provide useful documentation of your combinatorial test sets including statements of what and why something has to be tested.

1.
T20
Test
Techniques
10/6/16
15:00
Smart
Combinatorial
Testing
Presented
by:
Ingo
Phillip
Tricentis
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2. Ingo
Phillip
After
several
years
working
as
a
scientist
in
particle
physics
and
computational
fluid
dynamics,
Ingo
Philipp
immersed
himself
in
the
multifaceted
worlds
of
functional
software
testing
and
development.
Ingo
is
on
the
product
management
team
at
Tricentis
where
his
responsibilities
range
from
product
development
and
product
marketing
to
test
management,
test
conception,
test
design,
and
test
automation.
His
experiences
with
software
testing
embrace
the
application
of
agile
as
well
as
classic
testing
methodologies
in
the
insurance,
banking,
commerce,
telecommunications,
and
energy
sectors.

3. © 2016 by
Ingo Philipp
© 2016 by .© 2016 by .
Smart Combinatorial Testing

4. © 2016 by
It’s me, a problem!
Start End
software delivery cycle
67%
average level of redundancy in
enterprise test portfolios
40%
average risk coverage achieved in
enterprise test portfolios
90%
of all test automation is
UI test automation
80%
overall testing effort goes
into manual testing
55%
of systems only partially
accessible by Dev/Test
56%
of overall test effort goes into
test case maintenance
30%
of bugs found in acceptance
& production stage
50%
of manual testing goes into test
data preparation & organization
Albert Einstein, 1921
“The first step to solve a problem is to accept that you have one.”
Is it just about speed?

5. © 2016 by
By compressing the delivery cycle,
do problems just move closer to each other?
Is it just about speed?
Start End
software delivery cycle

6. © 2016 by
Start End
software delivery cycle
No, they mutually reinforce each other!
The devil is in
the combination!
It’s about speed@quality.
Hence, the biggest strength of DevOps is not solving
problems, but rather exposing buried problems.

7. © 2016 by
A
B
It’s about speed@quality.
Automate
Optimize
Manage

8. © 2016 by
Do the right things right.
Testing harder isn’t the answer, testing smarter is.
Wolfgang Platz

9. © 2016 by
Divide &
Conquer2
Structure &
Prioritize1
3
Explore &
Optimize
Smart
Combinatorial
Testing
Conceptual Model
Why to test at all?
Concrete Model
How to explore the missing what?
Abstract Model
What to test to cover the why?

10. © 2016 by
Low
Risk
High
Risk
Medium
Risk
80%20%
Business Risk
Coverage
Test
Cases
The time needed for
testing is infinitely larger
than the time available.
Critical Limit
Risk-Coverage Optimization
Have the right test cases.
Do the right things.

11. © 2016 by
User Stories
Product Backlog
User Story
User Story
User Story
User Story
User Story
User Story
User Story
Risk-Based Testing

12. © 2016 by
User
Stories
Epics
Epic
User Story
Epic
User Story
User Story
User Story
User Story
Product Backlog
Risk-Based Testing

13. © 2016 by
User
Stories
Epics
User Story
Theme
Epic
Epic
User Story
Epic
Epic
Risk-Based Testing
Product Backlog

14. © 2016 by
Sprint
Planning
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
Sprint #N
N.1 As a role, I want…
N.2 As a role, I want…
N.3 As a role, I want…
N.4 As a role, I want…
N.5 As a role, I want…
N.6 As a role, I want…
N.7 As a role, I want…
N.8 As a role, I want…
1 N
User Stories
tasks, spikes, chores
Sprint Backlogs
User
Stories
Epics
Product Backlog
I have this brand
new idea.
Risk-Based Testing

15. © 2016 by
Main Goal Intermediate Goals
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
User Stories
testable, measurable items
Epics
Sprint #N
N.1 As a role, I want…
N.2 As a role, I want…
N.3 As a role, I want…
N.4 As a role, I want…
N.5 As a role, I want…
N.6 As a role, I want…
N.7 As a role, I want…
N.8 As a role, I want…
Risk-Based Testing
1 N
Sprint ViewsProduct View
Sprint
Planning

16. © 2016 by
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
User Stories
testable, measurable items
Sprint #N
N.1 As a role, I want…
N.2 As a role, I want…
N.3 As a role, I want…
N.4 As a role, I want…
N.5 As a role, I want…
N.6 As a role, I want…
N.7 As a role, I want…
N.8 As a role, I want…
Traceability
Relationships
EpicsGovernance
Risk-Based Testing
1 N
Sprint ViewsProduct View

17. © 2016 by
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
Agile Team #1
Sprint #N
N.1 As a role, I want…
N.2 As a role, I want…
N.3 As a role, I want…
N.4 As a role, I want…
N.5 As a role, I want…
N.6 As a role, I want…
N.7 As a role, I want…
N.8 As a role, I want…
Governance
Risk-Based Testing
Traceability
Relationships
1 N
Sprint ViewsProduct View

18. © 2016 by
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
Agile Team #1
Agile Team #2
Agile Team #3
Agile Team #K
Governance
Risk-Based Testing
1 N•K
Sprint ViewsProduct View

19. © 2016 by
80%
27%
24%
1.5%
1.5%
53%
10%
10%
Securities Trading
Capture Order
Client Side Validation
Market Side Validation
Check Eligibility
Check Suitability
Check Availability
Rectify Order
Cancel Order
Business Risk
Contribution
Business Risk
Coverage %
39 12 18 31
60 2 6 32
33 17 50
100
42 21 37
34 5016
1025 25 40
18 22 3030
41 6 9 45
Product
Increment
Theme
Epic
User Story
Test Case
Governance
Risk-Based Testing
Product View

20. © 2016 by
It’s time to get insured.
Combinatorial Testing Approaches

21. © 2016 by
All Possible Combinations
Create Quote
Age
Gender
Payload [kg]
Vehicle Type
Country
Fuel Type
List Price
Mileage Per Year
Engine Performance
Start Date
Insurance Sum
Payment Option
Damage Insurance
Euro Protection
Defense Insurance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Antieconomical approach.
Test case count grows exponentially faster than risk coverage.
Test Objective is unknown.
Practical significance only for small scale testing missions.
Root cause analysis is a herculean task.
Test objective is the entire test case portfolio.
Exponential growth in test cases.
Time, resource & cost intensive.
Maximum risk coverage.
Equals exhaustive testing, and so leads to combinatorial explosion.
Good
Bad
Bad
Bad
Bad
32768test cases
≫ each with 2 instances ≪

22. © 2016 by
All Pairwise Combinations
Create Quote
Age
Gender
Payload [kg]
Vehicle Type
Country
Fuel Type
List Price
Mileage Per Year
Engine Performance
Start Date
Insurance Sum
Payment Option
Damage Insurance
Euro Protection
Defense Insurance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Numerous meaningless test cases.
Manual clean-up without decrease in risk coverage is virtually impossible.
All pairs are equally important.
Testing not focused around most important criteria.
Root cause analysis is a herculean task.
Test cases are highly condensed, and so hard to maintain.
Manifold test objective.
Multiple pairs covered in a test case, i.e. no unique test goal.
Masters combinatorial explosion.
Logarithmic growth in attributes & quadratic growth in instances.
Good
Bad
Bad
Bad
Bad
≫ each with 2 instances ≪
20test cases cover 420pairs

23. © 2016 by
Each Choice Coverage Criterion
Create Quote
Age
Gender
Payload [kg]
Vehicle Type
Country
Fuel Type
List Price
Mileage Per Year
Engine Performance
Start Date
Insurance Sum
Payment Option
Damage Insurance
Euro Protection
Defense Insurance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Enormous maintenance problems.
Doing a lot with a little is just elusive.
False statements about risk contribution.
Useless for risk-based approach on test case level.
Root cause analysis is a herculean task.
Maintainability, changeability & understandability are enemies.
Test Objective is unknown.
Test case goals are most versatile and out of all reason.
Minimal number of test cases.
Guaranteed that each instance is used at least once.
Good
Bad
Bad
Bad
Bad
≫ each with 2 instances ≪
2test cases cover 30singles

24. © 2016 by
Linear Expansion
Create Quote
Age
Gender
Payload [kg]
Vehicle Type
Country
Fuel Type
List Price
Mileage Per Year
Engine Performance
Start Date
Insurance Sum
Payment Option
Damage Insurance
Euro Protection
Defense Insurance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Case #1
(24,59)
Male
[1,1000]
Truck
>59
Case #2
Male
[1,1000]
Truck
Female
Case #3
(24,59)
[1,1000]
Truck
>1000
Case #4
(24,59)
Male
Truck
Case #5
(24,59)
Male
[1,1000]
Car
Case #6
(24,59)
Male
[1,1000]
Motorbike
“Nothing is perfect, life is messy, outcomes are uncertain, people are irrational, relationships are complex”
Creates slightly more test cases.
Linear increase in test case count up to about 95% risk coverage.
Good
Assigns a unique test objective.
Strongly supports changeability, maintainability & understandability.
Good
Enables to derive risk contribution.
Best to apply risk-based approach on test case level.
Good
Makes root cause analysis an easy task.
Test smarter, not harder and keep it simplistic instead of complex.
Good
Case #7
(24,59)
Male
[1,1000]
Trailer
Assumes attribute independence.
About 20% interdependent attributes, and so the logic partly breaks down.
Bad
≫ each with 2 instances ≪
16test cases cover 30singles

25. © 2016 by
Customized Linear Expansion
Create Quote
Age
Gender
Payload [kg]
Vehicle Type
Country
Fuel Type
List Price
Mileage Per Year
Engine Performance
Start Date
Insurance Sum
Payment Option
Damage Insurance
Euro Protection
Defense Insurance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Case #1
(24,59)
Male
[1,1000]
Truck
>59
Case #2
Male
[1,1000]
Truck
Female
Case #3
(24,59)
[1,1000]
Truck
>1000
Case #4
(24,59)
Male
Truck
Case #5
(24,59)
Male
[1,1000]
Car
Case #6
(24,59)
Male
[1,1000]
Motorbike
Case #7
(24,59)
Male
[1,1000]
Trailer
Meaningless
combinations
Missing
combinations
Business Rules
not linked to test cases
DesignLiving
Documentation
“Nothing is perfect, life is messy, outcomes are uncertain, people are irrational, relationships are complex”
Assumes attribute independence.
About 20% interdependent attributes, and so the logic partly breaks down.
Bad
≫ each with 2 instances ≪

26. © 2016 by
Risks
Your system under test.
Performance Issue
Usability Issue
Functional Issue
Stability Issue
Reliability IssueScalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
Why are we missing these issues?
≫ What do we have to do to proactively find them? ≪

27. © 2016 by
Risks
Automated Testing
I’m an automated test case.
Usability Issue
Stability Issue
Reliability IssueScalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
Performance Issue
Functional Issue

28. © 2016 by
Automated Testing
1 2 3 4 5 … … … … … … …
Risks

29. © 2016 by
Manual Testing
I’m the same test case executed manually.
Risks

30. © 2016 by
Risks
Story-Based
Motivating
Credible
Exploratory Testing
Exploratory Branching
New testing ideas continually occur
during exploratory testing.
New
Testing
Idea Exploratory Testing
Perfect counterpart to specification-based
automated & manual testing.
Test Cases vs. Scenarios
Pre-specified inputs vs. hypothetical situation

31. © 2016 by
Specification
Based Testing
Exploratory Testing
Exploratory Testing
Analyze Potential Risks
Problem vs. No Problem
Monitor Known Risks
Pass Result vs. Fail Result
Effective testing results from the combination of both.
*That’s an illustration. Don’t confuse it with reality.
Increase your testing cross
section. Actively attack risk.
Go beyond the obvious.
Diversify your testing.
Enrich your test design.
Find more critical bugs.
Fast error detection.
Provide rapid feedback.
Make your testing
intellectually rich.
Core Benefits

32. © 2016 by
Customer Case Study
If this doesn’t make an impact,
then its absence doesn’t make a difference.
Trey Smith

33. © 2016 by
100%
Manual
Testing
0%
Automated
Testing
Testing
Present
Customer Case Study
There’s a way to do it
better. Find it!
Project Charter
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution

34. © 2016 by
Risk Coverage Optimization Test Data Management
48
Hours
Execution
Single Agent
8
Hours
Execution
Multiple Agents
Customer Case Study
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution
89%
Business Risk
Coverage
1193
Manual
Test Cases
37%
Business Risk
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
53%
Business Risk
Coverage
Smoke Testing
34
Minutes
Execution
Smoke Testing
Test Case Count 7%
Automation (UI & API) & OSV Continuous Integration & Distributed Execution
75% Redundancy 50% Effort Test Data
64% API Tests

35. © 2016 by
8
Hours
Execution
Multiple Agents
Customer Case Study
72%
Critical Defect Reduction
Production Defects
89%
Business Risk
Coverage
1193
Manual
Test Cases
5
Weeks
Execution
2.5
Weeks
Execution
92%
Automated
Test Cases
34
Minutes
Execution
Smoke Testing
53%
Business Risk
Coverage
Smoke Testing
Test Case Count 7%
Test Case Design & Exploratory Testing
Risk Coverage Optimization Test Data Management Automation (UI & API) & OSV Continuous Integration & Distributed Execution
64% API Tests

36. © 2016 by
Bottom Line
The future starts today, not tomorrow.
Start remodeling your traditional approaches for greater agility today.