The document discusses pairwise testing, an effective combinatorial testing technique that focuses on identifying bugs triggered by interactions between pairs of parameters. It covers the implementation, advantages, limitations, and tools available for pairwise testing while emphasizing its role in reducing the number of test cases significantly. Although beneficial, the method may overlook interactions involving more than two variables, making it less suitable for mission-critical software.

1. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
Tes*ng:
What
It
Is,
When
to
Use
and
Not
to
Use
#PairOrNot
#PairWiseTes*ng
@XBOSo=
@philiplew
@QAIGlobal

2. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Meet Your Instructor
• Philip Lew
– Software developer
– Software product manager, DW product
– CEO, software QA services company
• Relevant specialties and education
– Software Usability / UX design and
evaluation
– Software Measurement and Metrics
– Cornell University B.S. and M.Eng in
Operations Research
2
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.

3. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Today’s
Discussion
3
1. The
nuts
and
bolts
of
pairwise
tes*ng
2. How
to
incorporate
into
your
test
design
and
planning
3. Introduc*on
to
tools
for
pairwise
tes*ng
– ACTS
4. When
and
when
not
to
use
pairwise
tes*ng
– Advantages
and
limita*ons
5. Going
beyond
pairwise
tes*ng,
taking
it
to
the
next
level
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.

4. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
WE
LIVE
INTO
OUR
EXPECTATIONS
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
4

5. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Session Spirit and Expectations
• I can’t read…
• Slides for you as a take-away (many)
– You can write me and I’ll send these to
you
• Interactive
– Lots of exercises and chances to learn
5
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.

6. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
HOW
MANY
OF
YOU
USE
A
COMBINATORIAL
TESTING
TECHNIQUE?
YES/NO
POLL
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
6

7. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Today’s
Discussion
7
1. The
nuts
and
bolts
of
pairwise
tes*ng
2. How
to
incorporate
into
your
test
design
and
planning
– Some
tools
for
pairwise
tes*ng
3. When
and
when
not
to
use
pairwise
tes*ng
– Advantages
and
limita*ons
4. Going
beyond
pairwise
tes*ng,
taking
it
to
the
next
level
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.

8. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
The
Problem
• Many
defects
only
occur
with
one
factor
or
variable.
• Or
when
a
combina*on
of
2
or
more
inputs
or
events
occur
that
interact
with
each
other.
• Tes*ng
every
combina*on
is
impossible.
• What
common
situa*ons
have
you
run
into
this
problem?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
8

9. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Some
Examples
• Plaaorm
stacks
• So=ware
configura*on
op*ons
• Mobile
plaaorm
• Any
others?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
9

10. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Typical
Problem
• An
applica*on
with
simple
list
box
with
10
elements
(Let's
say
0,1,2,3,4,5,6,7,8,9)
along
with
a
checkbox,
radio
bugon,
Text
Box
and
OK
Bugon.
The
Constraint
for
the
Text
box
is
it
can
accept
integer
values
only
between
1
and
100.
– List
Box
-­‐
0,1,2,3,4,5,6,7,8,9
– Check
Box
-­‐
Checked
or
Unchecked
– Radio
Bugon
-­‐
ON
or
OFF
– Text
Box
–
1
to
100
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
10

11. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
How
Many
Test
Cases?
• List
Box
=
10
• Check
Box
=
2
• Radio
Bugon
=
2
• Text
Box
=
100
• Total
Number
of
Test
Cases
using
Cartesian
Method
:
10*2*2*100
=
4000
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
11
How can we reduce the #
of test cases?

12. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Using
Equivalence
Par**oning
• We
can
consider
the
list
box
values
as
0
and
others
• Radio
bugon
and
check
box
values
cannot
be
reduced.
Each
one
of
them
will
have
2
combina*ons
(ON
or
OFF).
• The
Text
box
values
can
be
reduced
into
three
inputs
(Valid
Integer,
Invalid
Integer,
Alpha-­‐Special
Character).
• Calcula*ng
the
number
of
cases
:
2*2*2*3
=
24
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
12
That’s still a lot L

13. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Problems
with
Equivalence
Par**oning
• For
tes*ng
printers
instead
of
tes*ng
with
every
single
model
of
printer,
we
can
set
all
Hewleg-­‐
Packard
inkjet
printers
as
roughly
equivalent
as
a
par**on,
and
all
Canon
inkjets
as
another.
Or
sepasrate
by
printer
type,
laser
versus
inkjet…
• This
can
save
us
a
*me
and
energy,
BUT
• We
must
know
how
to
create
classes
-­‐
what
is
equivalent
to
what?
• We
must
understand
the
technology
and
domain
to
be
tested
to
create
valid
par**ons.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
13

14. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
Tes*ng
–
(All-­‐Pairs
Tes*ng)
• The
reasoning:
the
simplest
bugs
in
a
program
are
generally
triggered
by
a
single
input
parameter/
variable.
• The
next
simplest
category
of
bugs
consists
of
those
dependent
on
interac*ons
between
pairs
of
parameters,
which
can
be
discovered
with
pairwise/
all-­‐pairs
tes*ng.
• Bugs
involving
interac*ons
between
three
or
more
parameters
are
less
common,
and
also
more
expensive-­‐harder
to
find.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
14

15. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Let’s
Use
Pairwise
1. Order
the
parameters
such
that
one
with
most
number
of
values
is
the
first
and
the
least
is
placed
as
the
last
variable.
2. Now,
start
filling
the
table
column
by
column.
Textbox
(3).
3. The
next
column
under
discussion
would
be
check
box.
Again,
Check
box
can
take
2
values.
List
box
can
take
2
values.
4. Now,
we
need
to
ensure
that
we
cover
all
combina*ons
between
list
box
and
Check
box.
5. Now,
we
will
use
the
same
strategy
for
checking
the
Radio
Bugon.
It
can
take
2
values.
6. Verify
if
all
the
pair
values
are
covered.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
15
Listbox (0, Others)
Radio (On, Off)
Checkbox (Check, Uncheck)
Textbox(valid Int, Invalid int, Alpha)

16. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
Results
• Exhaus*ve
Combina*on:
>
4000
Test
Cases.
• Equivalence
par**oning:
24
Test
Cases.
• Pair
Wise:
6
Test
Cases.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
16
TEXT
BOX
LIST
BOX
CHECK
BOX
RADIO
BUTTON
VALID
INT
0
CHECK
ON
VALID
INT
OTHERS
UNCHECK
OFF
INVALID
INT
0
UNCHECK
ON
INVALID
INT
OTHERS
CHECK
OFF
ALPHA
0
CHECK
OFF
ALPHA
OTHERS
UNCHECK
OFF

17. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
LET’S
GO
AT
IT
AGAIN!
Actually,
enumera*ng
all
pairs
is
a
mathema*cal
challenge…
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
17

18. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
TesDng
Problem:
Test
System
S
Inputs
X
=
1,2
Y
=
Q,R
Z
=
5,6
There
are
23
=
8
possible
combina*ons
System S
X Y Z
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
18

19. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
TesDng
8 Test Cases
You can get rid of Test Case 2 (T2)
• {1, -, 5} is covered in T1
• {1, R, -} is covered in T4
• { -, R, 5} is covered in T6
Test
ID
X
Y
Z
T1
1
Q
5
T2
1
R
5
T3
1
Q
6
T4
1
R
6
T5
2
Q
5
T6
2
R
5
T7
2
Q
6
T8
2
R
6
1, R,
R, 5
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
19
Do
This
Now:
• How
many
more
test
cases
can
you
get
rid
of?
• List
the
Test
cases
you
would
keep.
• Time
yourself.

20. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
TesDng
Savings
Test
ID
X
Y
Z
T1
1
Q
5
T2
1
R
5
T3
1
Q
6
T4
1
R
6
T5
2
Q
5
T6
2
R
5
T7
2
Q
6
T8
2
R
6
Test
ID
X
Y
Z
T1
1
Q
5
T4
1
R
6
T6
2
R
5
T7
2
Q
6
8 Test Cases Pairwise testing
4 Test Cases
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
20
Darn, we only save ½
½ is a lot but what does this tell us?

21. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
TOOLS
MAKE
THINGS
EASIER
Luckily…
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
21

22. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
No
Lack
of
Tools
• ACTS
-­‐
AUTOMATED
COMBINATORIAL
TESTING
FOR
SOFTWARE
(ACTS)
• Automa*c
Efficient
Test
Generator
or
AETG™
• Allpairs
-­‐
Free
downloadable
executable
wrigen
in
python.
• All
Pairs
-­‐
Free.
Command-­‐line
executable
based
on
a
Perl
script.
By
Sa*sfice.
• Hexawise
-­‐
Comprehensive
pairwise
and
combinatorial
test
genera*on
solu*on.
So=ware
as
a
service.
Free
demo
accounts.
• Jenny
-­‐
Free.
Executable
(C
code).
• pairwise
-­‐
Free,
open
source
using
Ruby.
• PICT
(Pairwise
Independent
Combinatorial
Tes*ng)
tool
• rdExpert
–
commercial
so=ware.
Free
trial
available.
• SmartTest
Commercial,
executable
so=ware.
• Testcover
-­‐
Web
based.
Free
trial.
• VPTag
-­‐
Free
downloadable
executable.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
22

23. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
23

24. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
24

25. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
25

26. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
ACTS
–
Results
Output
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
26

27. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
ACTS
–
Results
Output
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
27

28. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
Tes*ng
1. Pairwise
tes*ng
protects
against
pairwise
(2
factor)
bugs.
2. Drama*cally
reduces
the
number
tests
to
perform.
3. Pairwise
bugs
represent
the
majority
of
combinatory
bugs
4. Available
tools
means
you
no
longer
need
to
create
these
tests
by
hand.
28
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
28
Good Stuff, That’s All I need to know right?

29. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
WE’RE
DONE
FOR
TODAY
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
29

30. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
LIMITATIONS
OF
PAIRWISE
TESTING
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
30

31. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Limita*ons
of
Pairwise
Tes*ng
1. Assumes
random
distribu*on
of
tests
with
no
focus
on
important
tests
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
31

32. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Applying
Simple
Network
Theory
• How
many
nodes
do
you
need
to
kill
to
take
down
the
network?
• Why
did
you
choose
the
nodes
you
did?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
32

33. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Blind
Implementa*on
of
Pairwise
Tes*ng
• Represents
a
random
agack
on
the
network
• Does
not
focus
on
hub
nodes
(important
areas
or
defect-­‐laden)
• What
are
your
‘hub’
nodes
in
your
so=ware?
– Where
is
it
vulnerable
to
failure?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
33

34. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Limita*ons
of
Pairwise
Tes*ng
1. Assumes
random
distribu*on
of
tests
with
no
focus
on
important
tests
2. Some
parameters
are
con*nuous
variables
and
need
to
be
discre*zed
to
dis*nct
values.
3. Some
variables
are
dependent
on
others.
4. How
to
determine
the
correct
result
that
should
be
expected
for
each
of
the
combina*ons
(000’s)
of
test
inputs?
34
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
34

35. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Pairwise
Tes*ng
Fails
When
• Not
selec*ng
the
right
input.
• You
don’t
have
a
good
enough
oracle
(you
don’t
know
what
is
the
correct
result).
• High
probability
combina*ons
not
covered
enough.
• Not
knowing
how
the
variables
interact
(constraints
and
dependencies).
35
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
35

36. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Where Were We?
________
________
_________
36
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
Problem
Pairwise
Testing
Combinatorial
Testing

37. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Combinatorial
Tes*ng
• Pairwise
Tes*ng
is
a
subset
of
Combinatorial
tes*ng
• Pairwise
(2-­‐way
combina*ons)
tes*ng
is
some*mes
used
to
obtain
reasonably
good
results
at
low
cost,
but
pairwise
tes*ng
may
miss
10%
to
40%
of
the
bugs.
• Not
sufficient
for
mission-­‐cri*cal
so=ware.
• Empirical
data
gathered
by
NIST
and
others
suggest
that
so=ware
failures
are
triggered
by
only
a
few
variables
interac*ng
(6
or
fewer).
• Combinatorial
tes*ng
beyond
2-­‐way
has
been
limited,
due
to
a
lack
of
good
algorithms
for
higher
interac*on
levels
(4-­‐6)
• New
algorithms
have
made
combinatorial
tes*ng
beyond
pairwise
prac*cal.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
37

38. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
38
https://en.wikipedia.org/wiki/All-pairs_testing

39. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
What
If?
• What
if
a
failure
is
triggered
by
a
combina*on
of
3,
4,
or
more
variables?
• Pairwise
tests
may
not
detect
these
defects.
• Is
tes*ng
all
4-­‐way
combina*ons
(T=4)
enough
to
detect
all
errors?
• What
degree
of
interac*on
(strength)
,
i.e.
2,3,4,5..
occurs
in
real
failures?
• What
is
my
tolerance
for
risk?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
39

40. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
T-­‐Way
Tes*ng
#
Interac*ng
Parameters
Leading
to
Failure
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
40
What is your risk tolerance? What is it dependent on?

41. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
“FLY
WITH
US,
WE
FOUND
90%
OF
THE
DEFECTS”.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
41
I’ll fly with you!

42. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
How
much
risk
can
we
tolerate?
• At
first
glance?
• 210
=
1,024
possible
combina*ons!
• To
reduce
that,
we
can
use
T-­‐Way
according
to
our
goals
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
42
About 90% with 3-Way Interactions

43. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
APPLYING
COMBINATORIAL
TESTING
TECHNIQUES
Input
Parameters
(Variables)
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
43

44. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Word
Processing
Input
Variables
• User
can
select
10
ways
to
modify
some
highlighted
text:
subscript,
superscript,
underline,
bold,
italic,
strikethrough,
emboss,
shadow,
small
caps,
or
all
caps.
• System
must
process
the
input
and
modify
the
text
on
the
screen
correctly.
• Most
op*ons
can
be
combined,
such
as
bold
and
small
caps,
but
some
are
incompa*ble,
such
as
subscript
and
superscript.
• How
many
combina*ons?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
44

45. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
APPLYING
COMBINATORIAL
TESTING
TECHNIQUES
Mobile
Tes*ng
Plaaorm
prolifera*on
AND
OS
op*ons
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
45

46. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Android
Op*on
Prolifera*on
Parameter
Values
#
Values
HardKeyboardHidden
No,
Undefined,
Yes
3
KeyboardHidden
No,
Undefined,
Yes
3
Keyboard
12Key,
NoKeys,
Querty,
Undefined
4
Naviga*onHidden
No,
Undefined,
Yes
3
Naviga*on
Dpad,
NoNav,
Trackball,
Undefined,
Wheel
5
Orienta*on
Landscape,
Portrait,
Square,
Undefined
4
ScreenLayout_Long
Mask,
No,
Undefined,
Yes
4
ScreenLayout_Size
Large,
Mask,
Normal,
Small,
Undefined
5
TouchScreen
Finger,
Notouch,
Stylus,
Undefined
4
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
46
3x3x4x3x5x4x4x5x4=? 172,800
What
Would
You
Reduce?

47. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
What
Would
You
Reduce?
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
47
Parameter
Values
#
Values
HardKeyboardHidden
No,
Undefined,
Yes
3
KeyboardHidden
No,
Undefined,
Yes
3
Keyboard
12Key,
NoKeys,
Querty,
Undefined
4
Naviga*onHidden
No,
Undefined,
Yes
3
Naviga*on
Dpad,
NoNav,
Trackball,
Undefined,
Wheel
5
Orienta*on
Landscape,
Portrait,
Square,
Undefined
4
ScreenLayout_Long
Mask,
No,
Undefined,
Yes
4
ScreenLayout_Size
Large,
Mask,
Normal,
Small,
Undefined
5
TouchScreen
Finger,
Notouch,
Stylus,
Undefined
4

48. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
How
Can
T-­‐Way
Help?
t
#
Tests
%
of
Total
(172k)
%
Defects
Found
2
29
.02
76%
3
137
.08
95%
4
625
0.4
97%
5
2532
1.5
99%
6
9168
5.3
100%
**
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
48
• The
savings
as
a
percentage
of
exhaus*ve
tes*ng
depends
on
the
number
of
parameters
and
the
#
of
values
for
the
parameters.
• With
larger
systems
the
savings
can
be
enormous.

49. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Key
Factors
When
Using
T-­‐Way
• Number
of
parameters
• Number
of
values
per
parameter
• T
(also
called
interac*ons,
where
T=2
for
pairwise)
• Number
of
tests
increases
rapidly
as
the
value
of
t
is
increased
• Rate
of
increase
depends
on
the
number
of
values
per
parameter.
– Binary
variables,
result
in
far
fewer
tests
than
parameters
with
many
values
each.
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
49

50. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
APPLYING
COMBINATORIAL
TESTING
TECHNIQUES
Configura*on
Tes*ng
-­‐
Let’s
go
back
and
look
at
this
again!
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
50

51. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Configura*on
Tes*ng
-­‐
Plaaorms
How
many
configura*ons
are
there?
• OS
(Windows,
Apple
OSX,
Linux)
• Browser
(IE,
Firefox)
• Protocol
(IPv4,
IPv6)
• Processor
(Intel,
AMD)
• Database
(MySQL,
Sybase,
Oracle)
• What
others
are
you
used
to?
• How
many
possible
plaaorms?
• 3x2x2x2x3=72
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
51

52. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Configura*on
Tes*ng
–
Plaaorms
Pairwise
Configura*ons
(WOW!)
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
52
Test
ID
OS
Browser
Protocol
CPU
DBMS
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1
1
Win
MySQL
IE
IPv4
Intel
10
2
Win
Sybase
FFX
IPv6
AMD
10
3
OSX
MySQL
FFX
IPv4
AMD
8
4
OSX
Sybase
IE
IPv6
Intel
8
5
Linux
Oracle
IE
IPv4
AMD
8
6
Linux
Oracle
FFX
IPv6
Intel
7
7
Win
MySQL
~IE
IPv6
~AMD
1
8
Win
Sybase
~FFX
IPv4
~Intel
1

53. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Variable
Interac*ons
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
53
• Some
combina*ons
don’t
exist
or
invalid
– OSX
and
IE
• Can’t
simply
delete
them,
why?
• Need
to
delete
and
supplement
• What
other
interac*ons
or
interdependencies
can
you
think
of?

54. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
USING
TOOLS
Using
ACTS
Demonstra*on
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
54

55. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Configura*ons
Revisited
Parameter
(variable)
Values
OS
Windows,
OSX,
Linux
Browser
IE,
Firefox
Protocol
IPv4,
IPv6
CPU
Intel,
AMD
DBMS
MySQL,
Sybase,
Oracle
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
55

56. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Using
ACTS
t
#
tests
%
of
Total
%
defects
found
2
10
14
76%
3
18
25
95%
4
36
50
97%
5
72
100
99%
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
56
We could test all these platforms. It’s a lot, but doable

57. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
ACTS
with
T=3
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
57

58. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Invalid
Combina*ons
-­‐
Interac*ons
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
58
Test
ID
OS
Browser
Protocol
CPU
DBMS
1
Win
IE
IPv4
Intel
MySQL
2
Win
FFX
IPv6
AMD
Sybase
3
Win
IE
IPv6
Intel
Oracle
4
OSX
FFX
IPv4
AMD
MySQL
5
OSX
IE
IPv4
Intel
Sybase
6
OSX
FFX
IPv4
Intel
Oracle
7
Linux
IE
IPv6
AMD
MySQL
8
Linux
FFX
IPv4
Intel
Sybase
9
Linux
FFX
IPv4
AMD
Oracle
10
OSX
FFX
IPv6
AMD
Oracle

59. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
ACTS
with
T=2
and
Contraints
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
59

60. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Using
Pairwise
and
T-­‐Way
Together
• Using
Pairwise
can
help
us
find
many
…
but
not
all
and
depending
on
our
risk
tolerance,
maybe
not
enough
• Using
T-­‐Way
can
help,
but
each
T+1
adds
a
lot
of
effort…
• Best
to
use
T-­‐Way
with
modified
set
of
parameters
and
values
according
to
our
experience
and
exper*se
based
on
the
situa*on
and
goals
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
60

61. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Understand
Pairwise
and
T-­‐Way
in
Perspec*ve
• Where
do
you
have
many
combina*ons?
• Where
do
you
have
dis*nct
inputs?
Either
naturally
or
ability
to
create
equivalence
classes.
• Where
do
you
have
many
parameter
values?
• Know
the
expected
results
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
61
Test
Strategy
Test
Design
T-­‐Way

62. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Combining
Combinatorial
Applica*ons
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
62
System
Under
Test
Inputs
(test
cases)
Configura*ons

63. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Numerous
Ways/Places
We
can
use
Combinatorial
Tes*ng
Telecomm
How
many
configura*ons
are
there?
• Types
of
calls
(local,
LD,
interna*onal)
• Types
of
payment
(phone
card,
800,
900,
credit
card,
monthly
bill)
• Access
(ISDN,
T1,
VOIP,
PBX)
• Billing
Server
(Windows,
Linux/MySQL,
Oracle)
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
63

64. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Reading
Material
1. All
Pairs
Tes*ng,
Wikipedia,
hgps://en.wikipedia.org/wiki/All-­‐pairs_tes*ng
2. AUTOMATED
COMBINATORIAL
TESTING
FOR
SOFTWARE
(ACTS),
NIST,
hgp://csrc.nist.gov/groups/SNS/acts/index.html
3. Combinatorial
Tes*ng:
Ra*onale
and
Impact,
Rick
Kuhn,
hgp://csrc.nist.gov/groups/SNS/acts/documents/kuhn-­‐icst-­‐14.pdf
4. Design
of
Experiments,
Wikipedia,
hgps://en.wikipedia.org/wiki/Design_of_experiments
5. Efficient
Tes*ng
Using
the
Pairwise
Approach,
Bernie
Berger,
hgp://www.s*ckyminds.com/ar*cle/efficient-­‐tes*ng-­‐using-­‐pairwise-­‐approach
6. An
Inves*ga*on
of
the
Applicability
of
Design
of
Experiments
to
So=ware
Tes*ng,
D.
Richard
Kuhn,
M.
Reilly,
hgp://csrc.nist.gov/staff/Kuhn/kuhn-­‐reilly-­‐02.pdf
7. Linked:
How
Everything
Is
Connected
to
Everything
Else
and
What
It
Means
for
Business,
Science,
and
Everyday
Life,
Albert-­‐Laszlo
Barabasi
8. Pairwise
Tes*ng,
A
Best
Prac*ce
That
Isn’t,
James
Bach,
hgp://www.tes*ngeduca*on.org/wtst5/PairwisePNSQC2004.pdf
9. Pairwise
Tes*ng
for
Dummies,
David
Fries,
hgp://blogs.msdn.com/b/davfries/archive/2004/06/17/158900.aspx
10. Pairwise
Tes*ng
in
the
Real
World,
hgps://msdn.microso=.com/en-­‐us/library/cc150619.aspx
11. Pairwise
Tes*ng,
Michael
Bolton,
hgp://www.developsense.com/pairwiseTes*ng.html
12. PRACTICAL
COMBINATORIAL
TESTING,
D.
Richard
Kuhn,
Raghu
N.
Kacker,
Yu
Lei,
Na*onal
Ins*tute
of
Standards
and
Technology
13. Tutorial
on
All
Pairs
Tes*ng,
hgp://www.tutorialspoint.com/so=ware_tes*ng_dic*onary/all_pairs_tes*ng.htm
©
2016
XBOSo=,
Inc.-­‐
All
Rights
Reserved.
64

65. #PairOrNot
#PairWiseTesting
@XBOSoft @philiplew
Thanks
Ques*ons
and
Answers
Philip Lew
@philiplew
philip.lew@xbosoft.com
Some resources: http://www.xbosoft.com/knowledge_center/
www.xbosoft.com