My talk in the STP Online Summit in March 2013. I take a deep dive into the PICT tool to combinatorially generate test cases, working through an example application. Originally published at http://www.softwaretestpro.com (where summit attendees can hear the recorded session).

1. 3/12/2013
To Infinity & Beyond! Using Combinatorial
Testing to Handle Complexity
Presented By:
Andy Tinkham
Magenic Technologies
Testing Everything is Impossible
 The number of test cases might as well be infinite
 Interacting variables have a multiplicative effect
1

2. 3/12/2013
Consider this screen…
How many test cases What about when What about all the
are needed to ensure additional settings rest of the
that this all works are added to the poll functionality of this
correctly? creation process? app?
Additional complexity
• MyVote runs on iOS, Windows 8, and
Windows Phone 8 (and multiple devices for
those systems!)
• Users can log in with Twitter, Facebook, and
Microsoft ID
• Testing every combination of these is already
over 18000 test cases
2

3. 3/12/2013
We need ways to manage this
Many ways we could handle this
Test everything (delaying
Give up release while we run all
18000+ test cases)
Choose a subset of tests
to run & hope we chose Automate the tests
well
Combine tests to get a
minimal test set that
covers the interactions
One way…
• Combinatorial testing
• Simply multiplying out the test
cases results in a lot of duplication
• Many bugs require interactions of
variables
• Solution: Create set of tests to
cover all pairs of variables
interacting
3

4. 3/12/2013
When to use combinatorial tests
• You’ve got a set of interacting variables that
result in a large number of test cases
• You understand these variables well
• The variables are independent or you can
explicitly define the dependencies
Pitfalls
• Choosing the wrong variables or values can
leave gaps in your testing
• Relying solely on combinatorial testing may
leave important combinations untested
• In at least 1 academic study, combinatorial
testing didn’t do much better than random
testing
4

5. 3/12/2013
Variables
• What might vary in a test case
• Focus on one piece of functionality not whole system (use
multiple models)
• Use domain analysis skills
• Look for anything that has an impact on the results for the
functionality you’re testing
• May already have identified many as part of previous test
design efforts
Identifying variables
Function Application Data
Input values Environment
parameters settings conditions
Data
User Input Configuration User settings User role
structures
Values from
Presence or Global
other Branch points Data state
absence of configuration
components
5

6. 3/12/2013
Identifying values
• Analyze variable list one by one
• Identify default values
• Identify equivalence classes for possible values
• Name equivalence classes rather than choosing
representative values for now
6

7. 3/12/2013
Example List for Add a Poll
• Question: Present, Not Present
• NumberOfAnswers: 0, 1, 2, 3, 4, 5
• CanSelectMultipleAnswers: Yes, No
• Image: None, JPG, GIF, PNG
• StartTime: None, Past, Present, Future
• EndTime: None, Past, Present, Future
Making the set
• List each value of the first variable
– Question: Present, Not Present
• Then add the second variables values so that
each pair of values occurs
– NumberOfAnswers: 0, 1, 2, 3, 4, 5
7

8. 3/12/2013
Question NumberOf CanSelectMultiple Image StartTime EndTime
Answers Answers
Present
Present
Present
Present
Present
Present
Not Pres
Not Pres
Not Pres
Not Pres
Not Pres
Not Pres
Question NumberOf CanSelectMultiple Image StartTime EndTime
Answers Answers
Present 0
Present 1
Present 2
Present 3
Present 4
Present 5
Not Pres 0
Not Pres 1
Not Pres 2
Not Pres 3
Not Pres 4
Not Pres 5
8

9. 3/12/2013
Continuing the set
• Repeat for the remaining variables, keeping
pair relationships up as you go
• When you can’t get a needed pair, first look
for partial rows and set unknown values if
possible
• If no partial row where you can set the
needed value, add a new row
– Mark currently non‐needed values as unknown
Question NumberOf CanSelectMultiple Image StartTime EndTime
Answers Answers
Present 0 Yes None None None
Present 1 No GIF Past Past
Present 2 Yes JPG Present Present
Present 3 No PNG Future Future
Present 4 Yes … … …
Present 5 No
Not Pres 0 No
Not Pres 1 Yes
Not Pres 2 No
Not Pres 3 Yes
Not Pres 4 No
Not Pres 5 Yes
9

10. 3/12/2013
Creating the set
For a detailed description of the manual
process, check out the Addendum to Chapter 3
in Kaner, Bach & Pettichord’s Lessons Learned in
Software Testing
Automating the process
• Luckily, we don’t have to do this by hand!
• Many tools available to create the test set for
us
– Bach’s AllPairs tool (satisfice.com)
– Hexawise
– Microsoft’s PICT
• Let’s look at PICT
10

11. 3/12/2013
PICT
• Free tool, developed at Microsoft
• Windows based
– Other platforms have similar tools, see
http://pairwise.org for more info
• Command‐line based
• Reads in text files containing variables, values
& context then outputs test cases
• See References slide for download &
documentation links
Setting up variables in PICT
• Create a text file
• List each variable on a separate line, followed
by a :
• List the values after the :, separated by
commas
11

12. 3/12/2013
Example
Question: Present, Not Present
Number Of Answers: 0, 1, 2, 3, 4, 5
Can Select Multiple Answers: Yes, No
Image: None, GIF, JPG, PNG
StartTime: None, Past, Present, Future
EndTime: None, Past, Present, Future
Output
Question Number Of Can Select Image Start Time End Time
Answers Multiple
Answers
Not Present 1 Yes PNG None Present
Not Present 5 No None Past None
Present 5 Yes GIF Present Future
Present 5 No JPG Future Past
Not Present 2 No JPG None Future
Not Present 3 Yes GIF Past Past
Present 0 No GIF None None
Present 4 No GIF Future Present
Not Present 0 Yes None Future Future
Present 1 No PNG Past Future
… (16 more) … … … … …
12

13. 3/12/2013
Adding constraints
• Can’t have one time stamp be None while the
other is set
• Don’t edit generated files – you might lose
coverage of all the pairs
• Instead, add constraints to PICT model
13

14. 3/12/2013
Example Model File
Question: Present, Not Present
Number Of Answers: 0, 1, 2, 3, 4, 5
Can Select Multiple Answers: Yes, No
Image: None, GIF, JPG, PNG
StartTime: None, Past, Present, Future
EndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";
IF [EndTime] = "None" THEN [StartTime] = "None";
Output
Question Number Of Can Select Image Start Time End Time
Answers Multiple
Answers
Not Present 4 Yes PNG Past Past
Present 5 No JPG Present Future
Not Present 0 Yes None Future Future
Not Present 4 No GIF None None
Present 2 Yes None Present Present
Present 5 No PNG Future Present
Not Present 4 Yes JPG Past Present
Present 3 Yes None None None
Present 0 No GIF Past Past
Not Present 3 No JPG Future Past
… (19 more) … … … … …
14

15. 3/12/2013
Positive & negative values
• Not all the test cases generated are expected
to pass due to invalid values
– Question Not Present
– Answers 0 & 1
• We may not want multiple invalid values in a
test case
• Solution: Tell PICT which values are invalid
Example Model
Question: Present, ~Not Present
Number Of Answers: ~0, ~1, 2, 3, 4, 5
Can Select Multiple Answers: Yes, No
Image: None, GIF, JPG, PNG
StartTime: None, Past, Present, Future
EndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";
IF [EndTime] = "None" THEN [StartTime] = "None";
15

16. 3/12/2013
Output
Question Number Of Can Select Image Start Time End Time
Answers Multiple
Answers
… (13 more) … … … … …
Present 3 Yes JPG Past Future
Present 5 Yes GIF None None
Present 4 No None None None
Present 4 No JPG Present Future
Present 4 No JPG Future Present
~Not Present 4 Yes PNG Past Past
Present ~0 Yes None Future Future
~Not Present 2 No JPG Past Present
~Not Present 3 Yes GIF None None
… (14 more) … … … … …
Weighting values
• Not every value occurs in the real world with
the same frequency
• Once pairs are satisfied, PICT still has some
“don’t care” values in table
• “Don’t care” values are filled in with random
selections from the variable
• We can weight the more likely variables so
PICT chooses them more often
16

17. 3/12/2013
Example Model
Question: Present, ~Not Present
Number Of Answers: ~0, ~1, 2, 3, 4 (10), 5
Can Select Multiple Answers: Yes, No (10)
Image: None, GIF, JPG (10), PNG
StartTime: None, Past, Present, Future
EndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";
IF [EndTime] = "None" THEN [StartTime] = "None";
Seeding Test Cases
• There may be some test combinations that we
want to ensure are included
– Default options
– Most common configuration
• If more than 2 variables involved in desired
combination, PICT may not automatically
generate that test
• We can give the test to PICT as a starting point
17

18. 3/12/2013
Creating seed file
• Make a separate file
• Format is same as output from PICT
– Lines tab‐delimited
– First line has variable names in order
– Remaining lines have desired test case values
• Seeded test cases can specify all values for test or
just some (PICT will populate the remaining
values
• File name is passed in to PICT command line call
with /e: option
Example Seed File
Question Number Of Can Select Image Start Time End Time
Answers Multiple
Answers
Not Present 4 Yes PNG Past Past
Present No GIF
18

19. 3/12/2013
Generating tests
• Install PICT
• Add directory with PICT.exe to PATH
environment variable
• From a Command Prompt, run
pict.exe model_file.txt [options] > output_file.txt
PICT Options
• /o:N ‐ Order of combinations (default: 2)
• /d:C ‐ Separator for values (default: ,)
• /a:C ‐ Separator for aliases (default: |)
• /n:C ‐ Negative value prefix (default: ~)
• /e:file ‐ File with seeding rows
• /r[:N] ‐ Randomize generation, N ‐ seed
• /c ‐ Case‐sensitive model evaluation
• /s ‐ Show model statistics
19

20. 3/12/2013
Test Automation with Combinatorics
• Generated test cases can be fed in to custom‐
built parser & executed
• Using names for equivalence classes allows
you to generate data on the fly
– Pick different values every time
– Find data that matches needs in current
environment
• Variable values can also be keyword actions
References
• PICT download
http://download.microsoft.com/download/f/5/5/f55484df‐8494‐48fa‐
8dbd‐8c6f76cc014b/pict33.msi
• PICT documentation
http://msdn.microsoft.com/en‐us/library/cc150619.aspx
• More information on pair‐wise testing
http://pairwise.org
• Bach & Schroeder’s comparison of pairwise &
randomly selected test cases
http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
20

21. 3/12/2013
Contact Information
Andy Tinkham
Magenic Technologies
andyt@magenic.com
http://magenic.com
http://ohours.org/andytinkham
http://testerthoughts.com
http://twitter.com/andytinkham
21