Software Testing and Analysis: Process, Principles, and Techniques (Pezzè and
Why Programs Fail (Zeller)
Software Testing: Principles and Practices(Desikan and Ramesh)
How We Test Software at Microsoft (Page, Johnston, and Rollison)
Reading: PJR Ch. 5
Functional testing uses the specification (formal or
informal) to systematically create tests
Can write tests before code is written
Finds faults that can elude other approaches
Best for catching “missing logic” bugs
Any description of program behavior serving as spec
Any level of granularity from unit to system testing
Functional Testing Goals
Problem: Cannot test all possible input combinations
Need to choose tests…
Key idea: Partition the input space into equivalence
All inputs within the equivalence class have the same
Sufficient to run one test in each equivalence class (not really).
Can be tricky to divide the input space into equivalence
Better to err on the side of too many tests than too few.
Systematic Functional Testing
Failure (valuable test case)
The space of possible input values
Functional Testing Approach
1. Identify independently testable features in the
2. Find equivalence classes for each feature identified
in step 1.
3. Apply boundary value analysis to create one or
more test case specifications for each equivalence
4. Select other potential interesting values based on
previous experience, intuition.
5. Apply combinatorial analyses to create test case
specifications that combine the independently
Independently Testable Features
Decompose subject under test into independently
testable features (ITF).
Each parameter may be an ITF.
Closely coupled parameters may be considered as one
For integration testing:
Each unit is an ITF.
For system testing:
ITFs are exposed through user interfaces or APIs.
ITFs refer to different “phases” of the program.
Class Problem: Testable Features
Consider a multi-
What are the
Need to divide the input space for each ITF into
Based on the desired / specified functionality of the
subject under test.
Computer science students need to meet with their
advisor if their GPA is less than 2.75 or if they are in their
first year at SU.
Income tax rate is dependent on income.
Volume discount if you buy more than 1000 units.
Class Problem: Next Date
The next date program
accepts a date as an
input and returns the
It only works on dates in
the years 1700-3000.
What are equivalence
values for each of the
Boundary Value Analysis
Possible test case
For ordered inputs, history suggests that values near
the boundary are most error prone.
What are good values for an input that expects
integers from -10 to 10?
Boundary Value Analysis
For each boundary n, want to have three cases:
n – 1, n, n +1
For a range of values from x to y, have at least six
x – 1, x, x + 1, y – 1, y , y + 1
Some values are outside of the range:
May also be part of another equivalence class (worry
about redundancy later)
May be illegal - should check to make sure it’s illegal
May be impossible to test
Example: drop-down menu
Is there another way to enter the value?
Boundary analysis can be applied to more situations
than handling numeric data. Examples?
What about “unbounded” data types?
Integers with no specified range
Size of “growing” data structures
What about hidden boundaries?
The size of the window in a GUI
Data type limits
Operating system limits
Within an equivalence class, we have identified
values near the boundary. However, we don’t want
to necessarily stop there.
Should have at least one test in the middle.
Sometimes the boundaries are either trivial or handled
differently in code.
Using prior experience and intuition, look at other
values likely to cause problems:
0 and negative numbers
values of a different type (if possible)
n +1 digit numbers when only n digits are needed
other values based on the subject under test
paying close attention to inputs considered invalid
Catalog Based Testing
Catalogs capture the experience of test designers by
listing important cases for each possible type of
A catalog lists kinds of elements that can occur in a
Each catalog entry is associated with a list of generic
test case specifications
catalog entry Boolean
two test case specifications: true, false
label if applicable only to: input, output, both
A simple catalog (part I)
Each enumerated value in/out
Some value outside the enumerated set in
large negative number in/out
small negative number in/out
small positive number in/out
large positive number in/out
A simple catalog (part II)
Range L ... U
A value between L and U in/out
Numeric Constant C
Any other constant compatible with C in
Non-Numeric Constant C
Any other constant compatible with C in
Some other compatible value in
A simple catalog (part III)
Sequence (such as a sorted linked list)
A single element in/out
More than one element in/out
Maximum length (if bounded) or very long in/out
Longer than maximum length (if bounded) in
Incorrectly terminated in
Traversal for item P in list
P occurs at beginning of list in
P occurs in interior of list in
P occurs at end of list in
P does not occur in list in
Insertion of item P in list
P inserted at beginning of list out
P inserted in interior of list out
P inserted at end of list out
Programs / units generally have more than one
Test cases need to take care of combinations of
A program may fail on a particular combination of inputs
Combinatorial testing: systematically generate
combinations to be tested
Uses the representative values for each ITF.
Example: Next Date
How many different combinations are there?
How could you legitimately reduce the number of
Generally each invalid input case needs to be executed
once (with all other inputs valid)
Overkill to test things like April 0, 3005
Some combinations of valid input values are invalid
Example: February 31
Suffices to test each of these once
Look for uninteresting cases – cases that can be grouped
Example: The year does not matter except in Feb. & Dec.
Error or invalid conditions
Input x is ignored when input y is 0
“Empty” data structure
Even when intelligent constraints are applied, the number
of resulting combinations can be too large.
Pairwise Testing: generate combinations that efficiently
cover all pairs of classes
Rationale: most failures are triggered by single values or
combinations of a few values.
Efficient – fewest number of test cases
Covering pairs reduces the number of test cases, but reveals most
N-way Combinatorial Testing: generate all n-tuples of
N=2 for pairwise testing
Combinatorial Testing of Configurations
A common use of combinatorial testing is for
Other pieces of software
Browser could be “IE” or “Firefox”
Operating system could be “Vista”, “XP”, or “Windows 7”
Systematically generate combinations to be tested
IE on Vista, IE on XP, Firefox on Vista, Firefox on OSX, ...
Example: Pairwise Testing
How many tests are needed such that each pair of
inputs is tested at least once?
OS: XP, Vista, Windows 7
Browser: IE, Firefox
Version of Library X: 1.0, 1.1, 1.2, 2.0
Has program Y installed: Yes, No
Example: One possible solution
Library X OS Browser Program Y?
1.0 XP IE Y
1.0 Vista Firefox Y
1.0 Windows 7 Firefox N
1.1 XP Firefox Y
1.1 Vista Firefox N
1.1 Windows 7 IE N
1.2 XP IE N
1.2 Vista IE Y
1.2 Windows 7 Firefox Y
2.0 XP Firefox N
2.0 Vista IE N
2.0 Windows 7 IE Y
Not always the case where the number of
combinations is the product of the number of cases
of the two inputs with the most cases.
In general, generating a list of combinations is hard
to do manually.
Too many combinations
Too many constraints to contend with
Process can be automated using tools
Tradeoff: manually specifying constraints vs. more tests