An introduction to black box testing, as part of the Advanced Software Engineering course at www.di.univaq.it/muccini/SE+/2012

1. Università degli Studi dell’Aquila
L21: Black Box/Functional Testing
Henry Muccini
DISIM, University of L’Aquila
www.henrymuccini.com, henry.muccini@univaq.it

2. The material in these slides may be freely reproduced and
distributed, partially or totally, as far as an explicit
reference or acknowledge to the material author is
preserved.
Partially based on material from Alex Orso, Mauro Pezze’,
and Michal Young.

3. AGENDA
White box testing
Representative values
Systematic vs Random Testing
Equivalence Partitioning
Category Partition Method

4. Where is the boundary between testing and other
(analysis/verification) methods?
What distinguishes software testing from “other” approaches
is execution. This requires the ability to:
→ launch the tests on some executable version (traceability problem),
and
→ analyse (observe and evaluate) the results (not granted for embedded
systems)
I am not saying testing is superior and we should disregard other
approaches: on the contrary, testing and other methods
should complement/support each other

5. Any software test campaign involves a trade-off
between
→ Limiting resource utilization (time, effort)
⇒ as few tests as possible
→ Augmenting our confidence
⇒ as many tests as possible
Two research challenges:
→ Determining a feasible and meaningful stopping rule
→ Evaluating test effectiveness (reliability, “coverage notion”,
....very tough problem

6. BLACK BOX/FUNCTIONAL TESTING

7. Functional testing: Deriving test cases from program
specifications
─ Functional refers to the source of information used in test case
design, not to what is tested
Also known as:
→ specification-based testing (from specifications)
→ black-box testing (no view of the code)
Functional specification = description of intended
program behavior
→ either formal or informal
(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

8. Exploit some knowledge to choose samples that are more
likely to include “special” or trouble-prone regions of the
input space
→ Failures are sparse in the whole input space ...
→ ... but we may find regions in which they are dense
Desirable case: Each fault leads to failures that are dense
(easy to find) in some class of inputs
→ sampling each class in the quasi-partition selects at least one
input that leads to a failure, revealing the fault
→ seldom guaranteed; we depend on experience-based heuristics
(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

9. Failures are sparse
Failure (valuable test case) ... but dense in some
in the space of
parts of the space
No failure possible inputs ...
The space of possible input values
(the haystack)
If we systematically test some Functional testing is one way of
cases from each part, we will drawing pink lines to isolate
include the dense parts regions with likely failures
(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

10. Random (uniform):
→ Pickpossible inputs uniformly
→ Avoids designer bias
─ A real problem: The test designer can make the same
logical mistakes and bad assumptions as the program
designer (especially if they are the same person)
→ But treats all inputs as equally valuable
(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

12. Defects are not distributed uniformly!
• Assume our printSum program fails if both
inputs are 0!
• Random sampling is unlikely to choose a = 0
and b = 0!
We want bias!!

13. Systematic (non-uniform):
→ Try to select inputs that are especially valuable
→ Usually by choosing representatives of classes that
are apt to fail often or not at all
Functional testing is systematic testing

15. •An independently testable feature (ITF) is a
functionality that can be tested independently of other
functionalities of the software under test.
•It need not correspond to a unit or subsystem of the
software.
•For example, a file sorting utility may be capable of
merging two sorted files, and it may be possible to test
the sorting and merging functionalities separately,
even though both features are implemented by much
of the same source code.

16. It is recommended to devise separate test cases for
each functionality of the system, whenever possible.
The design of different tests for different
functionalities can simplify the test generation
problem
Thus, identifying functional features does not
correspond to identifying single modules at the design
level, but rather to suitably slicing the specifications to
be able to attack their complexity incrementally
Feature-based view of the system

18. Functional testing uses the specification (formal or
informal) to partition the input space
→ E.g., specification of “roots” program suggests division
between cases with zero, one, and two real roots
Test each category, and boundaries between
categories
→ No guarantees, but experience suggests failures often lie at
the boundaries (as in the “roots” program)
(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

19. •Try to select inputs that are especially valuable!
• Usually by choosing representatives of equivalence classes!
•Assumption: inputs in different categories behave
equivalently
⇒I can select one input per category

20. Consider a software module that is intended to accept the
name of a grocery item and a list of the different sizes the
item comes in, specified in ounces. The specifications state
that the item name is to be alphabetic characters 2 to 15
characters in length. Each size may be a value in the range
of 1 to 48, whole numbers only. The sizes are to be
entered in ascending order (smaller sizes first). A
maximum of five sizes may be entered for each item. The
item name is to be entered first, followed by a comma,
then followed by a list of sizes. A comma will be used to
separate each size. Spaces (blanks) are to be ignored
anywhere in the input.
Taken from: http://users.csc.calpoly.edu/~jdalbey/205/Resources/grocerystore.

21. Consider a software module that is intended to accept the
name of a grocery item and a list of the different sizes the
item comes in, specified in ounces.
The specifications state that the item name is to be alphabetic
characters 2 to 15 characters in length. Each size may be a
value in the range of 1 to 48, whole numbers only.
The sizes are to be entered in ascending order (smaller sizes
first). A maximum of five sizes may be entered for each item.
The item name is to be entered first, followed by a comma,
then followed by a list of sizes. A comma will be used to
separate each size. Spaces (blanks) are to be ignored
anywhere in the input.

22. Item name is alphabetic (valid)
Item name is not alphabetic (invalid)
Item name is less than 2 characters in length (invalid)
Item name is 2 to 15 characters in length (valid)
Item name is greater than 15 characters in length (invalid)
Size value is less than 1 (invalid)
Size value is in the range 1 to 48 (valid)
Size value is greater than 48 (invalid)
Size value is a whole number (valid)
Size value is a decimal (invalid)
Size value is numeric (valid)
Size value includes nonnumeric characters (invalid)
Size values entered in ascending order (valid)
Size values entered in nonascending order (invalid)
No size values entered (invalid)
One to five size values entered (valid)
More than five sizes entered (invalid)
Item name is first (valid)
…

23. Expected
# Test Data Classes Covered
Outcome
1 xy,1 T 1,4,7,9,11,13,16,18,20,22
2 AbcDefghijklmno,1,2,3 ,4,48 T 1,4,7,9,11,13,16,18,20,23
3 a2x,1 F 2
4 A,1 F 3
5 abcdefghijklmnop F 5
6 Xy,0 F 6
7 XY,49 F 8
8 Xy,2.5 F 10
9 xy,2,1,3,4,5 F 14
10 Xy F 15
11 XY,1,2,3,4,5,6 F 17
12 1,Xy,2,3,4,5 F 19
13 XY2,3,4,5,6 F 21
14 AB,2#7 F 12

25. Item name is alphabetic (valid)
Item name is not alphabetic (invalid)
Item name is less than 2 characters in length (invalid)
Item name is 2 to 15 characters in length (valid)
Item name is greater than 15 characters in length (invalid)
Size value is less than 1 (invalid)
Size value is in the range 1 to 48 (valid)
Size value is greater than 48 (invalid)
Size value is a whole number (valid)
Size value is a decimal (invalid)
Size value is numeric (valid)
Size value includes nonnumeric characters (invalid)
Size values entered in ascending order (valid)
Size values entered in nonascending order (invalid)
No size values entered (invalid)
One to five size values entered (valid)
More than five sizes entered (invalid)
Item name is first (valid)
…

26. The main idea is to partition the input domain of
function being tested, and then select test data for
each class of the partition.
The problem of all the existing techniques is the lack of
systematic.
Solution: the Category Partition Method

27. Steps:
ITF (Independently testable feature)
Identify parameters and Environmental conditions
Parameters: Explicit input to the functional unit
Environmental Conditions: Characteristics of the
system’s state
Find categories that characterize each parameter and
environment condition.
Every category should be partitioned into distinct choices
Test Spec

28. Command: find (a pattern into a file)
Syntax: find <pattern> <file>
Function: The find command is used to locate one or more instance of a given
pattern in a text file. All lines in the file that contain the pattern are written to
standard output. A line containing the pattern is written only once, regardless of the
number of times the pattern occurs in it.
The pattern is any sequence of characters whose length does not exceed the
maximum length of a line in the file .To include a blank in the pattern, the entire
pattern must be enclosed in quotes (“).To include quotation mark in the pattern ,two
quotes in a row (“ “) must be used.

29. Example:
find john myfile
display lines in the file myfile which contain john
find “john smith” in myfile
display lines in the file myfile which contain john smith
find “john”” smith” in myfile
display lines in the file myfile which contain john” smith

30. Parameters (explicit input to the functional unit)
Example: size
find john myfile
display lines in the file myfile which contain john
quote emb blank
find “john smith” in myfile
display lines in the file myfile which contain john smith
emb quotes
find “john”” smith” in myfile
display lines in the file myfile which contain john” smith

31. Parameters:
Pattern size:
empty
single character
many character
longer than any line in the file
Quoting:
pattern is quoted
pattern is not quoted
pattern is improperly quoted
Embedded blanks:
no embedded blank
one embedded blank
several embedded blanks

32. Embedded quotes:
no embedded quotes
one embedded quotes
several embedded quotes
File name:
good file name
no file with this name
Environments (characteristics of the system’s state):
Number of occurrence of pattern in file:
none
exactly one Total Tests frames:
more than one 1944
Pattern occurrences on target line:
one
more than one

33. Pattern size : empty
Quoting : pattern is quoted
Embedded blanks : several embedded blanks
Embedded quotes : no embedded quote
File name : good file name
Number of occurrence of pattern in file : none
Pattern occurrence on target line : one
However, not all combinations of value classes
correspond to reasonable test case specifications!!!

34. • Properties
– [property A, B, …]
– A and B are property names
– E.g., [property Empty]
• Selector expression
– [if A]
– E.g., [if Empty]

35. Parameters:
Pattern size:
empty [ property Empty ]
single character [ property NonEmpty ]
many character [ property NonEmpty ]
longer than any line in the file [ property NonEmpty ]
Quoting:
pattern is quoted [ property quoted ]
pattern is not quoted [ if NonEmpty ]
pattern is improperly quoted [ if NonEmpty ]
Embedded blanks:
no embedded blank [ if NonEmpty ]
one embedded blank [ if NonEmpty and Quoted ]
several embedded blanks [ if NonEmpty and Quoted ]

36. Embedded quotes:
no embedded quotes [ if NonEmpty ]
one embedded quotes [ if NonEmpty ]
several embedded quotes [ if NonEmpty ]
File name:
good file name
no file with this name
Environments: Total Tests frames:
Number of occurrence of pattern in file: 678
none [ if NonEmpty ]
exactly one [ if NonEmpty ] [ property Match]
more than one [ if NonEmpty ] [ property Match ]
Pattern occurrences on target line:
one [ if Match ]
more than one [ if Match ]

37. The label [error] indicates a value class that need be
tried only once, in combination with non-error values
of other parameters.

38. Parameters:
Pattern size:
empty [ property Empty ]
single character [ property NonEmpty ]
many character [ property NonEmpty ]
longer than any line in the file [ error ]
Quoting:
pattern is quoted [ property quoted ]
pattern is not quoted [ if NonEmpty ]
pattern is improperly quoted [ error ]
Embedded blanks:
no embedded blank [ if NonEmpty ]
one embedded blank [ if NonEmpty and Quoted ]
several embedded blanks [ if NonEmpty and Quoted ]

39. Embedded quotes:
no embedded quotes [ if NonEmpty ]
one embedded quotes [ if NonEmpty ]
several embedded quotes [ if NonEmpty ]
[ single ]
File name:
good file name
no file with this name [ error ]
Environments: Total Tests frames:
Total Tests frames:
Number of occurrence of pattern in file: 40
125
none [ if NonEmpty ]
exactly one [ if NonEmpty ] [ property Match]
more than one [ if NonEmpty ] [ property Match ]
Pattern occurrences on target line: [ single ]
one [ if Match ]
more than one [ if Match ]
[ single ]

40. • Use a constraint solver
• Choose specific values that satisfy the
constraints

42. A completely different approach:
→We cannot realistically presume to find and remove the last
failure.
→Then, we should invest test resources to find out the
“biggest” ones.
“Goodness” here means to minimise the user
perceived faults, i.e.
→Tryto find earlier those bugs that are more frequently
encountered (neglect “small” bugs)
→Not suitable for safety critical applications

43. Identify:
• Typical usage scenario
• Frequency of those scenarios
Test more, more frequent scenarios