software engineering

1. Whitebox Testing
CS 420 - Spring 2007

2. A Testing Life Cycle
Requirement
Specs
Design
Error
Coding
Testing
Fix
Fault
Resolution
Fault
Isolation
Fault
Classification
Error
Fault
Fault
Fault
Error
incident

3. Terminology
• Error
– Represents mistakes made by people
• Fault
– Is result of error. May be categorized as
• Fault of Commission – we enter something into
representation that is incorrect
• Fault of Omission – Designer can make error of
omission, the resulting fault is that something is
missing that should have been present in the
representation

4. Cont…
• Failure
– Occurs when fault executes.
• Incident
– Behavior of fault. An incident is the
symptom(s) associated with a failure that
alerts user to the occurrence of a failure
• Test case
– Associated with program behavior. It carries
set of input and list of expected output

5. Cont…
• Verification
– Process of determining whether output of one
phase of development conforms to its
previous phase.
• Validation
– Process of determining whether a fully
developed system conforms to its SRS
document

6. Verification versus Validation
• Verification is concerned with phase
containment of errors
• Validation is concerned about the final
product to be error free

7. Relationship – program
behaviors
Program Behaviors
Specified
(expected)
Behavior
Programmed
(observed)
Fault Behavior
Of
Omission
Fault
Of
Commission
Correct portion

8. Classification of Test
• There are two levels of classification
– One distinguishes at granularity level
• Unit level
• System level
• Integration level
– Other classification (mostly for unit level) is
based on methodologies
• Black box (Functional) Testing
• White box (Structural) Testing

9. Relationship – Testing wrt
Behavior
Program Behaviors
5 6
Specified
(expected)
Behavior
Programmed
(observed)
Behavior
2
1
4 3
Test Cases
(Verified behavior)
8 7

10. Test methodologies
• Functional (Black box) inspects specified
behavior
• Structural (White box) inspects
programmed behavior

11. Functional Test cases
Specified Programmed
Test
Cases

12. Structural Test cases
Specified Programmed
Test
Cases

13. When to use what
• Few set of guidelines available
• A logical approach could be
– Prepare functional test cases as part of
specification. However they could be used
only after unit and/or system is available.
– Preparation of Structural test cases could be
part of implementation/code phase.
– Unit, Integration and System testing are
performed in order.

14. Unit testing – essence
• Applicable to modular design
– Unit testing inspects individual modules
• Locate error in smaller region
– In an integrated system, it may not be easier
to determine which module has caused fault
– Reduces debugging efforts

15. Test cases and Test suites
• Test case is a triplet [I, S, O] where
– I is input data
– S is state of system at which data will be input
– O is the expected output
• Test suite is set of all test cases
• Test cases are not randomly selected.
Instead even they need to be designed.

16. Need for designing test cases
• Almost every non-trivial system has an
extremely large input data domain thereby
making exhaustive testing impractical
• If randomly selected then test case may
loose significance since it may expose an
already detected error by some other test
case

17. Design of test cases
• Number of test cases do not determine the
effectiveness
• To detect error in following code
if(x>y) max = x; else max = x;
• {(x=3, y=2); (x=2, y=3)} will suffice
• {(x=3, y=2); (x=4, y=3); (x=5, y = 1)}
• Each test case should detect different errors

18. White-Box Testing
• Statement coverage
• Branch coverage
• Path coverage
• Condition coverage
• Data flow-based testing

19. Statement Coverage
• Statement coverage methodology:
– design test cases so that every statement in a
program is executed at least once.
• The principal idea:
– unless a statement is executed, we have no
way of knowing if an error exists in that
statement

20. Statement coverage criterion
• Observing that a statement behaves
properly for one input value:
– no guarantee that it will behave correctly for
all input values.

21. Example
• int f1(int x, int y){
1. while (x != y){
2. if (x>y) then
3. x=x-y;
4. else y=y-x;
5. }
6. return x; }

22. Euclid's GCD computation
algorithm
• By choosing the test set
{(x=3,y=3),(x=4,y=3), (x=3,y=4)}
– all statements are executed at least once.

23. Branch Coverage
• Test cases are designed such that:
– different branch conditions is given true and
false values in turn.
• Branch testing guarantees statement
coverage:
– a stronger testing compared to the statement
coverage-based testing.

24. Example
• Test cases for branch coverage can be:
{(x=3,y=3), (x=4,y=3), (x=3,y=4)}

25. Condition Coverage
• Test cases are designed such that:
– each component of a composite conditional
expression given both true and false values.
• Example
– Consider the conditional expression
((c1.and.c2).or.c3):
– Each of c1, c2, and c3 are exercised at least
once i.e. given true and false values.

26. Branch testing
• Branch testing is the simplest condition
testing strategy
• compound conditions appearing in
different branch statements are given true
and false values. (note: the entire
condition is given true and false values,
not ALL possible sub expressions)

27. Branch testing
• Condition testing
– stronger testing than branch testing:
• Branch testing
– stronger than statement coverage testing.

28. Condition coverage
• Consider a Boolean expression having n
components:
– for condition coverage we require 2n test
cases.
• practical only if n (the number of
component conditions) is small.

29. Path Coverage
• Design test cases such that:
– all linearly independent paths in the program
are executed at least once.
• Defined in terms of
– control flow graph (CFG) of a program.

30. Control flow graph (CFG)
• A control flow graph (CFG) describes:
– the sequence in which different instructions of
a program get executed.
– the way control flows through the program.

31. How to draw Control flow
graph?
• Number all the statements of a program.
• Numbered statements:
– represent nodes of the control flow graph.
• An edge from one node to another node
exists:
– if execution of the statement representing the
first node can result in transfer of control to
the other node.

32. Example
int f1(int x,int y){
1. while (x != y){
2. if (x>y) then
3. x=x-y;
4. else y=y-x;
5. }
6. return x; }

33. Example Control Flow Graph
1
2
3 4
5
6

34. Path
• A path through a program:
– A node and edge sequence from the starting
node to a terminal node of the control flow
graph.
– There may be several terminal nodes for
program.

35. Independent path
• Any path through the program:
– introducing at least one new node or one new
edge that is not included in any other
independent paths.
• It may be straight forward to identify
linearly independent paths of simple
programs. However For complicated
programs it is not so easy to determine the
number of independent paths.

36. McCabe's cyclomatic metric
• An upper bound:
– for the number of linearly independent paths
of a program
• Provides a practical way of determining:
– the maximum number of linearly independent
paths in a program.

37. McCabe's cyclomatic metric
• Given a control flow graph G,
cyclomatic complexity V(G):
– V(G)= E-N+2
• N is the number of nodes in G
• E is the number of edges in G

38. Example
• Cyclomatic complexity =
7 – 6 + 2 = 3.

39. Cyclomatic complexity
• The cyclomatic complexity of a program
provides:
– a lower bound on the number of test cases to
be designed to get coverage of all linearly
independent paths.
– only gives an indication of the minimum
number of test cases required.

40. Path Testing - Test Cases
• Draw control flow graph. (Loops can
cause an explosion in the number of
independent paths, so use looping values
of 0 and 1 to test loops).
• Determine V(G).
• Determine the set of linearly independent
paths.
• Prepare test cases:
– to force execution along each path

41. Example Control Flow Graph
1
2
3 4
5
6

42. Derivation of Test Cases
• Number of independent paths: 4
– 1, 6 test case (x=1, y=1)
– 1, 2, 3, 5, 1, 6 test case(x=1, y=2)
– 1, 2, 4, 5, 1, 6 test case(x=2, y=1)

43. An interesting application of
cyclomatic complexity
• Relationship exists between:
– McCabe's metric
– the number of errors existing in the code,
– the time required to find and correct the
errors.

44. Cyclomatic complexity
• Cyclomatic complexity of a program:
– also indicates the psychological complexity of
a program.
– difficulty level of understanding the program.

45. Cyclomatic complexity
• From maintenance perspective,
– limit cyclomatic complexity
• of modules to some reasonable value.
– Some software development organizations:
• restrict cyclomatic complexity of functions to some
max number.

46. Data-flow-based Testing
• Basic idea: test the connections between
variable definitions (“write”) and variable uses
(“read”)
• Starting point: variation of the control flow graph
– Each node represents a single statement, not a chain
of statements
• Set DEF(n) contains variables that are defined at
node n (i.e., they are written)
• Set USE(n): variables that are read

47. Example
Assume y is already initialized
1 s:= 0;
2 x:= 0;
3 while (x<y) {
4 x:=x+3;
5 y:=y+2;
6 if (x+y<10)
7 s:=s+x+y;
else
8 s:=s+x-y;
1 2 3
DEF(1) := {s}, USE(1) :=
DEF(2) := {x}, USE(2) :=
DEF(3) := , USE(3) := {x,y}
DEF(4) := {x}, USE(4) := {x}
DEF(5) := {y}, USE(5) := {y}
DEF(6) := , USE(6) := {x,y}
DEF(7) := {s}, USE(7) := {s,x,y}
DEF(8) := {s}, USE(8) := {s,x,y}
DEF(9) := , USE(9) :=
DEF(10) := , USE(10) :=
4
5
6
7 8
9
10

48. Reaching Definitions
A definition of variable x at node n1
reaches node n2 if and only if there is
a path between n1 and n2 that does
not contain a definition of x
1 2 3
DEF(1) := {s}, USE(1) :=
DEF(2) := {x}, USE(2) :=
DEF(3) := , USE(3) := {x,y}
DEF(4) := {x}, USE(4) := {x}
DEF(5) := {y}, USE(5) := {y}
DEF(6) := , USE(6) := {x,y}
DEF(7) := {s}, USE(7) := {s,x,y}
DEF(8) := {s}, USE(8) := {s,x,y}
4
5
6
7 8
9
10
Reaches
nodes
2,3,4,5,6,7,8,
but not 9 and
10.

49. Def-use Pairs
• A def-use pair (DU) for variable x is a pair of
nodes (n1,n2) such that
– x is in DEF(n1)
– The definition of x at n1 reaches n2
– x is in USE(n2)
• In other words, the value that is assigned to x at
n1 is used at n2
– Since the definition reaches n2, the value is not killed
along some path n1...n2.

50. Examples of Def-Use Pairs
1 2 3
DEF(1) := {s}, USE(1) :=
DEF(2) := {x}, USE(2) :=
DEF(3) := , USE(3) := {x,y}
DEF(4) := {x}, USE(4) := {x}
DEF(5) := {y}, USE(5) := {y}
DEF(6) := , USE(6) := {x,y}
DEF(7) := {s}, USE(7) := {s,x,y}
DEF(8) := {s}, USE(8) := {s,x,y}
4
5
6
7 8
9
10
Reaches nodes 2, 3, 4, 5, 6,
7, 8, but not 9,10
For this
definition, two
DU pairs:
1-7, 1-8

51. Data-flow-based Testing
• Identify all DU pairs and construct test cases
that cover these pairs
– Several variations with different “relative strength”
• All-DU-paths: For each DU pair (n1,n2) for x,
exercise all possible paths n1, n2 that are clear
of a definition of x
• All-uses: for each DU pair (n1,n2) for x,
exercise at least one path n1 n2 that is clear of
definitions of x

52. Data-flow-based Testing
• All-definitions: for each definition, cover at
least one DU pair for that definition
– i.e., if x is defined at n1, execute at least one path
n1..n2 such that x is in USE(n2) and the path is clear
of definitions of x
• Clearly, all-definitions is subsumed by all-uses
which is subsumed by all-DU-paths
• Motivation: see the effects of using the values
produced by computations
– Focuses on the data, while control-flow-based
testing focuses on the control