Software Engineering Testing techniques

1. Software Engineering
(CSI 321)
Testing techniques
1

2. Software Testing
• Testing is the process of executing a program with
the intent of finding errors.
• A good test case is one with a high probability of
finding an as-yet undiscovered error.
• A successful test is one that discovers an as-yet-
undiscovered error.
2

3. Testing Principles
• All tests should be traceable to customer requirements.
• Tests should be planned long before testing begins.
• The Pareto Principle applies to software testing(80% of all
errors are in 20% of the code)
• Testing should begin “in the small” and progress towards “in
the large”.
• Exhaustive testing is not possible.
• Testing should be conducted by an independent third party if
possible.
3

4. Software Testability Checklist
 What are the characteristics of software testability?
 Software testability is simply how easily a computer program can be
tested
1) Operability
– if it works better it can be tested more efficiently
1) Observability
– “what you see is what you test”
– Source code is accessible
– System states and variables are visible or query-able during
execution.
1) Controllability
– The better we can control the software, the more the testing can be
automated and optimized.
4

5. Software Testability Checklist
4) Decomposability
– By controlling the scope of testing, we can more quickly isolate
problems and perform smarter retesting.
5) Simplicity
–The less there is to test, the more quickly we can test it
6) Stability
– The fewer the changes, the fewer the disruptions to testing
7) Understandability
– The more information is known, the smarter the testing can be
done
5

6. Attributes of a Good Test
 What is a “good” test?
• The following are the attributes of a good test –
– A good test has a high probability of finding an error.
– A good test is not redundant.
– A good test should be “best of breed”.
– A good test should not be too simple or too complex.
6

7. Black-Box Testing & White-Box Testing
 Two basic categories of testing techniques –
1) White Box Testing
2) Black Box Testing
7

8. White Box Testing
• View components as transparent
• Based on knowledge of the internal logic
• Done by programmers (usually)
• AKA: Structural testing, Glass-box testing, Clear-box
testing
8

9. White Box Testing
• Knowing the internal workings of a product, tests are performed to check
the workings of all possible logic paths
• White box testing uses the control structure described as part of the
component level design to derive test cases.
• Using white box testing methods, the software engineer can derive test
cases that:
– Guarantee that all independent paths within a module have been
exercised at least once.
– Exercise all logical decisions on their true and false sides.
– Execute all loops at their boundaries and within their operational
bounds.
– Exercise internal data structures to ensure their validity.
9

10. Black-Box Testing
• View components as opaque
• Based on requirements and functionality
• Without any knowledge of internal design, code or
language.
• Structure of the program is not considered.
• Test cases are designed based on the requirements
or specification of the program or module.
• AKA Functional Testing, Behavioral Testing
10

11. Detailed Test Case Design
11
• A combination of black box and white techniques are applied
depending on the phase of testing - generally white box at the
unit testing level, black box during acceptance testing, and
both during integration and system testing.
 Black Box Methods
• Equivalence partitioning
• Boundary value analysis
• Error Guessing
 White Box Methods
• Basis path testing
• Statement Coverage
• Branch Coverage
• Condition Coverage
• Multiple-condition Coverage

12. Basis Path Testing
• Basis Path Testing is a white box test case design technique
that uses the algorithmic flow of the program to design tests.
• Basis path testing helps a tester to compute logical
complexity (also called cyclomatic complexity) measure, V(G),
of the code. This value of V(G), defines the maximum number
of test cases to be designed by identifying basis set of
execution paths to ensure that all statements are executed at
least once.
– Basis Set is the set of tests derived using Basis Path Testing. These are
set of test cases derived from the internal structure or specification to
ensure that 100% of a specified coverage criterion is achieved.
12

13. Basis Path Testing
• A White-box testing technique first proposed by Tom
McCabe.
• Determine the basis set of linearly independent
paths (the cardinality of this set is the program
cyclomatic complexity).
• Prepare test cases that will force the execution of
each path in the basis set.
13

14. BASIS PATH TESTING
• Goal: Each statement is executed at least once.
• Consider all paths of program execution. For this purpose it is
useful to represent a program as graph. Nodes are blocks of
code, directed edges are invocations of other code blocks and
represent flow of control. Nodes with two exiting rows
contain conditional statements. The graph contains initial and
terminal nodes.
• A path is a sequence of nodes connecting initial and terminal
nodes; an independent path is a path introducing at least one
new node.
14

15. BASIS PATH TESTING
• The basis path method enables the test case designer
to derive a logical complexity measure of a procedural
design and use this measure as a guide for defining a
basis set of execution paths.
• Test cases derived to exercise the basis set are
guaranteed to execute every statement in the program
at least one time during testing.

16. BASIS PATH TESTING
FLOW GRAPH NOTATION
A flow graph depicts logical control flow using the notation shown below:
Each structured construct corresponds to a flow graph symbol.

17. BASIS PATH TESTING
• Flow charts and flow graphs correspond to one another.
The flow chart depicts the program control structure.
The flow graph assumes no compound structures.

18. BASIS PATH TESTING
• Flow charts and flow graphs correspond to one another.
• Each circle, called a flow graph node, represents one or
more procedural statements.
• A sequence of process boxes and a decision diamond can
map to a single node.
• The arrows of a flow graph, called edges or links, represent
flow of control.
• An edge must terminate at a node.
• Areas bounded by edges are called regions.
18

19. BASIS PATH TESTING
19

20. BASIS PATH TESTING
Compound Conditions:
• Compound Conditions occur when one or more Boolean operators
(OR, AND, NAND, NOR) is/are present in a conditional statement.
• IF a OR b is represented as follows:
FIGURE 14.3

21. BASIS PATH TESTING
• Independent Program Paths: An independent program path is
any path through the program that introduces at least one
new set of processing statements or a new condition.
• When stated in terms of a flow graph, an independent path
must move along at least one edge that has not been
traversed before the path is defined.
• For example, a set of independent paths for the flow graph
illustrated in Figure 14.2b is:
• Path 1: 1-11
• Path 2: 1-2-3-4-5-10-1-11
• Path 3: 1-2-3-6-8-9-10-1-11
• Path 4: 1-2-3-6-7-9-10-1-11

22. BASIS PATH TESTING
• A basis set for a flow graph is the set of paths that cover
every statement in the program.
• Therefore Paths 1, 2, 3, & 4 are the basis set for the
previous figure.
• If tests are designed to force execution of these paths,
every statement is guaranteed to execute at least one
time. Every condition will have been executed on its true
and false sides.
• Note: A basis set is not unique. A number of basis sets
may be derived from a procedural design.
• How do we know how many paths to look for?

23. BASIS PATH TESTING
• The number of independent paths can be used to measure
the complexity of a program. It is called cyclomatic
complexity.
• Cyclomatic complexity is an upper bound for a base set of
path, i.e. a set of path that includes each node.
• If each path is translated into a test case, each statement
will be tested at least once.
• Cyclomatic complexity is a software metric that provides a
quantitative measure of the logical complexity of a
program.

24. What is the significance of the value of V(G)?
• Cyclomatic complexity, V(G) is a useful metric for
predicting those modules that are likely to be error
prone.
– Use it for test planning as well as test case design.
• Often components with a high V(G) are a high risk for
error and should be tested more completely.
• In general, V(G) > 10 is not acceptable.
24

25. BASIS PATH TESTING
 Cyclomatic complexity is computed using a number
of ways:
1)The number of regions corresponds to the cyclomatic
complexity.
2)Cyclomatic complexity, V(G), for a flow graph G, is defined as:
V(G) = E – N + 2, where E is the number of flow graph edges,
and N is the number of flow graph nodes.
3)Cyclomatic complexity, V(G), for a flow graph G, is defined as:
V(G) = P + 1, where P is the number of predicate nodes
contained in the flow graph G.
25

26. 26
Cyclomatic Complexity
11
1111
4,54,5
1010
8877
66
2,32,3
99
1
2
3
6
7 8
4
5
9 10
11
Flow chart Flow graph
Path 1: 1-11
Path 2: 1-2-3-4-5-10-1-11
Path 3: 1-2-3-6-8-9-10-1-11
Path 4: 1-2-3-6-7-9-10-1-11
R1
R2
R3
R4 (region)
Predicate node
Cyclomatic complexity
1. Flow graph has 4 regions
2. V(G) = 11 edges - 9 nodes +2 = 4
3. V(G) = 3 predicate nodes + 1 = 4

27. BASIS PATH TESTING
1. The flow graph has 4 regions. So, V(G) = 4
2. V(G) = 11 edges – 9 nodes +2 = 4
3. V(G) = 3 predicate nodes + 1 = 4
27

28. BASIS PATH TESTING
 Deriving Test Cases:
1. Using the design or code as a foundation, draw a
corresponding flow graph.
2. Determine the cyclomatic complexity of the resultant flow
graph.
3. Determine a basis set of linearly independent paths.
4. Prepare test cases that will force execution of each path in
the basis set.

29. BASIS PATH TESTING
Figure 14.4: PDL with nodes identified

30. BASIS PATH TESTING
Figure 14.5
V(G) = 6 regions
V(G) = 17 edges – 13 nodes + 2 = 6
V(G) = 5 predicate nodes + 1 = 6

31. BASIS PATH TESTING
• Paths:
• Path 1: 1-2-10-11-13
• Path 2: 1-2-10-12-13
• Path 3: 1-2-3-10-11-13
• Path 4: 1-2-3-4-5-8-9-2-…
• Path 5: 1-2-3-4-5-6-7-8-2….
• Path 6: 1-2-3-4-5-6-7-8-9-2…
• Prepare test cases that will force execution of each path in
the basis set.
31

32. BASIS PATH TESTING : Another example
32
R3

33. BASIS PATH TESTING : Another example
• What is the cyclomatic complexity, V(G)?
• What are the independent paths?
• What can you conclude from this?
33

34. BASIS PATH TESTING : Another example
• Conclusion – 1: By getting a value of V(G) = 3 we conclude that it is a “well
written” code, its “testability” is high and cost / effort to maintain is low.
• Conclusion – 2: There are 3 paths in this program which are independent
paths and they form a basis-set. These paths are described below:
• Path 1: 1 – 2 – 4 – 5 - 7
• Path 2: 1 – 3 – 4 – 5 - 7
• Path 3: 1 – 3 – 4 – 5 – 6 - 7
• Conclusion – 3: We can form another basis-set as described below:
• Path 1: 1 – 2 – 4 – 5 - 7
• Path 2: 1 – 3 – 4 – 5 - 7
• Path 3: 1 – 2 – 4 – 5 – 6 - 7
• Conclusion – 4: We must execute these paths at least once in order to test
the program thoroughly. Accordingly we can design the test cases.
34

35. Control Structure Testing
• White-box testing technique focusing on control structures
present in the software.
• The basis path testing technique is one of a number of
techniques for control structure testing. Basis path testing is
simple and effective; however, it is not sufficient in itself.
• Following are some variations on control structure testing;
they broaden testing coverage and improve quality of white-
box testing –
– Condition testing
– Data flow testing
– Loop testing
35

36. Control Structure Testing
 Condition testing:
• Condition Testing is a test case design method that exercises
the logical conditions contained in a program module.
• Focuses on testing each condition in the program to ensure
that it does not contain errors.
• Types of errors found include:
– Boolean operator error (OR, AND, NOT)
– Boolean variable error
– Boolean parenthesis error
– Relational operator error (>,<,=,!=,…)
– Arithmetic expression error
36

37. Control Structure Testing
 Condition testing:
• A simple condition is a Boolean variable or a relational expression, possibly
preceded with one NOT operator. A relational expression takes the form
E1<relational-operator> E2,
Where E1 and E2 are arithmetic expressions, and
<relational-operator> is one of the following <, <=, =, <>, > or >=
• A compound condition is composed of two or more simple conditions,
Boolean operators, and parentheses.
• Boolean operators allowed in a compound condition include OR, AND,
and NOT.
• Errors are much more common in the neighborhood of logical conditions
than they are in the locus of sequential programming.
37

38. Control Structure Testing
• Data Flow testing:
• Data flow testing is a form of structural testing and a White
Box testing technique that focuses on program variables and
the paths:
– From the point where a variable, v, is defined or assigned
a value
– To the point where that variable, v, is used
• Data flow testing method selects test paths of a program
according to the locations of definitions and uses of variables
in the program.
38

39. Control Structure Testing
 Data Flow testing:
• A program unit accepts inputs, performs
computations, assigns new values to variables, and
returns results. One can visualize of “flow” of data
values from one statement to another.
• A data value produced in one statement is expected
to be used later.
39

40. Control Structure Testing
 Data Flow testing:
• Anomaly: An anomaly is a deviant or abnormal way of doing
something.
• Data-flow anomalies represent the patterns of data usage which
may lead to an incorrect execution of the code.
• Examples of data flow anomaly:
a) It is an abnormal situation to successively assign two values to a
variable without using the first value==> Defined and then defined
again
b)It is abnormal to use a value of a variable before assigning a value
to the variable ==> Undefined but referenced
c) Another abnormal situation is to generate a data value and never
use it ==> Defined but not referenced
40

41. Control Structure Testing
 Data Flow testing:
• Question: What to do after detecting a data flow
anomaly?
• The programmers must analyze the causes of data
flow anomalies and eliminate them.
– Investigate the cause of the anomaly.
– To fix an anomaly, write new code or modify the
existing code.
41

42. Control Structure Testing
 Data Flow testing:
• Question: Does the presence of data flow anomaly always
mean that execution of the program will result in a failure?
• Answer: Not always…
– The presence of a data flow anomaly in a program does
not necessarily mean that execution of the program will
result in a failure.
– A data flow anomaly simply means that the program may
fail, and therefore the programmer must investigate the
cause of the anomaly.
42

43. Control Structure Testing
 Loop Testing:
• Loop testing is a white-box testing technique that focuses
exclusively on the validity of loop constructs.
• Four types of loop can be defined –
– Simple
– Nested
– Concatenated
– Unstructured
43

44. 44
Loop Types
Simple
Concatenated
Nested
Unstructured

45. Control Structure Testing
 Loop Testing:
• Simple Loops: The following tests should be performed for
simple loops, where n is the maximum number of allowable
passes through the loop:
– Skip the loop entirely
– Only one pass through the loop
– Two passes through the loop
– m passes through the loop where m < n
– n-1, n, n+1 passes through the loop
45

46. Control Structure Testing
 Loop Testing:
• Nested Loops: If we extended the simple loop test cases to nested loops,
the number of tests would grow geometrically. Instead use the following
scheme:
– Start at the innermost loop. Set all other loops to their minimum
values.
– Conduct simple loop tests for the innermost loop while holding the
outer loops at their minimum iteration parameter.
– Work outward, conducting tests for the next loop, but keeping all the
other outer loops at minimum values and other nested loops to
“typical” values.
– Continue until all loops have been tested.
46

47. Control Structure Testing
 Loop Testing:
• Concatenated Loops
– Concatenated loops can be tested using the
approach defined for simple loops, if each of the
loops is independent of each other.
– If the two loops are concatenated and the loop
counter for loop 1 is used as an initial value of
loop 2, then the loops are not independent.
47

48. Control Structure Testing
 Loop Testing:
• Unstructured Loops
– Can’t test unstructured loops effectively.
– Reflects very bad practice and should be
redesigned.
48

49. Black Box Testing

50. Black Box Testing
• Black box testing is also called behavioral/functional
testing.
• Black box tests are designed to validate functional
requirements without regard to the internal
workings of a program.
• Black box testing consists of a set of input conditions
that will fully exercise all the functional requirements
for a program.
50

51. Black Box Testing
• Derives test cases using requirements, spec, user manual.
• Internal implementation is not used.
• Module being tested is considered to be a ‘black box'
whose behaviour is only determined by its inputs and related
outputs.
• Tester is purely concerned with whether the module
performs the function it was intended to perform, as stated
in the requirements.
• Tester must exercise the normal behaviour, abnormal
behaviour, and error conditions.
• BBT is not an alternative to WBT. Rather it is a
complementary approach to WBT.
51

52. Black Box Testing
• Focuses on the functional requirements of the software
including functions, operations, external interfaces, external
data and information.
• Attempts to find errors in the following categories :
– Incorrect or missing functions
– Interface errors
– Errors in external database access
– Behavior or performance errors
– Initialization and termination errors
• Unlike WBT, which is performed early in the testing process,
BBT tends to be applied during later stages of testing.
52

53. Black Box Testing
 Black Box Testing Techniques
– Equivalence Partitioning
– Boundary Value Analysis
– Error Guessing
53

54. Equivalence Partitioning
• Exhaustive testing, where every possible program
execution sequence is tested, is impossible.
• We need to partition the class of all possible inputs to try
to simulate exhaustive testing - this technique is known
as equivalence partitioning.
• We divide up all possible inputs into equivalence classes.
• A test with one member of an equivalence class will be
the same as a test with all the members of that class.
• Reduces the total number of test cases to be developed.
54

55. Equivalence Partitioning
• Equivalence Partitioning is a black-box testing method that
divides the input domain of a program into classes of data
from which test cases are derived.
• The input domain of a program is partitioned into a finite
number of equivalence classes from which test cases are
derived.
• An equivalence class consists of a set of data that is treated
the same by the program or that should generate the same
result.
• Test case design is based on an evaluation of equivalence
classes for an input condition.
55

56. Equivalence Partitioning
• The equivalence classes are identified based on the
set of valid or invalid states for each input condition.
• An input condition can be
– A specific numeric value
– A range of values
– A set of related values
– A Boolean condition
56

57. Guidelines for defining equivalence classes
• If an input condition specifies a range(e.g. 1-12), one valid and
two invalid equivalence classes are defined(e.g. 5, 0, and 13)
• If an input condition requires a specific value(e.g. ‘m’) then
one valid and two invalid equivalence classes are defined(e.g.
‘m’, ‘f’, and ‘_’)
• If an input condition specifies a member of a set(e.g. ‘check
pin’, ‘check balance’, ‘withdraw cash’), then one valid and one
invalid equivalence classes are defined
• If an input condition is Boolean, one valid and one invalid
class are defined
57

58. Equivalence Partitioning: Example1
• Consider we are writing a program for computing letter
grades based on the numerical scores of students, where the
input variable is Score. The rule of computing the grade is as
follows:
58
Score Grade
90~100 A
80~89 B
70~79 C
60~69 D
0~59 F

59. Equivalence Partitioning: Example1
• The input domain of Score can be partitioned into 5
valid equivalence classes and 2 invalid equivalence
classes –
– Valid classes: 0~59, 60~69, 70~79, 80~89, 90~100
– Invalid classes: smaller than 0 and greater than 100
• Any data value within a class is considered
equivalence in terms of testing
• Using the equivalence partitioning testing, we can
reduce the test cases from 101 (assume 0 <= score
<= 100) to 7
59

60. Equivalence Partitioning: Example2
• A program which edits credit limits within a given
range ($10,000 - $15,000) would have 3 equivalence
classes:
– Less than $10,000 (invalid)
– Between $10,000 and $15,000 (valid)
– Greater than $15,000 (invalid)
60

61. Equivalence Partitioning
 Why we should use Equivalence Partitioning when
designing test cases?
– To reduce the number of test cases (by avoiding
the redundant cases), hence reducing the testing
time & cost
– To assure better test coverage, by testing all input
classes, hence improve the test efficiency.
61

62. Boundary Value Analysis (BVA)
• BVA extends equivalence partitioning by focusing on
data at the “edges” of an equivalence class.
• BVA is a test case design technique that
complements equivalence partitioning. Rather than
selecting any element of the equivalence class, BVA
leads to the selection of test cases at the “edges” of
the class.
• Unlike equivalence partition that derives test cases
only from input conditions, BVA derives test cases
from both input conditions and output domain.
62

63. Boundary Value Analysis (BVA)
• In addition to select test data “inside” an equivalence class,
data at the “edges” of the equivalence class also need to be
examined.
• Based on programming experience, more errors are found at
the boundaries of an input/output domain than in the
“center”.
• BVA is a technique that consists of developing test cases and
data that focus on the input and output boundaries of a given
function.
• BVA should be performed after identifying the Equivalence
Partitions of the software.
63

64. Boundary Value Analysis: Example
• In the same credit limit example, boundary analysis
would test:
– Low boundary plus or minus one ($9,999 and
$10,001)
– On the boundary ($10,000 and $15,000)
– Upper boundary plus or minus one ($14,999 and
$15,001)
64

65. Guidelines for BVA
1. If an input condition specifies a range [a, b], test cases
should be designed with value a and b, just above and
below a and b
Example: Integer D with input condition [-3, 5], BVA test
values are -3, 5, -2, 6, -4, 4
2. If an input condition specifies number values, test cases
should be developed to exercise the minimum and
maximum numbers. Values just above and below minimum
and maximum are also tested
Example: Enumerate data E with input condition: {3, 5, 100,
102}, BVA test values are 3, 102, 2, 4, 101, 103
65

66. Guidelines for BVA
3. Guidelines 1 and 2 are applied to output condition
4. If internal program data structures have prescribed
boundaries(e.g., an array has a defined limit of 100 entries)
make sure to design a test case to exercise the data
structure at its boundary.
– Array input condition: Empty, single element, full
element, out-of-boundary
– Search output condition: Element is inside array or the
element is not inside array
66

67. Error Guessing
• Based on the theory that test cases can be
developed based on experience of the Test Engineer.
• Test design technique in which the experience of the
testers is used to
– Guess the probable kinds and location of defects in a
system, and
– Design tests specifically to expose them
67

68. Error Guessing
• Identify potential errors and design test cases based on
intuition and experiences.
• Test cases can be derived by making a list of possible errors or
error-prone situations
– Empty or null lists
– Zero instances or occurrences
– Blanks or null strings
– Negative numbers
– Historical defects (need to maintain defect history)
68