Testing is the activity where the errors remaining from all the previous phases must be detected.
Hence testing plays a critical role for ensuring quality. Testing is of two types:
• Software inspections. Concerned with analysis of
the static system representation to discover problems (static verification)
– May be supplement by tool-based document and code analysis
• Software testing. Concerned with exercising and
observing product behaviour (dynamic verification)
– The system is executed with test data and its operational behaviour is observed
Static and dynamic testing
S ftw re
ins e ns
e uire e
m nts Hig ve
h-le l Fo a
rm l De ile
P g m
p cifica n
tio ds n
e ig se
p cification ds n
P to e
ro typ P gra
Types of testing
• Defect testing
– Tests designed to discover system defects.
– A successful defect test is one which reveals the presence of defects in a system.
– Covered in Chapter 23
• Validation testing
– Intended to show that the software meets its requirements.
– A successful test is one that shows that a requirements has been properly
Statistical Use testing
It is a part of Validation testing which amounts to testing software the way users intend to use it
• The specification for each increment is analyzed to define a set of stimuli (inputs or events)
that cause the software to change it’s behavior
• Test cases are generated for each set of stimuli according to the usage probability
Testing and debugging
• Defect testing and debugging are distinct
• Verification and validation is concerned with establishing the existence of defects in a
• Debugging is concerned with locating and
repairing these errors.
• Debugging involves formulating a hypothesis
about program behaviour then testing these
hypotheses to find the system error.
The debugging process
et Te t
p cifica n
s ca e
ca De ig
s n R p ir
ea R te t
rr r e o re a
rr r p ir e o
rr r p gr m
• These involve people examining the source representation with the aim of discovering
anomalies and defects.
• Inspections not require execution of a system so may be used before implementation.
• They may be applied to any representation of the system (requirements, design,
configuration data, test data, etc.).
• They have been shown to be an effective technique for discovering program errors.
• Many different defects may be discovered in a single inspection. In testing, one defect , may
mask another so several executions are required.
• The reuse domain and programming knowledge so reviewers are likely to have seen the
types of error that commonly arise.
The inspection process
e w F llow
Ind vid l
R w rk
p p ra n
re a tio
Ins e n
• System overview presented to inspection team.
• Code and associated documents are distributed to inspection team in advance.
• Inspection takes place and discovered errors are noted.
• Modifications are made to repair discovered errors.
• Re-inspection may or may not be required.
• Checklist of common errors should be used to
drive the inspection.
• Error checklists are programming language
dependent and reflect the characteristic errors that are likely to arise in the language.
• In general, the 'weaker' the type checking, the larger the checklist.
• Examples: Initialisation, Constant naming, loop
termination, array bounds, etc.
Inspections and testing
• Inspections and testing are complementary and not opposing verification techniques.
• Both should be used during the V & V process.
• Inspections can check conformance with a specification but not conformance with the
customer’s real requirements.
• Inspections cannot check non-functional characteristics such as performance, usability, etc.
Automated static analysis
• Static analysers are software tools for source text processing.
• They parse the program text and try to discover potentially erroneous conditions and bring
these to the attention of the V & V team.
• They are very effective as an aid to inspections - they are a supplement to but not a
replacement for inspections.
Stages of static analysis
• Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable
• Data use analysis. Detects uninitialized variables, variables written twice without an
intervening assignment, variables which are declared but never used, etc.
• Interface analysis. Checks the consistency of routine and procedure declarations and their
• Information flow analysis. Identifies the dependencies of output variables. Does not
detect anomalies itself but highlights information for code inspection or review
• Path analysis. Identifies paths through the program and sets out the statements executed in
that path. Again, potentially useful in the review process
• Both these stages generate vast amounts of information. They must be used with care.
Use of static analysis
• Particularly valuable when a language such as C is used which has weak typing and hence
many errors are undetected by the compiler,
• Less cost-effective for languages like Java that have strong type checking and can therefore
detect many errors during compilation.
Verification and formal methods
• Formal methods can be used when a mathematical specification of the system is produced.
• They are the ultimate static verification technique.
• They involve detailed mathematical analysis of the specification and may develop formal
arguments that a program conforms to its mathematical specification.
During dynamic testing, the software to be tested is executed with a set of test cases, and the
behavior of the system for the test cases is evaluated to determine if the system is performing as
expected. The behavior is evaluated using test oracles which are considered to have the correct
answer. These can be humans who decide the correct behavior of the system ,or can be generated
automatically using requirement specifications. Test cases are given to the test oracle (if it is a
human) and the program under testing. The output of the two is compared to find if program
behaved correctly for the test cases.
The V-model of development
Rq e e
e uir m nts S te
ys m Sys m
te De ile
p cifica n
p cifica n
tio ds n
e ig ds n
ys m S -s te
ub ys m Mo ulea
gra n inte tio
gra n unit co e
te t p n
te t p n
s la te t p n
s la a te t
p nce Sys m
te S -s te
ub ys m
s inte tio te t
gra n s inte a n te t
gr tio s
Psychology of testing
The basic purpose of testing is to detect the errors that may be present in the program. Hence the
intent should be to show that a program does not work, thus an intent of finding errors.
That is why most of the organizations, require a product to be tested by people not involved with
developing the program before finally delivering it to the customer. Also at times errors occur
because the programmer did not understand the specification correctly. Thus testing by external
person helps in resolving such errors.
This approach is suitable for earlier stages, where the objective is to reveal errors. However last
stages are meant for evaluating the product. Here test cases are selected to mimic user scenario.
Objectives of testing
1. Software Quality Improvement. Quality means conformance to the specified software design
2. Verification and validation: To ensure conformity with requirements.
3. Software reliability estimation
Test design refers to understanding the sources of test cases, test coverage, how to develop and
document test cases, and how to build and maintain test data. Two methods of test design are:
1. Black-box testing
2. White-box testing
Black box testing
In black box testing, the tester only knows the inputs that can be given to the system and what
output the system should give.
Also known as functional or behavioral testing, in black-box testing, the basis for deciding test cases
in functional testing is the requirements or specifications of the system or module.
To decide the test cases we use the following methods:
1. Equivalence class partitioning: It breaks classes into small parts and selects test cases from
each class, such partitioned. An equivalence class is formed of the inputs for which the
behavior is expected to be similar. Each group of inputs for which the behavior is expected
to be different from others is considered a separate equivalence class.
Equivalence classes are usually formed by considering each condition specified on an input as
specifying a valid equivalence class and one or more invalid equivalence classes. Another approach is
to consider any special case where behavior could be different.
2. Graph based testing methods or cause-effect testing is a technique that aids in selecting
combinations of input conditions in a systematic way, such that the number of test cases
does not become unmanageably large. Thus for n input conditions there will be 2n test cases.
A cause is a distinct input condition, and an effect is a distinct output condition. The
conditions are set such that they can be set to either true or false. A test case is generated
for each combination of conditions, which make some effect true.
3. Boundary value analysis selects 7 cases for all boundary value variables. The seven values
being min-1, min, min+1, max-1, max, max+1, nominal value. Thus for n variables there
would be 7n test cases
4. Specialized testing where special cases are provided for. It also includes testing of
documentation, real time systems, inter task testing etc.
1. Tester and programmer are independent
2. Testing from user’s point of view
3. Test cases can be prepared right after specifications are decided.
i. Duplication of test cases by tester and programmer
ii. Only sample testing can be done as all cases cannot be built and tested.
White box testing, also known as structural testing, is concerned with the implementation of the
requirements. It evaluates the programming structures and data structures used in the program. The
criterion for deciding test cases for white-box testing are:
1. Control flow based criterion: The control flow graph of a program is considered and
coverage of various aspects of the graph are specified. The coverage criterions are:
1. Statement coverage requires that the paths executed during testing include all the
nodes in the graph.
2. Branch coverage requires that each edge in the control flow graph be traversed at
3. Path coverage requires all possible paths in the control flow graph to be executed
2. Data flow based testing is conducted by creating a definition-use graph for the program. A
variable occurs in one of the three ways:
• def refers to the definition of the variable
• C-use represents computational use of the variable
• P-use represents predicate use, which is used for transfer of control
Based on these various paths can be defines, such as
• Global c-use
• Dcu(x, i) set of nodes such that each node has x is in c-use
• Dpu(x,i) set of nodes such that each node has x is in p-use
• All-p-uses includes a def-clear path w.r.t x from I to dpu(x, i)
• All-c-uses includes a def-clear path w.r.t x from I to dcu(x, i)
• Reveals errors in hidden code
• Forces reason in implementation
• Cases omitted can miss out code
It is performed by creating many mutants in the code by making simple changes to the program. By
testing we try to identify errors. If all the mutants planted are found, then we can assume that test
cases are sufficient, otherwise more test cases are created unless we find all the mutants. Steps for
mutant testing are as follows:
• Generate mutants for P. Suppose there are N mutants
• By executing each mutant and P on each test case in T, find how many mutants can be
distinguished by T. Let D be the number of mutants distinguished or dead.
• For each mutant that cannot be distinguished by T (live), find how many are equivalent to P.
Let E be the number of equivalent mutants.
• The mutation score is computed as D/(N-E)
• Add more test cases to T and continue testing until the mutation score is 1.
Test case specification
Requirement number Condition to be tested Test data and settings Expected o
Error, fault and failure
• Error refers to the discrepancy between a computed, observed or measured value and the
true, specified, or theoretically correct value.
• Fault is a condition that causes a system to fail in performing its required function. It is the
basic reason for the software malfunction and is synonymous with the term bug.
• Failure is the inability of a system or component to perform a required function according to
Presence of an error implies that a failure must have occurred, and the observance of a failure
implies a fault must be present. However, the presence of a fault does not imply that a failure must
During the testing process, only failures are observed, by which the presence of faults is deduced.
The actual faults are identified by separate activities, commonly referred to as “debugging”.
Defect logging and tracking
Severity of defect can be categorized as follows:
• Critical: Show stopper; affects a lot of users; can delay project
• Major: Has a large impact but workaround exists
• Minor: An isolated defect that manifests rarely and with little impact
• Cosmetic: Small mistakes that don’t impact the correct working
Defect analysis and prevention techniques
• Pareto Analysis states that 80% of the problems come from 20% of the possible sources.
Once these sources are identified, similar test cases can be prepared for the rest of the
• Casual analysis are drawn using cause-effect diagram. After the problem is stated, the major
categories of causes are identified. Then further brainstorming helps identify sub-causes to
these major causes.
• Develop and implement solution attacks the root cause of the defects. Preventive solutions
devised are implemented as project activities, assigned to team members. The effect of such
solutions is monitored.
Types of testing
• Unit testing is performed at the time of development. Errors found at this level can be
debugged easily. Each module is tested against the specifications produced during design for
the modules. It is typically done by the programmer. Some examples of unit testing are:
• Black-box approach
– Field level check
– Field level validation
– User interface check
– Function level check
• White box approach
– Statement coverage
– Decision coverage
– Condition coverage
– Path coverage
Advantages of unit testing
– Can be applied directly to object code
– Performance profilers implement this measure
– Not implementable to logical operators
– Insensitive to loop operators (number of iterations)
• Integration testing is performed to put together all the modules developed and unit tested.
It is done in either top-down method or bottom up method
• System testing is conducted after all the modules have been integrated and is followed by
performance testing, where the actual performance of the system is tested. System testing
is concerned with the compliance of requirements regarding the system. Techniques for
selecting test cases for system testing are as follows:
• Testing the system’s capabilities is more important than testing its
components; thus focus shifts on catastrophic failure than minor irritants
• Testing the usual is more important than testing the exotic
• If we are testing after modification, we should test old capabilities rather
than new ones.
Testing techniques can be structural or functional. Some of these are discussed as follows
Structural techniques for system testing:
• Stress testing: It executes the system in a manner that demands resources in abnormal
quantity, frequency or volume. It decides the higher limit parameters for the system.
• Recovery testing forces the software to fail in a variety of ways and verifies that the system
recovers in a proper manner.
• Compliance testing is conducted to ensure that the system confirms to the standards laid
down by the quality management system.
• Operations testing is performed to ensure that the system fits correctly in the organizations
Functional techniques for system testing:
• Security testing verifies that protection mechanism built in the system will protect it from
improper penetration. The role of system designer is to make penetration cost more than
the value of the information that will be obtained. It can also be a part of structural
• Manual support testing includes user documentation required by the clients and ensures
correct support is available.
• Control testing checks the control mechanism for correctness
• Parallel testing is conducted to ensure that all the versions have the required functionalities
and correctly display the changes incorporated in each version.
• Acceptance testing is performed with realistic data of the client to demonstrate that the
software is working satisfactorily. Testing here focuses on the external behavior of the
system. There are two types of acceptance testing:
• Alpha testing is performed at the developer’s site, but the user is present for testing
• Beta testing is performed at the client site and the system is tested for acceptability
Cleanroom software development
• The name is derived from the 'Cleanroom'
process in semiconductor fabrication. The
philosophy is defect avoidance rather than
• This software development process is based on:
– Incremental development;
– Formal specification;
– Static verification using correctness arguments;
– Statistical testing to determine program reliability.
Fo a y
rm ll Err r re o
o w rk
De fine C ns ct
o tru F rm ll y
o re structured v rify
m nts p g m
ro ra co e
De e p
o e a na
p r tio l De ig
s n Ts
ro s tica
tatis l inte te
s s te
Cleanroom process characteristics
• Formal specification using a state transition model.
• Incremental development where the customer prioritises increments.
• Structured programming - limited control and abstraction constructs are used in the
• Static verification using rigorous inspections.
• Statistical testing of the system
Formal specification and inspections
• The state based model is a system specification and the inspection process checks the
program against this model
• The programming approach is defined so that the correspondence between the model and
the system is clear.
• Mathematical arguments (not proofs) are used to increase confidence in the inspection
Cleanroom process teams
• Specification team. Responsible for developing
and maintaining the system specification.
• Development team. Responsible for
developing and verifying the software. The
software is NOT executed or even compiled
during this process.
• Certification team. Responsible for developing
a set of statistical tests to exercise the software
after development. Reliability growth models
used to determine when reliability is acceptable.
Certification requires the creation of three models
1. Sampling model: Software testing executes m random test cases and is certified if no failures
or a certified number of failures occur.
2. Component model: A system composed of n components is to be certified to determine the
probability that component I will fail prior to completion.
3. Certification model: The overall reliability of the system is projected and certified.
Cleanroom process evaluation
• The results of using the Cleanroom process have been very impressive with few discovered
faults in delivered systems.
• Independent assessment shows that the
process is no more expensive than other
• There were fewer errors than in a 'traditional' development process.
• However, the process is not widely used. It is not clear how this approach can be transferred
to an environment with less skilled or less
motivated software engineers.