MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING
Hemanta Chandra Bhatt
Novell Software Development (I) Ltd., Bangalore, INDIA
The development of a software product is essentially driven by requirements that are
derived from multiple sources such as study of customer needs, competition in the marketplace
and advancements in technology. The requirements thus derived are consolidated into a
software requirement specification (SRS) that drives the software life cycle.
For development, it is necessary that requirements at the SRS level be translated into
detailed engineering requirements during the design and implementation phases of the software
life cycle. This also brings out the need for how requirements at various phases have to be
specified to facilitate this.
Before it is released to customers, the software needs to be validated vis-a-vis the SRS.
Conventionally, validation is done through testing using test cases as the means. The mapping
of test cases to requirements can be used to measure - 1) the coverage of requirements and 2)
the effectiveness of testing in terms of coverage vis-a-vis the execution effort. The above
measures along with pass percentage of the number of test cases executed can be used to make
more objective decisions on when to stop testing and release the product.
This paper proposes metrics to measure the mapping of test cases to requirements and vice
versa. It also discusses the benefits of mapping test cases to requirements. A case study is also
included to illustrate the computation of the proposed metrics and their usage.
The development of a software product is essentially driven by requirements that are derived from
multiple sources such as:
a) Study of customer requirements - customers demanding products with enhanced functionality,
performance, reliability, etc.
b) Analysis of competitive products in the marketplace - multiple software vendors in the same
product segment releasing versions with enhanced functionality, performance, reliability, etc. in
c) Advancements in technology - newer technologies making it possible to develop products with
ever-increasing functionality and value to the customers.
The requirements gathered have to be analyzed and appropriately consolidated to arrive at a set of
technically and commercially feasible software requirement specification (SRS). The SRS is then used
to drive the development of the product right from design to its release.
Before the software is released to customers, it needs to be validated against the requirements.
Conventionally, testing is used for validation through the execution of a set of test cases. The mapping
of test cases to requirements can be used to measure the effectiveness of testing. Hammer, Theodore,
et al  described metrics to measure the effectiveness of requirements testing.
In Section 2, the terms - requirement and test case - are defined as they refer to in this paper. In
Section 3, the concept of mapping is described and metrics to measure the effectiveness of testing are
proposed. Finally in Section 4, a case study is presented that illustrates the computation and usage of
the proposed metrics.
2 EXPLANATION OF TERMS
In this section, the terms - requirement and test case - are explained as they refer to in this paper.
Requirement: A requirement is a statement with an associated level (L ) and priority number
( P ). A requirement R i with priority Pi at level Lk is written as R i Lk (Pi ) .
The level of a requirement is discussed in Section 3.1. The priority of a requirement indicates its
relative importance in terms of expected usage with respect to other requirements for the software.
R i will refer to R i Lk (Pi ) in the rest of the paper, for the sake of simplicity.
Test case: A test case is a set of executable steps with an associated level, the pass/fail definition
of each step, effort needed for execution ( E ) and Requirement Compliance Factor ( RCF ). RCF is
discussed in section 3.2. A test case T j with RCF RCF j at level Lk is written as T j Lk ( E j RCF j ) .
T j will refer to T j L k (E j RCF j ) in the rest of the paper for the sake of simplicity.
3 MEASURING THE EFFECTIVENESS OF TESTING
Measuring the effectiveness of testing involves the following steps:
a) Specification of requirements
b) Measuring the mapping of test cases to requirements
3.1 Specification of requirements
The specification of requirements affects the computation and usage of metrics to measure the
mapping of testing to requirements. The requirements at various phases of software life cycle vary in
terms of comprehensiveness and often have an overlap. The level of a requirement indicates the phase
it pertains to. William M. Wilson  suggested 10 attributes that are required for a meaningful
Consider an example, where requirements are specified at five levels. Let the requirements at
various levels be denoted by, R1 , R 2 , R3 , R 4 and R 5 the test case by T1 , T2 ,T3 , T4 and T5 respectively
as shown in Table 1.
Level Development Phase Testing Phase
Level 1 Customer Requirements ( R1 ) Customer Acceptance Testing ( T1 )
Level 2 Engineering Requirements ( R 2 ) System Testing ( T2 )
Level 3 High Level Design ( R 3 ) Integration Testing (T3 )
Level 4 Low Level Design ( R 4 ) Component Testing (T4 )
Level 5 Coding Requirements ( R 5 ) Unit Testing ( T5 )
At Level 1 the requirements are at a very abstract level whereas at Level 4 the requirements are
very detailed. In terms of test cases, a Level 4 requirement may be directly testable by two test cases
- one with a valid and other with an invalid input - but a Level 1 requirement may need several test
cases to test (that may test other requirements too).
3.2 Measuring the mapping test cases to requirements
Mapping means the evaluation of test cases in terms of coverage provided to the set of specified
requirements. The mapping of a test case to requirements can be measured in terms of:
a) No. of requirements it tests
b) The priority of the requirements it tests
c) The effort for its execution versus the coverage of requirements
Consider a software that has a set of m specified requirements,R i , i = 1 to m , to be tested. The
testing has to be done with a sample of test cases selected from the available population of test cases
containing n test cases T j , j = 1 to n .
For each test case T j , j = 1 to n , requirement compliance factor ( RCF j ) is defined as a measure
of the "coverage" provided by a test case to the set of requirements,R i , i = 1 to m . RCF j is
calculated as follows:
∑ Pi X i
RCF j =
(max m1 X i ) * ( ∑ Pi
Xi = 2, if test case T j tests requirement R i completely
1, if test case T j tests requirement R i partially
Pi = Priority number of Ri
RCF j can be used to rank the test cases in terms of the coverage provided to a given set of
requirements. This can be used in the selection of a sub-set of test cases from the available population
of test cases that would maximize the coverage provided to the set of specified requirements.
The RCF for testing can be computed as given below:
RCFTesting can be used to measure the coverage provided by a set of selected test cases to the set of
The effectiveness of the test case T j is defined as a measure of the coverage provided by a
test case vis -a-vis the effort needed for its execution. It is calculated as:
Effectiveness j =
Ej = Time required for executing the test case (in hours).
Effectiven ess j can be used to rank the test cases in terms of their effectiveness. This can be used in
the selection of a sub-set of test cases from the available population of test cases that would maximize
the coverage under the condition that the effort available for execution is constrained.
The effectiveness of testing can be computed as follows:
∑ Effectiveness j
Effectiven Testing =
Effectiven essTesting can be used to measure the effectiveness of testing in terms of coverage provided
by a set of test cases vis-a-vis the effort needed for their execution.
For each requirement R i , j = 1 to m , coverage factor (CFi ) is defined as the coverage
provided to it by a set of test cases T j , j = 1 to n vis-a-vis the effort needed for test execution. This
is the summation of effectiveness of all the test cases that test that requirement.
CFi = ∑ Y j Effectiven ess j
Yj = 2, if requirement R i is completely tested by test case T j
1, if requirement R i is partially tested by test case T j
The coverage factor for testing CF can calculated as follows:
CFTesting can be used to evaluate requirements in terms of coverage from the execution of a set of
selected test cases. This can be used in determining whether a requirement has been tested adequately
To summarize, the above metrics can be used for the following purposes:
a) Review of test cases
The RCF of the test cases can be used as a factor in the review of test cases.
b) Selection of test cases
The effectiveness of the test cases can be used to prioritize the test cases on the basis of returns
in terms of coverage versus the effort needed for execution. In case of reduced cycle time, the
selection of test cases can be done more objectively based on priority.
c) Setting acceptance criteria of the product from a level to the next higher level
Consider the example of transmittal of a product from level Lk to level Lk +1 . In a well-defined
process there should be clear and matching exit and entry criteria for both the levels. The criteria could
be defined as: “If the RCF of the product at the end of level Lk testing is 0.9 with respect to the level
Lk requirements, then it satisfies the entry criteria for level Lk +1 ".
d) Stopping criteria for testing
The decision on when to stop testing based on completeness of testing can be stated in terms of
not only the number of test cases (executed and passed) but also the coverage of requirements.
e) Making release decisions
The evaluation of the product in terms of coverage of requirements would help in making release
decisions more objectively.
4 CASE STUDY
In this section, the calculation of the metrics given in Section 3 and their usage has been
illustrated using a case study.
Consider the development of a Directory Server that has requirements specified at four levels as
shown in the Table 2. Note that the statements vary in detail from one level to the other. Each
requirement is assigned a priority number 1, 2 or 3. A requirement with priority number 1 is of low
priority, 2 of medium priority and 3 of highest priority.
L1 L2 L3 L4
R11: The search performance must be R21: The search performance must to R31:A new sub-string search for and existing R41:The search must return all the matching objects (3)
better than the previous version of the atleast 10% greater than that with the object in a database with 10000 objects must R42:Indexing must be done on most commonly sought
server (3) previous version of the server (3) be less than or equal to 2 seconds (3) attributes (2)
R43:The search results must be sorted by common name
R44:The output must be in Base64 encoding (3)
R32: A non-existent object search must take
less than or equal to 5 seconds (2)
R33: A referral should be given in case of an
unfruitful search (1)
R34: An all-attribute search for and existing
R35: The request complete listing of 10000
objects in the database must take less than or
equal to 10 seconds (2)
R22: The search algorithm must be
modified to make it more efficient (3)
R23: The search results must be cached and
the cache must be updated periodically (2)
L1 L2 L3 L4
R12: The search performance must be
better than that of the competitive
R13: The server should comply with the
industry standard (?) (2)
(Note: the number in bracket indicates the priority of the requirement)
Consider the level 3 requirements R1 to R5 that have to be tested. T1 to T10 is the available
population of test cases. Xi for each test case is evaluated based on how much it tests a requirement,
as specified in Section 3.2. Table 3 shows the metrics and calculations.
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 ity age CF
R i R1 1 2 2 1 2 0 1 0 2 1 3 12 3.35
e r R2 2 1 2 0 2 2 2 2 1 0 2 14 3.37
q e R3 0 2 1 2 2 0 2 1 0 2 1 12 2.99
u mR4 1 2 0 1 2 2 1 0 0 1 2 10 2.86
R5 1 0 0 0 1 1 0 0 0 2 2 5 1.42
10 10.60 2.80
(E) 2 4.5 5 2 1 8 7 3 2 6
RCF 0.55 0.70 0.55 0.35 0.90 0.50 0.55 0.25 0.40 0.55 0.53
Effectiveness 0.16 0.11
0.28 0.18 0.90 0.06 0.08 0.08 0.20 0.09 0.21
The ordering of test cases based on RCF can be used in the selection of a sample of test
cases from the population of available test cases that would maximize the coverage. In this case, test
case T5 has the maximum RCF and hence provides the highest coverage to the set of specified
Similarly, the ordering of requirements based on the coverage factor (CF) can be used to
determine the coverage acquired by a requirement from the execution of a sample of test cases. In the
above case, requirement R2 has the highest CF, which indicates that it acquires the highest coverage .
Decisions can be made based on specified targets for the various metrics described earlier in
this paper. In this example, for making decisions the following heuristics can be defined:
a) A requirement is “completely” met if and only if 100% of the test cases with X i = 2, testing the
requirement are pass and 80% of the test cases with X i = 1 pass.
In the above example, R1 is completely met if T2, T3, T5, T9 are passed and any three of T1,
T4, T7, T10 are passed.
a) A requirement is “partially” met if 80% of the test cases with X i = 2 are pass and more than 70%
of the test cases with X i = 1 pass.
In the above example, R1 is partially met if any three of T2, T3, T5, T9 are passed and any
three of T1, T4, T7, T10 pass.
b) The product can enter the next level is RCF is greater than 0.5. One additional criterion can be
that if the test coverage of all requirements should be greater than 3 for the product to enter next
level of testing.
In the above example, if R1, R2 and R3 are completely met, R4 partially met and R5 not met,
then RCF of the product at this level is 0.7. Based on the heuristic, it can enter the next level in the
The development of a software product is essentially driven by requirements. Before the software
is released to the customers, it needs to be validated against the requirements. Testing has been
conventionally used for validation through execution of test cases. The effectiveness of testing vis-à-
vis requirements can be measured in terms of mapping of test cases to requirements. This paper
explains metrics to measure the mapping. These metrics would help in making decisions on when to
stop testing and release the product in more objective manner as has been illustrated in this paper.
 Hammer, Theodore, et al, Measuring Requirements Testing,
 Wilson, M William, Writing Effective Requirements Specifications,
We would like to express our gratitude to the following persons from Novell Bangalore - Mr. J.
Veeraraghavan, Mr. Srinivasan Desikan, Mr. Jyoti Prakash Kurmi and Mr. Manu Sreenivasan - for
their valuable inputs towards this paper.
About the Authors
Chandana Pavuluru is working as a Senior Software Engineer with the System Testing Team at
Novell Bangalore for past 3 years.
Hemanta Chandra Bhatt is working as Senior Software Engineer with the Quality Engineering
Group at Novell Bangalore for the past 2 years.