MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING

                                           Submitted by

                ...
1    INTRODUCTION

    The development of a software product is essentially driven by requirements that are derived from

...
The level of a requirement is discussed in Section 3.1. The priority of a requirement indicates its

relative importance i...
Level           Development Phase                        Testing Phase

  Level 1         Customer Requirements ( R1 )    ...
calculated as follows:

                    m
                 ∑ Pi X i
                i =1
RCF j =
          (max m1 X i...
The effectiveness of testing can be computed as follows:

                            n
                            ∑ Effe...
selection of test cases can be done more objectively based on priority.

c) Setting acceptance criteria of the product fro...
L1                                        L2                                           L3                                 ...
L1                                       L2                                L3                                      L4

R12...
Consider the level 3 requirements R1 to R5 that have to be tested. T1 to T10 is the available

population of test cases. X...
a) A requirement is “partially” met if 80% of the test cases with X i = 2 are pass and more than 70%

    of the test case...
Acknowledgements

    We would like to express our gratitude to the following persons from Novell Bangalore - Mr. J.

Veer...
MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING
Upcoming SlideShare
Loading in …5
×

MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING

1,626 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,626
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING

  1. 1. MEASURING THE EFFECTIVENESS OF SOFTWARE TESTING Submitted by Chandana Pavuluru (pchandana@novell.com) Hemanta Chandra Bhatt (bhemanta@novell.com) Novell Software Development (I) Ltd., Bangalore, INDIA ABSTRACT The development of a software product is essentially driven by requirements that are derived from multiple sources such as study of customer needs, competition in the marketplace and advancements in technology. The requirements thus derived are consolidated into a software requirement specification (SRS) that drives the software life cycle. For development, it is necessary that requirements at the SRS level be translated into detailed engineering requirements during the design and implementation phases of the software life cycle. This also brings out the need for how requirements at various phases have to be specified to facilitate this. Before it is released to customers, the software needs to be validated vis-a-vis the SRS. Conventionally, validation is done through testing using test cases as the means. The mapping of test cases to requirements can be used to measure - 1) the coverage of requirements and 2) the effectiveness of testing in terms of coverage vis-a-vis the execution effort. The above measures along with pass percentage of the number of test cases executed can be used to make more objective decisions on when to stop testing and release the product. This paper proposes metrics to measure the mapping of test cases to requirements and vice versa. It also discusses the benefits of mapping test cases to requirements. A case study is also included to illustrate the computation of the proposed metrics and their usage.
  2. 2. 1 INTRODUCTION The development of a software product is essentially driven by requirements that are derived from multiple sources such as: a) Study of customer requirements - customers demanding products with enhanced functionality, performance, reliability, etc. b) Analysis of competitive products in the marketplace - multiple software vendors in the same product segment releasing versions with enhanced functionality, performance, reliability, etc. in quick succession. c) Advancements in technology - newer technologies making it possible to develop products with ever-increasing functionality and value to the customers. The requirements gathered have to be analyzed and appropriately consolidated to arrive at a set of technically and commercially feasible software requirement specification (SRS). The SRS is then used to drive the development of the product right from design to its release. Before the software is released to customers, it needs to be validated against the requirements. Conventionally, testing is used for validation through the execution of a set of test cases. The mapping of test cases to requirements can be used to measure the effectiveness of testing. Hammer, Theodore, et al [1] described metrics to measure the effectiveness of requirements testing. In Section 2, the terms - requirement and test case - are defined as they refer to in this paper. In Section 3, the concept of mapping is described and metrics to measure the effectiveness of testing are proposed. Finally in Section 4, a case study is presented that illustrates the computation and usage of the proposed metrics. 2 EXPLANATION OF TERMS In this section, the terms - requirement and test case - are explained as they refer to in this paper. Requirement: A requirement is a statement with an associated level (L ) and priority number ( P ). A requirement R i with priority Pi at level Lk is written as R i Lk (Pi ) .
  3. 3. The level of a requirement is discussed in Section 3.1. The priority of a requirement indicates its relative importance in terms of expected usage with respect to other requirements for the software. R i will refer to R i Lk (Pi ) in the rest of the paper, for the sake of simplicity. Test case: A test case is a set of executable steps with an associated level, the pass/fail definition of each step, effort needed for execution ( E ) and Requirement Compliance Factor ( RCF ). RCF is discussed in section 3.2. A test case T j with RCF RCF j at level Lk is written as T j Lk ( E j RCF j ) . T j will refer to T j L k (E j RCF j ) in the rest of the paper for the sake of simplicity. 3 MEASURING THE EFFECTIVENESS OF TESTING Measuring the effectiveness of testing involves the following steps: a) Specification of requirements b) Measuring the mapping of test cases to requirements 3.1 Specification of requirements The specification of requirements affects the computation and usage of metrics to measure the mapping of testing to requirements. The requirements at various phases of software life cycle vary in terms of comprehensiveness and often have an overlap. The level of a requirement indicates the phase it pertains to. William M. Wilson [2] suggested 10 attributes that are required for a meaningful requirement. Consider an example, where requirements are specified at five levels. Let the requirements at various levels be denoted by, R1 , R 2 , R3 , R 4 and R 5 the test case by T1 , T2 ,T3 , T4 and T5 respectively as shown in Table 1.
  4. 4. Level Development Phase Testing Phase Level 1 Customer Requirements ( R1 ) Customer Acceptance Testing ( T1 ) Level 2 Engineering Requirements ( R 2 ) System Testing ( T2 ) Level 3 High Level Design ( R 3 ) Integration Testing (T3 ) Level 4 Low Level Design ( R 4 ) Component Testing (T4 ) Level 5 Coding Requirements ( R 5 ) Unit Testing ( T5 ) Table 1 At Level 1 the requirements are at a very abstract level whereas at Level 4 the requirements are very detailed. In terms of test cases, a Level 4 requirement may be directly testable by two test cases - one with a valid and other with an invalid input - but a Level 1 requirement may need several test cases to test (that may test other requirements too). 3.2 Measuring the mapping test cases to requirements Mapping means the evaluation of test cases in terms of coverage provided to the set of specified requirements. The mapping of a test case to requirements can be measured in terms of: a) No. of requirements it tests b) The priority of the requirements it tests c) The effort for its execution versus the coverage of requirements Consider a software that has a set of m specified requirements,R i , i = 1 to m , to be tested. The testing has to be done with a sample of test cases selected from the available population of test cases containing n test cases T j , j = 1 to n . For each test case T j , j = 1 to n , requirement compliance factor ( RCF j ) is defined as a measure of the "coverage" provided by a test case to the set of requirements,R i , i = 1 to m . RCF j is
  5. 5. calculated as follows: m ∑ Pi X i i =1 RCF j = (max m1 X i ) * ( ∑ Pi i= ) Where, Xi = 2, if test case T j tests requirement R i completely 1, if test case T j tests requirement R i partially 0, otherwise Pi = Priority number of Ri RCF j can be used to rank the test cases in terms of the coverage provided to a given set of requirements. This can be used in the selection of a sub-set of test cases from the available population of test cases that would maximize the coverage provided to the set of specified requirements. The RCF for testing can be computed as given below: n ∑ RCFj j =1 RCFTesting = n RCFTesting can be used to measure the coverage provided by a set of selected test cases to the set of specified requirements. The effectiveness of the test case T j is defined as a measure of the coverage provided by a test case vis -a-vis the effort needed for its execution. It is calculated as: RCFj Effectiveness j = Ej Where, Ej = Time required for executing the test case (in hours). Effectiven ess j can be used to rank the test cases in terms of their effectiveness. This can be used in the selection of a sub-set of test cases from the available population of test cases that would maximize the coverage under the condition that the effort available for execution is constrained.
  6. 6. The effectiveness of testing can be computed as follows: n ∑ Effectiveness j j =1 Effectiven Testing = ess n Effectiven essTesting can be used to measure the effectiveness of testing in terms of coverage provided by a set of test cases vis-a-vis the effort needed for their execution. For each requirement R i , j = 1 to m , coverage factor (CFi ) is defined as the coverage provided to it by a set of test cases T j , j = 1 to n vis-a-vis the effort needed for test execution. This is the summation of effectiveness of all the test cases that test that requirement. n CFi = ∑ Y j Effectiven ess j j =1 Where, Yj = 2, if requirement R i is completely tested by test case T j 1, if requirement R i is partially tested by test case T j 0, otherwise The coverage factor for testing CF can calculated as follows: m ∑ CFi i =1 CFTesting = m CFTesting can be used to evaluate requirements in terms of coverage from the execution of a set of selected test cases. This can be used in determining whether a requirement has been tested adequately or not. To summarize, the above metrics can be used for the following purposes: a) Review of test cases The RCF of the test cases can be used as a factor in the review of test cases. b) Selection of test cases The effectiveness of the test cases can be used to prioritize the test cases on the basis of returns in terms of coverage versus the effort needed for execution. In case of reduced cycle time, the
  7. 7. selection of test cases can be done more objectively based on priority. c) Setting acceptance criteria of the product from a level to the next higher level Consider the example of transmittal of a product from level Lk to level Lk +1 . In a well-defined process there should be clear and matching exit and entry criteria for both the levels. The criteria could be defined as: “If the RCF of the product at the end of level Lk testing is 0.9 with respect to the level Lk requirements, then it satisfies the entry criteria for level Lk +1 ". d) Stopping criteria for testing The decision on when to stop testing based on completeness of testing can be stated in terms of not only the number of test cases (executed and passed) but also the coverage of requirements. e) Making release decisions The evaluation of the product in terms of coverage of requirements would help in making release decisions more objectively. 4 CASE STUDY In this section, the calculation of the metrics given in Section 3 and their usage has been illustrated using a case study. Consider the development of a Directory Server that has requirements specified at four levels as shown in the Table 2. Note that the statements vary in detail from one level to the other. Each requirement is assigned a priority number 1, 2 or 3. A requirement with priority number 1 is of low priority, 2 of medium priority and 3 of highest priority.
  8. 8. L1 L2 L3 L4 R11: The search performance must be R21: The search performance must to R31:A new sub-string search for and existing R41:The search must return all the matching objects (3) better than the previous version of the atleast 10% greater than that with the object in a database with 10000 objects must R42:Indexing must be done on most commonly sought server (3) previous version of the server (3) be less than or equal to 2 seconds (3) attributes (2) R43:The search results must be sorted by common name (3) R44:The output must be in Base64 encoding (3) R32: A non-existent object search must take less than or equal to 5 seconds (2) R33: A referral should be given in case of an unfruitful search (1) R34: An all-attribute search for and existing (2) R35: The request complete listing of 10000 objects in the database must take less than or equal to 10 seconds (2) R22: The search algorithm must be modified to make it more efficient (3) R23: The search results must be cached and the cache must be updated periodically (2)
  9. 9. L1 L2 L3 L4 R12: The search performance must be better than that of the competitive product (3) R13: The server should comply with the industry standard (?) (2) Table 2 (Note: the number in bracket indicates the priority of the requirement)
  10. 10. Consider the level 3 requirements R1 to R5 that have to be tested. T1 to T10 is the available population of test cases. Xi for each test case is evaluated based on how much it tests a requirement, as specified in Section 3.2. Table 3 shows the metrics and calculations. Test cases Prior Cover T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 ity age CF R i R1 1 2 2 1 2 0 1 0 2 1 3 12 3.35 e r R2 2 1 2 0 2 2 2 2 1 0 2 14 3.37 q e R3 0 2 1 2 2 0 2 1 0 2 1 12 2.99 u mR4 1 2 0 1 2 2 1 0 0 1 2 10 2.86 R5 1 0 0 0 1 1 0 0 0 2 2 5 1.42 10 10.60 2.80 Effort (E) 2 4.5 5 2 1 8 7 3 2 6 RCF 0.55 0.70 0.55 0.35 0.90 0.50 0.55 0.25 0.40 0.55 0.53 Effectiveness 0.16 0.11 0.28 0.18 0.90 0.06 0.08 0.08 0.20 0.09 0.21 Table 3 The ordering of test cases based on RCF can be used in the selection of a sample of test cases from the population of available test cases that would maximize the coverage. In this case, test case T5 has the maximum RCF and hence provides the highest coverage to the set of specified requirements. Similarly, the ordering of requirements based on the coverage factor (CF) can be used to determine the coverage acquired by a requirement from the execution of a sample of test cases. In the above case, requirement R2 has the highest CF, which indicates that it acquires the highest coverage . Decisions can be made based on specified targets for the various metrics described earlier in this paper. In this example, for making decisions the following heuristics can be defined: a) A requirement is “completely” met if and only if 100% of the test cases with X i = 2, testing the requirement are pass and 80% of the test cases with X i = 1 pass. In the above example, R1 is completely met if T2, T3, T5, T9 are passed and any three of T1, T4, T7, T10 are passed.
  11. 11. a) A requirement is “partially” met if 80% of the test cases with X i = 2 are pass and more than 70% of the test cases with X i = 1 pass. In the above example, R1 is partially met if any three of T2, T3, T5, T9 are passed and any three of T1, T4, T7, T10 pass. b) The product can enter the next level is RCF is greater than 0.5. One additional criterion can be that if the test coverage of all requirements should be greater than 3 for the product to enter next level of testing. In the above example, if R1, R2 and R3 are completely met, R4 partially met and R5 not met, then RCF of the product at this level is 0.7. Based on the heuristic, it can enter the next level in the process. 5 CONCLUSIONS The development of a software product is essentially driven by requirements. Before the software is released to the customers, it needs to be validated against the requirements. Testing has been conventionally used for validation through execution of test cases. The effectiveness of testing vis-à- vis requirements can be measured in terms of mapping of test cases to requirements. This paper explains metrics to measure the mapping. These metrics would help in making decisions on when to stop testing and release the product in more objective manner as has been illustrated in this paper. REFERENCES [1] Hammer, Theodore, et al, Measuring Requirements Testing, http://satc.gsfc.nasa.gov/support/ICSE_MAY97/icse/icse97.html [2] Wilson, M William, Writing Effective Requirements Specifications, http://satc.gsfc.nasa.gov/support/STC_APR97/write/writert.html
  12. 12. Acknowledgements We would like to express our gratitude to the following persons from Novell Bangalore - Mr. J. Veeraraghavan, Mr. Srinivasan Desikan, Mr. Jyoti Prakash Kurmi and Mr. Manu Sreenivasan - for their valuable inputs towards this paper. About the Authors Chandana Pavuluru is working as a Senior Software Engineer with the System Testing Team at Novell Bangalore for past 3 years. Hemanta Chandra Bhatt is working as Senior Software Engineer with the Quality Engineering Group at Novell Bangalore for the past 2 years.

×