Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Specification-basedregression test selection   with risk analysis       Zheng-Wen Shen         2005/03/14
Reference   Specification-based regression test selection with    risk analysis     Yanping Chen, Robert L. Probert, D. ...
Outline1.   Introduction2.   Background3.   Specification-based test case selection4.   Risk-based test case selection5.  ...
1. Introduction   Regression testing is essential to ensure software    quality.   Regression test selection attempts to...
White-box (code-based) V.S. Black-    box (specification-based)   Code-based regression test selection is good for    uni...
Two kinds of regression tests   Targeted Tests, which ensure that important    current customer features are still suppor...
2. Background   2.1 Regression testing   2.2 Controlled Regression Testing Assumption    (CRTA)   2.3 Risk Analysis   ...
2.1 Regression testing   Given a program P, a modified version P’, and a    set T of test cases used previously to test P...
2.2 Controlled Regression Testing          Assumption (CRTA)   When P’ is tested with T, we hold all factors that    migh...
2.3 Risk Analysis   Risk is anything that threatens the successful    achievement of a project’s goals.   Using risk met...
1.   P(f), the probability of a fault being present.2.   C(f), the cost (consequence or impact) of a     fault in the corr...
2.4 Requirement traceability   Traceability is a simple, common-sense book-    keeping property that can prevent a wide r...
The activity diagram and Test suite for the Get_Quote feature
Traceability matrix between the activity diagram and test suite
3. Specification-based test case                  selection   3.1 Changes in code   3.2 Changes in specification
3.1 Changes in code   Changes only happen in the implementation    without affecting the specification.   For a develope...
3.2 Changes in specification   Changes happen in requirements or design, the    activity diagram need to be modified acco...
A CFG-based regression test selection technique
Edge-coverage matrix for test suite T on CFG C
Regression analysis with the activity              diagram1.   We traverse the activity diagram to identify     affected e...
4. Risk-based test case selection   4.1 Motivation for risk-based selection.   4.2 Model-based selection of Safety Tests...
4.1 Motivation for risk-based                 selection   If the developer changes code that cannot be    identified by a...
Useful in two situations1.   After running the Targeted Tests, if time and     resources are available, testers may want t...
4.2 Model-based selection of Safety                  Tests    Uses Risk Exposure model for risk analysis.    There are f...
Step 1. Assess the cost of each test case.    Cost means the cost of the requirements     attributes that this test case ...
Cost of test cases
Step 2. Derive severity probability for each                      test case   How important or serious the defect is.
Step 3. Calculate Risk Exposure for each test                     case
Step 4. Select Safety Tests   We choose test cases that have the highest value    of RE(t) from the non-targeted tests to...
4.3 Risk-based end-to-end test               scenario selection   End-to-end scenarios simulate common user    profiles o...
    One end-to-end consists of several test cases.     Every test case tests one or more requirements     attributes.   ...
Traceability between test cases and scenarios
   Step 1. Calculate Risk Exposure for each    scenario   Step 2. Select scenario that has the highest RE(s)   Step 3. ...
Step 1. Calculate Risk Exposure for               each scenario   RE (s) = ΣRE (t)i , {1 ≤ i ≤ n | test case i is    cove...
Step 2. Select scenario that has the               highest RE(s)   The scenario with the highest RE(s) covers the    most...
Step 3. Update table and rebuild             table.
Rebuilt table of Risk Exposure for             scenarios
Step 4. Repeat Step 2 and Step 3 until  we run out of time and resources   The size of our test suite of scenarios is    ...
5. Evaluation   We applied our approach to three components    of IBM WebSphere Commerce 5.4. Each    component was owned...
Three assessment factors1.   Effectiveness2.   Cost-efficiency3.   Sensitivity to Risk
6. Conclusions   This technique is Customer-oriented and also Risk-    based.   It provides methods to obtain both Targt...
Upcoming SlideShare
Loading in …5
×

20050314 specification based regression test selection with risk analysis

790 views

Published on

Specification-based regression test selection with risk analysis

Published in: Technology
  • Be the first to comment

20050314 specification based regression test selection with risk analysis

  1. 1. Specification-basedregression test selection with risk analysis Zheng-Wen Shen 2005/03/14
  2. 2. Reference Specification-based regression test selection with risk analysis  Yanping Chen, Robert L. Probert, D. Paul Sims  Source IBM Centre for Advanced Studies Conference archive  Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research table of contents
  3. 3. Outline1. Introduction2. Background3. Specification-based test case selection4. Risk-based test case selection5. Evaluation6. Conclusions
  4. 4. 1. Introduction Regression testing is essential to ensure software quality. Regression test selection attempts to select a cost-minimized subset of test cases.
  5. 5. White-box (code-based) V.S. Black- box (specification-based) Code-based regression test selection is good for unit testing, but it has a scalability problem. Code-based regression techniques are not suitable for testing larger components at more abstract level. Code-based regression techniques require testers to access and understand the code. Code-based regression techniques are language- dependent.
  6. 6. Two kinds of regression tests Targeted Tests, which ensure that important current customer features are still supported adequately in the new release. Safety Tests, which are risk-directed, and ensure the potential problem areas are properly handled.
  7. 7. 2. Background 2.1 Regression testing 2.2 Controlled Regression Testing Assumption (CRTA) 2.3 Risk Analysis 2.4 Requirement traceability
  8. 8. 2.1 Regression testing Given a program P, a modified version P’, and a set T of test cases used previously to test P, regression analysis and testing techniques attempt to make use of a subset of T to gain sufficient confidence in the correctness of P’ with respect to behaviors from P retained in P’. With selective re-test techniques, we only return those test cases that test the affected entities of the modified program.
  9. 9. 2.2 Controlled Regression Testing Assumption (CRTA) When P’ is tested with T, we hold all factors that might influence the output of P’, except for the code in P’, constant with respect to their states when we tested P with T. Ideally, regression tests should be run while CRTA holds. Testers must identify and document uncontrollable factors and the test cases that are potentially affected by them.
  10. 10. 2.3 Risk Analysis Risk is anything that threatens the successful achievement of a project’s goals. Using risk metrics to quantitatively measure the quality of a test suite. Risk Exposure model by Stale Amland.  RE(f) = P(f) * C(f)
  11. 11. 1. P(f), the probability of a fault being present.2. C(f), the cost (consequence or impact) of a fault in the corresponding function if it occurs in operation.
  12. 12. 2.4 Requirement traceability Traceability is a simple, common-sense book- keeping property that can prevent a wide range of problem.  Know which test case verifies a given requirements attribute. We use the activity diagram to represent the desired system behavior.
  13. 13. The activity diagram and Test suite for the Get_Quote feature
  14. 14. Traceability matrix between the activity diagram and test suite
  15. 15. 3. Specification-based test case selection 3.1 Changes in code 3.2 Changes in specification
  16. 16. 3.1 Changes in code Changes only happen in the implementation without affecting the specification. For a developer, any changes in code should be documented in a code change history. For a tester, traceability needs to be established between the test case and the log of defects.  Test profile The tester can choose test cases using the traceability matrix.
  17. 17. 3.2 Changes in specification Changes happen in requirements or design, the activity diagram need to be modified accordingly. In a control flow graph (CFG), nodes are used to represent statements, and edges represent the control flow between the statement within a procedure. We find that the activity diagram is quite similar to the CFG except that the nodes of activity diagram describe activities instead of statements.
  18. 18. A CFG-based regression test selection technique
  19. 19. Edge-coverage matrix for test suite T on CFG C
  20. 20. Regression analysis with the activity diagram1. We traverse the activity diagram to identify affected edges.2. Then we select test cases that execute the affected edges based on the traceability matrix to create our Targeted Tests.
  21. 21. 4. Risk-based test case selection 4.1 Motivation for risk-based selection. 4.2 Model-based selection of Safety Tests. 4.3 Risk-based end-to-end test scenario selection.
  22. 22. 4.1 Motivation for risk-based selection If the developer changes code that cannot be identified by a test profile, and he does not update the code change history, the tester might miss some defects. To achieve enough confidence, we would like to include some test cases in addition to the Targeted Tests for our regression test suite.
  23. 23. Useful in two situations1. After running the Targeted Tests, if time and resources are available, testers may want to return some test cases to gain more confidence in the modified system.2. The time to delivery may be extremely short, Safety Tests provide some assurance that the remaining defects in the release will not bring about serious failures.
  24. 24. 4.2 Model-based selection of Safety Tests Uses Risk Exposure model for risk analysis. There are four main steps in our approach. 1. Assess the cost of each test case. 2. Derive severity probability for each test case. 3. Calculate Risk Exposure for each test case. 4. Select Safety Tests.
  25. 25. Step 1. Assess the cost of each test case. Cost means the cost of the requirements attributes that this test case covers. Two kinds of costs will be taken into consideration. 1. The consequences of a fault as seen by the customer, that is, losing market share because of faults. 2. The consequences of a fault as seen by the vendor, that is, high software maintenance cost because of faults.
  26. 26. Cost of test cases
  27. 27. Step 2. Derive severity probability for each test case How important or serious the defect is.
  28. 28. Step 3. Calculate Risk Exposure for each test case
  29. 29. Step 4. Select Safety Tests We choose test cases that have the highest value of RE(t) from the non-targeted tests to reach our coverage target.
  30. 30. 4.3 Risk-based end-to-end test scenario selection End-to-end scenarios simulate common user profiles of system uses. Some are equivalent to use cases.  More customer-directed.  Highly effective at finding regression faults.
  31. 31.  One end-to-end consists of several test cases. Every test case tests one or more requirements attributes. Selection strategy obeys two rules 1. Select scenarios that cover the most critical test cases. 2. Have the suite of scenarios cover as many test case as possible.
  32. 32. Traceability between test cases and scenarios
  33. 33.  Step 1. Calculate Risk Exposure for each scenario Step 2. Select scenario that has the highest RE(s) Step 3. Update table and rebuild table. Step 4. Repeat Step 2 and Step 3 until we run out of time and resources.
  34. 34. Step 1. Calculate Risk Exposure for each scenario RE (s) = ΣRE (t)i , {1 ≤ i ≤ n | test case i is covered by this scenario}
  35. 35. Step 2. Select scenario that has the highest RE(s) The scenario with the highest RE(s) covers the most critical test cases, that is, has the highest coverage of test cases. According to our selection rules, it should be included in the regression test suite. For example: s001 is put first into the regression test suite.
  36. 36. Step 3. Update table and rebuild table.
  37. 37. Rebuilt table of Risk Exposure for scenarios
  38. 38. Step 4. Repeat Step 2 and Step 3 until we run out of time and resources The size of our test suite of scenarios is dependent on the time and resources available. Regression testing is terminated whenever we run out of time and resources. The final selected regression test suite is the union of the Targeted Tests and the Safety Tests.
  39. 39. 5. Evaluation We applied our approach to three components of IBM WebSphere Commerce 5.4. Each component was owned by one experienced tester. (306 test cases) 65 defects were opened in the first pass. There were 9 defects found, which failed 28 test cases in the second pass.
  40. 40. Three assessment factors1. Effectiveness2. Cost-efficiency3. Sensitivity to Risk
  41. 41. 6. Conclusions This technique is Customer-oriented and also Risk- based. It provides methods to obtain both Targted Tests and Safety Tests. Future work  To decide when to stop regression testing.  Implementing our approach in a production test environment.

×