A Framework for Adaptive Regression Testing Jaymie Strecker Maryland Software Day 18 January 2008
Regression Testing <ul><li>As developers grow and change software, they inevitably introduce defects  </li></ul><ul><li>Re...
How Exactly to Regression Test? <ul><li>Lots of techniques for regression testing available </li></ul><ul><ul><li>Re-run, ...
Observation 1: Moving Target <ul><li>Application changes over time </li></ul><ul><li>So might the developers’ needs and re...
Observation 2: Learn As You Go <ul><li>Regression testing is iterative </li></ul><ul><li>Experience (data) from one iterat...
Non-adaptive Regression Testing Application Test suite Faults found
Adaptive Regression Testing Data repository Application Test suite Faults found Statistical model Profiles of test suites ...
Example with CrosswordSage <ul><li>Tester needs to test a new version of CrosswordSage (SourceForge) </li></ul>
Example with CrosswordSage <ul><li>From past iterations of regression testing, the tester has built up a data repository <...
Example: Statistical Model <ul><li>Tester picks some characteristics of test suites and faults that seem likely to affect ...
Example: Statistical Model <ul><li>Tester builds and assesses a statistical model from data in the repository </li></ul>Pr...
Example: New Test Strategy <ul><li>Predictions from our own experiment:  </li></ul><ul><ul><li>Increasing E3Cov by 10% mor...
Still To Do <ul><li>More experiments </li></ul><ul><ul><li>Find characteristics of test suites, faults, applications, etc....
Summary <ul><li>Framework for adaptive regression testing </li></ul><ul><ul><li>Ideal testing strategy is a moving target ...
Upcoming SlideShare
Loading in …5
×

PPT

779 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
779
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PPT

  1. 1. A Framework for Adaptive Regression Testing Jaymie Strecker Maryland Software Day 18 January 2008
  2. 2. Regression Testing <ul><li>As developers grow and change software, they inevitably introduce defects </li></ul><ul><li>Regression testing is a popular solution </li></ul><ul><ul><li>One step in iterative development process </li></ul></ul><ul><ul><li>Test each version of the software before releasing it </li></ul></ul>v.1 v.2 v.3 v.4 Test Test Test Test
  3. 3. How Exactly to Regression Test? <ul><li>Lots of techniques for regression testing available </li></ul><ul><ul><li>Re-run, prioritize, or reduce existing test suite </li></ul></ul><ul><ul><li>Any testing technique </li></ul></ul><ul><li>What is the best technique for my situation? </li></ul>v.1 v.2 v.3 v.4 Test Test Test Test
  4. 4. Observation 1: Moving Target <ul><li>Application changes over time </li></ul><ul><li>So might the developers’ needs and resource constraints </li></ul><ul><li>Shouldn’t the testing strategy change as well? </li></ul><ul><ul><li>What kind of test suite you want to use </li></ul></ul>v.1 v.2 v.3 v.4 Test Test Test Test
  5. 5. Observation 2: Learn As You Go <ul><li>Regression testing is iterative </li></ul><ul><li>Experience (data) from one iteration can be used to improve the next </li></ul><ul><ul><li>Ideal testing strategy is a moving target </li></ul></ul><ul><ul><li>One-size-fits-all strategy may not exist </li></ul></ul><ul><ul><ul><li>Tailor to developers’ unique situation </li></ul></ul></ul>v.1 v.2 v.3 v.4 Test Test Test Test Data Data Data
  6. 6. Non-adaptive Regression Testing Application Test suite Faults found
  7. 7. Adaptive Regression Testing Data repository Application Test suite Faults found Statistical model Profiles of test suites Profiles of faults Probabilities of fault detection
  8. 8. Example with CrosswordSage <ul><li>Tester needs to test a new version of CrosswordSage (SourceForge) </li></ul>
  9. 9. Example with CrosswordSage <ul><li>From past iterations of regression testing, the tester has built up a data repository </li></ul><ul><ul><li>Test cases and test suites used </li></ul></ul><ul><ul><li>Faults detected </li></ul></ul>
  10. 10. Example: Statistical Model <ul><li>Tester picks some characteristics of test suites and faults that seem likely to affect fault detection </li></ul>E3Cov Test suite’s event-triple coverage Mut Fault’s mutation type (method- or class-level) StmtDet Fault’s estimated probability of detection by statement-coverage-adequate test suite
  11. 11. Example: Statistical Model <ul><li>Tester builds and assesses a statistical model from data in the repository </li></ul>Pr ( Det ) = f ( E3Cov , Mut , StmtDet ) E3Cov Test suite’s event-triple coverage Mut Fault’s mutation type (method- or class-level) StmtDet Fault’s estimated probability of detection by statement-coverage-adequate test suite
  12. 12. Example: New Test Strategy <ul><li>Predictions from our own experiment: </li></ul><ul><ul><li>Increasing E3Cov by 10% more than doubles odds of fault detection </li></ul></ul><ul><ul><li>Faults more detectable with full statement coverage are also more detectable with full GUI event coverage </li></ul></ul><ul><ul><li>Class-level mutants more detectable </li></ul></ul>E3Cov Test suite’s event-triple coverage Mut Fault’s mutation type (method- or class-level) StmtDet Fault’s estimated probability of detection by statement-coverage-adequate test suite
  13. 13. Still To Do <ul><li>More experiments </li></ul><ul><ul><li>Find characteristics of test suites, faults, applications, etc. that are good predictors of fault detection </li></ul></ul><ul><ul><li>Validate our framework </li></ul></ul><ul><li>Tools and infrastructure to store data, build statistical models, and suggest testing strategies automatically </li></ul>
  14. 14. Summary <ul><li>Framework for adaptive regression testing </li></ul><ul><ul><li>Ideal testing strategy is a moving target </li></ul></ul><ul><ul><li>Tailor strategy to developers’ unique situation </li></ul></ul><ul><ul><li>Learn from previous iterations of testing how to improve the next iteration </li></ul></ul><ul><li>More information: </li></ul><ul><ul><li>J. Strecker and A. M. Memon. Faults’ context matters. SOQUA 2007. </li></ul></ul><ul><ul><li>J. Strecker and A. M. Memon. Relationships between test suites, faults, and fault detection in GUI testing. ICST 2008. </li></ul></ul><ul><ul><li>J. Strecker and A. M. Memon. Toward adaptive search-based regression testing. Submitted to JSME special issue on search-based software engineering. </li></ul></ul>

×