Successfully reported this slideshow.
Your SlideShare is downloading. ×

A Survey on Automatic Test Generation and Crash Reproduction

Upcoming SlideShare
Brief Introduction To Teseda
Brief Introduction To Teseda
Loading in …3
×

Check these out next

1 of 56 Ad
1 of 56 Ad
Advertisement

More Related Content

Viewers also liked (20)

Similar to A Survey on Automatic Test Generation and Crash Reproduction (20)

Advertisement
Advertisement

A Survey on Automatic Test Generation and Crash Reproduction

  1. 1. 1 A SURVEY ON AUTOMATIC TEST GENERATION AND CRASH REPRODUCTION PhD Qualifying Examination Ning Chen
  2. 2. 2 Outline • Automatic Test Generation • Feedback-directed Random Generation • Captured-Object based Generation • Genetic Evolution based Generation • Symbolic Execution • Dynamic Symbolic Execution • Automatic Crash Reproduction • Record-and-replay Approaches • Static analysis based Approaches • Summary
  3. 3. 3 Automatic Test Generation
  4. 4. 4 Automatic Test Generation Automatic Test Generation Symbolic Search Execution Based Based Feedback- Object- Genetic Symbolic Dynamic Symbolic directed capturing Evolution Execution Execution Common Challenge: My Approach Object Creation Challenge
  5. 5. 5 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  6. 6. 6 Feedback-directed Random Test Generation Randomly select a new method call Classes to test Error- Generate Execute Examine revealing test Method Method Execution cases Sequences Sequences Result Time Limit Regression test cases Properties Method to check Sequences Method Arguments If Sequence Legal Generated
  7. 7. 7 Feedback-directed Random Test Generation Randomly select a new Classes to method call test Error- Generate Execute revealing test Method Method cases Sequences Sequences Time Limit Regression test cases Properties Method to check Sequences Feedback to guide If Sequence Legal Generated
  8. 8. 8 Feedback-directed Random Test Generation • Advantages: • Scalable to large subject programs • Known to be easy to implement • Generates no or little redundant method call sequences • Challenges: • Generates too many test cases (thousands of test cases) • Cannot generate complex method sequences, thus may not achieve good coverage for programs with complex structures
  9. 9. 9 Feedback-directed Random Test Generation 01 void execute(tinySQLStatement stmt, String sqlStr) { 02 if (stmt.execute(sqlStr)) { 03 // target branch 04 } 05 } • cannot cover the target branch because it is difficult to generate the proper sql statement randomly • In short, Random approach fails to create test inputs that can reach deeper part of the subject program.
  10. 10. 10 Follow-up works Combining Dynamic Exec. with Random Generation: S. Artzi, M. D. Ernst, A. Kie˙zun, C. Pacheco, and J. H. Perkins. Finding the needles in the haystack: Generating legal test inputs for object-oriented programs. In 1st Workshop on Model-Based Testing and Object-Oriented, 2006. (Palulu) Combining Static Exec. with Random Generation: W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with mut- aware sequence recommendation. In ASE, 2010. (RecGen) Combining Dynamic + Static Exec. with Random Generation: S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, Combined static and dynamic automated test generation, in ISSTA, 2011 (Palus)
  11. 11. 11 Follow-up works • The common goal: more diverse and complex test inputs that can reach deeper part of the subject program. • However, there is an alternative…
  12. 12. 12 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT 10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  13. 13. 13 Object Captured Based Randomly select a new method call Classes to test Generate Execute Error- Method Method revealing test Sequences Sequences cases Time Limit Arguments Regression Method test cases Sequences Properties If Sequence Legal Generated to check Runtime Existing Test Captured Cases / Normal Objects Executions
  14. 14. 14 Object Captured Based • Advantages: • Can generate test inputs that are complex for random approaches • Can be integrated with other test generation approaches. • E.g. Seed objects for Random generation. • Challenges: • Fail to demonstrate an object generating method sequence. • Making a test case less useful. • Rely heavily on the quality and quantity of existing test cases. • Captured objects may contain sensitive private information. • James Clause et. al - ICSE2011
  15. 15. 15 Object Captured Based 01 void test_execute() { 02 SQLStatement stmt = load(“…/SQLStatement/001”); 03 String sqlStr = load(“…/String/123”); 02 if (stmt.execute(sqlStr)) { 03 // target branch 04 } 05 } • Apparently, such a test case would be less useful as it does not provide much information.
  16. 16. 16 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  17. 17. 17 Genetic Evolution Based • Adopts evolutionary computation techniques to synthesize method sequences for reaching higher code coverage. • Method sequences (chromosome) are crossed-over and mutated to produce different varieties of offspring. • The fitness score is computed according to the code coverage. • Method sequences that can achieve higher code coverage will be selected for future evolution.
  18. 18. 18 Genetic Evolution Based • Advantages: • Have stronger capabilities of generating complex test inputs over random approaches. • Challenges: • Might not scale to large subject programs. • Factors such as fitness functions, mutation operators, evolution algorithms can greatly affect the quality of the generations.
  19. 19. 19 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Search Based Test Generation Approaches
  20. 20. 20 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Symbolic Execution Based Test Generation Approaches
  21. 21. 21 Symbolic Execution • Proposed as early as the 70s but become realistic only in recent years • Generate tests by symbolically executing all paths in the program • Rely on constraint solvers to compute test inputs (i.e. models)
  22. 22. 22 Symbolic Execution 01 void foo(int x, int y) { 02 if (x > y + 1) { 03 // branch 1 04 } 05 else { 06 // branch 2 07 } 08 }
  23. 23. 23 Symbolic Execution 01 void foo(int x, int y) { Symbolic Input: X, Y 02 if (x > y + 1) { Path Condition: X > Y + 1 03 // branch 1 04 } 05 else { Constraint Solver -> X = 2, Y = 0 06 // branch 2 07 } Generate actual test case 08 }
  24. 24. 24 Symbolic Execution 01 void foo(int x) { Symbolic Input: X, Y 02 if (x > y + 1) { Path Condition: X <= Y + 1 03 // branch 1 04 } 05 else { Constraint Solver -> X = 0, Y = 0 06 // branch 2 07 } Generate actual test case 08 }
  25. 25. 25 Symbolic Execution • Advantages: • Feasible in recent years with the increase of computation power. • Capable of achieving higher structural coverage than search based techniques.
  26. 26. 26 Symbolic Execution • Challenges: • Constraint Solving Limitations: • Non-linear • Floating point, • Bit-vector, etc. • External Method Calls: • External Libraries, System Level Calls, etc. • Cannot Symbolically execution these external method calls. • The Object Creation Problem: • In OO, test inputs (objects) needs to be constructed in sequence of method calls.
  27. 27. 27 Follow-up work Complex Constraint Solving Challenge: C. S. P˘as˘areanu, N. Rungta, and W. Visser. Symbolic execution with mixed concrete-symbolic solving. In ISSTA ‟11 I. Erete and A. Orso. Optimizing constraint solving to better support symbolic execution. In CSTVA‟11 The Object Creation Problem: S. Thummalapenta, T. Xie, N. Tillmann, P. de Halleux, and W. Schulte. MSeqGen: Object-oriented unit-test generation via mining source code. In ESEC/FSE‟09
  28. 28. 28 Automatic Test Generation Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  29. 29. 29 Dynamic Symbolic Execution • Symbolic execution Ideally can achieve complete coverage, but usually not the case in reality • Recent years, Dynamic Symbolic Execution is proposed. • EXE/KLEE „05 • DART „05 • CUTE „05, JCUTE „06 • PEX „08 • DSE is designed to handle two major challenges in Symbolic Execution: • 1) Constraint Solving, and • 2) External Method Calls
  30. 30. 30 Dynamic Symbolic Execution 01 void foo(int x, int y) { 02 if (external (x) == y) { No source access for external() 03 // branch 1 04 } 05 else if (hash(x) > y) { hash() is complex arithmetic 06 // branch 2 07 } 08 } Key Idea: Combines Dynamic execution + Symbolic execution
  31. 31. 31 Dynamic Symbolic Execution 01 void foo(int x, int y) { Dynamic Exec. 02 if (external(x) == y) { external(0) == 1 03 // branch 1 04 } 05 else if (hash(x) < y) { hash(0) == 10 06 // branch 2 07 } 08 } First Iteration (Random Input): Dynamic Exec: x = 0 and y = 0
  32. 32. 32 Dynamic Symbolic Execution 01 void foo(int x, int y) { Symbolic Exec. 02 if (external(x) == y) { external(X) == Y 03 // branch 1 04 } external(0) == 1 05 else if (hash(x) < y) { 06 // branch 2 Constraint Solver: 07 } X == 0, Y == 1 08 } Second Iteration: Path Condition: external(X) == Y
  33. 33. 33 Dynamic Symbolic Execution 01 void foo(int x, int y) { Symbolic Exec. 02 if (external(x) == y) { external(X) != Y 03 // branch 1 04 } hash(X) < Y 05 else if (hash(x) < y) { 06 // branch 2 external(0) == 1 hash(0) == 10 07 } 08 } Constraint Solver: X == 0, Y == 11 Third Iteration: Path Condition: external(x) == y && hash(x) < y
  34. 34. 34 Dynamic Symbolic Execution • Advantage: • Can handle situations where traditional symbolic executions cannot handle. i.e. Constraint Solving, External Method Calls • Achieve higher coverage compared to only symbolic execution.
  35. 35. 35 Dynamic Symbolic Execution • Challenges: • A recent paper by Xiao ‟11. investigates the major challenges faced by a state-of-the-art DSE framework for C#, Pex. Project LOC Cov % Object Creation Problem SvnBridge 17.1K 56.26 11 (42.31%) xUnit 11.4K 15.54 8 (72.73%) Math.Net 3.5K 62.84 17 (70.83%) QuickGraph 8.3K 53.21 10 (100%) Total 40.3K 49.87 46 (64.79%) • The object creation challenge accounts for 65% of all cases that cause code blocks to be not covered by Pex.
  36. 36. 36 The Object Creation Problem void foo(Container container) { if (container.size >= 10) { Model: // not covered branch container.size = 10 … } The Object Creation Problem : How do we create and mutate a Container object into size = 10?
  37. 37. 37 Dynamic Symbolic Execution • In object-oriented programs, modifying a non-public field requires a set of legal method sequence • e.g. Container.size == 10 • Possible method sequence: Container.add(Object); … Container.add(Object); • But as the structure of the classes become more complex, so will the object creation problem. • Therefore, the object creation problem is an important, yet unsolved problem in automatic test generation.
  38. 38. 38 Demand-driven Object Generation • To address this OCP, I am working on a project for Demand-Driven Object Generation. • Goal: Given a target state -> generate a legal method sequence that can create and mutate object that satisfy this target state • Target state: container.size == 10 • Possible Output: container.add(new Object()); ten times
  39. 39. 39 Demand-driven Object Generation • The approach should be demand-driven, so only legal sequence satisfying the target state can be generated. • It is helpful to solve the object creation problem mentioned. • Thus, it can improve the coverage for dynamic symbolic execution approaches for object-oriented programs.
  40. 40. 40 Automatic Crash Reproduction
  41. 41. 41 Automatic Crash Reproduction Automatic Crash Reproduction Static Record- Analysis and-replay Based Common Challenges My Approach
  42. 42. 42 Automatic Crash Reproduction Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Categories Representatives Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10 Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09
  43. 43. 43 Record-and-Replay • Approach: • Monitoring Phase: Captures/Stores runtime heap & stack objects. • Test Generation Phase: Generates tests that loads the correct objects with the crashed methods. Original Program Execution Store from heap & stack Stored Objects Load as crashed method params Recreated Test Case
  44. 44. 44 Record-and-Replay • Advantage: • Can usually reproduce target crashes reliability • Challenges: • Require instrumentation in advance • Extra runtime overhead, i.e. ReCrash: • 13-64% performance overhead, and • 3-90% memory overhead • Extensive amount of data from users: objects in heap, running context, etc • Difficult to obtain ( > 100M ) • Privacy issues: James Clause et. al @ ICSE2011
  45. 45. 45 Record-and-Replay • Existing Frameworks: • Jrapture ‟00, • BugNet ’05, • ReCrash/ReCrashJ ‟08 • LEAP/LEAN ’10 • Existing approaches have been trying various techniques to reduce the runtime overhead yet maintain a good reproduction result.
  46. 46. 46 Automatic Crash Reproduction Categories Representatives Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11 Captured-Object Based MSeqGen’09, OCAT’10 Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09 Symbolic Execution King’76, Clarke’76, JPF’04 Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Categories Representatives Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10 Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09
  47. 47. 47 Static Analysis Approaches • Microsoft PSE ‘04 • Postmortem static analysis framework. • Explains the potential cause of a crash through symbolic execution. • XyLem ’09 • Null-dereference analysis tool for Java language. • Identifies the precise paths along which null values may flow to de- reference. • IBM SnuggleBug ‘09 • Demand-driven backward symbolic execution framework. • Computes the weakest preconditions of crashes. • But does not generate actual crash triggering objects from the computed preconditions.
  48. 48. 48 STAR: automatically reproduces failures • All of the above approaches either only finds error path, or computes the error triggering preconditions. • They do not support the full reproduction of a crash from the crash information. • We propose STAR, an automatic crash reproduction framework • Requires only minimal information (crash stack trace, crash type) • No client site deployment and recording necessary • No performance overhead • No need to collect any private data from user. • Supports both finding out the failure preconditions as well as recreating the actual failures.
  49. 49. 49 STAR: automatically reproduces failures crash precondition stack trace processing crash condition preparation computation 1 2 3 bug report x() at nnn {var == null} y() at yyy {true} z() at zzz … crash 4 test case generation reproduction program test TestGen: A novel test generation technique cases
  50. 50. 50 Backward Symbolic Execution • Goal: Given a crash location, compute at a public method entry, the precondition that can trigger the crash • There have been several previous work in this direction • XYLem ICSE ‟09 • SnuggleBug PLDI ‟09 • STAR has a more efficient backward symbolic execution engine that improves existing approaches with heuristics: • Pruning irrelevant branches • Smarter backtracking • Early detection of inner contradictions
  51. 51. 51 STAR: automatically reproduces failures crash precondition stack trace processing crash condition preparation computation 1 2 3 bug report x() at nnn {var == null} y() at yyy {true} z() at zzz … crash 4 test case generation reproduction program test TestGen: A novel test generation technique cases
  52. 52. 52 Crash Test Case Generation • Essentially, this is the same problem as the demand- driven method sequence generation we‟ve discussed. • Thus, we combine our efficient backward symbolic execution approach with the demand-driven method sequence generation to achieve automatic crash reproduction.
  53. 53. 53 Summary – Automatic Test Generation Categories Efficiency Coverage Complex Input Generation Feedback-directed Random Based Captured-Object Based Genetic Evolution Based Symbolic Execution Dynamic Symbolic Execution
  54. 54. 54 Summary – Automatic Test Generation Categories Efficiency Coverage Complex Input Generation Feedback-directed Random Based Captured-Object Based Genetic Evolution Based Symbolic Execution Dynamic Symbolic Execution Demand Driven Object Generation
  55. 55. 55 Summary – Crash Reproduction Categories Efficiency Coverage Complex Input Generation Feedback-directed Random Based Captured-Object Based Genetic Evolution Based Symbolic Execution Dynamic Symbolic Execution Categories Reproducibility Minimal No Performance Information Overhead Record-and-replay Static Analysis Approaches
  56. 56. 56 Summary – Crash Reproduction Categories Efficiency Coverage Complex Input Generation Feedback-directed Random Based Captured-Object Based Genetic Evolution Based Symbolic Execution Dynamic Symbolic Execution Categories Reproducibility Minimal No Performance Information Overhead Record-and-replay Static Analysis Approaches STAR

Editor's Notes

  • 1) external_method() is not analyzable 2) hash() contains complex mathematic computation
  • STAR consists of four main modules:Stack trace processing, Crash condition preparationSymbolic execution Test input generation
  • STAR consists of four main modules:Stack trace processing, Crash condition preparationSymbolic execution Test input generation

×