A Survey on Automatic Test Generation and Crash Reproduction

1,507 views

Published on

Ning Chen's Survey for his PhD Qualifying Examination.

Published in: Technology

A Survey on Automatic Test Generation and Crash Reproduction

  1. 1. 1A SURVEY ONAUTOMATIC TEST GENERATIONAND CRASH REPRODUCTIONPhD Qualifying ExaminationNing Chen
  2. 2. 2Outline• Automatic Test Generation • Feedback-directed Random Generation • Captured-Object based Generation • Genetic Evolution based Generation • Symbolic Execution • Dynamic Symbolic Execution• Automatic Crash Reproduction • Record-and-replay Approaches • Static analysis based Approaches• Summary
  3. 3. 3Automatic Test Generation
  4. 4. 4Automatic Test Generation Automatic Test Generation Symbolic Search Execution Based BasedFeedback- Object- Genetic Symbolic Dynamic Symbolic directed capturing Evolution Execution Execution Common Challenge: My Approach Object Creation Challenge
  5. 5. 5Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  6. 6. 6 Feedback-directed Random Test Generation Randomly select a new method callClasses to test Error- Generate Execute Examine revealing test Method Method Execution cases Sequences Sequences ResultTime Limit Regression test casesProperties Method to check Sequences Method Arguments If Sequence Legal Generated
  7. 7. 7 Feedback-directed Random Test Generation Randomly select a newClasses to method call test Error- Generate Execute revealing test Method Method cases Sequences SequencesTime Limit Regression test casesProperties Method to check Sequences Feedback to guide If Sequence Legal Generated
  8. 8. 8Feedback-directed Random Test Generation• Advantages: • Scalable to large subject programs • Known to be easy to implement • Generates no or little redundant method call sequences• Challenges: • Generates too many test cases (thousands of test cases) • Cannot generate complex method sequences, thus may not achieve good coverage for programs with complex structures
  9. 9. 9Feedback-directed Random Test Generation01 void execute(tinySQLStatement stmt, String sqlStr) {02 if (stmt.execute(sqlStr)) {03 // target branch04 }05 }• cannot cover the target branch because it is difficult to generate the proper sql statement randomly• In short, Random approach fails to create test inputs that can reach deeper part of the subject program.
  10. 10. 10Follow-up worksCombining Dynamic Exec. with Random Generation:S. Artzi, M. D. Ernst, A. Kie˙zun, C. Pacheco, and J. H. Perkins. Finding theneedles in the haystack: Generating legal test inputs for object-orientedprograms. In 1st Workshop on Model-Based Testing and Object-Oriented,2006. (Palulu)Combining Static Exec. with Random Generation:W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with mut-aware sequence recommendation. In ASE, 2010. (RecGen)Combining Dynamic + Static Exec. with Random Generation:S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, Combined static and dynamicautomated test generation, in ISSTA, 2011 (Palus)
  11. 11. 11Follow-up works• The common goal: more diverse and complex test inputs that can reach deeper part of the subject program.• However, there is an alternative…
  12. 12. 12Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT 10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  13. 13. 13 Object Captured Based Randomly select a new method callClasses to test Generate Execute Error- Method Method revealing test Sequences Sequences casesTime Limit Arguments Regression Method test cases SequencesProperties If Sequence Legal Generated to check Runtime Existing Test Captured Cases / Normal Objects Executions
  14. 14. 14Object Captured Based• Advantages: • Can generate test inputs that are complex for random approaches • Can be integrated with other test generation approaches. • E.g. Seed objects for Random generation.• Challenges: • Fail to demonstrate an object generating method sequence. • Making a test case less useful. • Rely heavily on the quality and quantity of existing test cases. • Captured objects may contain sensitive private information. • James Clause et. al - ICSE2011
  15. 15. 15Object Captured Based01 void test_execute() {02 SQLStatement stmt = load(“…/SQLStatement/001”);03 String sqlStr = load(“…/String/123”);02 if (stmt.execute(sqlStr)) {03 // target branch04 }05 }• Apparently, such a test case would be less useful as it does not provide much information.
  16. 16. 16Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  17. 17. 17Genetic Evolution Based• Adopts evolutionary computation techniques to synthesize method sequences for reaching higher code coverage.• Method sequences (chromosome) are crossed-over and mutated to produce different varieties of offspring.• The fitness score is computed according to the code coverage. • Method sequences that can achieve higher code coverage will be selected for future evolution.
  18. 18. 18Genetic Evolution Based• Advantages: • Have stronger capabilities of generating complex test inputs over random approaches.• Challenges: • Might not scale to large subject programs. • Factors such as fitness functions, mutation operators, evolution algorithms can greatly affect the quality of the generations.
  19. 19. 19Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Search Based Test Generation Approaches
  20. 20. 20Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08Symbolic Execution Based Test Generation Approaches
  21. 21. 21Symbolic Execution• Proposed as early as the 70s but become realistic only in recent years• Generate tests by symbolically executing all paths in the program• Rely on constraint solvers to compute test inputs (i.e. models)
  22. 22. 22Symbolic Execution01 void foo(int x, int y) {02 if (x > y + 1) {03 // branch 104 }05 else {06 // branch 207 }08 }
  23. 23. 23Symbolic Execution01 void foo(int x, int y) { Symbolic Input: X, Y02 if (x > y + 1) { Path Condition: X > Y + 103 // branch 104 }05 else { Constraint Solver -> X = 2, Y = 006 // branch 207 } Generate actual test case08 }
  24. 24. 24Symbolic Execution01 void foo(int x) { Symbolic Input: X, Y02 if (x > y + 1) { Path Condition: X <= Y + 103 // branch 104 }05 else { Constraint Solver -> X = 0, Y = 006 // branch 207 } Generate actual test case08 }
  25. 25. 25Symbolic Execution• Advantages: • Feasible in recent years with the increase of computation power. • Capable of achieving higher structural coverage than search based techniques.
  26. 26. 26Symbolic Execution• Challenges: • Constraint Solving Limitations: • Non-linear • Floating point, • Bit-vector, etc. • External Method Calls: • External Libraries, System Level Calls, etc. • Cannot Symbolically execution these external method calls. • The Object Creation Problem: • In OO, test inputs (objects) needs to be constructed in sequence of method calls.
  27. 27. 27Follow-up workComplex Constraint Solving Challenge:C. S. P˘as˘areanu, N. Rungta, and W. Visser. Symbolic execution with mixedconcrete-symbolic solving. In ISSTA ‟11I. Erete and A. Orso. Optimizing constraint solving to better support symbolicexecution. In CSTVA‟11The Object Creation Problem:S. Thummalapenta, T. Xie, N. Tillmann, P. de Halleux, and W. Schulte.MSeqGen: Object-oriented unit-test generation via mining source code. InESEC/FSE‟09
  28. 28. 28Automatic Test Generation Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
  29. 29. 29Dynamic Symbolic Execution• Symbolic execution Ideally can achieve complete coverage, but usually not the case in reality• Recent years, Dynamic Symbolic Execution is proposed. • EXE/KLEE „05 • DART „05 • CUTE „05, JCUTE „06 • PEX „08• DSE is designed to handle two major challenges in Symbolic Execution: • 1) Constraint Solving, and • 2) External Method Calls
  30. 30. 30Dynamic Symbolic Execution01 void foo(int x, int y) {02 if (external (x) == y) { No source access for external()03 // branch 104 }05 else if (hash(x) > y) { hash() is complex arithmetic06 // branch 207 }08 } Key Idea: Combines Dynamic execution + Symbolic execution
  31. 31. 31Dynamic Symbolic Execution01 void foo(int x, int y) { Dynamic Exec.02 if (external(x) == y) { external(0) == 103 // branch 104 }05 else if (hash(x) < y) { hash(0) == 1006 // branch 207 }08 } First Iteration (Random Input): Dynamic Exec: x = 0 and y = 0
  32. 32. 32Dynamic Symbolic Execution01 void foo(int x, int y) { Symbolic Exec.02 if (external(x) == y) { external(X) == Y03 // branch 104 } external(0) == 105 else if (hash(x) < y) {06 // branch 2 Constraint Solver:07 } X == 0, Y == 108 } Second Iteration: Path Condition: external(X) == Y
  33. 33. 33Dynamic Symbolic Execution01 void foo(int x, int y) { Symbolic Exec.02 if (external(x) == y) { external(X) != Y03 // branch 104 } hash(X) < Y05 else if (hash(x) < y) {06 // branch 2 external(0) == 1 hash(0) == 1007 }08 } Constraint Solver: X == 0, Y == 11 Third Iteration: Path Condition: external(x) == y && hash(x) < y
  34. 34. 34Dynamic Symbolic Execution• Advantage: • Can handle situations where traditional symbolic executions cannot handle. i.e. Constraint Solving, External Method Calls • Achieve higher coverage compared to only symbolic execution.
  35. 35. 35Dynamic Symbolic Execution• Challenges: • A recent paper by Xiao ‟11. investigates the major challenges faced by a state-of-the-art DSE framework for C#, Pex. Project LOC Cov % Object Creation Problem SvnBridge 17.1K 56.26 11 (42.31%) xUnit 11.4K 15.54 8 (72.73%) Math.Net 3.5K 62.84 17 (70.83%) QuickGraph 8.3K 53.21 10 (100%) Total 40.3K 49.87 46 (64.79%) • The object creation challenge accounts for 65% of all cases that cause code blocks to be not covered by Pex.
  36. 36. 36The Object Creation Problemvoid foo(Container container) { if (container.size >= 10) { Model: // not covered branch container.size = 10 … }The Object Creation Problem :How do we create and mutate a Container object into size = 10?
  37. 37. 37Dynamic Symbolic Execution• In object-oriented programs, modifying a non-public field requires a set of legal method sequence • e.g. Container.size == 10 • Possible method sequence: Container.add(Object); … Container.add(Object);• But as the structure of the classes become more complex, so will the object creation problem.• Therefore, the object creation problem is an important, yet unsolved problem in automatic test generation.
  38. 38. 38Demand-driven Object Generation• To address this OCP, I am working on a project for Demand-Driven Object Generation.• Goal: Given a target state -> generate a legal method sequence that can create and mutate object that satisfy this target state • Target state: container.size == 10 • Possible Output: container.add(new Object()); ten times
  39. 39. 39Demand-driven Object Generation• The approach should be demand-driven, so only legal sequence satisfying the target state can be generated.• It is helpful to solve the object creation problem mentioned.• Thus, it can improve the coverage for dynamic symbolic execution approaches for object-oriented programs.
  40. 40. 40Automatic Crash Reproduction
  41. 41. 41Automatic Crash Reproduction Automatic Crash Reproduction Static Record- Analysis and-replay Based Common Challenges My Approach
  42. 42. 42Automatic Crash Reproduction Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Categories RepresentativesRecord-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09
  43. 43. 43Record-and-Replay• Approach: • Monitoring Phase: Captures/Stores runtime heap & stack objects. • Test Generation Phase: Generates tests that loads the correct objects with the crashed methods. Original Program Execution Store from heap & stack Stored Objects Load as crashed method params Recreated Test Case
  44. 44. 44Record-and-Replay• Advantage: • Can usually reproduce target crashes reliability• Challenges: • Require instrumentation in advance • Extra runtime overhead, i.e. ReCrash: • 13-64% performance overhead, and • 3-90% memory overhead • Extensive amount of data from users: objects in heap, running context, etc • Difficult to obtain ( > 100M ) • Privacy issues: James Clause et. al @ ICSE2011
  45. 45. 45Record-and-Replay• Existing Frameworks: • Jrapture ‟00, • BugNet ’05, • ReCrash/ReCrashJ ‟08 • LEAP/LEAN ’10• Existing approaches have been trying various techniques to reduce the runtime overhead yet maintain a good reproduction result.
  46. 46. 46Automatic Crash Reproduction Categories RepresentativesFeedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11Captured-Object Based MSeqGen’09, OCAT’10Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09Symbolic Execution King’76, Clarke’76, JPF’04Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08 Categories RepresentativesRecord-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09
  47. 47. 47Static Analysis Approaches• Microsoft PSE ‘04 • Postmortem static analysis framework. • Explains the potential cause of a crash through symbolic execution.• XyLem ’09 • Null-dereference analysis tool for Java language. • Identifies the precise paths along which null values may flow to de- reference.• IBM SnuggleBug ‘09 • Demand-driven backward symbolic execution framework. • Computes the weakest preconditions of crashes. • But does not generate actual crash triggering objects from the computed preconditions.
  48. 48. 48STAR: automatically reproduces failures• All of the above approaches either only finds error path, or computes the error triggering preconditions.• They do not support the full reproduction of a crash from the crash information.• We propose STAR, an automatic crash reproduction framework • Requires only minimal information (crash stack trace, crash type) • No client site deployment and recording necessary • No performance overhead • No need to collect any private data from user. • Supports both finding out the failure preconditions as well as recreating the actual failures.
  49. 49. 49 STAR: automatically reproduces failures crash precondition stack trace processing crash condition preparation computation 1 2 3bug report x() at nnn {var == null} y() at yyy {true} z() at zzz … crash 4 test case generation reproductionprogram test TestGen: A novel test generation technique cases
  50. 50. 50Backward Symbolic Execution• Goal: Given a crash location, compute at a public method entry, the precondition that can trigger the crash• There have been several previous work in this direction • XYLem ICSE ‟09 • SnuggleBug PLDI ‟09• STAR has a more efficient backward symbolic execution engine that improves existing approaches with heuristics: • Pruning irrelevant branches • Smarter backtracking • Early detection of inner contradictions
  51. 51. 51 STAR: automatically reproduces failures crash precondition stack trace processing crash condition preparation computation 1 2 3bug report x() at nnn {var == null} y() at yyy {true} z() at zzz … crash 4 test case generation reproductionprogram test TestGen: A novel test generation technique cases
  52. 52. 52Crash Test Case Generation• Essentially, this is the same problem as the demand- driven method sequence generation we‟ve discussed.• Thus, we combine our efficient backward symbolic execution approach with the demand-driven method sequence generation to achieve automatic crash reproduction.
  53. 53. 53Summary – Automatic Test Generation Categories Efficiency Coverage Complex Input GenerationFeedback-directed Random BasedCaptured-Object BasedGenetic Evolution BasedSymbolic ExecutionDynamic Symbolic Execution
  54. 54. 54Summary – Automatic Test Generation Categories Efficiency Coverage Complex Input GenerationFeedback-directed Random BasedCaptured-Object BasedGenetic Evolution BasedSymbolic ExecutionDynamic Symbolic Execution Demand Driven Object Generation
  55. 55. 55Summary – Crash Reproduction Categories Efficiency Coverage Complex Input GenerationFeedback-directed Random BasedCaptured-Object BasedGenetic Evolution BasedSymbolic ExecutionDynamic Symbolic Execution Categories Reproducibility Minimal No Performance Information OverheadRecord-and-replayStatic Analysis Approaches
  56. 56. 56Summary – Crash Reproduction Categories Efficiency Coverage Complex Input GenerationFeedback-directed Random BasedCaptured-Object BasedGenetic Evolution BasedSymbolic ExecutionDynamic Symbolic Execution Categories Reproducibility Minimal No Performance Information OverheadRecord-and-replayStatic Analysis ApproachesSTAR

×