Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzzles (ASE 2012)

1,747 views

Published on

Ning Chen's ASE 2012 Presentation

  • Be the first to comment

  • Be the first to like this

Puzzle-Based Automatic Testing: Bringing Humans Into the Loop by Solving Puzzles (ASE 2012)

  1. 1. 1Puzzle-Based Automatic Testing:Bringing Humans Into the Loop bySolving PuzzlesNing Chen and Sunghun KimThe Hong Kong University of Science and Technology ASE 2012, Essen
  2. 2. 2Motivation• Many automatic test generation techniques introduced: • Randoop (C.Pacheco, ICSE 2007) • Pex (N.Tillmann, TAP 2008) • OCAT (H.Jaygarl, ISSTA 2010)• However, their coverage results are still not satisfactory when applied to complex object-oriented programs.
  3. 3. 3Coverage by Randoop Subject Branches Coverage Commons Math 7707 61.6% Commons Collections 5242 53.0%
  4. 4. 4Coverage by Pex Subject LOC Coverage SvnBridge 17.1K 56.2% xUnit 11.4K 15.5% Math.Net 3.5K 62.8% QuickGraph 8.3K 53.2% Total 40.3K 49.8% - by Xiao et.al (ICSE 2011)
  5. 5. 5Major Challenges• The Constraint Solving challenge: Test generators fail to solve path conditions to cover certain branches.• The Object Creation/Mutation challenge: Test generators cannot create and/or modify test inputs to desired object states;
  6. 6. 6Challenge 1 : The Constraint Solving problemvoid foo(int x, int y, int n) { int value = x << n; if (value < y && n > 2) { // not covered branch … } }What’s a model (solution) for the path condition: (x << n) < y && n > 2
  7. 7. 7Challenge 2: The Object Creation/Mutation Challengevoid foo(Container container) { Given Model: if (container.size() >= 10) { container.size() = 10 // not covered branch … }How do we create and mutate a Container object into size() = 10?
  8. 8. 8Can we leverage the human to help solve these challenges?
  9. 9. 9PAT PuzzleOverviews
  10. 10. 10Object Mutation Puzzle
  11. 11. 11Object Mutation Puzzle
  12. 12. 12Object Mutation Puzzle
  13. 13. 13Object Mutation Puzzle
  14. 14. 14Constraint Solving Puzzle
  15. 15. 15Constraint Solving Puzzle
  16. 16. 16Constraint Solving Puzzle
  17. 17. 17Approach
  18. 18. 18Architectural Design of PAT
  19. 19. 19Up-front Testing Runs
  20. 20. 20Up-front Testing Runs• Coverage report by existing automatic techniques • Avoid wasting human effort • PAT targets only not covered branches• Retrieve dynamic information • Runtime object instances • Method call sequences
  21. 21. 21Path Computation
  22. 22. 22Path Computation• The final result of this phase: • The models which will be used to generate the object mutation puzzles. • The constraints not solvable by SMT solver, which will be used to generate the constraint solving puzzles.
  23. 23. 23Object Mutation Puzzle
  24. 24. 24Extracting Sub-models• A complete model can contain object states for different inputs, we want to divide them into sub-models. Complete Model: in == not null in.readInt() == 1 this.currentState == null Sub-model 1: Sub-model 2: in == not null this.currentState == null in.readInt() == 1
  25. 25. 25Prioritizing Sub-models• One sub-model may appear in many models. Model 1: Model 2: in == not null in == not null in.readInt() ==1 in.readInt() == 0 this.currentState == null this.currentState == null• Sub-models are prioritized so that high frequent sub- models are ranked higher.
  26. 26. 26Generating Puzzles from Sub-models
  27. 27. 27Constraint Solving Puzzle
  28. 28. 28Extracting Error Related Constraints• From up-front testing runs, we can obtain many branches whose path conditions are not solvable by the SMT solver. Path Conditions: 1. this == not null 2. this.sums == not null 3. this.sums.length * this.sums.length <= 4096 4. this.sums.length > 0 5. this.n > 1 Error: feature not support: non-linear problem .
  29. 29. 29Extracting Error Related Constraints• From up-front testing runs, we can obtain many branches whose path conditions are not solvable by the SMT solver. Path Conditions: 1. this == not null 2. this.sums == not null 3. this.sums.length * this.sums.length <= 4096 4. this.sums.length > 0 5. this.n > 1 Error: feature not support: non-linear problem . Error related constraints: 1. this.sums.length * this.sums.length <= 4096 2. this.sums.length > 0
  30. 30. 30 30Grouping Constraint Sets• Error related constraint sets could be identical except only the variable names: Group 1 Group 2 Group 3 Constraint Set Constraint Constraint Set Sets
  31. 31. 31 31Grouping Constraint Sets• Error related constraint sets could be identical except only the variable names: Group 1 Group 2 Group 3 Constraint Set Constraint Constraint Set Sets Puzzle 3 Puzzle 1 Puzzle 2
  32. 32. 32Generating and Presenting Puzzles
  33. 33. 33Approach Overview
  34. 34. 34Test Case Generation from Solutions• Solutions from Constraint Solving Puzzles: • More models for not covered branches• Solutions from Object Mutation Puzzles: • Method call sequences to achieve the goal state • They can be directly translated to source code for generating one test input. • A test case can be generated when all test inputs are generated.
  35. 35. 35Evaluation
  36. 36. 36Evaluation Setup• Subjects: Subject Version Branches Commons Math 2.1 7707 Commons Collections 3.2.1 5242• Up-front testing runs: • Baseline techniques: Randoop + Symbolic execution module • Coverage: Subject Randoop Symbolic Commons Math 61.6% 64.4% Commons Collections 53.0% 56.7%
  37. 37. 37Research Question 1• How many PAT puzzles are solvable by humans?• Participants: • Eight computer science major graduate students.• Subject: Subject Version Branches Commons Math 2.1 7707
  38. 38. 38Research Question 1• Presenting Puzzles: • The same top 100 object mutation puzzles. • The same top 100 constraint solving puzzles. • Repeated solutions are counted only once.• Result: Puzzle Total Presented Solved Avg. Time Mutation 100 51 1 minute Constraint 100 72 1 minute
  39. 39. 39Research Question 2• How many people would play PAT voluntarily?• Participants: • We posted the links to PAT puzzles on Twitter and encourage people to participate.• Subject: Subject Version Branches Commons Collections 3.2.1 5242
  40. 40. 40Research Question 2• Puzzles: • The same top 100 object mutation puzzles. • The same top 100 constraint solving puzzles. • Repeated solutions are counted only once.• Result • In total, 120 people volunteered to play the puzzles Puzzle Total Presented Solved Avg. Time Mutation 100 24 1 minute Constraint 100 84 1 minute
  41. 41. 41Research Question 3• How much is the test coverage improved by the puzzle solutions of PAT?• We executed test cases generated from puzzles solutions from RQ1 and RQ2.• We measure the # of additional branches coverable by PAT over the baseline techniques (Randoop + Symbolic)
  42. 42. 42Research Question 3 75 70Branch Coverage (%) +534 (7.0%) 65 +217 (+2.8%) 60 +308 (5.8%) PAT 55 +190 (3.7%) Symbolic 50 4750 (61.6%) Randoop 2780 (53.0%) 45 40 ACM ACC Subjects
  43. 43. 43Research Question 4• How much manual test case writing effort can be saved with the help of PAT?
  44. 44. 44Research Question 4• How much manual test case writing effort can be saved with the help of PAT?
  45. 45. 45Conclusion• A novel framework to support software testing through puzzle solving by Internet users.• Two prototype puzzles have been introduced: • Constraint solving puzzles • Object mutation puzzles • More kinds of puzzles could be developed in the future• Evaluations show that PAT puzzles are solvable and can help improve non-trivial branch coverage on complex OO programs.

×