Nighthawk: A Two-Level Genetic-Random Unit Test Data Generator

3,987 views

Published on

A talk to ASE 2007 by Jamie Andrews, Felix C. H. Li Department of Computer Science University of Western Ontario; and Tim Menzies Lane Department of Computer Science West Virginia University

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,987
On SlideShare
0
From Embeds
0
Number of Embeds
961
Actions
Shares
0
Downloads
65
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Nighthawk: A Two-Level Genetic-Random Unit Test Data Generator

  1. 1. Nighthawk: A Two-Level Genetic-Random Unit Test Data Generator Jamie Andrews and Felix C. H. Li Department of Computer Science University of Western Ontario Tim Menzies Lane Department of Computer Science West Virginia University
  2. 2. Plan of Talk <ul><li>Randomized Unit Testing </li></ul><ul><li>Genetic Algorithms (GA) </li></ul><ul><li>Nighthawk: randomized testing level </li></ul><ul><li>Nighthawk: GA level </li></ul><ul><li>Empirical studies </li></ul>
  3. 3. Randomized Testing <ul><li>Generate inputs using randomization </li></ul><ul><li>Challenges: </li></ul><ul><ul><li>Thoroughness? </li></ul></ul><ul><ul><li>Test oracle? </li></ul></ul>
  4. 4. Effectiveness of Randomized Testing <ul><li>Analysis: </li></ul><ul><ul><li>Duran & Ntafos, Hamlet & Taylor, Jeng & Weyuker, ... </li></ul></ul><ul><li>Empirical: </li></ul><ul><ul><li>Miller (Fuzz, 1990s), Claessen & Hughes (QuickCheck, 2000), Andrews (ASE 2004), Pacheco et al., Groce et al. (ICSE 2007) </li></ul></ul><ul><li>Why? </li></ul><ul><ul><li>Generate many different test inputs cheaply </li></ul></ul><ul><ul><li>Effective and thorough if set up right </li></ul></ul>
  5. 5. Unit Testing <ul><li>Test case = seq of method calls </li></ul><ul><li>Each call possibly: </li></ul><ul><ul><li>Preceded by argument setup </li></ul></ul><ul><ul><li>Followed by result evaluation </li></ul></ul><ul><li>TreeMap t = new TreeMap(); </li></ul><ul><li>Employee e = new Employee(&quot;W&quot;); </li></ul><ul><li>t.put(e, 43); </li></ul><ul><li>t.remove(e); </li></ul><ul><li>assert t.size() == 0; </li></ul>How many TreeMaps to store? When to reuse TreeMaps?
  6. 6. Randomized Unit Testing <ul><li>Randomization of: </li></ul><ul><ul><li>Methods called </li></ul></ul><ul><ul><li>Arguments selected </li></ul></ul><ul><li>Challenges: </li></ul><ul><ul><li>Test oracle – JML, Java assertions, ... </li></ul></ul><ul><ul><li>Thoroughness? </li></ul></ul>
  7. 7. Example: TreeMap <ul><li>Create pool of n Employees </li></ul><ul><li>Randomly put(), remove() Employees </li></ul><ul><li>n =10000: remove() usually fails at first </li></ul><ul><ul><li>Doesn't cover emptying out tree </li></ul></ul><ul><li>n =2: tree doesn't get big </li></ul><ul><ul><li>Doesn't cover many branches </li></ul></ul><ul><li>n =30: &quot;just right&quot; </li></ul>
  8. 8. Genetic Algorithms (GA) <ul><li>Chromosomes encode solutions </li></ul>
  9. 9. Genetic Algorithms (GA) <ul><li>Mutation </li></ul>
  10. 10. Genetic Algorithms (GA) <ul><li>Recombination </li></ul>
  11. 11. Genetic Algorithms (GA) <ul><li>Fitness function chooses &quot;survivors&quot; </li></ul>
  12. 12. GAs and Testing <ul><li>Evolving individual test cases: </li></ul><ul><ul><li>Guo et al., FATES 2003 </li></ul></ul><ul><ul><li>Tonella, ISSTA 2004 </li></ul></ul>
  13. 13. Nighthawk: Randomized Testing Level <ul><li>Input: set M of target methods; chromosome c </li></ul><ul><li>Output: fitness of chromosome </li></ul><ul><li>Algorithm overview: select and run one randomized unit test case, measure coverage </li></ul><ul><li>Random choices partly controlled by chromosome c </li></ul>
  14. 14. Randomized Testing Level: Details <ul><li>Populate &quot;value pools&quot; for every relevant class </li></ul><ul><li>Repeat L times: </li></ul><ul><ul><li>Choose target method &quot;randomly&quot; </li></ul></ul><ul><ul><li>Choose receiver, params &quot;randomly&quot; from value pools </li></ul></ul><ul><ul><li>Call method </li></ul></ul><ul><ul><li>Place return value in value pool </li></ul></ul>
  15. 15. Value Pools and Methods TreeMap int Employee . . . . . . . . . . . . . . . . . . . . . . ... t.put(e, i); &quot;value reuse policy&quot;
  16. 16. GA Level: Chromosomes <ul><li>Chromosome = set of genes </li></ul><ul><li>Each gene controls aspect of randomized testing algorithm </li></ul>
  17. 17. Genes <ul><li>Genes answer questions like: </li></ul><ul><ul><li>How long is the test case? </li></ul></ul><ul><ul><li>How often do we choose method m? </li></ul></ul><ul><ul><li>How many value pools? </li></ul></ul><ul><ul><li>How do we construct int value pools? </li></ul></ul><ul><ul><li>What is the value reuse policy? </li></ul></ul><ul><ul><ul><li>Where do I get this parameter from? </li></ul></ul></ul><ul><ul><ul><li>Where do I put the result value? </li></ul></ul></ul>
  18. 18. Initialization
  19. 19. Cloning
  20. 20. Mutation, Recombination
  21. 21. Fitness Evaluation 4324 4300 4288 3696 3559 3331 3278 3277 3000
  22. 22. Sorting 4300 4288 3696 3559 3278 3000 4324 3331 3277
  23. 23. Retention 4300 4288 4324
  24. 24. Fitness Function (number of lines covered) * 1000 - (number of method calls) brake on test case length reward for high coverage
  25. 25. Empirical Evaluation <ul><li>Comparison to previous studies </li></ul><ul><li>Case study: Collection and Map classes </li></ul><ul><li>Comparison of option settings </li></ul>
  26. 26. Comparison to Previous Studies <ul><li>Compared to: </li></ul><ul><ul><li>Michael et al. TSE 2001: straight GA </li></ul></ul><ul><ul><li>Visser et al. ISSTA 2006: JPF with state matching </li></ul></ul><ul><ul><li>Pacheco et al. ICSE 2007: extending test sequences randomly </li></ul></ul><ul><li>Achieved same coverage when run with same constraints </li></ul><ul><li>Achieved more coverage when run with no constraints </li></ul>
  27. 27. Case Study: java.util <ul><li>Applied Nighthawk to all 16 Collection and Map classes from java.util 1.5.0 </li></ul><ul><li>Measured line coverage, clock time </li></ul><ul><li>Compared option settings </li></ul>
  28. 28. Results – Coverage (Lines) Enriched test wrappers Deep target analysis Both ... 325 (.92) 252 253 205 355 Hashtable 44 (.96) 26 40 24 46 HashSet 347 (.96) 305 265 238 360 HashMap 7 (.03) 10 9 7 239 EnumMap 140 (.93) 109 140 111 150 ArrayList ED PD EN PN SLOC Source
  29. 29. Results – Time (Clock Sec.) ... 157 110 110 8 Hashtable 39 27 29 25 HashSet 176 136 37 63 HashMap 5 6 9 3 EnumMap 48 29 91 75 ArrayList ED PD EN PN Source
  30. 30. Results - Analysis <ul><li>Enriched wrappers, Deep analysis better </li></ul><ul><ul><li>Overall coverage 82% of lines </li></ul></ul><ul><li>Deep analysis also took longer </li></ul><ul><ul><li>Still less than 100 sec/class avg </li></ul></ul><ul><li>EnumMap (3%): constructor expects enumerated type </li></ul><ul><ul><li>Customized wrapper: </li></ul></ul><ul><ul><ul><li>85% coverage of EnumMap </li></ul></ul></ul><ul><ul><ul><li>Raises overall coverage to 88% </li></ul></ul></ul>
  31. 31. Conclusions <ul><li>Metaheuristic search can find effective parameters for randomized testing </li></ul><ul><li>Only needs info about methods and parameter types </li></ul><ul><li>Efficiency acceptable </li></ul>
  32. 32. Future Work <ul><li>Which metaheuristic search approach? </li></ul><ul><ul><li>Genetic algorithms </li></ul></ul><ul><ul><li>Simulated annealing ... </li></ul></ul><ul><li>Which genes are really necessary? </li></ul><ul><li>Which coverage criterion? </li></ul>
  33. 33. Thank you!
  34. 34. Oracle: Test Wrapper Class <ul><li>Test wrapper class: methods that call methods of target class </li></ul><ul><li>public class TreeMapTestWrapper { </li></ul><ul><li>private TreeMap target; ... </li></ul><ul><li>void put(Object key, Object value) { </li></ul><ul><li>// insert preconditions </li></ul><ul><li>target.put(key, value); </li></ul><ul><li>// insert oracle assertions </li></ul><ul><li>} } </li></ul>
  35. 35. Enriched (E) Test Wrappers <ul><li>Add methods to test: </li></ul><ul><ul><li>Serialization </li></ul></ul><ul><ul><li>Typed equals() </li></ul></ul><ul><li>Typically cover more code </li></ul>
  36. 36. Normal (N) Target Analysis <ul><li>Red = call all methods </li></ul><ul><li>Green = call only constructors </li></ul>B: Classes of params of A methods A: Classes named by user All classes
  37. 37. Deep (D) Target Analysis <ul><li>Red = call all methods </li></ul><ul><li>Green = call only constructors </li></ul>B: Classes of params of A methods C: Classes of params of B methods A: Classes named by user All classes

×