Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evolutionary Testing of Stateful Systems: a Holistic Approach


Published on

Slides used in my Ph.D. dissertation on test input generation for stateful systems

Published in: Technology
  • Be the first to comment

Evolutionary Testing of Stateful Systems: a Holistic Approach

  1. 1. Evolutionary Testing of Stateful Systems:a Holistic ApproachMatteo Miraz Advisor: prof. L. BaresiFebuary 18, 2011 Coadvisor: prof. P. L. Lanzi
  2. 2. Motivations 2• Software systems permeate (almost) every aspect of our life • Software is buggy • In 2002 the costs related to software errors are estimated in 60 Billion USD 1999: NASA Mars Climate Orbiter ($125 million) 2011: iPhone Alarm DEI
  3. 3. Motivations 3Testing is an effective technique to increase the quality of software  Does not guarantee that the system is error-free  Is extremely expensive, as it requires up to half of the entire development effortStringent requirements on time-to-marketlimit the testing effort• We spoke with an important software consultant company, with major Italian banks as customers… • The good: they save up to half of the entire development effort • The bad: they do not test anything • The ugly: we had to explain them what testing is DEI
  4. 4. Related Work 4• The research community proposed several ways to automatically generate tests • Symbolic Execution • Search-Based Software Testing DEI
  5. 5. Symbolic Execution 5• The program is executed with symbolic values as input parameters (instead of concrete inputs) PC: a = 3• Paths are associated with a PC: a = 3 & log10(b) = 7 Path Condition (PC) • A PC constraints symbolic inputs to traverse the path • It is created incrementally: at each condition a formula is PC: a = 3 & log10(b) = 7 added to the Path Condition Constraint solver• A constraint solver is used to find concrete inputs for PCs toTest(3, 10 000 000) DEI
  6. 6. Search Based Software Testing: a survey 6• Goal: complete branch coverage• Paradigm “command and conquer”  The program is analyzed to identify branches  Each branch is considered separately from the others entry w = Math.log(b) if (a == 3) F T if (w == 7) F T // target exit DEI
  7. 7. Search Based Software Testing: a survey 7Once a target has been selected• Identify dependent branches• Search for inputs able to reach the target• Fitness Function: Fitness(a = 0, b = 0) = • Approach Level + 1 + norm(| 0 – 3 |) = 1.003 Normalized distance entry w = Math.log(b) Parameters Fitness distance: | 0 – 3 | a=0, b=0 1.003 if (a == 3) Approach Level: 1 T a=1, b=0 1.002 F if (w == 7) a=3, b=0 0.999 Approach Level: 0 F T a=4, b=0 1.001 // target a=3, b=10 0.005 a=3, b=100 0.003 exit DEI
  8. 8. Problems of Search Based Software Testing 8Usage of a single guidance Works on isolated functions Coarse / Misleading Modern systems are object-oriented void foo(int []a) { int flag = 0; for(int i=0; i<10; i++) if(a[i] == 23) flag = 1; if(flag == 1) { // target } } DEI
  9. 9. Targeting Stateful Systems 9• We Target Stateful systems (e.g., Java classes)• Feature – State Loop • A feature might put objects in “particular ” states • A “particular” state might enables enable other features DEI
  10. 10. Our Approach: TestFul 10 An individual of the Evolutionary algorithm (a Test) is a sequence of constructor and method calls on the test cluster (i.e., the class under test + all the required dependencies) We use a multi-objective EA, guided by: • The coverage of tests • Tests with a high coverage stress more deeply the class under test and put objects in more interesting states • Compactness of tests • Unnecessary long tests waste computational resources during their evaluation DEI
  11. 11. Improvements 11How can we improve How can we correctlythe efficiency? reward tests?• We use efficiency • We use enhancement complementary techniques: coverage criteria: • Local Search • Control Flow Graph • Seeding • Data Flow Graph • Fitness Inheritance • Behavioral DEI
  12. 12. Local Search 12• We hybridize the evolutionary algorithm with a step of local search • EA works at class level • Reaches complex state configurations • Local search works on methods • It focuses on the easiest missing element • It adopts a simpler search strategy (hill climbing) • Which element to improve? • One of the best elements • 5% of the population • 10% of the population DEI
  13. 13. Local Search 13• At each generation of the Evolutionary Algorithm, we pick the { 5% / 10% / best } test and we target the easiest branch to reach among those not exercised yet target DEI
  14. 14. Local Search 14• At each generation of the Evolutionary Algorithm, we pick the { 5% / 10% / best } test and we target the easiest branch to reach among those not exercised yet• We perform a local search by using a simple algorithm (hill climbing) • The guidance is provided by the following fitness function:• The result (if any) is merged in the evolutionary algorithm’s population DEI
  15. 15. Contribution of the Local Search 15 Simple Problem (Fraction) Complex Problem (Disjoint Set Fast) DEI
  16. 16. Seeding 16• Problem: • Testful starts the evolution from a population with a poor quality.• Solution: • Run an inexpensive technique (random search) for a short time, and use its result as initial population DEI
  17. 17. Fitness Inheritance 17• Problem: • Executing tests and collecting coverage is expensive• Solution: • Evaluate the “real” fitness only on a part of the population • Other individuals inherit the fitness of their parents • Which policy to select individuals to evaluate? • Uniform selection • Frontier selection: better tests are evaluate more often DEI
  18. 18. Improvements 18How can we improve How can we correctlythe efficiency? reward tests?• We use efficiency • We use enhancement complementary techniques: coverage criteria: • Local Search • Control Flow Graph • Seeding • Data Flow Graph • Fitness Inheritance • Behavioral DEI
  19. 19. Coverage of the Control-Flow Graph 19Testful aims to maximize:  The number of basic blocks executed (~ statement coverage)  The number of branches exercisedWe compared our approach against  jAutoTest: a Java Port of Bertrand Meyer’s Random Testing Approach  Randoop: Michael Ernst’s taboo search  etoc: Paolo Tonella’s Evolutionary Testing of ClassesOn a benchmark of classes from 10 min• Literature• Known software libraries• Independent testing benchmarks DEI
  20. 20. Coverage of the Data-Flow Graph 20Fault-detection effectiveness of Statement and Branch coverage has been disputed• Criteria on the coverage of the Data-Flow Graph have been proposed • Fit well on Object-Oriented systems • Data dependency • Statements might define the value of variable (e.g., v = 10 ) • Statements might use the value of some variables (e.g. print(v) ) • P-Use if the use happens in a predicate (e.g. if(v == 3) )• TestFul leverages Data-Flow information  Extend the fitness function with all def-use and all def-puse coverage  Improve Local Search and solve data-dependent problems (e.g., flag variable)• Tracking data-flow coverage is expensive! () DEI
  21. 21. Coverage of the Data-Flow Graph 21Performed an extensive empirical evaluation between• Java Path Finder (symbolic execution)• Testful only guided by Control-Flow information The structural coverage remains high, in spite of the higher monitoring cost TestFul outperforms JPF (e.g., statement coverage: 89% vs. 35%) The data-flow coverage increases a little, and its standard deviation lower significantly DEI
  22. 22. Some insights from our empirical evaluation… 22 «container classes are the de facto benchmark for testing software with internal state» [ Arcuri 2010 ] DEI
  23. 23. Some insights from our empirical evaluation… 23S. Mouchawrab, L. C. Briand, Y. Labiche, and M. Di Penta.Assessing, Comparing, and Combining State Machine-Based Testing andStructural Testing: A Series of Experiments.IEEE Transactions on Software Engineering, 2010 DEI
  24. 24. Coverage of the Behavioral Model 24White-Box coverage criteria judge tests by considering the covered code • If we execute the code that contains an error, we’ll likely spot it! • What if there is a high-level problem? (e.g., a feature is missing)Black-Box testing derives tests from the specification of the system • Adopts an outer perspective, is more resilient to high-level errors • Requires the specification of the system, often not available • We can both infer the behavioral model of the system, and reward tests according to their ability to thoroughly exercise the system DEI
  25. 25. Behavioral Coverage: Preliminary Results 25 DEI
  26. 26. Conclusions and Future Work 26• Contributions: • Search-Based Software Testing • Holistic Approach • Efficiency Enhancement Techniques • Complementary Coverage Criteria • Extensive Empirical Validation• Most interesting research directions • Use more sources to seed the evolutionary algorithm • Consider coarse-grained tests (e.g. integration testing) • Consider other types of stateful systems (e.g., services) DEI
  27. 27. Thank you for the attention… Questions? DEI