Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Algorithms For
Optimizing Test Cases
Presented by team 4
Jim Kile
Don Little
Samir Shah
Software Testing – So What?
 July 28 1962 – Mariner I space probe
 Mission control destroys the rocket
 1985-1987 – The...
What Is Test Case Optimization?
 Typically applies to unit test cases where
coverage approaches 100%
 Implies ordering e...
What Are Goals For Test Case
Execution?
 Increase the rate of early fault detection
and correction
Find bugs early so th...
Why Is Test Case Optimization
Important?
 Execution complexity
Non-linear problem
 Elapsed time to execute
 Computing ...
Who Develops Test Cases?
 Developers
 Dedicated quality assurance staff
 Automated test generation techniques
What Are Benefits Of Automated
Test Generation Techniques?
 No cognitive bias
 Better able to:
Generate test cases that...
Why Automated Test Optimization?
 Generating set of basic test cases - easy
Easily cover 50–70% of faults
 Improving a ...
What Is Difference Between Optimization
And Test Case Generation?
 Optimization problem
Single goal is sought
 Test cas...
What Are Benefits Of GA Vs. Random
Test Generation?
 Random test generation
Uniform distribution
 GA-based search proce...
How Do Genetic Algorithms Work?
 A GA operates on strings of digits called
chromosomes
 Each digit that makes up the chr...
How Do Genetic Algorithms Work?
 Each chromosome has a fitness value
 Fitness value determines probability of
survival i...
How Do Genetic Algorithms Work?
 Algorithm begins with random population
 Algorithm evolves incrementally – generation
...
Uses of GA’s In Testing
1) Generate a range of effectivetest data
with fault revealing power
 Both papers used this techn...
How Is Quality Of Test Cases
Evaluated?
 Number of detected injected faults
Killed by the test case
Otherwise, it’s ali...
How Does GA Work Specifically?
Example
 Genetic algorithm
Generates random population of binary digits
For example each...
How Does GA Work Specifically?
Crossover
Before crossover between gene 5 and 32
1) 111001110101 100101100110 001010111000
...
How Does Mutation Work Specifically?
 Mutation will randomly switch genes in
population
 Gene 23 in chromosome 1 was swi...
How Does Mutation Work?
Before mutation of gene 23 chromosome 1
1) 111001110101 100101100110 001010111000
After mutation
1...
Generation Of Next Population
 Based on a “roulette wheel”
 Where fitness determines the probability
of selection
Those...
Dynamic Software Testing Techniques
 Structural – first paper
Code coverage
 Boundary conditions
 Individual or combin...
First paper
Automatic test case optimization:
A bacteriologic algorithm
Benoit Baudry, Franck Fleurey, Jean-marc Jézéquel ...
Contribution
 Finding an optimal set of test cases
through revealing a test case’s “fault
revealing power”
 Building con...
Bacteriologic Algorithm
Theoretical Basis
 Adapted from genetic algorithms
 Inspired by evolutionary ecology and
bacteri...
Bacteriologic Algorithm
Framework
Bacteriologic Algorithm
Basic Functions
 Initialization
 Iterate incrementally creating new
generation
 Limitation
Onl...
Bacteriologic Algorithm
Initialization
 Initial test cases either written by hand or
automatically generated
 For the ex...
Bacteriologic Algorithm
Computing Fitness
 Tool used to generate test case mutants
 Uses the mutation score of a set of ...
Bacteriologic Algorithm
Memorization
 Used to compute relative fitness
 Test case mutation score relative to the solutio...
Bacteriologic Algorithm
Mutation
 Randomly selects test cases
Selection is weighted by relative fitness of the
test case...
Bacteriologic Algorithm
Mutation – Abstract Syntax Tree
 A finite, labeled directed tree
Nodes are labeled by operators
...
Bacteriologic Algorithm
Mutation – Abstract Syntax Tree Example
x = a + b;
y = a * b;
while (y > a) {
a++;
x = a + b;
}
Bacteriologic Algorithm
Filtering
 Filtering = removing
 Two different implementations
Delete any test case whose relat...
Bacteriologic Algorithm
Results
Bacteriologic Algorithm
Results
 Comparison with genetic algorithm
Both ran 50 times
 Genetic algorithm results
200 ge...
Bacteriologic Algorithm
Results
 Bacteriologic algorithm results
30 generations created
Average mutation score of 96
R...
Second paper
Breeding software test cases
with genetic algorithms
D. Berndt, J. Fisher, L. Johnson, J. Pinglikar And
A. Wa...
Focus
 Breeding software test cases using genetic
algorithms as part of a software testing cycle
 Uses automated test ge...
Genetic Algorithm
Simple Triangle Classification Program (TRITYP)
 Classify triangle by type
 Three sides of the triangl...
Genetic Algorithm
Approach
 Flaws were intentionally introduced into
data for testing purposes
 Errors introduced for sp...
Genetic Algorithm
Search Space Illegal/Legal Triangle
Genetic Algorithm
How does it work specifically?
 Generates random population of x, y and z
coordinates as binary digits
...
Genetic Algorithm
Fitness
 Relative fitness function
 Compares
Particular chromosome’s fitness
Historical information ...
Genetic Algorithm
Generating Software Test Cases
 Variety of sources into test case breeding with
genetic algorithms
 Po...
Genetic Algorithm
Breeding Software Test Cases
 Using genetic algorithms
 Evolving fitness function
Fossil record of or...
Genetic Algorithm
Novelty
 Measure of the uniqueness of particular test
case
 Quantified by measuring distance in
parame...
Genetic Algorithm
Proximity
 Measure of closeness to other test cases that
resulted in system failures
( )∑ ∑ −
2
* ijijp...
Genetic Algorithm
Severity
 Measure of the seriousness of a system
error
Genetic Algorithm
Diversity
 Used to avoid being trapped by local
maxima
 Generation of test cases diversity means
Emph...
Genetic Algorithm
Explorer
 Highly novel test case
 Spread across the lightly populated
regions of the test space
 Once...
Genetic Algorithm
Prospectors
 Somewhat unique test cases
 Near newfound errors
 Both novelty and proximity
 Prospecto...
Genetic Algorithm
Miners
 More fully probe error areas
Genetic Algorithm
Explorer, Prospectors & Miners
Genetic Algorithm
Fossil Record
 Contains all previously generated test
cases
 Information about type of error generated...
Genetic Algorithm
Fitness Functions and Fossil Records
 Genetic algorithm is used to generate
good test cases
 Compared ...
Genetic Algorithm
Visualizing the fossil record
 Three dimensional representation
 Using 200-by-200 cells to divide up t...
Genetic Algorithm
High Novelty & High Proximity
 Fairly high novelty
Forces the search to reward exploratory
behavior
 ...
Genetic Algorithm
Fossil Record
Genetic Algorithm
Low Novelty & Low Proximity
 Weights are reversed
 Little reward for novelty or proximity
 More unifo...
Genetic Algorithm
Fossil Record
Genetic Algorithm
Results
 Describes preliminary results from genetic algorithm
based approach to software test case bree...
Conclusions
 Structural versus functional
 Structural - Code coverage – First Paper
 Boundary conditions
 Individual o...
Upcoming SlideShare
Loading in …5
×

Optimizing test cases

7,490 views

Published on

Optimizing Test Cases

Published in: Technology, Business
  • Get Now to Read PDF Format === http://pdfugadagpaug.vizvaz.com/B00BAHPWA0-notion-de-climat-affectant-les-douze-cases-le-choix-du-bon-theme-chinois-traite-d-astrologie-chinoise-le-grand-jeu-ou-comment-negocier-son-destin-t-3.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1lite.top/R5an3 ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Who Else Wants To Cure Their uterine fibroids, Regain Their Natural Inner Balance and Achieve LASTING Freedom From PCOS Related Symptoms? ♣♣♣ https://bit.ly/2ONPXxg
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Amanda Leto- certified nutritionist and former Uterine Fibroids sufferer teaches you her Uterine Fibroids freedom step by step success system jam-packed with a valuable information on how to naturally and permanently eliminate your Uterine Fibroids from the ROOT and achieve LASTING freedom from PCOS related symptoms. ➤➤ http://t.cn/Aig7bySW
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Optimizing test cases

  1. 1. Algorithms For Optimizing Test Cases Presented by team 4 Jim Kile Don Little Samir Shah
  2. 2. Software Testing – So What?  July 28 1962 – Mariner I space probe  Mission control destroys the rocket  1985-1987 – Therac-25 medical accelerator  At least five patients die  Others seriously injured  November 2000 – National Cancer Institute Panama City  At least eight patients die  Another 20 receive significant overdoses  Physicians indicted for murder
  3. 3. What Is Test Case Optimization?  Typically applies to unit test cases where coverage approaches 100%  Implies ordering execution such that: Rate of fault detection is increased Amount of time to perform regression is reduced  Elimination of unnecessary test cases during regression runs
  4. 4. What Are Goals For Test Case Execution?  Increase the rate of early fault detection and correction Find bugs early so they can be corrected early  Regression test only those areas that have changed  Reduce the amount of time to execute full unit regression test suites
  5. 5. Why Is Test Case Optimization Important?  Execution complexity Non-linear problem  Elapsed time to execute  Computing resources  Human resources Wait time for test completion Late defect detection and correction
  6. 6. Who Develops Test Cases?  Developers  Dedicated quality assurance staff  Automated test generation techniques
  7. 7. What Are Benefits Of Automated Test Generation Techniques?  No cognitive bias  Better able to: Generate test cases that concentrate on error prone areas Produce highly novel test cases  Better coverage overall
  8. 8. Why Automated Test Optimization?  Generating set of basic test cases - easy Easily cover 50–70% of faults  Improving a test set’s quality - hard Improving to 90–100%  Time consuming and expensive
  9. 9. What Is Difference Between Optimization And Test Case Generation?  Optimization problem Single goal is sought  Test case generation No single goal Optimized coverage of code under test
  10. 10. What Are Benefits Of GA Vs. Random Test Generation?  Random test generation Uniform distribution  GA-based search process More focused test set Set focuses on identified flaws Some highly novel test cases
  11. 11. How Do Genetic Algorithms Work?  A GA operates on strings of digits called chromosomes  Each digit that makes up the chromosome is called a gene  Collection of such chromosomes makes up a population
  12. 12. How Do Genetic Algorithms Work?  Each chromosome has a fitness value  Fitness value determines probability of survival in next generation
  13. 13. How Do Genetic Algorithms Work?  Algorithm begins with random population  Algorithm evolves incrementally – generation  Produces a structure from iterative development  Reproduction  Combine with another chromosome (crossover)  Adjusted slightly (mutation)  Original chromosome may have a poor/ low fitness  Create offspring with much higher fitness
  14. 14. Uses of GA’s In Testing 1) Generate a range of effectivetest data with fault revealing power  Both papers used this technique 1) Introduce faults in software under test to determine effectives of test cases  Only one paper used this technique
  15. 15. How Is Quality Of Test Cases Evaluated?  Number of detected injected faults Killed by the test case Otherwise, it’s alive  Develop a score Errors killed by test case Divided by test set
  16. 16. How Does GA Work Specifically? Example  Genetic algorithm Generates random population of binary digits For example each chromosome may be 36 bits long  Each twelve bit segment representing one of the sides of a triangle  Chromosomes cross between gene 5 and 32
  17. 17. How Does GA Work Specifically? Crossover Before crossover between gene 5 and 32 1) 111001110101 100101100110 001010111000 2) 111101011010 100101101010 101110110100 After crossover 1) 111001011010 100101101010 101110111000 2) 111101110101 100101100110 001010110100
  18. 18. How Does Mutation Work Specifically?  Mutation will randomly switch genes in population  Gene 23 in chromosome 1 was switched from a 1 to 0
  19. 19. How Does Mutation Work? Before mutation of gene 23 chromosome 1 1) 111001110101 100101100110 001010111000 After mutation 1) 111001110101 100101100100 001010111000
  20. 20. Generation Of Next Population  Based on a “roulette wheel”  Where fitness determines the probability of selection Those with higher fitness more chance of offspring in the next generation in comparison to their less fit companions
  21. 21. Dynamic Software Testing Techniques  Structural – first paper Code coverage  Boundary conditions  Individual or combined statement traversal  Path coverage  Functional – second paper Confirms that a function from specification is correctly implemented No analysis of the structure of the program
  22. 22. First paper Automatic test case optimization: A bacteriologic algorithm Benoit Baudry, Franck Fleurey, Jean-marc Jézéquel And Yves Le Traon
  23. 23. Contribution  Finding an optimal set of test cases through revealing a test case’s “fault revealing power”  Building confidence in the test suite through “mutation analysis”
  24. 24. Bacteriologic Algorithm Theoretical Basis  Adapted from genetic algorithms  Inspired by evolutionary ecology and bacteriologic adaptation  Similarities in this problem domain Can’t generate a single perfect test suite
  25. 25. Bacteriologic Algorithm Framework
  26. 26. Bacteriologic Algorithm Basic Functions  Initialization  Iterate incrementally creating new generation  Limitation Only works on test cases of similar size
  27. 27. Bacteriologic Algorithm Initialization  Initial test cases either written by hand or automatically generated  For the experiment test cases were randomly generated  Initial size set to 25 nodes
  28. 28. Bacteriologic Algorithm Computing Fitness  Tool used to generate test case mutants  Uses the mutation score of a set of test cases as that set’s fitness function MS(T) = 100(d/(m - equiv))  Test cases are executed to determine how many mutants they can kill Global mutation score computed
  29. 29. Bacteriologic Algorithm Memorization  Used to compute relative fitness  Test case mutation score relative to the solution set’s mutation score  Test cases are selected whose relative score exceeds the memorization threshold )(}){(),( TCSMStcTCSMStcTCSrelMS −∪=
  30. 30. Bacteriologic Algorithm Mutation  Randomly selects test cases Selection is weighted by relative fitness of the test case  Selected cases and code are mutated to create new cases for the next generation  Code is represented by an abstract syntax tree Nodes in the tree are replaced
  31. 31. Bacteriologic Algorithm Mutation – Abstract Syntax Tree  A finite, labeled directed tree Nodes are labeled by operators Edges represent operands Leaves contain variables or constants  Used in a parser  Range of all possible structures defined by the syntax
  32. 32. Bacteriologic Algorithm Mutation – Abstract Syntax Tree Example x = a + b; y = a * b; while (y > a) { a++; x = a + b; }
  33. 33. Bacteriologic Algorithm Filtering  Filtering = removing  Two different implementations Delete any test case whose relative mutation score is equal to 0  That is the function kills no mutant that the test cases in the solution set haven’t killed Reduce the coverage matrix by deleting redundant test cases
  34. 34. Bacteriologic Algorithm Results
  35. 35. Bacteriologic Algorithm Results  Comparison with genetic algorithm Both ran 50 times  Genetic algorithm results 200 generations created Average mutation score of 85 Required executing an average of 480,000 test cases
  36. 36. Bacteriologic Algorithm Results  Bacteriologic algorithm results 30 generations created Average mutation score of 96 Required executing an average of 46,375 test cases
  37. 37. Second paper Breeding software test cases with genetic algorithms D. Berndt, J. Fisher, L. Johnson, J. Pinglikar And A. Watkins
  38. 38. Focus  Breeding software test cases using genetic algorithms as part of a software testing cycle  Uses automated test generation techniques  Evolving fitness function  Relies on fossil record of organisms  Search behaviors  Novelty  Proximity  Severity
  39. 39. Genetic Algorithm Simple Triangle Classification Program (TRITYP)  Classify triangle by type  Three sides of the triangle Parameters x, y, and z Range 0 – 2000  Search space Illegal / legal triangle
  40. 40. Genetic Algorithm Approach  Flaws were intentionally introduced into data for testing purposes  Errors introduced for specific ranges of x and y parameters X coordinate between 500 - 1000 Y coordinate between 0 – 500 Result in error
  41. 41. Genetic Algorithm Search Space Illegal/Legal Triangle
  42. 42. Genetic Algorithm How does it work specifically?  Generates random population of x, y and z coordinates as binary digits  Each chromosome is 36 bits long Each twelve bit segment representing one of the sides of a triangle
  43. 43. Genetic Algorithm Fitness  Relative fitness function  Compares Particular chromosome’s fitness Historical information from the fossil record
  44. 44. Genetic Algorithm Generating Software Test Cases  Variety of sources into test case breeding with genetic algorithms  Powerful evolutionary  Naturally parallel computational engine  Balance fitness with diversity  Wide variety of test cases can be bred  Concepts of novelty, proximity and severity  Used to create a relative or changing fitness function
  45. 45. Genetic Algorithm Breeding Software Test Cases  Using genetic algorithms  Evolving fitness function Fossil record of organisms Interesting search behaviors  Novelty  Proximity  Severity
  46. 46. Genetic Algorithm Novelty  Measure of the uniqueness of particular test case  Quantified by measuring distance in parameter space from previous invocations stored in the fossil record ( )∑ ∑ − 2 * ijijn fck
  47. 47. Genetic Algorithm Proximity  Measure of closeness to other test cases that resulted in system failures ( )∑ ∑ − 2 * ijijp eck
  48. 48. Genetic Algorithm Severity  Measure of the seriousness of a system error
  49. 49. Genetic Algorithm Diversity  Used to avoid being trapped by local maxima  Generation of test cases diversity means Emphasizing novelty Downplaying proximity  Simple rules complex behavior Explorers Prospectors Miners
  50. 50. Genetic Algorithm Explorer  Highly novel test case  Spread across the lightly populated regions of the test space  Once an error is discovered - fitness function encourages more thorough testing of the region
  51. 51. Genetic Algorithm Prospectors  Somewhat unique test cases  Near newfound errors  Both novelty and proximity  Prospectors uncover additional errors  Fitness function will reward points that are simply near other errors
  52. 52. Genetic Algorithm Miners  More fully probe error areas
  53. 53. Genetic Algorithm Explorer, Prospectors & Miners
  54. 54. Genetic Algorithm Fossil Record  Contains all previously generated test cases  Information about type of error generated, if any  Provides an context for changing notions of fitness  Relative rather than absolute fitness function
  55. 55. Genetic Algorithm Fitness Functions and Fossil Records  Genetic algorithm is used to generate good test cases  Compared to fossil records  Points awarded for novelty & proximity  Causes population to evolve
  56. 56. Genetic Algorithm Visualizing the fossil record  Three dimensional representation  Using 200-by-200 cells to divide up the x-y parameter space  Bars represent the number of organisms
  57. 57. Genetic Algorithm High Novelty & High Proximity  Fairly high novelty Forces the search to reward exploratory behavior  High proximity Fosters the collection of points where errors are detected Three seeded range errors
  58. 58. Genetic Algorithm Fossil Record
  59. 59. Genetic Algorithm Low Novelty & Low Proximity  Weights are reversed  Little reward for novelty or proximity  More uniformly distribution
  60. 60. Genetic Algorithm Fossil Record
  61. 61. Genetic Algorithm Results  Describes preliminary results from genetic algorithm based approach to software test case breeding  Project’s reliance on relative or changing fitness function using fossil record  Fossil record records past organisms allowing any current fitness calculations to be influenced by past generations  Three factors are developed for the fitness function: novelty proximity and severity  Interplay of these factors produces fairly complex search behaviors
  62. 62. Conclusions  Structural versus functional  Structural - Code coverage – First Paper  Boundary conditions  Individual or combined statement traversal  Path coverage  Functional – specification – Second Paper  Confirms that a function from specification is correctly implemented  No analysis of the structure of the program  GA algorithm generation  Generate effective test data – Both Papers  Introduction of software faults – First Paper

×