Software Testing and Reliability Test Assessment and Enhancement Aditya P. Mathur Purdue University August 12-16 @ Guidant Corporation Minneapolis/St Paul, MN Last update: August 13, 2002 Graduate Assistants : Ramkumar Natarajan Baskar Sridharan
It is required to write a program that takes in the name of a person as a string and searches for the name in a file of names. The program must output the record ID which matches the given name. In case of no match a -1 is returned.
What coverage domains can be identified from this requirement?
The rate at which new faults are discovered reduces as test adequacy with respect to a finite coverage domain increases ; it reduces to zero when the coverage domain has been exhausted.
coverage 0 1
Saturation Effect: Fault View Testing Effort Remaining Faults 0 N Functional t f s t f e t d s M t df e t m e
Saturation Effect: Reliability View Functional, Decision, Dataflow, and Mutation tsting provide various test assessment criteria. True reliability (R) Estimated reliability (R’) Saturation region Reliability Testing Effort R’ f R’ d R’ df R’ m Functional R f t f s t f e Decision R d t d s t d e Dataflow R df t df s t df e Mutation R m t m s t f e
One can develop a test strategy based on any of the criteria discussed.
A test strategy based on the statement coverage criterion will begin by evaluating a test set T against this criterion. Then new tests will be added to T until all the statements are covered, i.e. T satisfies the criterion.
c-use stands for computational use and p-use for predicate-use .
Both c- and p-uses affect the flow of control: p-uses directly as their values are used in evaluating conditions and c-uses indirectly as their values are used to compute other variables which in turn affect the outcome of condition evaluation.
For the above graph generate a test set that satisfies
the branch coverage criterion
the all-defs criterion - for definitions of all variables at least one use (c- or p- use) must be exercised .
the all-uses criterion- all p-uses and all c-uses of all variable definitions be covered .
Develop the tests incrementally, i.e. by modifying the previous test set!
SUDS processing: Phase I P, Program under test Preprocess, compile and instrument . trace file upon execution . atac files generate Instrumented version of P (executable) generate Test set input Program output upon execution
ATAC processing: phase II coverage analyzer .atac files .trace file control flow and data flow coverage values
Given a program P, a mutant of P is obtained by making a simple change in P.
What is zpush ? 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; Program 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/ zpush(x); 7. else 8. z=y; Mutant
Another mutant 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; Program 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z < 1) 6. z =z/x; 7. else 8. z=y; Mutant
Now suppose that a mutant of P is obtained by changing x=x+1 to x=abs(x)+1 .
This mutant is equivalent to P as no test case can distinguish it from P.
Mutation Testing Procedure Given P and a test set T: 1. Generate mutants 2. Compile P and the mutants 3. Execute P and the mutants on each test case. 4. Determine equivalent mutants.. 5. Determine mutation score. 6. If mutation score is not 1 then improve the test set and repeat from step 3.