1.
Software Testing and Reliability Test Assessment and Enhancement Aditya P. Mathur Purdue University August 12-16 @ Guidant Corporation Minneapolis/St Paul, MN Last update: August 13, 2002 Graduate Assistants : Ramkumar Natarajan Baskar Sridharan
A Data Flow Oriented Program Testing Strategy , Janusz Laski and Bogden Korel, IEEE Transactions on Software Engineering, Vol. SE-9, pp. 347-354, May 1983.
Hints on Test Data Selection: Help for the Practicing Programmer , Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward, IEEE Computer, pp 34-41, April 1978.
Mutation Testing , Aditya P. Mathur, in Encyclopedia of Software Engineering, Ed. John Marcianiak, Wiley Interscience, Vol 1pp. 707-712.
Test assessment and improvement is applicable throughout the testing process and during all stages of software development.
Hence we say that test assessment and improvement helps in the improvement of software reliability .
11.
Test Assessment Procedure Yes Improve T No Measure adequacy of T w.r.t. C. 2 Is T adequate? 3 Yes 4 More testing is warranted ? 5 Select an adequacy criterion C. 1 Develop T 0 No Done 6
To formulate and understand the coverage principle, we need to understand:
coverage domains
coverage elements
A coverage domain is a finite domain , related to the program under test, that we want to cover . Coverage elements are the individual elements of this domain
14.
The Coverage Principle (contd.) Coverage Domains Coverage Elements Requirements Classes Functions Interface mutations Exceptions
Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved confidence in the reliability of the system under test.
It is required to write a program that takes in the name of a person as a string and searches for the name in a file of names. The program must output the record ID which matches the given name. In case of no match a -1 is returned.
What coverage domains can be identified from this requirement?
One measure is the size of T. By this measure a test set with a larger number of test cases corresponds to higher effort than one with a lesser number of test cases.
Each coverage criterion has its error detection ability. This is also known as the error detection effectiveness or simply effectiveness of the criterion.
One measure of the effectiveness of criterion C is the fraction of faults guaranteed to be revealed by a test T that satisfies C.
Another measure is the probability that at least fraction f of the faults in P will be revealed by test T that satisfies C.
Unfortunately there is no absolute measure of the effectiveness of any given coverage criterion for a general class of programs and for arbitrary test sets.
The rate at which new faults are discovered reduces as test adequacy with respect to a finite coverage domain increases ; it reduces to zero when the coverage domain has been exhausted.
coverage 0 1
27.
Saturation Effect: Fault View Testing Effort Remaining Faults 0 N Functional t f s t f e t d s M t df e t m e
28.
Saturation Effect: Reliability View Functional, Decision, Dataflow, and Mutation tsting provide various test assessment criteria. True reliability (R) Estimated reliability (R’) Saturation region Reliability Testing Effort R’ f R’ d R’ df R’ m Functional R f t f s t f e Decision R d t d s t d e Dataflow R df t df s t df e Mutation R m t m s t f e
Control flow graph (CFG) of a program is a representation of the flow of execution within the program.
More formally, a CFG G is:
G=(N,A)
where N: set of nodes and A: set of arcs
There is a unique entry node e n in N.
There is a unique exit node ex in N. A node represents a single statement or a block .
A block is a single-entry-single-exit sequence of instructions that are always executed in a sequence without any diversion of path except at the end of the block.
For any two adjacent arcs a i = ( n,m ) and a j = ( p,q ), m=p .
A path is considered executable or feasible if there exists a test case which causes this path to be traversed during program execution , otherwise the path is unexecutable or infeasible .
Draw a CFG for the following program and identify all paths. :
1. scanf (x,y); if (y<0) 2. pow=0-y; 3. else pow=y; 4. z=1.0; 5. while (pow !=0) 6. {z=z*x; pow=pow-1;} 7. if (y<0) 8. z=1.0/z; 9. printf(z); What does the above program compute ?
34.
Control-flow Graph 2 3 pow=0-y; else pow=y; 4 z=1.0; 5 while (pow !=0) {z=z*x; pow=pow-1;} 6 7 if (y<0) 8 9 z=1.0/z; printf(z); 1 scanf (x,y); if (y<0) en ex
Restated in terms of G, T is adequate w.r.t. the statement coverage criterion if each node in N is on at least one of the paths traversed when P is executed on each element of T.
In G there may be nodes which correspond to conditions in P. Such nodes, also called condition nodes, contain branches in P.
Each such node is considered covered if during some execution of P, the condition evaluates to true and false ; these executions of P need not be the same.
The coverage domain consists of all branches in G. Restated, in terms of the control flow graph, it is the set of all arcs exiting the condition nodes.
A test T satisfies the branch coverage criterion if upon execution of P on each element of T, each branch of P has been executed at least once.
A compound condition is considered covered if all of its constituent elementary conditions evaluate to true and false, respectively, during some execution of P.
A test set T is adequate w.r.t. condition coverage if all conditions in P are covered when P is executed on elements of T.
Improve T from the previous exercise so that it is adequate w.r.t. the condition coverage criterion for the check function and does not reveal the error .
As mentioned before, a path through a program is a sequence of statements such that the entry node of the program CFG is the first node on the path and the exit node is the last one on the path.
Is this definition equivalent to the one given earlier ?
A test set T is considered adequate w.r.t. the path coverage criterion if all paths in P are executed at least once upon execution on each element of T.
Class exercise:
Construct T for set_z such that T is adequate w.r.t. the path coverage criterion and does not reveal the error .
One can develop a test strategy based on any of the criteria discussed.
Example:
A test strategy based on the statement coverage criterion will begin by evaluating a test set T against this criterion. Then new tests will be added to T until all the statements are covered, i.e. T satisfies the criterion.
The largest element is saved in R0 and R3 points to the location of R0 in a .
The completion of one iteration of the outer loop ensures that the sub-array a[0:R1-1] has been sorted and that a[R1-1] is greater than or equal to any element of a[R1:N].
c-use stands for computational use and p-use for predicate-use .
Both c- and p-uses affect the flow of control: p-uses directly as their values are used in evaluating conditions and c-uses indirectly as their values are used to compute other variables which in turn affect the outcome of condition evaluation.
A path from node i to node j is said to be def-clear w.r.t. a variable x if there is no def of x in the nodes along the path from node i to node j . Nodes i and j may have a def of x .
A def-clear path from node i to edge ( j,k ) is one in which no node on the path has a def of x .
75.
def-use graph exercise 0. int set_z(x,y); { 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; } Draw a def-use graph for the following program.
For the above graph generate a test set that satisfies
the branch coverage criterion
the all-defs criterion - for definitions of all variables at least one use (c- or p- use) must be exercised .
the all-uses criterion- all p-uses and all c-uses of all variable definitions be covered .
Develop the tests incrementally, i.e. by modifying the previous test set!
78.
SUDS processing: Phase I P, Program under test Preprocess, compile and instrument . trace file upon execution . atac files generate Instrumented version of P (executable) generate Test set input Program output upon execution
79.
ATAC processing: phase II coverage analyzer .atac files .trace file control flow and data flow coverage values
The process of program development is considered as iterative whereby an initial version of the program is refined by making simple, or a combination of simple changes, towards the final version.
Given a program P, a mutant of P is obtained by making a simple change in P.
What is zpush ? 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; Program 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/ zpush(x); 7. else 8. z=y; Mutant
83.
Another mutant 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; Program 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z < 1) 6. z =z/x; 7. else 8. z=y; Mutant
Now suppose that a mutant of P is obtained by changing x=x+1 to x=abs(x)+1 .
This mutant is equivalent to P as no test case can distinguish it from P.
92.
Mutation Testing Procedure Given P and a test set T: 1. Generate mutants 2. Compile P and the mutants 3. Execute P and the mutants on each test case. 4. Determine equivalent mutants.. 5. Determine mutation score. 6. If mutation score is not 1 then improve the test set and repeat from step 3.
Be the first to comment