1. Science and Information Conference 2015
July 28-30, 2015 | London, UK
1 | P a g e
www.conference.thesai.org
Analyzing Test Case Quality with Mutation Testing
Approach
Zainab Nayyar, Nazish Rafique, Nousheen Hashmi, Nadia Rashid, Saba Awan
Department of Computer Engineering, College of EME, National University of Sciences and Technology (NUST),
H-12, Islamabad, Pakistan
zainabnayyar786@gmail.com, nazishrafique16@gmail.com, nsheen25@gmail.com, nadia1.rashid@gmail.com,
sabaawan90@gmail.com
Abstract— Testing is the most important process for the
validation and verification of the industrialized products.
Similarly, in the field of software development it counts a lot.
Sometimes it becomes difficult to judge the quality of test cases
especially in the case of large and complex projects, where not
even a single fault is tolerable and can have more consequences.
So the solution lies in performing a mutation testing process
which is a type of white box testing. In the mutation testing
process, the quality of test cases is analyzed by just making
mutants of the original code and then by killing them. In this
paper, the whole process of mutation testing approach is
analyzed and then the process is applied on a piece of code to give
the complete step by step description of making and killing the
mutants.
Keywords— Testing, White box testing, Mutation testing,
Mutants, Test cases, Test case quality.
I. INTRODUCTION
Testing is the process which is carried out to check whether
the code is working according to the given specifications or
not. Testing involves black box testing and white box testing.
White box testing is a structural testing which aims to check
the internal structure of the code, logical and semantic errors
of the code. Mutation testing technique is a kind of white box
testing. Mutation testing is a technique of software testing
proposed originally by Hamlet [1]. The mutation testing is the
structural method of testing whose objective is to access and
improve the test suites competence and also to calculate the
total faults that are there in system under testing. Mutation
testing aims to assess test case quality by introducing faults in
programs to check whether the test cases will highlight them.
Further, this process aims to utilize these evaluations to assist
in building more competent tests for the original program.
Mutation is said to put a little change in the program and these
minor changes are aimed to model defects at low level that
occur during coding of systems process. Operators of mutation
are generally based on theoretical or empirical fault models
[2] and numerous experimentation iterations are normally
required to build and distill mutation operators set. Mutation
Operators have been built for many programming languages,
such as C [3], Java [4] [5] and Fortran 77 [6] [7].
Mutant operators help to create alternative program versions
under the test and test designs which compose these program
mutants to act in a different way as compared to the original
program. Measure of quality is how many mutants can be
killed by the test set. Mutants on test set should be run and
testers should examine live mutants and to kill these mutants
to design appropriate test sets. Furthermore, there can be
mutants who cannot be killed, i.e. for all tests the program
responds in same way; they are called equivalent mutants and
they show that the code have semantic errors and is logically
weak and not able to identify faults. Discovering these
mutants is mostly done manually. If for a given test suit, no
test case can possibly differentiate among original version of
program and mutant then the mutant remains alive. An
effective test case is the one that kills as many mutants as
possible.
Many researches have been done on mutation testing for the
identification of faults in the program or to check the
effectiveness of test cases. In order to apply mutation on new
technologies, the key is to design efficient and effective
mutation operators. . In this paper, we shall analyze mutation
testing by applying it on different pieces of code to evaluate
the effectiveness and quality of test cases
II. RELATED WORK
Following are the relevant studies which have been made for
applying mutation testing process:
In [8], the ability of test cases for the detection of hand
seeded faults (semantic errors), automatically generated and
real world faults. The mutants were generated on the basis of
hand seeded and real time code. It was seen that hand seeded
faults were difficult to detect then real world faults.
Many researchers had shown that some of the generated
mutants were sometimes not needed. The problem addressed
in [9] [13] was to reduce the number of mutants by
maintaining the test case quality. Higher order mutation
technique was discussed which was reduced the mutants to
some extent but that did not give the exact and accurate
solution so a theoretical framework called dynamic
subsumption was used to reduce the mutants. This graphical
model was used to visualize the repetition between mutants.
Subsumption graphs [9] [13] were used to analyze the
relationships between different mutants. Subsumption was
used in a way in which a mutant subsumes another if at least
one test kills the first and every test that kills the first also kills
the second. This was used to remove the redundancy in
mutants.

2. Science and Information Conference 2015
July 28-30, 2015 | London, UK
2 | P a g e
www.conference.thesai.org
As mutation testing is a very costly process so in [10] the
emphasis was put on a question that is this mandatory to
generate multiple sets of test cases? How cost and
effectiveness of result effected by different sets of test cases?
Results shown that there could be significant differences for
individual subject programs among different test sets chosen
for the same adequacy criterion. These differences were
observed for both effectiveness (mutation score) and cost
(number of tests). This result makes a case for choosing
multiple test sets during experimentation. Statement deletion
in mutation testing has been introduced from the beginning.
This technique helped in reducing the cost of mutation testing.
Different approaches have been introduced so far. Lin Deng et
al. proposed a technique in which they used a SDL (statement
deletion) operator was implemented [11]. It was basically used
to delete the statements in the code and make mutants in order
to generate tests. This operator was applied to 40 different
Java classes. Tests were created to eliminate the non-
equivalent SDL mutants. This minimizes the cost of mutation
testing hence making it more attractive. In [14], the same
technique was used with one OP-mutation in which a single
strong operator was used to kill the test cases and reduce costs,
but it was applied on C language. The experimentations were
made on 39 different programs.
Jingyu Hu et al in [12] presented a study of class level
mutants. Class level mutant operators change the features
related to object oriented programming. Non-equivalent
mutants are being killed and it showed that how hard it is to
kill individual mutants. This study showed that class level
mutants kills by less number of tests as compared to the
statement level mutants.
In [15], the author proposed the genetic algorithms and co-
evolution process to solve two main problems in mutation
testing. First problem is the detection of equivalent mutants
and the second one is the generation of huge number of
mutants. A fitness function was designed to show possible
ways to detect equivalent mutants. In the proposed
methodology, the idea is to apply genetic algorithm to evolve
subsets of mutants from the large number of mutants
generated for original program. Further, to evolve subsets of
test cases from pool of test cases generated randomly, same
technique was applied. And the last step was to co-evolve sets
of mutants and sets of test cases.
To reduce the cost of mutation testing, statement deletion
operator technique is used to delete entire statement from
programs. In [16], an innovative mutation operators were
introduced which deletes some part of the statements. Three
types of operators were defined in the paper. Operator deletion
which deletes the arithmetic and relational operators, variable
deletion operators which removes variable and constant
deletion which deletes constants. The study has shown that
with the use of deletion operators we require less number of
tests than traditional set of operators.
III. MUTATION TESTING PROCESS
In mutation testing, mutants are created within the original
piece of code and test cases are applied on both the original
code and the new mutated code. The output of mutants and the
original code is compared; if they are similar then that means
that the mutants remain alive and test data is needed to be
redesigned. If the out is different, then the mutants in the code
will be killed, verifying the good quality and effectiveness of
the test cases. As shown in the figure below.
Fig. 1. Mutation Testing Process
A. Value Mutation
In this type of mutation, values of variables or constants
are changed to test the behavior of original code and mutant.
TABLE I. VALUE MUTATION
Original Code: Mutant
Int x;
Int y;
Int sum;
Cin>>x;
Cin>>y;
Sum= x+ y;
Sum++;
Cout<<sum;
Int x;
Int y;
Int sum;
Cin>>x;
Cin>>y;
Sum= x+ y;
Sum--;
Cout<<sum;
Test case 1:
x=5, y=9;
TABLE II. OUTPUT
Value of sum =15 Value of sum =13
In the above code different values are assigned to the variables
i.e. x=5 and y=15. The above code simply adds two values but

3. Science and Information Conference 2015
July 28-30, 2015 | London, UK
3 | P a g e
www.conference.thesai.org
in the original code the values are incrementing but to check
whether the code is working correctly and identifying hand
seeded faults or not the increment operators are changed to
decrement operators in the mutant. In the output it is shown
that the output of both the codes is not same due to the change
of increment and decrement operators and the test case is able
to detect faults in the code.
B. Statement Mutation
In this type, the output of original code and mutants are
compared by deleting or swapping the entire statements.
TABLE III. STATEMENT MUTATION EXAMPLE 1
Original Code: Mutant
Int x;
Int y;
Cin>>x;
Cin>>y;
If(x>y)
{
Cout<< ”x is greater”;
}
Else if(y>x)
{
Cout<< ”y is greater”;
}
Else
{
Cout<<“x and y are equal”;
}
Int x;
Int y;
Cin>>x;
Cin>>y;
If(x>y)
{
Cout<< ”x is greater”;
}
//Else if(y>x)
// {
//Cout<< ”y is greater”;
//}
Else
{
Cout<<“x and y are equal”;
}
Test case 1:
x=5, y=9;
TABLE IV. OUTPUT I
y is greater Program terminate
TABLE V. STATEMENT MUTATION EXAMPLE 2
Original Code: Mutant
Public void testif ()
{
Int a, b, c , t;
If (a<5)
{
T= t+ b+ c;
}
Else if (a>20)
{
t= t + a +c;
b++;
}}
Public void testif ()
{
Int a, b, c , t;
If (a<5)
{
T= t+ b+ c;
}
Else if (a>20)
{
t= t + a +c;
b++;
b++; }}
Test case 1:
A= 7; b= 1, c=2, t=3;
TABLE VI. OUTPUT 2
T=12
B=2
T=12
B=3
C. Decision Mutation
In this type, conditional statements are modified like >
replaces the < sign.
TABLE VII. DECISION MUTATION EXAMPLE
Original Code: Mutant
Public void testwhile ()
{
Int a, b, c, t;
While (a<5)
{
t= t + b + c;
a++;
}
}
Public void testwhile ()
{
Int a, b, c, t;
While (true)
{
t= t + b + c;
a++;
}
}
Test case 1:
a= 4, b= 3, c= 2, t=1;
Test case 2:
a=5, b= 4, c=2, t=1;
TABLE VIII. OUTPUT
When a>5
T=6;
A=5;
When a>5
Infinite Loop
IV. CONCLUSION
Mutation testing is helpful for verification and validation
of critical requirements without which system cannot
act/operate properly. The mutation testing applied on small
piece of codes in this paper, illustrates the different types of
mutation operators that can be applied to identify the accuracy
of the test case.
It not only ensures the good quality of test cases but also
helps to detect faults of programs. Through mutation testing,
effective test cases can be made. Though, it is considered as a
costly process but one time cost is better than the system
failure after deployment

4. Science and Information Conference 2015
July 28-30, 2015 | London, UK
4 | P a g e
www.conference.thesai.org
REFERENCES
[1] Hamlet, R.G. Testing programs with the aid of a compiler. Pages 279-
290. IEEE Transactions on Software Engineering..
[2] J. Offutt, R. Alexander, Y. Wu, Q. Xiao, and C. Hutchinson. A fault
model for subtype inheritance and polymorphism. Pages 84–93. 12th
International Symposium on Software Reliability Engineering.
November 2001.
[3] M. E. Delamaro and J. C. Maldonado. Proteum-A tool for the assessment
of test adequacy for C programs. Pages 79–95. Performability in
Computing Systems. July 1996.
[4] S. Kim, J. A. Clark, and J. A. Mc Dermid. Investigating the effectiveness
of object-oriented strategies with the mutation method. Pages 4–100.
Mutation Testing in the Twentieth and the Twenty First Centuries.
October 2000. December 2001.
[5] Y.-S. Ma, J. Offutt, and Y.-R. Kwon. MuJava : An automated class
mutation system. Volume 15, no. 2, pages 97–133. Software Testing,
Verification, and Reliability. June 2005.
[6] R. A. De Millo and J. Offutt. Constraint-based automatic test data
generation. Volume 17, no. 9, pages 900–910. IEEE Transactions on
Software Engineering. September 1991.
[7] K. N. King and J. Offutt. A Fortran language system for mutation-based
software testing. Volume 21, no. 7, pages 685–718. Software-Practice
and Experience. July 1991.
[8] J. H. Andrews, L. C. Briand, Y. Labiche. Is Mutation an Appropriate
Tool for Testing Experiments? pages 402-411. IEEE International
Conference on Software Engineering (ICSE). 2005.
[9] Paul Ammann, Marcio E. Delamaro and Jeff Offutt. Establishing
Theoretical Minimal Sets of Mutants. 7th IEEE International Conference
on Software Testing, Verification and Validation (ICST 2014). March
2014.
[10] Marcio E. Delamaro and Jeff Offutt. Assessing the Influence of Multiple
Test Case Selection on Mutation Experiments. Tenth Workshop on
Mutation Analysis (Mutation 2014). March 2014.
[11] .Lin Deng, Jeff Offutt and Nan Li, Empirical Evaluation of the
Statement Deletion Mutation Operator, Software Engineering. 6th
IEEE International Conference on Software Testing, Verification and
Validation (ICST). March 2011
[12]. Jingyu Hu, Nan Li and Jeff Offutt. An Analysis of OO Mutation
Operators,Software Engineering. IEEE Fourth International
Conference on Software Testing, Verification and Validation
Workshops (ICSTW). Pages 334-341. 21-25 March 2011.
[13]. Bob Kurtzy, Paul Ammanny, Marcio E. Delamaro, Jeff Offutty and Lin
Dengy. Mutant Subsumption Graphs. Tenth Workshop on Mutation
Analysis. March 2014.
[14]. Marcio E. Delamaro, Lin Deng, Vinicius H. S. Durelli, Nan Li and Jeff
Offutty. Experimental Evaluation of SDL and One-Op Mutation for C.
IEEE 7th International Conference on Software Testing, Verification
and Validation. April 2014.
[15]. Konstantinos Adamopoulos, Mark Harman and Robert M. Hierons,How
to Overcome the Equivalent Mutant Problem and Achieve Tailored
Selective Mutation Using Co-evolution
[16]. Marcio E. Delamaro, Jeff Offutt, Paul Ammann, Designing Deletion
Mutation Operators