SlideShare a Scribd company logo
1 of 4
Download to read offline
Science and Information Conference 2015
July 28-30, 2015 | London, UK
1 | P a g e
www.conference.thesai.org
Analyzing Test Case Quality with Mutation Testing
Approach
Zainab Nayyar, Nazish Rafique, Nousheen Hashmi, Nadia Rashid, Saba Awan
Department of Computer Engineering, College of EME, National University of Sciences and Technology (NUST),
H-12, Islamabad, Pakistan
zainabnayyar786@gmail.com, nazishrafique16@gmail.com, nsheen25@gmail.com, nadia1.rashid@gmail.com,
sabaawan90@gmail.com
Abstract— Testing is the most important process for the
validation and verification of the industrialized products.
Similarly, in the field of software development it counts a lot.
Sometimes it becomes difficult to judge the quality of test cases
especially in the case of large and complex projects, where not
even a single fault is tolerable and can have more consequences.
So the solution lies in performing a mutation testing process
which is a type of white box testing. In the mutation testing
process, the quality of test cases is analyzed by just making
mutants of the original code and then by killing them. In this
paper, the whole process of mutation testing approach is
analyzed and then the process is applied on a piece of code to give
the complete step by step description of making and killing the
mutants.
Keywords— Testing, White box testing, Mutation testing,
Mutants, Test cases, Test case quality.
I. INTRODUCTION
Testing is the process which is carried out to check whether
the code is working according to the given specifications or
not. Testing involves black box testing and white box testing.
White box testing is a structural testing which aims to check
the internal structure of the code, logical and semantic errors
of the code. Mutation testing technique is a kind of white box
testing. Mutation testing is a technique of software testing
proposed originally by Hamlet [1]. The mutation testing is the
structural method of testing whose objective is to access and
improve the test suites competence and also to calculate the
total faults that are there in system under testing. Mutation
testing aims to assess test case quality by introducing faults in
programs to check whether the test cases will highlight them.
Further, this process aims to utilize these evaluations to assist
in building more competent tests for the original program.
Mutation is said to put a little change in the program and these
minor changes are aimed to model defects at low level that
occur during coding of systems process. Operators of mutation
are generally based on theoretical or empirical fault models
[2] and numerous experimentation iterations are normally
required to build and distill mutation operators set. Mutation
Operators have been built for many programming languages,
such as C [3], Java [4] [5] and Fortran 77 [6] [7].
Mutant operators help to create alternative program versions
under the test and test designs which compose these program
mutants to act in a different way as compared to the original
program. Measure of quality is how many mutants can be
killed by the test set. Mutants on test set should be run and
testers should examine live mutants and to kill these mutants
to design appropriate test sets. Furthermore, there can be
mutants who cannot be killed, i.e. for all tests the program
responds in same way; they are called equivalent mutants and
they show that the code have semantic errors and is logically
weak and not able to identify faults. Discovering these
mutants is mostly done manually. If for a given test suit, no
test case can possibly differentiate among original version of
program and mutant then the mutant remains alive. An
effective test case is the one that kills as many mutants as
possible.
Many researches have been done on mutation testing for the
identification of faults in the program or to check the
effectiveness of test cases. In order to apply mutation on new
technologies, the key is to design efficient and effective
mutation operators. . In this paper, we shall analyze mutation
testing by applying it on different pieces of code to evaluate
the effectiveness and quality of test cases
II. RELATED WORK
Following are the relevant studies which have been made for
applying mutation testing process:
In [8], the ability of test cases for the detection of hand
seeded faults (semantic errors), automatically generated and
real world faults. The mutants were generated on the basis of
hand seeded and real time code. It was seen that hand seeded
faults were difficult to detect then real world faults.
Many researchers had shown that some of the generated
mutants were sometimes not needed. The problem addressed
in [9] [13] was to reduce the number of mutants by
maintaining the test case quality. Higher order mutation
technique was discussed which was reduced the mutants to
some extent but that did not give the exact and accurate
solution so a theoretical framework called dynamic
subsumption was used to reduce the mutants. This graphical
model was used to visualize the repetition between mutants.
Subsumption graphs [9] [13] were used to analyze the
relationships between different mutants. Subsumption was
used in a way in which a mutant subsumes another if at least
one test kills the first and every test that kills the first also kills
the second. This was used to remove the redundancy in
mutants.
Science and Information Conference 2015
July 28-30, 2015 | London, UK
2 | P a g e
www.conference.thesai.org
As mutation testing is a very costly process so in [10] the
emphasis was put on a question that is this mandatory to
generate multiple sets of test cases? How cost and
effectiveness of result effected by different sets of test cases?
Results shown that there could be significant differences for
individual subject programs among different test sets chosen
for the same adequacy criterion. These differences were
observed for both effectiveness (mutation score) and cost
(number of tests). This result makes a case for choosing
multiple test sets during experimentation. Statement deletion
in mutation testing has been introduced from the beginning.
This technique helped in reducing the cost of mutation testing.
Different approaches have been introduced so far. Lin Deng et
al. proposed a technique in which they used a SDL (statement
deletion) operator was implemented [11]. It was basically used
to delete the statements in the code and make mutants in order
to generate tests. This operator was applied to 40 different
Java classes. Tests were created to eliminate the non-
equivalent SDL mutants. This minimizes the cost of mutation
testing hence making it more attractive. In [14], the same
technique was used with one OP-mutation in which a single
strong operator was used to kill the test cases and reduce costs,
but it was applied on C language. The experimentations were
made on 39 different programs.
Jingyu Hu et al in [12] presented a study of class level
mutants. Class level mutant operators change the features
related to object oriented programming. Non-equivalent
mutants are being killed and it showed that how hard it is to
kill individual mutants. This study showed that class level
mutants kills by less number of tests as compared to the
statement level mutants.
In [15], the author proposed the genetic algorithms and co-
evolution process to solve two main problems in mutation
testing. First problem is the detection of equivalent mutants
and the second one is the generation of huge number of
mutants. A fitness function was designed to show possible
ways to detect equivalent mutants. In the proposed
methodology, the idea is to apply genetic algorithm to evolve
subsets of mutants from the large number of mutants
generated for original program. Further, to evolve subsets of
test cases from pool of test cases generated randomly, same
technique was applied. And the last step was to co-evolve sets
of mutants and sets of test cases.
To reduce the cost of mutation testing, statement deletion
operator technique is used to delete entire statement from
programs. In [16], an innovative mutation operators were
introduced which deletes some part of the statements. Three
types of operators were defined in the paper. Operator deletion
which deletes the arithmetic and relational operators, variable
deletion operators which removes variable and constant
deletion which deletes constants. The study has shown that
with the use of deletion operators we require less number of
tests than traditional set of operators.
III. MUTATION TESTING PROCESS
In mutation testing, mutants are created within the original
piece of code and test cases are applied on both the original
code and the new mutated code. The output of mutants and the
original code is compared; if they are similar then that means
that the mutants remain alive and test data is needed to be
redesigned. If the out is different, then the mutants in the code
will be killed, verifying the good quality and effectiveness of
the test cases. As shown in the figure below.
Fig. 1. Mutation Testing Process
A. Value Mutation
In this type of mutation, values of variables or constants
are changed to test the behavior of original code and mutant.
TABLE I. VALUE MUTATION
Original Code: Mutant
Int x;
Int y;
Int sum;
Cin>>x;
Cin>>y;
Sum= x+ y;
Sum++;
Cout<<sum;
Int x;
Int y;
Int sum;
Cin>>x;
Cin>>y;
Sum= x+ y;
Sum--;
Cout<<sum;
Test case 1:
x=5, y=9;
TABLE II. OUTPUT
Value of sum =15 Value of sum =13
In the above code different values are assigned to the variables
i.e. x=5 and y=15. The above code simply adds two values but
Science and Information Conference 2015
July 28-30, 2015 | London, UK
3 | P a g e
www.conference.thesai.org
in the original code the values are incrementing but to check
whether the code is working correctly and identifying hand
seeded faults or not the increment operators are changed to
decrement operators in the mutant. In the output it is shown
that the output of both the codes is not same due to the change
of increment and decrement operators and the test case is able
to detect faults in the code.
B. Statement Mutation
In this type, the output of original code and mutants are
compared by deleting or swapping the entire statements.
TABLE III. STATEMENT MUTATION EXAMPLE 1
Original Code: Mutant
Int x;
Int y;
Cin>>x;
Cin>>y;
If(x>y)
{
Cout<< ”x is greater”;
}
Else if(y>x)
{
Cout<< ”y is greater”;
}
Else
{
Cout<<“x and y are equal”;
}
Int x;
Int y;
Cin>>x;
Cin>>y;
If(x>y)
{
Cout<< ”x is greater”;
}
//Else if(y>x)
// {
//Cout<< ”y is greater”;
//}
Else
{
Cout<<“x and y are equal”;
}
Test case 1:
x=5, y=9;
TABLE IV. OUTPUT I
y is greater Program terminate
TABLE V. STATEMENT MUTATION EXAMPLE 2
Original Code: Mutant
Public void testif ()
{
Int a, b, c , t;
If (a<5)
{
T= t+ b+ c;
}
Else if (a>20)
{
t= t + a +c;
b++;
}}
Public void testif ()
{
Int a, b, c , t;
If (a<5)
{
T= t+ b+ c;
}
Else if (a>20)
{
t= t + a +c;
b++;
b++; }}
Test case 1:
A= 7; b= 1, c=2, t=3;
TABLE VI. OUTPUT 2
T=12
B=2
T=12
B=3
C. Decision Mutation
In this type, conditional statements are modified like >
replaces the < sign.
TABLE VII. DECISION MUTATION EXAMPLE
Original Code: Mutant
Public void testwhile ()
{
Int a, b, c, t;
While (a<5)
{
t= t + b + c;
a++;
}
}
Public void testwhile ()
{
Int a, b, c, t;
While (true)
{
t= t + b + c;
a++;
}
}
Test case 1:
a= 4, b= 3, c= 2, t=1;
Test case 2:
a=5, b= 4, c=2, t=1;
TABLE VIII. OUTPUT
When a>5
T=6;
A=5;
When a>5
Infinite Loop
IV. CONCLUSION
Mutation testing is helpful for verification and validation
of critical requirements without which system cannot
act/operate properly. The mutation testing applied on small
piece of codes in this paper, illustrates the different types of
mutation operators that can be applied to identify the accuracy
of the test case.
It not only ensures the good quality of test cases but also
helps to detect faults of programs. Through mutation testing,
effective test cases can be made. Though, it is considered as a
costly process but one time cost is better than the system
failure after deployment
Science and Information Conference 2015
July 28-30, 2015 | London, UK
4 | P a g e
www.conference.thesai.org
REFERENCES
[1] Hamlet, R.G. Testing programs with the aid of a compiler. Pages 279-
290. IEEE Transactions on Software Engineering..
[2] J. Offutt, R. Alexander, Y. Wu, Q. Xiao, and C. Hutchinson. A fault
model for subtype inheritance and polymorphism. Pages 84–93. 12th
International Symposium on Software Reliability Engineering.
November 2001.
[3] M. E. Delamaro and J. C. Maldonado. Proteum-A tool for the assessment
of test adequacy for C programs. Pages 79–95. Performability in
Computing Systems. July 1996.
[4] S. Kim, J. A. Clark, and J. A. Mc Dermid. Investigating the effectiveness
of object-oriented strategies with the mutation method. Pages 4–100.
Mutation Testing in the Twentieth and the Twenty First Centuries.
October 2000. December 2001.
[5] Y.-S. Ma, J. Offutt, and Y.-R. Kwon. MuJava : An automated class
mutation system. Volume 15, no. 2, pages 97–133. Software Testing,
Verification, and Reliability. June 2005.
[6] R. A. De Millo and J. Offutt. Constraint-based automatic test data
generation. Volume 17, no. 9, pages 900–910. IEEE Transactions on
Software Engineering. September 1991.
[7] K. N. King and J. Offutt. A Fortran language system for mutation-based
software testing. Volume 21, no. 7, pages 685–718. Software-Practice
and Experience. July 1991.
[8] J. H. Andrews, L. C. Briand, Y. Labiche. Is Mutation an Appropriate
Tool for Testing Experiments? pages 402-411. IEEE International
Conference on Software Engineering (ICSE). 2005.
[9] Paul Ammann, Marcio E. Delamaro and Jeff Offutt. Establishing
Theoretical Minimal Sets of Mutants. 7th IEEE International Conference
on Software Testing, Verification and Validation (ICST 2014). March
2014.
[10] Marcio E. Delamaro and Jeff Offutt. Assessing the Influence of Multiple
Test Case Selection on Mutation Experiments. Tenth Workshop on
Mutation Analysis (Mutation 2014). March 2014.
[11] .Lin Deng, Jeff Offutt and Nan Li, Empirical Evaluation of the
Statement Deletion Mutation Operator, Software Engineering. 6th
IEEE International Conference on Software Testing, Verification and
Validation (ICST). March 2011
[12]. Jingyu Hu, Nan Li and Jeff Offutt. An Analysis of OO Mutation
Operators,Software Engineering. IEEE Fourth International
Conference on Software Testing, Verification and Validation
Workshops (ICSTW). Pages 334-341. 21-25 March 2011.
[13]. Bob Kurtzy, Paul Ammanny, Marcio E. Delamaro, Jeff Offutty and Lin
Dengy. Mutant Subsumption Graphs. Tenth Workshop on Mutation
Analysis. March 2014.
[14]. Marcio E. Delamaro, Lin Deng, Vinicius H. S. Durelli, Nan Li and Jeff
Offutty. Experimental Evaluation of SDL and One-Op Mutation for C.
IEEE 7th International Conference on Software Testing, Verification
and Validation. April 2014.
[15]. Konstantinos Adamopoulos, Mark Harman and Robert M. Hierons,How
to Overcome the Equivalent Mutant Problem and Achieve Tailored
Selective Mutation Using Co-evolution
[16]. Marcio E. Delamaro, Jeff Offutt, Paul Ammann, Designing Deletion
Mutation Operators

More Related Content

What's hot

Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...
Chakkrit (Kla) Tantithamthavorn
 
A software fault localization technique based on program mutations
A software fault localization technique based on program mutationsA software fault localization technique based on program mutations
A software fault localization technique based on program mutations
Tao He
 
A survey of software testing
A survey of software testingA survey of software testing
A survey of software testing
Tao He
 
Unit Testing Guidelines
Unit Testing GuidelinesUnit Testing Guidelines
Unit Testing Guidelines
Joel Hooks
 
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
Chakkrit (Kla) Tantithamthavorn
 
AI-Driven Software Quality Assurance in the Age of DevOps
AI-Driven Software Quality Assurance in the Age of DevOpsAI-Driven Software Quality Assurance in the Age of DevOps
AI-Driven Software Quality Assurance in the Age of DevOps
Chakkrit (Kla) Tantithamthavorn
 
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
Chakkrit (Kla) Tantithamthavorn
 

What's hot (20)

Software testing using genetic algorithms
Software testing using genetic algorithmsSoftware testing using genetic algorithms
Software testing using genetic algorithms
 
Comprehensive Testing Tool for Automatic Test Suite Generation, Prioritizatio...
Comprehensive Testing Tool for Automatic Test Suite Generation, Prioritizatio...Comprehensive Testing Tool for Automatic Test Suite Generation, Prioritizatio...
Comprehensive Testing Tool for Automatic Test Suite Generation, Prioritizatio...
 
Practices and Tools for Better Software Testing
Practices and Tools for  Better Software TestingPractices and Tools for  Better Software Testing
Practices and Tools for Better Software Testing
 
Software testing lab manual
Software testing lab manualSoftware testing lab manual
Software testing lab manual
 
Software Testing 1/5
Software Testing 1/5Software Testing 1/5
Software Testing 1/5
 
Dynamic Radius Species Conserving Genetic Algorithm for Test Generation for S...
Dynamic Radius Species Conserving Genetic Algorithm for Test Generation for S...Dynamic Radius Species Conserving Genetic Algorithm for Test Generation for S...
Dynamic Radius Species Conserving Genetic Algorithm for Test Generation for S...
 
On the application of SAT solvers for Search Based Software Testing
On the application of SAT solvers for Search Based Software TestingOn the application of SAT solvers for Search Based Software Testing
On the application of SAT solvers for Search Based Software Testing
 
On The Relation of Test Smells to Software Code Quality
On The Relation of Test Smells to Software Code QualityOn The Relation of Test Smells to Software Code Quality
On The Relation of Test Smells to Software Code Quality
 
Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...Towards a Better Understanding of the Impact of Experimental Components on De...
Towards a Better Understanding of the Impact of Experimental Components on De...
 
Making property-based testing easier to read for humans
Making property-based testing easier to read for humansMaking property-based testing easier to read for humans
Making property-based testing easier to read for humans
 
A software fault localization technique based on program mutations
A software fault localization technique based on program mutationsA software fault localization technique based on program mutations
A software fault localization technique based on program mutations
 
Test-Driven Development
Test-Driven DevelopmentTest-Driven Development
Test-Driven Development
 
A survey of software testing
A survey of software testingA survey of software testing
A survey of software testing
 
Software testing: an introduction - 2017
Software testing: an introduction - 2017Software testing: an introduction - 2017
Software testing: an introduction - 2017
 
Unit Testing Guidelines
Unit Testing GuidelinesUnit Testing Guidelines
Unit Testing Guidelines
 
An Introduction to Test Driven Development
An Introduction to Test Driven Development An Introduction to Test Driven Development
An Introduction to Test Driven Development
 
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
Explainable Artificial Intelligence (XAI) 
to Predict and Explain Future Soft...
 
AI-Driven Software Quality Assurance in the Age of DevOps
AI-Driven Software Quality Assurance in the Age of DevOpsAI-Driven Software Quality Assurance in the Age of DevOps
AI-Driven Software Quality Assurance in the Age of DevOps
 
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
An Empirical Comparison of Model Validation Techniques for Defect Prediction ...
 
Search-based testing of procedural programs:iterative single-target or multi-...
Search-based testing of procedural programs:iterative single-target or multi-...Search-based testing of procedural programs:iterative single-target or multi-...
Search-based testing of procedural programs:iterative single-target or multi-...
 

Similar to SAIConference_PAPER

A mutation testing analysis and regression
A mutation testing analysis and regressionA mutation testing analysis and regression
A mutation testing analysis and regression
ijfcstjournal
 
A MUTATION TESTING ANALYSIS AND REGRESSION TESTING
A MUTATION TESTING ANALYSIS AND REGRESSION TESTINGA MUTATION TESTING ANALYSIS AND REGRESSION TESTING
A MUTATION TESTING ANALYSIS AND REGRESSION TESTING
ijfcstjournal
 
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
Donghwan Shin
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
IJERD Editor
 

Similar to SAIConference_PAPER (20)

DYNAMUT: A MUTATION TESTING TOOL FOR INDUSTRY-LEVEL EMBEDDED SYSTEM APPLICATIONS
DYNAMUT: A MUTATION TESTING TOOL FOR INDUSTRY-LEVEL EMBEDDED SYSTEM APPLICATIONSDYNAMUT: A MUTATION TESTING TOOL FOR INDUSTRY-LEVEL EMBEDDED SYSTEM APPLICATIONS
DYNAMUT: A MUTATION TESTING TOOL FOR INDUSTRY-LEVEL EMBEDDED SYSTEM APPLICATIONS
 
50120140502017
5012014050201750120140502017
50120140502017
 
A mutation testing analysis and regression
A mutation testing analysis and regressionA mutation testing analysis and regression
A mutation testing analysis and regression
 
A MUTATION TESTING ANALYSIS AND REGRESSION TESTING
A MUTATION TESTING ANALYSIS AND REGRESSION TESTINGA MUTATION TESTING ANALYSIS AND REGRESSION TESTING
A MUTATION TESTING ANALYSIS AND REGRESSION TESTING
 
Dc35579583
Dc35579583Dc35579583
Dc35579583
 
Software testing strategy
Software testing strategySoftware testing strategy
Software testing strategy
 
C41041120
C41041120C41041120
C41041120
 
H047054064
H047054064H047054064
H047054064
 
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Cap...
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
 
Testing
TestingTesting
Testing
 
Bd36334337
Bd36334337Bd36334337
Bd36334337
 
JS Fest 2018. Андрей Кучеренко. Мутационное тестирование в JavaScript
JS Fest 2018. Андрей Кучеренко. Мутационное тестирование в JavaScriptJS Fest 2018. Андрей Кучеренко. Мутационное тестирование в JavaScript
JS Fest 2018. Андрей Кучеренко. Мутационное тестирование в JavaScript
 
Effectiveness of test case
Effectiveness of test caseEffectiveness of test case
Effectiveness of test case
 
Thetheoryofsoftwaretesting
ThetheoryofsoftwaretestingThetheoryofsoftwaretesting
Thetheoryofsoftwaretesting
 
Mutation testing
Mutation testingMutation testing
Mutation testing
 
Testing embedded system through optimal mining technique (OMT) based on multi...
Testing embedded system through optimal mining technique (OMT) based on multi...Testing embedded system through optimal mining technique (OMT) based on multi...
Testing embedded system through optimal mining technique (OMT) based on multi...
 
Specification based or black box techniques
Specification based or black box techniques Specification based or black box techniques
Specification based or black box techniques
 
A Complexity Based Regression Test Selection Strategy
A Complexity Based Regression Test Selection StrategyA Complexity Based Regression Test Selection Strategy
A Complexity Based Regression Test Selection Strategy
 
QUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONS
QUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONSQUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONS
QUALITY METRICS OF TEST SUITES IN TESTDRIVEN DESIGNED APPLICATIONS
 

SAIConference_PAPER

  • 1. Science and Information Conference 2015 July 28-30, 2015 | London, UK 1 | P a g e www.conference.thesai.org Analyzing Test Case Quality with Mutation Testing Approach Zainab Nayyar, Nazish Rafique, Nousheen Hashmi, Nadia Rashid, Saba Awan Department of Computer Engineering, College of EME, National University of Sciences and Technology (NUST), H-12, Islamabad, Pakistan zainabnayyar786@gmail.com, nazishrafique16@gmail.com, nsheen25@gmail.com, nadia1.rashid@gmail.com, sabaawan90@gmail.com Abstract— Testing is the most important process for the validation and verification of the industrialized products. Similarly, in the field of software development it counts a lot. Sometimes it becomes difficult to judge the quality of test cases especially in the case of large and complex projects, where not even a single fault is tolerable and can have more consequences. So the solution lies in performing a mutation testing process which is a type of white box testing. In the mutation testing process, the quality of test cases is analyzed by just making mutants of the original code and then by killing them. In this paper, the whole process of mutation testing approach is analyzed and then the process is applied on a piece of code to give the complete step by step description of making and killing the mutants. Keywords— Testing, White box testing, Mutation testing, Mutants, Test cases, Test case quality. I. INTRODUCTION Testing is the process which is carried out to check whether the code is working according to the given specifications or not. Testing involves black box testing and white box testing. White box testing is a structural testing which aims to check the internal structure of the code, logical and semantic errors of the code. Mutation testing technique is a kind of white box testing. Mutation testing is a technique of software testing proposed originally by Hamlet [1]. The mutation testing is the structural method of testing whose objective is to access and improve the test suites competence and also to calculate the total faults that are there in system under testing. Mutation testing aims to assess test case quality by introducing faults in programs to check whether the test cases will highlight them. Further, this process aims to utilize these evaluations to assist in building more competent tests for the original program. Mutation is said to put a little change in the program and these minor changes are aimed to model defects at low level that occur during coding of systems process. Operators of mutation are generally based on theoretical or empirical fault models [2] and numerous experimentation iterations are normally required to build and distill mutation operators set. Mutation Operators have been built for many programming languages, such as C [3], Java [4] [5] and Fortran 77 [6] [7]. Mutant operators help to create alternative program versions under the test and test designs which compose these program mutants to act in a different way as compared to the original program. Measure of quality is how many mutants can be killed by the test set. Mutants on test set should be run and testers should examine live mutants and to kill these mutants to design appropriate test sets. Furthermore, there can be mutants who cannot be killed, i.e. for all tests the program responds in same way; they are called equivalent mutants and they show that the code have semantic errors and is logically weak and not able to identify faults. Discovering these mutants is mostly done manually. If for a given test suit, no test case can possibly differentiate among original version of program and mutant then the mutant remains alive. An effective test case is the one that kills as many mutants as possible. Many researches have been done on mutation testing for the identification of faults in the program or to check the effectiveness of test cases. In order to apply mutation on new technologies, the key is to design efficient and effective mutation operators. . In this paper, we shall analyze mutation testing by applying it on different pieces of code to evaluate the effectiveness and quality of test cases II. RELATED WORK Following are the relevant studies which have been made for applying mutation testing process: In [8], the ability of test cases for the detection of hand seeded faults (semantic errors), automatically generated and real world faults. The mutants were generated on the basis of hand seeded and real time code. It was seen that hand seeded faults were difficult to detect then real world faults. Many researchers had shown that some of the generated mutants were sometimes not needed. The problem addressed in [9] [13] was to reduce the number of mutants by maintaining the test case quality. Higher order mutation technique was discussed which was reduced the mutants to some extent but that did not give the exact and accurate solution so a theoretical framework called dynamic subsumption was used to reduce the mutants. This graphical model was used to visualize the repetition between mutants. Subsumption graphs [9] [13] were used to analyze the relationships between different mutants. Subsumption was used in a way in which a mutant subsumes another if at least one test kills the first and every test that kills the first also kills the second. This was used to remove the redundancy in mutants.
  • 2. Science and Information Conference 2015 July 28-30, 2015 | London, UK 2 | P a g e www.conference.thesai.org As mutation testing is a very costly process so in [10] the emphasis was put on a question that is this mandatory to generate multiple sets of test cases? How cost and effectiveness of result effected by different sets of test cases? Results shown that there could be significant differences for individual subject programs among different test sets chosen for the same adequacy criterion. These differences were observed for both effectiveness (mutation score) and cost (number of tests). This result makes a case for choosing multiple test sets during experimentation. Statement deletion in mutation testing has been introduced from the beginning. This technique helped in reducing the cost of mutation testing. Different approaches have been introduced so far. Lin Deng et al. proposed a technique in which they used a SDL (statement deletion) operator was implemented [11]. It was basically used to delete the statements in the code and make mutants in order to generate tests. This operator was applied to 40 different Java classes. Tests were created to eliminate the non- equivalent SDL mutants. This minimizes the cost of mutation testing hence making it more attractive. In [14], the same technique was used with one OP-mutation in which a single strong operator was used to kill the test cases and reduce costs, but it was applied on C language. The experimentations were made on 39 different programs. Jingyu Hu et al in [12] presented a study of class level mutants. Class level mutant operators change the features related to object oriented programming. Non-equivalent mutants are being killed and it showed that how hard it is to kill individual mutants. This study showed that class level mutants kills by less number of tests as compared to the statement level mutants. In [15], the author proposed the genetic algorithms and co- evolution process to solve two main problems in mutation testing. First problem is the detection of equivalent mutants and the second one is the generation of huge number of mutants. A fitness function was designed to show possible ways to detect equivalent mutants. In the proposed methodology, the idea is to apply genetic algorithm to evolve subsets of mutants from the large number of mutants generated for original program. Further, to evolve subsets of test cases from pool of test cases generated randomly, same technique was applied. And the last step was to co-evolve sets of mutants and sets of test cases. To reduce the cost of mutation testing, statement deletion operator technique is used to delete entire statement from programs. In [16], an innovative mutation operators were introduced which deletes some part of the statements. Three types of operators were defined in the paper. Operator deletion which deletes the arithmetic and relational operators, variable deletion operators which removes variable and constant deletion which deletes constants. The study has shown that with the use of deletion operators we require less number of tests than traditional set of operators. III. MUTATION TESTING PROCESS In mutation testing, mutants are created within the original piece of code and test cases are applied on both the original code and the new mutated code. The output of mutants and the original code is compared; if they are similar then that means that the mutants remain alive and test data is needed to be redesigned. If the out is different, then the mutants in the code will be killed, verifying the good quality and effectiveness of the test cases. As shown in the figure below. Fig. 1. Mutation Testing Process A. Value Mutation In this type of mutation, values of variables or constants are changed to test the behavior of original code and mutant. TABLE I. VALUE MUTATION Original Code: Mutant Int x; Int y; Int sum; Cin>>x; Cin>>y; Sum= x+ y; Sum++; Cout<<sum; Int x; Int y; Int sum; Cin>>x; Cin>>y; Sum= x+ y; Sum--; Cout<<sum; Test case 1: x=5, y=9; TABLE II. OUTPUT Value of sum =15 Value of sum =13 In the above code different values are assigned to the variables i.e. x=5 and y=15. The above code simply adds two values but
  • 3. Science and Information Conference 2015 July 28-30, 2015 | London, UK 3 | P a g e www.conference.thesai.org in the original code the values are incrementing but to check whether the code is working correctly and identifying hand seeded faults or not the increment operators are changed to decrement operators in the mutant. In the output it is shown that the output of both the codes is not same due to the change of increment and decrement operators and the test case is able to detect faults in the code. B. Statement Mutation In this type, the output of original code and mutants are compared by deleting or swapping the entire statements. TABLE III. STATEMENT MUTATION EXAMPLE 1 Original Code: Mutant Int x; Int y; Cin>>x; Cin>>y; If(x>y) { Cout<< ”x is greater”; } Else if(y>x) { Cout<< ”y is greater”; } Else { Cout<<“x and y are equal”; } Int x; Int y; Cin>>x; Cin>>y; If(x>y) { Cout<< ”x is greater”; } //Else if(y>x) // { //Cout<< ”y is greater”; //} Else { Cout<<“x and y are equal”; } Test case 1: x=5, y=9; TABLE IV. OUTPUT I y is greater Program terminate TABLE V. STATEMENT MUTATION EXAMPLE 2 Original Code: Mutant Public void testif () { Int a, b, c , t; If (a<5) { T= t+ b+ c; } Else if (a>20) { t= t + a +c; b++; }} Public void testif () { Int a, b, c , t; If (a<5) { T= t+ b+ c; } Else if (a>20) { t= t + a +c; b++; b++; }} Test case 1: A= 7; b= 1, c=2, t=3; TABLE VI. OUTPUT 2 T=12 B=2 T=12 B=3 C. Decision Mutation In this type, conditional statements are modified like > replaces the < sign. TABLE VII. DECISION MUTATION EXAMPLE Original Code: Mutant Public void testwhile () { Int a, b, c, t; While (a<5) { t= t + b + c; a++; } } Public void testwhile () { Int a, b, c, t; While (true) { t= t + b + c; a++; } } Test case 1: a= 4, b= 3, c= 2, t=1; Test case 2: a=5, b= 4, c=2, t=1; TABLE VIII. OUTPUT When a>5 T=6; A=5; When a>5 Infinite Loop IV. CONCLUSION Mutation testing is helpful for verification and validation of critical requirements without which system cannot act/operate properly. The mutation testing applied on small piece of codes in this paper, illustrates the different types of mutation operators that can be applied to identify the accuracy of the test case. It not only ensures the good quality of test cases but also helps to detect faults of programs. Through mutation testing, effective test cases can be made. Though, it is considered as a costly process but one time cost is better than the system failure after deployment
  • 4. Science and Information Conference 2015 July 28-30, 2015 | London, UK 4 | P a g e www.conference.thesai.org REFERENCES [1] Hamlet, R.G. Testing programs with the aid of a compiler. Pages 279- 290. IEEE Transactions on Software Engineering.. [2] J. Offutt, R. Alexander, Y. Wu, Q. Xiao, and C. Hutchinson. A fault model for subtype inheritance and polymorphism. Pages 84–93. 12th International Symposium on Software Reliability Engineering. November 2001. [3] M. E. Delamaro and J. C. Maldonado. Proteum-A tool for the assessment of test adequacy for C programs. Pages 79–95. Performability in Computing Systems. July 1996. [4] S. Kim, J. A. Clark, and J. A. Mc Dermid. Investigating the effectiveness of object-oriented strategies with the mutation method. Pages 4–100. Mutation Testing in the Twentieth and the Twenty First Centuries. October 2000. December 2001. [5] Y.-S. Ma, J. Offutt, and Y.-R. Kwon. MuJava : An automated class mutation system. Volume 15, no. 2, pages 97–133. Software Testing, Verification, and Reliability. June 2005. [6] R. A. De Millo and J. Offutt. Constraint-based automatic test data generation. Volume 17, no. 9, pages 900–910. IEEE Transactions on Software Engineering. September 1991. [7] K. N. King and J. Offutt. A Fortran language system for mutation-based software testing. Volume 21, no. 7, pages 685–718. Software-Practice and Experience. July 1991. [8] J. H. Andrews, L. C. Briand, Y. Labiche. Is Mutation an Appropriate Tool for Testing Experiments? pages 402-411. IEEE International Conference on Software Engineering (ICSE). 2005. [9] Paul Ammann, Marcio E. Delamaro and Jeff Offutt. Establishing Theoretical Minimal Sets of Mutants. 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014). March 2014. [10] Marcio E. Delamaro and Jeff Offutt. Assessing the Influence of Multiple Test Case Selection on Mutation Experiments. Tenth Workshop on Mutation Analysis (Mutation 2014). March 2014. [11] .Lin Deng, Jeff Offutt and Nan Li, Empirical Evaluation of the Statement Deletion Mutation Operator, Software Engineering. 6th IEEE International Conference on Software Testing, Verification and Validation (ICST). March 2011 [12]. Jingyu Hu, Nan Li and Jeff Offutt. An Analysis of OO Mutation Operators,Software Engineering. IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). Pages 334-341. 21-25 March 2011. [13]. Bob Kurtzy, Paul Ammanny, Marcio E. Delamaro, Jeff Offutty and Lin Dengy. Mutant Subsumption Graphs. Tenth Workshop on Mutation Analysis. March 2014. [14]. Marcio E. Delamaro, Lin Deng, Vinicius H. S. Durelli, Nan Li and Jeff Offutty. Experimental Evaluation of SDL and One-Op Mutation for C. IEEE 7th International Conference on Software Testing, Verification and Validation. April 2014. [15]. Konstantinos Adamopoulos, Mark Harman and Robert M. Hierons,How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution [16]. Marcio E. Delamaro, Jeff Offutt, Paul Ammann, Designing Deletion Mutation Operators