Testing throughout the software life cycle 2 hadnan
MuFinal
1. MUJAVA: AN
AUTOMATED CLASS
MUTATION SYSTEM
Y. S. Ma, J. Offutt and Y. R.
Kwon
Journal of Software Testing,
Verification and Reliability, 2005
André Policarpo,
Carina Antunes, 73161
Diogo Nicolau, 72935
1
2. OUTLINE
• Introduction
• State Of The Art
• Mutation Testing
• MuJava
• Results
• Conclusion
Keywords : Object-Oriented Programs, Mutation Testing,
Software Testing
2
4. INTRODUCTION
• Software testing is used to produce highly
reliable systems
• Tests can be created to verify the correctness
of the implementation of a given software
system, but the creation of tests still poses the
question whether the tests are correct and
sufficiently cover the requirements that have
originated the implementation.
5. • Mutation testing (or Mutation
analysis or Program mutation) is used to design
new software tests and evaluate the quality of
existing software tests.
• The purpose is to help the tester develop
effective tests or locate weaknesses in the test
data used for the program or in sections of the
code that are seldom or never accessed during
execution.
Do you
have an
objective
to hit ?
5
6. MUTATION TESTING
OVERVIEW
• Simple faults are introduced into the program by
creating a set of faulty versions, called mutants.
• Test cases are used to execute the mutants with
the goal of causing the each mutant to produce
incorrect output.
• A test case that is able to detect the change (i.e.
one of the tests fails), and distinguish the program
from one or more mutants is considered to be
effective at finding faults in the program and the
mutant is said to be killed.
6
7. 7
• These faults are
introduced into the
program by
applying mutation
operators, which
describe syntactic
changes to the
programming
language.
Original
progra
m
Mutant
Mutant
operato
r
8. • Object-oriented software testing poses
additional problems due to key OO features such
as inheritance, encapsulation and
polymorphism.
• Testing efforts for OO software found to be
increased compared to testing procedural
software.
• A major difference for testers is that OO
software changes the levels at which testing is
performed. In OO software, unit and integration
level testing can be classified into four levels:
intra-method, inter-method, intra-class, inter-
class (definitions by Harrold and Rothermel and
Gallagher and Offutt).
8
9. Intra-method level:
• Faults occur when the functionality of a
implemented incorrectly.
• Testing within classes corresponds to unit
in conventional programs.
• Researches have assumed that traditional
mutation operators for procedural programs
suffice for this level.
Inter-method level:
• Faults are made on the connections between
pairs of methods of a single class.
• Testing at this level is equivalent to
testing of procedural language programs.
• Interface mutation, which evaluates how well
interactions between various units have been
tested, is applicable to this level.
9
10. Intra-class level:
• Tests are constructed for a single class, with
purpose of testing the class as a whole.
• It is a specialization of the traditional unit
module testing.
• It tests the interactions of public methods of
class when they are called in various
Inter-class level:
• More than one class is tested in
look for faults in how they are integrated.
• It specializes in the traditional integration
and subsystem testing.
• Most faults related to polymorphism,
and access are found.
10
12. STATE OF THE ART
• Additional Mutation operators that handle new types
of faults introduced by OO-specific features were
needed.
• It is time consuming technique, hence requires
automated tools.
• Several approaches have been developed to
reduce the computational expense of the mutation
testing. Three strategies, do fewer, do smarter, and
do faster.
12
13. 13
do fewer: run fewer mutant programs
without incurring intolerable loss in
effectiveness.
do smarter: distribute the computational
expense over several machines or factor the
expense over several executions by
retaining state information between runs.
do faster: generate and run mutant
programs as quickly as possible.
14. 14
• This paper primarily focuses on studying
the inter-class or object-oriented operators.
• It presents a do faster method for OO inter-
class mutation testing : proposes a solution
focused on performance, to generate
mutants faster since previous tools where
very time-consuming.
16. MUTATION TESTING -
OPERATORS
• The authors developed a set of 24 mutation
operators for Java.
• Mutation operators are only applied in situations
where the mutated program will still compile.
16
17. MUTATION TESTING -
OPERATORS
• The first 4 groups are based on language features
that are common to all OO languages: Information
Hiding (Access Control), Inheritance,
Polymorphism, Overloading.
• The 5th group includes languages features that are
Java-specific and the last group of mutation
operators are based on common OO programming
mistakes.
17
18. 18
Information Hiding (Access Control)
• Common source of mistakes among OO
programmers: Semantic of various access levels
poorly understood.
• Access variables and methods are not always
considered during design.
19. 19
Information Hiding (Access Control) (cont.)
• The purpose of the AMC operator is to guide
testers to generate test cases that ensure that
accessibility is correct.
20. 20
Inheritance
• Seven mutation operators have been defined to
the various aspects of using inheritance,
variable hiding, method overriding, the use of
and definition of constructors.
24. 24
Polymorphism
• Polymorphism allows the behaviour of an
reference to be different depending the actual
Therefore, it is important to identify and
program with all possible type bindings.
25. 25
Polymorphism (cont.)
• PMD – Member variable declaration with parent
type
• PRV – Reference assignment with other
type
26. 26
Overloading
• Method overloading allows 2 or more
the same class or type family to have the same
name as long as they have different argument
signatures.
28. 28
Java-Specific Features
• Some object-oriented features are not
to all object-oriented languages. This group
operators attempt to ensure correct use of
features supported in Java.
33. 33
Mutation Testing : outcomes
• Killed : mutant is killed if a test fails
the mutated code)
• Lived : A mutant didn’t trigger a failing
remains live because it is equivalent to the
original program i.e. it is functionally
the original program or the test data is
inadequate to kill the mutant.
34. 34
Mutation Testing : outcomes
• There may be surviving mutants that cannot be
killed, these are called Equivalent Mutants,
syntactically different they are indistinguishable
testing.
36. 36
Mutation Testing: process
• To make the test suit effective (kill the
mutants), new test cases should be added
to detect the remaining non-equivalent
live mutants.
𝑇𝑒𝑠𝑡 𝑠𝑢𝑖𝑡 𝑎𝑑𝑒𝑞𝑢𝑎𝑐𝑦 =
(𝑁𝑟 𝑘𝑖𝑙𝑙𝑒𝑑 𝑚𝑢𝑡𝑎𝑛𝑡𝑠) ÷ (𝑇𝑜𝑡𝑎𝑙 𝑁𝑟 𝑛𝑜𝑛 −
𝑒𝑞𝑢𝑖𝑣𝑎𝑙𝑒𝑛𝑡 𝑚𝑢𝑡𝑎𝑛𝑡𝑠)
38. MuJAVA
38
• MuJava is a automated mutation system for
Java programs. It automatically generates
mutants for mutation testing.
• Tests are supplied by the users as sequences
of method calls to the classes under test that
must return a string that is used to compare
the original result against the mutants result.
41. MuJAVA
41
• MuJava distinguishes the generation of
behavioral mutants (traditional, non-OO) and
structural mutants (change the structure of the
program).
• Mujava uses Mutant Schemata Generator (MSG) to
generate one “meta-mutant” program at the source
level that incorporates many mutants for
behavioral mutants.
• Bytecode translation is used for class mutation
operators that change the structure of the
program, so that no additional compilation is
required.
44. 44
Example of mutation for structural mutant
• Using bytecode transformation
• The IOD mutation operator deletes overriding
methods. It generates a mutant by deleting temp() in
Child.
47. 47
Mutation Process For Behavioral Mutant
• Uses MSG + reflection technique
• Reflection is a natural way to implement
mutation analysis for several reasons:
1. It provides an API to easily change the
behavior of a program during execution.
2. It lets programmers extract OO-related
Information about a class by providing an
object.
Reflection is the ability of a program to observe and
possibly modify its high level structure.
48. 48
Example of mutation for behavioral mutant
• A unique ID is assigned to each method and to
each statement of the program.
• Each mutant is then associated with a MethodID
and StatementID.
Only when a
method whose
methodID is 3 is
executed with a
mutant whose
methodID is 3 and
the statementID is
8, the
metamutated form
of that statement
is executed.
53. 53
BCEL – Byte Code Engineering Library
• 264 Java classes (without abstract and interface
classes)
• 3,812 mutants
• Average of 14.44 mutants per class
• 1.53 times more behavioral than structural
mutants
55. 55
MSG vs Separate Compilation
Speedup = Compilation Time / MSG
method Time
• Overhead in both MSG and Separate
Compilation method
• Both use JVM’s class loaders
• But MSG only needs to load the metamutant
56. 56
Bytecode Translation vs Separate Compilation
Speedup = Compilation Time / Bytecode
Translation Time
• High increase in mutant generation
performance
• Same performance in execution
• Bytecode Translation – memory.
• Compilation – source code, more disk accesses.
57. 57
Results Overview
• MSG Method:
• Average speedup of 6.79
• Only one compilation (metamutant)
• Only one class load (metaclass)
• More efficient than bytecode translation
• Only usable for behavioral mutants
• Bytecode Translation:
• Average speedup of 4.26
• But only speeds up mutant generation
• Usable for both behavioral and structural mutants
59. CONCLUSION
• Mutation testing is a powerful technique
for the assessment and enhancement of
tests.
• The effectiveness of mutation testing
depends heavily on the types of faults that
the mutation operators are designed to
represent. Therefore, the quality of the
mutation operators is key to mutation
testing.
• MuJava allows the tester to enter and run
tests, and evaluates the mutation coverage 59
Editor's Notes
Although a powerful and useful abstraction mechanism, incorrect use of inheritance can lead to a number of faults. We define five mutation operators to try to test the various aspects of using inheritance, covering variable shadowing, method overriding, the use of super, and definition of constructors. Variable shadowing can cause instance variables that are defined in a subclass to shadow (or hide) member variables of the parent. However, this powerful feature can cause an incorrect variable to be accessed. Thus it is necessary to ensure that the correct variable is accessed when variable shadowing is used, which is the intent of the IHD and IHI mutation operators.
• IHD – Hiding variable deletion: The IHD operator deletes a hiding variable, a variable in a subclass that has the same name and type as a variable in the parent class. This causes references to that variable to access the variable defined in the parent (or ancestor). This mutant can only be killed by a test case that is able to show that the reference to the parent variable is incorrect.
• IHI – Hiding variable insertion: The IHI operator inserts a hiding variable into a subclass. It is a reverse case of IHD. By inserting a hiding variable, two variables (a hiding variable and a hidden variable) of the same name become to be exist. Newly defined and overriding methods in a subclass reference the hiding variable although inherited methods reference the hidden variable as before.
• IOD – Overriding method deletion: The IOD operator deletes an entire declaration of an overriding method in a subclass so that references to the method uses the parent’s version. The mutant act as if there is no overriding method for the method.
• IOP – Overridden method calling position change: Sometimes, an overriding method in a child class needs to call the method it overrides in the parent class. This may happen if the parent’s method uses a private variable v, which means the method in the child class may not modify v directly. However, an easy mistake to make is to call the parent’s version at the wrong time, which can cause incorrect state behavior. The IOP operator moves calls to overridden methods to the first and last statements of the method and up and down one statement.
Object references can have different types with different executions. That is, object references may refer to objects whose actual types differ from their declared types. The actual type can be from any type that is a subclass of the declared type. Polymorphism allows the behavior of an object reference to be different depending the actual type. Therefore, it is important to identify and exercise the program with all possible type bindings. The polymorphism mutation operators are designed to ensure this type of testing.
• PMD – Member variable declaration with parent class type: The PMD operator changes the declared type of an object reference to the parent of the original declared type. The instantiation will still be valid (it will still be a descendant of the new declared type). To kill this mutant, a test case must cause the behavior of the object to be incorrect with the new declared type. In the example below, class Parent is the parent of class Child.
• PRV – Reference assignment with other compatible type: Object references can refer to objects of types that are descendants of its declared type. The PRV operator changes operands of a reference assignment to be assigned to objects of subclasses. In the example below, obj is of type Object, and in the original code it is given an object of type String. In the mutated code, it is given an object of type Integer
Method overloading allows two or more methods of the same class to have the same name as long as they have different argument signatures. Just as with method overriding, it is important for testers to ensure that a method invocation invokes the correct method with appropriate parameters. Three mutation operators are defined to test various aspects of method overloading.
• OMR – Overloading method contents change: The OMR operator is designed to check that overloaded methods are invoked appropriately. The OMR operator replaces the body of a method with the body of another method that has the same name. This is accomplished by using the keyword this.
• OMD – Overloading method deletion: The OMD operator deletes overloading method declarations, one at a time in turn. If the mutant still works correctly without the deleted method, there may be an error in invoking one of the overloading methods; the incorrect method may be invoked or an incorrect parameter type conversion has occurred. The POD operator ensures coverage of overloaded methods, that is, all the overloaded methods must be invoked at least once.
Some object-oriented features are not common to all object-oriented languages. This group of operators attempt to ensure correct use of such features supported in Java. Also, mistakes that a programmer often makes when writing object-oriented software are considered here.
• JTD – this keyword deletion: The JTD operator deletes uses of the keyword this. It models a reverse case of JTI.
• JDC – Java-supported default constructor deletion: Java creates default constructors if a class contains no constructors. The JDC operator forces Java to create a default constructor by deleting the implemented default constructor. It is designed to check if the user-defined default constructor is implemented properly.
• EOA – Reference assignment and content assignment replacement: Object references in Java are always through pointers. Although pointers in Java are typed, which is considered to help prevent certain types of faults, there are still mistakes that programmers can make. One common mistake is that of using an object reference instead of the contents of the object the pointer references. The EOA operator replaces an assignment of a pointer reference with a copy of the object, using the Java convention of a clone() method. The clone() method duplicates the contents of an object, creating and returning a reference to a new object.
• EAM – Accessor method change: The EAM operator changes an accessor method name for other compatible accessor method names, where compatible means that the signatures are the same. This type of mistake occurs because classes with multiple instance variables may wind up having many accessor methods with the same signature and very similar names. As a result, programmers easily get them confused. To kill this mutant a test case will have to produce incorrect output as a result of calling the wrong method.
• EMM – Modifier method change: The EMM operator does the same as EMM, except it works with modifier methods instead of accessor methods.
