MuJava is an automated mutation testing tool for Java programs that generates mutants faster than previous tools. It uses two techniques - metamutant generation and bytecode translation - to quickly generate behavioral and structural mutants, respectively. Metamutant generation creates a single "meta-mutant" program incorporating many mutants to test method behavior, while bytecode translation directly modifies class files to test structural changes. The paper presents the mutation operators and processes used by MuJava and demonstrates its speed improvements over separate compilation, with an average speedup of 6.79 times for metamutants.
Synthesizing Knowledge from Software Development ArtifactsJeongwhan Choi
The content was created from "The Art and Science of Analyzing Software Data"
O Baysal, Kononenko, O. (Oleksii), Holmes, R. (Reid), and Godfrey, M.W. (Michael W.), “Synthesizing Knowledge from Software Development Artifacts”, 2015.
EXPERIMENTAL EVALUATION AND RESULT DISCUSSION OF METAMORPHIC TESTING AUTOMATI...IAEME Publication
Metamorphic Testing is an attribute relations based testing, used to mitigate the test oracle problem in testing complex non-testable programs. MTAF stands for Metamorphic Testing Automation Framework, introduced to eliminate the human intervention in creating test cases, mapping the relations, executing the statements and identifying the errors from input programs. MTAF is especially designed to address the test oracle problem of two most popular non-testable program domains are Multi Precision Arithmetic (MPA) and Graph Theory (GT) applications. In this paper, the researcher explains the results of conducted experiments and identified bug information with MTAF. Several Multi Precision Arithmetic and Graph Theory related hidden bugs are discussed in this paper to show the performance of MTAF.
Unit 3 Control Flow Testing contains Concept of CFT, Generate Test Input Data, Activities of Generating Test Input Data, Control Flow Graph, Path Selection Criteria, Techniques for Path Selection statement wise, branch wise, predicate wise etc. and Generating Test Input.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Learning Analytics: What it is, where we are, and where we could goDoug Clow
Talk given at the Computers and Learning Research Group (CALRG) annual conference, 12 June 2013, at The Open University, UK.
This presentation briefly reviews learning analytics, using some key examples. It then assesses what the OU is doing, and then sets out some ideas for what the OU could do in future to harness the potential of data about our learners to improve their learning.
Synthesizing Knowledge from Software Development ArtifactsJeongwhan Choi
The content was created from "The Art and Science of Analyzing Software Data"
O Baysal, Kononenko, O. (Oleksii), Holmes, R. (Reid), and Godfrey, M.W. (Michael W.), “Synthesizing Knowledge from Software Development Artifacts”, 2015.
EXPERIMENTAL EVALUATION AND RESULT DISCUSSION OF METAMORPHIC TESTING AUTOMATI...IAEME Publication
Metamorphic Testing is an attribute relations based testing, used to mitigate the test oracle problem in testing complex non-testable programs. MTAF stands for Metamorphic Testing Automation Framework, introduced to eliminate the human intervention in creating test cases, mapping the relations, executing the statements and identifying the errors from input programs. MTAF is especially designed to address the test oracle problem of two most popular non-testable program domains are Multi Precision Arithmetic (MPA) and Graph Theory (GT) applications. In this paper, the researcher explains the results of conducted experiments and identified bug information with MTAF. Several Multi Precision Arithmetic and Graph Theory related hidden bugs are discussed in this paper to show the performance of MTAF.
Unit 3 Control Flow Testing contains Concept of CFT, Generate Test Input Data, Activities of Generating Test Input Data, Control Flow Graph, Path Selection Criteria, Techniques for Path Selection statement wise, branch wise, predicate wise etc. and Generating Test Input.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Learning Analytics: What it is, where we are, and where we could goDoug Clow
Talk given at the Computers and Learning Research Group (CALRG) annual conference, 12 June 2013, at The Open University, UK.
This presentation briefly reviews learning analytics, using some key examples. It then assesses what the OU is doing, and then sets out some ideas for what the OU could do in future to harness the potential of data about our learners to improve their learning.
What is Learning Analytics about? A Survey of Different Methods Used in 2013-...Mohammad Khalil
What is Learning Analytics about? A Survey of Different Methods Used in 2013-2015
in Innovation Arabia 9 Conference in Dubai, UAE
Full Publication: https://www.researchgate.net/publication/298068696_What_is_Learning_Analytics_about_A_Survey_of_Different_Methods_Used_in_2013-2015
Full Citation:
Khalil, M. & Ebner, M. (2016). “What is Learning Analytics about? A Survey of Different Methods Used in 2013-2015”. In Proceedings of Smart Learning Conference, Dubai, UAE, 7-9 March, 2016 (pp. 294-304). Dubai: HBMSU Publishing House
Lessons learnt at building recommendation services at industry scaleDomonkos Tikk
Industry day keynote presentation held at ECIR 2016, Padova. The talk presents algorithmic, technical and business challenges Gravity R&D encountered from building a recommender system vendor company from being a top Netflix Prize contender.
about mutation testing and demonstration of muJava. muJava is automated tool for mutation testing of java programs. It tests the test cases. hence good to enhance and checking effectiveness of your test suites.
Strategic Approach to Software Testing, Strategic Issues, Test Conventional Software, Test Strategies for Object-Oriented Software, Test Strategies for WebApps, Validation Testing, System Testing, The Art of Debugging, Software Testing Fundamentals, White-Box Testing, Basis Path Testing,
Control Structure Testing
How to Make the Most of Regression and Unit Testing.pdfAbhay Kumar
Understanding the differences between regression testing and unit testing is paramount for maintaining the robustness of any software. Although both serve the purpose of vulnerability scanning to detect threats, they vary in terms of their test objectives, stages in the development process, and the scope of the code they cover. Let's delve into these distinctions to grasp their roles better.
Difference in Test Objectives:
- While similar, Unit and regression testing have distinct targets within your code. Unit testing, conducted by programmers, assesses individual components, validating the correct functionality of each variable, function, and object.
- On the other hand, regression testing (often termed QA testing) occurs after programmers complete work on specific features. It acts as a system-wide check, ensuring untouched components function as expected. While unit tests provide the precision of individual functions and variables, regression tests collaborate to ascertain that the entire system functions optimally.
Difference in the Development Phase:
- The timing of unit and regression tests sets them apart. Unit tests are conducted during the development phase, where developers run them after implementing changes to confirm no adverse impacts.
- Conversely, regression testing is performed before the feature's production release. It comprises unit tests, integration tests, and various other testing types. Testers are responsible for executing regression testing. Automated regression testing, a key step in continuous integration/continuous delivery, quickly detects if recent code changes have disrupted the existing code.
Difference in Code Coverage:
- A unit test concentrates on a single unit, method, or function, examining one element at a time. It doesn't account for how these units interact, necessitating integration tests. This approach provides swift feedback due to its focused testing nature.
- In contrast, regression tests validate if alterations to existing functionalities have affected other parts of the system by testing against predefined scenarios, ensuring correct integration of units. Given the comprehensive testing involved, it generally consumes more time.
Unit and regression testing are vital pillars in the software development journey. Regular execution of these tests is key to minimizing bugs and refining code quality.
Regression testing conducted post-unit testing before a software release, ensures system integrity despite changes. On the other hand, unit testing meticulously validates new functions, ensuring precise code execution.
This lecture is about the detail definition of software quality and quality assurance. Provide details about software tesing and its types. Clear the basic concepts of software quality and software testing.
The development process adopted for a project will depend on the project's aims and goals.
The models specify the various stages of the process and the order in which they are carried out.
The way testing is organized must fit the development life cycle or it will fail to deliver its benefit.
This ppt covers the following
A strategic approach to testing
Test strategies for conventional software
Test strategies for object-oriented software
Validation testing
System testing
The art of debugging
A strategy for software testing integrates the design of software test cases into a well-planned series of steps that result in successful development of the software.
1. MUJAVA: AN
AUTOMATED CLASS
MUTATION SYSTEM
Y. S. Ma, J. Offutt and Y. R.
Kwon
Journal of Software Testing,
Verification and Reliability, 2005
André Policarpo,
Carina Antunes, 73161
Diogo Nicolau, 72935
1
2. OUTLINE
• Introduction
• State Of The Art
• Mutation Testing
• MuJava
• Results
• Conclusion
Keywords : Object-Oriented Programs, Mutation Testing,
Software Testing
2
4. INTRODUCTION
• Software testing is used to produce highly
reliable systems
• Tests can be created to verify the correctness
of the implementation of a given software
system, but the creation of tests still poses the
question whether the tests are correct and
sufficiently cover the requirements that have
originated the implementation.
5. • Mutation testing (or Mutation
analysis or Program mutation) is used to design
new software tests and evaluate the quality of
existing software tests.
• The purpose is to help the tester develop
effective tests or locate weaknesses in the test
data used for the program or in sections of the
code that are seldom or never accessed during
execution.
Do you
have an
objective
to hit ?
5
6. MUTATION TESTING
OVERVIEW
• Simple faults are introduced into the program by
creating a set of faulty versions, called mutants.
• Test cases are used to execute the mutants with
the goal of causing the each mutant to produce
incorrect output.
• A test case that is able to detect the change (i.e.
one of the tests fails), and distinguish the program
from one or more mutants is considered to be
effective at finding faults in the program and the
mutant is said to be killed.
6
7. 7
• These faults are
introduced into the
program by
applying mutation
operators, which
describe syntactic
changes to the
programming
language.
Original
progra
m
Mutant
Mutant
operato
r
8. • Object-oriented software testing poses
additional problems due to key OO features such
as inheritance, encapsulation and
polymorphism.
• Testing efforts for OO software found to be
increased compared to testing procedural
software.
• A major difference for testers is that OO
software changes the levels at which testing is
performed. In OO software, unit and integration
level testing can be classified into four levels:
intra-method, inter-method, intra-class, inter-
class (definitions by Harrold and Rothermel and
Gallagher and Offutt).
8
9. Intra-method level:
• Faults occur when the functionality of a
implemented incorrectly.
• Testing within classes corresponds to unit
in conventional programs.
• Researches have assumed that traditional
mutation operators for procedural programs
suffice for this level.
Inter-method level:
• Faults are made on the connections between
pairs of methods of a single class.
• Testing at this level is equivalent to
testing of procedural language programs.
• Interface mutation, which evaluates how well
interactions between various units have been
tested, is applicable to this level.
9
10. Intra-class level:
• Tests are constructed for a single class, with
purpose of testing the class as a whole.
• It is a specialization of the traditional unit
module testing.
• It tests the interactions of public methods of
class when they are called in various
Inter-class level:
• More than one class is tested in
look for faults in how they are integrated.
• It specializes in the traditional integration
and subsystem testing.
• Most faults related to polymorphism,
and access are found.
10
12. STATE OF THE ART
• Additional Mutation operators that handle new types
of faults introduced by OO-specific features were
needed.
• It is time consuming technique, hence requires
automated tools.
• Several approaches have been developed to
reduce the computational expense of the mutation
testing. Three strategies, do fewer, do smarter, and
do faster.
12
13. 13
do fewer: run fewer mutant programs
without incurring intolerable loss in
effectiveness.
do smarter: distribute the computational
expense over several machines or factor the
expense over several executions by
retaining state information between runs.
do faster: generate and run mutant
programs as quickly as possible.
14. 14
• This paper primarily focuses on studying
the inter-class or object-oriented operators.
• It presents a do faster method for OO inter-
class mutation testing : proposes a solution
focused on performance, to generate
mutants faster since previous tools where
very time-consuming.
16. MUTATION TESTING -
OPERATORS
• The authors developed a set of 24 mutation
operators for Java.
• Mutation operators are only applied in situations
where the mutated program will still compile.
16
17. MUTATION TESTING -
OPERATORS
• The first 4 groups are based on language features
that are common to all OO languages: Information
Hiding (Access Control), Inheritance,
Polymorphism, Overloading.
• The 5th group includes languages features that are
Java-specific and the last group of mutation
operators are based on common OO programming
mistakes.
17
18. 18
Information Hiding (Access Control)
• Common source of mistakes among OO
programmers: Semantic of various access levels
poorly understood.
• Access variables and methods are not always
considered during design.
19. 19
Information Hiding (Access Control) (cont.)
• The purpose of the AMC operator is to guide
testers to generate test cases that ensure that
accessibility is correct.
20. 20
Inheritance
• Seven mutation operators have been defined to
the various aspects of using inheritance,
variable hiding, method overriding, the use of
and definition of constructors.
24. 24
Polymorphism
• Polymorphism allows the behaviour of an
reference to be different depending the actual
Therefore, it is important to identify and
program with all possible type bindings.
25. 25
Polymorphism (cont.)
• PMD – Member variable declaration with parent
type
• PRV – Reference assignment with other
type
26. 26
Overloading
• Method overloading allows 2 or more
the same class or type family to have the same
name as long as they have different argument
signatures.
28. 28
Java-Specific Features
• Some object-oriented features are not
to all object-oriented languages. This group
operators attempt to ensure correct use of
features supported in Java.
33. 33
Mutation Testing : outcomes
• Killed : mutant is killed if a test fails
the mutated code)
• Lived : A mutant didn’t trigger a failing
remains live because it is equivalent to the
original program i.e. it is functionally
the original program or the test data is
inadequate to kill the mutant.
34. 34
Mutation Testing : outcomes
• There may be surviving mutants that cannot be
killed, these are called Equivalent Mutants,
syntactically different they are indistinguishable
testing.
36. 36
Mutation Testing: process
• To make the test suit effective (kill the
mutants), new test cases should be added
to detect the remaining non-equivalent
live mutants.
𝑇𝑒𝑠𝑡 𝑠𝑢𝑖𝑡 𝑎𝑑𝑒𝑞𝑢𝑎𝑐𝑦 =
(𝑁𝑟 𝑘𝑖𝑙𝑙𝑒𝑑 𝑚𝑢𝑡𝑎𝑛𝑡𝑠) ÷ (𝑇𝑜𝑡𝑎𝑙 𝑁𝑟 𝑛𝑜𝑛 −
𝑒𝑞𝑢𝑖𝑣𝑎𝑙𝑒𝑛𝑡 𝑚𝑢𝑡𝑎𝑛𝑡𝑠)
38. MuJAVA
38
• MuJava is a automated mutation system for
Java programs. It automatically generates
mutants for mutation testing.
• Tests are supplied by the users as sequences
of method calls to the classes under test that
must return a string that is used to compare
the original result against the mutants result.
41. MuJAVA
41
• MuJava distinguishes the generation of
behavioral mutants (traditional, non-OO) and
structural mutants (change the structure of the
program).
• Mujava uses Mutant Schemata Generator (MSG) to
generate one “meta-mutant” program at the source
level that incorporates many mutants for
behavioral mutants.
• Bytecode translation is used for class mutation
operators that change the structure of the
program, so that no additional compilation is
required.
44. 44
Example of mutation for structural mutant
• Using bytecode transformation
• The IOD mutation operator deletes overriding
methods. It generates a mutant by deleting temp() in
Child.
47. 47
Mutation Process For Behavioral Mutant
• Uses MSG + reflection technique
• Reflection is a natural way to implement
mutation analysis for several reasons:
1. It provides an API to easily change the
behavior of a program during execution.
2. It lets programmers extract OO-related
Information about a class by providing an
object.
Reflection is the ability of a program to observe and
possibly modify its high level structure.
48. 48
Example of mutation for behavioral mutant
• A unique ID is assigned to each method and to
each statement of the program.
• Each mutant is then associated with a MethodID
and StatementID.
Only when a
method whose
methodID is 3 is
executed with a
mutant whose
methodID is 3 and
the statementID is
8, the
metamutated form
of that statement
is executed.
53. 53
BCEL – Byte Code Engineering Library
• 264 Java classes (without abstract and interface
classes)
• 3,812 mutants
• Average of 14.44 mutants per class
• 1.53 times more behavioral than structural
mutants
55. 55
MSG vs Separate Compilation
Speedup = Compilation Time / MSG
method Time
• Overhead in both MSG and Separate
Compilation method
• Both use JVM’s class loaders
• But MSG only needs to load the metamutant
56. 56
Bytecode Translation vs Separate Compilation
Speedup = Compilation Time / Bytecode
Translation Time
• High increase in mutant generation
performance
• Same performance in execution
• Bytecode Translation – memory.
• Compilation – source code, more disk accesses.
57. 57
Results Overview
• MSG Method:
• Average speedup of 6.79
• Only one compilation (metamutant)
• Only one class load (metaclass)
• More efficient than bytecode translation
• Only usable for behavioral mutants
• Bytecode Translation:
• Average speedup of 4.26
• But only speeds up mutant generation
• Usable for both behavioral and structural mutants
59. CONCLUSION
• Mutation testing is a powerful technique
for the assessment and enhancement of
tests.
• The effectiveness of mutation testing
depends heavily on the types of faults that
the mutation operators are designed to
represent. Therefore, the quality of the
mutation operators is key to mutation
testing.
• MuJava allows the tester to enter and run
tests, and evaluates the mutation coverage 59
Editor's Notes
Although a powerful and useful abstraction mechanism, incorrect use of inheritance can lead to a number of faults. We define five mutation operators to try to test the various aspects of using inheritance, covering variable shadowing, method overriding, the use of super, and definition of constructors. Variable shadowing can cause instance variables that are defined in a subclass to shadow (or hide) member variables of the parent. However, this powerful feature can cause an incorrect variable to be accessed. Thus it is necessary to ensure that the correct variable is accessed when variable shadowing is used, which is the intent of the IHD and IHI mutation operators.
• IHD – Hiding variable deletion: The IHD operator deletes a hiding variable, a variable in a subclass that has the same name and type as a variable in the parent class. This causes references to that variable to access the variable defined in the parent (or ancestor). This mutant can only be killed by a test case that is able to show that the reference to the parent variable is incorrect.
• IHI – Hiding variable insertion: The IHI operator inserts a hiding variable into a subclass. It is a reverse case of IHD. By inserting a hiding variable, two variables (a hiding variable and a hidden variable) of the same name become to be exist. Newly defined and overriding methods in a subclass reference the hiding variable although inherited methods reference the hidden variable as before.
• IOD – Overriding method deletion: The IOD operator deletes an entire declaration of an overriding method in a subclass so that references to the method uses the parent’s version. The mutant act as if there is no overriding method for the method.
• IOP – Overridden method calling position change: Sometimes, an overriding method in a child class needs to call the method it overrides in the parent class. This may happen if the parent’s method uses a private variable v, which means the method in the child class may not modify v directly. However, an easy mistake to make is to call the parent’s version at the wrong time, which can cause incorrect state behavior. The IOP operator moves calls to overridden methods to the first and last statements of the method and up and down one statement.
Object references can have different types with different executions. That is, object references may refer to objects whose actual types differ from their declared types. The actual type can be from any type that is a subclass of the declared type. Polymorphism allows the behavior of an object reference to be different depending the actual type. Therefore, it is important to identify and exercise the program with all possible type bindings. The polymorphism mutation operators are designed to ensure this type of testing.
• PMD – Member variable declaration with parent class type: The PMD operator changes the declared type of an object reference to the parent of the original declared type. The instantiation will still be valid (it will still be a descendant of the new declared type). To kill this mutant, a test case must cause the behavior of the object to be incorrect with the new declared type. In the example below, class Parent is the parent of class Child.
• PRV – Reference assignment with other compatible type: Object references can refer to objects of types that are descendants of its declared type. The PRV operator changes operands of a reference assignment to be assigned to objects of subclasses. In the example below, obj is of type Object, and in the original code it is given an object of type String. In the mutated code, it is given an object of type Integer
Method overloading allows two or more methods of the same class to have the same name as long as they have different argument signatures. Just as with method overriding, it is important for testers to ensure that a method invocation invokes the correct method with appropriate parameters. Three mutation operators are defined to test various aspects of method overloading.
• OMR – Overloading method contents change: The OMR operator is designed to check that overloaded methods are invoked appropriately. The OMR operator replaces the body of a method with the body of another method that has the same name. This is accomplished by using the keyword this.
• OMD – Overloading method deletion: The OMD operator deletes overloading method declarations, one at a time in turn. If the mutant still works correctly without the deleted method, there may be an error in invoking one of the overloading methods; the incorrect method may be invoked or an incorrect parameter type conversion has occurred. The POD operator ensures coverage of overloaded methods, that is, all the overloaded methods must be invoked at least once.
Some object-oriented features are not common to all object-oriented languages. This group of operators attempt to ensure correct use of such features supported in Java. Also, mistakes that a programmer often makes when writing object-oriented software are considered here.
• JTD – this keyword deletion: The JTD operator deletes uses of the keyword this. It models a reverse case of JTI.
• JDC – Java-supported default constructor deletion: Java creates default constructors if a class contains no constructors. The JDC operator forces Java to create a default constructor by deleting the implemented default constructor. It is designed to check if the user-defined default constructor is implemented properly.
• EOA – Reference assignment and content assignment replacement: Object references in Java are always through pointers. Although pointers in Java are typed, which is considered to help prevent certain types of faults, there are still mistakes that programmers can make. One common mistake is that of using an object reference instead of the contents of the object the pointer references. The EOA operator replaces an assignment of a pointer reference with a copy of the object, using the Java convention of a clone() method. The clone() method duplicates the contents of an object, creating and returning a reference to a new object.
• EAM – Accessor method change: The EAM operator changes an accessor method name for other compatible accessor method names, where compatible means that the signatures are the same. This type of mistake occurs because classes with multiple instance variables may wind up having many accessor methods with the same signature and very similar names. As a result, programmers easily get them confused. To kill this mutant a test case will have to produce incorrect output as a result of calling the wrong method.
• EMM – Modifier method change: The EMM operator does the same as EMM, except it works with modifier methods instead of accessor methods.