Mutation testing (MT) targets the assessment of test cases by measuring their efficiency to detect faults. This technique involves modifying the program under test to emulate programming faults, and assessing whether the existing test cases detect such mutations. MT has been extensively studied since the 70's, and many tools have been proposed for widely used languages like C, Java, Fortran, Ada and SQL; and for notations like Petri-nets. However, building MT tools is costly and error-prone, which may prevent their development for new programming and domain-specific (modelling) languages. In this paper, we propose a framework called Wodel-Test to reduce the effort to create MT tools. For this purpose, it follows a model-driven approach by which MT tools are synthesized from a high-level description. This description makes use of the domain-specific language Wodel to define and execute model mutations. Wodel is language-independent, as it allows the creation of mutation operators for any language defined by a meta-model. Starting from the definition of the mutation operators, Wodel-Test generates a MT environment which parses the program under test into a model, applies the mutation operators, and evaluates the test-suite against the generated mutants, offering a rich collection of MT metrics. We report on an evaluation of the approach based on the creation of MT tools for Java and the Atlas transformation language.

1. WODEL-TEST:
A MODEL-BASED FRAMEWORK
FOR LANGUAGE-INDEPENDENT
MUTATION TESTING
P. Gómez-Abajo , E. Guerra, J. de Lara
Modelling&Software Engineering Research Group
http://miso.es
Universidad Autónoma de Madrid (Spain)
October 2021
Mercedes G. Merayo
Design and Testing of Reliable Systems Research Group
http://antares.sip.ucm.es/testing
Universidad Complutense de Madrid (Spain)

2. MOTIVATION
• Current mutation testing tools are
• Specific por a language
• Encoded by hand
• Incur high-costs of maintenance
2
We propose the Wodel-Test framework – a post-
processing extension to Wodel for the generation of
mutation testing tools
We generate two mutation testing tools
Wodel-Test for Java & Wodel-Test for ATL

3. WHAT IS MUTATION
TESTING?
• Approach of software testing to
assess the quality of test suites
• Injection of syntax changes in a
program by using mutation
operators
• The mutations introduced
emulate common programming
faults
• Facilitates to improve the
quality of the test-suites and
the mutation operators set
3

4. WODEL-TEST PROCESS
4

5. WODEL-TEST PROCESS
4
Wodel-Test for
ATL demo

6. EVALUATION
5
RQ1
Does Wodel-Test allow creating MT tools with similar
capabilities to existing MT tools developed by hand?
• Comparison between Wodel-Test for Java with existing mutation testing tools
for Java
•
• Generation of Wodel-Test for ATL with the mutation operators introduced by
Troya et al.[1]
[1] Troya, J., Bergmayr, A., Burgueño, L. y Wimmer, M.: Towards systematic mutations for and with ATL model
transformations. International conference on software testing, verification and validation workshops (ICSTW),
pp. 1–10, 2015
RQ2
How effective is Wodel-Test to specify MT tools?

7. RQ1 EVALUATION
6
Major Javalanche PITest LittleDarwin muJava Wodel-Test/Java
No. of operators 30
(default)
19 40 28 47 77 (default)
Op. extensibility Yes
(DSL)
No API No No Yes (DSL)
Mutated artifact Bytecode Bytecode Bytecode AST Bytecode Model
Mutant source No No Yes Yes Yes Yes
Equivalent
detection
No Yes
(dynamic inv.)
No No No Yes (TCE)
Report type CSV HTML HTML HTML GUI Interactive views
No. of mutants
Mutation score
Killed/live mutants Number Number Number Number Number, list Number, list
Covered operators Number Number Number Number, list Number, list
Mutants by class
Mutants by test
Tests by mutants
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓

  
   
✓
   
✓
✓
✓
✓
✓
✓
✓
✓
✓

RQ1
Similar capabilities?

8. • Application of the mutation testing process to the functional-matrix1 project:
• 1https://github.com/soursop/functional-matrix-operator
MT Tool Killed/lived
mutants
Mutation
score (%)
Run-time Time by
mutant (s)
Major 1638 (331/864) 20.21 13 h 42 m 56 s 30.14
PITest 918 (321/597) 34.97 1 h 11 m 20 s 4.66
LittleDarwin 439 (130/309) 29.61 3 h 30 m 39 s 28.79
muJava 2589 (557/2032) 21.51 5 h 30 m 10 s 7.65
Wodel-Test/Java 4756 (985/3771) 20.71 3 h 24 m 23 s 2.58
7
RQ1
Similar capabilities?
RQ1 EVALUATION

9. • Application of the mutation testing process to the functional-matrix1 project:
• 1https://github.com/soursop/functional-matrix-operator
MT Tool Killed/lived
mutants
Mutation
score (%)
Run-time Time by
mutant (s)
Major 1638 (331/864) 20.21 13 h 42 m 56 s 30.14
PITest 918 (321/597) 34.97 1 h 11 m 20 s 4.66
LittleDarwin 439 (130/309) 29.61 3 h 30 m 39 s 28.79
muJava 2589 (557/2032) 21.51 5 h 30 m 10 s 7.65
Wodel-Test/Java 4756 (985/3771) 20.71 3 h 24 m 23 s 2.58
7
RQ1
Similar capabilities?
RQ1 EVALUATION
RQ1: Yes, Wodel-Test generates
MT tools with similar capabilities

10. RQ2 EVALUATION
• Wodel-Test for ATL with the mutation operators introduced by
Troya et al.
• The proposal by Troya et al. provides a mutant generator
• Wodel-Test for ATL provides the complete functionality of a
mutation testing tool for this language
8
RQ2
How effective?

11. • Mutation operators for ATL
Concept Mutation operator LOC in Wodel LOC in ATL
Matched rule Creation
Deletion
Name change
1
1
1
-
-
-
In pattern element Addition
Deletion
Class change
Name change
6
1
4
1
14
-
-
-
Filter Addition
Deletion
Condition change
10
1
4
-
-
-
Out pattern element Addition
Deletion
Type change
Name change
6
1
4
1
-
6
-
-
Binding Addition
Deletion
Value change
Feature change
6
1
2
6
-
3
-
-
9
RQ2 EVALUATION
RQ2
How effective?

12. • Mutation operators for ATL
Concept Mutation operator LOC in Wodel LOC in ATL
Matched rule Creation
Deletion
Name change
1
1
1
-
-
-
In pattern element Addition
Deletion
Class change
Name change
6
1
4
1
14
-
-
-
Filter Addition
Deletion
Condition change
10
1
4
-
-
-
Out pattern element Addition
Deletion
Type change
Name change
6
1
4
1
-
6
-
-
Binding Addition
Deletion
Value change
Feature change
6
1
2
6
-
3
-
-
9
RQ2 EVALUATION
RQ2
How effective?
RQ2: Yes, Wodel-Test is effective

13. CONCLUSIONS & FUTURE WORK
10
• Wodel-Test provides a similar functionality to other existing mutation
testing tools for Java
• Wodel-Test can be a better option when we need to
• Access the source code of the mutants
• Reason which mutants reduce the mutation score and why
• Test new mutation operators
• The existing mutation testing tools require two orders of magnitude more
LOC than the same specification using Wodel-Test
• Future work
• Apply Wodel-Test to larger programs
• Optimize the mutation testing process
• Usability evaluation

14. 11
Wodel & Wodel-Test websites:
http://gomezabajo.github.io/Wodel/
http://gomezabajo.github.io/Wodel/Wodel-Test/

15. MUTATION TESTING TOOL
FOR JAVA
12

16. GENERATED
MUTATION TESTING TOOL
1. Project explorer
2. Mutant program example
3. Test-suite
4. Global results of the mutation
testing process
13

17. 14
WODEL-TEST ARCHITECTURE