Mutation testing involves introducing small changes or mutations to code and checking if tests detect these changes. It is more effective than just measuring code coverage as it verifies test functionality rather than just execution. The goal is to "kill" or have tests detect all mutations, though some mutations may produce invalid code. It is a time-consuming process so techniques like parallelization and fast tests can help. Mutation testing helps ensure tests thoroughly verify code works as intended.

1. Mutation testing
Igor Gajowiak
July 2019
Igor Gajowiak 1 / 13

2. Why do we test our code ?
To ensure that code works properly
To have regression base for future development
Igor Gajowiak 2 / 13

3. What does test the tests ?
Code coverage index
Mutation testing ! ! !
Igor Gajowiak 3 / 13

4. Mutation testing
1 Prerequisites : Code base, test suite.
2 Run the test suite
3 Introduce small change to the code base - mutation
4 Run test suite
5 If the first test run succeeds and the second one not, then
it’s great !
Igor Gajowiak 4 / 13

5. Examples
Before :
1 f o r ( int i = 0; i < 100; ++i )
2 {
3 . . .
4 }
After :
1 f o r ( int i = 0; i > 100; ++i )
2 {
3 . . .
4 }
Igor Gajowiak 5 / 13

6. Examples
Before :
1 i f ( a && b)
2 {
3 c = 1;
4 }
After :
1 i f ( a | | b)
2 {
3 c = 1;
4 }
Igor Gajowiak 6 / 13

7. Result interpretation
Survived - tests did not detect change
Killed - tests detected change in the code
Incompetent - mutation produced flawed code
Igor Gajowiak 7 / 13

8. What is the goal ?
KILL THEM ALL
Igor Gajowiak 8 / 13

9. Code coverage vs mutation testing
Was a line executed ?
VS
Was a functionality verified ?
Igor Gajowiak 9 / 13

10. Examples of mutations
+ −
∗ /
< >
<
+ + i − − i
! = ==
continue break
|| &&
true false
Igor Gajowiak 10 / 13

11. It is time-consuming process
EXECUTION_TIME =
NUMBER_OF_MUTANTS * TEST_SUITE_EXECUTION_TIME
Solutions :
Parallelize
Write fast tests
Igor Gajowiak 11 / 13

12. Detecting incompetent mutants
1 while ( true )
2 {
3 . . .
4 i f ( condition )
5 {
6 break ;
7 }
8 . . .
9 }
1 while ( true )
2 {
3 . . .
4 i f ( condition )
5 {
6 continue ;
7 }
8 . . .
9 }
Igor Gajowiak 12 / 13

13. Stryker
Stryker - https://stryker-mutator.io
Supported mutations - https://github.com/stryker-
mutator/stryker-net/blob/master/docs/Mutators.md
Igor Gajowiak 13 / 13