The document discusses mutation testing as a way to ensure the quality of test code. Mutation testing works by modifying the source code in small ways (called mutations) and checking if tests can detect the changes. This helps identify weaknesses or gaps in test coverage that code coverage metrics alone may miss. The document recommends using a tool like PIT for Java projects, which automatically generates mutations and analyzes whether tests can kill the mutated code or if mutants survive undetected. Mutation testing is slow but can help improve test quality by finding overlooked cases.

1. MUTATION TESTING
TO THE RESCUE OF
YOUR TESTS
NICOLAS FRÄNKEL

2. @nicolas_frankel #mutationtesting
2

3. HYBRIS, AN SAP COMPANY
@nicolas_frankel #mutationtesting
3

4. ME, MYSELF AND I
@nicolas_frankel #mutationtesting
By day
• Consultant
By night
• Teacher
• Book author
• Conference speaker
• Blogger
4

5. MANY KINDS OF TESTING
@nicolas_frankel #mutationtesting
Unit Testing
Integration Testing
End-to-end Testing
Performance Testing
Penetration Testing
Exploratory Testing
etc.
5

6. THEIR ONLY SINGLE GOAL
@nicolas_frankel #mutationtesting
Ensure the quality of the
production code
6

7. THE PROBLEM
@nicolas_frankel #mutationtesting
How to check the quality of
the testing code?
7

8. CODE COVERAGE
@nicolas_frankel #mutationtesting
“Code coverage is a
measure used to describe the
degree to which the source
code of a program is tested”
--Wikipedia
http://en.wikipedia.org/wiki/Co
de_coverage
8

9. MEASURING CODE COVERAGE
@nicolas_frankel #mutationtesting
Check whether a source
code line is executed during a
test
• Or Branch Coverage
9

10. COMPUTING CODE COVERAGE
CC =
Lexecuted
Ltotal
*100
CC: Code Coverage
(in percent)
Lexecuted: Number of
executed lines of code
Ltotal: Number of total
lines of code
@nicolas_frankel #mutationtesting
10

11. JAVA TOOLS FOR CODE COVERAGE
@nicolas_frankel #mutationtesting
JaCoCo
Clover
Cobertura
etc.
11

12. 100% CODE COVERAGE?
@nicolas_frankel #mutationtesting
“Is 100% code coverage
realistic? Of course it is. If you
can write a line of code, you can
write another that tests it.”
Robert Martin (Uncle Bob)
https://twitter.com/unclebobmarti
n/status/55966620509667328
12

13. ASSERT-LESS TESTING
@Test
public void add_should_add() {
new Math().add(1, 1);
}
@nicolas_frankel #mutationtesting
But, where is the
assert?
As long as the Code Coverage is
OK…
13

14. CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Any metric can be gamed!
Code coverage is a metric…
⇒ Code coverage can be
gamed
• On purpose
• Or by accident
14

15. CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Code Coverage lulls you into
a false sense of security…
15

16. THE PROBLEM STILL STANDS
@nicolas_frankel #mutationtesting
Code coverage cannot
ensure test quality
• Is there another way?
Mutation Testing to the
rescue!
16

17. THE CAST
@nicolas_frankel #mutationtesting
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”
17

18. public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
18

19. STANDARD TESTING
@nicolas_frankel #mutationtesting
✔Execute Test
19

20. MUTATION TESTING
@nicolas_frankel #mutationtesting
?Execute SAME Test
MUTATION
20

21. MUTATION TESTING
@nicolas_frankel #mutationtesting
✗
✔Execute SAME Test
Execute SAME Test
Mutant
Killed
Mutant
Survived
21

22. KILLED OR SURVIVING?
@nicolas_frankel #mutationtesting
Surviving means changing the
source code did not change the test
result
• It’s bad!
Killed means changing the source
code changed the test result
• It’s good
22

23. TEST THE CODE
@nicolas_frankel #mutationtesting
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
23

24. ✔Execute Test
SURVIVING MUTANT
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}
24

25. TEST THE CODE
@nicolas_frankel #mutationtesting
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
25

26. KILLED MUTANT
@nicolas_frankel #mutationtesting
✗Execute SAME Test
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
26

27. MUTATION TESTING IN JAVA
@nicolas_frankel #mutationtesting
PIT is a tool for Mutation
testing
Available as
• Command-line tool
• Ant target
• Maven plugin
27

28. MUTATORS
@nicolas_frankel #mutationtesting
Mutators are patterns
applied to source code to
produce mutations
28

29. PIT MUTATORS SAMPLE
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
@nicolas_frankel #mutationtesting
29

30. IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Conditionals Boundary
• Potential serious bug hiding
there
• if (foo > bar)
30

31. IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
 Void Method Call
• Assert.checkNotNull()
• connection.close()
31

32. REMEMBER
@nicolas_frankel #mutationtesting
 It’s not because the IDE
generates code safely that
it will never change
• equals()
• hashCode()
32

33. ENOUGH TALK…
@nicolas_frankel #mutationtesting
33

34. SH… HAPPENS
@nicolas_frankel #mutationtesting
 False positives
 Imperfect
 Sloooooooooooooooow
34

35. PIT IS IMPERFECT
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
@nicolas_frankel #mutationtesting
35

36. PIT IS IMPERFECT
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
@nicolas_frankel #mutationtesting
36

37. WHY SO SLOW?
@nicolas_frankel #mutationtesting
Analyze test code
For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
37

38. WORKAROUNDS
Increase number of threads  
Set a limited a set of mutators
Limit scope of target classes
Limit number of tests
Limit dependency distance
Use incremental analysis  
Don’t bind to the test phase  
Use scmMutationCoverage
@nicolas_frankel #mutationtesting
38

39. INCREMENTAL ANALYSIS
 Metadata stored between runs
 Mutant will not be checked again, if:
• killed, and neither class nor test have changed
• survived, and there are no new/changed tests for it
@nicolas_frankel #mutationtesting
39

40. THE SILVER BULLET?
@nicolas_frankel #mutationtesting
Checks the relevance of
your unit tests
Points out potential bugs
40

41. THE REST IS UP TO YOU
@nicolas_frankel #mutationtesting
Validate the assembled application
• Integration Testing
Check the performance
• Performance Testing
Look out for display bugs
• End-to-end testing
Etc.
41

42. TESTING IS ABOUT ROI
@nicolas_frankel #mutationtesting
Don’t test to achieve 100%
coverage
Test because it saves money
in the long run
Prioritize:
• Business-critical code
• Complex code
42

43. Q&A
@nicolas_frankel #mutationtesting
http://blog.frankel.ch/
@nicolas_frankel
https://git.io/vznQK
43