You probably have a lot of tests but their goal is to ensure quality of your production code. What about the quality of your testing code? Code coverage? Nah!

1. MUTATION TESTING
TO THE RESCUE OF
YOUR TESTS
NICOLAS FRÄNKEL

2. ME, MYSELF AND I
@nicolas_frankel #mutationtesting
(Java) Developer & Architect
• As Consultant

3. HYBRIS, AN SAP COMPANY
3

4. MANY KINDS OF TESTING
@nicolas_frankel #mutationtesting
Unit Testing
Integration Testing
End-to-end Testing
Performance Testing
Penetration Testing
Exploratory Testing
etc.

5. THEIR ONLY SINGLE GOAL
@nicolas_frankel #mutationtesting
Ensure the quality of the
production code

6. THE PROBLEM
@nicolas_frankel #mutationtesting
How to check the quality of
the testing code?

7. CODE COVERAGE
@nicolas_frankel #mutationtesting
“Code coverage is a
measure used to describe the
degree to which the source
code of a program is tested”
--Wikipedia
http://en.wikipedia.org/wiki/Co
de_coverage

8. MEASURING CODE COVERAGE
@nicolas_frankel #mutationtesting
Check whether a source
code line is executed during a
test
• Or Branch Coverage

9. COMPUTING CODE COVERAGE
CC =
Lexecuted
Ltotal
*100
CC: Code Coverage
(in percent)
Lexecuted: Number of
executed lines of code
Ltotal: Number of total
lines of code
@nicolas_frankel #mutationtesting

10. JAVA TOOLS FOR CODE COVERAGE
@nicolas_frankel #mutationtesting
JaCoCo
Clover
Cobertura
etc.

11. 100% CODE COVERAGE?
@nicolas_frankel #mutationtesting
“Is 100% code coverage
realistic? Of course it is. If you
can write a line of code, you can
write another that tests it.”
Robert Martin (Uncle Bob)
https://twitter.com/unclebobmarti
n/status/55966620509667328

12. ASSERT-LESS TESTING
@Test
public void add_should_add() {
new Math().add(1, 1);
}
@nicolas_frankel #mutationtesting
But, where is the
assert?
As long as the Code Coverage is
OK…

13. CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Any metric can be gamed!
Code coverage is a metric…
⇒ Code coverage can be
gamed
• On purpose
• Or by accident

14. CODE COVERAGE AS A MEASURE OF QUALITY
@nicolas_frankel #mutationtesting
Code Coverage lulls you into
a false sense of security…

15. THE PROBLEM STILL STANDS
@nicolas_frankel #mutationtesting
Code coverage cannot
ensure test quality
• Is there another way?
Mutation Testing to the
rescue!

16. THE CAST
@nicolas_frankel #mutationtesting
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”

17. public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}

18. STANDARD TESTING
@nicolas_frankel #mutationtesting
✔Execute Test

19. MUTATION TESTING
@nicolas_frankel #mutationtesting
?Execute SAME Test
MUTATION

20. MUTATION TESTING
@nicolas_frankel #mutationtesting
✗
✔Execute SAME Test
Execute SAME Test
Mutant
Killed
Mutant
Survived

21. KILLED OR SURVIVING?
@nicolas_frankel #mutationtesting
Surviving means changing the
source code did not change the test
result
• It’s bad!
Killed means changing the source
code changed the test result
• It’s good

22. TEST THE CODE
@nicolas_frankel #mutationtesting
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}

23. ✔Execute Test
SURVIVING MUTANT
@nicolas_frankel #mutationtesting
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}

24. TEST THE CODE
@nicolas_frankel #mutationtesting
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}

25. KILLED MUTANT
@nicolas_frankel #mutationtesting
✗Execute SAME Test
@Test
public void add_should_add() {
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}

26. MUTATION TESTING IN JAVA
@nicolas_frankel #mutationtesting
PIT is a tool for Mutation
testing
Available as
• Command-line tool
• Ant target
• Maven plugin

27. MUTATORS
@nicolas_frankel #mutationtesting
Mutators are patterns
applied to source code to
produce mutations

28. PIT MUTATORS SAMPLE
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
@nicolas_frankel #mutationtesting

29. IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
Conditionals Boundary
• Potential serious bug hiding
there
• if (foo > bar)

30. IMPORTANT MUTATORS
@nicolas_frankel #mutationtesting
 Void Method Call
• Assert.checkNotNull()
• connection.close()

31. REMEMBER
@nicolas_frankel #mutationtesting
 It’s not because the IDE
generates code safely that
it will never change
• equals()
• hashCode()

32. ENOUGH TALK…
@nicolas_frankel #mutationtesting

33. SH… HAPPENS
@nicolas_frankel #mutationtesting
 False positives
 Imperfect
 Sloooooooooooooooow

34. PIT IS IMPERFECT
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
@nicolas_frankel #mutationtesting

35. PIT IS IMPERFECT
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
@nicolas_frankel #mutationtesting

36. WHY SO SLOW?
@nicolas_frankel #mutationtesting
Analyze test code
For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results

37. WORKAROUNDS
Increase number of threads  
Set a limited a set of mutators
Limit scope of target classes
Limit number of tests
Limit dependency distance
Use incremental analysis  
Don’t bind to the test phase  
Use scmMutationCoverage

38. INCREMENTAL ANALYSIS
 Metadata stored between runs
 Mutant will not be checked again, if:
• timed out, and class has not changed
• killed, and neither class nor test have changed
• survived, and there are no new/changed tests for it
@nicolas_frankel #mutationtesting

39. THE SILVER BULLET?
@nicolas_frankel #mutationtesting
Checks the relevance of
your unit tests
Points out potential bugs

40. THE REST IS UP TO YOU
@nicolas_frankel #mutationtesting
Validate the assembled application
• Integration Testing
Check the performance
• Performance Testing
Look out for display bugs
• End-to-end testing
Etc.

41. TESTING IS ABOUT ROI
@nicolas_frankel #mutationtesting
Don’t test to achieve 100%
coverage
Test because it saves money
in the long run
Prioritize:
• Business-critical code
• Complex code

42. Q&A
@nicolas_frankel #springboot
http://blog.frankel.ch/
@nicolas_frankel
http://frankel.in/
https://git.io/vznQK