Mutation testing can help identify weaknesses in test cases by deliberately introducing faults into code and checking if tests catch the mutations. While code coverage metrics can report high coverage for poor tests, mutation testing works by changing the code in small ways, like changing a boolean operator from AND to OR, and seeing if tests fail as a result. If tests pass despite a mutation, the tests are insufficient. Mutation testing tools exist for Java, like PIT, and Ruby, like Mutant, that automate the process of mutation testing to evaluate test quality. However, mutation testing also has limitations as some mutations may not actually change program behavior, confounding the results.

1. Mutation Testing
Chris Sinjakli

2. Testing is a good thing
But how do we know our tests are
good?

3. Code coverage is a start
But it can give a “good” score with
really dreadful tests

4. Really dreadful tests
public int addTwoNumbers(int a, int b) {
return a – b;
}
...
@Test
public void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);
assertTrue(true);
}
Coverage: 100%
Usefulness: 0

6. A contrived example
But how could we detect it?

7. Mutation Testing!
“Who watches the watchmen?”

8. If you can change the code, and a
test doesn’t fail, either the code is
never run or the tests are wrong.

9. Going with our previous example
public int addTwoNumbers(int a, int b) {
return a – b;
}
Let’s change something
...
@Test
public void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);
assertTrue(true);
}

10. Going with our previous example
public int addTwoNumbers(int a, int b) {
return a + b;
}
...
This still passes
@Test
public void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);
assertTrue(true);
}

11. So it caught a really rubbish test
How about something slightly less
obvious?

12. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a && b) {
return 42;
}
else {
return 0;
}
}
@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
} Coverage: 100%
Usefulness: >0
But still wrong

13. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a && b) {
return 42;
} Mutate
else {
return 0;
}
}
@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
}

14. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a || b) {
return 42;
}
else {
return 0;
}
}
@Test
public void testBothFalse() { Passing tests
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
}

15. Mutation testing caught our
mistake
:D

16. Useful technique
But still has its flaws

17. The downfall of mutation
(Equivalent Mutants)
int index = 0
while (someCondition) {
doStuff();
index++;
if (index == 100) {
break;
}
}
Mutates to
int index = 0
while (someCondition) {
doStuff();
index++;
if (index >= 100) {
break;
}
}
But the programs are equivalent, so no test will fail

18. Tools
Some Java, then some Ruby

19. Java
• Loads of tools to choose from
• Bytecode vs source mutation
• Will look at PIT (seems like one of the better
ones)

20. PIT - pitest.org
• Works with “everything”
– Command line
– Ant
– Maven
• Bytecode level mutations (faster)
• Very customisable
– Exclude classes/packages from mutation
– Choose which mutations you want
– Timeouts
• Makes pretty HTML reports (line/mutation coverage)

21. Ruby

22. Ruby
• Mutant seems to be the new favourite
• Runs in Rubinius (1.8 or 1.9 mode)
• Only supports RSpec
• Easy to set up
rvm install rbx-head
rvm use rbx-head
gem install mutant
• And easy to use
mutate “ClassName#method_to_test” spec

23. Summary
• Seems like it could identify areas of weakness
in our tests
• At the same time, could be very noisy
• Might be worth just trying it against an
existing project and seeing what happens

24. Questions?