Extended version of my brown bag lightning talk on Mutation Testing. Given at IPRUG on 3rd July 2012.

1. Mutation Testing
Chris Sinjakli

2. Testing is a good thing
But how do we know our tests are
good?

3. Code coverage is a start
But it can give a “good” score with
really dreadful tests

4. Really dreadful tests
class Adder
def self.add (x, y)
return x - y
end
end
describe Adder do
it "should add the two arguments" do
Adder.add(1, 1)
end
end
Coverage: 100%
Usefulness: 0

6. A contrived example
But how could we detect it?

7. Mutation Testing!
“Who watches the watchmen?”

8. If you can change the code, and a
test doesn’t fail, either the code is
never run or the tests are wrong.

9. How?
1. Run test suite
2. Change code (mutate)
3. Run test suite again
If tests now fail, mutant dies. Otherwise it
survives.

10. Going with our previous example
class Adder
def self.add (x, y)
return x - y
end
end Let’s change something
describe Adder do
it "should add the two arguments" do
Adder.add(1, 1)
end
end

11. Going with our previous example
class Adder
def self.add (x, y)
return x + y
end
end
This still passes
describe Adder do
it "should add the two arguments" do
Adder.add(1, 1)
end
end

12. Success
We know something is wrong

13. So what? It caught a really
rubbish test
How about something slightly less
obvious?

14. Slightly less obvious (and I mean slightly)
class ConditionChecker
def self.check(a, b)
if a && b
return 42
else
return 0
end
end
end
describe ConditionChecker do
it "should return 42 when both arguments are true" do
ConditionChecker.check(true, true).should == 42
end
it "should return 0 when both arguments are false" do
ConditionChecker.check(false, false).should == 0
end
end Coverage: 100%
Usefulness: >0
But still wrong

15. Slightly less obvious (and I mean slightly)
class ConditionChecker
def self.check(a, b)
if a && b
return 42
else Mutate
return 0
end
end
end
describe ConditionChecker do
it "should return 42 when both arguments are true" do
ConditionChecker.check(true, true).should == 42
end
it "should return 0 when both arguments are false" do
ConditionChecker.check(false, false).should == 0
end
end

16. Slightly less obvious (and I mean slightly)
class ConditionChecker
def self.check(a, b)
if a || b
return 42
else
return 0
end
end
end
describe ConditionChecker do Passing tests
it "should return 42 when both arguments are true" do
ConditionChecker.check(true, true).should == 42
end
it "should return 0 when both arguments are false" do
ConditionChecker.check(false, false).should == 0
end
end

17. Mutation testing caught our
mistake
:D

18. Useful technique
But still has its flaws

19. The downfall of mutation
(Equivalent Mutants)
index = 0
while index != 100 do
doStuff()
index += 1
end
Mutates to
index = 0
while index < 100 do
doStuff()
index += 1
end
But the programs are equivalent, so no test will fail

20. There is no possible test which
can “kill” the mutant
The programs are equivalent

21. Also (potentially)
• Infinite loops
• More memory used
• Compile/run time errors – tools should
minimise these

22. How bad is it?
• Good paper assessing the problem [SZ10]
• Took 7 widely used, “large” projects
• Found:
– 15 mins to assess one mutation
– 45% uncaught mutations are equivalent
– Better tested project -> worse signal-to-noise ratio

23. Can we detect the equivalents?
• Not in the general case [BA82]
• Some specific cases can be detected
– Using compiler optimisation techniques [BS79]
– Using mathematical constraints [DO91]
– Line coverage changes [SZ10]
• All heuristic algorithms – not seen any
claiming to kill all equivalent mutants

24. Tools
Some Ruby, then a Java one I liked

25. Ruby
• Looked into Heckle
• Seemed unmaintained (nothing since 2009)
• Then I saw...

26. Ruby

27. Ruby
• Mutant seems to be the new favourite
• Runs in Rubinius (1.8 or 1.9 mode)
• Only supports RSpec
• Easy to set up
rvm install rbx-head
rvm use rbx-head
gem install mutant
• And easy to use
mutate “ClassName#method_to_test” spec

28. Java
• Loads of tools to choose from
• Bytecode vs source mutation
• Will look at PIT (seems like one of the better
ones)

29. PIT - pitest.org
• Works with “everything”
– Command line
– Ant
– Maven
• Bytecode level mutations (faster)
• Very customisable
– Exclude classes/packages from mutation
– Choose which mutations you want
– Timeouts
• Makes pretty HTML reports (line/mutation coverage)

30. Summary
• Can point at weak areas in your tests
• At the same time, can be prohibitively noisy
• Try it and see

31. Questions?

32. References
• [BA82] - T. A. Budd and D. Angluin. Two notions of correctness and
their relation to testing. Acta Informatica, 18(1):31-45, November
1982.
• [BS79] - D. Baldwin and F. Sayward. Heuristics for determining
equivalence of program mutations. Research report 276,
Department of Computer Science, Yale University, 1979.
• [DO91] - R. A. DeMillo and A. J. O
utt. Constraint-based automatic test data generation. IEEE
Transactions on Software Engineering, 17(9):900-910, September
1991.
• [SZ10] - D. Schuler and A. Zeller. (Un-)Covering Equivalent Mutants.
Third International Conference on Software Testing, Verification and
Validation (ICST), pages 45-54. April 2010.

33. Also interesting
• [AHH04] – K. Adamopoulos, M. Harman and R. M. Hierons. How to
Overcome the Equivalent Mutant Problem and Achieve Tailored
Selective Mutation Using Co-evolution. Genetic and Evolutionary
Computation -- GECCO 2004, pages 1338-1349. 2004.