This document discusses mutation testing, including basics, relation to other testing techniques, examples, mutation testing tools, and personal experiences. Mutation testing involves modifying code in small ways (e.g., changing < to >=) and seeing if tests detect the changes. It helps ensure tests thoroughly validate code behavior and semantics. The document recommends using mutation testing from the start of a project to write high-quality tests and code.

1. © Computas AS 26.01.15
Mutation Testing
with Mutant and PIT
Filip van Laenen & Markus Schirp
OOP 2015
2015-01-26

2. 2 © Computas AS 26.01.15
Agenda
• Basics of mutation testing
• Relation to other testing techniques
• Example
• Mutation testing techniques
• Mutation testing tools
• Personal experiences and recommendations

3. 3 © Computas AS 26.01.15
Basics of
Mutation Testing

4. 4 © Computas AS 26.01.15
Mutation Testing in a Nutshell
Seeking The Summoner @ The Daily WTF
http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx

5. 5 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

6. 6 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

7. 7 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)
• Unit tests guard the source code
• But who guards the guardians?
• Do the unit tests cover all source code?
• Lines?
• Branches?
• Paths?
• Semantics?
• Do the unit tests test the right things?
Mutation testing tests the tests!

8. 8 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

9. 9 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

10. 10 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

11. 11 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

12. 12 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

13. 13 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

14. 14 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)
int max(int a, int b) {
return (a < b) ? b : a;
}
int max(int a, int b) {
return (a <= b) ? b : a;
}

15. 15 © Computas AS 26.01.15
Mutation Testing in a Nutshell (cont'd)

16. “
16 © Computas AS 26.01.15
Does Mutation Testing Work?
In practice, if the software contains a
fault, there will usually be a set of
mutants that can only be killed by a test
case that also detects that fault.
Geist et. al., “Estimation and Enhancement of Real-time
Software Reliability through Mutation Analysis,” 1992

17. “
17 © Computas AS 26.01.15
Does Mutation Testing Work? (cont'd)
Complex faults are coupled to simple
faults in such a way that a test data set
that detects all simple faults in a program
will detect most complex faults.
K. Wah, “Fault Coupling in Finite Bijective Functions,”
1995

18. 18 © Computas AS 26.01.15
Does Mutation Testing Work? (cont'd)
• “Generated mutants are similar to real faults.”
• Andrews, Briand, Labiche, ICSE 2005
• “Mutation testing is more powerful than
statement or branch coverage.”
• Walsh, Ph.D. Thesis, State University of New York at
Binghampton, 1985
• “Mutation testing is superior to data flow
coverage criteria.”
• Frankl, Weiss, Hu, Journal of Systems and Software,
1997

19. 19 © Computas AS 26.01.15
Relation to Other
Testing Techniques

20. 20 © Computas AS 26.01.15
Relation to Other Testing Techniques
• Unit tests
• Test-Driven Development (TDD)
• Test coverage
• Static code analysis
• Fuzz testing

21. 21 © Computas AS 26.01.15
Relation to Other Testing Techniques

22. 22 © Computas AS 26.01.15
Is Mutation Testing New?
• R. Lipton, “Fault Diagnosis of Computer
Programs,” 1971
• R. Lipton et. al., “Hints on Test Data Selection:
Help for the Practicing Programmer,” 1978
• Historical obstacles:
• No unit testing
• No TDD
• Inefficient implementation
• Hence time-consuming
• No integration with IDEs

23. 23 © Computas AS 26.01.15
Practical Example
of Mutation Testing

24. 24 © Computas AS 26.01.15
Code Along…
https://github.com/computas-fvl/oop2015
git clone
https://github.com/computas-fvl/oop2015.git
mvn clean site:site
org.pitest:pitest-maven:mutationCoverage

25. 25 © Computas AS 26.01.15
Mutation Testing
Techniques

26. 26 © Computas AS 26.01.15
Mutation Testing Techniques
• Three aspects:
• Mutation injection
• Mutation types
• Unit test selection per mutant
• Key properties:
• Efficiency
• Performance
• Coverage

27. 27 © Computas AS 26.01.15
Mutation Injection
• Source code mutation
• Intermediate code mutation
• Binary code mutation
• Caveats:
• De-mutation (for reporting)
• Compilation errors
• Invalid binary code
• Leaked state

28. 28 © Computas AS 26.01.15
Mutation Types
• Some mutations never change behaviour
• Constants reused by unit tests
• Log messages
• Exception messages
• Some mutations can change behaviour
• Switching between < and ≠
• Switching between < and ≤
• Some mutations always change behaviour
• Switching between < and ≥

29. 29 © Computas AS 26.01.15
Mutation Types (cont'd)
int max(int a, int b) {
return (a < b) ? b : a;
}
int max(int a, int b) {
return (a <= b) ? b : a;
}
int max(int a, int b) {
return (a >= b) ? b : a;
}

30. 30 © Computas AS 26.01.15
Mutation Types (cont'd)
for (int i = 0; i < 10; i++) …
for (int i = 0; i != 10; i++) …
for (int i = 0; i >= 10; i++) …

31. 31 © Computas AS 26.01.15
Mutation Types Guaranteed to Change
Behaviour
• Negation of the comparison
• Switching between = and ≠
• Switching between < and ≥
• Switching between > and ≤
• Negation of boolean conditions
• Adding a ¬, ! or ~
• Shortcutting boolean conditions
• Replacement with True or False
*

32. 32 © Computas AS 26.01.15
Unit Test Selection
• Goal: find the unit test that “kills” the mutant
• Selection aids:
• Hints
• Name/package matching
• Code coverage tools
• Incremental mutation testing
• Other heuristics

33. 33 © Computas AS 26.01.15
Unit Test Selection (cont'd)
• System:
• 50 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 50 × 20 × 1ms = 1s
• 10 mutants per class:
• Brute-force: 10 × 50 × 1s = 6m 20s
• Educated: 10 × 50 × 20 × 1ms = 10s

34. 34 © Computas AS 26.01.15
Unit Test Selection (cont'd)
• System:
• 500 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 500 × 20 × 1ms = 10s
• 10 mutants per class:
• Brute-force: 10 × 500 × 10s = 13h 53m 20s
• Educated: 10 × 500 × 20 × 1ms = 1m 40s

35. 35 © Computas AS 26.01.15
Complexity
• f: Number of function points
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × f²
• Educated force: τ × μ × φ × f
• Ideal: τ × μ × f

36. 36 © Computas AS 26.01.15
Complexity (cont'd)
• c: Number of classes
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²
• Educated force: τ × μ × φ² × c
• Ideal: τ × μ × φ × c

37. 37 © Computas AS 26.01.15
Loops
for (int i = 0; i < 10; i++) …
for (int i = 0; i < 10; i--) …

38. 38 © Computas AS 26.01.15
Infinite Loops
• Terminate mutants that take too long to run
• What's “too long”?
• Ruins the performance
• Can be hard to predict

39. 39 © Computas AS 26.01.15
Other Problems
• Recursion:
• Stack overflows
• Out of memory exceptions
• Syntax errors
• Leaked state
• Segmentation faults

40. 40 © Computas AS 26.01.15
Mutation Testing
Tools

41. 41 © Computas AS 26.01.15
Mutation Testing Tools
• Java:
• PIT
• Ruby:
• Mutant
• Heckle
• PHP
• Humbug
• C#:
• NinjaTurtles

42. 42 © Computas AS 26.01.15
PIT
• Java
• Junit & TestNG
• Maven or command-line
• Operates on byte code
• Large set of mutators
• Also possibly equivalent mutators available
• Highly configurable
• Sensible defaults
• http://pitest.org/

43. 43 © Computas AS 26.01.15
PIT Mutators
• Conditionals Boundary
• Negate Conditionals
• Remove Conditionals*
• Math
• Increments
• Invert Negatives
• Inline Constant*
• Return Values
• Void Method Calls
• Non Void Method Calls*
• Constructor Calls*

44. 44 © Computas AS 26.01.15
PIT Sample Report

45. 45 © Computas AS 26.01.15
Jester
• Java
• JUnit
• Usually run from the command-line
• Grester for Maven2
• Operates on source code
• Runs all unit tests on all classes
• Reporting and documentation could be better
• http://jester.sourceforge.net/
• http://sourceforge.net/projects/grester/

46. 46 © Computas AS 26.01.15
Jester Sample Report Overview

47. 47 © Computas AS 26.01.15
Jester Sample Detailed Report

48. 48 © Computas AS 26.01.15
Pester and Nester
• Pester
• Jester for Python
• PyUnit
• Nester
• Port of Jester for C#
• NUnit
• Integrated with Visual Studio
• But outdated…
• http://nester.sourceforge.net/

49. 49 © Computas AS 26.01.15
Nester Sample Report

50. 50 © Computas AS 26.01.15
Jumble
• Java
• JUnit
• Run from the command-line
• Operates on byte code
• Runs unit tests on a class
• Reporting and documentation could be better
• Claims to be faster than Jester
• http://jumble.sourceforge.net/index.html

51. 51 © Computas AS 26.01.15
Jumble Sample Report
Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
..
M FAIL: Foo:31: negated conditional
M FAIL: Foo:33: negated conditional
M FAIL: Foo:34: - -> +
M FAIL: Foo:35: negated conditional
......
Score: 67%

52. 52 © Computas AS 26.01.15
Mutant
• Ruby 1.9 up to 2.2
• Rspec2 and Rspec3
• Usually run from the command-line
• Good to-the-point reporting
• Good performance
• OK documentation
• Active project
• https://github.com/mbj/mutant

53. 53 © Computas AS 26.01.15
Heckle
• Ruby 1.8
• Doesn't work on Ruby 1.9
• Test::Unit and rSpec
• Usually run from the command-line
• Runs a set of unit tests on a class or a method
• Good to-the-point reporting
• Acceptable performance
• Virtually no documentation
• http://rubyforge.org/projects/seattlerb/
• http://docs.seattlerb.org/heckle/

54. 54 © Computas AS 26.01.15
Heckle Mutations
• Booleans
• Numbers
• Strings
• Symbols
• Ranges
• Regexes
• Branches (if, while, unless, until)

55. 55 © Computas AS 26.01.15
Humbug
• PHP 5.4 or greater
• PHPUnit
• Usually run from the command-line
• OK documentation
• Active project
• https://github.com/padraic/humbug

56. 56 © Computas AS 26.01.15
NinjaTurtles
• .Net:
• C#
• Visual Basic/VB.NET
• Any other .NET language code
• Doesn't seem much alive
• http://www.mutation-testing.net/

57. 57 © Computas AS 26.01.15
NinjaTurtles Mutations
• Sequence point deletion
• Arithmetic operator substitution (*, /, +, -, %)
• Bitwise operator substitution (&, |, ^)
• Branch substituion (condition, always and never
branch)
• Conditional boundary substition (< and <=, >
and >=)
• Substitution of reads from variables,
parameters and fields of the same type
• Substitution of writes to variables of the same
type

58. 58 © Computas AS 26.01.15
Personal Experiences
and Recommendations

59. 59 © Computas AS 26.01.15
Experiences and Recommendations
• Use mutation testing from day 1
• Start on a small code base
• Keep things simple
• Have small interfaces
• Select a good tool
• Configurable
• Flexible
• One that can output the mutant

60. 60 © Computas AS 26.01.15
Experiences and Recommendations
(cont'd)
• Believe the tool
• Or try to proof that the tool is wrong
• Fix the problem
• Don't turn mutation testing off
• Embrace the “more than 100%” test coverage
• Semantic coverage
• Less code
• More unit tests
• More intelligent unit tests

61. 61 © Computas AS 26.01.15
Improvements

62. 62 © Computas AS 26.01.15
Improvements
• Integration with more unit testing frameworks
• Better unit test–source code mapping
• Better heuristics
• Parallellisation
• Better reporting
• IDE integration
• Building tool integration