Oop 2015 – Mutation Testing

1. © Computas AS 26.01.15 Mutation Testing with Mutant and PIT Filip van Laenen & Markus Schirp OOP 2015 2015-01-26

2. 2 © Computas AS 26.01.15 Agenda • Basics of mutation testing • Relation to other testing techniques • Example • Mutation testing techniques • Mutation testing tools • Personal experiences and recommendations

3. 3 © Computas AS 26.01.15 Basics of Mutation Testing

4. 4 © Computas AS 26.01.15 Mutation Testing in a Nutshell Seeking The Summoner @ The Daily WTF http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx

5. 5 © Computas AS 26.01.15 Mutation Testing in a Nutshell (cont'd)

7. 7 © Computas AS 26.01.15 Mutation Testing in a Nutshell (cont'd) • Unit tests guard the source code • But who guards the guardians? • Do the unit tests cover all source code? • Lines? • Branches? • Paths? • Semantics? • Do the unit tests test the right things? Mutation testing tests the tests!

14. 14 © Computas AS 26.01.15 Mutation Testing in a Nutshell (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; }

16. “ 16 © Computas AS 26.01.15 Does Mutation Testing Work? In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault. Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992

17. “ 17 © Computas AS 26.01.15 Does Mutation Testing Work? (cont'd) Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults. K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995

18. 18 © Computas AS 26.01.15 Does Mutation Testing Work? (cont'd) • “Generated mutants are similar to real faults.” • Andrews, Briand, Labiche, ICSE 2005 • “Mutation testing is more powerful than statement or branch coverage.” • Walsh, Ph.D. Thesis, State University of New York at Binghampton, 1985 • “Mutation testing is superior to data flow coverage criteria.” • Frankl, Weiss, Hu, Journal of Systems and Software, 1997

19. 19 © Computas AS 26.01.15 Relation to Other Testing Techniques

20. 20 © Computas AS 26.01.15 Relation to Other Testing Techniques • Unit tests • Test-Driven Development (TDD) • Test coverage • Static code analysis • Fuzz testing

21. 21 © Computas AS 26.01.15 Relation to Other Testing Techniques

22. 22 © Computas AS 26.01.15 Is Mutation Testing New? • R. Lipton, “Fault Diagnosis of Computer Programs,” 1971 • R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978 • Historical obstacles: • No unit testing • No TDD • Inefficient implementation • Hence time-consuming • No integration with IDEs

23. 23 © Computas AS 26.01.15 Practical Example of Mutation Testing

24. 24 © Computas AS 26.01.15 Code Along… https://github.com/computas-fvl/oop2015 git clone https://github.com/computas-fvl/oop2015.git mvn clean site:site org.pitest:pitest-maven:mutationCoverage

25. 25 © Computas AS 26.01.15 Mutation Testing Techniques

26. 26 © Computas AS 26.01.15 Mutation Testing Techniques • Three aspects: • Mutation injection • Mutation types • Unit test selection per mutant • Key properties: • Efficiency • Performance • Coverage

27. 27 © Computas AS 26.01.15 Mutation Injection • Source code mutation • Intermediate code mutation • Binary code mutation • Caveats: • De-mutation (for reporting) • Compilation errors • Invalid binary code • Leaked state

28. 28 © Computas AS 26.01.15 Mutation Types • Some mutations never change behaviour • Constants reused by unit tests • Log messages • Exception messages • Some mutations can change behaviour • Switching between < and ≠ • Switching between < and ≤ • Some mutations always change behaviour • Switching between < and ≥

29. 29 © Computas AS 26.01.15 Mutation Types (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; } int max(int a, int b) { return (a >= b) ? b : a; }

30. 30 © Computas AS 26.01.15 Mutation Types (cont'd) for (int i = 0; i < 10; i++) … for (int i = 0; i != 10; i++) … for (int i = 0; i >= 10; i++) …

31. 31 © Computas AS 26.01.15 Mutation Types Guaranteed to Change Behaviour • Negation of the comparison • Switching between = and ≠ • Switching between < and ≥ • Switching between > and ≤ • Negation of boolean conditions • Adding a ¬, ! or ~ • Shortcutting boolean conditions • Replacement with True or False *

32. 32 © Computas AS 26.01.15 Unit Test Selection • Goal: find the unit test that “kills” the mutant • Selection aids: • Hints • Name/package matching • Code coverage tools • Incremental mutation testing • Other heuristics

33. 33 © Computas AS 26.01.15 Unit Test Selection (cont'd) • System: • 50 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 50 × 20 × 1ms = 1s • 10 mutants per class: • Brute-force: 10 × 50 × 1s = 6m 20s • Educated: 10 × 50 × 20 × 1ms = 10s

34. 34 © Computas AS 26.01.15 Unit Test Selection (cont'd) • System: • 500 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 500 × 20 × 1ms = 10s • 10 mutants per class: • Brute-force: 10 × 500 × 10s = 13h 53m 20s • Educated: 10 × 500 × 20 × 1ms = 1m 40s

35. 35 © Computas AS 26.01.15 Complexity • f: Number of function points • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × f² • Educated force: τ × μ × φ × f • Ideal: τ × μ × f

36. 36 © Computas AS 26.01.15 Complexity (cont'd) • c: Number of classes • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c² • Educated force: τ × μ × φ² × c • Ideal: τ × μ × φ × c

37. 37 © Computas AS 26.01.15 Loops for (int i = 0; i < 10; i++) … for (int i = 0; i < 10; i--) …

38. 38 © Computas AS 26.01.15 Infinite Loops • Terminate mutants that take too long to run • What's “too long”? • Ruins the performance • Can be hard to predict

39. 39 © Computas AS 26.01.15 Other Problems • Recursion: • Stack overflows • Out of memory exceptions • Syntax errors • Leaked state • Segmentation faults

40. 40 © Computas AS 26.01.15 Mutation Testing Tools

41. 41 © Computas AS 26.01.15 Mutation Testing Tools • Java: • PIT • Ruby: • Mutant • Heckle • PHP • Humbug • C#: • NinjaTurtles

42. 42 © Computas AS 26.01.15 PIT • Java • Junit & TestNG • Maven or command-line • Operates on byte code • Large set of mutators • Also possibly equivalent mutators available • Highly configurable • Sensible defaults • http://pitest.org/

43. 43 © Computas AS 26.01.15 PIT Mutators • Conditionals Boundary • Negate Conditionals • Remove Conditionals* • Math • Increments • Invert Negatives • Inline Constant* • Return Values • Void Method Calls • Non Void Method Calls* • Constructor Calls*

44. 44 © Computas AS 26.01.15 PIT Sample Report

45. 45 © Computas AS 26.01.15 Jester • Java • JUnit • Usually run from the command-line • Grester for Maven2 • Operates on source code • Runs all unit tests on all classes • Reporting and documentation could be better • http://jester.sourceforge.net/ • http://sourceforge.net/projects/grester/

48. 48 © Computas AS 26.01.15 Pester and Nester • Pester • Jester for Python • PyUnit • Nester • Port of Jester for C# • NUnit • Integrated with Visual Studio • But outdated… • http://nester.sourceforge.net/

50. 50 © Computas AS 26.01.15 Jumble • Java • JUnit • Run from the command-line • Operates on byte code • Runs unit tests on a class • Reporting and documentation could be better • Claims to be faster than Jester • http://jumble.sourceforge.net/index.html

51. 51 © Computas AS 26.01.15 Jumble Sample Report Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s .. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional ...... Score: 67%

52. 52 © Computas AS 26.01.15 Mutant • Ruby 1.9 up to 2.2 • Rspec2 and Rspec3 • Usually run from the command-line • Good to-the-point reporting • Good performance • OK documentation • Active project • https://github.com/mbj/mutant

53. 53 © Computas AS 26.01.15 Heckle • Ruby 1.8 • Doesn't work on Ruby 1.9 • Test::Unit and rSpec • Usually run from the command-line • Runs a set of unit tests on a class or a method • Good to-the-point reporting • Acceptable performance • Virtually no documentation • http://rubyforge.org/projects/seattlerb/ • http://docs.seattlerb.org/heckle/

54. 54 © Computas AS 26.01.15 Heckle Mutations • Booleans • Numbers • Strings • Symbols • Ranges • Regexes • Branches (if, while, unless, until)

55. 55 © Computas AS 26.01.15 Humbug • PHP 5.4 or greater • PHPUnit • Usually run from the command-line • OK documentation • Active project • https://github.com/padraic/humbug

56. 56 © Computas AS 26.01.15 NinjaTurtles • .Net: • C# • Visual Basic/VB.NET • Any other .NET language code • Doesn't seem much alive • http://www.mutation-testing.net/

57. 57 © Computas AS 26.01.15 NinjaTurtles Mutations • Sequence point deletion • Arithmetic operator substitution (*, /, +, -, %) • Bitwise operator substitution (&, |, ^) • Branch substituion (condition, always and never branch) • Conditional boundary substition (< and <=, > and >=) • Substitution of reads from variables, parameters and fields of the same type • Substitution of writes to variables of the same type

59. 59 © Computas AS 26.01.15 Experiences and Recommendations • Use mutation testing from day 1 • Start on a small code base • Keep things simple • Have small interfaces • Select a good tool • Configurable • Flexible • One that can output the mutant

60. 60 © Computas AS 26.01.15 Experiences and Recommendations (cont'd) • Believe the tool • Or try to proof that the tool is wrong • Fix the problem • Don't turn mutation testing off • Embrace the “more than 100%” test coverage • Semantic coverage • Less code • More unit tests • More intelligent unit tests

62. 62 © Computas AS 26.01.15 Improvements • Integration with more unit testing frameworks • Better unit test–source code mapping • Better heuristics • Parallellisation • Better reporting • IDE integration • Building tool integration

Oop 2015 – Mutation Testing

Oop 2015 – Mutation Testing

Recommended

More Related Content

Viewers also liked

Viewers also liked(6)

Similar to Oop 2015 – Mutation Testing

Similar to Oop 2015 – Mutation Testing(20)

More from Filip Van Laenen

More from Filip Van Laenen(17)

Recently uploaded

Recently uploaded(20)

Oop 2015 – Mutation Testing