Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

1,139 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,139
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

  1. 1. © Computas AS 11.03.14 Mutation Testing with PIT Filip van Laenen Booster 2014 2014-03-13
  2. 2. 2 © Computas AS 11.03.14 Agenda • Basics of mutation testing • Relation to other testing techniques • Example • Mutation testing techniques • Mutation testing tools • Personal experiences and recommendations
  3. 3. 3 © Computas AS 11.03.14 Basics of Mutation Testing
  4. 4. 4 © Computas AS 11.03.14 Mutation Testing in a Nutshell Seeking The Summoner @ The Daily WTF http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx
  5. 5. 5 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  6. 6. 6 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  7. 7. 7 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd) • Unit tests guard the source code • But who guards the guardians? • Do the unit tests cover all source code? • Lines? • Branches? • Paths? • Do the unit tests test the right things? Mutation testing tests the tests!
  8. 8. 8 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  9. 9. 9 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  10. 10. 10 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  11. 11. 11 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  12. 12. 12 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  13. 13. 13 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  14. 14. 14 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; }
  15. 15. 15 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)
  16. 16. “ 16 © Computas AS 11.03.14 Does Mutation Testing Work? In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault. Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992
  17. 17. “ 17 © Computas AS 11.03.14 Does Mutation Testing Work? (cont'd) Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults. K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995
  18. 18. 18 © Computas AS 11.03.14 Does Mutation Testing Work? (cont'd) • “Generated mutants are similar to real faults.” • Andrews, Briand, Labiche, ICSE 2005 • “Mutation testing is more powerful than statement or branch coverage.” • Walsh, Ph.D. Thesis, State University of New York at Binghampton, 1985 • “Mutation testing is superior to data flow coverage criteria.” • Frankl, Weiss, Hu, Journal of Systems and Software, 1997
  19. 19. 19 © Computas AS 11.03.14 Relation to Other Testing Techniques
  20. 20. 20 © Computas AS 11.03.14 Relation to Other Testing Techniques • Unit tests • Test-Driven Development (TDD) • Test coverage • Static code analysis • Fuzz testing
  21. 21. 21 © Computas AS 11.03.14 Relation to Other Testing Techniques
  22. 22. 22 © Computas AS 11.03.14 Is Mutation Testing New? • R. Lipton, “Fault Diagnosis of Computer Programs,” 1971 • R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978 • Historical obstacles: • No unit testing • No TDD • Inefficient implementation • Hence time-consuming • No integration with IDEs
  23. 23. 23 © Computas AS 11.03.14 Practical Example of Mutation Testing
  24. 24. 24 © Computas AS 11.03.14 Code Along… https://github.com/computas-fvl/booster2014 git clone https://github.com/computas-fvl/booster2014.git mvn clean site:site org.pitest:pitest-maven:mutationCoverage
  25. 25. 25 © Computas AS 11.03.14 Mutation Testing Techniques
  26. 26. 26 © Computas AS 11.03.14 Mutation Testing Techniques • Three aspects: • Mutation injection • Mutation types • Unit test selection per mutant • Key properties: • Efficiency • Performance
  27. 27. 27 © Computas AS 11.03.14 Mutation Injection • Source code mutation • Binary code mutation • Caveats: • De-mutation • Compilation errors • Invalid binary code
  28. 28. 28 © Computas AS 11.03.14 Mutation Types • Some mutations never change behaviour • Constants reused by unit tests • Log messages • Some mutations can change behaviour • Switching between < and ≠ • Switching between < and ≤ • Some mutations always change behaviour • Switching between < and ≥
  29. 29. 29 © Computas AS 11.03.14 Mutation Types (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; } int max(int a, int b) { return (a >= b) ? b : a; }
  30. 30. 30 © Computas AS 11.03.14 Mutation Types (cont'd) for (int i = 0; i < 10; i++) … for (int i = 0; i != 10; i++) … for (int i = 0; i >= 10; i++) …
  31. 31. 31 © Computas AS 11.03.14 Mutation Types Guaranteed to Change Behaviour • Negation of the comparison • Switching between = and ≠ • Switching between < and ≥ • Switching between > and ≤ • Negation of boolean conditions • Adding a ¬, ! or ~ • Shortcutting boolean conditions • Replacement with True or False *
  32. 32. 32 © Computas AS 11.03.14 Unit Test Selection • Goal: find the unit test that “kills” the mutant • Selection aids: • Hints • Name/package matching • Code coverage tools • Automatic learning • Other heuristics
  33. 33. 33 © Computas AS 11.03.14 Unit Test Selection (cont'd) • System: • 50 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 50 × 20 × 1ms = 1s • 10 mutants per class: • Brute-force: 10 × 50 × 1s = 6m 20s • Educated: 10 × 50 × 20 × 1ms = 10s
  34. 34. 34 © Computas AS 11.03.14 Unit Test Selection (cont'd) • System: • 500 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 500 × 20 × 1ms = 10s • 10 mutants per class: • Brute-force: 10 × 500 × 10s = 13h 53m 20s • Educated: 10 × 500 × 20 × 1ms = 1m 40s
  35. 35. 35 © Computas AS 11.03.14 Complexity • f: Number of function points • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × f² • Educated force: τ × μ × φ × f • Ideal: τ × μ × f
  36. 36. 36 © Computas AS 11.03.14 Complexity (cont'd) • c: Number of classes • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c² • Educated force: τ × μ × φ² × c • Ideal: τ × μ × φ × c
  37. 37. 37 © Computas AS 11.03.14 Loops for (int i = 0; i < 10; i++) … for (int i = 0; i < 10; i--) …
  38. 38. 38 © Computas AS 11.03.14 Infinite Loops • Terminate mutants that take too long to run • What's “too long”? • Ruins the performance • Can be hard to predict
  39. 39. 39 © Computas AS 11.03.14 Other Problems • Recursion: • Stack overflows • Out of memory exceptions • Syntax errors • Segmentation faults
  40. 40. 40 © Computas AS 11.03.14 Mutation Testing Tools
  41. 41. 41 © Computas AS 11.03.14 Mutation Testing Tools • Java: • PIT • Jester (Nester, Pester) • Jumble • Ruby: • Mutant • Heckle • C#: • NinjaTurtles
  42. 42. 42 © Computas AS 11.03.14 PIT • Java • Junit & TestNG • Maven or command-line • Operates on byte code • Large set of mutators • Also possibly equivalent mutators available • Highly configurable • Sensible defaults • http://pitest.org/
  43. 43. 43 © Computas AS 11.03.14 PIT Mutators • Conditionals Boundary • Negate Conditionals • Remove Conditionals* • Math • Increments • Invert Negatives • Inline Constant* • Return Values • Void Method Calls • Non Void Method Calls* • Constructor Calls*
  44. 44. 44 © Computas AS 11.03.14 PIT Sample Report
  45. 45. 45 © Computas AS 11.03.14 Jester • Java • JUnit • Usually run from the command-line • Grester for Maven2 • Operates on source code • Runs all unit tests on all classes • Reporting and documentation could be better • http://jester.sourceforge.net/ • http://sourceforge.net/projects/grester/
  46. 46. 46 © Computas AS 11.03.14 Jester Sample Report Overview
  47. 47. 47 © Computas AS 11.03.14 Jester Sample Detailed Report
  48. 48. 48 © Computas AS 11.03.14 Pester and Nester • Pester • Jester for Python • PyUnit • Nester • Port of Jester for C# • NUnit • Integrated with Visual Studio • But outdated… • http://nester.sourceforge.net/
  49. 49. 49 © Computas AS 11.03.14 Nester Sample Report
  50. 50. 50 © Computas AS 11.03.14 Jumble • Java • JUnit • Run from the command-line • Operates on byte code • Runs unit tests on a class • Reporting and documentation could be better • Claims to be faster than Jester • http://jumble.sourceforge.net/index.html
  51. 51. 51 © Computas AS 11.03.14 Jumble Sample Report Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s .. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional ...... Score: 67%
  52. 52. 52 © Computas AS 11.03.14 Mutant • Ruby 1.9 and 2.0 • rspec2 • Usually run from the command-line • Good to-the-point reporting • Good performance • OK documentation • Active project • https://github.com/mbj/mutant
  53. 53. 53 © Computas AS 11.03.14 Heckle • Ruby 1.8 • Doesn't work on Ruby 1.9 • Test::Unit and rSpec • Usually run from the command-line • Runs a set of unit tests on a class or a method • Good to-the-point reporting • Good performance • Virtually no documentation • http://rubyforge.org/projects/seattlerb/ • http://docs.seattlerb.org/heckle/
  54. 54. 54 © Computas AS 11.03.14 Heckle Mutations • Booleans • Numbers • Strings • Symbols • Ranges • Regexes • Branches (if, while, unless, until)
  55. 55. 55 © Computas AS 11.03.14 NinjaTurtles • .Net: • C# • Visual Basic/VB.NET • Any other .NET language code • Doesn't seem much alive • http://www.mutation-testing.net/
  56. 56. 56 © Computas AS 11.03.14 NinjaTurtles Mutations • Sequence point deletion • Arithmetic operator substitution (*, /, +, -, %) • Bitwise operator substitution (&, |, ^) • Branch substituion (condition, always and never branch) • Conditional boundary substition (< and <=, > and >=) • Substitution of reads from variables, parameters and fields of the same type • Substitution of writes to variables of the same type
  57. 57. 57 © Computas AS 11.03.14 Personal Experiences and Recommendations
  58. 58. 58 © Computas AS 11.03.14 Experiences and Recommendations • Use mutation testing from day 1 • Start on a small code base • Keep number of unit tests per class low • Have small classes • Select a good tool • Configurable • Flexible • One that can output the mutant
  59. 59. 59 © Computas AS 11.03.14 Experiences and Recommendations (cont'd) • Believe the tool • Or try to proof that the tool is wrong • Fix the problem • Don't turn mutation testing off • Embrace the “more than 100%” test coverage • Path coverage • Less code • More unit tests • More intelligent unit tests
  60. 60. 60 © Computas AS 11.03.14 Improvements
  61. 61. 61 © Computas AS 11.03.14 Improvements • Integration with more unit testing frameworks • Better unit test–source code mapping • Better heuristics • Parallellisation • Better reporting • IDE integration • Building tool integration
  62. 62. 62 © Computas AS 11.03.14 Questions? Computas AS Lysaker Torg 45, pb 482 N-1327 Lysaker NORWAY Tel +47-67 83 10 00 Fax +47-67 83 10 01 Org.nr: NO 986 352 325 MVA www.computas.com Contact: fvl@computas.com @filipvanlaenen

×