Mutation testing (OOP 2012, 2012-JAN-24)

2,950 views

Published on

My presentation about mutation testing at the OOP 2012 conference in Munich, Germany on 24 Januray 2012.

Notes:
* PIT now integrates with TestNG too

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,950
On SlideShare
0
From Embeds
0
Number of Embeds
56
Actions
Shares
0
Downloads
84
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Mutation testing (OOP 2012, 2012-JAN-24)

  1. 1. Mutation Testing Filip van Laenen OOP 2012 2012-01-24 © Computas AS 27.01.12
  2. 2. Agenda• Basics of mutation testing• Relation to other testing techniques• Example• Mutation testing techniques• Mutation testing tools• Personal experiences and recommendations• Improvements• Questions and comments 2 © Computas AS 27.01.12
  3. 3. Basics ofMutation Testing 3 © Computas AS 27.01.12
  4. 4. Mutation Testing in a Nutshell Seeking The Summoner @ The Daily WTF http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx 4 © Computas AS 27.01.12
  5. 5. Mutation Testing in a Nutshell (contd) 5 © Computas AS 27.01.12
  6. 6. Mutation Testing in a Nutshell (contd) 6 © Computas AS 27.01.12
  7. 7. Mutation Testing in a Nutshell (contd)• Unit tests guard the source code• But who guards the guardians? • Do the unit tests cover all source code? • Lines? • Branches? • Paths? • Do the unit tests test the right things? Mutation testing tests the tests! 7 © Computas AS 27.01.12
  8. 8. Mutation Testing in a Nutshell (contd) 8 © Computas AS 27.01.12
  9. 9. Mutation Testing in a Nutshell (contd) 9 © Computas AS 27.01.12
  10. 10. Mutation Testing in a Nutshell (contd) 10 © Computas AS 27.01.12
  11. 11. Mutation Testing in a Nutshell (contd) 11 © Computas AS 27.01.12
  12. 12. Mutation Testing in a Nutshell (contd) 12 © Computas AS 27.01.12
  13. 13. Mutation Testing in a Nutshell (contd) 13 © Computas AS 27.01.12
  14. 14. Mutation Testing in a Nutshell (contd)def max(a, b) { return (a < b) ? b : a;}def max(a, b) { return (a ≤ b) ? b : a;} 14 © Computas AS 27.01.12
  15. 15. Mutation Testing in a Nutshell (contd) 15 © Computas AS 27.01.12
  16. 16. Does Mutation Testing Work?“ In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault. Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992 16 © Computas AS 27.01.12
  17. 17. Does Mutation Testing Work? (contd)“ Complex faults are coupled to simple faults in such a way that a test data setthat detects all simple faults in a program will detect most complex faults. K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995 17 © Computas AS 27.01.12
  18. 18. Does Mutation Testing Work? (contd)• “Generated mutants are similar to real faults.” • Andrews, Briand, Labiche, ICSE 2005• “Mutation testing is more powerful than statement or branch coverage.” • Walsh, Ph.D. Thesis, State University of New York at Binghampton, 1985• “Mutation testing is superior to data flow coverage criteria.” • Frankl, Weiss, Hu, Journal of Systems and Software, 1997 18 © Computas AS 27.01.12
  19. 19. Relation to OtherTesting Techniques 19 © Computas AS 27.01.12
  20. 20. Relation to Other Testing Techniques• Unit tests• Test-Driven Development (TDD)• Test coverage• Static code analysis• Fuzz testing 20 © Computas AS 27.01.12
  21. 21. Relation to Other Testing Techniques 21 © Computas AS 27.01.12
  22. 22. Is Mutation Testing New?• R. Lipton, “Fault Diagnosis of Computer Programs,” 1971• R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978• Historical obstacles: • No unit testing • No TDD • Time-consuming • No integration with IDEs 22 © Computas AS 27.01.12
  23. 23. Practical Exampleof Mutation Testing 23 © Computas AS 27.01.12
  24. 24. Practical Example# Returns the maximum of a.# @param a An array of integers.def max(a) { return …;} 24 © Computas AS 27.01.12
  25. 25. Practical Example (contd)Omitted: max(null)Omitted: max([])def max(a) { return …;} 25 © Computas AS 27.01.12
  26. 26. Practical Example (contd)Assertion: max([0]) = 0def max(a) { return 0;} 26 © Computas AS 27.01.12
  27. 27. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1def max(a) { return 0;} 27 © Computas AS 27.01.12
  28. 28. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1def max(a) { return a.first;} 28 © Computas AS 27.01.12
  29. 29. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { return a.first;} 29 © Computas AS 27.01.12
  30. 30. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { nit t ests! m ← a.first; U age! cover foreach (e ∈ a) line 100% age? if (e > m) over nch c m ← e; 100% b ra return m;} 30 © Computas AS 27.01.12
  31. 31. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) m ← e; return m;} 31 © Computas AS 27.01.12
  32. 32. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (true) m ← e; return m;} 32 © Computas AS 27.01.12
  33. 33. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { return a.last;} 33 © Computas AS 27.01.12
  34. 34. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { return a.last;} 34 © Computas AS 27.01.12
  35. 35. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) m ← e; return m;} 35 © Computas AS 27.01.12
  36. 36. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { m ← -∞; foreach (e ∈ a) if (e > m) m ← e; return m;} 36 © Computas AS 27.01.12
  37. 37. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) ← Implicit else-branch! m ← e; return m;} 37 © Computas AS 27.01.12
  38. 38. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← -∞; foreach (e ∈ a) ? rage if (e > m) h cove ← Implicitcelse-branch! ran m ← e; 10 0% b return m;} 38 © Computas AS 27.01.12
  39. 39. Practical Example (contd)• 100% test coverage may be illusory • Line coverage • Branch coverage • Path coverage• TDD principles easily broken • Even if youre very careful 39 © Computas AS 27.01.12
  40. 40. Mutation Testing Techniques 40 © Computas AS 27.01.12
  41. 41. Mutation Testing Techniques• Three aspects: • Mutation injection • Mutation types • Unit test selection per mutant• Key properties: • Efficiency • Performance 41 © Computas AS 27.01.12
  42. 42. Mutation Injection• Source code mutation• Binary code mutation• Caveats: • De-mutation • Compilation errors • Invalid binary code 42 © Computas AS 27.01.12
  43. 43. Mutation Types• Some mutations never change behaviour • Constants reused by unit tests • Log messages• Some mutations can change behaviour • Switching between < and ≠ • Switching between < and ≤• Some mutations always change behaviour • Switching between < and ≥ 43 © Computas AS 27.01.12
  44. 44. Mutation Types (contd)def max(a, b) { return (a < b) ? b : a;}def max(a, b) { return (a ≤ b) ? b : a;}def max(a, b) { return (a ≥ b) ? b : a;} 44 © Computas AS 27.01.12
  45. 45. Mutation Types (contd)for (i ← 0; i < 10; i++) …for (i ← 0; i ≠ 10; i++) …for (i ← 0; i ≥ 10; i++) … 45 © Computas AS 27.01.12
  46. 46. Mutation Types Guaranteed to ChangeBehaviour *• Negation of the comparison • Switching between = and ≠ • Switching between < and ≥ • Switching between > and ≤• Negation of boolean conditions • Adding a ¬, ! or ~• Shortcutting boolean conditions • Replacement with True or False 46 © Computas AS 27.01.12
  47. 47. Unit Test Selection• Goal: find the unit test that “kills” the mutant• Selection aids: • Hints • Name/package matching • Code coverage tools • Automatic learning • Other heuristics 47 © Computas AS 27.01.12
  48. 48. Unit Test Selection (contd)• System: • 50 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 50 × 20 × 1ms = 1s• 10 mutants per class: • Brute-force: 10 × 50 × 1s = 6m 20s • Educated: 10 × 50 × 20 × 1ms = 10s 48 © Computas AS 27.01.12
  49. 49. Unit Test Selection (contd)• System: • 500 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 500 × 20 × 1ms = 10s• 10 mutants per class: • Brute-force: 10 × 500 × 10s = 13h 53m 20s • Educated: 10 × 500 × 20 × 1ms = 1m 40s 49 © Computas AS 27.01.12
  50. 50. Complexity• f: Number of function points• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1• Brute-force: (f × τ) × (f × μ) = τ × μ × f²• Educated force: τ × μ × φ × f• Ideal: τ × μ × f 50 © Computas AS 27.01.12
  51. 51. Complexity (contd)• c: Number of classes• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²• Educated force: τ × μ × φ² × c• Ideal: τ × μ × φ × c 51 © Computas AS 27.01.12
  52. 52. Loopsfor (i ← 0; i < 10; i++) …for (i ← 0; i < 10; i--) … 52 © Computas AS 27.01.12
  53. 53. Infinite Loops• Terminate mutants that take too long to run • Whats too long?• Ruins the performance• Can be hard to predict 53 © Computas AS 27.01.12
  54. 54. Other Problems• Recursion: • Stack overflows • Out of memory exceptions• Syntax errors• Segmentation faults 54 © Computas AS 27.01.12
  55. 55. Mutation Testing Tools 55 © Computas AS 27.01.12
  56. 56. Mutation Testing Tools• Ruby: Heckle• Java: • Jester • Jumble • PIT• C#: Nester• Python: Pester 56 © Computas AS 27.01.12
  57. 57. Heckle• Ruby• Test::Unit and rSpec• Usually run from the command-line• Runs a set of unit tests on a class or a method• Good to-the-point reporting• Good performance• Virtually no documentation• http://rubyforge.org/projects/seattlerb/• http://docs.seattlerb.org/heckle/ 57 © Computas AS 27.01.12
  58. 58. Heckle Mutations• Booleans• Numbers• Strings• Symbols• Ranges• Regexes• Branches (if, while, unless, until) 58 © Computas AS 27.01.12
  59. 59. Heckle Sample OutputInitial tests pass. Lets rumble.******************************************************************** Greeter#greet loaded with 3 possible mutations*****************************************************************3 mutations remaining...2 mutations remaining...1 mutations remaining...No mutants survived. Cool! 59 © Computas AS 27.01.12
  60. 60. My Heckle Sample Outputfilip@filip-laptop:~/github/wruf$ rake heckle(in /home/filip/github/wruf)Doing mutation testing on 15 method(s) of FlickrSearcher againsttest/flickr_searcher_unit_test.rb: o FlickrSearcher#convert_photo_info [1/15] o FlickrSearcher#create_form_data_to_get_info_about_photo [2/15] o FlickrSearcher#create_form_data_to_get_info_about_user [3/15] o FlickrSearcher#create_form_data_to_search_photos [4/15] o FlickrSearcher#do_rest_request [5/15] o FlickrSearcher#get_author [6/15]… o FlickrSearcher#get_photo_info [12/15] o FlickrSearcher#get_photo_url [13/15] o FlickrSearcher#get_ref_url [14/15] o FlickrSearcher#get_ref_url_from_xml_photo_info [15/15]Checked 192 mutations, and no issues were found in FlickrSearcher. 60 © Computas AS 27.01.12
  61. 61. My Heckle Sample Output (contd)Doing mutation testing on 7 method(s) of WrufSettings againsttest/wruf_settings_unit_test.rb: o WrufSettings#dimensions [1/7] o WrufSettings#dimensions= [2/7] o WrufSettings#hours [3/7] o WrufSettings#hours= [4/7] o WrufSettings#tags [5/7] o WrufSettings#tolerance [6/7] o WrufSettings#tolerance= [7/7]Checked 0 mutations, and no issues were found in WrufSettings. 61 © Computas AS 27.01.12
  62. 62. Heckle Sample Report--- original+++ mutation def calculate_precision(rescaled_number) if (rescaled_number < 9.995) then return 2 else if (rescaled_number < 99.95) then return 1 else if (rescaled_number >= 99.95) then return 0 else- return -1+ return -70 end end end end 62 © Computas AS 27.01.12
  63. 63. Jester• Java• JUnit• Usually run from the command-line • Grester for Maven2• Operates on source code• Runs all unit tests on all classes• Reporting and documentation could be better• http://jester.sourceforge.net/• http://sourceforge.net/projects/grester/ 63 © Computas AS 27.01.12
  64. 64. Jester Sample Report Overview 64 © Computas AS 27.01.12
  65. 65. Jester Sample Detailed Report 65 © Computas AS 27.01.12
  66. 66. Pester and Nester• Pester • Jester for Python • PyUnit• Nester • Port of Jester for C# • NUnit • Integrated with Visual Studio • But outdated… • http://nester.sourceforge.net/ 66 © Computas AS 27.01.12
  67. 67. Nester Sample Report 67 © Computas AS 27.01.12
  68. 68. Jumble• Java• JUnit• Run from the command-line• Operates on byte code• Runs unit tests on a class• Reporting and documentation could be better• Claims to be faster than Jester• http://jumble.sourceforge.net/index.html 68 © Computas AS 27.01.12
  69. 69. Jumble Sample ReportMutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s..M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67% 69 © Computas AS 27.01.12
  70. 70. PIT• Java• JUnit• Maven or command-line• Operates on byte code• Large set of mutators • Also possibly equivalent mutators available• Highly configurable• Sensible defaults• http://pitest.org/ 70 © Computas AS 27.01.12
  71. 71. PIT Mutators• Conditionals Boundary• Negate Conditionals• Math• Increments• Invert Negatives• Inline Constant*• Return Values• Void Method Call• Non Void Method Call*• Constructor Call* 71 © Computas AS 27.01.12
  72. 72. PIT Sample Report 72 © Computas AS 27.01.12
  73. 73. Personal Experiencesand Recommendations 73 © Computas AS 27.01.12
  74. 74. Experiences and Recommendations• Use mutation testing from day 1 • Start on a small code base• Keep number of unit tests per class low • Have small classes• Select a good tool • Configurable • Flexible • One that can output the mutant 74 © Computas AS 27.01.12
  75. 75. Experiences and Recommendations(contd)• Believe the tool • Or try to proof that the tool is wrong• Fix the problem • Dont turn mutation testing off• Embrace the “more than 100%” test coverage • Path coverage • Less code • More unit tests • More intelligent unit tests 75 © Computas AS 27.01.12
  76. 76. Improvements 76 © Computas AS 27.01.12
  77. 77. Improvements• Integration with more unit testing frameworks• Better unit test–source code mapping • Better heuristics• Parallellisation• Better reporting• IDE integration• Building tool integration 77 © Computas AS 27.01.12
  78. 78. The Ideal Mutation Testing Tool™• Easy integration with building tools• Easy integration with IDEs• Support for all unit testing frameworks• Human-aided unit test selection heuristics• Full parallellisation• Good source code mutation reporting 78 © Computas AS 27.01.12
  79. 79. Questions?Contact: fvl@computas.com @filipvanlaenen Computas AS Tel +47-67 83 10 00 Lysaker Torg 45, pb 482 Fax +47-67 83 10 01 N-1327 Lysaker Org.nr: NO 986 352 325 MVA NORWAY www.computas.com 79 © Computas AS 27.01.12

×