Your SlideShare is downloading. ×
  • Like
Mutation testing (OOP 2012, 2012-JAN-24)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Mutation testing (OOP 2012, 2012-JAN-24)

  • 1,781 views
Published

My presentation about mutation testing at the OOP 2012 conference in Munich, Germany on 24 Januray 2012. …

My presentation about mutation testing at the OOP 2012 conference in Munich, Germany on 24 Januray 2012.

Notes:
* PIT now integrates with TestNG too

Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,781
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
47
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Mutation Testing Filip van Laenen OOP 2012 2012-01-24 © Computas AS 27.01.12
  • 2. Agenda• Basics of mutation testing• Relation to other testing techniques• Example• Mutation testing techniques• Mutation testing tools• Personal experiences and recommendations• Improvements• Questions and comments 2 © Computas AS 27.01.12
  • 3. Basics ofMutation Testing 3 © Computas AS 27.01.12
  • 4. Mutation Testing in a Nutshell Seeking The Summoner @ The Daily WTF http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx 4 © Computas AS 27.01.12
  • 5. Mutation Testing in a Nutshell (contd) 5 © Computas AS 27.01.12
  • 6. Mutation Testing in a Nutshell (contd) 6 © Computas AS 27.01.12
  • 7. Mutation Testing in a Nutshell (contd)• Unit tests guard the source code• But who guards the guardians? • Do the unit tests cover all source code? • Lines? • Branches? • Paths? • Do the unit tests test the right things? Mutation testing tests the tests! 7 © Computas AS 27.01.12
  • 8. Mutation Testing in a Nutshell (contd) 8 © Computas AS 27.01.12
  • 9. Mutation Testing in a Nutshell (contd) 9 © Computas AS 27.01.12
  • 10. Mutation Testing in a Nutshell (contd) 10 © Computas AS 27.01.12
  • 11. Mutation Testing in a Nutshell (contd) 11 © Computas AS 27.01.12
  • 12. Mutation Testing in a Nutshell (contd) 12 © Computas AS 27.01.12
  • 13. Mutation Testing in a Nutshell (contd) 13 © Computas AS 27.01.12
  • 14. Mutation Testing in a Nutshell (contd)def max(a, b) { return (a < b) ? b : a;}def max(a, b) { return (a ≤ b) ? b : a;} 14 © Computas AS 27.01.12
  • 15. Mutation Testing in a Nutshell (contd) 15 © Computas AS 27.01.12
  • 16. Does Mutation Testing Work?“ In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault. Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992 16 © Computas AS 27.01.12
  • 17. Does Mutation Testing Work? (contd)“ Complex faults are coupled to simple faults in such a way that a test data setthat detects all simple faults in a program will detect most complex faults. K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995 17 © Computas AS 27.01.12
  • 18. Does Mutation Testing Work? (contd)• “Generated mutants are similar to real faults.” • Andrews, Briand, Labiche, ICSE 2005• “Mutation testing is more powerful than statement or branch coverage.” • Walsh, Ph.D. Thesis, State University of New York at Binghampton, 1985• “Mutation testing is superior to data flow coverage criteria.” • Frankl, Weiss, Hu, Journal of Systems and Software, 1997 18 © Computas AS 27.01.12
  • 19. Relation to OtherTesting Techniques 19 © Computas AS 27.01.12
  • 20. Relation to Other Testing Techniques• Unit tests• Test-Driven Development (TDD)• Test coverage• Static code analysis• Fuzz testing 20 © Computas AS 27.01.12
  • 21. Relation to Other Testing Techniques 21 © Computas AS 27.01.12
  • 22. Is Mutation Testing New?• R. Lipton, “Fault Diagnosis of Computer Programs,” 1971• R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978• Historical obstacles: • No unit testing • No TDD • Time-consuming • No integration with IDEs 22 © Computas AS 27.01.12
  • 23. Practical Exampleof Mutation Testing 23 © Computas AS 27.01.12
  • 24. Practical Example# Returns the maximum of a.# @param a An array of integers.def max(a) { return …;} 24 © Computas AS 27.01.12
  • 25. Practical Example (contd)Omitted: max(null)Omitted: max([])def max(a) { return …;} 25 © Computas AS 27.01.12
  • 26. Practical Example (contd)Assertion: max([0]) = 0def max(a) { return 0;} 26 © Computas AS 27.01.12
  • 27. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1def max(a) { return 0;} 27 © Computas AS 27.01.12
  • 28. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1def max(a) { return a.first;} 28 © Computas AS 27.01.12
  • 29. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { return a.first;} 29 © Computas AS 27.01.12
  • 30. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { nit t ests! m ← a.first; U age! cover foreach (e ∈ a) line 100% age? if (e > m) over nch c m ← e; 100% b ra return m;} 30 © Computas AS 27.01.12
  • 31. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) m ← e; return m;} 31 © Computas AS 27.01.12
  • 32. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (true) m ← e; return m;} 32 © Computas AS 27.01.12
  • 33. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { return a.last;} 33 © Computas AS 27.01.12
  • 34. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { return a.last;} 34 © Computas AS 27.01.12
  • 35. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) m ← e; return m;} 35 © Computas AS 27.01.12
  • 36. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2Assertion: max([2, 1]) = 2def max(a) { m ← -∞; foreach (e ∈ a) if (e > m) m ← e; return m;} 36 © Computas AS 27.01.12
  • 37. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← a.first; foreach (e ∈ a) if (e > m) ← Implicit else-branch! m ← e; return m;} 37 © Computas AS 27.01.12
  • 38. Practical Example (contd)Assertion: max([0]) = 0Assertion: max([1]) = 1Assertion: max([1, 2]) = 2def max(a) { m ← -∞; foreach (e ∈ a) ? rage if (e > m) h cove ← Implicitcelse-branch! ran m ← e; 10 0% b return m;} 38 © Computas AS 27.01.12
  • 39. Practical Example (contd)• 100% test coverage may be illusory • Line coverage • Branch coverage • Path coverage• TDD principles easily broken • Even if youre very careful 39 © Computas AS 27.01.12
  • 40. Mutation Testing Techniques 40 © Computas AS 27.01.12
  • 41. Mutation Testing Techniques• Three aspects: • Mutation injection • Mutation types • Unit test selection per mutant• Key properties: • Efficiency • Performance 41 © Computas AS 27.01.12
  • 42. Mutation Injection• Source code mutation• Binary code mutation• Caveats: • De-mutation • Compilation errors • Invalid binary code 42 © Computas AS 27.01.12
  • 43. Mutation Types• Some mutations never change behaviour • Constants reused by unit tests • Log messages• Some mutations can change behaviour • Switching between < and ≠ • Switching between < and ≤• Some mutations always change behaviour • Switching between < and ≥ 43 © Computas AS 27.01.12
  • 44. Mutation Types (contd)def max(a, b) { return (a < b) ? b : a;}def max(a, b) { return (a ≤ b) ? b : a;}def max(a, b) { return (a ≥ b) ? b : a;} 44 © Computas AS 27.01.12
  • 45. Mutation Types (contd)for (i ← 0; i < 10; i++) …for (i ← 0; i ≠ 10; i++) …for (i ← 0; i ≥ 10; i++) … 45 © Computas AS 27.01.12
  • 46. Mutation Types Guaranteed to ChangeBehaviour *• Negation of the comparison • Switching between = and ≠ • Switching between < and ≥ • Switching between > and ≤• Negation of boolean conditions • Adding a ¬, ! or ~• Shortcutting boolean conditions • Replacement with True or False 46 © Computas AS 27.01.12
  • 47. Unit Test Selection• Goal: find the unit test that “kills” the mutant• Selection aids: • Hints • Name/package matching • Code coverage tools • Automatic learning • Other heuristics 47 © Computas AS 27.01.12
  • 48. Unit Test Selection (contd)• System: • 50 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 50 × 20 × 1ms = 1s• 10 mutants per class: • Brute-force: 10 × 50 × 1s = 6m 20s • Educated: 10 × 50 × 20 × 1ms = 10s 48 © Computas AS 27.01.12
  • 49. Unit Test Selection (contd)• System: • 500 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 500 × 20 × 1ms = 10s• 10 mutants per class: • Brute-force: 10 × 500 × 10s = 13h 53m 20s • Educated: 10 × 500 × 20 × 1ms = 1m 40s 49 © Computas AS 27.01.12
  • 50. Complexity• f: Number of function points• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1• Brute-force: (f × τ) × (f × μ) = τ × μ × f²• Educated force: τ × μ × φ × f• Ideal: τ × μ × f 50 © Computas AS 27.01.12
  • 51. Complexity (contd)• c: Number of classes• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²• Educated force: τ × μ × φ² × c• Ideal: τ × μ × φ × c 51 © Computas AS 27.01.12
  • 52. Loopsfor (i ← 0; i < 10; i++) …for (i ← 0; i < 10; i--) … 52 © Computas AS 27.01.12
  • 53. Infinite Loops• Terminate mutants that take too long to run • Whats too long?• Ruins the performance• Can be hard to predict 53 © Computas AS 27.01.12
  • 54. Other Problems• Recursion: • Stack overflows • Out of memory exceptions• Syntax errors• Segmentation faults 54 © Computas AS 27.01.12
  • 55. Mutation Testing Tools 55 © Computas AS 27.01.12
  • 56. Mutation Testing Tools• Ruby: Heckle• Java: • Jester • Jumble • PIT• C#: Nester• Python: Pester 56 © Computas AS 27.01.12
  • 57. Heckle• Ruby• Test::Unit and rSpec• Usually run from the command-line• Runs a set of unit tests on a class or a method• Good to-the-point reporting• Good performance• Virtually no documentation• http://rubyforge.org/projects/seattlerb/• http://docs.seattlerb.org/heckle/ 57 © Computas AS 27.01.12
  • 58. Heckle Mutations• Booleans• Numbers• Strings• Symbols• Ranges• Regexes• Branches (if, while, unless, until) 58 © Computas AS 27.01.12
  • 59. Heckle Sample OutputInitial tests pass. Lets rumble.******************************************************************** Greeter#greet loaded with 3 possible mutations*****************************************************************3 mutations remaining...2 mutations remaining...1 mutations remaining...No mutants survived. Cool! 59 © Computas AS 27.01.12
  • 60. My Heckle Sample Outputfilip@filip-laptop:~/github/wruf$ rake heckle(in /home/filip/github/wruf)Doing mutation testing on 15 method(s) of FlickrSearcher againsttest/flickr_searcher_unit_test.rb: o FlickrSearcher#convert_photo_info [1/15] o FlickrSearcher#create_form_data_to_get_info_about_photo [2/15] o FlickrSearcher#create_form_data_to_get_info_about_user [3/15] o FlickrSearcher#create_form_data_to_search_photos [4/15] o FlickrSearcher#do_rest_request [5/15] o FlickrSearcher#get_author [6/15]… o FlickrSearcher#get_photo_info [12/15] o FlickrSearcher#get_photo_url [13/15] o FlickrSearcher#get_ref_url [14/15] o FlickrSearcher#get_ref_url_from_xml_photo_info [15/15]Checked 192 mutations, and no issues were found in FlickrSearcher. 60 © Computas AS 27.01.12
  • 61. My Heckle Sample Output (contd)Doing mutation testing on 7 method(s) of WrufSettings againsttest/wruf_settings_unit_test.rb: o WrufSettings#dimensions [1/7] o WrufSettings#dimensions= [2/7] o WrufSettings#hours [3/7] o WrufSettings#hours= [4/7] o WrufSettings#tags [5/7] o WrufSettings#tolerance [6/7] o WrufSettings#tolerance= [7/7]Checked 0 mutations, and no issues were found in WrufSettings. 61 © Computas AS 27.01.12
  • 62. Heckle Sample Report--- original+++ mutation def calculate_precision(rescaled_number) if (rescaled_number < 9.995) then return 2 else if (rescaled_number < 99.95) then return 1 else if (rescaled_number >= 99.95) then return 0 else- return -1+ return -70 end end end end 62 © Computas AS 27.01.12
  • 63. Jester• Java• JUnit• Usually run from the command-line • Grester for Maven2• Operates on source code• Runs all unit tests on all classes• Reporting and documentation could be better• http://jester.sourceforge.net/• http://sourceforge.net/projects/grester/ 63 © Computas AS 27.01.12
  • 64. Jester Sample Report Overview 64 © Computas AS 27.01.12
  • 65. Jester Sample Detailed Report 65 © Computas AS 27.01.12
  • 66. Pester and Nester• Pester • Jester for Python • PyUnit• Nester • Port of Jester for C# • NUnit • Integrated with Visual Studio • But outdated… • http://nester.sourceforge.net/ 66 © Computas AS 27.01.12
  • 67. Nester Sample Report 67 © Computas AS 27.01.12
  • 68. Jumble• Java• JUnit• Run from the command-line• Operates on byte code• Runs unit tests on a class• Reporting and documentation could be better• Claims to be faster than Jester• http://jumble.sourceforge.net/index.html 68 © Computas AS 27.01.12
  • 69. Jumble Sample ReportMutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s..M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67% 69 © Computas AS 27.01.12
  • 70. PIT• Java• JUnit• Maven or command-line• Operates on byte code• Large set of mutators • Also possibly equivalent mutators available• Highly configurable• Sensible defaults• http://pitest.org/ 70 © Computas AS 27.01.12
  • 71. PIT Mutators• Conditionals Boundary• Negate Conditionals• Math• Increments• Invert Negatives• Inline Constant*• Return Values• Void Method Call• Non Void Method Call*• Constructor Call* 71 © Computas AS 27.01.12
  • 72. PIT Sample Report 72 © Computas AS 27.01.12
  • 73. Personal Experiencesand Recommendations 73 © Computas AS 27.01.12
  • 74. Experiences and Recommendations• Use mutation testing from day 1 • Start on a small code base• Keep number of unit tests per class low • Have small classes• Select a good tool • Configurable • Flexible • One that can output the mutant 74 © Computas AS 27.01.12
  • 75. Experiences and Recommendations(contd)• Believe the tool • Or try to proof that the tool is wrong• Fix the problem • Dont turn mutation testing off• Embrace the “more than 100%” test coverage • Path coverage • Less code • More unit tests • More intelligent unit tests 75 © Computas AS 27.01.12
  • 76. Improvements 76 © Computas AS 27.01.12
  • 77. Improvements• Integration with more unit testing frameworks• Better unit test–source code mapping • Better heuristics• Parallellisation• Better reporting• IDE integration• Building tool integration 77 © Computas AS 27.01.12
  • 78. The Ideal Mutation Testing Tool™• Easy integration with building tools• Easy integration with IDEs• Support for all unit testing frameworks• Human-aided unit test selection heuristics• Full parallellisation• Good source code mutation reporting 78 © Computas AS 27.01.12
  • 79. Questions?Contact: fvl@computas.com @filipvanlaenen Computas AS Tel +47-67 83 10 00 Lysaker Torg 45, pb 482 Fax +47-67 83 10 01 N-1327 Lysaker Org.nr: NO 986 352 325 MVA NORWAY www.computas.com 79 © Computas AS 27.01.12