Mutation Testing<br />Hernán Wilkinson<br />UBA - 10Pines<br />hernan.wilkinson@gmail.com<br />Nicolás Chillo<br />UBA<br ...
WhatisMutationTesting?<br />Techniquetoverifythequality of thetests<br />
WhatisMutationTesting?<br />Verify Quality of…<br />Verify Quality of…<br />Source Code <br />Tests<br />Mutation<br />Tes...
How does it work?1st Step: Create the Mutant<br />Mutation Process<br />The Source <br />Code<br />The “Mutant”<br />The M...
Examples<br />DebitCard&gt;&gt;= anotherDebitCard<br />^(type = anotherDebitCard type) <br />	and: [ number = anotherDebit...
Examples<br />Purchase&gt;&gt;netPaid<br />   ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt...
Why?How does it help?<br />
How does it work?2nd Step: Try to Kill the Mutant<br />A Killer<br />tries to kill the Mutant!<br />The “Mutant”<br />All ...
Meaning…<br />The Mutant Survives  The case generated by the mutant is not tested<br />The Mutant Dies  The case generat...
Example: Themutantsurvives<br />DebitCard&gt;&gt;= anotherDebitCard<br />   ^(type = anotherDebitCard type) and: [ number ...
Example: Themutantdies<br />DebitCard&gt;&gt;= anotherDebitCard<br />   ^(type = anotherDebitCard type) and: [ number = an...
Example: Themutantsurvives<br />Purchase&gt;&gt;netPaid<br />   ^self totalPaid – self totalRefunded<br />Change #- with #...
Example: Themutantdies<br />Purchase&gt;&gt;netPaid<br />   ^self totalPaid – self totalRefunded<br />Change #- with #+<br...
How does it work? - Summary<br /><ul><li>Changesthe original sourcecodewithspecial “operators” togenerate “Mutants”
Runthe test suite relatedtothechangedcode
Ifa test errorsorfailsKillsthemutant
Ifalltestsrun TheMutantsurvives
SurvivingMutants show nottestedcases</li></ul>The Important Thing!<br />
MuTalk<br />MutationTestingToolfor Smalltalk (Pharo and Squeak)<br />
Demo<br />
MuTalk – How does it work?<br /><ul><li>Runs the test to be sure that all run
For each method m
For each operator o
Changes mAST using o
Compiles mutated code
Changes method dictionary
Run the tests</li></li></ul><li>MuTalk – Operators<br /><ul><li>Boolean messages
Remove #not
Replace #and: with #eqv:
Replace #and: with #nand:
Replace #and: with #or:
Replace #and: with #secondArgResult:
Replace #and: with false
Replace #or: First Condition with false
Replace #or: Second Condition with false
Replace #or: with #and:
Replace #or: with #xor:</li></li></ul><li>MuTalk – Operators<br /><ul><li>Magnitude messages
Replace #'<=' with #<
Replace #'<=' with #=
Replace #'<=' with #>
Replace #'>=' with #=
Replace #'>=' with #>
Replace #'~=' with #=
Replace #< with #>
Replace #= with #'~='
Upcoming SlideShare
Loading in...5
×

Mutation Testing

2,124

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,124
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
64
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Mutation Testing

  1. 1. Mutation Testing<br />Hernán Wilkinson<br />UBA - 10Pines<br />hernan.wilkinson@gmail.com<br />Nicolás Chillo<br />UBA<br />nchillo@gmail.com<br />Gabriel Brunstein<br />UBA<br />gaboto@gmail.com<br />
  2. 2. WhatisMutationTesting?<br />Techniquetoverifythequality of thetests<br />
  3. 3. WhatisMutationTesting?<br />Verify Quality of…<br />Verify Quality of…<br />Source Code <br />Tests<br />Mutation<br />Testing<br />
  4. 4. How does it work?1st Step: Create the Mutant<br />Mutation Process<br />The Source <br />Code<br />The “Mutant”<br />The Mutation “Operator”<br />
  5. 5. Examples<br />DebitCard&gt;&gt;= anotherDebitCard<br />^(type = anotherDebitCard type) <br /> and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />CreditCard&gt;&gt;= anotherDebitCard<br />^(type = anotherDebitCard type) <br /> or: [ number = anotherDebitCard number ]<br />
  6. 6. Examples<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />
  7. 7. Why?How does it help?<br />
  8. 8. How does it work?2nd Step: Try to Kill the Mutant<br />A Killer<br />tries to kill the Mutant!<br />The “Mutant”<br />All tests run  The Mutant Survives!!!<br />The Test Suite<br />A test fails or errors  The Mutant Dies<br />
  9. 9. Meaning…<br />The Mutant Survives  The case generated by the mutant is not tested<br />The Mutant Dies  The case generated by the mutant is tested<br />
  10. 10. Example: Themutantsurvives<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]<br />DebitCardTest&gt;&gt;testDebitCardWithSameNumberShouldBeEqual<br />self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).<br />
  11. 11. Example: Themutantdies<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]<br />DebitCardTest&gt;&gt;testDebitCardWithSameNumberShouldBeEqual<br />self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).<br />DebitCardTest &gt;&gt;testDebitCardWithDifferentNumberShouldBeDifferent<br />self deny: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 789).<br />
  12. 12. Example: Themutantsurvives<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />Purchase&gt;&gt;testNetPaid<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />
  13. 13. Example: Themutantdies<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />Purchase&gt;&gt;testNetPaidWithOutRefunds Renamed!<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />Purchase&gt;&gt;testNetPaidWithRefunds<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />purchase addRefundFor: 10 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />
  14. 14. How does it work? - Summary<br /><ul><li>Changesthe original sourcecodewithspecial “operators” togenerate “Mutants”
  15. 15. Runthe test suite relatedtothechangedcode
  16. 16. Ifa test errorsorfailsKillsthemutant
  17. 17. Ifalltestsrun TheMutantsurvives
  18. 18. SurvivingMutants show nottestedcases</li></ul>The Important Thing!<br />
  19. 19. MuTalk<br />MutationTestingToolfor Smalltalk (Pharo and Squeak)<br />
  20. 20. Demo<br />
  21. 21. MuTalk – How does it work?<br /><ul><li>Runs the test to be sure that all run
  22. 22. For each method m
  23. 23. For each operator o
  24. 24. Changes mAST using o
  25. 25. Compiles mutated code
  26. 26. Changes method dictionary
  27. 27. Run the tests</li></li></ul><li>MuTalk – Operators<br /><ul><li>Boolean messages
  28. 28. Remove #not
  29. 29. Replace #and: with #eqv:
  30. 30. Replace #and: with #nand:
  31. 31. Replace #and: with #or:
  32. 32. Replace #and: with #secondArgResult:
  33. 33. Replace #and: with false
  34. 34. Replace #or: First Condition with false
  35. 35. Replace #or: Second Condition with false
  36. 36. Replace #or: with #and:
  37. 37. Replace #or: with #xor:</li></li></ul><li>MuTalk – Operators<br /><ul><li>Magnitude messages
  38. 38. Replace #'<=' with #<
  39. 39. Replace #'<=' with #=
  40. 40. Replace #'<=' with #>
  41. 41. Replace #'>=' with #=
  42. 42. Replace #'>=' with #>
  43. 43. Replace #'~=' with #=
  44. 44. Replace #< with #>
  45. 45. Replace #= with #'~='
  46. 46. Replace #> with #<
  47. 47. Replace #max: with #min:
  48. 48. Replace #min: with #max:</li></li></ul><li>MuTalk – Operators<br /><ul><li>Collection messages
  49. 49. Remove at:ifAbsent:
  50. 50. Replace #reject: with #select:
  51. 51. Replace #select: with #reject:
  52. 52. Replace Reject block with [:each | false]
  53. 53. Replace Reject block with [:each | true]
  54. 54. Replace Select block with [:each | false]
  55. 55. Replace Select block with [:each | true]
  56. 56. Replace detect: block with [:each | false] when #detect:ifNone:
  57. 57. Replace detect: block with [:each | true] when #detect:ifNone:
  58. 58. Replace do block with [:each |]
  59. 59. Replace ifNone: block with [] when #detect:ifNone:
  60. 60. Replace inject:aValueinto:aBlock with aValue
  61. 61. Replace sortBlock:aBlock with sortBlock:[:a :b| true]</li></li></ul><li>MuTalk – Operators<br /><ul><li>Number messages
  62. 62. Replace #* with #/
  63. 63. Replace #+ with #-
  64. 64. Replace #- with #+
  65. 65. Replace #/ with #*</li></li></ul><li>MuTalk – Operators<br /><ul><li>Flow control messages
  66. 66. Remove Exception Handler Operator
  67. 67. Replace #ifFalse: receiver with false
  68. 68. Replace #ifFalse: receiver with true
  69. 69. Replace #ifFalse: with #ifTrue:
  70. 70. Replace #ifFalse:IfTrue: receiver with false
  71. 71. Replace #ifFalse:IfTrue: receiver with true
  72. 72. Replace #ifTrue: receiver with false
  73. 73. Replace #ifTrue: receiver with true
  74. 74. Replace #ifTrue: with #ifFalse:
  75. 75. Replace #ifTrue:ifFalse: receiver with false
  76. 76. Replace #ifTrue:ifFalse: receiver with true</li></li></ul><li>Why is not widely used?<br />
  77. 77. Is not new … - History<br />Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs”<br />Generally accepted in 1978, R. Lipton et al, “Hints on test data selection: Help for the practicing programmer”<br />
  78. 78. Why is not widely used?<br />Maturity Problem: Because Testing is not widely used YET!<br />(Although it is increasing)<br />
  79. 79. Why is not widely used?<br />Integration Problem: Inability to successfully integrate it into the software development process<br />(TDD plays a key role now)<br />
  80. 80. Why is not widely used?<br />Technical Problem: It is a Brute Force technique!<br />
  81. 81. Technical Problems<br /><ul><li>Bruteforcetechnique</li></ul>N x M<br />N = number of tests<br />M = number of mutants<br />
  82. 82. Aconcagua<br /><ul><li>Number of Tests: 666
  83. 83. Number of Mutants: 1005
  84. 84. Time to create a mutant/compile/link/run: 10 secs. each aprox.?
  85. 85. Total time:
  86. 86. 6693300 seconds
  87. 87. 1859 hours, 15 minutes</li></li></ul><li>Another way of doing it…<br />CreditCard&gt;&gt;= anotherCreditCard<br /> ^(anotherCreditCardisKindOf: self class) and: [ number = anotherCreditCard number ]<br />CreditCard&gt;&gt;= anotherCreditCard<br />MutantId = 12 ifTrue: [ ^(anotherCreditCardisKindOf: self class) or: [ number = anotherCreditCard number ].<br />MutantId = 13 ifTrue: [ ^(anotherCreditCardisKindOf: self class) nand: [ number = anotherCreditCard number ].<br />MutantId = 14 ifTrue: [ ^(anotherCreditCardisKindOf: self class) eqv: [ number = anotherCreditCard number ].<br />
  88. 88. Aconcagua<br /><ul><li>Number of Tests: 666
  89. 89. Number of Mutants: 1005
  90. 90. Time to create the metamutant/compile/link: 2 minutes?
  91. 91. Time to run the tests per mutant: 1 sec
  92. 92. Total time:
  93. 93. 1125 seconds
  94. 94. 18 minutes 45 seconds</li></li></ul><li>MuTalk OptimizationsRunning Strategies<br />Mutate all methods, run all tests per mutant<br /><ul><li>Create a mutant for each method
  95. 95. Run all the test for each mutant
  96. 96. Disadvantage: Slower strategy</li></ul>Mutate covered methods, run all tests per mutant<br /><ul><li>Takes coverage running all tests
  97. 97. Mutate only covered methods
  98. 98. Run all methods per mutant
  99. 99. Relies on coverage</li></ul>Mutate all methods, run only test that cover mutated method<br /><ul><li>Run coverage keeping for each method the tests that covered it
  100. 100. Create a mutant for each method
  101. 101. For each mutant, run only the tests that covered the original method</li></ul>Mutate covered methods, run only test that covered mutated methods<br /><ul><li>Run coverage keeping for each method the tests that covered it
  102. 102. Create a mutant for only covered methods
  103. 103. For each mutant, run only the tests that covered the original method</li></li></ul><li>MuTalk - Aconcagua Statistics<br /><ul><li>Mutate All, Run All: 1 minute, 6 seconds
  104. 104. Mutate Covered, Run Covering: 36 seconds
  105. 105. Result:
  106. 106. 545 Killed
  107. 107. 6 Terminated
  108. 108. 83 Survived</li></li></ul><li>More Statistics<br />
  109. 109. MuTalk OptimizationsTerminated Mutants<br />Try to kill the Mutant!<br />The killer has to be<br />“Terminated”<br />The Test Suite<br />
  110. 110. MuTalk - Terminated Mutants<br /><ul><li>Take the time it runs each test the first time
  111. 111. If the test takes more thant 3 times, terminate it</li></li></ul><li>Let’s redefine MuTalk as…<br />MutationTestingToolfor Smalltalk (Pharo and Squeak) that uses meta-facilitiestorunfaster and provideinmediatefeedback<br />
  112. 112. Work in progress<br /><ul><li>Operators Categorization based on how useful they are to detect errors
  113. 113. Filter Operators on View
  114. 114. Cancel process</li></li></ul><li>Future work<br /><ul><li>Make Operators more “inteligent”
  115. 115. a = b ifTrue: [ … ]
  116. 116. a = b ifFalse: [] is equivalent to a ~= b ifTrue: []
  117. 117. Suggest tests using not killed mutants
  118. 118. Use MuTalk to test MuTalk? </li></li></ul><li>Why does it work?<br />“Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” (Coupling effect)<br />Demonstrated in 1995, K. Wah, “Fault coupling in finite bijective functions”<br />
  119. 119. Why does it work?<br />“In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault”<br />Geist et al, “Estimation and enhancement of real-time software reliability through mutation analysis”, 1992<br />
  120. 120. More Statistics…<br />
  121. 121. How does it compare to coverage?<br /><ul><li>Does not replaces coverage because some methods do not generate mutants
  122. 122. But:
  123. 123. Mutants on not covered methods will survive
  124. 124. It provides better insight than coverage
  125. 125. Method Coverage fails with long methods/conditions/loops/etc.</li></li></ul><li>Questions?<br />
  126. 126. MuTalk - MutationTestingfor Smalltalk<br />Hernán Wilkinson<br />UBA - 10Pines<br />hernan.wilkinson@gmail.com<br />Nicolás Chillo<br />UBA<br />nchillo@gmail.com<br />Gabriel Brunstein<br />UBA<br />gaboto@gmail.com<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×