Upcoming SlideShare
×

# Mutation Testing

2,124

Published on

2 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total Views
2,124
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
64
0
Likes
2
Embeds 0
No embeds

No notes for slide

### Mutation Testing

1. 1. Mutation Testing<br />Hernán Wilkinson<br />UBA - 10Pines<br />hernan.wilkinson@gmail.com<br />Nicolás Chillo<br />UBA<br />nchillo@gmail.com<br />Gabriel Brunstein<br />UBA<br />gaboto@gmail.com<br />
2. 2. WhatisMutationTesting?<br />Techniquetoverifythequality of thetests<br />
3. 3. WhatisMutationTesting?<br />Verify Quality of…<br />Verify Quality of…<br />Source Code <br />Tests<br />Mutation<br />Testing<br />
4. 4. How does it work?1st Step: Create the Mutant<br />Mutation Process<br />The Source <br />Code<br />The “Mutant”<br />The Mutation “Operator”<br />
5. 5. Examples<br />DebitCard&gt;&gt;= anotherDebitCard<br />^(type = anotherDebitCard type) <br /> and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />CreditCard&gt;&gt;= anotherDebitCard<br />^(type = anotherDebitCard type) <br /> or: [ number = anotherDebitCard number ]<br />
6. 6. Examples<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />
7. 7. Why?How does it help?<br />
8. 8. How does it work?2nd Step: Try to Kill the Mutant<br />A Killer<br />tries to kill the Mutant!<br />The “Mutant”<br />All tests run  The Mutant Survives!!!<br />The Test Suite<br />A test fails or errors  The Mutant Dies<br />
9. 9. Meaning…<br />The Mutant Survives  The case generated by the mutant is not tested<br />The Mutant Dies  The case generated by the mutant is tested<br />
10. 10. Example: Themutantsurvives<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]<br />DebitCardTest&gt;&gt;testDebitCardWithSameNumberShouldBeEqual<br />self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).<br />
11. 11. Example: Themutantdies<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]<br />Operator: Change #and: by #or:<br />DebitCard&gt;&gt;= anotherDebitCard<br /> ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]<br />DebitCardTest&gt;&gt;testDebitCardWithSameNumberShouldBeEqual<br />self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).<br />DebitCardTest &gt;&gt;testDebitCardWithDifferentNumberShouldBeDifferent<br />self deny: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 789).<br />
12. 12. Example: Themutantsurvives<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />Purchase&gt;&gt;testNetPaid<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />
13. 13. Example: Themutantdies<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid – self totalRefunded<br />Change #- with #+<br />Purchase&gt;&gt;netPaid<br /> ^self totalPaid + self totalRefunded<br />Purchase&gt;&gt;testNetPaidWithOutRefunds Renamed!<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />Purchase&gt;&gt;testNetPaidWithRefunds<br />| purchase |<br />purchase := Purchase for: 20 * euros.<br />purchase addRefundFor: 10 * euros.<br />self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)<br />
14. 14. How does it work? - Summary<br /><ul><li>Changesthe original sourcecodewithspecial “operators” togenerate “Mutants”
15. 15. Runthe test suite relatedtothechangedcode
16. 16. Ifa test errorsorfailsKillsthemutant
17. 17. Ifalltestsrun TheMutantsurvives
18. 18. SurvivingMutants show nottestedcases</li></ul>The Important Thing!<br />
19. 19. MuTalk<br />MutationTestingToolfor Smalltalk (Pharo and Squeak)<br />
20. 20. Demo<br />
21. 21. MuTalk – How does it work?<br /><ul><li>Runs the test to be sure that all run
22. 22. For each method m
23. 23. For each operator o
24. 24. Changes mAST using o
25. 25. Compiles mutated code
26. 26. Changes method dictionary
27. 27. Run the tests</li></li></ul><li>MuTalk – Operators<br /><ul><li>Boolean messages
28. 28. Remove #not
29. 29. Replace #and: with #eqv:
30. 30. Replace #and: with #nand:
31. 31. Replace #and: with #or:
32. 32. Replace #and: with #secondArgResult:
33. 33. Replace #and: with false
34. 34. Replace #or: First Condition with false
35. 35. Replace #or: Second Condition with false
36. 36. Replace #or: with #and:
37. 37. Replace #or: with #xor:</li></li></ul><li>MuTalk – Operators<br /><ul><li>Magnitude messages
38. 38. Replace #'<=' with #<
39. 39. Replace #'<=' with #=
40. 40. Replace #'<=' with #>
41. 41. Replace #'>=' with #=
42. 42. Replace #'>=' with #>
43. 43. Replace #'~=' with #=
44. 44. Replace #< with #>
45. 45. Replace #= with #'~='
46. 46. Replace #> with #<
47. 47. Replace #max: with #min:
48. 48. Replace #min: with #max:</li></li></ul><li>MuTalk – Operators<br /><ul><li>Collection messages
49. 49. Remove at:ifAbsent:
50. 50. Replace #reject: with #select:
51. 51. Replace #select: with #reject:
52. 52. Replace Reject block with [:each | false]
53. 53. Replace Reject block with [:each | true]
54. 54. Replace Select block with [:each | false]
55. 55. Replace Select block with [:each | true]
56. 56. Replace detect: block with [:each | false] when #detect:ifNone:
57. 57. Replace detect: block with [:each | true] when #detect:ifNone:
58. 58. Replace do block with [:each |]
59. 59. Replace ifNone: block with [] when #detect:ifNone:
60. 60. Replace inject:aValueinto:aBlock with aValue
61. 61. Replace sortBlock:aBlock with sortBlock:[:a :b| true]</li></li></ul><li>MuTalk – Operators<br /><ul><li>Number messages
62. 62. Replace #* with #/
63. 63. Replace #+ with #-
64. 64. Replace #- with #+
65. 65. Replace #/ with #*</li></li></ul><li>MuTalk – Operators<br /><ul><li>Flow control messages
66. 66. Remove Exception Handler Operator
67. 67. Replace #ifFalse: receiver with false
68. 68. Replace #ifFalse: receiver with true
69. 69. Replace #ifFalse: with #ifTrue:
70. 70. Replace #ifFalse:IfTrue: receiver with false
71. 71. Replace #ifFalse:IfTrue: receiver with true
72. 72. Replace #ifTrue: receiver with false
73. 73. Replace #ifTrue: receiver with true
74. 74. Replace #ifTrue: with #ifFalse:
75. 75. Replace #ifTrue:ifFalse: receiver with false
76. 76. Replace #ifTrue:ifFalse: receiver with true</li></li></ul><li>Why is not widely used?<br />
77. 77. Is not new … - History<br />Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs”<br />Generally accepted in 1978, R. Lipton et al, “Hints on test data selection: Help for the practicing programmer”<br />
78. 78. Why is not widely used?<br />Maturity Problem: Because Testing is not widely used YET!<br />(Although it is increasing)<br />
79. 79. Why is not widely used?<br />Integration Problem: Inability to successfully integrate it into the software development process<br />(TDD plays a key role now)<br />
80. 80. Why is not widely used?<br />Technical Problem: It is a Brute Force technique!<br />
81. 81. Technical Problems<br /><ul><li>Bruteforcetechnique</li></ul>N x M<br />N = number of tests<br />M = number of mutants<br />
82. 82. Aconcagua<br /><ul><li>Number of Tests: 666
83. 83. Number of Mutants: 1005
84. 84. Time to create a mutant/compile/link/run: 10 secs. each aprox.?
85. 85. Total time:
86. 86. 6693300 seconds
87. 87. 1859 hours, 15 minutes</li></li></ul><li>Another way of doing it…<br />CreditCard&gt;&gt;= anotherCreditCard<br /> ^(anotherCreditCardisKindOf: self class) and: [ number = anotherCreditCard number ]<br />CreditCard&gt;&gt;= anotherCreditCard<br />MutantId = 12 ifTrue: [ ^(anotherCreditCardisKindOf: self class) or: [ number = anotherCreditCard number ].<br />MutantId = 13 ifTrue: [ ^(anotherCreditCardisKindOf: self class) nand: [ number = anotherCreditCard number ].<br />MutantId = 14 ifTrue: [ ^(anotherCreditCardisKindOf: self class) eqv: [ number = anotherCreditCard number ].<br />
88. 88. Aconcagua<br /><ul><li>Number of Tests: 666
89. 89. Number of Mutants: 1005
90. 90. Time to create the metamutant/compile/link: 2 minutes?
91. 91. Time to run the tests per mutant: 1 sec
92. 92. Total time:
93. 93. 1125 seconds
94. 94. 18 minutes 45 seconds</li></li></ul><li>MuTalk OptimizationsRunning Strategies<br />Mutate all methods, run all tests per mutant<br /><ul><li>Create a mutant for each method
95. 95. Run all the test for each mutant
96. 96. Disadvantage: Slower strategy</li></ul>Mutate covered methods, run all tests per mutant<br /><ul><li>Takes coverage running all tests
97. 97. Mutate only covered methods
98. 98. Run all methods per mutant
99. 99. Relies on coverage</li></ul>Mutate all methods, run only test that cover mutated method<br /><ul><li>Run coverage keeping for each method the tests that covered it
100. 100. Create a mutant for each method
101. 101. For each mutant, run only the tests that covered the original method</li></ul>Mutate covered methods, run only test that covered mutated methods<br /><ul><li>Run coverage keeping for each method the tests that covered it
102. 102. Create a mutant for only covered methods
103. 103. For each mutant, run only the tests that covered the original method</li></li></ul><li>MuTalk - Aconcagua Statistics<br /><ul><li>Mutate All, Run All: 1 minute, 6 seconds
104. 104. Mutate Covered, Run Covering: 36 seconds
105. 105. Result:
106. 106. 545 Killed
107. 107. 6 Terminated
108. 108. 83 Survived</li></li></ul><li>More Statistics<br />
109. 109. MuTalk OptimizationsTerminated Mutants<br />Try to kill the Mutant!<br />The killer has to be<br />“Terminated”<br />The Test Suite<br />
110. 110. MuTalk - Terminated Mutants<br /><ul><li>Take the time it runs each test the first time
111. 111. If the test takes more thant 3 times, terminate it</li></li></ul><li>Let’s redefine MuTalk as…<br />MutationTestingToolfor Smalltalk (Pharo and Squeak) that uses meta-facilitiestorunfaster and provideinmediatefeedback<br />
112. 112. Work in progress<br /><ul><li>Operators Categorization based on how useful they are to detect errors
113. 113. Filter Operators on View
114. 114. Cancel process</li></li></ul><li>Future work<br /><ul><li>Make Operators more “inteligent”
115. 115. a = b ifTrue: [ … ]
116. 116. a = b ifFalse: [] is equivalent to a ~= b ifTrue: []
117. 117. Suggest tests using not killed mutants
118. 118. Use MuTalk to test MuTalk? </li></li></ul><li>Why does it work?<br />“Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” (Coupling effect)<br />Demonstrated in 1995, K. Wah, “Fault coupling in finite bijective functions”<br />
119. 119. Why does it work?<br />“In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault”<br />Geist et al, “Estimation and enhancement of real-time software reliability through mutation analysis”, 1992<br />
120. 120. More Statistics…<br />
121. 121. How does it compare to coverage?<br /><ul><li>Does not replaces coverage because some methods do not generate mutants
122. 122. But:
123. 123. Mutants on not covered methods will survive
124. 124. It provides better insight than coverage
125. 125. Method Coverage fails with long methods/conditions/loops/etc.</li></li></ul><li>Questions?<br />
126. 126. MuTalk - MutationTestingfor Smalltalk<br />Hernán Wilkinson<br />UBA - 10Pines<br />hernan.wilkinson@gmail.com<br />Nicolás Chillo<br />UBA<br />nchillo@gmail.com<br />Gabriel Brunstein<br />UBA<br />gaboto@gmail.com<br />
1. #### A particular slide catching your eye?

Clipping is a handy way to collect important slides you want to go back to later.