• Like
Mutation Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Mutation Testing

  • 1,936 views
Published

 

Published in Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,936
On SlideShare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
54
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Mutation Testing
    Hernán Wilkinson
    UBA - 10Pines
    hernan.wilkinson@gmail.com
    Nicolás Chillo
    UBA
    nchillo@gmail.com
    Gabriel Brunstein
    UBA
    gaboto@gmail.com
  • 2. WhatisMutationTesting?
    Techniquetoverifythequality of thetests
  • 3. WhatisMutationTesting?
    Verify Quality of…
    Verify Quality of…
    Source Code
    Tests
    Mutation
    Testing
  • 4. How does it work?1st Step: Create the Mutant
    Mutation Process
    The Source
    Code
    The “Mutant”
    The Mutation “Operator”
  • 5. Examples
    DebitCard>>= anotherDebitCard
    ^(type = anotherDebitCard type)
    and: [ number = anotherDebitCard number ]
    Operator: Change #and: by #or:
    CreditCard>>= anotherDebitCard
    ^(type = anotherDebitCard type)
    or: [ number = anotherDebitCard number ]
  • 6. Examples
    Purchase>>netPaid
    ^self totalPaid – self totalRefunded
    Change #- with #+
    Purchase>>netPaid
    ^self totalPaid + self totalRefunded
  • 7. Why?How does it help?
  • 8. How does it work?2nd Step: Try to Kill the Mutant
    A Killer
    tries to kill the Mutant!
    The “Mutant”
    All tests run  The Mutant Survives!!!
    The Test Suite
    A test fails or errors  The Mutant Dies
  • 9. Meaning…
    The Mutant Survives  The case generated by the mutant is not tested
    The Mutant Dies  The case generated by the mutant is tested
  • 10. Example: Themutantsurvives
    DebitCard>>= anotherDebitCard
    ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]
    Operator: Change #and: by #or:
    DebitCard>>= anotherDebitCard
    ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]
    DebitCardTest>>testDebitCardWithSameNumberShouldBeEqual
    self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).
  • 11. Example: Themutantdies
    DebitCard>>= anotherDebitCard
    ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ]
    Operator: Change #and: by #or:
    DebitCard>>= anotherDebitCard
    ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]
    DebitCardTest>>testDebitCardWithSameNumberShouldBeEqual
    self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).
    DebitCardTest >>testDebitCardWithDifferentNumberShouldBeDifferent
    self deny: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 789).
  • 12. Example: Themutantsurvives
    Purchase>>netPaid
    ^self totalPaid – self totalRefunded
    Change #- with #+
    Purchase>>netPaid
    ^self totalPaid + self totalRefunded
    Purchase>>testNetPaid
    | purchase |
    purchase := Purchase for: 20 * euros.
    self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)
  • 13. Example: Themutantdies
    Purchase>>netPaid
    ^self totalPaid – self totalRefunded
    Change #- with #+
    Purchase>>netPaid
    ^self totalPaid + self totalRefunded
    Purchase>>testNetPaidWithOutRefunds Renamed!
    | purchase |
    purchase := Purchase for: 20 * euros.
    self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)
    Purchase>>testNetPaidWithRefunds
    | purchase |
    purchase := Purchase for: 20 * euros.
    purchase addRefundFor: 10 * euros.
    self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)
  • 14. How does it work? - Summary
    • Changesthe original sourcecodewithspecial “operators” togenerate “Mutants”
    • 15. Runthe test suite relatedtothechangedcode
    • 16. Ifa test errorsorfailsKillsthemutant
    • 17. Ifalltestsrun TheMutantsurvives
    • 18. SurvivingMutants show nottestedcases
    The Important Thing!
  • 19. MuTalk
    MutationTestingToolfor Smalltalk (Pharo and Squeak)
  • 20. Demo
  • 21. MuTalk – How does it work?
    • Runs the test to be sure that all run
    • 22. For each method m
    • 23. For each operator o
    • 24. Changes mAST using o
    • 25. Compiles mutated code
    • 26. Changes method dictionary
    • 27. Run the tests
  • MuTalk – Operators
    • Boolean messages
    • 28. Remove #not
    • 29. Replace #and: with #eqv:
    • 30. Replace #and: with #nand:
    • 31. Replace #and: with #or:
    • 32. Replace #and: with #secondArgResult:
    • 33. Replace #and: with false
    • 34. Replace #or: First Condition with false
    • 35. Replace #or: Second Condition with false
    • 36. Replace #or: with #and:
    • 37. Replace #or: with #xor:
  • MuTalk – Operators
    • Magnitude messages
    • 38. Replace #'<=' with #<
    • 39. Replace #'<=' with #=
    • 40. Replace #'<=' with #>
    • 41. Replace #'>=' with #=
    • 42. Replace #'>=' with #>
    • 43. Replace #'~=' with #=
    • 44. Replace #< with #>
    • 45. Replace #= with #'~='
    • 46. Replace #> with #<
    • 47. Replace #max: with #min:
    • 48. Replace #min: with #max:
  • MuTalk – Operators
    • Collection messages
    • 49. Remove at:ifAbsent:
    • 50. Replace #reject: with #select:
    • 51. Replace #select: with #reject:
    • 52. Replace Reject block with [:each | false]
    • 53. Replace Reject block with [:each | true]
    • 54. Replace Select block with [:each | false]
    • 55. Replace Select block with [:each | true]
    • 56. Replace detect: block with [:each | false] when #detect:ifNone:
    • 57. Replace detect: block with [:each | true] when #detect:ifNone:
    • 58. Replace do block with [:each |]
    • 59. Replace ifNone: block with [] when #detect:ifNone:
    • 60. Replace inject:aValueinto:aBlock with aValue
    • 61. Replace sortBlock:aBlock with sortBlock:[:a :b| true]
  • MuTalk – Operators
    • Number messages
    • 62. Replace #* with #/
    • 63. Replace #+ with #-
    • 64. Replace #- with #+
    • 65. Replace #/ with #*
  • MuTalk – Operators
    • Flow control messages
    • 66. Remove Exception Handler Operator
    • 67. Replace #ifFalse: receiver with false
    • 68. Replace #ifFalse: receiver with true
    • 69. Replace #ifFalse: with #ifTrue:
    • 70. Replace #ifFalse:IfTrue: receiver with false
    • 71. Replace #ifFalse:IfTrue: receiver with true
    • 72. Replace #ifTrue: receiver with false
    • 73. Replace #ifTrue: receiver with true
    • 74. Replace #ifTrue: with #ifFalse:
    • 75. Replace #ifTrue:ifFalse: receiver with false
    • 76. Replace #ifTrue:ifFalse: receiver with true
  • Why is not widely used?
  • 77. Is not new … - History
    Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs”
    Generally accepted in 1978, R. Lipton et al, “Hints on test data selection: Help for the practicing programmer”
  • 78. Why is not widely used?
    Maturity Problem: Because Testing is not widely used YET!
    (Although it is increasing)
  • 79. Why is not widely used?
    Integration Problem: Inability to successfully integrate it into the software development process
    (TDD plays a key role now)
  • 80. Why is not widely used?
    Technical Problem: It is a Brute Force technique!
  • 81. Technical Problems
    • Bruteforcetechnique
    N x M
    N = number of tests
    M = number of mutants
  • 82. Aconcagua
    • Number of Tests: 666
    • 83. Number of Mutants: 1005
    • 84. Time to create a mutant/compile/link/run: 10 secs. each aprox.?
    • 85. Total time:
    • 86. 6693300 seconds
    • 87. 1859 hours, 15 minutes
  • Another way of doing it…
    CreditCard&gt;&gt;= anotherCreditCard
    ^(anotherCreditCardisKindOf: self class) and: [ number = anotherCreditCard number ]
    CreditCard&gt;&gt;= anotherCreditCard
    MutantId = 12 ifTrue: [ ^(anotherCreditCardisKindOf: self class) or: [ number = anotherCreditCard number ].
    MutantId = 13 ifTrue: [ ^(anotherCreditCardisKindOf: self class) nand: [ number = anotherCreditCard number ].
    MutantId = 14 ifTrue: [ ^(anotherCreditCardisKindOf: self class) eqv: [ number = anotherCreditCard number ].
  • 88. Aconcagua
    • Number of Tests: 666
    • 89. Number of Mutants: 1005
    • 90. Time to create the metamutant/compile/link: 2 minutes?
    • 91. Time to run the tests per mutant: 1 sec
    • 92. Total time:
    • 93. 1125 seconds
    • 94. 18 minutes 45 seconds
  • MuTalk OptimizationsRunning Strategies
    Mutate all methods, run all tests per mutant
    • Create a mutant for each method
    • 95. Run all the test for each mutant
    • 96. Disadvantage: Slower strategy
    Mutate covered methods, run all tests per mutant
    • Takes coverage running all tests
    • 97. Mutate only covered methods
    • 98. Run all methods per mutant
    • 99. Relies on coverage
    Mutate all methods, run only test that cover mutated method
    • Run coverage keeping for each method the tests that covered it
    • 100. Create a mutant for each method
    • 101. For each mutant, run only the tests that covered the original method
    Mutate covered methods, run only test that covered mutated methods
    • Run coverage keeping for each method the tests that covered it
    • 102. Create a mutant for only covered methods
    • 103. For each mutant, run only the tests that covered the original method
  • MuTalk - Aconcagua Statistics
    • Mutate All, Run All: 1 minute, 6 seconds
    • 104. Mutate Covered, Run Covering: 36 seconds
    • 105. Result:
    • 106. 545 Killed
    • 107. 6 Terminated
    • 108. 83 Survived
  • More Statistics
  • 109. MuTalk OptimizationsTerminated Mutants
    Try to kill the Mutant!
    The killer has to be
    “Terminated”
    The Test Suite
  • 110. MuTalk - Terminated Mutants
    • Take the time it runs each test the first time
    • 111. If the test takes more thant 3 times, terminate it
  • Let’s redefine MuTalk as…
    MutationTestingToolfor Smalltalk (Pharo and Squeak) that uses meta-facilitiestorunfaster and provideinmediatefeedback
  • 112. Work in progress
    • Operators Categorization based on how useful they are to detect errors
    • 113. Filter Operators on View
    • 114. Cancel process
  • Future work
    • Make Operators more “inteligent”
    • 115. a = b ifTrue: [ … ]
    • 116. a = b ifFalse: [] is equivalent to a ~= b ifTrue: []
    • 117. Suggest tests using not killed mutants
    • 118. Use MuTalk to test MuTalk?
  • Why does it work?
    “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” (Coupling effect)
    Demonstrated in 1995, K. Wah, “Fault coupling in finite bijective functions”
  • 119. Why does it work?
    “In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault”
    Geist et al, “Estimation and enhancement of real-time software reliability through mutation analysis”, 1992
  • 120. More Statistics…
  • 121. How does it compare to coverage?
    • Does not replaces coverage because some methods do not generate mutants
    • 122. But:
    • 123. Mutants on not covered methods will survive
    • 124. It provides better insight than coverage
    • 125. Method Coverage fails with long methods/conditions/loops/etc.
  • Questions?
  • 126. MuTalk - MutationTestingfor Smalltalk
    Hernán Wilkinson
    UBA - 10Pines
    hernan.wilkinson@gmail.com
    Nicolás Chillo
    UBA
    nchillo@gmail.com
    Gabriel Brunstein
    UBA
    gaboto@gmail.com