Mutation Testing.pdf
•
0 likes•107 views
Slide deck from my Dreamforce '22 session on Mutation Testing. Includes links to the bbmutants CLI plug-in and source code.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Mutation Testing.pdf
Similar to Mutation Testing.pdf (20)
More from Keir Bowden
More from Keir Bowden (20)
Recently uploaded
Recently uploaded (20)
Mutation Testing.pdf
- 1. Release the Mutants And validate your tests Keir Bowden, CTO BrightGen @bob_buzzard
- 2. About me CTO at BrightGen Salesforce MVP 14 years Salesforce 35 years experience
- 3. Agenda The need for mutation testing What is mutation testing? Mutation testing in practice bbmutants CLI Plug-in Key Learnings
- 4. The need for mutation testing
- 6. The Need
- 7. The Need
- 8. The Need
- 9. The Need
- 10. The Need
- 11. The Need
- 12. Quis custodiet ipsos custodes? Who will guard the guards themselves? How do you tests your tests? Changing code should cause test failures Enter the mutants!
- 13. What is Mutation Testing?
- 14. 1. Competent Programmer ● Code is close to correct ● Most faults are simple mistakes 2. Coupling Effects ● Simple faults are coupled to complex faults ● Fixing simple faults will fix complex faults ● Tests that detect mutants would also detect real bugs Two Hypotheses
- 15. Fault based testing - intentionally break code 1. Test original class 2. Create mutant class - change one aspect 3. Test mutant class 4. Compare test results “Open Box” testing - knows about internals Mutation testing R. Lipton, “Fault Diagnosis of Computer Programs”, Student Report, Carnegie Mellon University, 1971
- 17. Mutant Killed ● Changing code breaks tests ● Mutant is covered by test case Mutant Survived ● Changing code did not break test ● Mutant is not covered by test case, or ● Mutant is equivalent to original class Mutation Testing
- 18. Syntax changed Functionality did not Suppress (aka delete) them Equivalent Mutant
- 19. Mutations Type Example Boolean Operator Replace && with || Comparison Operator Replace == with !=, >=, <= Increment Replace ++ with -- Method Call Replace System.now() with System.now.addHours(17) Constant Replace 250000 with 275000 Return Values Replace return true with return false Delete Code Remove lines containing DML statements
- 20. Mutation Testing in Practice
- 21. One change per mutant Avoid multiple mutations of the same line Limit mutations of a single class Large test suite means fewer mutants ● Target specific classes Generating Mutants
- 22. Generate new mutants for each cycle ● They can’t get any deader ● Avoids maintenance overhead Chance of regenerating identical mutants ● Change tactics ● Target only updated files Generating Mutants
- 23. Computationally expensive For each mutant ● Replace original code ● Deploy mutant ● Run entire test suite ● Restore original code Execute, evaluate, repeat ● Execute every mutant, every time Executing Mutants
- 24. Some mutants will not deploy ● Not a weakness of tests ● Developers should spot this! ● Continuous integration will catch this Suppress (aka delete) them Executing Mutants
- 25. Expensive ● Human Oracle required ● Manual checks for all survivors Suppress equivalents Calculate mutation score ● Indicator of test quality Fix tests ● Ensure original code is still green Evaluating, or Counting the Bodies killed mutants total mutants Mutation Score * 100
- 26. Well suited to automation ● What changes is less important than something changed ● Simple “mistakes” introduced ● Coupling effect says real bugs would be found Repeated deploy/execute tests Automation
- 27. New scratch org for cycle Source tracking detects changes ● Replace production class with mutant on filesystem ● Reverse after testing CLI to push source and run tests SFDX is Key
- 29. bbmutants:generate Once per cycle Configurable ● Target specific files ● Limit mutants per file ● Specify strings to replace ● Specify values to replace Generate Mutants
- 30. Avoid repetition - randomise: ● Starting line in file (if limiting mutants) ● Choice of mutation ● Choice of constant ● Choice of comparison operator Generate Mutants
- 31. bbmutants:execute Deploy and run tests ● Delete mutants that won’t deploy List survivors Calculate mutation score Iterative, after fixing tests ● Delete equivalents Execute Mutants
- 32. Plug-in Demo
- 33. Key Learnings
- 34. Allow sufficient time Easier with smaller files ● Fewer mutants Mutation scores won’t surprise you! ● Lots of test effort = high score ● Focus on coverage = low score Key Learnings
- 35. Mutation test at the end of development Automatically generated perform as well as hand crafted Constants are less helpful than expected ● Tests typically reference production values Removing DML didn’t break tests ● Tendency to check returned value over database contents Key Learnings
- 36. Key Learnings Pros Cons Detect untested code Vast number of potential mutants Measurable mutation score Manual verification of surviving mutants Improve test suite quality Full test run for each mutant Tests how the code doesn’t work Knowledge of code internals required
- 37. CLI Plug-in on NPM - https://bobbuzz.me.uk/mutationplugin sfdx plugins:install bbmutation Source on Github - https://bobbuzz.me.uk/mutationsrc Resources
- 39. Connect with Developer Trailblazers from Anywhere Join the Community www.salesforce.com/devcommunity/ #SFDevsAMA
- 40. 40 Thank you