Download as PDF, PPTX
Uploaded byKeir Bowden
Mutation Testing.pdf
The document outlines the importance and practice of mutation testing in software development, using the bbmutants CLI plug-in as a tool for generating and executing mutants to test code reliability. It discusses the principles behind mutation testing, including the detection of untested code and the evaluation of tests through mutation scores. Key takeaways include the need for continuous integration and the benefits of automating the mutation testing process to enhance the quality of test suites.
Mutation Testing.pdf
- 1. Release the Mutants And validateyour tests Keir Bowden, CTO BrightGen @bob_buzzard
- 2. About me CTO atBrightGen Salesforce MVP 14 years Salesforce 35 years experience
- 3. Agenda The need formutation testing What is mutation testing? Mutation testing in practice bbmutants CLI Plug-in Key Learnings
- 4. The need for mutationtesting
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12. Quis custodiet ipsoscustodes? Who will guard the guards themselves? How do you tests your tests? Changing code should cause test failures Enter the mutants!
- 13. What is MutationTesting?
- 14. 1. Competent Programmer ●Code is close to correct ● Most faults are simple mistakes 2. Coupling Effects ● Simple faults are coupled to complex faults ● Fixing simple faults will fix complex faults ● Tests that detect mutants would also detect real bugs Two Hypotheses
- 15. Fault based testing- intentionally break code 1. Test original class 2. Create mutant class - change one aspect 3. Test mutant class 4. Compare test results “Open Box” testing - knows about internals Mutation testing R. Lipton, “Fault Diagnosis of Computer Programs”, Student Report, Carnegie Mellon University, 1971
- 16.
- 17. Mutant Killed ● Changingcode breaks tests ● Mutant is covered by test case Mutant Survived ● Changing code did not break test ● Mutant is not covered by test case, or ● Mutant is equivalent to original class Mutation Testing
- 18. Syntax changed Functionality didnot Suppress (aka delete) them Equivalent Mutant
- 19. Mutations Type Example Boolean OperatorReplace && with || Comparison Operator Replace == with !=, >=, <= Increment Replace ++ with -- Method Call Replace System.now() with System.now.addHours(17) Constant Replace 250000 with 275000 Return Values Replace return true with return false Delete Code Remove lines containing DML statements
- 20. Mutation Testing inPractice
- 21. One change permutant Avoid multiple mutations of the same line Limit mutations of a single class Large test suite means fewer mutants ● Target specific classes Generating Mutants
- 22. Generate new mutantsfor each cycle ● They can’t get any deader ● Avoids maintenance overhead Chance of regenerating identical mutants ● Change tactics ● Target only updated files Generating Mutants
- 23. Computationally expensive For eachmutant ● Replace original code ● Deploy mutant ● Run entire test suite ● Restore original code Execute, evaluate, repeat ● Execute every mutant, every time Executing Mutants
- 24. Some mutants willnot deploy ● Not a weakness of tests ● Developers should spot this! ● Continuous integration will catch this Suppress (aka delete) them Executing Mutants
- 25. Expensive ● Human Oraclerequired ● Manual checks for all survivors Suppress equivalents Calculate mutation score ● Indicator of test quality Fix tests ● Ensure original code is still green Evaluating, or Counting the Bodies killed mutants total mutants Mutation Score * 100
- 26. Well suited toautomation ● What changes is less important than something changed ● Simple “mistakes” introduced ● Coupling effect says real bugs would be found Repeated deploy/execute tests Automation
- 27. New scratch orgfor cycle Source tracking detects changes ● Replace production class with mutant on filesystem ● Reverse after testing CLI to push source and run tests SFDX is Key
- 28.
- 29. bbmutants:generate Once per cycle Configurable ●Target specific files ● Limit mutants per file ● Specify strings to replace ● Specify values to replace Generate Mutants
- 30. Avoid repetition -randomise: ● Starting line in file (if limiting mutants) ● Choice of mutation ● Choice of constant ● Choice of comparison operator Generate Mutants
- 31. bbmutants:execute Deploy and runtests ● Delete mutants that won’t deploy List survivors Calculate mutation score Iterative, after fixing tests ● Delete equivalents Execute Mutants
- 32.
- 33.
- 34. Allow sufficient time Easierwith smaller files ● Fewer mutants Mutation scores won’t surprise you! ● Lots of test effort = high score ● Focus on coverage = low score Key Learnings
- 35. Mutation test atthe end of development Automatically generated perform as well as hand crafted Constants are less helpful than expected ● Tests typically reference production values Removing DML didn’t break tests ● Tendency to check returned value over database contents Key Learnings
- 36. Key Learnings Pros Cons Detectuntested code Vast number of potential mutants Measurable mutation score Manual verification of surviving mutants Improve test suite quality Full test run for each mutant Tests how the code doesn’t work Knowledge of code internals required
- 37. CLI Plug-in onNPM - https://bobbuzz.me.uk/mutationplugin sfdx plugins:install bbmutation Source on Github - https://bobbuzz.me.uk/mutationsrc Resources
- 39. Connect with Developer Trailblazersfrom Anywhere Join the Community www.salesforce.com/devcommunity/ #SFDevsAMA
- 40.