Recommended
More Related Content
Similar to Mutation Testing: The Swiss army knife of Testing
Similar to Mutation Testing: The Swiss army knife of Testing (20)
Recently uploaded
Recently uploaded (20)
Mutation Testing: The Swiss army knife of Testing
- 2. Who am I? What mutation can do for you “Attacking” bugs with mutation
- 3. Research Associate (postdoc) @ marinos.kintis@uni.lu https://kintism.github.io Marinos Kintis • Mutation Testing • E2E testing • Test case prioritisation Prof. Yves Le Traon Head of Serval Dr. Mike Papadakis Renaud Rwemalika
- 5. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 6. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 7. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3 The Major tool http://mutation-testing.org/
- 8. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3 The Major tool http://mutation-testing.org/ More tools can be found here: https://mutationtesting.uni.lu/tools.php
- 9. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3 A comparison of the effectiveness of Java mutation tools can be found here: http://pages.cs.aueb.gr/~kintism/papers/preprint1.pdf The Major tool http://mutation-testing.org/ More tools can be found here: https://mutationtesting.uni.lu/tools.php
- 10. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 15. MUTATION OPERATORS AND THEIR IMPLEMENTATION BE CAREFUL Preprint: http://pages.cs.aueb.gr/~kintism/papers/preprint1.pdf
- 16. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 17. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 18. the tests
- 19. Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3
- 21. 0% 100% HOW GOOD ARE MY TESTS? Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests
- 22. HOW CAN I GENERATE GOOD TESTS?
- 23. HOW CAN I GENERATE GOOD TESTS?
- 25. Many diverse
- 26. Test case generation Test suite reduction/selection Testing techniques comparison Test case minimisation Test case prioritisation Fault localisation Automated Program repair Genetic improvement Testing Compiler Optimisations Test oracle creation Models Testing Network protocols Testing Security Policies Mutation Cost Reduction GUI Testing
- 30. Superior than other testing criteria If the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault Geist et al., “Estimation and enhancement of real-time software reliability through mutation analysis”, IEEE Trans. on Computers, 1992
- 31. Superior than other testing criteria Mutation makes us think about other implementations of the SUT that seem correct based on the available tests
- 32. Who am I? What mutation can do for you “Attacking” bugs with mutation
- 33. VERSUS
- 34. VERSUS Is there a bug in your code?
- 35. VERSUS
- 36. VERSUS Unlikely! I have very high branch coverage! Is there a bug in your code?
- 37. Look again!!!
- 38. Look again!!! Branch coverage CANNOT provide any help in finding the bug
- 40. minutesOffset <= 0 This mutant can only be detected by tests that also find the bug
- 41. minutesOffset <= 0 This mutant can only be killed by tests that also find the bug Yep! My mutants helped me find it!!! Is there a bug in your code?
- 42. What are the benefits of applying mutation after achieving 100% branch coverage? Mutation is NOT costly after the satisfaction of the BC coverage 2 bugs found that were missed by the BC tests Improvements in the source code and test code during the analysis of mutants
- 44. Test case generation Test suite reduction/ selection Testing techniques comparison Test case minimisation Test case prioritisation Fault localisation Automated Program repair Genetic improvement Testing Compiler Optimisations Test oracle creation Models Testing Network protocols Testing Security Policies Mutation Cost Reduction GUI Testing Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3 minutesOffset <= 0 This mutant can only be killed by tests that also find the bug Yep! My mutants helped me find it!!! Is there a bug in your code? What are the benefits of applying mutation after achieving 100% branch coverage? Mutation is NOT costly after the satisfaction of the BC coverage 2 bugs found that were missed by the BC tests I mprovements in the source code and test code during the analysis of mutants
- 45. Test case generation Test suite reduction/ selection Testing techniques comparison Test case minimisation Test case prioritisation Fault localisation Automated Program repair Genetic improvement Testing Compiler Optimisations Test oracle creation Models Testing Network protocols Testing Security Policies Mutation Cost Reduction GUI Testing Mutation Testing Available Tests 1 Test quality 0% 100% 2 3 4 3 minutesOffset <= 0 This mutant can only be killed by tests that also find the bug Yep! My mutants helped me find it!!! Is there a bug in your code? What are the benefits of applying mutation after achieving 100% branch coverage? Mutation is NOT costly after the satisfaction of the BC coverage 2 bugs found that were missed by the BC tests I mprovements in the source code and test code during the analysis of mutants Thank you! marinos.kintis@uni.lu https://kintism.github.io