Seeding Bugs to Find Bugs: Mutation Testing Revisited
How do you know your test suite is "good enough"? One of the best ways to tell is _mutation testing_. Mutation testing seeds artificial defects (mutations) into a program and checks whether your test suite finds them. If it does not, this means your test suite is not adequate yet.
Despite its effectiveness, mutation testing has two issues. First, it requires large computing resources to re-run the test suite again and again. Second, and this is worse, a mutation to the program can keep the program's semantics unchanged -- and thus cannot be detected by any test. Such _equivalent mutants_ act as false positives; they have to be assessed and isolated manually, which is an extremely tedious task.
In this talk, I present the JAVALANCHE framework for mutation testing of Java programs, which addresses both the problems of efficiency and equivalent mutants. First, JAVALANCHE is built for efficiency from the ground up, manipulating byte code directly and allowing mutation testing of programs that are several orders of magnitude larger than earlier research subjects. Second, JAVALANCHE addresses the problem of equivalent mutants by assessing the _impact_ of mutations on dynamic invariants: The more invariants impacted by a mutation, the more likely it is to be useful for improving test suites.
We have evaluated JAVALANCHE on seven industrial-size programs, confirming its effectiveness. With less than 3% of equivalent mutants, our approach provides a precise and fully automatic measure of the adequacy of a test suite -- making mutation testing, finally, applicable in practice.
Joint work with David Schuler and Valentin Dallmeier.
Andreas Zeller is computer science professor at Saarland University; he researches large programs and their history, and has developed a number of methods to determine the causes of program failures - on open-source programs as well as in industrial contexts at IBM, Microsoft, SAP and others. His book "Why Programs Fail" has received the Software Development Magazine productivity award in 2006.