Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mutation testing in Java

1,038 views

Published on

"Mutation testing in Java" presented on conference "33rd Degree 4 Charity" in Wrocław

Published in: Software
  • Be the first to comment

Mutation testing in Java

  1. 1. https://github.com/wlk/mutation-testing-demo Mutation Testing in Java Wojciech Langiewicz @ 33rd Degree 4 Charity 1
  2. 2. https://github.com/wlk/mutation-testing-demo How do you make sure your tests are any good? 2
  3. 3. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How do you make sure your tests are any good? 3 ● Code Review ● TDD ● Code coverage ○ What does it really measure? ○ What does it prove? ○ Line/Statement/Branch coverage ○ Tests without assertions
  4. 4. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutation Testing 4 ● Technique to measure quality of tests ● Injects a fault into a system and uses our tests to find it ● Proposed in 1971 by Richard Lipton ● Competent programmer hypothesis ● Few new concepts: ○ Mutants ○ Mutation Operators ○ Generating Mutants ○ Mutation Coverage
  5. 5. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutation Testing - problems 5 ● Forgotten for many years (only some academic work) ○ Performance problems ○ Lack of tooling ● Performance problem: ○ Test suite takes 5 minutes to run ○ 500 classes, 10 tests per class, testing each class takes 0.6s ○ Naive: 10 mutants per class gives 10 * 5 * 500 ~ 70 hours ○ Fast: 10 * 0.6 * 500 = 50 minutes
  6. 6. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How It Works Original code Mutant Run Tests 6 Tests Pass - mutation wasn’t found Tests Fail - mutation was found
  7. 7. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo How It Works if ( i >= 0 ) { return "foo"; } else { return "bar"; } if ( i > 0 ) { return "foo"; } else { return "bar"; } Original code Mutant Run Tests 7 Tests Pass - mutation wasn’t found Tests Fail - mutation was found
  8. 8. https://github.com/wlk/mutation-testing-demo Mutation Operators (Mutators) 8
  9. 9. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Conditionals Boundary Mutator 9 < <= > >= <= < >= > if (a < b) { // do something } if (a <= b) { // do something }
  10. 10. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Negate Conditionals Mutator 10 == != <= >= < > != ++ > < >= <= if (a == b) { // do something } if (a != b) { // do something }
  11. 11. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Void Method Call 11 public void someVoidMethod(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void someVoidMethod(int i) { // does something } public int foo() { int i = 5; // don’t do anything return i; }
  12. 12. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutator: Constructor Call 12 public Object foo() { Object o = new Object(); return o; } public Object foo() { Object o = null; return o; }
  13. 13. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Mutators: Many More ● Replace constants ● Replace return values to defaults ● And many others 13
  14. 14. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Tooling 14 ● PIT - http://pitest.org/ ● Ruby: Mutant: https://github.com/mbj/mutant ● Popular in communities where testing (TDD) is already popular
  15. 15. https://github.com/wlk/mutation-testing-demo PIT 15
  16. 16. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo PIT Features 16 ● Bytecode modifications (to avoid recompilation) ● Integrates easily with: ○ Java 6, 7, 8 ○ JUnit, TestNG ○ Eclipse, Intellij ○ Gradle, Maven, Ant ○ Sonar, Jenkins ○ Mocking frameworks ● For each mutation, it tries to minimize number of tests to run ● Allows to choose which mutators we want to use ● Doesn’t work with Scala ● Doesn’t store mutated code ● Generates simple HTML report, or XML report for other tools
  17. 17. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Performance Nowadays: 17 ● Not really a problem on CI server when using modern tools (PIT) ● PIT can analyze only changed code (looking at your SCM) ● Practical tests: ○ Apache commons-math: ○ 177k lines of code ○ 109k lines of tests ○ 8 minutes to test ○ PIT takes 1:15h with 4 threads
  18. 18. https://github.com/wlk/mutation-testing-demo Demo 18
  19. 19. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Problems With Mutation Testing 19 Mutants can be dangerous: if(false){ Runtime.getRuntime().exec("rm -rf /"); } Equivalent Mutants: int i = 2; if ( i >= 1 ) { return "foo"; } // is equivalent to int i = 2; if ( i > 1 ) { return "foo"; } Defensive programming: if(i > 0){ throw new IllegalArgumentException( “argument i must be positive” ); } return Math.sqrt(i);
  20. 20. https://github.com/wlk/mutation-testing-demohttps://github.com/wlk/mutation-testing-demo Summary 20 ● Mutation testing tests your tests ● Code coverage gives you false sense of security ● PIT is extremely easy to introduce into Java project
  21. 21. https://github.com/wlk/mutation-testing-demo Time for questions 21
  22. 22. https://github.com/wlk/mutation-testing-demo Thank You! 22

×