Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Can
You
Trust
Your
Tests?
2016 Vaidas Pilkauskas & Tadas Ščerbinskas
An Introduction to Mutation Testing
Thanks to co-author
Tadas Ščerbinskas
VilniusRB organizer
@tadassce
Van Halen
Band Tour Rider
Van Halen’s 1982
Tour Rider
Agenda
1. Test quality & code coverage
2. Mutation testing in theory
3. Mutation testing in practice
Prod vs. Test code quality
Code has bugs.
Tests are code.
Tests have bugs.
Test quality
Readable
Focused
Concise
Well named
“Program testing can be used to show
the presence of bugs, but never to
show their absence!”
- Edsger W. Dijkstra
Code Coverage
Types of Code Coverage
Lines
Branches
Instructions
Cyclomatic Complexity
Methods
& more
Lines
String foo() {
return "a" + "b";
}
assertThat(a.foo(), is("ab"))
Lines
String foo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.foo(true), is("a"));
Branches
String foo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.foo(true), is("a"));
assertThat(a.foo(false), is...
SUCCESS: 26/26 (100%) Tests passed
Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage ...
Code Coverage can be gamed
On purpose or by accident
Mutation testing
Mutation testing
Changes your program code and
expects your tests to fail.
What exactly is a mutation?
Boolean isFoo(int a) {
return a == foo;
}
Boolean isFoo(int a) {
return a != foo;
}
Boolean is...
Terminology
Applying a mutation to some code creates a mutant.
If test passes - mutant has survived.
If test fails - mutan...
Failing is the new
passing
array = [a, b, c]
max(array) == ???
// test
assertThat(Max.of(0), equalTo(0)); ✘
// test
assertThat(Max.of(0), equalTo(0)); ✔
// implementation
public static int of(int... integers) {
return 0;
}
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✘
// implementation
public static int of(i...
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
// implementation
public static int of(i...
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalTo(2)); ✘
...
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalTo(2)); ✔
...
Coverage
Mutation
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalT...
Mutation
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalT...
Mutation
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalT...
Mutation
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalT...
Mutation
// test
assertThat(Max.of(0), equalTo(0)); ✔
assertThat(Max.of(1), equalTo(1)); ✔
assertThat(Max.of(1, 2), equalT...
Baby steps matter
Tests’ effectiveness is measured
by number of killed mutants by
your test suite.
It’s like hiring a white-hat hacker to try to break into
your server and making sure you detect it.
What if mutant survives
● Simplify your code
● Add additional tests
● TDD - minimal amount of code to pass
the test
Challenges
1. High computation cost - slow
2. Equivalent mutants - false negatives
3. Infinite loops
Equivalent mutations
// Original
int i = 0;
while (i != 10) {
doSomething();
i += 1;
}
// Mutant
int i = 0;
while (i < 10)...
Infinite Runtime
// Original
while (expression)
doSomething();
// Mutant
while (true)
doSomething();
● Let’s say we have codebase with:
● 300 classes
● around 10 tests per class
● 1 test runs around 1ms
● total test suite r...
Speeding it up
Run only tests that cover the mutation
● 300 classes
● 10 tests per class
● 10 mutations per class
● 1ms te...
Speeding it up
During development run tests that
cover only your current changes
● Continuous integration
● TDD with mutation testing only
on new changes
● Add mutation testing to your
legacy project, bu...
Tools
● Ruby - Mutant
● Java - PIT
● And many tools for other
languages
Pit
● Active project
● Good reporting
● No Scala support :(
Pit
PIT originally stood for Parallel Isolated
Test.
Now it stands for PIT.
Test Selection
● Line coverage
● Test execution speed
● Test naming convention
Running PIT
● Command line tool
● Maven
● Ant
● Gradle
PIT supports many mocking
frameworks
● Mockito
● EasyMock
● JMock
● PowerMock
● JMockit
Maven usage
mvn org.pitest:pitest-maven:mutationCoverage
With SCM plugin
mvn org.pitest:pitest-maven:scmMutationCoverage 
...
Configuration
● Dependency distance
● Limit mutation number per class
● Activate/deactivate mutators
● Extension points - ...
What about TDD?
● Influences code style
● Helps to spot dead features
● Puts emphasis on minimal code
to pass the test
Summary
Code coverage highlights code that is not
tested.
It shows which code you have executed in tests.
Summary
Mutation testing highlights code that is
tested.
It shows which code you have asserted in tests.
How much we suck
Vaidas Pilkauskas
@liucijus
● Vilnius JUG co-founder
● Vilnius Scala leader
● Coderetreat facilitator
● Mountain bicycle r...
Q&A
Credits
A lot of presentation content is based on
work by these guys
● Markus Schirp - author of Mutant
● Henry Coles - au...
Can you trust your tests?
Can you trust your tests?
Can you trust your tests?
Can you trust your tests?
Can you trust your tests?
Upcoming SlideShare
Loading in …5
×

Can you trust your tests?

531 views

Published on

Nobody argues these days that unit tests are useful and provide valuable feedback about your code. But who watches the watchmen? Or, in other words, who is testing tests themselves? Let's talk about test code quality, code coverage and introduce mutation based testing techniques. Ideas behind mutation testing are nothing new in academic world, but only now become recognized by software developers while programming their production systems. This talk will look into how mutation testing can be used to improve your test-driven development cycle and will try to answer if existing tooling is mature enough to be used while coding your daily tasks.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Can you trust your tests?

  1. 1. Can You Trust Your Tests? 2016 Vaidas Pilkauskas & Tadas Ščerbinskas An Introduction to Mutation Testing
  2. 2. Thanks to co-author Tadas Ščerbinskas VilniusRB organizer @tadassce
  3. 3. Van Halen
  4. 4. Band Tour Rider
  5. 5. Van Halen’s 1982 Tour Rider
  6. 6. Agenda 1. Test quality & code coverage 2. Mutation testing in theory 3. Mutation testing in practice
  7. 7. Prod vs. Test code quality Code has bugs. Tests are code. Tests have bugs.
  8. 8. Test quality Readable Focused Concise Well named
  9. 9. “Program testing can be used to show the presence of bugs, but never to show their absence!” - Edsger W. Dijkstra
  10. 10. Code Coverage
  11. 11. Types of Code Coverage Lines Branches Instructions Cyclomatic Complexity Methods & more
  12. 12. Lines String foo() { return "a" + "b"; } assertThat(a.foo(), is("ab"))
  13. 13. Lines String foo(boolean arg) { return arg ? "a" : "b"; } assertThat(a.foo(true), is("a"));
  14. 14. Branches String foo(boolean arg) { return arg ? "a" : "b"; } assertThat(a.foo(true), is("a")); assertThat(a.foo(false), is("b"));
  15. 15. SUCCESS: 26/26 (100%) Tests passed
  16. 16. Can you trust 100% coverage? Code coverage can only show what is not tested. For interpreted languages 100% code coverage is equivalent to compilation.
  17. 17. Code Coverage can be gamed On purpose or by accident
  18. 18. Mutation testing
  19. 19. Mutation testing Changes your program code and expects your tests to fail.
  20. 20. What exactly is a mutation? Boolean isFoo(int a) { return a == foo; } Boolean isFoo(int a) { return a != foo; } Boolean isFoo(int a) { return true; } Boolean isFoo(int a) { return null; } > > >
  21. 21. Terminology Applying a mutation to some code creates a mutant. If test passes - mutant has survived. If test fails - mutant is killed.
  22. 22. Failing is the new passing
  23. 23. array = [a, b, c] max(array) == ???
  24. 24. // test assertThat(Max.of(0), equalTo(0)); ✘
  25. 25. // test assertThat(Max.of(0), equalTo(0)); ✔ // implementation public static int of(int... integers) { return 0; }
  26. 26. // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✘ // implementation public static int of(int... integers) { return 0; }
  27. 27. // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ // implementation public static int of(int... integers) { return integers[0]; }
  28. 28. // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✘ // implementation public static int of(int... integers) { return integers[0]; }
  29. 29. // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ // implementation public static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max; }
  30. 30. Coverage
  31. 31. Mutation // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ // implementation public static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max; }
  32. 32. Mutation // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ // implementation public static int of(int... integers) { int max = integers[0]; for (int i : integers) if (true) max = i; return max; }
  33. 33. Mutation // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ // implementation public static int of(int... integers) { return integers[integers.length - 1]; }
  34. 34. Mutation // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ assertThat(Max.of(2, 1), equalTo(2)); ✘ // implementation public static int of(int... integers) { return integers[integers.length - 1]; }
  35. 35. Mutation // test assertThat(Max.of(0), equalTo(0)); ✔ assertThat(Max.of(1), equalTo(1)); ✔ assertThat(Max.of(1, 2), equalTo(2)); ✔ assertThat(Max.of(2, 1), equalTo(2)); ✔ // implementation public static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max; }
  36. 36. Baby steps matter
  37. 37. Tests’ effectiveness is measured by number of killed mutants by your test suite.
  38. 38. It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.
  39. 39. What if mutant survives ● Simplify your code ● Add additional tests ● TDD - minimal amount of code to pass the test
  40. 40. Challenges 1. High computation cost - slow 2. Equivalent mutants - false negatives 3. Infinite loops
  41. 41. Equivalent mutations // Original int i = 0; while (i != 10) { doSomething(); i += 1; } // Mutant int i = 0; while (i < 10) { doSomething(); i += 1; }
  42. 42. Infinite Runtime // Original while (expression) doSomething(); // Mutant while (true) doSomething();
  43. 43. ● Let’s say we have codebase with: ● 300 classes ● around 10 tests per class ● 1 test runs around 1ms ● total test suite runtime is about 3s Is it slow? Let’s do 10 mutations per class ● We get 3000 (300 classes * 10 mutations) mutations ● runtime with all mutations is 150 minutes (3s * 3000)
  44. 44. Speeding it up Run only tests that cover the mutation ● 300 classes ● 10 tests per class ● 10 mutations per class ● 1ms test runtime ● total mutation runtime 10 tests * 10 mutations * 1 ms * 300 classes = 30 s
  45. 45. Speeding it up During development run tests that cover only your current changes
  46. 46. ● Continuous integration ● TDD with mutation testing only on new changes ● Add mutation testing to your legacy project, but do not fail a build - produce a warning report Usage scenarios
  47. 47. Tools ● Ruby - Mutant ● Java - PIT ● And many tools for other languages
  48. 48. Pit ● Active project ● Good reporting ● No Scala support :(
  49. 49. Pit PIT originally stood for Parallel Isolated Test. Now it stands for PIT.
  50. 50. Test Selection ● Line coverage ● Test execution speed ● Test naming convention
  51. 51. Running PIT ● Command line tool ● Maven ● Ant ● Gradle
  52. 52. PIT supports many mocking frameworks ● Mockito ● EasyMock ● JMock ● PowerMock ● JMockit
  53. 53. Maven usage mvn org.pitest:pitest-maven:mutationCoverage With SCM plugin mvn org.pitest:pitest-maven:scmMutationCoverage -Dinclude=ADDED,UNKNOWN -DmutationThreshold=85
  54. 54. Configuration ● Dependency distance ● Limit mutation number per class ● Activate/deactivate mutators ● Extension points - Java SPI
  55. 55. What about TDD? ● Influences code style ● Helps to spot dead features ● Puts emphasis on minimal code to pass the test
  56. 56. Summary Code coverage highlights code that is not tested. It shows which code you have executed in tests.
  57. 57. Summary Mutation testing highlights code that is tested. It shows which code you have asserted in tests.
  58. 58. How much we suck
  59. 59. Vaidas Pilkauskas @liucijus ● Vilnius JUG co-founder ● Vilnius Scala leader ● Coderetreat facilitator ● Mountain bicycle rider ● Snowboarder About me
  60. 60. Q&A
  61. 61. Credits A lot of presentation content is based on work by these guys ● Markus Schirp - author of Mutant ● Henry Coles - author of PIT ● Filip Van Laenen - working on a book

×