Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Can
You
Trust
Your
Tests?
2015 Vaidas Pilkauskas & Tadas Ščerbinskas
Van Halen
Band Tour Rider
Van Halen’s 1982
Tour Rider
Agenda
1. Test quality & code coverage
2. Mutation testing in theory
3. Mutation testing in practice
Prod vs. Test code quality
Code has bugs.
Tests are code.
Tests have bugs.
Test quality
Readable
Focused
Concise
Well named
“Program testing can be used to show
the presence of bugs, but never to
show their absence!”
- Edsger W. Dijkstra
Code Coverage
Types of Code Coverage
Lines
Branches
Instructions
Cyclomatic Complexity
Methods
& more
Lines
boolean doFoo() {
return "a" + "b";
}
assertThat(a.doFoo(), is("ab"));
Lines
boolean doFoo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.doFoo(true), is("a"));
SUCCESS: 26/26 (100%) Tests passed
Branches
boolean doFoo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.doFoo(true), is("a"));
assertThat(a.doFoo(fal...
Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage ...
Code Coverage can be gamed
On purpose or by accident
Mutation testing
Mutation testing
Changes your program code and
expects your tests to fail.
Wait! What exactly is a mutation?
def isFoo(a) {
return a == foo;
}
def isFoo(a) {
return a != foo;
}
def isFoo(a) {
retur...
Terminology
Applying a mutation to some code creates a mutant.
If test passes - mutant has survived.
If test fails - mutan...
Failing is the new
passing
array = [a, b, c]
max(array) == ???
//test
max([0]) == 0
//implementation
max(a) {
return 0
}
//test
max([0]) == 0
max([1]) == 1
//implementation
max(a) {
return a.first
}
//test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
//implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
ret...
Coverage
Mutation
// test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
// implementation
max(a) {
m = a.first
for (e in a)
if (e > m...
Mutation
// test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
// implementation
max(a) {
m = a.first
for (e in a)
if (true)...
// test
max([0]) == 0
max([1]) == 1
max([0, 2]) == 2
// implementation
max(a) {
return a.last
}
Baby steps matter
// test
max([0]) == 0
max([1]) == 1
max([0, 2]) == 2
max([2, 1]) == 2
// implementation
max(a) {
m = a.first
for (e in a)
...
Tests’ effectiveness is measured by
number of killed mutants by your
test suite.
It’s like hiring a white-hat hacker to try to break into
your server and making sure you detect it.
What if mutant survives
● Simplify your code
● Add additional tests
● TDD - minimal amount of code to pass the
test
Challenges
1. High computation cost - slow
2. Equivalent mutants - false negatives
3. Infinite loops
Equivalent mutations
// Original
int i = 0;
while (i != 10) {
doSomething();
i += 1;
}
// Mutant
int i = 0;
while (i < 10)...
Infinite Runtime
// Original
while (expression)
doSomething();
// Mutant
while (true)
doSomething();
Disadvantages
● Can slow down your TDD rhythm
● May be very noisy
Let’s say we have codebase with:
● 300 classes
● around 10 tests per class
● 1 test runs around 1ms
● total test suite run...
Speeding it up
Run only tests that cover the mutation
● 300 classes
● 10 tests per class
● 10 mutations per class
● 1ms te...
Speeding it up
During development run tests that cover
only your current changes
Usage scenarios
● Continuous integration
● TDD with mutation testing only on new
changes
● Add mutation testing to your le...
Tools
● Ruby - Mutant
● Java - PIT
● And many tools for other languages
Summary
● Code coverage highlights code that is
definitely not tested
● Mutation testing highlights code that
definitely i...
About us
Vaidas Pilkauskas
@liucijus
● Vilnius JUG co-
founder
● Vilnius Scala leader
● Coderetreat facilitator
● Mountain...
Credits
A lot of presentation content is based on
work by these guys
● Markus Schirp - author of Mutant
● Henry Coles - au...
Q&A
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Upcoming SlideShare
Loading in …5
×

Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?

472 views

Published on

Agile Tour Kaunas 2015
http://agileturas.lt/kaunas

Published in: Leadership & Management
  • Be the first to comment

  • Be the first to like this

Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?

  1. 1. Can You Trust Your Tests? 2015 Vaidas Pilkauskas & Tadas Ščerbinskas
  2. 2. Van Halen
  3. 3. Band Tour Rider
  4. 4. Van Halen’s 1982 Tour Rider
  5. 5. Agenda 1. Test quality & code coverage 2. Mutation testing in theory 3. Mutation testing in practice
  6. 6. Prod vs. Test code quality Code has bugs. Tests are code. Tests have bugs.
  7. 7. Test quality Readable Focused Concise Well named
  8. 8. “Program testing can be used to show the presence of bugs, but never to show their absence!” - Edsger W. Dijkstra
  9. 9. Code Coverage
  10. 10. Types of Code Coverage Lines Branches Instructions Cyclomatic Complexity Methods & more
  11. 11. Lines boolean doFoo() { return "a" + "b"; } assertThat(a.doFoo(), is("ab"));
  12. 12. Lines boolean doFoo(boolean arg) { return arg ? "a" : "b"; } assertThat(a.doFoo(true), is("a"));
  13. 13. SUCCESS: 26/26 (100%) Tests passed
  14. 14. Branches boolean doFoo(boolean arg) { return arg ? "a" : "b"; } assertThat(a.doFoo(true), is("a")); assertThat(a.doFoo(false), is("b"));
  15. 15. Can you trust 100% coverage? Code coverage can only show what is not tested. For interpreted languages 100% code coverage is kind of like full compilation.
  16. 16. Code Coverage can be gamed On purpose or by accident
  17. 17. Mutation testing
  18. 18. Mutation testing Changes your program code and expects your tests to fail.
  19. 19. Wait! What exactly is a mutation? def isFoo(a) { return a == foo; } def isFoo(a) { return a != foo; } def isFoo(a) { return true; } def isFoo(a) { return null; } > > >
  20. 20. Terminology Applying a mutation to some code creates a mutant. If test passes - mutant has survived. If test fails - mutant is killed.
  21. 21. Failing is the new passing
  22. 22. array = [a, b, c] max(array) == ???
  23. 23. //test max([0]) == 0 //implementation max(a) { return 0 }
  24. 24. //test max([0]) == 0 max([1]) == 1 //implementation max(a) { return a.first }
  25. 25. //test max([0]) == 0 max([1]) == 1 max([0, 2] == 2 //implementation max(a) { m = a.first for (e in a) if (e > m) m = e return m }
  26. 26. Coverage
  27. 27. Mutation // test max([0]) == 0 max([1]) == 1 max([0, 2] == 2 // implementation max(a) { m = a.first for (e in a) if (e > m) m = e return m }
  28. 28. Mutation // test max([0]) == 0 max([1]) == 1 max([0, 2] == 2 // implementation max(a) { m = a.first for (e in a) if (true) m = e return m }
  29. 29. // test max([0]) == 0 max([1]) == 1 max([0, 2]) == 2 // implementation max(a) { return a.last } Baby steps matter
  30. 30. // test max([0]) == 0 max([1]) == 1 max([0, 2]) == 2 max([2, 1]) == 2 // implementation max(a) { m = a.first for (e in a) if (e > m) m = e }
  31. 31. Tests’ effectiveness is measured by number of killed mutants by your test suite.
  32. 32. It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.
  33. 33. What if mutant survives ● Simplify your code ● Add additional tests ● TDD - minimal amount of code to pass the test
  34. 34. Challenges 1. High computation cost - slow 2. Equivalent mutants - false negatives 3. Infinite loops
  35. 35. Equivalent mutations // Original int i = 0; while (i != 10) { doSomething(); i += 1; } // Mutant int i = 0; while (i < 10) { doSomething(); i += 1; }
  36. 36. Infinite Runtime // Original while (expression) doSomething(); // Mutant while (true) doSomething();
  37. 37. Disadvantages ● Can slow down your TDD rhythm ● May be very noisy
  38. 38. Let’s say we have codebase with: ● 300 classes ● around 10 tests per class ● 1 test runs around 1ms ● total test suite runtime is about 3s Is it really slow? Let’s do 10 mutations per class ● We get 3000 (300 * 10) mutations ● runtime with all mutations is 150 minutes (3s * 3000)
  39. 39. Speeding it up Run only tests that cover the mutation ● 300 classes ● 10 tests per class ● 10 mutations per class ● 1ms test runtime ● total mutation runtime 10 * 10 * 1 * 300 = 30s
  40. 40. Speeding it up During development run tests that cover only your current changes
  41. 41. Usage scenarios ● Continuous integration ● TDD with mutation testing only on new changes ● Add mutation testing to your legacy project, but do not fail a build - produce warning report
  42. 42. Tools ● Ruby - Mutant ● Java - PIT ● And many tools for other languages
  43. 43. Summary ● Code coverage highlights code that is definitely not tested ● Mutation testing highlights code that definitely is tested ● Given non equivalent mutations, good test suite should work the same as a hash function
  44. 44. About us Vaidas Pilkauskas @liucijus ● Vilnius JUG co- founder ● Vilnius Scala leader ● Coderetreat facilitator ● Mountain bicycle rider ● Snowboarder Tadas Ščerbinskas @tadassce ● VilniusRB co- organizer ● RubyConfLT co- organizer ● RailsGirls Vilnius & Berlin coach ● Various board sports’ enthusiast
  45. 45. Credits A lot of presentation content is based on work by these guys ● Markus Schirp - author of Mutant ● Henry Coles - author of PIT ● Filip Van Laenen - working on a book
  46. 46. Q&A

×