The document discusses mutation testing as a way to evaluate test quality and coverage. Mutation testing works by modifying the code being tested in small ways (called mutants) and checking if tests detect the changes. This helps identify code that is not fully covered by tests. While it can improve test quality, mutation testing is computationally expensive and challenges include equivalent mutants and infinite loops. The document provides an overview of how mutation testing works and some tools available to implement it.

1. Can
You
Trust
Your
Tests?
2015 Vaidas Pilkauskas & Tadas Ščerbinskas

2. Van Halen

3. Band Tour Rider

4. Van Halen’s 1982
Tour Rider

6. Agenda
1. Test quality & code coverage
2. Mutation testing in theory
3. Mutation testing in practice

7. Prod vs. Test code quality
Code has bugs.
Tests are code.
Tests have bugs.

9. Test quality
Readable
Focused
Concise
Well named

10. “Program testing can be used to show
the presence of bugs, but never to
show their absence!”
- Edsger W. Dijkstra

11. Code Coverage

12. Types of Code Coverage
Lines
Branches
Instructions
Cyclomatic Complexity
Methods
& more

15. Lines
boolean doFoo() {
return "a" + "b";
}
assertThat(a.doFoo(), is("ab"));

16. Lines
boolean doFoo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.doFoo(true), is("a"));

17. SUCCESS: 26/26 (100%) Tests passed

18. Branches
boolean doFoo(boolean arg) {
return arg ? "a" : "b";
}
assertThat(a.doFoo(true), is("a"));
assertThat(a.doFoo(false), is("b"));

19. Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage is
kind of like full compilation.

20. Code Coverage can be gamed
On purpose or by accident

21. Mutation testing

22. Mutation testing
Changes your program code and
expects your tests to fail.

23. Wait! What exactly is a mutation?
def isFoo(a) {
return a == foo;
}
def isFoo(a) {
return a != foo;
}
def isFoo(a) {
return true;
}
def isFoo(a) {
return null;
}
>
>
>

24. Terminology
Applying a mutation to some code creates a mutant.
If test passes - mutant has survived.
If test fails - mutant is killed.

25. Failing is the new
passing

26. array = [a, b, c]
max(array) == ???

27. //test
max([0]) == 0
//implementation
max(a) {
return 0
}

28. //test
max([0]) == 0
max([1]) == 1
//implementation
max(a) {
return a.first
}

29. //test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
//implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
return m
}

30. Coverage

31. Mutation
// test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
// implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
return m
}

32. Mutation
// test
max([0]) == 0
max([1]) == 1
max([0, 2] == 2
// implementation
max(a) {
m = a.first
for (e in a)
if (true)
m = e
return m
}

33. // test
max([0]) == 0
max([1]) == 1
max([0, 2]) == 2
// implementation
max(a) {
return a.last
}
Baby steps matter

34. // test
max([0]) == 0
max([1]) == 1
max([0, 2]) == 2
max([2, 1]) == 2
// implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
}

35. Tests’ effectiveness is measured by
number of killed mutants by your
test suite.

36. It’s like hiring a white-hat hacker to try to break into
your server and making sure you detect it.

37. What if mutant survives
● Simplify your code
● Add additional tests
● TDD - minimal amount of code to pass the
test

38. Challenges
1. High computation cost - slow
2. Equivalent mutants - false negatives
3. Infinite loops

39. Equivalent mutations
// Original
int i = 0;
while (i != 10) {
doSomething();
i += 1;
}
// Mutant
int i = 0;
while (i < 10) {
doSomething();
i += 1;
}

40. Infinite Runtime
// Original
while (expression)
doSomething();
// Mutant
while (true)
doSomething();

41. Disadvantages
● Can slow down your TDD rhythm
● May be very noisy

42. Let’s say we have codebase with:
● 300 classes
● around 10 tests per class
● 1 test runs around 1ms
● total test suite runtime is about 3s
Is it really slow?
Let’s do 10 mutations per class
● We get 3000 (300 * 10) mutations
● runtime with all mutations is
150 minutes (3s * 3000)

43. Speeding it up
Run only tests that cover the mutation
● 300 classes
● 10 tests per class
● 10 mutations per class
● 1ms test runtime
● total mutation runtime
10 * 10 * 1 * 300 = 30s

44. Speeding it up
During development run tests that cover
only your current changes

45. Usage scenarios
● Continuous integration
● TDD with mutation testing only on new
changes
● Add mutation testing to your legacy
project, but do not fail a build -
produce warning report

46. Tools
● Ruby - Mutant
● Java - PIT
● And many tools for other languages

47. Summary
● Code coverage highlights code that is
definitely not tested
● Mutation testing highlights code that
definitely is tested
● Given non equivalent mutations, good test
suite should work the same as a hash
function

48. About us
Vaidas Pilkauskas
@liucijus
● Vilnius JUG co-
founder
● Vilnius Scala leader
● Coderetreat facilitator
● Mountain bicycle
rider
● Snowboarder
Tadas Ščerbinskas
@tadassce
● VilniusRB co-
organizer
● RubyConfLT co-
organizer
● RailsGirls Vilnius &
Berlin coach
● Various board sports’
enthusiast

49. Credits
A lot of presentation content is based on
work by these guys
● Markus Schirp - author of Mutant
● Henry Coles - author of PIT
● Filip Van Laenen - working on a book

50. Q&A