Can You Trust Your Tests? (Agile Tour 2015 Kaunas)
Oct. 6, 2015•0 likes•676 views
Download to read offline
Report
Software
Nobody argues these days that unit tests are useful and provide valuable feedback about your code. But who watches the watchmen? Let's talk about test code quality, code coverage and introduce mutation based testing techniques.
18. Can you trust 100% coverage?
Code coverage can only show what is not tested.
For interpreted languages 100% code coverage is
kind of like full compilation.
33. Mutation
// test
max([0]) == 0 ✔
max([1]) == 1 ✔
max([0, 2]) == 2 ✔
// implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
return m
}
34. Mutation
// test
max([0]) == 0 ✔
max([1]) == 1 ✔
max([0, 2]) == 2 ✔
// implementation
max(a) {
m = a.first
for (e in a)
if (true)
m = e
return m
}
37. // implementation
max(a) {
m = a.first
for (e in a)
if (e > m)
m = e
return m
}
// test
max([0]) == 0 ✔
max([1]) == 1 ✔
max([0, 2]) == 2 ✔
max([2, 1]) == 2 ✔
43. Equivalent mutations
// Original
int i = 0;
while (i != 10) {
doSomething();
i += 1;
}
// Mutant
int i = 0;
while (i < 10) {
doSomething();
i += 1;
}
46. Let’s say we have codebase with:
● 300 classes
● around 10 tests per class
● 1 test runs around 1ms
● total test suite runtime is about 3s
Is it really slow?
Let’s do 10 mutations per class
● We get 3000 (300 * 10) mutations
● runtime with all mutations is
150 minutes (3s * 3000)
47. Speeding it up
Run only tests that cover the mutation
● 300 classes
● 10 tests per class
● 10 mutations per class
● 1ms test runtime
● total mutation runtime
10 * 10 * 1 * 300 = 30s
49. ● Continuous integration
● TDD with mutation testing only
on new changes
● Add mutation testing to your
legacy project, but do not fail a
build - produce warning report
Usage scenarios
50. Tools
● Ruby - Mutant
● Java - PIT
● And many tools for other
languages
51. Summary
● Code coverage highlights code that is
definitely not tested
● Mutation testing highlights code that
definitely is tested
● Given non equivalent mutations, good
test suite should work the same as a
hash function
52. Vaidas Pilkauskas
@liucijus
● Vilnius JUG co-founder
● Vilnius Scala leader
● Coderetreat facilitator
● Mountain bicycle rider
● Snowboarder
About us
Tadas Ščerbinskas
@tadassce
● VilniusRB co-organizer
● RubyConfLT co-
organizer
● RailsGirls Vilnius & Berlin
coach
● Various board sports’
enthusiast
53. Credits
A lot of presentation content is based on
work by these guys
● Markus Schirp - author of Mutant
● Henry Coles - author of PIT
● Filip Van Laenen - working on a book