Mutation Testing with PIT
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast

About me
● Integration Team Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast

I wrote code for the
spacecraft.

How do you that it
works?

Hmm...

I just know

I feel it

I feel it

I feel it

I feel it

$327.6 million

I wrote code for the
spacecraft.

How do you that it
works?

I wrote unit tests!

How do you know
that your tests work?

Hmm...

really?

code

code
test

code
test
test

code
test
test
test

code
test
test
test
test

Does testing make
any sense at all?

return a + b;

return a + b;

Coverage does not
prove that you have
TESTED every line.
All it proves is that you
have EXECUTED every
line. And that is a big
difference.

public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}

@Test
public void
force100PercentCoverage() {
calculator.sum(0, 0);
}

Richard Lipton found the
good answer in 1971

"If you want to know if a test
suite has properly checked
some code, introduce a bug"

return a + b;

return a - b;

Is it the whole idea behind the
mutation testing?

Actually, Yes!

● Artificial bug: MUTATION OPERATION

● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT

● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant

● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant
● When the test succeeds on mutant:
mutant SURVIVED

return a - b;
KILLED SURVIVED

killing is good

My code is much more
complex than adding two
numbers!

Mutation
Operators

Math
+ to - * to /
- to + / to *

Boundary
< to <= > to >=
<= to < <= to >

Negate
== to !=
>= to <

Complex
Remove IFs

Complex
Remove IFs Remove method calls

Complex
Remove IFs
Modify return
Remove method calls

Complex
Remove IFs
Modify constantsModify return
Remove method calls

Do I need to change the
code on my own?

PIT Mutation Testing

public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}
@Test
public
void force100PercentCoverage() {
calculator.sum(0, 0);
}

DEMO

> MathMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

@Test public void
Should_ReturnSum_When_SummingTwoValues() {
// given
int a = 1;
int b = 2;
// when
int result = calculator.sum(a, b);
// then
assertThat(result).isEqualTo(3);
}

> MathMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0

Do I need to read the
console?

DEMO

How about bigger
projects?

Hazelcast Kubernetes Plugin
● 5000 LOC
● 12 classes

DEMO

Why NASA engineers
didn't use it in 1998?

1971
Richard
Lipton

1971
Richard
Lipton
2000

1971
Richard
Lipton
2000 2012

1
Equivalent
Mutants

if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}

if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}

2
Slow
Performance

return a + b;

return a + b;

return a + b;
return a - b
return 0

return a + b;
return a - b
return 0

:)

What about my
team?

Main class paradox

// given

// given
// when

// given
// when
?

What to do with people
who write tests without
assertions?

Do people really use
mutation testing?

Test your tests

Thank You!
Rafał Leszko
@RafalLeszko
rafalleszko.com