Mutants to the rescue:
How effective are your
unit tests?

@DevPaco
Paco van Beckhoven
- Software engineer
- Automated warehouse team @ Picnic
About me

@DevPaco
“Quality”

@DevPaco
“The degree to which a system, component, or process meets
specified requirements” – IEEE
Software quality

@DevPaco
ISO/IEC 25010

@DevPaco
ISO/IEC 25010

@DevPaco

@DevPaco
Unit test example
public class Calculator {
public int multiply(int a, int b) {
return a * b;
}
}

@DevPaco
Unit test example
public class Calculator {
public int multiply(int a, int b) {
return a * b;
}
}
class CalculatorTest {
@Test
void testMultiply() {
assertEquals(20, new Calculator().multiply(4, 5));
}
@Test
void testMultiplyWithZero() {
assertEquals(0, new Calculator().multiply(0, 5));
}
}

@DevPaco
Maintenance

@DevPaco
Documentation

@DevPaco
Quick feedback
https://www.rainerhahnekamp.com/en/tag/agile/

@DevPaco
“ Always code as if the guy who ends up maintaining
your code will be a violent psychopath who knows
where you live ”
John F. Woods

@DevPaco

@DevPaco

@DevPaco
• Projects evolve, grow
• Tests evolve, sometimes missed when refactoring
• Test code often not monitored
The problem

@DevPaco
Code coverage
“The degree to which the source code of a program is
executed when a particular test suite runs”
Monitoring Tests

@DevPaco
Example
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

@DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

@DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));

@DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));

@DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));

@DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

@DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
testZ: submit(proposal, 0, Instant.now().minusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

@DevPaco
Pros:
• Helps you write more/better tests
• Easy/cheap to measure
• Shows what you didn’t test
• Shows that what you did test, didn’t crash
Code coverage

@DevPaco
But:
• Can be misleading
• Doesn’t give any guarantees
Code coverage

@DevPaco
But:
• Can be misleading
• Doesn’t give any guarantees
Code coverage
@Test
public void shouldReturnSuccessOnValidSubmit() {
proposalService.submit(randomProposal(), 0, tomorrow());
}

@DevPaco
• Testing the tests
• “Bug detection ability”
Test effectiveness
https://shirt.woot.com/offers/bug-hunt

@DevPaco
• Simulating bugs
• Typical programming errors
Mutation testing

@DevPaco
Step 1. Generating mutants
MUTATOR
Production code The mutant
Altered version of the
production code

@DevPaco
Example
boolean isAllowedToApplyForDriversLicense(int age) {
return age < 17;
}
boolean isAllowedToApplyForDriversLicense(int age) {
return age >= 17;
}

@DevPaco
Example
return age < 17; // negate conditional
return age > 17; // change conditional boundaries
return true; // true returns
return false; // false returns
boolean isAllowedToApplyForDriversLicense(int age) {
return age >= 17;
}

@DevPaco
Math mutators
int getTotalNumberOfLayers() {
return layers.size() + numberOfAdditionalLayers;
}
int getTotalNumberOfLayers() {
return layers.size() - numberOfAdditionalLayers;
}

@DevPaco
Removing void statements
MetaData createMetaData(Instant eventTimestamp) {
MetaData meta = new MetaData();
meta.setPublished(eventTimestamp);
return meta;
}
MetaData createMetaData(Instant eventTimestamp) {
MetaData meta = new MetaData();
return meta;
}

@DevPaco
Step 2. Mutation analysis
❌ Mutant survived
✅ Mutant killed
+ Tests

@DevPaco
The first generation mutation tools (for Java)
- Jester
- Jumble
- MuJava
Mutation tools

@DevPaco
• Created by Henry Coles in 2011
• Byte code mutation
• Optimized for coverage
• Easy to use
• Actively maintained/supported
PIT / Pitest

@DevPaco
1. Add the plugin
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.7.5</version>
</plugin>
2. Run pitest
> mvn clean verify pitest:mutationCoverage
Getting started

@DevPaco
Demo

@DevPaco
Reality

@DevPaco
• Limit number of mutants
<targetClasses>
<param>org.jfree.chart.util.*</param>
</targetClasses>
• Limit number of executed tests
<targetTests>
<param>org.jfree.chart.util.*</param>
</targetTests>
• Increase Threads
<threads>8</threads>
Start small!

@DevPaco
Tweaking performance

@DevPaco
• Reduce number of tests to run
<configuration>
<excludedTestClasses>foo.bar.acceptance.*</excludedTestClasses>
<excludedTestClasses>foo.bar.database.*</excludedTestClasses>
<excludedTestClasses>foo.bar.slooow.*</excludedTestClasses>
</configuration>
Tweaking performance

@DevPaco
• Exclude code from mutation
<configuration>
<excludedClasses>foo.bar.generated.*</excludedClasses>
<excludedClasses>*pdfgenerator.*</excludedClasses>
<excludedClasses>foo.bar.database.*</excludedClasses>
<excludedMethods>*ThreadManager.run*</excludedMethods>
</configuration>
Tweaking performance

@DevPaco
• Incremental analysis
• Experimental feature
• Reuse results from last run
Tweaking performance

@DevPaco
Keeping your reports clean
<configuration>
<avoidCallsTo>
<avoidCallsTo>java.util.logging</avoidCallsTo>
<avoidCallsTo>org.apache.log4j</avoidCallsTo>
<avoidCallsTo>org.slf4j</avoidCallsTo>
<avoidCallsTo>org.apache.commons.logging</avoidCallsTo>
</avoidCallsTo>
</configuration>
log.info(”Execution took {} seconds", end – start);
Tweaking performance

@DevPaco
Tweaking Mutators

@DevPaco
Tweaking Mutators

@DevPaco
Tweaking Mutators

@DevPaco
Equivalent mutants
MUTATOR
Replace == with >=
int i = 0;
while (true) {
i++;
if (i == MAX) {
break;
}
}
int i = 0;
while (true) {
i++;
if (i >= MAX) {
break;
}
}

@DevPaco
• Add the maven goal to your build step
mvn org.pitest:pitest-maven:mutationCoverage
• Plugins available for Jenkins/Sonar
• Pitest can be configured to break the pipeline
Adding mutation testing to CI

@DevPaco
When to consider mutation testing?

@DevPaco
Are there lives at stake?
Are you building software for a rocket?
Is it a self driving car?
When to consider mutation testing?

@DevPaco
Are you using code coverage?
What is the cost of fixing a bug?
Does the team think it’s a good idea?
When to consider mutation testing?

@DevPaco
• Code coverage: “How much of the code is tested”
• Mutation testing: “How proper is the code tested”
• Start small
• Tweak for performance!
Summary

@DevPaco
Thank you!
https://pitest.org/
Mutation testing for Java, Kotlin(paid)
https://stryker-mutator.io/
Supports JavaScript (typescript), C#, Scala
https://github.com/boxed/mutmut
Mutation testing tool for Python
@DevPaco
For questions, feedback, suggestions