Mutation Testing with PIT

Mutation Testing with PIT
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast
About me
● Cloud-Native Tech Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland
About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
@Hazelcast
Mutation Testing with PIT
Mutation Testing with PIT
I wrote code for the
spacecraft.
How do you that it
works?
Hmm...
I just know
I feel it
I feel it
I feel it
I feel it
Mutation Testing with PIT
Mutation Testing with PIT
$327.6 million
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
I wrote code for the
spacecraft.
How do you that it
works?
I wrote unit tests!
How do you know
that your tests work?
Hmm...
really?
code
code
test
code
test
test
code
test
test
test
code
test
test
test
test
Mutation Testing with PIT
Does testing make
any sense at all?
return a + b;
return a + b;
Coverage does not
prove that you have
TESTED every line.
All it proves is that you
have EXECUTED every
line. And that is a big
difference.
public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}
@Test
public void
force100PercentCoverage() {
calculator.sum(0, 0);
}
Richard Lipton found the
good answer in 1971
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
"If you want to know if a test
suite has properly checked
some code, introduce a bug"
return a + b;
return a - b;
Is it the whole idea behind the
mutation testing?
Actually, Yes!
● Artificial bug: MUTATION OPERATION
● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant
● Artificial bug: MUTATION OPERATION
● Code with artificial bug: MUTANT
● When the test fails on mutant: KILLED the
mutant
● When the test succeeds on mutant:
mutant SURVIVED
return a - b;
KILLED SURVIVED
killing is good
My code is much more
complex than adding two
numbers!
Mutation
Operators
Math
+ to - * to /
- to + / to *
Boundary
< to <= > to >=
<= to < <= to >
Negate
== to !=
>= to <
Complex
Remove IFs
Complex
Remove IFs Remove method calls
Complex
Remove IFs
Modify return
Remove method calls
Complex
Remove IFs
Modify constants
Modify return
Remove method calls
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Do I need to change the
code on my own?
PIT Mutation Testing
public class Calculator {
public int sum(int a, int b) {
return a + b;
}
}
@Test
public
void force100PercentCoverage() {
calculator.sum(0, 0);
}
DEMO
> MathMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
@Test public void
Should_ReturnSum_When_SummingTwoValues() {
// given
int a = 1;
int b = 2;
// when
int result = calculator.sum(a, b);
// then
assertThat(result).isEqualTo(3);
}
> MathMutator
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0
> TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0
> STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
Do I need to read the
console?
DEMO
How about bigger
projects?
Hazelcast Kubernetes Plugin
● 5000 LOC
● 12 classes
DEMO
Why NASA engineers
didn't use it in 1998?
1971
Richard
Lipton
1971
Richard
Lipton
2000
1971
Richard
Lipton
2000 2012
1
Equivalent
Mutants
if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}
if (a >= 100) {
throw new IllegalArgumentException();
}
if (a < 100) {
fooBar();
}
2
Slow
Performance
Mutation Testing with PIT
Mutation Testing with PIT
return a + b;
return a + b;
return a + b;
return a - b
return 0
return a + b;
return a - b
return 0
:)
What about my
team?
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Main class paradox
Mutation Testing with PIT
Mutation Testing with PIT
// given
// given
// when
// given
// when
?
What to do with people
who write tests without
assertions?
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Do people really use
mutation testing?
Mutation Testing with PIT
Mutation Testing with PIT
Test your tests
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Mutation Testing with PIT
Thank You!
Rafał Leszko
@RafalLeszko
rafalleszko.com
1 of 122

Recommended

Mutation testing with PIT by
Mutation testing with PITMutation testing with PIT
Mutation testing with PITRafał Leszko
140 views122 slides
Is your C# optimized by
Is your C# optimizedIs your C# optimized
Is your C# optimizedWoody Pewitt
1.1K views23 slides
Java performance by
Java performanceJava performance
Java performanceSergey D
745 views18 slides
Introduction to Reactive Extensions without saying functional reactive by
Introduction to Reactive Extensions without saying functional reactiveIntroduction to Reactive Extensions without saying functional reactive
Introduction to Reactive Extensions without saying functional reactiveTetsuharu OHZEKI
611 views34 slides
SNP STEAM Academy 2017 Class #12 by
SNP STEAM Academy 2017 Class #12SNP STEAM Academy 2017 Class #12
SNP STEAM Academy 2017 Class #12Markus Van Kempen
66 views43 slides
Lecture05 by
Lecture05Lecture05
Lecture05elearning_portal
617 views15 slides

More Related Content

What's hot

Navigating the xDD Alphabet Soup by
Navigating the xDD Alphabet SoupNavigating the xDD Alphabet Soup
Navigating the xDD Alphabet SoupDror Helper
968 views73 slides
Python Programming Essentials - M11 - Comparison and Logical Operators by
Python Programming Essentials - M11 - Comparison and Logical OperatorsPython Programming Essentials - M11 - Comparison and Logical Operators
Python Programming Essentials - M11 - Comparison and Logical OperatorsP3 InfoTech Solutions Pvt. Ltd.
1.4K views17 slides
Klee introduction by
Klee  introductionKlee  introduction
Klee introductionGeorgiana T.
1.8K views7 slides
4. Обработка ошибок, исключения, отладка by
4. Обработка ошибок, исключения, отладка4. Обработка ошибок, исключения, отладка
4. Обработка ошибок, исключения, отладкаDEVTYPE
1.1K views34 slides
Monads in Swift by
Monads in SwiftMonads in Swift
Monads in SwiftVincent Pradeilles
7.9K views33 slides
Боремся с NPE вместе с Kotlin, Павел Шацких СберТех by
Боремся с NPE вместе с Kotlin, Павел Шацких СберТехБоремся с NPE вместе с Kotlin, Павел Шацких СберТех
Боремся с NPE вместе с Kotlin, Павел Шацких СберТехСбертех | SberTech
167 views33 slides

What's hot(20)

Navigating the xDD Alphabet Soup by Dror Helper
Navigating the xDD Alphabet SoupNavigating the xDD Alphabet Soup
Navigating the xDD Alphabet Soup
Dror Helper968 views
Klee introduction by Georgiana T.
Klee  introductionKlee  introduction
Klee introduction
Georgiana T.1.8K views
4. Обработка ошибок, исключения, отладка by DEVTYPE
4. Обработка ошибок, исключения, отладка4. Обработка ошибок, исключения, отладка
4. Обработка ошибок, исключения, отладка
DEVTYPE1.1K views
Боремся с NPE вместе с Kotlin, Павел Шацких СберТех by Сбертех | SberTech
Боремся с NPE вместе с Kotlin, Павел Шацких СберТехБоремся с NPE вместе с Kotlin, Павел Шацких СберТех
Боремся с NPE вместе с Kotlin, Павел Шацких СберТех
關於測試,我說的其實是...... by hugo lu
關於測試,我說的其實是......關於測試,我說的其實是......
關於測試,我說的其實是......
hugo lu8.6K views
Building unit tests correctly by Dror Helper
Building unit tests correctlyBuilding unit tests correctly
Building unit tests correctly
Dror Helper1.4K views
Project_Euler_No_104_Pandigital_Fibonacci_ends by ? ?
Project_Euler_No_104_Pandigital_Fibonacci_endsProject_Euler_No_104_Pandigital_Fibonacci_ends
Project_Euler_No_104_Pandigital_Fibonacci_ends
? ?315 views
JavaScript Control Statements I by Reem Alattas
JavaScript Control Statements IJavaScript Control Statements I
JavaScript Control Statements I
Reem Alattas3.4K views
Chapter 5.3 by sotlsoc
Chapter 5.3Chapter 5.3
Chapter 5.3
sotlsoc278 views

Similar to Mutation Testing with PIT

Mutation Testing by
Mutation TestingMutation Testing
Mutation TestingChris Sinjakli
685 views24 slides
Building unit tests correctly with visual studio 2013 by
Building unit tests correctly with visual studio 2013Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013Dror Helper
2.8K views46 slides
Kill the mutants and test your tests - Roy van Rijn by
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnNLJUG
1.5K views55 slides
Kill the mutants - A better way to test your tests by
Kill the mutants - A better way to test your testsKill the mutants - A better way to test your tests
Kill the mutants - A better way to test your testsRoy van Rijn
9K views55 slides
Java tutorial PPT by
Java tutorial PPTJava tutorial PPT
Java tutorial PPTIntelligo Technologies
177.7K views55 slides
Java tutorial PPT by
Java tutorial  PPTJava tutorial  PPT
Java tutorial PPTIntelligo Technologies
6.6K views55 slides

Similar to Mutation Testing with PIT(20)

Building unit tests correctly with visual studio 2013 by Dror Helper
Building unit tests correctly with visual studio 2013Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013
Dror Helper2.8K views
Kill the mutants and test your tests - Roy van Rijn by NLJUG
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van Rijn
NLJUG1.5K views
Kill the mutants - A better way to test your tests by Roy van Rijn
Kill the mutants - A better way to test your testsKill the mutants - A better way to test your tests
Kill the mutants - A better way to test your tests
Roy van Rijn9K views
MUTANTS KILLER - PIT: state of the art of mutation testing system by Tarin Gamberini
MUTANTS KILLER - PIT: state of the art of mutation testing system MUTANTS KILLER - PIT: state of the art of mutation testing system
MUTANTS KILLER - PIT: state of the art of mutation testing system
Tarin Gamberini3.1K views
Pi j1.3 operators by mcollison
Pi j1.3 operatorsPi j1.3 operators
Pi j1.3 operators
mcollison144 views
The state of PHPUnit by Edorian
The state of PHPUnitThe state of PHPUnit
The state of PHPUnit
Edorian2.1K views
TDD reloaded - JUGTAA 24 Ottobre 2012 by Pietro Di Bello
TDD reloaded - JUGTAA 24 Ottobre 2012TDD reloaded - JUGTAA 24 Ottobre 2012
TDD reloaded - JUGTAA 24 Ottobre 2012
Pietro Di Bello483 views
Surviving javascript.pptx by Tamas Rev
Surviving javascript.pptxSurviving javascript.pptx
Surviving javascript.pptx
Tamas Rev546 views
Tdd with python unittest for embedded c by Benux Wei
Tdd with python unittest for embedded cTdd with python unittest for embedded c
Tdd with python unittest for embedded c
Benux Wei5K views
Mutation Testing: Start Hunting The Bugs by Ari Waller
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The Bugs
Ari Waller107 views
Extreme Swift by Movel
Extreme SwiftExtreme Swift
Extreme Swift
Movel1K views
Chapter i(introduction to java) by Chhom Karath
Chapter i(introduction to java)Chapter i(introduction to java)
Chapter i(introduction to java)
Chhom Karath695 views

More from Rafał Leszko

Build Your Kubernetes Operator with the Right Tool! by
Build Your Kubernetes Operator with the Right Tool!Build Your Kubernetes Operator with the Right Tool!
Build Your Kubernetes Operator with the Right Tool!Rafał Leszko
193 views92 slides
Distributed Locking in Kubernetes by
Distributed Locking in KubernetesDistributed Locking in Kubernetes
Distributed Locking in KubernetesRafał Leszko
2K views90 slides
Architectural patterns for high performance microservices in kubernetes by
Architectural patterns for high performance microservices in kubernetesArchitectural patterns for high performance microservices in kubernetes
Architectural patterns for high performance microservices in kubernetesRafał Leszko
377 views90 slides
Architectural caching patterns for kubernetes by
Architectural caching patterns for kubernetesArchitectural caching patterns for kubernetes
Architectural caching patterns for kubernetesRafał Leszko
447 views88 slides
Architectural patterns for caching microservices by
Architectural patterns for caching microservicesArchitectural patterns for caching microservices
Architectural patterns for caching microservicesRafał Leszko
216 views91 slides
[jLove 2020] Where is my cache architectural patterns for caching microservi... by
[jLove 2020] Where is my cache  architectural patterns for caching microservi...[jLove 2020] Where is my cache  architectural patterns for caching microservi...
[jLove 2020] Where is my cache architectural patterns for caching microservi...Rafał Leszko
203 views86 slides

More from Rafał Leszko(20)

Build Your Kubernetes Operator with the Right Tool! by Rafał Leszko
Build Your Kubernetes Operator with the Right Tool!Build Your Kubernetes Operator with the Right Tool!
Build Your Kubernetes Operator with the Right Tool!
Rafał Leszko193 views
Distributed Locking in Kubernetes by Rafał Leszko
Distributed Locking in KubernetesDistributed Locking in Kubernetes
Distributed Locking in Kubernetes
Rafał Leszko2K views
Architectural patterns for high performance microservices in kubernetes by Rafał Leszko
Architectural patterns for high performance microservices in kubernetesArchitectural patterns for high performance microservices in kubernetes
Architectural patterns for high performance microservices in kubernetes
Rafał Leszko377 views
Architectural caching patterns for kubernetes by Rafał Leszko
Architectural caching patterns for kubernetesArchitectural caching patterns for kubernetes
Architectural caching patterns for kubernetes
Rafał Leszko447 views
Architectural patterns for caching microservices by Rafał Leszko
Architectural patterns for caching microservicesArchitectural patterns for caching microservices
Architectural patterns for caching microservices
Rafał Leszko216 views
[jLove 2020] Where is my cache architectural patterns for caching microservi... by Rafał Leszko
[jLove 2020] Where is my cache  architectural patterns for caching microservi...[jLove 2020] Where is my cache  architectural patterns for caching microservi...
[jLove 2020] Where is my cache architectural patterns for caching microservi...
Rafał Leszko203 views
Where is my cache architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache  architectural patterns for caching microservices by exampleWhere is my cache  architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by example
Rafał Leszko357 views
Architectural caching patterns for kubernetes by Rafał Leszko
Architectural caching patterns for kubernetesArchitectural caching patterns for kubernetes
Architectural caching patterns for kubernetes
Rafał Leszko258 views
Build your operator with the right tool by Rafał Leszko
Build your operator with the right toolBuild your operator with the right tool
Build your operator with the right tool
Rafał Leszko138 views
5 levels of high availability from multi instance to hybrid cloud by Rafał Leszko
5 levels of high availability  from multi instance to hybrid cloud5 levels of high availability  from multi instance to hybrid cloud
5 levels of high availability from multi instance to hybrid cloud
Rafał Leszko155 views
Where is my cache? Architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache? Architectural patterns for caching microservices by exampleWhere is my cache? Architectural patterns for caching microservices by example
Where is my cache? Architectural patterns for caching microservices by example
Rafał Leszko460 views
5 Levels of High Availability: From Multi-instance to Hybrid Cloud by Rafał Leszko
5 Levels of High Availability: From Multi-instance to Hybrid Cloud5 Levels of High Availability: From Multi-instance to Hybrid Cloud
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
Rafał Leszko516 views
Where is my cache architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache architectural patterns for caching microservices by exampleWhere is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by example
Rafał Leszko92 views
Where is my cache architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache architectural patterns for caching microservices by exampleWhere is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by example
Rafał Leszko728 views
Where is my cache? Architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache? Architectural patterns for caching microservices by exampleWhere is my cache? Architectural patterns for caching microservices by example
Where is my cache? Architectural patterns for caching microservices by example
Rafał Leszko556 views
[DevopsDays India 2019] Where is my cache? Architectural patterns for caching... by Rafał Leszko
[DevopsDays India 2019] Where is my cache? Architectural patterns for caching...[DevopsDays India 2019] Where is my cache? Architectural patterns for caching...
[DevopsDays India 2019] Where is my cache? Architectural patterns for caching...
Rafał Leszko344 views
Where is my cache? Architectural patterns for caching microservices by example by Rafał Leszko
Where is my cache? Architectural patterns for caching microservices by exampleWhere is my cache? Architectural patterns for caching microservices by example
Where is my cache? Architectural patterns for caching microservices by example
Rafał Leszko510 views
Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019 by Rafał Leszko
Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019
Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019
Rafał Leszko138 views
Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018 by Rafał Leszko
Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018
Stream Processing with Hazelcast Jet - Voxxed Days Thessaloniki 19.11.2018
Rafał Leszko1K views
Mutation Testing - Voxxed Days Cluj-Napoca 2017 by Rafał Leszko
Mutation Testing - Voxxed Days Cluj-Napoca 2017Mutation Testing - Voxxed Days Cluj-Napoca 2017
Mutation Testing - Voxxed Days Cluj-Napoca 2017
Rafał Leszko206 views

Recently uploaded

EV Charging App Case by
EV Charging App Case EV Charging App Case
EV Charging App Case iCoderz Solutions
10 views1 slide
Flask-Python by
Flask-PythonFlask-Python
Flask-PythonTriloki Gupta
10 views12 slides
predicting-m3-devopsconMunich-2023.pptx by
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxTier1 app
10 views24 slides
Electronic AWB - Electronic Air Waybill by
Electronic AWB - Electronic Air Waybill Electronic AWB - Electronic Air Waybill
Electronic AWB - Electronic Air Waybill Freightoscope
6 views1 slide
.NET Deserialization Attacks by
.NET Deserialization Attacks.NET Deserialization Attacks
.NET Deserialization AttacksDharmalingam Ganesan
7 views50 slides
How Workforce Management Software Empowers SMEs | TraQSuite by
How Workforce Management Software Empowers SMEs | TraQSuiteHow Workforce Management Software Empowers SMEs | TraQSuite
How Workforce Management Software Empowers SMEs | TraQSuiteTraQSuite
7 views3 slides

Recently uploaded(20)

predicting-m3-devopsconMunich-2023.pptx by Tier1 app
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptx
Tier1 app10 views
Electronic AWB - Electronic Air Waybill by Freightoscope
Electronic AWB - Electronic Air Waybill Electronic AWB - Electronic Air Waybill
Electronic AWB - Electronic Air Waybill
Freightoscope 6 views
How Workforce Management Software Empowers SMEs | TraQSuite by TraQSuite
How Workforce Management Software Empowers SMEs | TraQSuiteHow Workforce Management Software Empowers SMEs | TraQSuite
How Workforce Management Software Empowers SMEs | TraQSuite
TraQSuite7 views
aATP - New Correlation Confirmation Feature.pptx by EsatEsenek1
aATP - New Correlation Confirmation Feature.pptxaATP - New Correlation Confirmation Feature.pptx
aATP - New Correlation Confirmation Feature.pptx
EsatEsenek1222 views
Bootstrapping vs Venture Capital.pptx by Zeljko Svedic
Bootstrapping vs Venture Capital.pptxBootstrapping vs Venture Capital.pptx
Bootstrapping vs Venture Capital.pptx
Zeljko Svedic16 views
Streamlining Your Business Operations with Enterprise Application Integration... by Flexsin
Streamlining Your Business Operations with Enterprise Application Integration...Streamlining Your Business Operations with Enterprise Application Integration...
Streamlining Your Business Operations with Enterprise Application Integration...
Flexsin 5 views
Introduction to Git Source Control by John Valentino
Introduction to Git Source ControlIntroduction to Git Source Control
Introduction to Git Source Control
John Valentino8 views
Dapr Unleashed: Accelerating Microservice Development by Miroslav Janeski
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice Development
Miroslav Janeski16 views
JioEngage_Presentation.pptx by admin125455
JioEngage_Presentation.pptxJioEngage_Presentation.pptx
JioEngage_Presentation.pptx
admin1254559 views
Top-5-production-devconMunich-2023-v2.pptx by Tier1 app
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
Tier1 app9 views
Transport Management System - Shipment & Container Tracking by Freightoscope
Transport Management System - Shipment & Container TrackingTransport Management System - Shipment & Container Tracking
Transport Management System - Shipment & Container Tracking
Freightoscope 6 views

Mutation Testing with PIT