SlideShare a Scribd company logo
1 of 55
Download to read offline
KILL THE MUTANTS 
a better way to test your tests
ABOUT ME 
• Roy van Rijn 
• Mutants: 
• Nora 
• Lucas 
• Works for
let's do a 
SHOW OF HANDS
WHO DOES 
• Unit testing 
• Test-driven development (TDD) 
• Continuous integration 
• Measure code coverage 
• Mutation testing
UNIT TESTING 
• Prove your code works 
• Instant regression tests 
• Improve code design 
• Has become a mainstream practise over the last 10 years
CONTINUOUS INTEGRATION 
• Automate testing 
• Maintain a single source repository 
• Collect statistics
CODE COVERAGE 
• Measure the lines (or branches) that are executed during testing
CODE COVERAGE 
• How did they test your car?
CODE COVERAGE 
• Who has seen (or written?) tests 
• without verifications or assertions? 
• just to fake and boost coverage? 
• 100% branch coverage proves nothing
QUIS CUSTODIET IPSOS CUSTODES? 
Who watches the watchmen?
MUTATION TESTING 
• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize) 
• A better way to measure the quality of your tests 
• Surge of interest in the 1980s 
• Time to revive this interest!
TERMINOLOGY: MUTATION 
• A mutation is a (small) change in your codebase, for example:
TERMINOLOGY: MUTANT 
• A mutant is a mutated version of your class
MUTATION TESTING 
• Generate (a lot of) mutants of your codebase 
• Run (some of) your unit tests 
• Check the outcome!
OUTCOME #1: KILLED 
• A mutant is killed if a test fails (detecting the mutated code) 
• This proves the mutated code is properly tested
OUTCOME #2: LIVED 
• A mutant didn’t trigger a failing test…
OUTCOME #3: TIMED OUT 
• The mutant caused the program loop, get stuck
OTHER OUTCOMES 
• NON-VIABLE 
• JVM could not load the mutant bytecode 
• MEMORY ERROR 
• JVM ran out of memory during test 
• RUN ERROR 
• An error but none of the above.
FAULT INJECTION? 
• With fault injection you test code 
• Inject faults/mutations and see how the system reacts 
• With mutation testing you test your tests 
• Inject faults/mutations and see how the tests react
TOOLING 
• μJava: http://cs.gmu.edu/~offutt/mujava/ (inactive) 
• Jester: http://jester.sourceforge.net/ (inactive) 
• Jumble: http://jumble.sourceforge.net/ (inactive) 
• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive) 
• PIT: http://pitest.org/
USING PIT 
• PIT uses configurable ‘mutators' 
• ASM (bytecode manipulation) is used to mutate your code 
• No mutated code is stored, it can't interfere with your code 
• Generates reports with test results
MUTATORS: CONDITION BOUNDARY 
> into >= 
< into <= 
>= into > 
<= into <
MUTATORS: NEGATE CONDITIONALS 
== into != 
!= into == 
<= into > 
>= into < 
< into >= 
> into <=
MUTATORS: REMOVE CONDITIONALS 
into 
if(true) { 
//something 
} 
if(a == b) { 
//something 
}
MUTATORS: MATH 
+ into - 
- into + 
* into / 
/ into * 
% into * 
& into | 
<< into >> 
>> into << 
>>> into <<< 
a++ into a-- 
a-- into a++
MUTATORS: MANY MORE 
• Replacing return values (return a; becomes return 0;) 
• Removal of void invocations (doSomething(); is removed) 
• Some enabled by default, others are optional/configurable
MUTATION TESTING IS SLOW? 
• Speed was unacceptable in the 80's 
• Mutation testing is still CPU intensive 
• But PIT has a lot of methods to speed it up!
WHICH TESTS TO RUN? 
• PIT uses code coverage to decide which tests to run: 
• A mutation is on a line covered by 3 tests? Only run those.
SIMPLE EXAMPLE 
• 100 classes 
• 10 unit tests per class 
• 2 ms per unit test 
• Total time (all tests): 100 x 10 x 2ms = 2s
SIMPLE EXAMPLE 
• Total time (all tests): 100 x 10 x 2ms = 2s 
• 8 mutants per class, 100 classes x 8 = 800 mutants 
• Brute force: 800 x 2s = 26m40s 
• Smart testing: 800 x 10 x 2ms = 16s
LONGER EXAMPLE 
• Total time (all tests): 1000 x 10 x 2ms = 20s 
• 8 mutants per class, 1000 classes x 8 = 8000 mutants 
• Brute force: 8000 x 20s = 1d20h26m40s…!!! 
• Smart testing: 8000 x 10 x 2ms = 2m40s
PERFORMANCE TIPS 
• Write fast tests 
• Good separation or concerns 
• Use small classes, keep amount of unit tests per class low
INCREMENTAL ANALYSIS 
• Experimental feature 
• Incremental analysis keeps track of: 
• Changes in the codebase 
• Previous results
HOW ABOUT MOCKING? 
• PIT has support for: 
• Mockito, EasyMock, JMock, PowerMock and JMockit
HOW TO USE PIT? 
• Standalone Java process 
• Build: Ant task, Maven plugin 
• CI: Sonarqube plugin, Gradle plugin 
• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
STANDALONE JAVA 
java -cp <your classpath including pit jar and dependencies> 
org.pitest.mutationtest.commandline.MutationCoverageReport 
--reportDir /somePath/ 
--targetClasses com.your.package.tobemutated* 
--targetTests com.your.package.* 
--sourceDirs /sourcePath/
MAVEN PLUGIN 
<plugin> 
<groupId>org.pitest</groupId> 
<artifactId>pitest-maven</artifactId> 
<version>1.0.0</version> 
<configuration> 
<targetClasses> 
<param>com.your.package.tobemutated*</param> 
</targetClasses> 
<jvmArgs>…</jvmArgs> 
</configuration> 
</plugin> 
Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage
EXAMPLE 
Let’s kill some mutants… or be killed.
USE CASE 
The price of an item is 17 euro 
If you buy 20 or more, all items cost 15 euro 
If you have a coupon, all items cost 15 euro
CODE 
public int getPrice(int amountOfThings, boolean coupon) { 
if (amountOfThings >= 20 || coupon) { 
return amountOfThings * 15; 
} 
return amountOfThings * 17; 
}
TEST #1 
@Test 
public void testNormalPricing() { 
//Not enough for discount: 
int amount = 1; 
Assert.assertEquals(17, businessLogic.getPrice(amount, false)); 
}
BRANCH COVERAGE 
public int getPrice(int amountOfThings, boolean coupon) { 
if (amountOfThings >= 20 || coupon) { 
return amountOfThings * 15; 
} 
return amountOfThings * 17; 
}
TEST #2 
@Test 
public void testDiscountPricingByAmount() { 
//Enough for discount: 
int amount = 100; 
Assert.assertEquals(1500, businessLogic.getPrice(amount, false)); 
}
BRANCH COVERAGE 
public int getPrice(int amountOfThings, boolean coupon) { 
if (amountOfThings >= 20 || coupon) { 
return amountOfThings * 15; 
} 
return amountOfThings * 17; 
}
TEST #3 
@Test 
public void testDiscountWithCoupon() { 
//Not enough for discount, but coupon: 
int amount = 1; 
Assert.assertEquals(15, businessLogic.getPrice(amount, true)); 
}
BRANCH COVERAGE 
public int getPrice(int amountOfThings, boolean coupon) { 
if (amountOfThings >= 20 || coupon) { 
return amountOfThings * 15; 
} 
return amountOfThings * 17; 
}
PIT RESULT
PIT RESULT 
> org.pitest.mutationtest…ConditionalsBoundaryMutator 
>> Generated 1 Killed 0 (0%) 
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 0 
PIT tells us: Changing >= into > doesn’t trigger a failing test
TEST #4 
@Test 
public void testDiscountAmountCornerCase() { 
//Just enough for discount, mutation into > should fail this test 
int amount = 20; 
Assert.assertEquals(300, businessLogic.getPrice(amount, true)); 
}
BRANCH COVERAGE 
public int getPrice(int amountOfThings, boolean coupon) { 
if (amountOfThings >= 20 || coupon) { 
return amountOfThings * 15; 
} 
return amountOfThings * 17; 
}
PIT RESULT
PIT RESULT 
> org.pitest.mutationtest…ConditionalsBoundaryMutator 
>> Generated 1 Killed 0 (0%) 
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 0 
STILL WRONG!?
DID YOU SPOT THE BUG? 
@Test 
public void testDiscountAmountCornerCase() { 
//Just enough for discount, mutation into > should fail this test 
int amount = 20; 
Assert.assertEquals(300, businessLogic.getPrice(amount, true)); 
}
SUMMARY 
• Mutation testing automatically tests your tests 
• Mutation testing can find bugs in your tests 
• Code coverage is wrong, gives a false sense of security 
• Mutation testing with PIT is easy to implement
QUESTIONS?

More Related Content

What's hot

Mutation Testing and MuJava
Mutation Testing and MuJavaMutation Testing and MuJava
Mutation Testing and MuJavaKrunal Parmar
 
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
 Tips for Writing Better Charters for Exploratory Testing Sessions by Michael... Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...TEST Huddle
 
TestNG - The Next Generation of Unit Testing
TestNG - The Next Generation of Unit TestingTestNG - The Next Generation of Unit Testing
TestNG - The Next Generation of Unit TestingBethmi Gunasekara
 
Top 50 Software Testing Interview Questions & Answers | Edureka
Top 50 Software Testing Interview Questions & Answers | EdurekaTop 50 Software Testing Interview Questions & Answers | Edureka
Top 50 Software Testing Interview Questions & Answers | EdurekaEdureka!
 
Unit tests & TDD
Unit tests & TDDUnit tests & TDD
Unit tests & TDDDror Helper
 
An Introduction to Unit Testing
An Introduction to Unit TestingAn Introduction to Unit Testing
An Introduction to Unit TestingJoe Tremblay
 
An introduction to Google test framework
An introduction to Google test frameworkAn introduction to Google test framework
An introduction to Google test frameworkAbner Chih Yi Huang
 
Unit Testing in Python
Unit Testing in PythonUnit Testing in Python
Unit Testing in PythonHaim Michael
 
[Webinar] Qt Test-Driven Development Using Google Test and Google Mock
[Webinar] Qt Test-Driven Development Using Google Test and Google Mock[Webinar] Qt Test-Driven Development Using Google Test and Google Mock
[Webinar] Qt Test-Driven Development Using Google Test and Google MockICS
 

What's hot (20)

Unit test
Unit testUnit test
Unit test
 
testng
testngtestng
testng
 
Unit Testing
Unit TestingUnit Testing
Unit Testing
 
O papel do qa (testador) em um time ágil
O papel do qa (testador) em um time ágilO papel do qa (testador) em um time ágil
O papel do qa (testador) em um time ágil
 
Mutation Testing and MuJava
Mutation Testing and MuJavaMutation Testing and MuJava
Mutation Testing and MuJava
 
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
 Tips for Writing Better Charters for Exploratory Testing Sessions by Michael... Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
Tips for Writing Better Charters for Exploratory Testing Sessions by Michael...
 
Python unit testing
Python unit testingPython unit testing
Python unit testing
 
TestNG - The Next Generation of Unit Testing
TestNG - The Next Generation of Unit TestingTestNG - The Next Generation of Unit Testing
TestNG - The Next Generation of Unit Testing
 
Regression testing
Regression testingRegression testing
Regression testing
 
Workshop unit test
Workshop   unit testWorkshop   unit test
Workshop unit test
 
Top 50 Software Testing Interview Questions & Answers | Edureka
Top 50 Software Testing Interview Questions & Answers | EdurekaTop 50 Software Testing Interview Questions & Answers | Edureka
Top 50 Software Testing Interview Questions & Answers | Edureka
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Unit tests & TDD
Unit tests & TDDUnit tests & TDD
Unit tests & TDD
 
An Introduction to Unit Testing
An Introduction to Unit TestingAn Introduction to Unit Testing
An Introduction to Unit Testing
 
Robot Framework
Robot FrameworkRobot Framework
Robot Framework
 
Junit 4.0
Junit 4.0Junit 4.0
Junit 4.0
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
An introduction to Google test framework
An introduction to Google test frameworkAn introduction to Google test framework
An introduction to Google test framework
 
Unit Testing in Python
Unit Testing in PythonUnit Testing in Python
Unit Testing in Python
 
[Webinar] Qt Test-Driven Development Using Google Test and Google Mock
[Webinar] Qt Test-Driven Development Using Google Test and Google Mock[Webinar] Qt Test-Driven Development Using Google Test and Google Mock
[Webinar] Qt Test-Driven Development Using Google Test and Google Mock
 

Viewers also liked

Kill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnNLJUG
 
I.T.A.K.E Unconference - Mutation testing to the rescue of your tests
I.T.A.K.E Unconference - Mutation testing to the rescue of your testsI.T.A.K.E Unconference - Mutation testing to the rescue of your tests
I.T.A.K.E Unconference - Mutation testing to the rescue of your testsNicolas Fränkel
 
Mutation Testing
Mutation TestingMutation Testing
Mutation Testing10Pines
 
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014Samir Bessalah
 
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system Tarin Gamberini
 
Automation using-phing
Automation using-phingAutomation using-phing
Automation using-phingRajat Pandit
 
Efficient Parallel Testing with Docker
Efficient Parallel Testing with DockerEfficient Parallel Testing with Docker
Efficient Parallel Testing with DockerLaura Frank Tacho
 
How good are your tests?
How good are your tests?How good are your tests?
How good are your tests?Noam Shaish
 
Testing with Docker
Testing with DockerTesting with Docker
Testing with Dockertoffermann
 
Using Docker for Testing
Using Docker for TestingUsing Docker for Testing
Using Docker for TestingMukta Aphale
 
National Civic Summit - BBB Civic Ethics - Lisa Jemtrud
National Civic Summit - BBB Civic Ethics - Lisa JemtrudNational Civic Summit - BBB Civic Ethics - Lisa Jemtrud
National Civic Summit - BBB Civic Ethics - Lisa JemtrudNational Civic Summit
 
cchandler_portfolioFeb16.3
cchandler_portfolioFeb16.3cchandler_portfolioFeb16.3
cchandler_portfolioFeb16.3Charles Chandler
 
Newsbrands and social media
Newsbrands and social media Newsbrands and social media
Newsbrands and social media Newsworks
 
Guia de estudio séptimo
Guia de estudio séptimoGuia de estudio séptimo
Guia de estudio séptimojairo
 
Online audience research
Online audience researchOnline audience research
Online audience researchchino rambo
 
Resume update executive it
Resume  update executive itResume  update executive it
Resume update executive itAjaya Mohanty
 
6° básico b semana del lunes 07 al 11 de noviembre
6° básico b semana del lunes 07 al 11 de noviembre  6° básico b semana del lunes 07 al 11 de noviembre
6° básico b semana del lunes 07 al 11 de noviembre Colegio Camilo Henríquez
 

Viewers also liked (20)

Kill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van Rijn
 
I.T.A.K.E Unconference - Mutation testing to the rescue of your tests
I.T.A.K.E Unconference - Mutation testing to the rescue of your testsI.T.A.K.E Unconference - Mutation testing to the rescue of your tests
I.T.A.K.E Unconference - Mutation testing to the rescue of your tests
 
Mutation testing in Java
Mutation testing in JavaMutation testing in Java
Mutation testing in Java
 
Mutation Testing
Mutation TestingMutation Testing
Mutation Testing
 
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014Algebird : Abstract Algebra for big data analytics. Devoxx 2014
Algebird : Abstract Algebra for big data analytics. Devoxx 2014
 
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
 
Automation using-phing
Automation using-phingAutomation using-phing
Automation using-phing
 
Efficient Parallel Testing with Docker
Efficient Parallel Testing with DockerEfficient Parallel Testing with Docker
Efficient Parallel Testing with Docker
 
How good are your tests?
How good are your tests?How good are your tests?
How good are your tests?
 
Testing with Docker
Testing with DockerTesting with Docker
Testing with Docker
 
Using Docker for Testing
Using Docker for TestingUsing Docker for Testing
Using Docker for Testing
 
Javantura v3 - Mutation Testing for everyone – Nicolas Fränkel
Javantura v3 - Mutation Testing for everyone – Nicolas FränkelJavantura v3 - Mutation Testing for everyone – Nicolas Fränkel
Javantura v3 - Mutation Testing for everyone – Nicolas Fränkel
 
National Civic Summit - BBB Civic Ethics - Lisa Jemtrud
National Civic Summit - BBB Civic Ethics - Lisa JemtrudNational Civic Summit - BBB Civic Ethics - Lisa Jemtrud
National Civic Summit - BBB Civic Ethics - Lisa Jemtrud
 
cchandler_portfolioFeb16.3
cchandler_portfolioFeb16.3cchandler_portfolioFeb16.3
cchandler_portfolioFeb16.3
 
Newsbrands and social media
Newsbrands and social media Newsbrands and social media
Newsbrands and social media
 
Guia de estudio séptimo
Guia de estudio séptimoGuia de estudio séptimo
Guia de estudio séptimo
 
Pdcs 1
Pdcs 1Pdcs 1
Pdcs 1
 
Online audience research
Online audience researchOnline audience research
Online audience research
 
Resume update executive it
Resume  update executive itResume  update executive it
Resume update executive it
 
6° básico b semana del lunes 07 al 11 de noviembre
6° básico b semana del lunes 07 al 11 de noviembre  6° básico b semana del lunes 07 al 11 de noviembre
6° básico b semana del lunes 07 al 11 de noviembre
 

Similar to Kill the mutants - A better way to test your tests

Cpp Testing Techniques Tips and Tricks - Cpp Europe
Cpp Testing Techniques Tips and Tricks - Cpp EuropeCpp Testing Techniques Tips and Tricks - Cpp Europe
Cpp Testing Techniques Tips and Tricks - Cpp EuropeClare Macrae
 
MUTANTS KILLER - PIT: state of the art of mutation testing system
MUTANTS KILLER - PIT: state of the art of mutation testing system MUTANTS KILLER - PIT: state of the art of mutation testing system
MUTANTS KILLER - PIT: state of the art of mutation testing system Tarin Gamberini
 
Qt test framework
Qt test frameworkQt test framework
Qt test frameworkICS
 
Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013Dror Helper
 
Testing the waters of iOS
Testing the waters of iOSTesting the waters of iOS
Testing the waters of iOSKremizas Kostas
 
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...TEST Huddle
 
Wix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxWix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxEfrat Attas
 
New types of tests for Java projects
New types of tests for Java projectsNew types of tests for Java projects
New types of tests for Java projectsVincent Massol
 
Implementing Quality on a Java Project
Implementing Quality on a Java ProjectImplementing Quality on a Java Project
Implementing Quality on a Java ProjectVincent Massol
 
Renaissance of JUnit - Introduction to JUnit 5
Renaissance of JUnit - Introduction to JUnit 5Renaissance of JUnit - Introduction to JUnit 5
Renaissance of JUnit - Introduction to JUnit 5Jimmy Lu
 
We Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentWe Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentAll Things Open
 
Mutation testing
Mutation testingMutation testing
Mutation testing기영 이
 
Level Up Your Salesforce Unit Testing
Level Up Your Salesforce Unit TestingLevel Up Your Salesforce Unit Testing
Level Up Your Salesforce Unit TestingGordon Bockus
 
Principles and patterns for test driven development
Principles and patterns for test driven developmentPrinciples and patterns for test driven development
Principles and patterns for test driven developmentStephen Fuqua
 
Test Driven Development with Sql Server
Test Driven Development with Sql ServerTest Driven Development with Sql Server
Test Driven Development with Sql ServerDavid P. Moore
 
Oop 2015 – Mutation Testing
Oop 2015 – Mutation TestingOop 2015 – Mutation Testing
Oop 2015 – Mutation TestingFilip Van Laenen
 
An Introduction to unit testing
An Introduction to unit testingAn Introduction to unit testing
An Introduction to unit testingSteven Casey
 

Similar to Kill the mutants - A better way to test your tests (20)

Test driven development
Test driven developmentTest driven development
Test driven development
 
Cpp Testing Techniques Tips and Tricks - Cpp Europe
Cpp Testing Techniques Tips and Tricks - Cpp EuropeCpp Testing Techniques Tips and Tricks - Cpp Europe
Cpp Testing Techniques Tips and Tricks - Cpp Europe
 
MUTANTS KILLER - PIT: state of the art of mutation testing system
MUTANTS KILLER - PIT: state of the art of mutation testing system MUTANTS KILLER - PIT: state of the art of mutation testing system
MUTANTS KILLER - PIT: state of the art of mutation testing system
 
Qt test framework
Qt test frameworkQt test framework
Qt test framework
 
Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013Building unit tests correctly with visual studio 2013
Building unit tests correctly with visual studio 2013
 
Testing the waters of iOS
Testing the waters of iOSTesting the waters of iOS
Testing the waters of iOS
 
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...
Testing As A Bottleneck - How Testing Slows Down Modern Development Processes...
 
Wix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxWix Automation - The False Positive Paradox
Wix Automation - The False Positive Paradox
 
New types of tests for Java projects
New types of tests for Java projectsNew types of tests for Java projects
New types of tests for Java projects
 
Implementing Quality on a Java Project
Implementing Quality on a Java ProjectImplementing Quality on a Java Project
Implementing Quality on a Java Project
 
Renaissance of JUnit - Introduction to JUnit 5
Renaissance of JUnit - Introduction to JUnit 5Renaissance of JUnit - Introduction to JUnit 5
Renaissance of JUnit - Introduction to JUnit 5
 
We Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End DevelopmentWe Are All Testers Now: The Testing Pyramid and Front-End Development
We Are All Testers Now: The Testing Pyramid and Front-End Development
 
Mutation testing
Mutation testingMutation testing
Mutation testing
 
Level Up Your Salesforce Unit Testing
Level Up Your Salesforce Unit TestingLevel Up Your Salesforce Unit Testing
Level Up Your Salesforce Unit Testing
 
des mutants dans le code.pdf
des mutants dans le code.pdfdes mutants dans le code.pdf
des mutants dans le code.pdf
 
Unit testing basics
Unit testing basicsUnit testing basics
Unit testing basics
 
Principles and patterns for test driven development
Principles and patterns for test driven developmentPrinciples and patterns for test driven development
Principles and patterns for test driven development
 
Test Driven Development with Sql Server
Test Driven Development with Sql ServerTest Driven Development with Sql Server
Test Driven Development with Sql Server
 
Oop 2015 – Mutation Testing
Oop 2015 – Mutation TestingOop 2015 – Mutation Testing
Oop 2015 – Mutation Testing
 
An Introduction to unit testing
An Introduction to unit testingAn Introduction to unit testing
An Introduction to unit testing
 

Recently uploaded

^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbankkasambamuno
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?NYGGS Automation Suite
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Henry Schreiner
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...drm1699
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypseTomasz Kowalczewski
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Andrea Goulet
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto
^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto
^Clinic ^%[+27788225528*Abortion Pills For Sale In sowetokasambamuno
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIInflectra
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdfSelfMade bd
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringPrakhyath Rai
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jNeo4j
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphNeo4j
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfWSO2
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfICS
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In hararekasambamuno
 

Recently uploaded (20)

^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto
^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto
^Clinic ^%[+27788225528*Abortion Pills For Sale In soweto
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
^Clinic ^%[+27788225528*Abortion Pills For Sale In harare
 

Kill the mutants - A better way to test your tests

  • 1. KILL THE MUTANTS a better way to test your tests
  • 2. ABOUT ME • Roy van Rijn • Mutants: • Nora • Lucas • Works for
  • 3. let's do a SHOW OF HANDS
  • 4. WHO DOES • Unit testing • Test-driven development (TDD) • Continuous integration • Measure code coverage • Mutation testing
  • 5. UNIT TESTING • Prove your code works • Instant regression tests • Improve code design • Has become a mainstream practise over the last 10 years
  • 6. CONTINUOUS INTEGRATION • Automate testing • Maintain a single source repository • Collect statistics
  • 7. CODE COVERAGE • Measure the lines (or branches) that are executed during testing
  • 8. CODE COVERAGE • How did they test your car?
  • 9. CODE COVERAGE • Who has seen (or written?) tests • without verifications or assertions? • just to fake and boost coverage? • 100% branch coverage proves nothing
  • 10. QUIS CUSTODIET IPSOS CUSTODES? Who watches the watchmen?
  • 11. MUTATION TESTING • Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize) • A better way to measure the quality of your tests • Surge of interest in the 1980s • Time to revive this interest!
  • 12. TERMINOLOGY: MUTATION • A mutation is a (small) change in your codebase, for example:
  • 13. TERMINOLOGY: MUTANT • A mutant is a mutated version of your class
  • 14. MUTATION TESTING • Generate (a lot of) mutants of your codebase • Run (some of) your unit tests • Check the outcome!
  • 15. OUTCOME #1: KILLED • A mutant is killed if a test fails (detecting the mutated code) • This proves the mutated code is properly tested
  • 16. OUTCOME #2: LIVED • A mutant didn’t trigger a failing test…
  • 17. OUTCOME #3: TIMED OUT • The mutant caused the program loop, get stuck
  • 18. OTHER OUTCOMES • NON-VIABLE • JVM could not load the mutant bytecode • MEMORY ERROR • JVM ran out of memory during test • RUN ERROR • An error but none of the above.
  • 19. FAULT INJECTION? • With fault injection you test code • Inject faults/mutations and see how the system reacts • With mutation testing you test your tests • Inject faults/mutations and see how the tests react
  • 20. TOOLING • μJava: http://cs.gmu.edu/~offutt/mujava/ (inactive) • Jester: http://jester.sourceforge.net/ (inactive) • Jumble: http://jumble.sourceforge.net/ (inactive) • javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive) • PIT: http://pitest.org/
  • 21. USING PIT • PIT uses configurable ‘mutators' • ASM (bytecode manipulation) is used to mutate your code • No mutated code is stored, it can't interfere with your code • Generates reports with test results
  • 22. MUTATORS: CONDITION BOUNDARY > into >= < into <= >= into > <= into <
  • 23. MUTATORS: NEGATE CONDITIONALS == into != != into == <= into > >= into < < into >= > into <=
  • 24. MUTATORS: REMOVE CONDITIONALS into if(true) { //something } if(a == b) { //something }
  • 25. MUTATORS: MATH + into - - into + * into / / into * % into * & into | << into >> >> into << >>> into <<< a++ into a-- a-- into a++
  • 26. MUTATORS: MANY MORE • Replacing return values (return a; becomes return 0;) • Removal of void invocations (doSomething(); is removed) • Some enabled by default, others are optional/configurable
  • 27. MUTATION TESTING IS SLOW? • Speed was unacceptable in the 80's • Mutation testing is still CPU intensive • But PIT has a lot of methods to speed it up!
  • 28. WHICH TESTS TO RUN? • PIT uses code coverage to decide which tests to run: • A mutation is on a line covered by 3 tests? Only run those.
  • 29. SIMPLE EXAMPLE • 100 classes • 10 unit tests per class • 2 ms per unit test • Total time (all tests): 100 x 10 x 2ms = 2s
  • 30. SIMPLE EXAMPLE • Total time (all tests): 100 x 10 x 2ms = 2s • 8 mutants per class, 100 classes x 8 = 800 mutants • Brute force: 800 x 2s = 26m40s • Smart testing: 800 x 10 x 2ms = 16s
  • 31. LONGER EXAMPLE • Total time (all tests): 1000 x 10 x 2ms = 20s • 8 mutants per class, 1000 classes x 8 = 8000 mutants • Brute force: 8000 x 20s = 1d20h26m40s…!!! • Smart testing: 8000 x 10 x 2ms = 2m40s
  • 32. PERFORMANCE TIPS • Write fast tests • Good separation or concerns • Use small classes, keep amount of unit tests per class low
  • 33. INCREMENTAL ANALYSIS • Experimental feature • Incremental analysis keeps track of: • Changes in the codebase • Previous results
  • 34. HOW ABOUT MOCKING? • PIT has support for: • Mockito, EasyMock, JMock, PowerMock and JMockit
  • 35. HOW TO USE PIT? • Standalone Java process • Build: Ant task, Maven plugin • CI: Sonarqube plugin, Gradle plugin • IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
  • 36. STANDALONE JAVA java -cp <your classpath including pit jar and dependencies> org.pitest.mutationtest.commandline.MutationCoverageReport --reportDir /somePath/ --targetClasses com.your.package.tobemutated* --targetTests com.your.package.* --sourceDirs /sourcePath/
  • 37. MAVEN PLUGIN <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.0.0</version> <configuration> <targetClasses> <param>com.your.package.tobemutated*</param> </targetClasses> <jvmArgs>…</jvmArgs> </configuration> </plugin> Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage
  • 38. EXAMPLE Let’s kill some mutants… or be killed.
  • 39. USE CASE The price of an item is 17 euro If you buy 20 or more, all items cost 15 euro If you have a coupon, all items cost 15 euro
  • 40. CODE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  • 41. TEST #1 @Test public void testNormalPricing() { //Not enough for discount: int amount = 1; Assert.assertEquals(17, businessLogic.getPrice(amount, false)); }
  • 42. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  • 43. TEST #2 @Test public void testDiscountPricingByAmount() { //Enough for discount: int amount = 100; Assert.assertEquals(1500, businessLogic.getPrice(amount, false)); }
  • 44. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  • 45. TEST #3 @Test public void testDiscountWithCoupon() { //Not enough for discount, but coupon: int amount = 1; Assert.assertEquals(15, businessLogic.getPrice(amount, true)); }
  • 46. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  • 48. PIT RESULT > org.pitest.mutationtest…ConditionalsBoundaryMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 PIT tells us: Changing >= into > doesn’t trigger a failing test
  • 49. TEST #4 @Test public void testDiscountAmountCornerCase() { //Just enough for discount, mutation into > should fail this test int amount = 20; Assert.assertEquals(300, businessLogic.getPrice(amount, true)); }
  • 50. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  • 52. PIT RESULT > org.pitest.mutationtest…ConditionalsBoundaryMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 STILL WRONG!?
  • 53. DID YOU SPOT THE BUG? @Test public void testDiscountAmountCornerCase() { //Just enough for discount, mutation into > should fail this test int amount = 20; Assert.assertEquals(300, businessLogic.getPrice(amount, true)); }
  • 54. SUMMARY • Mutation testing automatically tests your tests • Mutation testing can find bugs in your tests • Code coverage is wrong, gives a false sense of security • Mutation testing with PIT is easy to implement