SlideShare a Scribd company logo
1 of 51
Download to read offline
Mutate	and	Test	Your	Tests
Author: Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
1
baudry@kth.se caroline.landry@inria.fr
Speaker: Caroline Landry
Project Technical Manager
INRIA, France
POSS	17	- 2 Caroline Landry, INRIABenoit Baudry, KTH
Test	Your	Tests
•What	do	you	expect	from	test	cases?
• Cover	requirements
• Stress	the	application
• Prevent	regressions
• Reveal	bugs
POSS	17	- 3 Caroline Landry, INRIABenoit Baudry, KTH
Test	Your	Tests
•What	do	you	expect	from	test	cases?
• Cover	requirements
• Stress	the	application
• Prevent	regressions
• Reveal	bugs
POSS	17	- 4 Caroline Landry, INRIABenoit Baudry, KTH
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
POSS	17	- 5 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
POSS	17	- 6 Caroline Landry, INRIABenoit Baudry, KTH
Coverage
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
POSS	17	- 7 Caroline Landry, INRIABenoit Baudry, KTH
Coverage
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
POSS	17	- 8 Caroline Landry, INRIABenoit Baudry, KTH
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are	these	test	cases	good	at	
detecting	bugs?
POSS	17	- 9 Caroline Landry, INRIABenoit Baudry, KTH
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are	these	test	cases	good	at	
detecting	bugs?
Let’s	mutate	our	code	to	see.
POSS	17	- 10 Caroline Landry, INRIABenoit Baudry, KTH
Mutation	analysis
•Tests	are	good	if	they	can	detect	bugs
•Principle:	generate	bugs	and	test	the	tests
POSS	17	- 11 Caroline Landry, INRIABenoit Baudry, KTH
Mutation	analysis
•Tests	are	good	if	they	can	detect	bugs
•Principle:	generate	bugs	and	test	the	tests
•Mutation	operators
•Mutant	=	Program	with	one	mutation
POSS	17	- 12 Caroline Landry, INRIABenoit Baudry, KTH
Mutation	analysis
Inputs : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)
POSS	17	- 13 Caroline Landry, INRIABenoit Baudry, KTH
PIT
•Open	source
•Actively	developed	and	maintained
•State	of	the	art	
•For	Java
•Integrates	with	major	build	systems
POSS	17	- 14 Caroline Landry, INRIABenoit Baudry, KTH
PIT	mutation	operators
•Conditions
•Constants
•Return
•Delete	method	calls
•Constructor	calls
POSS	17	- 15 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
POSS	17	- 16 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
n != 0
POSS	17	- 17 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
n != 0 return 1+1
POSS	17	- 18 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
n != 0 return 1+1
<
POSS	17	- 19 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 20 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 21 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 22 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 23 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 24 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation	reveals bugs	in	the	test	suite
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
Bugs	in	the	test	suite:
- Weak	oracle
- Missing	input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 25 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
POSS	17	- 26 Caroline Landry, INRIABenoit Baudry, KTH
PIT	– reporting
POSS	17	- 27 Caroline Landry, INRIABenoit Baudry, KTH
PIT	– reporting
POSS	17	- 28 Caroline Landry, INRIABenoit Baudry, KTH
Project #LOC #Mutants Time	(h) Score	(%)
Amazon	Web	Services	SDK 1703892 2141690 04:25:35 76.28
XWiki	Rendering	Engine 104727 112609 01:59:55 50.89
Apache	Commons	Math 180951 104786 03:22:18 83.81
JFreeChart 134353 89592 00:41:28 58.04
Apache	PdfBox 137099 79763 06:20:25 58.89
Java	Git 128874 78316 16:02:00 73.86
SCIFIO 55347 62768 03:12:11 45.92
Joda-Time 85911 31233 00:16:32 81.65
Apache	Commons	Lang 60733 30361 00:21:02 86.17
Apache	Commons	Collections 51632 20394 00:05:41 85.94
Urban	Airship	Client	Library 40885 17345 00:11:31 82.26
SAT4J 26415 17067 11:11:04 68.58
ImageJ	Common 23868 15592 00:29:09 54.77
jsoup 17839 14054 00:12:49 78.34
Jaxen	XPath	Engine 21204 12210 00:24:40 67.13
POSS	17	- 29 Caroline Landry, INRIABenoit Baudry, KTH
Descartes
•Mutation	operators:	extreme	mutation
•Active	development
•Open	source
•Compared	to	PIT
• Less	mutants
• Different	type	of	feedback
• Same	framework
POSS	17	- 30 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes	- example
POSS	17	- 31 Caroline Landry, INRIABenoit Baudry, KTH
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes	- example
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT	:	7	mutants
Descartes	:	2	mutants
POSS	17	- 32 Caroline Landry, INRIABenoit Baudry, KTH
Scalability	of	extreme	mutation
public static boolean isValidXmlChar(int ch){
return (ch == 0x9)
|| (ch == 0xA)
|| (ch == 0xD)
|| (ch >= 0x20 && ch <= 0xD7FF)
|| (ch >= 0xE000 && ch <= 0xFFFD
|| (ch >= 0x10000 && ch <= 0x10FFFF);
}
PIT	:	45	mutants
Descartes	:	2	mutants
POSS	17	- 33 Caroline Landry, INRIABenoit Baudry, KTH
Project Mutants	PIT Mutants	Descartes Time	PIT Time	Descartes
Amazon	Web	Services	SDK 2141690 161758 04:25:35 01:27:30
XWiki	Rendering	Engine 112609 5534 01:59:55 00:10:50
Apache	Commons	Math 104786 7150 03:22:18 00:08:30
JFreeChart 89592 7210 00:41:28 00:05:26
Apache	PdfBox 79763 7559 06:20:25 00:42:11
Java	Git 78316 7152 16:02:00 00:56:07
SCIFIO 62768 3627 03:12:11 00:15:26
Joda-Time 31233 4525 00:16:32 00:04:13
Apache	Commons	Lang 30361 3872 00:21:02 00:02:18
Apache	Commons	Collections 20394 3558 00:05:41 00:01:48
Urban	Airship	Client	Library 17345 3082 00:11:31 00:09:38
SAT4J 17067 2296 11:11:04 00:56:42
ImageJ	Common 15592 1947 00:29:09 00:04:08
jsoup 14054 1566 00:12:49 00:02:45
POSS	17	- 34 Caroline Landry, INRIABenoit Baudry, KTH
Coarser	grain	than	PIT
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
long fact(int n){
return 0;}
long fact(int n){
return 1;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith0Test() {
assertEqual(1, fact(0));}
POSS	17	- 35 Caroline Landry, INRIABenoit Baudry, KTH
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
POSS	17	- 36 Caroline Landry, INRIABenoit Baudry, KTH
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
POSS	17	- 37 Caroline Landry, INRIABenoit Baudry, KTH
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
POSS	17	- 38 Caroline Landry, INRIABenoit Baudry, KTH
Wrong	test
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
POSS	17	- 39 Caroline Landry, INRIABenoit Baudry, KTH
protected final void prepareSocket(final SSLSocket socket) { }}
POSS	17	- 40 Caroline Landry, INRIABenoit Baudry, KTH
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
f.prepareSocket(new TestSSLSocket() {
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, new String[] {
"TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" }));
}});}}}
protected final void prepareSocket(final SSLSocket socket) { }}
POSS	17	- 41 Caroline Landry, INRIABenoit Baudry, KTH
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
f.prepareSocket(new TestSSLSocket() {
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, new String[] {
"TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" }));
}});}}}
protected final void prepareSocket(final SSLSocket socket) { }}
POSS	17	- 42 Caroline Landry, INRIABenoit Baudry, KTH
Missing	oracle
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
f.prepareSocket(new TestSSLSocket() {
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, new String[] {
"TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" }));
}});}}}
protected final void prepareSocket(final SSLSocket socket) { }}
POSS	17	- 43 Caroline Landry, INRIABenoit Baudry, KTH
private boolean isLongOption(final String token){
if (...){return false;} //Covered 27 times
...
if (...){return true;} //Not covered
else if (...){return true;} //Covered once
return false; //Covered 7 times
}
POSS	17	- 44 Caroline Landry, INRIABenoit Baudry, KTH
private boolean isLongOption(final String token){
return false;}
private boolean isLongOption(final String token){
return true;}
POSS	17	- 45 Caroline Landry, INRIABenoit Baudry, KTH
...
if (condition && isLongOption(value)){
action1(value);}
else{
action2(value);}
...
private boolean isLongOption(final String token){
return false;}
private boolean isLongOption(final String token){
return true;}
POSS	17	- 46 Caroline Landry, INRIABenoit Baudry, KTH
...
if (condition && isLongOption(value)){
action1(value);}
else{
action2(value);}
...
private boolean isLongOption(final String token){
return false;}
private boolean isLongOption(final String token){
return true;}
POSS	17	- 47 Caroline Landry, INRIABenoit Baudry, KTH
...
if (condition && isLongOption(value)){
action1(value);}
else{
action2(value);}
...
private boolean isLongOption(final String token){
return false;}
private boolean isLongOption(final String token){
return true;}
Testability	issue
POSS	17	- 48 Caroline Landry, INRIABenoit Baudry, KTH
15
285
4 15 54 28 64 132 23
115
14
594 369
92 9 40
158
458 116
65
2592
69
2
14 28 19 27 55 10
17
12
97
85
31 2
14
11
61 20
69
197
274
1516
175 408 1178 610 1840 4245 459
344
585
2931 2237
2596 252 714
456 1726 425
2004
1618
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Tested
Weak pseudo-tested
Strong pseudo-tested
POSS	17	- 49 Caroline Landry, INRIABenoit Baudry, KTH
Descartes	– future	
•Incremental	Descartes	in	the	CI
•Github	hook
•Support	multi	module	Maven	projects
•Enhanced	diagnosis
•Integrate	into	a	test	augmentation	pipeline
POSS	17	- 50 Caroline Landry, INRIABenoit Baudry, KTH
Conclusion
•Mutation	analysis
• Automatic	generation	of	mutants
• Evaluate	the	test	suite
•Bugs	in	test	suites
• Oracle
• Input	space	coverage
• Testability	
• Indirectly	tested	code
Feedback	welcome!
• https://github.com/STAMP-project/pitest-descartes
• https://github.com/hcoles/pitest
• http://stamp-project.eu/
baudry@kth.se
caroline.landry@inria.fr

More Related Content

What's hot

Threshold and Proactive Pseudo-Random Permutations
Threshold and Proactive Pseudo-Random PermutationsThreshold and Proactive Pseudo-Random Permutations
Threshold and Proactive Pseudo-Random PermutationsAleksandr Yampolskiy
 
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...Alex Pruden
 
Class 17: Golden Sneezewort
Class 17: Golden SneezewortClass 17: Golden Sneezewort
Class 17: Golden SneezewortDavid Evans
 
Make Testing Groovy
Make Testing GroovyMake Testing Groovy
Make Testing GroovyPaul King
 
Automatically Describing Program Structure and Behavior (PhD Defense)
Automatically Describing Program Structure and Behavior (PhD Defense)Automatically Describing Program Structure and Behavior (PhD Defense)
Automatically Describing Program Structure and Behavior (PhD Defense)Ray Buse
 
JDK1.7 features
JDK1.7 featuresJDK1.7 features
JDK1.7 featuresindia_mani
 
Story of static code analyzer development
Story of static code analyzer developmentStory of static code analyzer development
Story of static code analyzer developmentAndrey Karpov
 
Practical Introduction to Java Memory Model
Practical Introduction to Java Memory ModelPractical Introduction to Java Memory Model
Practical Introduction to Java Memory ModelDmitry Degrave
 
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy Approach
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy ApproachMulti-Index Bi-Criterion Transportation Problem: A Fuzzy Approach
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy ApproachIJAEMSJORNAL
 
CBSE Question Paper Computer Science with C++ 2011
CBSE Question Paper Computer Science with C++ 2011CBSE Question Paper Computer Science with C++ 2011
CBSE Question Paper Computer Science with C++ 2011Deepak Singh
 
ZK Study Club: Sumcheck Arguments and Their Applications
ZK Study Club: Sumcheck Arguments and Their ApplicationsZK Study Club: Sumcheck Arguments and Their Applications
ZK Study Club: Sumcheck Arguments and Their ApplicationsAlex Pruden
 
Accelerating Habanero-Java Program with OpenCL Generation
Accelerating Habanero-Java Program with OpenCL GenerationAccelerating Habanero-Java Program with OpenCL Generation
Accelerating Habanero-Java Program with OpenCL GenerationAkihiro Hayashi
 
Introduction to ParSeq: to make asynchronous java easier
Introduction to ParSeq: to make asynchronous java easierIntroduction to ParSeq: to make asynchronous java easier
Introduction to ParSeq: to make asynchronous java easierJunchuan Wang
 
LeetCode Solutions In Java .pdf
LeetCode Solutions In Java .pdfLeetCode Solutions In Java .pdf
LeetCode Solutions In Java .pdfzupsezekno
 
A Configurable CEGAR Framework with Interpolation-Based Refinements
A Configurable CEGAR Framework with Interpolation-Based RefinementsA Configurable CEGAR Framework with Interpolation-Based Refinements
A Configurable CEGAR Framework with Interpolation-Based RefinementsAkos Hajdu
 
Dependency Injection in Episerver and .Net
Dependency Injection in Episerver and .NetDependency Injection in Episerver and .Net
Dependency Injection in Episerver and .NetValdis Iljuconoks
 
A survey on Fully Homomorphic Encryption
A survey on Fully Homomorphic EncryptionA survey on Fully Homomorphic Encryption
A survey on Fully Homomorphic Encryptioniosrjce
 
zkStudy Club: Subquadratic SNARGs in the Random Oracle Model
zkStudy Club: Subquadratic SNARGs in the Random Oracle ModelzkStudy Club: Subquadratic SNARGs in the Random Oracle Model
zkStudy Club: Subquadratic SNARGs in the Random Oracle ModelAlex Pruden
 

What's hot (20)

Threshold and Proactive Pseudo-Random Permutations
Threshold and Proactive Pseudo-Random PermutationsThreshold and Proactive Pseudo-Random Permutations
Threshold and Proactive Pseudo-Random Permutations
 
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...
zkStudyClub: PLONKUP & Reinforced Concrete [Luke Pearson, Joshua Fitzgerald, ...
 
Class 17: Golden Sneezewort
Class 17: Golden SneezewortClass 17: Golden Sneezewort
Class 17: Golden Sneezewort
 
Make Testing Groovy
Make Testing GroovyMake Testing Groovy
Make Testing Groovy
 
Algorithm
AlgorithmAlgorithm
Algorithm
 
Automatically Describing Program Structure and Behavior (PhD Defense)
Automatically Describing Program Structure and Behavior (PhD Defense)Automatically Describing Program Structure and Behavior (PhD Defense)
Automatically Describing Program Structure and Behavior (PhD Defense)
 
JDK1.7 features
JDK1.7 featuresJDK1.7 features
JDK1.7 features
 
Story of static code analyzer development
Story of static code analyzer developmentStory of static code analyzer development
Story of static code analyzer development
 
Practical Introduction to Java Memory Model
Practical Introduction to Java Memory ModelPractical Introduction to Java Memory Model
Practical Introduction to Java Memory Model
 
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy Approach
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy ApproachMulti-Index Bi-Criterion Transportation Problem: A Fuzzy Approach
Multi-Index Bi-Criterion Transportation Problem: A Fuzzy Approach
 
CBSE Question Paper Computer Science with C++ 2011
CBSE Question Paper Computer Science with C++ 2011CBSE Question Paper Computer Science with C++ 2011
CBSE Question Paper Computer Science with C++ 2011
 
ZK Study Club: Sumcheck Arguments and Their Applications
ZK Study Club: Sumcheck Arguments and Their ApplicationsZK Study Club: Sumcheck Arguments and Their Applications
ZK Study Club: Sumcheck Arguments and Their Applications
 
Accelerating Habanero-Java Program with OpenCL Generation
Accelerating Habanero-Java Program with OpenCL GenerationAccelerating Habanero-Java Program with OpenCL Generation
Accelerating Habanero-Java Program with OpenCL Generation
 
Introduction to ParSeq: to make asynchronous java easier
Introduction to ParSeq: to make asynchronous java easierIntroduction to ParSeq: to make asynchronous java easier
Introduction to ParSeq: to make asynchronous java easier
 
LeetCode Solutions In Java .pdf
LeetCode Solutions In Java .pdfLeetCode Solutions In Java .pdf
LeetCode Solutions In Java .pdf
 
Grails Transactions
Grails TransactionsGrails Transactions
Grails Transactions
 
A Configurable CEGAR Framework with Interpolation-Based Refinements
A Configurable CEGAR Framework with Interpolation-Based RefinementsA Configurable CEGAR Framework with Interpolation-Based Refinements
A Configurable CEGAR Framework with Interpolation-Based Refinements
 
Dependency Injection in Episerver and .Net
Dependency Injection in Episerver and .NetDependency Injection in Episerver and .Net
Dependency Injection in Episerver and .Net
 
A survey on Fully Homomorphic Encryption
A survey on Fully Homomorphic EncryptionA survey on Fully Homomorphic Encryption
A survey on Fully Homomorphic Encryption
 
zkStudy Club: Subquadratic SNARGs in the Random Oracle Model
zkStudy Club: Subquadratic SNARGs in the Random Oracle ModelzkStudy Club: Subquadratic SNARGs in the Random Oracle Model
zkStudy Club: Subquadratic SNARGs in the Random Oracle Model
 

Similar to Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

The CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsThe CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsCaroline Landry
 
STAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Project
 
Pooja Sharma , BCA Third Year
Pooja Sharma , BCA Third YearPooja Sharma , BCA Third Year
Pooja Sharma , BCA Third YearDezyneecole
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveSanderSlideShare
 
Hardik Jadam , BCA Third Year
Hardik Jadam , BCA Third YearHardik Jadam , BCA Third Year
Hardik Jadam , BCA Third YearDezyneecole
 
Vinita Vaishnav , BCA Third Year
Vinita Vaishnav , BCA Third YearVinita Vaishnav , BCA Third Year
Vinita Vaishnav , BCA Third YearDezyneecole
 
8. Recursion.pptx
8. Recursion.pptx8. Recursion.pptx
8. Recursion.pptxAbid523408
 
Shivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearShivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearDezyneecole
 
A scrupulous code review - 15 bugs in C++ code
A scrupulous code review - 15 bugs in C++ codeA scrupulous code review - 15 bugs in C++ code
A scrupulous code review - 15 bugs in C++ codePVS-Studio LLC
 
Modularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart ContractsModularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart ContractsFacultad de Informática UCM
 
Ronak Kachhawa , BCA Third Year
Ronak Kachhawa , BCA Third YearRonak Kachhawa , BCA Third Year
Ronak Kachhawa , BCA Third YearDezyneecole
 
C++ Code as Seen by a Hypercritical Reviewer
C++ Code as Seen by a Hypercritical ReviewerC++ Code as Seen by a Hypercritical Reviewer
C++ Code as Seen by a Hypercritical ReviewerAndrey Karpov
 
Shivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearShivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearDezyneecole
 
Pointless Pointers - How to make our interfaces efficient?
Pointless Pointers - How to make our interfaces efficient?Pointless Pointers - How to make our interfaces efficient?
Pointless Pointers - How to make our interfaces efficient?Mateusz Pusz
 
Hypercritical C++ Code Review
Hypercritical C++ Code ReviewHypercritical C++ Code Review
Hypercritical C++ Code ReviewAndrey Karpov
 
4.3 real time game physics
4.3 real time game physics4.3 real time game physics
4.3 real time game physicsSayed Ahmed
 
Mathematical Reasoning in Discrete Mathmatics.pptx
Mathematical Reasoning in Discrete Mathmatics.pptxMathematical Reasoning in Discrete Mathmatics.pptx
Mathematical Reasoning in Discrete Mathmatics.pptxgbikorno
 

Similar to Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017 (20)

The CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsThe CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestions
 
Stamp breizhcamp 2019
Stamp breizhcamp 2019Stamp breizhcamp 2019
Stamp breizhcamp 2019
 
STAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Descartes Presentation
STAMP Descartes Presentation
 
Pooja Sharma , BCA Third Year
Pooja Sharma , BCA Third YearPooja Sharma , BCA Third Year
Pooja Sharma , BCA Third Year
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra derive
 
Hardik Jadam , BCA Third Year
Hardik Jadam , BCA Third YearHardik Jadam , BCA Third Year
Hardik Jadam , BCA Third Year
 
Vinita Vaishnav , BCA Third Year
Vinita Vaishnav , BCA Third YearVinita Vaishnav , BCA Third Year
Vinita Vaishnav , BCA Third Year
 
8. Recursion.pptx
8. Recursion.pptx8. Recursion.pptx
8. Recursion.pptx
 
hw-sol.pdf
hw-sol.pdfhw-sol.pdf
hw-sol.pdf
 
Shivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearShivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third Year
 
A scrupulous code review - 15 bugs in C++ code
A scrupulous code review - 15 bugs in C++ codeA scrupulous code review - 15 bugs in C++ code
A scrupulous code review - 15 bugs in C++ code
 
Modularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart ContractsModularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart Contracts
 
Ronak Kachhawa , BCA Third Year
Ronak Kachhawa , BCA Third YearRonak Kachhawa , BCA Third Year
Ronak Kachhawa , BCA Third Year
 
C++ Code as Seen by a Hypercritical Reviewer
C++ Code as Seen by a Hypercritical ReviewerC++ Code as Seen by a Hypercritical Reviewer
C++ Code as Seen by a Hypercritical Reviewer
 
Shivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third YearShivani Chouhan , BCA Third Year
Shivani Chouhan , BCA Third Year
 
Pointless Pointers - How to make our interfaces efficient?
Pointless Pointers - How to make our interfaces efficient?Pointless Pointers - How to make our interfaces efficient?
Pointless Pointers - How to make our interfaces efficient?
 
Hypercritical C++ Code Review
Hypercritical C++ Code ReviewHypercritical C++ Code Review
Hypercritical C++ Code Review
 
4.3 real time game physics
4.3 real time game physics4.3 real time game physics
4.3 real time game physics
 
CPP Quiz
CPP QuizCPP Quiz
CPP Quiz
 
Mathematical Reasoning in Discrete Mathmatics.pptx
Mathematical Reasoning in Discrete Mathmatics.pptxMathematical Reasoning in Discrete Mathmatics.pptx
Mathematical Reasoning in Discrete Mathmatics.pptx
 

More from STAMP Project

Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenSTAMP Project
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP Project
 
20181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday201820181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday2018STAMP Project
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018STAMP Project
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17STAMP Project
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP Project
 

More from STAMP Project (7)

Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, Sweden
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
 
20181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday201820181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday2018
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017
 

Recently uploaded

Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 

Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

  • 1. Mutate and Test Your Tests Author: Benoit Baudry Project Coordinator and Scientific Leader KTH, Sweden 1 baudry@kth.se caroline.landry@inria.fr Speaker: Caroline Landry Project Technical Manager INRIA, France
  • 2. POSS 17 - 2 Caroline Landry, INRIABenoit Baudry, KTH Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs
  • 3. POSS 17 - 3 Caroline Landry, INRIABenoit Baudry, KTH Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs
  • 4. POSS 17 - 4 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 5. POSS 17 - 5 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 6. POSS 17 - 6 Caroline Landry, INRIABenoit Baudry, KTH Coverage long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 7. POSS 17 - 7 Caroline Landry, INRIABenoit Baudry, KTH Coverage @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  • 8. POSS 17 - 8 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs?
  • 9. POSS 17 - 9 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see.
  • 10. POSS 17 - 10 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests
  • 11. POSS 17 - 11 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests •Mutation operators •Mutant = Program with one mutation
  • 12. POSS 17 - 12 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis Inputs : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M)
  • 13. POSS 17 - 13 Caroline Landry, INRIABenoit Baudry, KTH PIT •Open source •Actively developed and maintained •State of the art •For Java •Integrates with major build systems
  • 14. POSS 17 - 14 Caroline Landry, INRIABenoit Baudry, KTH PIT mutation operators •Conditions •Constants •Return •Delete method calls •Constructor calls
  • 15. POSS 17 - 15 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  • 16. POSS 17 - 16 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0
  • 17. POSS 17 - 17 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1
  • 18. POSS 17 - 18 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 <
  • 19. POSS 17 - 19 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 20. POSS 17 - 20 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 21. POSS 17 - 21 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 22. POSS 17 - 22 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith0Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 23. POSS 17 - 23 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 24. POSS 17 - 24 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 25. POSS 17 - 25 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 26. POSS 17 - 26 Caroline Landry, INRIABenoit Baudry, KTH PIT – reporting
  • 27. POSS 17 - 27 Caroline Landry, INRIABenoit Baudry, KTH PIT – reporting
  • 28. POSS 17 - 28 Caroline Landry, INRIABenoit Baudry, KTH Project #LOC #Mutants Time (h) Score (%) Amazon Web Services SDK 1703892 2141690 04:25:35 76.28 XWiki Rendering Engine 104727 112609 01:59:55 50.89 Apache Commons Math 180951 104786 03:22:18 83.81 JFreeChart 134353 89592 00:41:28 58.04 Apache PdfBox 137099 79763 06:20:25 58.89 Java Git 128874 78316 16:02:00 73.86 SCIFIO 55347 62768 03:12:11 45.92 Joda-Time 85911 31233 00:16:32 81.65 Apache Commons Lang 60733 30361 00:21:02 86.17 Apache Commons Collections 51632 20394 00:05:41 85.94 Urban Airship Client Library 40885 17345 00:11:31 82.26 SAT4J 26415 17067 11:11:04 68.58 ImageJ Common 23868 15592 00:29:09 54.77 jsoup 17839 14054 00:12:49 78.34 Jaxen XPath Engine 21204 12210 00:24:40 67.13
  • 29. POSS 17 - 29 Caroline Landry, INRIABenoit Baudry, KTH Descartes •Mutation operators: extreme mutation •Active development •Open source •Compared to PIT • Less mutants • Different type of feedback • Same framework
  • 30. POSS 17 - 30 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example
  • 31. POSS 17 - 31 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example long fact(int n){ return 0;} long fact(int n){ return 1;} PIT : 7 mutants Descartes : 2 mutants
  • 32. POSS 17 - 32 Caroline Landry, INRIABenoit Baudry, KTH Scalability of extreme mutation public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants
  • 33. POSS 17 - 33 Caroline Landry, INRIABenoit Baudry, KTH Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45
  • 34. POSS 17 - 34 Caroline Landry, INRIABenoit Baudry, KTH Coarser grain than PIT long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));}
  • 35. POSS 17 - 35 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}
  • 36. POSS 17 - 36 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 37. POSS 17 - 37 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 38. POSS 17 - 38 Caroline Landry, INRIABenoit Baudry, KTH Wrong test bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 39. POSS 17 - 39 Caroline Landry, INRIABenoit Baudry, KTH protected final void prepareSocket(final SSLSocket socket) { }}
  • 40. POSS 17 - 40 Caroline Landry, INRIABenoit Baudry, KTH @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  • 41. POSS 17 - 41 Caroline Landry, INRIABenoit Baudry, KTH @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  • 42. POSS 17 - 42 Caroline Landry, INRIABenoit Baudry, KTH Missing oracle @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  • 43. POSS 17 - 43 Caroline Landry, INRIABenoit Baudry, KTH private boolean isLongOption(final String token){ if (...){return false;} //Covered 27 times ... if (...){return true;} //Not covered else if (...){return true;} //Covered once return false; //Covered 7 times }
  • 44. POSS 17 - 44 Caroline Landry, INRIABenoit Baudry, KTH private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  • 45. POSS 17 - 45 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  • 46. POSS 17 - 46 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  • 47. POSS 17 - 47 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;} Testability issue
  • 48. POSS 17 - 48 Caroline Landry, INRIABenoit Baudry, KTH 15 285 4 15 54 28 64 132 23 115 14 594 369 92 9 40 158 458 116 65 2592 69 2 14 28 19 27 55 10 17 12 97 85 31 2 14 11 61 20 69 197 274 1516 175 408 1178 610 1840 4245 459 344 585 2931 2237 2596 252 714 456 1726 425 2004 1618 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Tested Weak pseudo-tested Strong pseudo-tested
  • 49. POSS 17 - 49 Caroline Landry, INRIABenoit Baudry, KTH Descartes – future •Incremental Descartes in the CI •Github hook •Support multi module Maven projects •Enhanced diagnosis •Integrate into a test augmentation pipeline
  • 50. POSS 17 - 50 Caroline Landry, INRIABenoit Baudry, KTH Conclusion •Mutation analysis • Automatic generation of mutants • Evaluate the test suite •Bugs in test suites • Oracle • Input space coverage • Testability • Indirectly tested code