SlideShare a Scribd company logo
1
22-Mar-2019
Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
Caroline Landry
Project Technical Manager
INRIA, France
The CI as a partner for test improvement suggestions
2
Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
Caroline Landry
Project Technical Manager
INRIA, France
22-Mar-2019
This work was partially supported by the EU Project
STAMP ICT-16-10 No.731529
•4 research institutions
•5 companies
•1 open source
consortium
•Automated Testing in
DevOps
DevOps
4
DevOps
5
• Better quality
DevOps
6
• Better quality
• Shorter release cycles
DevOps
7
• Better quality
• Shorter release cycles
• Reconnect Ops to Dev
DevOps
8
Automation• Better quality
• Shorter release cycles
• Reconnect Ops to Dev
DevOps
9
Step acceptance & Feedback
Automation• Better quality
• Shorter release cycles
• Reconnect Ops to Dev
DevOps
10
unit
perf.
fuzzing
loggingdep. inj.
UI
CI
pertur-
bation
fault
recov.
IDEs
libraries
container
IDS
VMs
cluster
config.
DevOps
11
unit
perf.
fuzzing
logging
UI
pertur-
bation
fault
recov.
IDS
cluster
config.
IDEs
Continuous testing
DevOps – Testing in Dev
12
Continuous Integration
- Unit, integration testing
- Coverage, mutation
Automatic test improvement
- Test amplification
- Test fixing
IDE
- Linters
- Completion
DevOps – Testing in Ops
13
Crash analysis
- Reproduction
- Localization
Online experiments
- Chaos
- A/B testing
14
STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
https://dictionary.cambridge.org/dictionary/english/amplify
15
STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test assets
16
https://dictionary.cambridge.org/dictionary/english/amplify
STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test assets
 Test assets: test cases, configuration files, production logs
 Effect metrics: mutation score, feature interactions
 Automatic amplification
17
https://dictionary.cambridge.org/dictionary/english/amplify
DevOps – STAMP focus
18
DevOps – STAMP focus
19
DevOps – STAMP focus
20
DevOps – STAMP focus
21
Unit test amplification
22
Descartes &
DSpot
Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
23
Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
24
Example
25
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
26
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Coverage
Example
27
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Coverage
Example
28
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
Coverage
Example
29
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
Coverage
Is these test suite good at
detecting bugs?
Example
30
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
Coverage
Is these test suite good at
detecting bugs?
Let’s mutate our code to see.
Mutation Analysis
31
Mutation Analysis
32
inject fault
Mutant 1
Mutation Analysis
33
inject fault
Mutant 1
Run
Mutation Analysis
34
inject fault
Mutant 1
Run ✗ Mutant 1
Mutation Analysis
35
inject fault
Mutant 1
Run ✗ Mutant 1
Killed

Mutation Analysis
36
inject fault
Mutant 1
Mutant 2
Run ✗ Mutant 1
Killed

Mutation Analysis
37
inject fault
Mutant 1
Mutant 2
Run ✗ Mutant 1
Killed

Mutation Analysis
38
inject fault
Mutant 1
Mutant 2
Run ✗
✗
Mutant 1
Mutant 2
Killed

Mutation Analysis
39
inject fault
Mutant 1
Mutant 2
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed

Mutation Analysis
40
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed

Mutation Analysis
41
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed

Mutation Analysis
42
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
Mutant 1
Mutant 2
Mutant 3
Killed

Killed

Mutation Analysis
43
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
Mutant 1
Mutant 2
Mutant 3
Killed

Alive

Killed

Mutation Analysis
44
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
 Mutation score = 66%
Mutant 1
Mutant 2
Mutant 3
Killed

Alive

Killed

Example
45
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
long fact(int n) {
if (n != 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
46
✗
✓
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
47
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
Example
48
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Example
49
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
Example
50
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
51
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
 Mutation score = 71%
Test suite:
• Weak oracle
• Missing input
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
52
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
@Test
factorialWith5Test() {
assertEqual(120, fact(5));
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith0Test() {
assertEqual(1, fact(0));
}
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Mutation Analysis
•Tests are good if they can detect bugs
•Mutation operators
• Based on common faults
•PIT or PITest
• Open source, in active development and production ready
• Integrates with major build systems
• State of the art mutation testing
• Extensible via plugins
• Concurrent execution
• Test selection
53
Limitations of mutation testing
•Expensive computation
•Huge number of mutants
•Presence of equivalent mutants
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
Limitations of mutation testing
•Expensive computation
•Huge number of mutants
•Presence of equivalent mutants
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
 Extreme Mutation
Extreme mutation
•Proposed in 2016 by Niedermayr et al.
•Remove the body of the method
•Replace by a single return
•Less mutants
•Most equivalent mutants can be detected
56
R. Niedermayr, E. Juergens, and S. Wagner, “Will my tests tell me if I break this code?,” in Proceedings of the International Workshop on Continuous
Software Evolution and Delivery, 2016, pp. 23–29.
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
57
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
58
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
long fact(int n) {
return 0;
}
long fact(int n) {
return 1;
}
Example
59
Descartes I mutate therefore I am
• A mutation engine for PIT
• Implements extreme mutation
• Finds pseudo-tested methods
• Identify weaknesses in your tests
• Compute mutation score
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
Pseudo-tested methods
•A method executed by the test suite
•No extreme mutation is detected
•Found in well tested projects
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
62Apache Commons Collections
A pseudo-tested method
63Apache Commons Collections
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
64Apache Commons Collections
Pseudo-tested
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
65Apache Commons Collections
No exception is thrown
A fail is needed here
Pseudo-tested
class SingletonListIteratorTest {
...
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
...
}
Descartes results
66
Descartes reports
67
Descartes reports
68
Descartes reports
69
Descartes reports
70
Descartes reports
71
How do you correct the
weaknesses in your tests ?
Let’s amplify existing
test cases
DSpot
•Automatically enhances existing JUnit test suites
•Generate new assertions or new test cases
72
Dspot principle
73
x
Dspot principle
74
Test criterion
x
Dspot principle
75
Test criterion
(Mutation score)
x
Dspot principle
76
Test criterion
(Mutation score)
x
Dspot principle
77
Test criterion
(Mutation score)
x
Dspot principle
78
Test criterion
(Mutation score)
Dspot – How it works ?
79
Mutation
Analysis
✗
✓
✗
✗
✓
Dspot – How it works ?
80
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
amplify
Dspot – How it works ?
81
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify
Dspot – How it works ?
82
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify
Dspot – How it works ?
83
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify
Dspot – How it works ?
84
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify
Dspot – How it works ?
85
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify
Dspot – How it works ?
86
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
✗
✗
✗
✗
✗
amplify
87
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case
88
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case
Remove a method call
89
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
assertFalse(benjamin.isHungry());
}
Example
Original
test case
Remove existing assertions
Remove a method call
90
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
91
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test
benjamin.isHungry()true
benjamin.isHappy()false
92
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test
benjamin.isHungry()true
benjamin.isHappy()false
93
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test
assertTrue(benjamin.isHungry());
assertFalse(benjamin.isHappy());Generate assertions
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
assertTrue(benjamin.isHungry());
assertFalse(benjamin.isHappy());
} 94
Example
Original
test case
Amplified
test case
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
95
DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
The pull request loop
Collaborative
platform
96
The pull request loop
Collaborative
platformpull req.
97
The pull request loop
Collaborative
platformpull req.
code
98
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
99
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
100
The pull request loop
pull req.
code
analysesfeedback
101
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
102
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
103
Novel analyses
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
104
Novel analyses
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
105
Novel analyses
The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
106
Novel analyses
Integration
•Command line
•Eclipse plugin
•Maven plugin
•Gradle plugin
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
Integration
•Jenkins
• Plugin to monitor score and pseudo-tested methods
• Xwiki
• Same strategy used with the code coverage
• Threshold on the mutation score
• https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescar
tes
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
Integration
•In the CI as a service
• GitHub Application
• Find pseudo-tested methods in pull-requests
• Feedback using GitHub Check Runs API
Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 -
Montpellier
More
•Open Source tools
• https://github.com/STAMP-project
•Public website
• https://www.stamp-project.eu
•Medium on Descartes
• https://medium.com/@almyre/short-circuiting-method-
executions-to-assess-test-quality-2d3fda45bc7f
•Publications
• https://www.stamp-project.eu/view/main/publications
110
111

More Related Content

Similar to Stamp breizhcamp 2019

Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your Tests
STAMP Project
 
Mutation testing
Mutation testingMutation testing
Mutation testing
기영 이
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
STAMP Project
 
Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016
Gerald Muecke
 
Modularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart ContractsModularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart Contracts
Facultad de Informática UCM
 
Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017
Gerald Muecke
 
DSR Testing (Part 1)
DSR Testing (Part 1)DSR Testing (Part 1)
DSR Testing (Part 1)
Steve Upton
 
Mutation Testing: Leaving the Stone Age. FOSDEM 2017
Mutation Testing: Leaving the Stone Age. FOSDEM 2017Mutation Testing: Leaving the Stone Age. FOSDEM 2017
Mutation Testing: Leaving the Stone Age. FOSDEM 2017
Alex Denisov
 
Mutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The Bugs
Ari Waller
 
Mateusz Bryła - Mutation testing
Mateusz Bryła - Mutation testingMateusz Bryła - Mutation testing
Mateusz Bryła - Mutation testing
kraqa
 
NIO-ICSE2022.pptx
NIO-ICSE2022.pptxNIO-ICSE2022.pptx
NIO-ICSE2022.pptx
DavidWei89
 
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Agile Lietuva
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra derive
SanderSlideShare
 
GW SDAB Dev Tools 2012
GW SDAB Dev Tools 2012GW SDAB Dev Tools 2012
GW SDAB Dev Tools 2012
Hervé Vũ Roussel
 
Bayesian Experimental Design for Stochastic Kinetic Models
Bayesian Experimental Design for Stochastic Kinetic ModelsBayesian Experimental Design for Stochastic Kinetic Models
Bayesian Experimental Design for Stochastic Kinetic Models
Colin Gillespie
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
Lionel Briand
 
White-box Unit Test Generation with Microsoft IntelliTest
White-box Unit Test Generation with Microsoft IntelliTestWhite-box Unit Test Generation with Microsoft IntelliTest
White-box Unit Test Generation with Microsoft IntelliTest
Dávid Honfi
 
Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*
Intel® Software
 
"Quantum" performance effects
"Quantum" performance effects"Quantum" performance effects
"Quantum" performance effects
Sergey Kuksenko
 
05 junit
05 junit05 junit
05 junit
mha4
 

Similar to Stamp breizhcamp 2019 (20)

Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your Tests
 
Mutation testing
Mutation testingMutation testing
Mutation testing
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
 
Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016
 
Modularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart ContractsModularity for Accurate Static Analysis of Smart Contracts
Modularity for Accurate Static Analysis of Smart Contracts
 
Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017
 
DSR Testing (Part 1)
DSR Testing (Part 1)DSR Testing (Part 1)
DSR Testing (Part 1)
 
Mutation Testing: Leaving the Stone Age. FOSDEM 2017
Mutation Testing: Leaving the Stone Age. FOSDEM 2017Mutation Testing: Leaving the Stone Age. FOSDEM 2017
Mutation Testing: Leaving the Stone Age. FOSDEM 2017
 
Mutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The Bugs
 
Mateusz Bryła - Mutation testing
Mateusz Bryła - Mutation testingMateusz Bryła - Mutation testing
Mateusz Bryła - Mutation testing
 
NIO-ICSE2022.pptx
NIO-ICSE2022.pptxNIO-ICSE2022.pptx
NIO-ICSE2022.pptx
 
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
Vaidas Pilkauskas and Tadas Ščerbinskas - Can you trust your tests?
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra derive
 
GW SDAB Dev Tools 2012
GW SDAB Dev Tools 2012GW SDAB Dev Tools 2012
GW SDAB Dev Tools 2012
 
Bayesian Experimental Design for Stochastic Kinetic Models
Bayesian Experimental Design for Stochastic Kinetic ModelsBayesian Experimental Design for Stochastic Kinetic Models
Bayesian Experimental Design for Stochastic Kinetic Models
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
 
White-box Unit Test Generation with Microsoft IntelliTest
White-box Unit Test Generation with Microsoft IntelliTestWhite-box Unit Test Generation with Microsoft IntelliTest
White-box Unit Test Generation with Microsoft IntelliTest
 
Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*
 
"Quantum" performance effects
"Quantum" performance effects"Quantum" performance effects
"Quantum" performance effects
 
05 junit
05 junit05 junit
05 junit
 

Recently uploaded

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLESINTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
anfaltahir1010
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
The Third Creative Media
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Project Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdfProject Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdf
Karya Keeper
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
Maitrey Patel
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
aisafed42
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
Alberto Brandolini
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 

Recently uploaded (20)

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLESINTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
INTRODUCTION TO AI CLASSICAL THEORY TARGETED EXAMPLES
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Project Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdfProject Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdf
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabhQuarter 3 SLRP grade 9.. gshajsbhhaheabh
Quarter 3 SLRP grade 9.. gshajsbhhaheabh
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 

Stamp breizhcamp 2019

  • 1. 1 22-Mar-2019 Benoit Baudry Project Coordinator and Scientific Leader KTH, Sweden Caroline Landry Project Technical Manager INRIA, France
  • 2. The CI as a partner for test improvement suggestions 2 Benoit Baudry Project Coordinator and Scientific Leader KTH, Sweden Caroline Landry Project Technical Manager INRIA, France 22-Mar-2019
  • 3. This work was partially supported by the EU Project STAMP ICT-16-10 No.731529 •4 research institutions •5 companies •1 open source consortium •Automated Testing in DevOps
  • 6. DevOps 6 • Better quality • Shorter release cycles
  • 7. DevOps 7 • Better quality • Shorter release cycles • Reconnect Ops to Dev
  • 8. DevOps 8 Automation• Better quality • Shorter release cycles • Reconnect Ops to Dev
  • 9. DevOps 9 Step acceptance & Feedback Automation• Better quality • Shorter release cycles • Reconnect Ops to Dev
  • 12. DevOps – Testing in Dev 12 Continuous Integration - Unit, integration testing - Coverage, mutation Automatic test improvement - Test amplification - Test fixing IDE - Linters - Completion
  • 13. DevOps – Testing in Ops 13 Crash analysis - Reproduction - Localization Online experiments - Chaos - A/B testing
  • 14. 14
  • 15. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something https://dictionary.cambridge.org/dictionary/english/amplify 15
  • 16. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets 16 https://dictionary.cambridge.org/dictionary/english/amplify
  • 17. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets  Test assets: test cases, configuration files, production logs  Effect metrics: mutation score, feature interactions  Automatic amplification 17 https://dictionary.cambridge.org/dictionary/english/amplify
  • 18. DevOps – STAMP focus 18
  • 19. DevOps – STAMP focus 19
  • 20. DevOps – STAMP focus 20
  • 21. DevOps – STAMP focus 21
  • 23. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 23
  • 24. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 24
  • 25. Example 25 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; }
  • 26. Example 26 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Coverage
  • 27. Example 27 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } Coverage
  • 28. Example 28 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); } Coverage
  • 29. Example 29 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); } Coverage Is these test suite good at detecting bugs?
  • 30. Example 30 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); } Coverage Is these test suite good at detecting bugs? Let’s mutate our code to see.
  • 35. Mutation Analysis 35 inject fault Mutant 1 Run ✗ Mutant 1 Killed 
  • 36. Mutation Analysis 36 inject fault Mutant 1 Mutant 2 Run ✗ Mutant 1 Killed 
  • 37. Mutation Analysis 37 inject fault Mutant 1 Mutant 2 Run ✗ Mutant 1 Killed 
  • 38. Mutation Analysis 38 inject fault Mutant 1 Mutant 2 Run ✗ ✗ Mutant 1 Mutant 2 Killed 
  • 39. Mutation Analysis 39 inject fault Mutant 1 Mutant 2 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  • 40. Mutation Analysis 40 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  • 41. Mutation Analysis 41 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✗ Mutant 1 Mutant 2 Killed  Killed 
  • 42. Mutation Analysis 42 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗ Mutant 1 Mutant 2 Mutant 3 Killed  Killed 
  • 43. Mutation Analysis 43 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗ Mutant 1 Mutant 2 Mutant 3 Killed  Alive  Killed 
  • 44. Mutation Analysis 44 inject fault Mutant 1 Mutant 2 Mutant 3 Run ✗ ✓ ✗  Mutation score = 66% Mutant 1 Mutant 2 Mutant 3 Killed  Alive  Killed 
  • 45. Example 45 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); }
  • 46. long fact(int n) { if (n != 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 46 ✗ ✓ @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); }
  • 47. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 47 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  • 48. Example 48 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 49. Example 49 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } @Test factorialWith0Test() { assertEqual(1, fact(0)); }
  • 50. Example 50 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); } long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; }
  • 51. Example 51 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); }  Mutation score = 71% Test suite: • Weak oracle • Missing input long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; }
  • 52. Example 52 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1 @Test factorialWith5Test() { assertEqual(120, fact(5)); } @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0)); } long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; }
  • 53. Mutation Analysis •Tests are good if they can detect bugs •Mutation operators • Based on common faults •PIT or PITest • Open source, in active development and production ready • Integrates with major build systems • State of the art mutation testing • Extensible via plugins • Concurrent execution • Test selection 53
  • 54. Limitations of mutation testing •Expensive computation •Huge number of mutants •Presence of equivalent mutants Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 55. Limitations of mutation testing •Expensive computation •Huge number of mutants •Presence of equivalent mutants Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier  Extreme Mutation
  • 56. Extreme mutation •Proposed in 2016 by Niedermayr et al. •Remove the body of the method •Replace by a single return •Less mutants •Most equivalent mutants can be detected 56 R. Niedermayr, E. Juergens, and S. Wagner, “Will my tests tell me if I break this code?,” in Proceedings of the International Workshop on Continuous Software Evolution and Delivery, 2016, pp. 23–29.
  • 57. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 57 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  • 58. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } Example 58 n!=0 return 1+1 i < n !(i<=n) i-- result/i result+1
  • 59. long fact(int n) { if (n == 0) { return 1; } long result = 1; for (int i = 2; i <= n; i++) { result = result * i; } return result; } long fact(int n) { return 0; } long fact(int n) { return 1; } Example 59
  • 60. Descartes I mutate therefore I am • A mutation engine for PIT • Implements extreme mutation • Finds pseudo-tested methods • Identify weaknesses in your tests • Compute mutation score Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 61. Pseudo-tested methods •A method executed by the test suite •No extreme mutation is detected •Found in well tested projects Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 62. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 62Apache Commons Collections
  • 63. A pseudo-tested method 63Apache Commons Collections class SingletonListIteratorTest { ... @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} ... } class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... }
  • 64. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 64Apache Commons Collections Pseudo-tested class SingletonListIteratorTest { ... @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} ... }
  • 65. class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... } A pseudo-tested method 65Apache Commons Collections No exception is thrown A fail is needed here Pseudo-tested class SingletonListIteratorTest { ... @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} ... }
  • 71. Descartes reports 71 How do you correct the weaknesses in your tests ? Let’s amplify existing test cases
  • 72. DSpot •Automatically enhances existing JUnit test suites •Generate new assertions or new test cases 72
  • 79. Dspot – How it works ? 79 Mutation Analysis ✗ ✓ ✗ ✗ ✓
  • 80. Dspot – How it works ? 80 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ amplify
  • 81. Dspot – How it works ? 81 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  • 82. Dspot – How it works ? 82 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  • 83. Dspot – How it works ? 83 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  • 84. Dspot – How it works ? 84 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  • 85. Dspot – How it works ? 85 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ amplify
  • 86. Dspot – How it works ? 86 Amplified Test 1 Amplified Test 2 Amplified Test 3 Mutation Analysis ✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ amplify
  • 87. 87 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); } Example Original test case
  • 88. 88 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); } Example Original test case Remove a method call
  • 89. 89 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); assertFalse(benjamin.isHungry()); } Example Original test case Remove existing assertions Remove a method call
  • 90. 90 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test
  • 91. 91 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test
  • 92. benjamin.isHungry()true benjamin.isHappy()false 92 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test
  • 93. benjamin.isHungry()true benjamin.isHappy()false 93 @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.isHungry(); Log.log(benjamin, id : "benjamin"); } Example Original test case Instrument the test Run the instrumented test assertTrue(benjamin.isHungry()); assertFalse(benjamin.isHappy());Generate assertions
  • 94. @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); assertTrue(benjamin.isHungry()); assertFalse(benjamin.isHappy()); } 94 Example Original test case Amplified test case @Test public void html() { Food kouignAmann = new Food("KouignAmann"); PhD benjamin = new PhD("Benjamin"); benjamin.eat(kouignAmann); assertFalse(benjamin.isHungry()); }
  • 95. 95 DSpot Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
  • 96. The pull request loop Collaborative platform 96
  • 97. The pull request loop Collaborative platformpull req. 97
  • 98. The pull request loop Collaborative platformpull req. code 98
  • 99. The pull request loop Collaborative platformpull req. code analysesfeedback 99
  • 100. The pull request loop Collaborative platformpull req. code analysesfeedback 100
  • 101. The pull request loop pull req. code analysesfeedback 101
  • 102. The pull request loop Collaborative platformpull req. code analysesfeedback 102
  • 103. The pull request loop Collaborative platformpull req. code analysesfeedback 103 Novel analyses
  • 104. The pull request loop Collaborative platformpull req. code analysesfeedback 104 Novel analyses
  • 105. The pull request loop Collaborative platformpull req. code analysesfeedback 105 Novel analyses
  • 106. The pull request loop Collaborative platformpull req. code analysesfeedback 106 Novel analyses
  • 107. Integration •Command line •Eclipse plugin •Maven plugin •Gradle plugin Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 108. Integration •Jenkins • Plugin to monitor score and pseudo-tested methods • Xwiki • Same strategy used with the code coverage • Threshold on the mutation score • https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescar tes Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 109. Integration •In the CI as a service • GitHub Application • Find pseudo-tested methods in pull-requests • Feedback using GitHub Check Runs API Vera-Pérez, Baudry, Massol - ASE 2018 - 03/09/2018 - Montpellier
  • 110. More •Open Source tools • https://github.com/STAMP-project •Public website • https://www.stamp-project.eu •Medium on Descartes • https://medium.com/@almyre/short-circuiting-method- executions-to-assess-test-quality-2d3fda45bc7f •Publications • https://www.stamp-project.eu/view/main/publications 110
  • 111. 111