This document describes the STAMP project which aims to increase test automation in DevOps. It focuses on test amplification using techniques like mutation analysis and automatic test improvement. The project involves 4 research institutions, 5 companies and an open source consortium with 516 person months. It will develop tools like Descartes and DSpot which use extreme mutation and suggest improvements to existing test cases.

1. 1

2. Software Testing AMPlification
H2020 LEIT RIA - ICT-10-2016 – Software Technology
2016/12/01 – 2019/11/30
2

3. DevOps
3

4. DevOps – automatic development
unit
perf.
fuzzing
loggingdep. inj.
UI
CI
pertur-
bation
fault
recov.
IDEs
libraries
container
IDS
VMs
cluster
config.
4

5. DevOps – continuous testing
unit
perf.
fuzzing
logging
UI
pertur-
bation
fault
recov.
IDS
cluster
config.
5

6. DevOps – STAMP focus
unit
logging
config.
6

7. Human – bot collaboration
Collaboration
platform
pull req.
7

8. Human – bot collaboration
Collaboration
platform
pull req.
code
8

9. Human – bot collaboration
Collaboration
platform
pull req.
code
analyses
9

10. Human – bot collaboration
Collaboration
platform
pull req.
code
analyses
feedback
10

11. Human – bot collaboration
pull req.
code
analyses
feedback
11

12. Human – bot collaboration
Collaboration
platform
pull req.
code
analyses
feedback
12
STAMP tools

13. Human – bot collaboration
Collaboration
platform
pull req.
code
analyses
feedback
STAMP tools 13

14. DevOps – STAMP context
•Test automation
•Human – bot collaboration
14

15. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
https://dictionary.cambridge.org/dictionary/english/amplify
15

16. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test
assets
16
https://dictionary.cambridge.org/dictionary/english/amplify

17. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test
assets
 Test assets: test cases, configuration files, production logs
 Effect metrics: mutation score, feature interactions
 Automatic amplification
17
https://dictionary.cambridge.org/dictionary/english/amplify

18. Unit test amplification
•Premise: unit test cases exist
• Can run automatically
• Developpers capture essential behaviors and expectations
• They are not sufficient
•Intuition: unit test cases can be automatically amplified
• Automatic analysis and transformations
• Automatic assessment and selection
• In the continuous integration pipeline
18

19. 19
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}

20. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
20
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}

21. Coverage
21
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}

22. 22
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Are these test cases good at
detecting bugs?

23. 23
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.

24. Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
• Conditions
• Constants
• Return
• Delete method calls
• Constructor call
24

25. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
25

26. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
26
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

27. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
27
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

28. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
28
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

29. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
29
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

30. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation reveals bugs in the test suite
30
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

31. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
31
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

32. Descartes – extreme Java
mutation
32

33. Descartes
•Mutation operators: extreme mutation
•Active, open source development
•Pitest plugin
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework
33

34. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
34

35. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
35
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT : 7 mutants
Descartes : 2 mutants

36. Assess test cases effectiveness
36Apache Commons Collections
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new UnsupportedOperationException();
}
...
}

37. Assess test cases effectiveness
37Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
throw
new UnsupportedOperationException();
}
}

38. Assess test cases effectiveness
38Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
throw
new UnsupportedOperationException();
}
}
✔

39. Descartes
39Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
}
}
Remove the whole body of method

40. Descartes
40Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
}
}
✔

41. Descartes
41Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
}
}
✔Pseudo-tested
method

42. Descartes
42Apache Commons Collections
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
class SingletonListIterator
implements Iterator<Node> {
void add() {
}
}
✔Pseudo-tested
method
No exception is thrown
A fail is needed here

43. Descartes
43

44. <plugins>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.0</version>
<dependencies>
<dependency>
<groupId>eu.stamp-project</groupId>
<artifactId>descartes</artifactId>
<version>1.2</version>
</dependency>
</dependencies>
<configuration>
<verbose>true</verbose>
<mutationEngine>descartes</mutationEngine>
<mutators>
<mutator>true</mutator>
<mutator>”A string”</mutator>
<mutator>null</mutator>
<mutator>empty</mutator>
</mutators>
<outputFormats>
<value>ISSUES</value>
<value>METHODS</value>
<value>JSON</value>
</outputFormats>
</configuration>
</plugin>
</plugins>
PITest as a plugin
Descartes to be used by PITest
Regular PITest configuration
Descartes as the mutation engine
Values to use in extreme transforms. Supports literals for
primitive types, String, null and empty for empty arrays.
Reporting options. Custom reports ISSUES and METHODS for
testing issues and method categorization. JSON, HTML, XML
for regular PITest report.

45. 45

46. public void testIterationOrder() {
for (int size = 1; size < 1000; size++) {
List<Integer> other = new ArrayList<Integer>(size);
for (int i = 0; i < size; i++) {other.add(i);}
TreeList<Integer> l = new TreeList<Integer>(other);
ListIterator<Integer> it = l.listIterator();
int i = 0;
while (it.hasNext()) {
Integer val = it.next();
assertEquals(i++, val.intValue());
}
}
}
Original test case input
(data and functions)
oracle
(expected properties)
Benoit Baudry, KTH, ICES DevOps 46

47. Unit testing
Benoit Baudry, KTH, ICES DevOps
P
state
Input space Observation space
47

48. Unit test amplification
Benoit Baudry, KTH, ICES DevOps
P
state
Input space Observation space
48

49. 49
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}

50. 50
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}
@Test
public void html_add33() throws Exception {
Attribute attr = new Attribute("key", "value &");
Assert.assertEquals("key="value &"", attr.html());
Assert.assertEquals("key="value &"", attr.toString());
Assert.assertEquals("key", attr.getKey());
Assert.assertEquals("value &", attr.getValue());
}

51. 51

52. 52

53. 53
DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.

54. Test amplification
•Descartes: spots weak tests
•Dspot: suggests improvements of existing tests
•Available as Maven plugins
•Method
• Developer machine
• In the CI
•Future work
• Integration in PR-based development
• Develop work on advanced Maven dependency management
54

55. Open source tooling
•https://github.com/STAMP-project/pitest-descartes
•https://github.com/STAMP-project/dspot
•Two Maven plugins
55

56. More information
• An Approach and Benchmark to Detect Behavioral Changes of
Commits in Continuous Integration
• https://arxiv.org/pdf/1902.08482
• Automatic Test Improvement with DSpot: a Study with Ten Mature
Open-Source Projects
• https://arxiv.org/pdf/1811.08330
• A Comprehensive Study of Pseudo-tested Methods
• https://arxiv.org/pdf/1807.05030
• http://stamp-project.eu/
• https://github.com/KTH/devops-course/
• baudry@kth.se
56

57. •4 res. institutions
•5 companies
•1 open source
consortium
•516 p.m
•To increase test
automation in
DevOps