Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

Universiteit Antwerpen
"Next Level" Test Automation
Keynote VST 2020
3rd Workshop on Validation, Analysis and Evolution of Software Tests
February 2020
© Prof. Serge Demeyer

Books
• 2 books
• 3 proceedings (editor) Best Teacher’s Award
Top Publications
Spin Off &
Start Up

Ericsson, Bombardier,
Saab, System Verification,
Empear, Verifyter, KTH,
MDH, RISE Comiq, EfiCode,
Ponsse, Siili,
Qentinel, Symbio,
Uni.Oulu, VTT
Axini, Testwerk,
TNO, Open Uni. AKKA, Expleo,
EKS, FFT,
Fraunhofer,
IFAK, OFFIS,
Parasoft
Alerion,
Prodevelop,,
Uni.Mandragon Kuveyt Bank,
Saha BT
The TESTOMAT project will allow software teams to
increase the development speed without sacrificing quality
To achieve this goal, the project will advance the state-of-the-art in test
automation for software teams moving towards a more agile development
process.

Industry 4.0 Internet of Things

variability
"lot size 1"
safety
IEC 62061 (mechatronics)
RTCA/DO-178C (avionics)
IEC 62304 (medical)
ISO 26262 (automotive)
agility
rapid customer feedback
faster release cycles

Six decades into the computer revolution, four
decades since the invention of the microprocessor,
and two decades into the rise of the modern Internet,
all of the technology required to transform industries
through software ﬁnally works and can be widely
delivered at global scale.

Software Testing is the process of executing a program
or system with the intent of ﬁnding errors.
(Myers, Glenford J., The art of software testing. Wiley, 1979)

"Next Level" Test Automation (VST Keynote)
Context
11
Continuous Integration
Continuous Delivery
Continuous Deployment
DevOps
Tesla
“over-the-air” updates
± once every month
Amazon deploys to
production
± every 11,6 seconds
September 2015
Amazon Web Services
suffered major disruption.
NetFlix recovers quickly!
(Chaos Monkey)

Lean Manufacturing
12
Eliminate Waste
(e.g. Spaghetti Diagrams)
© Christoph Roser on AllAboutLean.com
By Dr Ian Mitchell - Own work©
Smooth Flow
(e.g. KanBan Boards)

Integration Hell
13
Requirement 
Collection
Analysis
Design
Implementation
Testing
Deployment
N
otsm
ooth
Lots
ofW
aste

Continuous Integration Pipeline
14
<<Breaking the Build>>
version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate

[Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of
Rapid Releases on Software Quality: The Case of Firefox, Empirical Software
Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x
1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0
8.0
9.0
Traditional Release Cycle Rapid Release Cycle
(a) Time Line of Major Versions of FireFox
(b) Time Line of Minor Versions of FireFox
Figure 1. Timeline of FireFox versions.
channels are respectively 100,000 for NIGHTLY, 1 million
for AURORA, 10 million for BETA and 100+ millions for
a major Firefox version [11]. NIGHTLY reaches Firefox
developers and contributors, while other channels (i.e., AU-
RORA and BETA) recruit external users for testing. The
source code on AURORA is tested by web developers who
are interested in the latest standards, and by Firefox add-on
developers who are willing to experiment with new browser
APIs. The BETA channel is tested by Firefox’s regular beta
by bug triaging developers and assigned for fixing. When
a developer fixes a bug, he typically submits a patch to
Bugzilla. Once approved, the patch code is integrated into
the source code of Firefox on the corresponding channel and
migrated through the other channels for release. Bugs that
take too long to get fixed and hence miss a scheduled release
are picked up by the next release’s channel.
III. STUDY DESIGN

[Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of
Rapid Releases on Software Quality: The Case of Firefox, Empirical Software
Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x
✓ bugs are fixed faster
(but … harder bugs propagated to later releases)
✓ amount of pre- & post-release bugs ± the same
✓ the program crashes earlier
(perhaps due to recent features)
3.6 4.0 5.0 7.0
8.0
9.0
Rapid Release Cycle
rs and assigned for fixing. When
he typically submits a patch to
the patch code is integrated into
on the corresponding channel and
er channels for release. Bugs that
nd hence miss a scheduled release
release’s channel.
TUDY DESIGN
earch questions:
ease cycle affect the
erence in the number
control for the time
lease dates. However,
tly lower for versions
les, i.e., failures seem
cycle affect the fixing
ter for versions devel-
e cycle affect software
d release model are
.e., the proportion of
ersions that possibly
5.0 NIGHTLY 6.0 NIGHTLY 7.0 NIGHTLY 8.0 NIGHTLY
5.0 AURORA 6.0 AURORA 7.0 AURORA
5.0 BETA 6.0 BETA
5.0 MAIN
New Feature Development
6 Weeks 6 Weeks 6 Weeks 6 Weeks
Figure 2. Development and Release Process of Mozilla Firefox
major release was made. Figure 1(b) shows the release dates
of the minor versions of Firefox.
With the advent of shorter release cycles in March 2011,
new features need to be tested and delivered to users faster.
To achieve this goal, Firefox changed its development pro-
cess. First, versions are no longer supported in parallel, i.e.,

Plan
• Context
• Test Strategy: When Should I Test?
- V-Model
- Fit tables (Acceptance Testing)
- Flipping the V
- 4 Quadrants
• Test Quality: How Good Are Your Tests?
• Test Analytics: Exploiting the DevOps pipeline.
• Test Research: Tips for PhD Students
17

V-Model
18
Requirements Acceptance Tests
Design Integration Tests
Coding Unit Tests
Architecture System Tests
Requirements Architecture Design Coding Testing
Test Design Test Execution
Acceptance
Test Cases
System Test
Cases
Integration Test
Cases
Unit Test
Cases
Unit Tests
Integration
Tests
System Tests
Acceptance
Tests
Integration hell?

Fit Tables
19
Browse Music
Play Music
Browse Music
start eg.music.browser
enter library
check total songs 37
Browse Music
enter select 1
check title Akila
check artist Toure Kunda
enter select 2
check title American Tango
check artist Weather Report
check album Mysterious Traveller
check year 1974
Example: Acceptance Test Cases
http://ﬁt.c2.com
Play Music
start eg.music.Realtime
press play
check status loading
pause 2
check status playing

Test Case Management
20
Smoke Test

Scrum — Feedback Loop
21
Product
Backlog
Sprint
Backlog
Sprint
Execution
Working Increment
of Product
24h
Sprint
Planning
Sprint
Review
Sprint
Retrospective

Behaviour Driven (User Stories)
22
As a <user role>
I want to <goal>
so that <beneﬁt>.
• …
• …
• …
As a clerk
I want to calculate stampage
so that goods get shipped fast.
• Verify with nearby address
• Verify with overseas address
• Verify with parcels <= 1kg
• Verify with fragile parcel
Template
Example
Conditions of Satisfaction

Flipping the V
23
Acceptance Tests
(GUI Tests)
System Tests
Integration Tests
Unit Tests
70%
20%
10%
10%
20%
70%
Test Automation

Flipping the V in Practice
24
Acceptance Tests
(GUI Tests)
System Tests
Integration Tests
Unit Tests

SupportingTeam
Functional Tests
Examples
Story Tests
Prototypes
Simulation
Exploratory Testing
Scenarios
Usability Testing
Acceptance Testing
Alpha / Beta
Unit Tests
Integration Tests
Performance Testing
Load Testing
Security Testing
“ility” Testing
Technology Facing
Business Facing
CritiqueProduct
Automated & Manual Manual
Automated Tools

Plan
• Context
- Coverage (Control flow, Statement, Branch, Path)
- Tests vs. Faults
* Reach, Infect, Propagate, Reveal
- Mutation Analysis
- Mutation Operators
- Case studies
* Scaling up: the cloud
27

How good are your tests?
version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate

Code Coverage
34
Test System Under Test
(SUT)
Code is instrumented
Test executes the code
Instrumented code gets
executed
The traces are stored:
• When statement is executed
- CFG (Control Flow Graph)
• When data is changed
- DFG (Data Flow Graph)

import org.junit.Test;
import static org.junit.Assert.assertEquals;
public class TestEmployeeDetails {
EmpBusinessLogic empBusinessLogic = new EmpBusinessLogic();
EmployeeDetails employee = new EmployeeDetails();
//happy day scenario for calculation of appraisal and salary
@Test
public void testCalculateAppriasal() {
employee.setName("Rajeev");
employee.setAge(25);
employee.setMonthlySalary(8000);
double appraisal = empBusinessLogic.calculateAppraisal(employee);
double salary = empBusinessLogic.calculateYearlySalary(employee);
}
}
assertionless
test

The RIPR Model
36
Reach
Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43(4):1-1 · January 2016
Infect
Propagate
Reveal

/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}

/**
*/
{
if (x[i] == y)
return i;
return -1;
}
x = null; y = 5

/**
*/
{
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 3

/**
*/
{
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 25

/**
*/
{
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 2

Mutation Testing
47
State of the Art State of the Practice
Number of mutation testing publications per year

int method(int v1, int v2)
{
if (v1 <v2)
return 1;
return -1;
}
int method(int v1, int v2)
{
if (v1 >=v2)
return 1;
return -1;
}

Operator Description Example
Before After
CBM
Mutates the boundary
conditions
a > b a >= b
IM Mutates increment operators a++ a−−
INM Inverts negation operator −a a
MM
Mutates arithmetic & logical
operators
a & b a | b
NCM
Negates a conditional
operator
a == b a != b
RVM
Mutates the return value of a
function
return true return false
VMCM Removes a void method call voidCall(x) –
Competent Programmer
Hypothesis
(Program is close to correct)
Coupling Effect
(Test suites capable of detecting simple errors 
will also detect complex errors)

Industrial Case Study
50
• 83K lines of code
• Complicated structure
• Lots of legacy code
• Lots of black-box tests
Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”.
Software Tools for Technology Transfer

http://littledarwin.parsai.net

Industrial Case
52
Unit tests only !
Segmentation
Percentage
020406080100
Mutation Coverage
Branch Coverage

CI
D
evelop
Build
Test
W
ay too slow
We witnessed 48 hours of mutation testing time on a
test suite comprising 272 unit tests and 5,258 lines of
test code for testing a project with 48,873 lines of
production code.
Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the
Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018

Master
1) Initial test Build
2) ∀ files to mutate:
queue file names
3a) Generate mutants
4a) Execute mutants
3b) Store mutants
4b) Store results
3c) Queue mutant
references
5) Process results

0
1h
2h
3h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers
0
12h
1d
1d 12h
2d
2d 12h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers

https://github.com/joakim-brannstrom/dextool

Presence of defect
+ reach the defect
+ infect the program state
+ observable on output
coverage
mutants
Mutation Testing
= Actionable !

Mutation Testing @ google
58
version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate
DevOps
TriCoder CodeCritique

Mutation Testing @ google
59
MutationTesting
Reported beneﬁts
• stronger tests,
• more effective debugging,
• prevention of bugs,
• improved code quality.
An Industrial Application of Mutation Testing: Lessons,
Challenges, and Research Directions, Goran Petrovic ́
Marko Ivankovic, Bob Kurtz Paul Ammann, René Just. ICST
Proceedings

Plan
• Context
• Test Analytics: Exploiting the DevOps pipeline
- Test Automation Maturity Model (TAIM)
- (Spectrum Based) Fault Localisation
- Test Amplification
- Bug Reports
- Stack Overflow
60

version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate
Artiﬁcial
Intelligence
Inside

Henri Heiskanen, Mika Maunumaa, and Mika Katara.
A test process improvement model for automated test
generation. In Oscar Dieste, Andreas Jedlitschka, and
Natalia Juristo, editors, Product-Focused Software
Process Improvement: 13th International Conference,
PROFES 2012, Madrid, Spain, June 13-15, 2012
Proceedings, pages 17--31, Berlin, Heidelberg, 2012.
Springer Berlin Heidelberg.
Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl,
and Kristian Wiklund. Towards a test automation
improvement model (TAIM). In Proceedings of the
2014 IEEE International Conference on Software
Testing, Verification, and Validation Workshops,
ICSTW '14, pages 337--342, Washington, DC, USA,
2014. IEEE Computer Society.
Ana Paula C. C. Furtado, Silvio R. L. Meira, and
Marcos Wanderley Gomes. Towards a maturity model
in software testing automation. In Proceedings of
ICSEA 2014 : The Ninth International Conference on
Software Engineering Advances, ICSEA2014, pages
282 - 285. IARIA, 2014.
Sigrid Eldh. Test Automation Improvement Model —
TAIM 2.0. In Proceedinsg NEXTA 2020 (Workshop on
Next Level Test Automation)
Test Process Improvement
for Automated Test
Generation
Version: 1.01.51 (2010‐05‐06) [Draft]
TPI is an evaluation framework to evaluate thematurity of a software testing
process. We have modified the original TPI framework to evaluate test processes
that areusing Automated Test Generation methodsand tools.
Henri Heiskanen, Mika Maunumaa, Mika Katara
10.2.2010
Tampere University of Technology, Department of Software Systems

Spectrum Based Fault Localisation
wastedeffort
7
46
56
122
48
2
104
124
74
6
109
9
83
68
116
90
86
27
32
97
128
16
115
81
52
57
60
42
62
63
69
75
23
26
88
65
77
84
73
1
133
51
113
76
117
53
61
38
34
105
29
72
78
96
43
103
33
4
87
49
30
85
15
112
25
118
125
82
20
110
132
39
45
89
71
10
114
5
24
64
35
54
40
3
127
70
28
120
98
121
37
19
100
11
99
108
14
17
131
50
95
66
58
41
80
92
93
47
44
91
67
8
126
31
36
79
55
13
21
22
106
119
123
130
102
111
129
12
18
101
94
107
59
7
46
56
122
48
2
104
124
74
6
109
9
83
68
116
90
86
27
32
97
128
16
115
81
52
57
60
42
62
63
69
75
23
26
88
65
77
84
73
1
133
51
113
76
117
53
61
38
34
105
29
72
78
96
43
103
33
4
87
49
30
85
15
112
25
118
125
82
20
110
132
39
45
89
71
10
114
5
24
64
35
54
40
3
127
70
28
120
98
121
37
19
100
11
99
108
14
17
131
50
95
66
58
41
80
92
93
47
44
91
67
8
126
31
36
79
55
13
21
22
106
119
123
130
102
111
129
12
18
101
94
107
59
11513515517519511151140116511901
11513515517519511151140116511901
Patterned Spectrum Analysis
Raw Spectrum Analysis
wastedeffort
3
15
2
8
9
23
1
14
17
5
18
16
4
7
26
21
25
22
6
13
24
10
12
19
20
27
3
15
2
8
9
23
1
14
17
5
18
16
4
7
26
21
25
22
6
13
24
10
12
19
20
27
151101151201251301351401451
151101151201251301351401451
Patterned Spectrum Analysis
Raw Spectrum Analysis
Artiﬁcial
Intelligence
Inside

Test Ampliﬁcation
68
Benjamin Danglot, Oscar Vera-Pérez, Benoit Baudry, Martin Monperrus. Automatic Test Improvement with DSpot: a Study with Ten
Mature Open-Source Projects. Empirical Software Engineering, Springer Verlag, 2019, pp.1-35. 10.1007/s10664-019-09692-y .
Mehrdad Abdi, Henrique Rocha and Serge Demeyer. Test Ampliﬁcation in the Pharo Smalltalk Ecosystem. Proceedings IWST 2019
(International Workshop on Smalltalk Technologies)
input generation
+ assertion generation
testWithdraw
|b|
b := SmallBank new. 
b deposit: 100. 
self assert: b balance equals: 100.
b withdraw: 30. 
self assert: b balance equals: 70
Genetic
Algorithms
Inside
testWithdraw_12
| b |
b := SmallBank new.
b deposit: 100.
b withdraw: SmallInteger maxVal.
self assert: b balance equals: 100

Description text Mining
Stack Traces Link to source code
Product/Component
Speciﬁc vocabulary
Suggestions?
Artiﬁcial
Intelligence
Inside

Question Cases Precision Recall
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
How long will it take to fix
this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of this
bug? (**)
Mozilla, Eclipse, Gnome
mozilla, eclipse:67% -
73%
gnome:
75%-82%
mozilla, eclipse:50% -
75%
gnome:
68%-84%
Promising results but …
• how much training is needed? (cross-project training?)
• how reliable is the data? (estimates, severity, assigned-to)
• does this generalise? (on industrial scale?)
replication is needed
(*) In CSMR2012 Proceedings
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
Irrelevant for
Practitioners
(**) In CSMR2011; MSR 2010 Proceedings
Artificial
Intelligence
Inside

Q&A support
71

In CHI2016 Proceedings
Artiﬁcial
Intelligence
Inside

Plan
• Context
- Descriptive Statistics
- Unit Tests vs. Integration Tests
73

Descriptive Statistics
74
Semi-automatic Test Case Expansion for Mutation Testing
ZhongXiLu,StenVercammen,andSergeDemeyer—UniversityofAntwerp,Belgium . . . . . . . . . . . . . . . . . . . . . . 1
An Early Investigation of Unit Testing Practices of Component-Based Software Systems
Georg Buchgeher, Stefan Fischer, Michael Moser, and Josef Pichler — Software Competence Center
Hagenberg, Austria; University ofAppliedSciencesUpperAustria,Austria ............................................ 12

Test Code Quality
75
At minimum include some code coverage
measurement + assertion density.

Unit Tests vs. Integration Tests (1/2)
76
Further analysis conﬁrmed that it is worthwhile to treat unit
tests and integration tests differently: we discovered that unit
tests cause more breaking builds, that ﬁxing the defects
exposed by unit tests takes longer and implies more
coordination between team members.

Unit Tests vs. Integration Tests (2/2)
77
Hence, if we apply the IEEE deﬁnition to Java a unit test is a test that tests
only units from within one package (i.e., related units). On the other hand, an
integration test tests units from more than one package.

version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate
Artiﬁcial
Intelligence
Inside
If your AI system cannot motivate its decision
practitioners will not accept it.

Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

Similar to Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests) (20)

More from University of Antwerp

More from University of Antwerp (11)

Recently uploaded

Recently uploaded (20)

Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)