SlideShare a Scribd company logo
Universiteit Antwerpen
"Next Level" Test Automation
Keynote VST 2020
3rd Workshop on Validation, Analysis and Evolution of Software Tests
February 2020
© Prof. Serge Demeyer
Books
• 2 books
• 3 proceedings (editor) Best Teacher’s Award
Top Publications
Spin Off &
Start Up
Ericsson, Bombardier,
Saab, System Verification,
Empear, Verifyter, KTH,
MDH, RISE Comiq, EfiCode,
Ponsse, Siili,
Qentinel, Symbio,
Uni.Oulu, VTT
Axini, Testwerk,
TNO, Open Uni. AKKA, Expleo,
EKS, FFT,
Fraunhofer,
IFAK, OFFIS,
Parasoft
Alerion,
Prodevelop,,
Uni.Mandragon Kuveyt Bank,
Saha BT
The TESTOMAT project will allow software teams to
increase the development speed without sacrificing quality
To achieve this goal, the project will advance the state-of-the-art in test
automation for software teams moving towards a more agile development
process.
Industry 4.0 Internet of Things
variability
"lot size 1"
safety
IEC 62061 (mechatronics)
RTCA/DO-178C (avionics)
IEC 62304 (medical)
ISO 26262 (automotive)
agility
rapid customer feedback
faster release cycles
Six decades into the computer revolution, four
decades since the invention of the microprocessor,
and two decades into the rise of the modern Internet,
all of the technology required to transform industries
through software finally works and can be widely
delivered at global scale.
Software Testing is the process of executing a program
or system with the intent of finding errors.
(Myers, Glenford J., The art of software testing. Wiley, 1979)
"Next Level" Test Automation (VST Keynote)
Context
11
Continuous Integration
Continuous Delivery
Continuous Deployment
DevOps
Tesla
“over-the-air” updates
± once every month
Amazon deploys to
production
± every 11,6 seconds
September 2015
Amazon Web Services
suffered major disruption.
NetFlix recovers quickly!
(Chaos Monkey)
"Next Level" Test Automation (VST Keynote)
Lean Manufacturing
12
Eliminate Waste
(e.g. Spaghetti Diagrams)
© Christoph Roser on AllAboutLean.com
By Dr Ian Mitchell - Own work©
Smooth Flow
(e.g. KanBan Boards)
"Next Level" Test Automation (VST Keynote)
Integration Hell
13
Requirement

Collection
Analysis
Design
Implementation
Testing
Deployment
N
otsm
ooth
Lots
ofW
aste
"Next Level" Test Automation (VST Keynote)
Continuous Integration Pipeline
14
<<Breaking the Build>>
version	
control
build
developer	
tests
deploy
scenario	
tests
deploy	to	
production
measure	&	
validate
[Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of
Rapid Releases on Software Quality: The Case of Firefox, Empirical Software
Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x
1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0
8.0
9.0
Traditional Release Cycle Rapid Release Cycle
(a) Time Line of Major Versions of FireFox
(b) Time Line of Minor Versions of FireFox
Figure 1. Timeline of FireFox versions.
channels are respectively 100,000 for NIGHTLY, 1 million
for AURORA, 10 million for BETA and 100+ millions for
a major Firefox version [11]. NIGHTLY reaches Firefox
developers and contributors, while other channels (i.e., AU-
RORA and BETA) recruit external users for testing. The
source code on AURORA is tested by web developers who
are interested in the latest standards, and by Firefox add-on
developers who are willing to experiment with new browser
APIs. The BETA channel is tested by Firefox’s regular beta
by bug triaging developers and assigned for fixing. When
a developer fixes a bug, he typically submits a patch to
Bugzilla. Once approved, the patch code is integrated into
the source code of Firefox on the corresponding channel and
migrated through the other channels for release. Bugs that
take too long to get fixed and hence miss a scheduled release
are picked up by the next release’s channel.
III. STUDY DESIGN
[Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of
Rapid Releases on Software Quality: The Case of Firefox, Empirical Software
Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x
✓ bugs are fixed faster
(but … harder bugs propagated to later releases)
✓ amount of pre- & post-release bugs ± the same
✓ the program crashes earlier
(perhaps due to recent features)
3.6 4.0 5.0 7.0
8.0
9.0
Rapid Release Cycle
rs and assigned for fixing. When
he typically submits a patch to
the patch code is integrated into
on the corresponding channel and
er channels for release. Bugs that
nd hence miss a scheduled release
release’s channel.
TUDY DESIGN
earch questions:
ease cycle affect the
erence in the number
control for the time
lease dates. However,
tly lower for versions
les, i.e., failures seem
cycle affect the fixing
ter for versions devel-
e cycle affect software
d release model are
.e., the proportion of
ersions that possibly
5.0 NIGHTLY 6.0 NIGHTLY 7.0 NIGHTLY 8.0 NIGHTLY
5.0 AURORA 6.0 AURORA 7.0 AURORA
5.0 BETA 6.0 BETA
5.0 MAIN
New Feature Development
6 Weeks 6 Weeks 6 Weeks 6 Weeks
Figure 2. Development and Release Process of Mozilla Firefox
major release was made. Figure 1(b) shows the release dates
of the minor versions of Firefox.
With the advent of shorter release cycles in March 2011,
new features need to be tested and delivered to users faster.
To achieve this goal, Firefox changed its development pro-
cess. First, versions are no longer supported in parallel, i.e.,
"Next Level" Test Automation (VST Keynote)
Plan
• Context
• Test Strategy: When Should I Test?
- V-Model
- Fit tables (Acceptance Testing)
- Flipping the V
- 4 Quadrants
• Test Quality: How Good Are Your Tests?
• Test Analytics: Exploiting the DevOps pipeline.
• Test Research: Tips for PhD Students
17
"Next Level" Test Automation (VST Keynote)
V-Model
18
Requirements Acceptance Tests
Design Integration Tests
Coding Unit Tests
Architecture System Tests
Requirements Architecture Design Coding Testing
Test Design Test Execution
Acceptance
Test Cases
System Test
Cases
Integration Test
Cases
Unit Test
Cases
Unit Tests
Integration
Tests
System Tests
Acceptance
Tests
Integration hell?
"Next Level" Test Automation (VST Keynote)
Fit Tables
19
Browse Music
Play Music
Browse Music
start eg.music.browser
enter library
check total songs 37
Browse Music
enter select 1
check title Akila
check artist Toure Kunda
enter select 2
check title American Tango
check artist Weather Report
check album Mysterious Traveller
check year 1974
Example: Acceptance Test Cases
http://fit.c2.com
Play Music
start eg.music.Realtime
press play
check status loading
pause 2
check status playing
"Next Level" Test Automation (VST Keynote)
Test Case Management
20
Smoke Test
"Next Level" Test Automation (VST Keynote)
Scrum — Feedback Loop
21
Product
Backlog
Sprint
Backlog
Sprint
Execution
Working Increment
of Product
24h
Sprint
Planning
Sprint
Review
Sprint
Retrospective
"Next Level" Test Automation (VST Keynote)
Behaviour Driven (User Stories)
22
As a <user role>
I want to <goal>
so that <benefit>.
• …
• …
• …
As a clerk
I want to calculate stampage
so that goods get shipped fast.
• Verify with nearby address
• Verify with overseas address
• Verify with parcels <= 1kg
• Verify with fragile parcel
Template
Example
Conditions of Satisfaction
"Next Level" Test Automation (VST Keynote)
Flipping the V
23
Acceptance Tests
(GUI Tests)
System Tests
Integration Tests
Unit Tests
70%
20%
10%
10%
20%
70%
Test Automation
"Next Level" Test Automation (VST Keynote)
Flipping the V in Practice
24
Acceptance Tests
(GUI Tests)
System Tests
Integration Tests
Unit Tests
SupportingTeam
Functional Tests
Examples
Story Tests
Prototypes
Simulation
Exploratory Testing
Scenarios
Usability Testing
Acceptance Testing
Alpha / Beta
Unit Tests
Integration Tests
Performance Testing
Load Testing
Security Testing
“ility” Testing
Technology Facing
Business Facing
CritiqueProduct
Automated & Manual Manual
Automated Tools
unit tests?
"Next Level" Test Automation (VST Keynote)
Plan
• Context
• Test Strategy: When Should I Test?
• Test Quality: How Good Are Your Tests?
- Coverage (Control flow, Statement, Branch, Path)
- Tests vs. Faults
* Reach, Infect, Propagate, Reveal
- Mutation Analysis
- Mutation Operators
- Case studies
* Scaling up: the cloud
• Test Analytics: Exploiting the DevOps pipeline.
• Test Research: Tips for PhD Students
27
How good are your tests?
version	
control
build
developer	
tests
deploy
scenario	
tests
deploy	to	
production
measure	&	
validate
90,000 test executions/day
c++
java
"Next Level" Test Automation (VST Keynote)
Code Coverage
34
Test System Under Test
(SUT)
Code is instrumented
Test executes the code
Instrumented code gets
executed
The traces are stored:
• When statement is executed
- CFG (Control Flow Graph)
• When data is changed
- DFG (Data Flow Graph)
import org.junit.Test;
import static org.junit.Assert.assertEquals;
public class TestEmployeeDetails {
EmpBusinessLogic empBusinessLogic = new EmpBusinessLogic();
EmployeeDetails employee = new EmployeeDetails();
//happy day scenario for calculation of appraisal and salary
@Test
public void testCalculateAppriasal() {
employee.setName("Rajeev");
employee.setAge(25);
employee.setMonthlySalary(8000);
double appraisal = empBusinessLogic.calculateAppraisal(employee);
double salary = empBusinessLogic.calculateYearlySalary(employee);
}
}
assertionless
test
"Next Level" Test Automation (VST Keynote)
The RIPR Model
36
Reach
Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43(4):1-1 · January 2016 
Infect
Propagate
Reveal
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
x = null; y = 5
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 3
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 25
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
/**
* Find last index of element
* @param x array to search
* @param y element to look for
* @return last index of y in x, if absent -1
* @throws NullPointerException if x is null
*/
public static int findLast(int [] x, int y)
{
for (int i=x.length-1; i>0; i--)
if (x[i] == y)
return i;
return -1;
}
x = [2,3,5]; y = 2
"Next Level" Test Automation (VST Keynote)
Mutation Testing
47
State of the Art State of the Practice
Number of mutation testing publications per year
int method(int v1, int v2)
{
if (v1 <v2)
return 1;
return -1;
}
int method(int v1, int v2)
{
if (v1 >=v2)
return 1;
return -1;
}
Operator Description Example
Before After
CBM
Mutates the boundary
conditions
a > b a >= b
IM Mutates increment operators a++ a−−
INM Inverts negation operator −a a
MM
Mutates arithmetic & logical
operators
a & b a | b
NCM
Negates a conditional
operator
a == b a != b
RVM
Mutates the return value of a
function
return true return false
VMCM Removes a void method call voidCall(x) –
Competent Programmer
Hypothesis
(Program is close to correct)
Coupling Effect
(Test suites capable of detecting simple errors

will also detect complex errors)
"Next Level" Test Automation (VST Keynote)
Industrial Case Study
50
• 83K lines of code
• Complicated structure
• Lots of legacy code
• Lots of black-box tests
Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”.
Software Tools for Technology Transfer
http://littledarwin.parsai.net
"Next Level" Test Automation (VST Keynote)
Industrial Case
52
Unit tests only !
Segmentation
Percentage
020406080100
Mutation Coverage
Branch Coverage
CI
D
evelop
Build
Test
W
ay too slow
We witnessed 48 hours of mutation testing time on a
test suite comprising 272 unit tests and 5,258 lines of
test code for testing a project with 48,873 lines of
production code.
Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the
Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018
Master
1) Initial test Build
2) ∀ files to mutate:
queue file names
3a) Generate mutants
4a) Execute mutants
3b) Store mutants
4b) Store results
3c) Queue mutant
references
5) Process results
0
1h
2h
3h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers
0
12h
1d
1d 12h
2d
2d 12h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers
https://github.com/joakim-brannstrom/dextool
Presence of defect
+ reach the defect
+ infect the program state
+ observable on output
coverage
mutants
Mutation Testing
= Actionable !
"Next Level" Test Automation (VST Keynote)
Mutation Testing @ google
58
version	
control
build
developer	
tests
deploy
scenario	
tests
deploy	to	
production
measure	&	
validate
DevOps
TriCoder CodeCritique
"Next Level" Test Automation (VST Keynote)
Mutation Testing @ google
59
MutationTesting
Reported benefits
• stronger tests,
• more effective debugging,
• prevention of bugs,
• improved code quality.
An Industrial Application of Mutation Testing: Lessons,
Challenges, and Research Directions, Goran Petrovic ́
Marko Ivankovic, Bob Kurtz Paul Ammann, René Just. ICST
Proceedings
"Next Level" Test Automation (VST Keynote)
Plan
• Context
• Test Strategy: When Should I Test?
• Test Quality: How Good Are Your Tests?
• Test Analytics: Exploiting the DevOps pipeline
- Test Automation Maturity Model (TAIM)
- (Spectrum Based) Fault Localisation
- Test Amplification
- Bug Reports
- Stack Overflow
• Test Research: Tips for PhD Students
60
version	
control
build
developer	
tests
deploy
scenario	
tests
deploy	to	
production
measure	&	
validate
Artificial
Intelligence
Inside
© TMMi © TPI next
Henri Heiskanen, Mika Maunumaa, and Mika Katara.
A test process improvement model for automated test
generation. In Oscar Dieste, Andreas Jedlitschka, and
Natalia Juristo, editors, Product-Focused Software
Process Improvement: 13th International Conference,
PROFES 2012, Madrid, Spain, June 13-15, 2012
Proceedings, pages 17--31, Berlin, Heidelberg, 2012.
Springer Berlin Heidelberg.
Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl,
and Kristian Wiklund. Towards a test automation
improvement model (TAIM). In Proceedings of the
2014 IEEE International Conference on Software
Testing, Verification, and Validation Workshops,
ICSTW '14, pages 337--342, Washington, DC, USA,
2014. IEEE Computer Society.
Ana Paula C. C. Furtado, Silvio R. L. Meira, and
Marcos Wanderley Gomes. Towards a maturity model
in software testing automation. In Proceedings of
ICSEA 2014 : The Ninth International Conference on
Software Engineering Advances, ICSEA2014, pages
282 - 285. IARIA, 2014.
Sigrid Eldh. Test Automation Improvement Model —
TAIM 2.0. In Proceedinsg NEXTA 2020 (Workshop on
Next Level Test Automation)
Test Process Improvement 
for Automated Test 
Generation 
Version: 1.01.51 (2010‐05‐06)  [Draft] 
TPI is an evaluation framework to evaluate thematurity of a software testing
process. We have modified the original TPI framework to evaluate test processes
that areusing Automated Test Generation methodsand tools.
Henri Heiskanen, Mika Maunumaa, Mika Katara 
10.2.2010 
Tampere University of Technology, Department of Software Systems 
Fault Localisation: Debugger
Spectrum Based Fault Localisation
wastedeffort
7
46
56
122
48
2
104
124
74
6
109
9
83
68
116
90
86
27
32
97
128
16
115
81
52
57
60
42
62
63
69
75
23
26
88
65
77
84
73
1
133
51
113
76
117
53
61
38
34
105
29
72
78
96
43
103
33
4
87
49
30
85
15
112
25
118
125
82
20
110
132
39
45
89
71
10
114
5
24
64
35
54
40
3
127
70
28
120
98
121
37
19
100
11
99
108
14
17
131
50
95
66
58
41
80
92
93
47
44
91
67
8
126
31
36
79
55
13
21
22
106
119
123
130
102
111
129
12
18
101
94
107
59
7
46
56
122
48
2
104
124
74
6
109
9
83
68
116
90
86
27
32
97
128
16
115
81
52
57
60
42
62
63
69
75
23
26
88
65
77
84
73
1
133
51
113
76
117
53
61
38
34
105
29
72
78
96
43
103
33
4
87
49
30
85
15
112
25
118
125
82
20
110
132
39
45
89
71
10
114
5
24
64
35
54
40
3
127
70
28
120
98
121
37
19
100
11
99
108
14
17
131
50
95
66
58
41
80
92
93
47
44
91
67
8
126
31
36
79
55
13
21
22
106
119
123
130
102
111
129
12
18
101
94
107
59
11513515517519511151140116511901
11513515517519511151140116511901
Patterned Spectrum Analysis
Raw Spectrum Analysis
wastedeffort
3
15
2
8
9
23
1
14
17
5
18
16
4
7
26
21
25
22
6
13
24
10
12
19
20
27
3
15
2
8
9
23
1
14
17
5
18
16
4
7
26
21
25
22
6
13
24
10
12
19
20
27
151101151201251301351401451
151101151201251301351401451
Patterned Spectrum Analysis
Raw Spectrum Analysis
Artificial
Intelligence
Inside
unit tests?
"Next Level" Test Automation (VST Keynote)
Test Amplification
68
Benjamin Danglot, Oscar Vera-Pérez, Benoit Baudry, Martin Monperrus. Automatic Test Improvement with DSpot: a Study with Ten
Mature Open-Source Projects. Empirical Software Engineering, Springer Verlag, 2019, pp.1-35. 10.1007/s10664-019-09692-y .
Mehrdad Abdi, Henrique Rocha and Serge Demeyer. Test Amplification in the Pharo Smalltalk Ecosystem. Proceedings IWST 2019
(International Workshop on Smalltalk Technologies)
input generation
+ assertion generation
testWithdraw
|b|
b := SmallBank new.

b deposit: 100.

self assert: b balance equals: 100.
b withdraw: 30.

self assert: b balance equals: 70
Genetic
Algorithms
Inside
testWithdraw_12
| b |
b := SmallBank new.
b deposit: 100.
b withdraw: SmallInteger maxVal.
self assert: b balance equals: 100
Description text Mining
Stack Traces Link to source code
Product/Component
Specific vocabulary
Suggestions?
Artificial
Intelligence
Inside
Question Cases Precision Recall
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
How long will it take to fix
this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of this
bug? (**)
Mozilla, Eclipse, Gnome
mozilla, eclipse:67% -
73%
gnome:
75%-82%
mozilla, eclipse:50% -
75%
gnome:
68%-84%
Promising results but …
• how much training is needed? (cross-project training?)
• how reliable is the data? (estimates, severity, assigned-to)
• does this generalise? (on industrial scale?)
replication is needed
(*) In CSMR2012 Proceedings
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
Irrelevant for
Practitioners
(**) In CSMR2011; MSR 2010 Proceedings
Artificial
Intelligence
Inside
"Next Level" Test Automation (VST Keynote)
Q&A support
71
In CHI2016 Proceedings
Artificial
Intelligence
Inside
"Next Level" Test Automation (VST Keynote)
Plan
• Context
• Test Strategy: When Should I Test?
• Test Quality: How Good Are Your Tests?
• Test Analytics: Exploiting the DevOps pipeline.
• Test Research: Tips for PhD Students
- Descriptive Statistics
- Unit Tests vs. Integration Tests
73
"Next Level" Test Automation (VST Keynote)
Descriptive Statistics
74
Semi-automatic Test Case Expansion for Mutation Testing
ZhongXiLu,StenVercammen,andSergeDemeyer—UniversityofAntwerp,Belgium . . . . . . . . . . . . . . . . . . . . . . 1
An Early Investigation of Unit Testing Practices of Component-Based Software Systems
Georg Buchgeher, Stefan Fischer, Michael Moser, and Josef Pichler — Software Competence Center
Hagenberg, Austria; University ofAppliedSciencesUpperAustria,Austria ............................................ 12
"Next Level" Test Automation (VST Keynote)
Test Code Quality
75
At minimum include some code coverage
measurement + assertion density.
"Next Level" Test Automation (VST Keynote)
Unit Tests vs. Integration Tests (1/2)
76
Further analysis confirmed that it is worthwhile to treat unit
tests and integration tests differently: we discovered that unit
tests cause more breaking builds, that fixing the defects
exposed by unit tests takes longer and implies more
coordination between team members.
"Next Level" Test Automation (VST Keynote)
Unit Tests vs. Integration Tests (2/2)
77
Hence, if we apply the IEEE definition to Java a unit test is a test that tests
only units from within one package (i.e., related units). On the other hand, an
integration test tests units from more than one package.
version	
control
build
developer	
tests
deploy
scenario	
tests
deploy	to	
production
measure	&	
validate
Artificial
Intelligence
Inside
If your AI system cannot motivate its decision
practitioners will not accept it.

More Related Content

What's hot

How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
How do you implement Continuous Delivery?: Part 5 - Deployment Patterns How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
ThoughtWorks Studios
 
Continuous delivery @åf consult
Continuous delivery @åf consultContinuous delivery @åf consult
Continuous delivery @åf consult
Tomas Riha
 
Increasing Quality with DevOps
Increasing Quality with DevOpsIncreasing Quality with DevOps
Increasing Quality with DevOps
Coveros, Inc.
 
Continuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 PlatformContinuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 Platform
Techsophy Inc.
 
Infrastructure
InfrastructureInfrastructure
Infrastructure
Yujuan Jiang
 
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
CURVC Corp
 
Mobile Quality Assurance
Mobile Quality AssuranceMobile Quality Assurance
Mobile Quality Assurance
Nudge Software Inc.
 
Personalized Defect Prediction
Personalized Defect PredictionPersonalized Defect Prediction
Personalized Defect Prediction
Sung Kim
 
How do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about PipelinesHow do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about Pipelines
Thoughtworks
 
How to Learn The History of Software Testing
How to Learn The History of Software Testing How to Learn The History of Software Testing
How to Learn The History of Software Testing
Keizo Tatsumi
 
Mobile Apps development best practices. TDD, CI, CD
Mobile Apps development best practices. TDD, CI, CDMobile Apps development best practices. TDD, CI, CD
Mobile Apps development best practices. TDD, CI, CD
GlobalLogic Ukraine
 
Balance Change and Control of Continuous Delivery at Scale
Balance Change and Control of Continuous Delivery at ScaleBalance Change and Control of Continuous Delivery at Scale
Balance Change and Control of Continuous Delivery at Scale
Plutora
 
Meenakshi Pal_16
Meenakshi Pal_16Meenakshi Pal_16
Meenakshi Pal_16
Meenakshi Pal
 
Delivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous IntegrationDelivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous Integration
Aspire Systems
 
Manual testing
Manual testingManual testing
Manual testing
Krishna Gurjar
 
The History of Software Testing
The History of Software TestingThe History of Software Testing
The History of Software Testing
Yaroslavtseva Iryna
 
Vinodhkumar
VinodhkumarVinodhkumar
Vinodhkumar
Bysani Vinodhkumar
 
DevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroDevOps - Boldly Go for Distro
DevOps - Boldly Go for Distro
Paul Boos
 
Enforcing Quality with DevOps Pipeline Gates
Enforcing Quality with DevOps Pipeline GatesEnforcing Quality with DevOps Pipeline Gates
Enforcing Quality with DevOps Pipeline Gates
Michael King
 
Tccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcityTccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcity
Baskin Tapkan
 

What's hot (20)

How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
How do you implement Continuous Delivery?: Part 5 - Deployment Patterns How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
How do you implement Continuous Delivery?: Part 5 - Deployment Patterns
 
Continuous delivery @åf consult
Continuous delivery @åf consultContinuous delivery @åf consult
Continuous delivery @åf consult
 
Increasing Quality with DevOps
Increasing Quality with DevOpsIncreasing Quality with DevOps
Increasing Quality with DevOps
 
Continuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 PlatformContinuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 Platform
 
Infrastructure
InfrastructureInfrastructure
Infrastructure
 
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
SonarQube와 함께하는 소프트웨어 품질 세미나 - 지속적인 코드 인스펙션 SonarQube 활용 방안
 
Mobile Quality Assurance
Mobile Quality AssuranceMobile Quality Assurance
Mobile Quality Assurance
 
Personalized Defect Prediction
Personalized Defect PredictionPersonalized Defect Prediction
Personalized Defect Prediction
 
How do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about PipelinesHow do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about Pipelines
 
How to Learn The History of Software Testing
How to Learn The History of Software Testing How to Learn The History of Software Testing
How to Learn The History of Software Testing
 
Mobile Apps development best practices. TDD, CI, CD
Mobile Apps development best practices. TDD, CI, CDMobile Apps development best practices. TDD, CI, CD
Mobile Apps development best practices. TDD, CI, CD
 
Balance Change and Control of Continuous Delivery at Scale
Balance Change and Control of Continuous Delivery at ScaleBalance Change and Control of Continuous Delivery at Scale
Balance Change and Control of Continuous Delivery at Scale
 
Meenakshi Pal_16
Meenakshi Pal_16Meenakshi Pal_16
Meenakshi Pal_16
 
Delivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous IntegrationDelivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous Integration
 
Manual testing
Manual testingManual testing
Manual testing
 
The History of Software Testing
The History of Software TestingThe History of Software Testing
The History of Software Testing
 
Vinodhkumar
VinodhkumarVinodhkumar
Vinodhkumar
 
DevOps - Boldly Go for Distro
DevOps - Boldly Go for DistroDevOps - Boldly Go for Distro
DevOps - Boldly Go for Distro
 
Enforcing Quality with DevOps Pipeline Gates
Enforcing Quality with DevOps Pipeline GatesEnforcing Quality with DevOps Pipeline Gates
Enforcing Quality with DevOps Pipeline Gates
 
Tccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcityTccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcity
 

Similar to Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
University of Antwerp
 
Cloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct servicesCloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct services
André Agostinho
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"
Stephanie Locke
 
Developing software at scale cs 394 may 2011
Developing software at scale   cs 394 may 2011Developing software at scale   cs 394 may 2011
Developing software at scale cs 394 may 2011
Todd Warren
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
Mike Brittain
 
Continuous Integration & the Release Maturity Model
Continuous Integration & the Release Maturity Model Continuous Integration & the Release Maturity Model
Continuous Integration & the Release Maturity Model
cPrime | Project Management | Agile | Consulting | Staffing | Training
 
The Cloudification Perspectives of Search-based Software Testing
The Cloudification Perspectives of Search-based Software TestingThe Cloudification Perspectives of Search-based Software Testing
The Cloudification Perspectives of Search-based Software Testing
Sebastiano Panichella
 
Control source code quality using the SonarQube platform
Control source code quality using the SonarQube platformControl source code quality using the SonarQube platform
Control source code quality using the SonarQube platform
PVS-Studio
 
SDLC & DevOps Transformation with Agile
SDLC & DevOps Transformation with AgileSDLC & DevOps Transformation with Agile
SDLC & DevOps Transformation with Agile
Abdel Moneim Emad
 
2016 quali continuous testing quest for quality conference
2016 quali continuous testing quest for quality conference2016 quali continuous testing quest for quality conference
2016 quali continuous testing quest for quality conference
QualiQuali
 
WebRTC Status Update - 2017Q2
WebRTC Status Update - 2017Q2WebRTC Status Update - 2017Q2
WebRTC Status Update - 2017Q2
Alexandre Gouaillard
 
Value stream mapping for DevOps
Value stream mapping for DevOpsValue stream mapping for DevOps
Value stream mapping for DevOps
Marc Hornbeek
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
CIVEL Benoit
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1
CIVEL Benoit
 
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer ToolsDevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
Amazon Web Services
 
Ibm innovate ci for system z
Ibm innovate ci for system zIbm innovate ci for system z
Ibm innovate ci for system z
Rosalind Radcliffe
 
A Declarative Approach for Performance Tests Execution in Continuous Software...
A Declarative Approach for Performance Tests Execution in Continuous Software...A Declarative Approach for Performance Tests Execution in Continuous Software...
A Declarative Approach for Performance Tests Execution in Continuous Software...
Vincenzo Ferme
 
Ovp Introduction Presentation (04 Feb 10)
Ovp Introduction Presentation (04 Feb 10)Ovp Introduction Presentation (04 Feb 10)
Ovp Introduction Presentation (04 Feb 10)
simon56
 
Continuous Delivery with a PaaS Application
Continuous Delivery with a PaaS ApplicationContinuous Delivery with a PaaS Application
Continuous Delivery with a PaaS Application
Mark Rendell
 
Bottlenecks rel b works and rel c planning
Bottlenecks rel b works and rel c planningBottlenecks rel b works and rel c planning
Bottlenecks rel b works and rel c planning
Jun Li
 

Similar to Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests) (20)

Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs — Exploiting Automated Tests to In...
 
Cloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct servicesCloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct services
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"
 
Developing software at scale cs 394 may 2011
Developing software at scale   cs 394 may 2011Developing software at scale   cs 394 may 2011
Developing software at scale cs 394 may 2011
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
Continuous Integration & the Release Maturity Model
Continuous Integration & the Release Maturity Model Continuous Integration & the Release Maturity Model
Continuous Integration & the Release Maturity Model
 
The Cloudification Perspectives of Search-based Software Testing
The Cloudification Perspectives of Search-based Software TestingThe Cloudification Perspectives of Search-based Software Testing
The Cloudification Perspectives of Search-based Software Testing
 
Control source code quality using the SonarQube platform
Control source code quality using the SonarQube platformControl source code quality using the SonarQube platform
Control source code quality using the SonarQube platform
 
SDLC & DevOps Transformation with Agile
SDLC & DevOps Transformation with AgileSDLC & DevOps Transformation with Agile
SDLC & DevOps Transformation with Agile
 
2016 quali continuous testing quest for quality conference
2016 quali continuous testing quest for quality conference2016 quali continuous testing quest for quality conference
2016 quali continuous testing quest for quality conference
 
WebRTC Status Update - 2017Q2
WebRTC Status Update - 2017Q2WebRTC Status Update - 2017Q2
WebRTC Status Update - 2017Q2
 
Value stream mapping for DevOps
Value stream mapping for DevOpsValue stream mapping for DevOps
Value stream mapping for DevOps
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1
 
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer ToolsDevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
 
Ibm innovate ci for system z
Ibm innovate ci for system zIbm innovate ci for system z
Ibm innovate ci for system z
 
A Declarative Approach for Performance Tests Execution in Continuous Software...
A Declarative Approach for Performance Tests Execution in Continuous Software...A Declarative Approach for Performance Tests Execution in Continuous Software...
A Declarative Approach for Performance Tests Execution in Continuous Software...
 
Ovp Introduction Presentation (04 Feb 10)
Ovp Introduction Presentation (04 Feb 10)Ovp Introduction Presentation (04 Feb 10)
Ovp Introduction Presentation (04 Feb 10)
 
Continuous Delivery with a PaaS Application
Continuous Delivery with a PaaS ApplicationContinuous Delivery with a PaaS Application
Continuous Delivery with a PaaS Application
 
Bottlenecks rel b works and rel c planning
Bottlenecks rel b works and rel c planningBottlenecks rel b works and rel c planning
Bottlenecks rel b works and rel c planning
 

More from University of Antwerp

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
University of Antwerp
 
AI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience ReportsAI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience Reports
University of Antwerp
 
Test Amplification in Python — An Industrial Experience Report
       Test Amplification in Python — An Industrial Experience Report       Test Amplification in Python — An Industrial Experience Report
Test Amplification in Python — An Industrial Experience Report
University of Antwerp
 
Technical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-UpsTechnical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-Ups
University of Antwerp
 
Social Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant ForksSocial Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant Forks
University of Antwerp
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
University of Antwerp
 
VST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdfVST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdf
University of Antwerp
 
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
University of Antwerp
 
Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
Reproducible Crashes: Fuzzing Pharo by Mutating the Test MethodsReproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
University of Antwerp
 
Test Automation Maturity: A Self-Assessment Tool
Test Automation Maturity: A Self-Assessment ToolTest Automation Maturity: A Self-Assessment Tool
Test Automation Maturity: A Self-Assessment Tool
University of Antwerp
 
Saner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblindSaner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblind
University of Antwerp
 

More from University of Antwerp (11)

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
AI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience ReportsAI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience Reports
 
Test Amplification in Python — An Industrial Experience Report
       Test Amplification in Python — An Industrial Experience Report       Test Amplification in Python — An Industrial Experience Report
Test Amplification in Python — An Industrial Experience Report
 
Technical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-UpsTechnical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-Ups
 
Social Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant ForksSocial Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant Forks
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
 
VST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdfVST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdf
 
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
Formal Verification of Developer Tests: a Research Agenda Inspired by Mutatio...
 
Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
Reproducible Crashes: Fuzzing Pharo by Mutating the Test MethodsReproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods
 
Test Automation Maturity: A Self-Assessment Tool
Test Automation Maturity: A Self-Assessment ToolTest Automation Maturity: A Self-Assessment Tool
Test Automation Maturity: A Self-Assessment Tool
 
Saner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblindSaner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblind
 

Recently uploaded

IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE  AND ITS BENIFITS.pptxIMPORTANCE OF ALGAE  AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
OmAle5
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
PsychoTech Services
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Sérgio Sacani
 
Immunotherapy presentation from clinical immunology
Immunotherapy presentation from clinical immunologyImmunotherapy presentation from clinical immunology
Immunotherapy presentation from clinical immunology
VetriVel359477
 
2001_Book_HumanChromosomes - Genéticapdf
2001_Book_HumanChromosomes - Genéticapdf2001_Book_HumanChromosomes - Genéticapdf
2001_Book_HumanChromosomes - Genéticapdf
lucianamillenium
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
QusayMaghayerh
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Sérgio Sacani
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
RAYMUNDONAVARROCORON
 
Signatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coastsSignatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coasts
Sérgio Sacani
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
frank0071
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
Sérgio Sacani
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
PirithiRaju
 
Alternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart AgricultureAlternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Embracing Deep Variability For Reproducibility and Replicability
Embracing Deep Variability For Reproducibility and ReplicabilityEmbracing Deep Variability For Reproducibility and Replicability
Embracing Deep Variability For Reproducibility and Replicability
University of Rennes, INSA Rennes, Inria/IRISA, CNRS
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
Scintica Instrumentation
 
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdfHolsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
frank0071
 
gastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptxgastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptx
Shekar Boddu
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
PirithiRaju
 
Quality assurance B.pharm 6th semester BP606T UNIT 5
Quality assurance B.pharm 6th semester BP606T UNIT 5Quality assurance B.pharm 6th semester BP606T UNIT 5
Quality assurance B.pharm 6th semester BP606T UNIT 5
vimalveerammal
 

Recently uploaded (20)

IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE  AND ITS BENIFITS.pptxIMPORTANCE OF ALGAE  AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
 
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...Discovery of An Apparent Red, High-Velocity Type Ia Supernova at  𝐳 = 2.9  wi...
Discovery of An Apparent Red, High-Velocity Type Ia Supernova at 𝐳 = 2.9 wi...
 
Immunotherapy presentation from clinical immunology
Immunotherapy presentation from clinical immunologyImmunotherapy presentation from clinical immunology
Immunotherapy presentation from clinical immunology
 
2001_Book_HumanChromosomes - Genéticapdf
2001_Book_HumanChromosomes - Genéticapdf2001_Book_HumanChromosomes - Genéticapdf
2001_Book_HumanChromosomes - Genéticapdf
 
Introduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptxIntroduction_Ch_01_Biotech Biotechnology course .pptx
Introduction_Ch_01_Biotech Biotechnology course .pptx
 
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
 
Signatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coastsSignatures of wave erosion in Titan’s coasts
Signatures of wave erosion in Titan’s coasts
 
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
Juaristi, Jon. - El canon espanol. El legado de la cultura española a la civi...
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
 
11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf11.1 Role of physical biological in deterioration of grains.pdf
11.1 Role of physical biological in deterioration of grains.pdf
 
Alternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart AgricultureAlternate Wetting and Drying - Climate Smart Agriculture
Alternate Wetting and Drying - Climate Smart Agriculture
 
Embracing Deep Variability For Reproducibility and Replicability
Embracing Deep Variability For Reproducibility and ReplicabilityEmbracing Deep Variability For Reproducibility and Replicability
Embracing Deep Variability For Reproducibility and Replicability
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
 
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdfHolsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
Holsinger, Bruce W. - Music, body and desire in medieval culture [2001].pdf
 
gastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptxgastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptx
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
 
Quality assurance B.pharm 6th semester BP606T UNIT 5
Quality assurance B.pharm 6th semester BP606T UNIT 5Quality assurance B.pharm 6th semester BP606T UNIT 5
Quality assurance B.pharm 6th semester BP606T UNIT 5
 

Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software Tests)

  • 1. Universiteit Antwerpen "Next Level" Test Automation Keynote VST 2020 3rd Workshop on Validation, Analysis and Evolution of Software Tests February 2020 © Prof. Serge Demeyer
  • 2. Books • 2 books • 3 proceedings (editor) Best Teacher’s Award Top Publications Spin Off & Start Up
  • 3.
  • 4. Ericsson, Bombardier, Saab, System Verification, Empear, Verifyter, KTH, MDH, RISE Comiq, EfiCode, Ponsse, Siili, Qentinel, Symbio, Uni.Oulu, VTT Axini, Testwerk, TNO, Open Uni. AKKA, Expleo, EKS, FFT, Fraunhofer, IFAK, OFFIS, Parasoft Alerion, Prodevelop,, Uni.Mandragon Kuveyt Bank, Saha BT The TESTOMAT project will allow software teams to increase the development speed without sacrificing quality To achieve this goal, the project will advance the state-of-the-art in test automation for software teams moving towards a more agile development process.
  • 6. variability "lot size 1" safety IEC 62061 (mechatronics) RTCA/DO-178C (avionics) IEC 62304 (medical) ISO 26262 (automotive) agility rapid customer feedback faster release cycles
  • 7. Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software finally works and can be widely delivered at global scale.
  • 8.
  • 9.
  • 10. Software Testing is the process of executing a program or system with the intent of finding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979)
  • 11. "Next Level" Test Automation (VST Keynote) Context 11 Continuous Integration Continuous Delivery Continuous Deployment DevOps Tesla “over-the-air” updates ± once every month Amazon deploys to production ± every 11,6 seconds September 2015 Amazon Web Services suffered major disruption. NetFlix recovers quickly! (Chaos Monkey)
  • 12. "Next Level" Test Automation (VST Keynote) Lean Manufacturing 12 Eliminate Waste (e.g. Spaghetti Diagrams) © Christoph Roser on AllAboutLean.com By Dr Ian Mitchell - Own work© Smooth Flow (e.g. KanBan Boards)
  • 13. "Next Level" Test Automation (VST Keynote) Integration Hell 13 Requirement
 Collection Analysis Design Implementation Testing Deployment N otsm ooth Lots ofW aste
  • 14. "Next Level" Test Automation (VST Keynote) Continuous Integration Pipeline 14 <<Breaking the Build>> version control build developer tests deploy scenario tests deploy to production measure & validate
  • 15. [Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of Rapid Releases on Software Quality: The Case of Firefox, Empirical Software Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x 1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0 8.0 9.0 Traditional Release Cycle Rapid Release Cycle (a) Time Line of Major Versions of FireFox (b) Time Line of Minor Versions of FireFox Figure 1. Timeline of FireFox versions. channels are respectively 100,000 for NIGHTLY, 1 million for AURORA, 10 million for BETA and 100+ millions for a major Firefox version [11]. NIGHTLY reaches Firefox developers and contributors, while other channels (i.e., AU- RORA and BETA) recruit external users for testing. The source code on AURORA is tested by web developers who are interested in the latest standards, and by Firefox add-on developers who are willing to experiment with new browser APIs. The BETA channel is tested by Firefox’s regular beta by bug triaging developers and assigned for fixing. When a developer fixes a bug, he typically submits a patch to Bugzilla. Once approved, the patch code is integrated into the source code of Firefox on the corresponding channel and migrated through the other channels for release. Bugs that take too long to get fixed and hence miss a scheduled release are picked up by the next release’s channel. III. STUDY DESIGN
  • 16. [Khom2014] Khomh, F. Adams, B, Dhaliwal, T and Zou, Y Understanding the Impact of Rapid Releases on Software Quality: The Case of Firefox, Empirical Software Engineering, Springer. http://link.springer.com/article/10.1007/s10664-014-9308-x ✓ bugs are fixed faster (but … harder bugs propagated to later releases) ✓ amount of pre- & post-release bugs ± the same ✓ the program crashes earlier (perhaps due to recent features) 3.6 4.0 5.0 7.0 8.0 9.0 Rapid Release Cycle rs and assigned for fixing. When he typically submits a patch to the patch code is integrated into on the corresponding channel and er channels for release. Bugs that nd hence miss a scheduled release release’s channel. TUDY DESIGN earch questions: ease cycle affect the erence in the number control for the time lease dates. However, tly lower for versions les, i.e., failures seem cycle affect the fixing ter for versions devel- e cycle affect software d release model are .e., the proportion of ersions that possibly 5.0 NIGHTLY 6.0 NIGHTLY 7.0 NIGHTLY 8.0 NIGHTLY 5.0 AURORA 6.0 AURORA 7.0 AURORA 5.0 BETA 6.0 BETA 5.0 MAIN New Feature Development 6 Weeks 6 Weeks 6 Weeks 6 Weeks Figure 2. Development and Release Process of Mozilla Firefox major release was made. Figure 1(b) shows the release dates of the minor versions of Firefox. With the advent of shorter release cycles in March 2011, new features need to be tested and delivered to users faster. To achieve this goal, Firefox changed its development pro- cess. First, versions are no longer supported in parallel, i.e.,
  • 17. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? - V-Model - Fit tables (Acceptance Testing) - Flipping the V - 4 Quadrants • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students 17
  • 18. "Next Level" Test Automation (VST Keynote) V-Model 18 Requirements Acceptance Tests Design Integration Tests Coding Unit Tests Architecture System Tests Requirements Architecture Design Coding Testing Test Design Test Execution Acceptance Test Cases System Test Cases Integration Test Cases Unit Test Cases Unit Tests Integration Tests System Tests Acceptance Tests Integration hell?
  • 19. "Next Level" Test Automation (VST Keynote) Fit Tables 19 Browse Music Play Music Browse Music start eg.music.browser enter library check total songs 37 Browse Music enter select 1 check title Akila check artist Toure Kunda enter select 2 check title American Tango check artist Weather Report check album Mysterious Traveller check year 1974 Example: Acceptance Test Cases http://fit.c2.com Play Music start eg.music.Realtime press play check status loading pause 2 check status playing
  • 20. "Next Level" Test Automation (VST Keynote) Test Case Management 20 Smoke Test
  • 21. "Next Level" Test Automation (VST Keynote) Scrum — Feedback Loop 21 Product Backlog Sprint Backlog Sprint Execution Working Increment of Product 24h Sprint Planning Sprint Review Sprint Retrospective
  • 22. "Next Level" Test Automation (VST Keynote) Behaviour Driven (User Stories) 22 As a <user role> I want to <goal> so that <benefit>. • … • … • … As a clerk I want to calculate stampage so that goods get shipped fast. • Verify with nearby address • Verify with overseas address • Verify with parcels <= 1kg • Verify with fragile parcel Template Example Conditions of Satisfaction
  • 23. "Next Level" Test Automation (VST Keynote) Flipping the V 23 Acceptance Tests (GUI Tests) System Tests Integration Tests Unit Tests 70% 20% 10% 10% 20% 70% Test Automation
  • 24. "Next Level" Test Automation (VST Keynote) Flipping the V in Practice 24 Acceptance Tests (GUI Tests) System Tests Integration Tests Unit Tests
  • 25. SupportingTeam Functional Tests Examples Story Tests Prototypes Simulation Exploratory Testing Scenarios Usability Testing Acceptance Testing Alpha / Beta Unit Tests Integration Tests Performance Testing Load Testing Security Testing “ility” Testing Technology Facing Business Facing CritiqueProduct Automated & Manual Manual Automated Tools
  • 27. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? - Coverage (Control flow, Statement, Branch, Path) - Tests vs. Faults * Reach, Infect, Propagate, Reveal - Mutation Analysis - Mutation Operators - Case studies * Scaling up: the cloud • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students 27
  • 28. How good are your tests? version control build developer tests deploy scenario tests deploy to production measure & validate
  • 29.
  • 30.
  • 31.
  • 34. "Next Level" Test Automation (VST Keynote) Code Coverage 34 Test System Under Test (SUT) Code is instrumented Test executes the code Instrumented code gets executed The traces are stored: • When statement is executed - CFG (Control Flow Graph) • When data is changed - DFG (Data Flow Graph)
  • 35. import org.junit.Test; import static org.junit.Assert.assertEquals; public class TestEmployeeDetails { EmpBusinessLogic empBusinessLogic = new EmpBusinessLogic(); EmployeeDetails employee = new EmployeeDetails(); //happy day scenario for calculation of appraisal and salary @Test public void testCalculateAppriasal() { employee.setName("Rajeev"); employee.setAge(25); employee.setMonthlySalary(8000); double appraisal = empBusinessLogic.calculateAppraisal(employee); double salary = empBusinessLogic.calculateYearlySalary(employee); } } assertionless test
  • 36. "Next Level" Test Automation (VST Keynote) The RIPR Model 36 Reach Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43(4):1-1 · January 2016  Infect Propagate Reveal
  • 37. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 38. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 39. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = null; y = 5
  • 40. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 41. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = [2,3,5]; y = 3
  • 42. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 43. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = [2,3,5]; y = 25
  • 44. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 45. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; }
  • 46. /** * Find last index of element * @param x array to search * @param y element to look for * @return last index of y in x, if absent -1 * @throws NullPointerException if x is null */ public static int findLast(int [] x, int y) { for (int i=x.length-1; i>0; i--) if (x[i] == y) return i; return -1; } x = [2,3,5]; y = 2
  • 47. "Next Level" Test Automation (VST Keynote) Mutation Testing 47 State of the Art State of the Practice Number of mutation testing publications per year
  • 48. int method(int v1, int v2) { if (v1 <v2) return 1; return -1; } int method(int v1, int v2) { if (v1 >=v2) return 1; return -1; }
  • 49. Operator Description Example Before After CBM Mutates the boundary conditions a > b a >= b IM Mutates increment operators a++ a−− INM Inverts negation operator −a a MM Mutates arithmetic & logical operators a & b a | b NCM Negates a conditional operator a == b a != b RVM Mutates the return value of a function return true return false VMCM Removes a void method call voidCall(x) – Competent Programmer Hypothesis (Program is close to correct) Coupling Effect (Test suites capable of detecting simple errors
 will also detect complex errors)
  • 50. "Next Level" Test Automation (VST Keynote) Industrial Case Study 50 • 83K lines of code • Complicated structure • Lots of legacy code • Lots of black-box tests Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”. Software Tools for Technology Transfer
  • 52. "Next Level" Test Automation (VST Keynote) Industrial Case 52 Unit tests only ! Segmentation Percentage 020406080100 Mutation Coverage Branch Coverage
  • 53. CI D evelop Build Test W ay too slow We witnessed 48 hours of mutation testing time on a test suite comprising 272 unit tests and 5,258 lines of test code for testing a project with 48,873 lines of production code. Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018
  • 54. Master 1) Initial test Build 2) ∀ files to mutate: queue file names 3a) Generate mutants 4a) Execute mutants 3b) Store mutants 4b) Store results 3c) Queue mutant references 5) Process results
  • 55. 0 1h 2h 3h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers 0 12h 1d 1d 12h 2d 2d 12h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers
  • 57. Presence of defect + reach the defect + infect the program state + observable on output coverage mutants Mutation Testing = Actionable !
  • 58. "Next Level" Test Automation (VST Keynote) Mutation Testing @ google 58 version control build developer tests deploy scenario tests deploy to production measure & validate DevOps TriCoder CodeCritique
  • 59. "Next Level" Test Automation (VST Keynote) Mutation Testing @ google 59 MutationTesting Reported benefits • stronger tests, • more effective debugging, • prevention of bugs, • improved code quality. An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions, Goran Petrovic ́ Marko Ivankovic, Bob Kurtz Paul Ammann, René Just. ICST Proceedings
  • 60. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline - Test Automation Maturity Model (TAIM) - (Spectrum Based) Fault Localisation - Test Amplification - Bug Reports - Stack Overflow • Test Research: Tips for PhD Students 60
  • 62. © TMMi © TPI next
  • 63. Henri Heiskanen, Mika Maunumaa, and Mika Katara. A test process improvement model for automated test generation. In Oscar Dieste, Andreas Jedlitschka, and Natalia Juristo, editors, Product-Focused Software Process Improvement: 13th International Conference, PROFES 2012, Madrid, Spain, June 13-15, 2012 Proceedings, pages 17--31, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. Sigrid Eldh, Kenneth Andersson, Andreas Ermedahl, and Kristian Wiklund. Towards a test automation improvement model (TAIM). In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW '14, pages 337--342, Washington, DC, USA, 2014. IEEE Computer Society. Ana Paula C. C. Furtado, Silvio R. L. Meira, and Marcos Wanderley Gomes. Towards a maturity model in software testing automation. In Proceedings of ICSEA 2014 : The Ninth International Conference on Software Engineering Advances, ICSEA2014, pages 282 - 285. IARIA, 2014. Sigrid Eldh. Test Automation Improvement Model — TAIM 2.0. In Proceedinsg NEXTA 2020 (Workshop on Next Level Test Automation) Test Process Improvement  for Automated Test  Generation  Version: 1.01.51 (2010‐05‐06)  [Draft]  TPI is an evaluation framework to evaluate thematurity of a software testing process. We have modified the original TPI framework to evaluate test processes that areusing Automated Test Generation methodsand tools. Henri Heiskanen, Mika Maunumaa, Mika Katara  10.2.2010  Tampere University of Technology, Department of Software Systems 
  • 64.
  • 66. Spectrum Based Fault Localisation wastedeffort 7 46 56 122 48 2 104 124 74 6 109 9 83 68 116 90 86 27 32 97 128 16 115 81 52 57 60 42 62 63 69 75 23 26 88 65 77 84 73 1 133 51 113 76 117 53 61 38 34 105 29 72 78 96 43 103 33 4 87 49 30 85 15 112 25 118 125 82 20 110 132 39 45 89 71 10 114 5 24 64 35 54 40 3 127 70 28 120 98 121 37 19 100 11 99 108 14 17 131 50 95 66 58 41 80 92 93 47 44 91 67 8 126 31 36 79 55 13 21 22 106 119 123 130 102 111 129 12 18 101 94 107 59 7 46 56 122 48 2 104 124 74 6 109 9 83 68 116 90 86 27 32 97 128 16 115 81 52 57 60 42 62 63 69 75 23 26 88 65 77 84 73 1 133 51 113 76 117 53 61 38 34 105 29 72 78 96 43 103 33 4 87 49 30 85 15 112 25 118 125 82 20 110 132 39 45 89 71 10 114 5 24 64 35 54 40 3 127 70 28 120 98 121 37 19 100 11 99 108 14 17 131 50 95 66 58 41 80 92 93 47 44 91 67 8 126 31 36 79 55 13 21 22 106 119 123 130 102 111 129 12 18 101 94 107 59 11513515517519511151140116511901 11513515517519511151140116511901 Patterned Spectrum Analysis Raw Spectrum Analysis wastedeffort 3 15 2 8 9 23 1 14 17 5 18 16 4 7 26 21 25 22 6 13 24 10 12 19 20 27 3 15 2 8 9 23 1 14 17 5 18 16 4 7 26 21 25 22 6 13 24 10 12 19 20 27 151101151201251301351401451 151101151201251301351401451 Patterned Spectrum Analysis Raw Spectrum Analysis Artificial Intelligence Inside
  • 68. "Next Level" Test Automation (VST Keynote) Test Amplification 68 Benjamin Danglot, Oscar Vera-Pérez, Benoit Baudry, Martin Monperrus. Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. Empirical Software Engineering, Springer Verlag, 2019, pp.1-35. 10.1007/s10664-019-09692-y . Mehrdad Abdi, Henrique Rocha and Serge Demeyer. Test Amplification in the Pharo Smalltalk Ecosystem. Proceedings IWST 2019 (International Workshop on Smalltalk Technologies) input generation + assertion generation testWithdraw |b| b := SmallBank new.
 b deposit: 100.
 self assert: b balance equals: 100. b withdraw: 30.
 self assert: b balance equals: 70 Genetic Algorithms Inside testWithdraw_12 | b | b := SmallBank new. b deposit: 100. b withdraw: SmallInteger maxVal. self assert: b balance equals: 100
  • 69. Description text Mining Stack Traces Link to source code Product/Component Specific vocabulary Suggestions? Artificial Intelligence Inside
  • 70. Question Cases Precision Recall Who should fix this bug? Eclipse, Firefox, gcc eclipse: 57% firefox: 64% gcc: 6% — How long will it take to fix this bug? (*) JBoss depends on the component many similar reports: off by one hour few similar reports: off by 7 hours What is the severity of this bug? (**) Mozilla, Eclipse, Gnome mozilla, eclipse:67% - 73% gnome: 75%-82% mozilla, eclipse:50% - 75% gnome: 68%-84% Promising results but … • how much training is needed? (cross-project training?) • how reliable is the data? (estimates, severity, assigned-to) • does this generalise? (on industrial scale?) replication is needed (*) In CSMR2012 Proceedings Who should fix this bug? Eclipse, Firefox, gcc eclipse: 57% firefox: 64% gcc: 6% — Irrelevant for Practitioners (**) In CSMR2011; MSR 2010 Proceedings Artificial Intelligence Inside
  • 71. "Next Level" Test Automation (VST Keynote) Q&A support 71
  • 73. "Next Level" Test Automation (VST Keynote) Plan • Context • Test Strategy: When Should I Test? • Test Quality: How Good Are Your Tests? • Test Analytics: Exploiting the DevOps pipeline. • Test Research: Tips for PhD Students - Descriptive Statistics - Unit Tests vs. Integration Tests 73
  • 74. "Next Level" Test Automation (VST Keynote) Descriptive Statistics 74 Semi-automatic Test Case Expansion for Mutation Testing ZhongXiLu,StenVercammen,andSergeDemeyer—UniversityofAntwerp,Belgium . . . . . . . . . . . . . . . . . . . . . . 1 An Early Investigation of Unit Testing Practices of Component-Based Software Systems Georg Buchgeher, Stefan Fischer, Michael Moser, and Josef Pichler — Software Competence Center Hagenberg, Austria; University ofAppliedSciencesUpperAustria,Austria ............................................ 12
  • 75. "Next Level" Test Automation (VST Keynote) Test Code Quality 75 At minimum include some code coverage measurement + assertion density.
  • 76. "Next Level" Test Automation (VST Keynote) Unit Tests vs. Integration Tests (1/2) 76 Further analysis confirmed that it is worthwhile to treat unit tests and integration tests differently: we discovered that unit tests cause more breaking builds, that fixing the defects exposed by unit tests takes longer and implies more coordination between team members.
  • 77. "Next Level" Test Automation (VST Keynote) Unit Tests vs. Integration Tests (2/2) 77 Hence, if we apply the IEEE definition to Java a unit test is a test that tests only units from within one package (i.e., related units). On the other hand, an integration test tests units from more than one package.