The document outlines the principles and process of conducting experiments in empirical software engineering, emphasizing the importance of clearly stating hypotheses and scoping experiments. It defines key terminology such as dependent and independent variables, factors, treatments, subjects, and objects, while also detailing the various stages of the experimental process from planning to data analysis and result presentation. The content serves as a guide for understanding experimentation methodology and highlights the need for careful execution and documentation.

1. ‹#› Het begint met een idee
The experimental process
Ivano Malavolta

2. Vrije Universiteit Amsterdam
Experiment principles
Terminology
The experimental process
2 Ivano Malavolta / S2 group / Empirical software engineering
Roadmap

3. Vrije Universiteit Amsterdam
3 Ivano Malavolta / S2 group / Empirical software engineering
Experiment principles

4. Vrije Universiteit Amsterdam
4 Ivano Malavolta / S2 group / Empirical software engineering
Running example
Image encoding: which algorithm should I
use to optimize battery usage?
○ PNG
○ JPEG
○ …

5. Vrije Universiteit Amsterdam
5 Ivano Malavolta / S2 group / Empirical software engineering
from the previous lecture...
MAIN DRIVERS of experimentation:
● to be able to scientifically assess an idea
● generalize

6. Vrije Universiteit Amsterdam
6 Ivano Malavolta / S2 group / Empirical software engineering
Experiment principles
Theory
Observation
Cause EffectCausation
e.g. green encoding algorithm e.g. better energy efficiency
Treatment Experiment Outcome
e.g. JPEG, PNG e.g. energy consumption per image

7. Vrije Universiteit Amsterdam
7 Ivano Malavolta / S2 group / Empirical software engineering
Terminology

8. Vrije Universiteit Amsterdam
8 Ivano Malavolta / S2 group / Empirical software engineering
Terminology
● Dependent variables: quantities observed in the study
(a.k.a. response, output variables)
○ e.g. energy consumption, image quality
● Independent variables: quantities that we are able to
manipulate/control (a.k.a. input variables)
○ e.g. encoding algorithm, size of image, operating system

9. Vrije Universiteit Amsterdam
9 Ivano Malavolta / S2 group / Empirical software engineering
Terminology
● Factor: an independent variable that we deliberately
manipulate/control
○ e.g. image encoding algorithm
● Treatment: a specific value of a factor
○ e.g. JPEG, PNG

10. Vrije Universiteit Amsterdam
10 Ivano Malavolta / S2 group / Empirical software engineering
Terminology
● Subject: who applies the treatment
○ e.g. software developer, software architect, user
● Object: the receiver of the treatment
○ e.g. software application

11. Vrije Universiteit Amsterdam
11 Ivano Malavolta / S2 group / Empirical software engineering
Terminology
● Trial: combination of subject, object and treatment
○ e.g. developer John develops app X that encodes
images using the JPEG algorithm
● Experiment: a combination of several trials to observe the
effects of the treatments
○ e.g. ???

12. Vrije Universiteit Amsterdam
12 Ivano Malavolta / S2 group / Empirical software engineering
Let’s put them all together...

13. Vrije Universiteit Amsterdam
13 Ivano Malavolta / S2 group / Empirical software engineering
The experimental process

14. Vrije Universiteit Amsterdam
14 Ivano Malavolta / S2 group / Empirical software engineering
0 - It all starts with an idea
● Main idea behind the experiment
● The hypotheses must be stated clearly
○ Not formally, just clearly
Idea

15. Vrije Universiteit Amsterdam
15 Ivano Malavolta / S2 group / Empirical software engineering
1 - Experiment scoping
Experiment
scoping
● We scope the experiment by stating the
problem, objectives, and goals
● We will use the Goal-Question-Metric
(GQM) framework
Idea

16. Vrije Universiteit Amsterdam
16 Ivano Malavolta / S2 group / Empirical software engineering
2 - Experiment planning
● Define context
● Formulate hypotheses
● Identify input and output variables
● Design the study
● Instrumentation
● Analyze validity threats
Experiment
scoping
Experiment
planning
Idea

17. Vrije Universiteit Amsterdam
17 Ivano Malavolta / S2 group / Empirical software engineering
3 - Experiment operation
● Preparation
○ Guidelines, subjects training,
code instrumentation, ...
● Execution
○ aka measurements
collection
● Data Validation
Experiment
scoping
Experiment
planning
Idea
Experiment
operation

18. Vrije Universiteit Amsterdam
18 Ivano Malavolta / S2 group / Empirical software engineering
4 - Analysis & interpretation
● Understanding the data
○ descriptive statistics
○ exploratory data analysis
(e.g. box plots, scatter plots, … )
● Hypothesis testing
● Results interpretation
Experiment
scoping
Experiment
planning
Idea
Experiment
operation
Analysis &
interpretation

19. Vrije Universiteit Amsterdam
19 Ivano Malavolta / S2 group / Empirical software engineering
5 - Presentation & package
● Document the results
● Prepare replication package
● Sum up lessons learned
● Write down reflections
Experiment
scoping
Experiment
planning
Idea
Experiment
operation
Analysis &
interpretation
Presentation &
package

20. Vrije Universiteit Amsterdam
● You have the basics on experimentation principles and
terminology
▪ Theory VS observation
▪ Variables, treatments, factors, …
● Overview of the experimental process
▪ It is not a pure waterfall model
> But after operation you cannot come back!
20 Ivano Malavolta / S2 group / Empirical software engineering
What this lecture means to you?

21. Vrije Universiteit Amsterdam
21 Ivano Malavolta / S2 group / Empirical software engineering
Readings
Chapter 6