This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master program, of the Vrije Universiteit Amsterdam. http://www.ivanomalavolta.com

1. ‹#› Het begint met een idee
Empirical software engineering
Ivano Malavolta

2. Vrije Universiteit Amsterdam
Empirical software engineering
- definition
- dimensions
- types of studies
2 Ivano Malavolta / S2 group / Empirical software engineering
Roadmap

3. Vrije Universiteit Amsterdam
Scientific use of quantitative and qualitative data to
- understand and
- improve
software products and software development processes
[Victor Basili]
Data is central to address any research question
Issues related to validity addressed continuously
3 Ivano Malavolta / S2 group / Empirical software engineering
Empirical software engineering

4. Vrije Universiteit Amsterdam
4 Ivano Malavolta / S2 group / Empirical software engineering
Intuition
It is an application of the scientific method
○ ask yourself a question
○ observe a phenomenon
○ formulate an hypothesis
○ build a prediction
○ setup an experiment
○ perform analysis on your results

5. Vrije Universiteit Amsterdam
Anecdotal evidence or “common-sense” often not good enough
- Anecdotes often insufficient to support decisions in the industry
- Practitioners need better advice on how and when to use methodologies
Evidence important for successful technology transfer
- Systematic gathering of evidence
- Wide dissemination of evidence
5 Ivano Malavolta / S2 group / Empirical software engineering
Why empirical studies?

6. Vrije Universiteit Amsterdam
● Tested on a single system (lack of generalization)
● Data analysis not statistical
● No indications on how many runs were performed, error rate,
etc.
6 Ivano Malavolta / S2 group / Empirical software engineering
How NOT to do experiments

7. Vrije Universiteit Amsterdam
7 Ivano Malavolta / S2 group / Empirical software engineering
How NOT to do experiments

8. Vrije Universiteit Amsterdam
8
How NOT to do experiments
IE9
Chrome
FireFox
Opera
Safari

9. Vrije Universiteit Amsterdam
9 Ivano Malavolta / S2 group / Empirical software engineering

10. Vrije Universiteit Amsterdam
“In the lab” versus “in the wild” studies
Qualitative versus quantitative studies
Primary versus secondary studies
10 Ivano Malavolta / S2 group / Empirical software engineering
Dimensions of empirical studies

11. Vrije Universiteit Amsterdam
Common “in the lab” methods
– Controlled experiments
– Literature reviews
– Simulations
Common “in the wild” methods
– Quasi-experiments
– Case studies
– Survey research
– Ethnographies
– Action research
11 Ivano Malavolta / S2 group / Empirical software engineering
“In the lab” VS “in the wild” studies

12. Vrije Universiteit Amsterdam
12 Ivano Malavolta / S2 group / Empirical software engineering
Examples

13. Vrije Universiteit Amsterdam
Qualitative research
studying objects in their natural setting and letting the findings emerge from
the observations
– inductive process (usually discovers the “why”)
– the subject is the person
Quantitative research
quantifying a relationship or to compare two or more groups with the aim to
identify a cause-effect relationship
– fixed implied factors
– focus on quantitative data → promotes comparison and statistical
analyses
13 Ivano Malavolta / S2 group / Empirical software engineering
Qualitative VS quantitative studies

14. Vrije Universiteit Amsterdam
14 Ivano Malavolta / S2 group / Empirical software engineering
Example
They are
complementary

15. Vrije Universiteit Amsterdam
Primary studies
empirical studies in which we directly make measurements or observations
about the objects of interest, whether by surveys, experiments, case studies,
etc.
Secondary studies
empirical studies that do not generate any data from direct measurements,
but:
– analyze a set of primary studies
– usually seek to aggregate the results from primary studies in order to
provide stronger forms of evidence about a phenomenon
15 Ivano Malavolta / S2 group / Empirical software engineering
Primary VS secondary studies

16. Vrije Universiteit Amsterdam
16 Ivano Malavolta / S2 group / Empirical software engineering
Examples

17. Vrije Universiteit Amsterdam
17 Ivano Malavolta / S2 group / Empirical software engineering
… and what about this?

18. Vrije Universiteit Amsterdam
Survey
Case study
Experiment
18 Ivano Malavolta / S2 group / Empirical software engineering
Types of empirical studies

19. Vrije Universiteit Amsterdam
Def: a system for collecting information from or about people to
describe, compare or explain their knowledge, attitudes and
behavior
Often an investigation performed in retrospect
Interviews and questionnaires are the primary means of
gathering qualitative or quantitative data
Surveys are done by taking a sample which is representative from
the population to be studied
19 Ivano Malavolta / S2 group / Empirical software engineering
Survey

20. Vrije Universiteit Amsterdam
20 Ivano Malavolta / S2 group / Empirical software engineering
Example

21. Vrije Universiteit Amsterdam
1. Architects identification
– Identify preliminary set of ALs
– Systematic search
2. Planning the survey
3. Designing the survey
4. Analyzing the Data
– vertical analysis (and coding) + horizontal analysis
21 Ivano Malavolta / S2 group / Empirical software engineering
Example

22. Vrije Universiteit Amsterdam
Def: an empirical enquiry to investigate one instance (or a small
number of instances) of a contemporary software engineering
phenomenon within its real-life context, especially when the
boundary between phenomenon and context cannot be clearly
specified
Observational study
Data collected to track a specific attribute or establishing
relationships between different attributes
Multivariate statistical analysis is often applied
22 Ivano Malavolta / S2 group / Empirical software engineering
Case study

23. Vrije Universiteit Amsterdam
23 Ivano Malavolta / S2 group / Empirical software engineering
Example

24. Vrije Universiteit Amsterdam
Def: an empirical enquiry that manipulates one factor or variable
of the studied setting
1. Identify and understand the variables that play a role in software
development, and the connections between variables
2. Learn cause-effect relationships between the development process
and the obtained products
3. Establish laws and theories about software construction that explain
development behaviour
24
Experiment

25. Vrije Universiteit Amsterdam
Experiment
Models key characteristics of a reality in a controlled
environment and manipulating them iteratively to investigate the
impact of such variations and get a better understanding of a
phenomenon
Laboratory
Simplified and controllable reality where the phenomenon under
study can be manipulated
CONTROL - CONTROL - CONTROL
25 Ivano Malavolta / S2 group / Empirical software engineering
A more intuitive definition...

26. Vrije Universiteit Amsterdam
26
Example

27. Vrije Universiteit Amsterdam
You know the basics of empirical software engineering now
We can make claims that can be eventually measured
Many dimensions and types of research
• we will focus on experiments
27 Ivano Malavolta / S2 group / Empirical software engineering
What this lecture means to you?

28. Vrije Universiteit Amsterdam
28 Ivano Malavolta / S2 group / Empirical software engineering
Readings
Chapters 1 and 2
+ the MobileSoft_2017 article on Canvas