The document discusses the goal-question-metric (GQM) approach for defining research goals in software engineering experiments, emphasizing the importance of scoping and quantifying goals effectively. It provides a framework to analyze goals through questions and metrics, illustrated by examples related to image encoding algorithms and Android app success. The lecture concludes with guidelines for defining experiment goals and preparing for assignments.

1. 1 Het begint met een idee
Experiment scoping
Ivano Malavolta

2. Vrije Universiteit Amsterdam
Experiment goal definition
The Goal-Question-Metric approach
A final Example
2 Ivano Malavolta / S2 group / Empirical software engineering
Roadmap

3. Vrije Universiteit Amsterdam
3 Ivano Malavolta / S2 group / Empirical software engineering
Experiment scoping
[1] V.R. Basili, "Software Modeling and Measurement: The Goal Question Metric Paradigm," Computer
Science Technical Report Series, CS-TR-2956 (UMIACS-TR-92-96), University of Maryland, College Park,
MD, September 1992.
● Research approach: Goal-Question-Metric [1]
● Define the goal of the study

4. Vrije Universiteit Amsterdam
4 Ivano Malavolta / S2 group / Empirical software engineering
Goal GoalConceptual level
Operational level
Quantitative level
Question Question Question
Metric Metric Metric Metric
Overview

5. Vrije Universiteit Amsterdam
5 Ivano Malavolta / S2 group / Empirical software engineering
Experiment goal definition

6. Vrije Universiteit Amsterdam
6 Ivano Malavolta / S2 group / Empirical software engineering
How to define a goal?
Analyze Object(s) of study
for the purpose of Purpose
with respect to their Quality focus
from the point of view of Perspective
in the context of Context

7. Vrije Universiteit Amsterdam
7 Ivano Malavolta / S2 group / Empirical software engineering
Goal definition framework
Object of
study
Purpose Quality focus Perspective Context
Product
Process
Model
Metric
Theory
Characterize
Monitor
Evaluate
Predict
Control
Change
Effectiveness
Cost
Reliability
Maintainability
Portability
Performance
Understandability
Developer
Maintainer
Project
manager
Corporate
manager
Customer
User
Researcher
Environmental
conditions

8. Vrije Universiteit Amsterdam
8 Ivano Malavolta / S2 group / Empirical software engineering
Image encoding example - goal
Analyze
for the purpose of
with respect to their
from the point of view of
in the context of

9. Vrije Universiteit Amsterdam
9 Ivano Malavolta / S2 group / Empirical software engineering
Image encoding example - goal
Analyze Encoding algorithms
for the purpose of Evaluation
with respect to their Energy Efficiency
from the point of view of Software Developer
in the context of
Mobile Software
Applications

10. Vrije Universiteit Amsterdam
10 Ivano Malavolta / S2 group / Empirical software engineering
Image encoding example - goal
"Analyze image encoding algorithms for the purpose
of evaluation with respect to their energy efficiency
from the point of view of a software developer in the
context of mobile software applications".

11. Vrije Universiteit Amsterdam
11 Ivano Malavolta / S2 group / Empirical software engineering
From goals to questions
● Questions are at the operational level
● Answering a question should tell you if
the goal has been attained
● Questions should be quantifiable

12. Vrije Universiteit Amsterdam
12 Ivano Malavolta / S2 group / Empirical software engineering
Question examples
Analyze Encoding algorithms
for the purpose of Evaluation
with respect to their Energy Efficiency
from the point of view of Software Developer
in the context of
Mobile Software
Applications
● Does the energy consumption of a mobile
device vary, when using different image
encoding algorithms?
● Which encoding algorithm is more
efficient?
● Which algorithm provides the best
quality of the image?

13. Vrije Universiteit Amsterdam
13 Ivano Malavolta / S2 group / Empirical software engineering
Other examples...
Analyze Encoding algorithms
for the purpose of Evaluation
with respect to their Energy Efficiency
from the point of view of Software Developer
in the context of
Mobile Software
Applications
● Which algorithm is most used in Android
applications?
● What is the effort of changing algorithm
implementation?
● How does the encoding algorithm affect
energy consumption?

14. Vrije Universiteit Amsterdam
14 Ivano Malavolta / S2 group / Empirical software engineering
From questions to metrics
● Metrics are at the quantitative level
● Metrics can be objective (e.g. energy consumption) or
subjective (e.g. readability of code)

15. Vrije Universiteit Amsterdam
15 Ivano Malavolta / S2 group / Empirical software engineering
Examples of metrics
● Does the energy consumption of a mobile device vary, when using different
encoding algorithms?
Metric: Energy Consumption (Joule or Watthour)
● What is the efficiency of the encoding algorithm?
Metric: Number of rendered pixels over time (pixels/s)
● What is the effort of changing algorithm implementation?
Metric: Effort (person/hours, money)

16. Vrije Universiteit Amsterdam
16 Ivano Malavolta / S2 group / Empirical software engineering
A final example

17. Vrije Universiteit Amsterdam
17 Ivano Malavolta / S2 group / Empirical software engineering
Goal
Objects of study: Android apps
Purpose: to understand how the APIs fault- and
change-proneness could have impacted the apps’ success
Quality focus: success of those apps in terms of ratings
expressed by users on the market
Perspective: app developers’
Context: 7,097 free apps from the Google Play Market

18. Vrije Universiteit Amsterdam
18 Ivano Malavolta / S2 group / Empirical software engineering
Questions
Q1: Does the fault-proneness of APIs affect the success of
Android Apps?
Q2: Does the change-proneness of APIs affect the success of
Android Apps?

19. Vrije Universiteit Amsterdam
19 Ivano Malavolta / S2 group / Empirical software engineering
Metrics
Main goal
Q1 (fault-proneness) Q2 (change-proneness)
Average rating in
the Google Play
store
#bugs fixed in
the APIs used
by the app
#changes in the
signatures of the
methods used by
the app

20. Vrije Universiteit Amsterdam
You know how to:
● define the goal of an experiment
● precisely scope an experiment
○ … and have an hint about how to plan it
● You are able to work on assignment 1:
○ Describe your goal using the provided template
○ Develop a GQM for your experiment
○ Fill in Sections 1 and 2 of the report
20 Ivano Malavolta / S2 group / Empirical software engineering
What this lecture means to you?

21. Vrije Universiteit Amsterdam
21 Ivano Malavolta / S2 group / Empirical software engineering
Readings
Chapter 7
+ papers with examples (Koziolek, Basili) on Blackboard

22. Vrije Universiteit Amsterdam
22 Ivano Malavolta / S2 group / Empirical software engineering
Some contents of this lecture extracted from:
● Giuseppe Procaccianti’s lectures at VU
Acknowledgements