This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.infn.it/. http://www.ivanomalavolta.com

1. Ivano Malavolta
RESEARCH
in software engineering
VRIJE
UNIVERSITEIT
AMSTERDAM

3. Hello
Empirical software engineering
+ Software Architecture + MDE
+
applied to
Complex systems
Autonomous robots
Mobile-enabled systems
If you think good architecture is expensive,
try bad architecture.
... Brian Foote and Joseph Yoder

4. Course overview (from my perspective)
Module name #hours Instructor
Introduction to the course 2 Inverardi
SE principles, Eclipse IDE, MDE 8 Iovino
Research in software engineering 2 Malavolta
Collaborative software development 2 Malavolta
Software architecture 6 Malavolta
Modern development paradigms 2 Malavolta
Code generation, applications of MDE, LAB, etc. 10 Iovino
Principles of software testing 4 Bertolino
Homework 1
Christmas break
Homework 2

5. Homework 1
Tasks
• create an AADL specification describing the architecture of a chosen system
• write an Acceleo program that takes as input the AADL model and produces
an HTML5 report or other artifact
• write a report describing the performed activities
Goals
• to understand what are the advantages of SE principles like abstraction and
separation of concerns
• to concretely understand what architectural modeling means
• to be exposed to concerns and issues related to dependability
• to understand how to use MDE techniques like model transformations and
code generators
Tentative deadline
20/12/2016

6. Homework 2
Tasks
• pick a set of articles related to a chosen research theme
• carefully read them and explore the state of the art about the research
theme
• write a report about your findings
• make a presentation to the classroom
Goals
• to study a specific area of software engineering that may be of interest to
you
• to design a scientific paper
• to train yourself in reading and writing scientific papers
Tentative deadline
20/01/2017
Research themes will
be available soon

7. Roadmap
Software engineering research
Empirical strategies
Writing good research papers

8. Software engineering research

9. The application of engineering to software
We deal with software systems that are:
– large and complex
– built by teams
– exist in many versions
– last many years
– undergo changes
Programming skills are not enough
“Physicist example”
Software engineering
Programmer:
complete program
Software engineer:
software component

10. Software engineering VS
computer science
Computer Science
– Computability, algorithms and complexity, programming
languages, data structures, databases, artificial intelligence,
etc.
Software Engineering
– The APPLICATION of computer science, mathematics,
project management to build high quality software

11. My vision about research
Research
Theory Industrial projectsProgramming Experimentation
Ivano Malavolta. Research Statement. November 2013. http://goo.gl/99N5AS

12. Research objective
Key objectives
• Quality àutility + functional correctness
• Cost à development + use
• Timeliness à good-enough result, when it’s needed
Address problems that affect practical software
Real world
practical PROBLEM
Real world
practical SOLUTION
?

13. Research objective: example

14. Example

15. Research strategy
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

16. Research product: example

17. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

18. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?

19. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?
Validation task 2
Does the product
help to solve the practical problem?

20. Validation of the results: example

21. SE research process
Research
questions
Research
validation
Research
results

22. Types of research questions
FEASIBILITY
CHARACTERIZATION
METHOD/MEANS
GENERALIZATION
DISCRIMINATION
Does X exist, and what is it?
Is it possible to do X at all?
What are the characteristics of X?
What exactly do we mean by X?
What are the varieties of X, and how are
they related?
How can we do X?
What is a better way to do X?
How can we automate doing X?
Is X always true of Y?
Given X, what will Y be?
How do I decide whether X or Y?

23. LAB

24. SE research process
Research
questions
Research
results
Research
validation

25. Research results
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

26. Types of research results
QUALITATIVE &
DESCRIPTIVE
MODELS
TECHNIQUES
SYSTEM
EMPIRICAL
MODELS
ANALYTIC
MODELS
Report interesting observations
Generalize from (real-life) examples
Structure a problem area; ask good questions
Invent new ways to do some tasks, including
implementation techniques
Develop ways to select from alternatives
Embody result in a system, using the system
both for insight and as carrier of results
Develop empirical predictive models from
observed data
Develop structural models that permit formal
analysis

27. LAB

28. SE research process
Research
questions
Research
results
Research
validation

29. Research validation
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?
Validation task 2
Does the result
help to solve the practical problem?

30. Types of research validation
PERSUASION I thought hard about this, and I believe…
IMPLEMENTATION Here is a prototype of a system that…
EVALUATION Given these criteria, the object rates as…
ANALYSIS Given the facts, here are consequences…
Formal model Rigorous derivation and proof
Empirical model Data on use in controlled situation
EXPERIENCE Report on use in practice
Qualitative model Narrative
Decision criteria Comparison of systems in actual use
Empirical model Data, usually statistical, on practice

31. LAB

32. Building blocks for research
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

33. Is this a good plan?
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

34. A common good plan
Feasibility
Characterization
Can X be
done better?
Generalization
Selection
Qualitative model
Technique
Build Y
Empirical model
Analytic model
Persuasion
Implementation
Measure Y,
compare to X
Analysis
Experience
Question Result Validation

35. Is this a good plan?
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

36. A common, but bad, plan
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

37. Two other good plans
Can X be done
at all?
Characterization
Is X always
true of Y?
Selection
Qualitative model
Technique
Build a Y
that does X
Empirical model
Formally model
Y, prove X
“Look it works!”
Implementation
Check proof
Experience
Question Result Validation
Method/means Evaluation

38. LAB
Choose a research paper and try to map it into the building
blocks of SE research
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

39. “NO-NO”s for software engineering
research
• Assume that a result demonstrated fro a 10K-line system
will scale to a 500K-line system
• Expect everyone to do things “my way”
• Believe functional correctness is sufficient
• Assume the existence of a complete, consistent
specification
• Just build things without extracting enduring lessons
• Devise a solution in ignorance of how the world really
works

40. How do you trust a research then?
1. What are the problems from the real world?
– Are they “real” and widespread?
2. Are the solutions general? What are their limits?
Real world
practical PROBLEM
Real world
practical SOLUTION
?
EMPIRICAL SOFTWARE ENGINEERING

41. Empirical strategies*
*We will have an advanced course on this topic

42. Empirical software engineering
Scientific use of quantitative and qualitative data to
– understand and
– improve
software products and software development processes
Data is central to address any research question
Issues related to validity addressed continuously
[Victor Basili]

43. Intuition
• You apply the scientific method
– ask yourself a question
– observe a phenomenon
– formulate an hypothesis
– build a prediction
– setup an experiment
– perform analysis on your results

44. Example
End Users’ Perception of Hybrid Mobile Apps
in the Google Play Store
Ivano Malavolta, Stefano Ruberto
Tommaso Soru, Valerio Terragni
Ivano Malavolta
Gran Sasso Science Institute (Italy)
ivano.malavolta@gssi.infn.it
ABSTRACT
Recently, companies like IBM and Adobe and a growing
community of developers advocate hybrid mobile apps
development as a possible solution to mobile platforms
fragmentation. Hybrid mobile apps are consistent across
platforms and built on web standards.
In this study, we present an empirical investigation into mobile
hybrid apps. Our goal is to identify and analyse the traits and
distinctions of publicly available hybrid mobile apps from end
users’ perspective. The study has been conducted by mining
11,917 free apps and 3,041,315 reviews from the Google Play
Store, and analyzing them from the end users’ perception
perspective. The results of this study build an objective and
reproducible snapshot about how hybrid mobile development is
performing “in the wild” in real projects, thus establishing a base
for future methods and techniques for developing hybrid mobile
apps.
FINDINGS
• hybrid development frameworks are perceived as better
suited for data-intensive mobile apps, whereas they
perform poorly when dealing with low-level, platform-
specific features
• end users value hybrid and native apps similarly
• in some categories, end users perceive native apps better
than hybrid apps with respect to performance and the
presence of bugs
RESEARCH QUESTIONS
What is the difference between hybrid and native mobile apps
as perceived by end users?
– RQ1: What is the difference in the perceived value between
hybrid and native mobile apps?
– RQ2: What is the difference in the perceived performance
between hybrid and native mobile apps?
– RQ3: What is the difference in the perceived bugginess
between hybrid and native mobile apps?
– RQ4: What is the difference in the initial download overhead
between hybrid and native mobile apps?

45. Dimensions of empirical studies
“In the lab” versus “in the wild” studies
Qualitative versus quantitative studies
Primary versus secondary studies

46. “In the lab” versus “in the wild”
studies
Common “in the lab” methods
– Controlled experiments
– Literature reviews
– Simulations
Common “in the wild” methods
– Quasi-experiments
– Case studies
– Survey research
– Ethnographies
– Action research

47. Examples

48. Qualitative versus quantitative
studies
Qualitative research
studying objects in their natural setting and letting the
findings emerge from the observations
– inductive process
– the subject is the person
Quantitative research
quantifying a relationship with the aim to identify a cause-
effect relationship
– fixed implied factors
– focus on collected quantitative data à promotes comparison and
statistical analyses
They are
complementary

49. Primary versus secondary studies
Primary studies
empirical studies in which we directly make measurements
or observations about the objects of interest, whether by
surveys, experiments, case studies, etc.
Secondary studies
empirical studies that do not generate any data from direct
measurements, but:
– analyze a set of primary studies
– usually seek to aggregate the results from these in order to
provide stronger forms of evidence about a phenomenon

50. Examples

51. …and what about this?

52. Types of empirical studies
• Survey
• Case study
• Experiment

53. Survey
Def: a system for collecting information from or about
people to describe, compare or explain their knowledge,
attitudes and behavior
Often an investigation performed in retrospect
Interviews and questionnaires are the primary means of
gathering qualitative or quantitative data
These are done through taking a sample which is
representative from the population to be studied

54. Example: our survey on arch. languages
1. ALs Identification
– Definition of a preliminary set of ALs
– Systematic search
2. Planning the Survey
3. Designing the survey
4. Analyzing the Data
– vertical analysis (and coding) + horizontal analysis

55. Case study
Def: an empirical enquiry to investigate one instance (or a
small number of instances) of a contemporary software
engineering phenomenon within its real-life context,
especially when the boundary between phenomenon and
context cannot be clearly specified
Observational study
Data collected to track a specific attribute or establishing
relationships between different attributes
Multivariate statistical analysis is often applied

56. Example

57. Experiment
Def: an empirical enquiry that manipulates one factor or
variable of the studied setting
1. Identify and understand the variables that play a role in software
development, and the connections between variables
2. Learn cause-effect relationships between the development
process and the obtained products
3. Establish laws and best practices about software construction

58. Experiment
process

59. Example
http://dl.acm.org/citation.cfm?id=2491411.2491428

60. Examples of empirical studies in SE
• Nejati, Javad, and Aruna Balasubramanian. "An in-depth study of
mobile browser performance." Proceedings of the 25th International
Conference on World Wide Web. International World Wide Web
Conferences Steering Committee, 2016.
• Hassan, Safwat, Weiyi Shang, and Ahmed E. Hassan. "An empirical
study of emergency updates for top android mobile apps." Empirical
Software Engineering (2016): 1-42.
• Tufano, Michele, et al. "An empirical investigation into the nature of
test smells." Proceedings of the 31st IEEE/ACM International
Conference on Automated Software Engineering. ACM, 2016.
• Linares-Vásquez, Mario, et al. "Mining energy-greedy api usage
patterns in android apps: an empirical study." Proceedings of the 11th
Working Conference on Mining Software Repositories. ACM, 2014.
• Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago.
"Empirical evaluation of two best practices for energy-efficient
software development." Journal of Systems and Software 117 (2016):
185-198.

61. How to have an impact in reality?
This is called technology transfer

62. Writing good software
engineering papers

63. Let’s reconsider our SE research
process…
Research
questions
Research
results
Research
validation

64. Research questions
The program committee looks for
– a clear statement of the specific problem you solved
– the question about software development you answered
– an explanation of how the answer will help solve an important
software engineering problem
You'll devote most of your paper to describing your result,
but you should begin by explaining what question you
are answering and why the answer matters
Research
questions

65. Research results
Explain precisely
– what you have contributed to the catalogue of software
engineering knowledge
– how this is useful beyond your own project

66. Why should the reader believe your
result?
Show evidence that your result
• is valid
• helps to solve the problem

67. A couple of words on the abstract of
a paper
• People judge papers by their abstracts
– usually decide whether to read the whole paper based on
abstract
• It's important for the abstract to tell the whole story
– background and context
– goals
– method
– results
– conclusion
• Often it is the only part freely accessible

68. Example of an abstract structure:
1. Two or three sentences about the current state of the art,
identifying a particular problem
2. One or two sentences about what this paper contributes to
improving the situation
3. One or two sentences about the specific result of the paper
and the main idea behind it
4. A sentence about how the result is demonstrated or
defended

69. Coming back to the initial example…
State of
the art
Overall
contribution
Specific
results
Validation
✗ ✓✓✗ ✗

70. Second try…
State of
the art
Overall
contribution
Specific
results
Validation

71. What this lecture means to you?
Software engineering != computer science
You now know how to carry on research in SE
Don’t focus on the “size” of the problem, but on
– the relevance (the practical, but also the theoretical!)
– the accuracy in the investigation (problem and evaluation
research)
When conducting empirical research, don’t make claims
you cannot eventually measure

72. Suggested readings about scientific
writing
• Shaw, M. (2003), Writing Good Software Engineering Research
Papers, in Lori A. Clarke; Laurie Dillon & Walter F. Tichy, ed., 'ICSE' ,
IEEE Computer Society, , pp. 726-737 .
• Shaw, M. (2002), 'What makes good research in software
engineering?', STTT 4 (1) , 1-7 .

73. References
http://link.springer.com/book/10.1007%2F978-3-642-29044-2

74. Contact
Ivano Malavolta |
Assistant professor
Vrije Universiteit Amsterdam
iivanoo
i.malavolta@vu.nl
www.ivanomalavolta.com

75. Acknowledgement
Some parts of this lecture are inspired by:
• Ivica Crnkovic’s lecture on software engineering research at
Mälardalen University (Sweden)
• Matthias Galster ‘s tutorial titled “Introduction to Empirical Research
Methodologies” at ECSA 2014