Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
RESEARCH 
in software engineering 
Ivano Malavolta
Roadmap 
Software engineering research 
Empirical strategies 
Writing good research papers 
Homework
Software engineering research 
Some contents of this part of lecture extracted from Ivica Crnkovic’s lecture on 
software ...
What makes good research? 
is it HARD? 
is it USEFUL? 
is it ELEGANT? 
These are all 
orthogonal and 
equally respectful 
...
My vision about research 
Research 
Theory Programming Industrial projects Experimentation 
Ivano Malavolta. Research Stat...
The basic characteristic of SE 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
?
Research objectives 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
Key objectives 
• Quality àutility as...
Research objectives: example
Example
Research strategy 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
Research setting 
IDEALIZED PROBLEM 
Res...
Research product: example
Validation of the results 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
Research setting 
IDEALIZED PROB...
Validation of the results 
Real world 
practical PROBLEM 
Real world 
Validation task 2 
Does the product 
help to solve t...
Validation of the results: example
SE research process 
Research 
questions 
Research 
validation 
Research 
results
Types of research questions 
FEASIBILITY 
CHARACTERIZATION 
METHOD/MEANS 
GENERALIZATION 
DISCRIMINATION 
Does X exist, an...
Example: software architecture 
The software architecture of a program or computing system is the 
structure or structures...
Example: SA research questions 
FEASIBILITY 
CHARACTERIZATION 
METHOD/MEANS 
GENERALIZATION 
DISCRIMINATION 
Is it possibl...
SE research process 
Research 
questions 
Research 
results 
Research 
validation
Research results 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
Research setting 
IDEALIZED PROBLEM 
Rese...
Types of research results 
QUALITATIVE & 
DESCRIPTIVE 
MODELS 
TECHNIQUES 
SYSTEM 
EMPIRICAL 
MODELS 
ANALYTIC 
MODELS 
Re...
Example: SA research results 
QUALITATIVE & 
DESCRIPTIVE 
MODELS 
TECHNIQUES 
SYSTEM 
EMPIRICAL 
MODELS 
ANALYTIC 
MODELS ...
SE research process 
Research 
questions 
Research 
results 
Research 
validation
Research validation 
Real world 
practical PROBLEM 
Real world 
Validation task 2 
Does the result 
help to solve the prac...
Types of research validation 
PERSUASION 
IMPLEMENTATION 
EVALUATION 
ANALYSIS 
Formal model 
Empirical model 
EXPERIENCE ...
Example: SA research validation 
PERSUASION 
IMPLEMENTATION 
EVALUATION 
ANALYSIS 
Formal model 
Empirical model 
EXPERIEN...
“NO-NO”s for software engineering 
research 
• Assume that a result demonstrated fro a 10K-line system 
will scale to a 50...
Building blocks for research 
Question Result Validation 
Feasibility 
Characterization 
Method/means 
Generalization 
Sel...
Is this a good plan? 
Question Result Validation 
Feasibility 
Characterization 
Method/means 
Generalization 
Selection 
...
A common good plan 
Question Result Validation 
Feasibility 
Characterization 
Can X be 
done better? 
Generalization 
Sel...
Is this a good plan? 
Question Result Validation 
Feasibility 
Characterization 
Method/means 
Generalization 
Selection 
...
A common, but bad, plan 
Question Result Validation 
Feasibility 
Characterization 
Method/means 
Generalization 
Selectio...
Two other good plans 
Question Result Validation 
Can X be done 
at all? 
Characterization 
Method/means Evaluation 
Is X ...
How do you trust a research then? 
Real world 
practical PROBLEM 
Real world 
practical SOLUTION 
? 
1. What are the probl...
*We will have a dedicated course on this topic 
Empirical strategies* 
Some contents of this part of lecture extracted fro...
Empirical software engineering 
Scientific use of quantitative and qualitative data to 
– understand and 
– improve 
softw...
Why empirical studies? 
Anecdotal evidence or “common-sense” often not good 
enough 
– Anecdotes often insufficient to sup...
Dimensions of empirical studies 
“In the lab” versus “in the wild” studies 
Qualitative versus quantitative studies 
Prima...
“In the lab” versus “in the wild” studies 
Common “in the lab” methods 
– Controlled experiments 
– Literature reviews 
– ...
Examples
Qualitative versus quantitative studies 
Qualitative research 
studying objects in their natural setting and letting the 
...
Primary versus secondary studies 
Primary studies 
empirical studies in which we directly make measurements 
or observatio...
Examples
…and what about this?
Types of empirical studies 
• Survey 
• Case study 
• Experiment
Survey 
Def: a system for collecting information from or about people 
to describe, compare or explain their knowledge, at...
Example: our survey on arch. languages 
1. ALs Identification 
– Definition of a preliminary set of ALs 
– Systematic sear...
Case study 
Def: an empirical enquiry to investigate one instance (or a 
small number of instances) of a contemporary soft...
Example
Experiment 
Def: an empirical enquiry that manipulates one factor or 
variable of the studied setting. 
1. Identify and un...
Experiment 
process
Example 
http://dl.acm.org/citation.cfm?id=2491411.2491428
What to choose?
How to have an impact in reality? 
This is called technology transfer
Writing good software 
engineering papers 
Contents of this part of lecture extracted from Ivica Crnkovic’s lecture on 
so...
Research Papers 
The basic and most important activity of the research 
• Visible results, quality stamp 
• Means for comm...
A good research paper should 
answer a number of questions 
What, precisely, was your contribution? 
– What question did y...
Let’s reconsider our SE research 
process… 
Research 
questions 
Research 
results 
Research 
validation
What do program committees 
look for? 
The program committee looks for 
Research 
questions 
– a clear statement of the sp...
Research results 
Explain precisely 
– what you have contributed to the store of software engineering 
knowledge 
– how th...
What do program committees look 
for? 
The program committee looks for 
– interesting, novel, exciting results that signif...
What do program committees look 
for? What’s new here? 
Use verbs that shows 
RESULTS, not only efforts
Philosophical moment
What has been done before? How is 
your work different or better? 
• What existing technology does your research build on?...
Explain the relation to other work 
clearly 
70
What, precisely, is the result? 
• Explain what your result is and how it works. Be concrete 
and specific. Use examples. ...
Why should the reader believe your 
result? 
Show evidence that your result is valid—that it actually helps 
to solve the ...
73
What do program committees look for? Why 
should the reader believe your result? 
• If you claim to improve on prior art, ...
A couple of words on the abstract of 
a paper 
People judge papers by their abstracts and read the abstract 
in order to d...
Example of an abstract structure: 
1. Two or three sentences about the current state of the art, 
identifying a particular...
Coming back to the initial example… 
✓✗ ✓ ✗ ✓ 
State of 
the art 
Overall 
contribution 
Specific 
results Validation
Second try… 
State of 
the art 
Overall 
contribution 
Specific 
results Validation
Homework
Homework 
ICSE 2014 features a "Future of Software Engineering" track, 
which provides delegates with a unique opportunity...
Homework 
GOALS: 
1. to have the chance to study a specific area of software 
engineering that may be of interest to you 
...
What this lecture means to you? 
You now know how to carry on research in SE 
Don’t focus on the “size” of the problem, bu...
Suggested readings 
1. Checking App Behavior Against App Descriptions (Alessandra Gorla, 
Ilaria Tavecchia, Florian Gross,...
References 
http://link.springer.com/book/10.1007%2F978-3-642-29044-2
Contact Ivano Malavolta | 
Post-doc researcher 
Gran Sasso Science Institute 
iivanoo 
ivano.malavolta@gssi.infn.it 
www.i...
RESEARCH in software engineering
RESEARCH in software engineering
RESEARCH in software engineering
RESEARCH in software engineering
RESEARCH in software engineering
Upcoming SlideShare
Loading in …5
×

RESEARCH in software engineering

4,240 views

Published on

This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.infn.it/.

http://www.ivanomalavolta.com

Published in: Technology
  • If you’re looking for a great essay service then you should check out Digitalessay.net. A friend of mine asked them to write a whole dissertation for him and he said it turned out great! Afterwards I also ordered an essay from them and I was very happy with the work I got too.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

RESEARCH in software engineering

  1. 1. RESEARCH in software engineering Ivano Malavolta
  2. 2. Roadmap Software engineering research Empirical strategies Writing good research papers Homework
  3. 3. Software engineering research Some contents of this part of lecture extracted from Ivica Crnkovic’s lecture on software engineering research at Mälardalen University (Sweden)
  4. 4. What makes good research? is it HARD? is it USEFUL? is it ELEGANT? These are all orthogonal and equally respectful Very little chances that you will excel in all three axes We are young researchers, don’t refuse usefulness, why limit your impact to dusty publications? http://goo.gl/d1YM9v
  5. 5. My vision about research Research Theory Programming Industrial projects Experimentation Ivano Malavolta. Research Statement. November 2013. http://goo.gl/99N5AS
  6. 6. The basic characteristic of SE Real world practical PROBLEM Real world practical SOLUTION ?
  7. 7. Research objectives Real world practical PROBLEM Real world practical SOLUTION Key objectives • Quality àutility as well as functional correctness • Cost à both of development and of use • Timeliness à good-enough result, when it’s needed Address problems that affect practical software
  8. 8. Research objectives: example
  9. 9. Example
  10. 10. Research strategy Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Research product (technique, method, model, system, …)
  11. 11. Research product: example
  12. 12. Validation of the results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Validation task 1 Does the product solve the idealized problem? Research product (technique, method, model, system, …)
  13. 13. Validation of the results Real world practical PROBLEM Real world Validation task 2 Does the product help to solve the practical problem? practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Validation task 1 Does the product solve the idealized problem? Research product (technique, method, model, system, …)
  14. 14. Validation of the results: example
  15. 15. SE research process Research questions Research validation Research results
  16. 16. Types of research questions FEASIBILITY CHARACTERIZATION METHOD/MEANS GENERALIZATION DISCRIMINATION Does X exist, and what is it? Is it possible to do X at all? What are the characteristics of X? What exactly do we mean by X? What are the varieties of X, and how are they related? How can we do X? What is a better way to do X? How can we automate doing X? Is X always true of Y? Given X, what will Y be? How do I decide whether X or Y?
  17. 17. Example: software architecture The software architecture of a program or computing system is the structure or structures of the system, which comprise software components, the externally visible properties of those components and the relationships among them System subsystem Subsystem component component component L. Bass, P. Clements, R. Kazman, Software Architecture In Practise, Addison Wesley, 1998
  18. 18. Example: SA research questions FEASIBILITY CHARACTERIZATION METHOD/MEANS GENERALIZATION DISCRIMINATION Is it possible to automatically generate code from an architectural specification? What are the important concepts for modeling software architectures? How can we exploit domain knowledge to improve software development? What patterns capture and explain a significant set of architectural constructs? How can a designer make tradeoff choices among architectural alternatives?
  19. 19. SE research process Research questions Research results Research validation
  20. 20. Research results Real world practical PROBLEM Real world practical SOLUTION Research setting IDEALIZED PROBLEM Research product (technique, method, model, system, …)
  21. 21. Types of research results QUALITATIVE & DESCRIPTIVE MODELS TECHNIQUES SYSTEM EMPIRICAL MODELS ANALYTIC MODELS Report interesting observations Generalize from (real-life) examples Structure a problem area; ask good questions Invent new ways to do some tasks, including implementation techniques Develop ways to select from alternatives Embody result in a system, using the system both for insight and as carrier of results Develop empirical predictive models from observed data Develop structural models that permit formal analysis
  22. 22. Example: SA research results QUALITATIVE & DESCRIPTIVE MODELS TECHNIQUES SYSTEM EMPIRICAL MODELS ANALYTIC MODELS Early architectural models Architectural patterns Domain-specific software architectures UML to support object-oriented design Architectural languages Communication metrics as indicator of impact on project complexity Formal specification of higher-level architecture for simulation
  23. 23. SE research process Research questions Research results Research validation
  24. 24. Research validation Real world practical PROBLEM Real world Validation task 2 Does the result help to solve the practical problem? practical SOLUTION Research setting IDEALIZED PROBLEM Research setting SOLUTION to IDEALIZED PROBLEM Validation task 1 Does the product solve the idealized problem? Research product (technique, method, model, system, …)
  25. 25. Types of research validation PERSUASION IMPLEMENTATION EVALUATION ANALYSIS Formal model Empirical model EXPERIENCE Qualitative model Decision criteria Empirical model I thought hard about this, and I believe… Here is a prototype of a system that… Given these criteria, the object rates as… Given the facts, here are consequences… Rigorous derivation and proof Data on use in controlled situation Report on use in practice Narrative Comparison of systems in actual use Data, usually statistical, on practice
  26. 26. Example: SA research validation PERSUASION IMPLEMENTATION EVALUATION ANALYSIS Formal model Empirical model EXPERIENCE Qualitative model Decision criteria Empirical model Early architectural models Early architectural languages Taxonomies, performance improvement Formal schedulability analysis User interface structure Architectural patterns Domain-specific architectures Communication and project complexity
  27. 27. “NO-NO”s for software engineering research • Assume that a result demonstrated fro a 10K-line system will scale to a 500K-line system • Expect everyone to do things “my way” • Believe functional correctness is sufficient • Assume the existence of a complete, consistent specification • Just build things without extracting enduring lessons • Devise a solution in ignorance of how the world really works
  28. 28. Building blocks for research Question Result Validation Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience
  29. 29. Is this a good plan? Question Result Validation Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience
  30. 30. A common good plan Question Result Validation Feasibility Characterization Can X be done better? Generalization Selection Qualitative model Technique Build Y Empirical model Analytic model Persuasion Implementation Measure Y, compare to X Analysis Experience
  31. 31. Is this a good plan? Question Result Validation Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience
  32. 32. A common, but bad, plan Question Result Validation Feasibility Characterization Method/means Generalization Selection Qualitative model Technique System Empirical model Analytic model Persuasion Implementation Evaluation Analysis Experience
  33. 33. Two other good plans Question Result Validation Can X be done at all? Characterization Method/means Evaluation Is X always true of Y? Selection Qualitative model Technique Build a Y that does X Empirical model Formally model Y, prove X “Look it works!” Implementation Check proof Experience
  34. 34. How do you trust a research then? Real world practical PROBLEM Real world practical SOLUTION ? 1. What are the problems from the real world? – Are they general? – What are the elements of them? 2. Are the solutions general? What are their limits? EMPIRICAL SOFTWARE ENGINEERING
  35. 35. *We will have a dedicated course on this topic Empirical strategies* Some contents of this part of lecture extracted from Matthias Galster ‘s tutorial titled “Introduction to Empirical Research Methodologies” at ECSA 2014
  36. 36. Empirical software engineering Scientific use of quantitative and qualitative data to – understand and – improve software products and software development processes [Victor Basili] Data is central to address any research question Issues related to validity addressed continuously
  37. 37. Why empirical studies? Anecdotal evidence or “common-sense” often not good enough – Anecdotes often insufficient to support decisions in the industry – Practitioners need better advice on how and when to use methodologies Evidence important for successful technology transfer – systematic gathering of evidence – wide dissemination of evidence
  38. 38. Dimensions of empirical studies “In the lab” versus “in the wild” studies Qualitative versus quantitative studies Primary versus secondary studies
  39. 39. “In the lab” versus “in the wild” studies Common “in the lab” methods – Controlled experiments – Literature reviews – Simulations Common “in the wild” methods – Quasi-experiments – Case studies – Survey research – Ethnographies – Action research
  40. 40. Examples
  41. 41. Qualitative versus quantitative studies Qualitative research studying objects in their natural setting and letting the findings emerge from the observations – inductive process – the subject is the person They are complementary Quantitative research quantifying a relationship or to compare two or more groups with the aim to identify a cause-effect relationship – fixed implied factors – focus on collected quantitative data à promotes comparison and statistical analyses
  42. 42. Primary versus secondary studies Primary studies empirical studies in which we directly make measurements or observations about the objects of interest, whether by surveys, experiments, case studies, etc. Secondary studies empirical studies that do not generate any data from direct measurements, but: – analyze a set of primary studies – usually seek to aggregate the results from these in order to provide stronger forms of evidence about a phenomenon
  43. 43. Examples
  44. 44. …and what about this?
  45. 45. Types of empirical studies • Survey • Case study • Experiment
  46. 46. Survey Def: a system for collecting information from or about people to describe, compare or explain their knowledge, attitudes and behavior Often an investigation performed in retrospect Interviews and questionnaires are the primary means of gathering qualitative or quantitative data These are done through taking a sample which is representative from the population to be studied
  47. 47. Example: our survey on arch. languages 1. ALs Identification – Definition of a preliminary set of ALs – Systematic search 2. Planning the Survey 3. Designing the survey 4. Analyzing the Data – vertical analysis (and coding) + horizontal analysis
  48. 48. Case study Def: an empirical enquiry to investigate one instance (or a small number of instances) of a contemporary software engineering phenomenon within its real-life context, especially when the boundary between phenomenon and context cannot be clearly specified Observational study Data collected to track a specific attribute or establishing relationships between different attributes Multivariate statistical analysis is often applied
  49. 49. Example
  50. 50. Experiment Def: an empirical enquiry that manipulates one factor or variable of the studied setting. 1. Identify and understand the variables that play a role in software development, and the connections between variables 2. Learn cause-effect relationships between the development process and the obtained products 3. Establish laws and theories about software construction that explain development behaviour
  51. 51. Experiment process
  52. 52. Example http://dl.acm.org/citation.cfm?id=2491411.2491428
  53. 53. What to choose?
  54. 54. How to have an impact in reality? This is called technology transfer
  55. 55. Writing good software engineering papers Contents of this part of lecture extracted from Ivica Crnkovic’s lecture on software engineering research papers writing at Mälardalen University (Sweden)
  56. 56. Research Papers The basic and most important activity of the research • Visible results, quality stamp • Means for communications with other researchers
  57. 57. A good research paper should answer a number of questions What, precisely, was your contribution? – What question did you answer? – Why should the reader care? – What larger question does this address? What is your new result? – What new knowledge have you contributed that the reader can use elsewhere? – What previous work (yours or someone else’s) do you build on? What do you provide a superior alternative to? – How is your result different from and better than this prior work? – What, precisely and in detail, is your new result? Why should the reader believe your result? – What standard should be used to evaluate your claim? – What concrete evidence shows that your result satisfies your claim? If you answer these questions clearly, you’ll probably communicate your result well
  58. 58. Let’s reconsider our SE research process… Research questions Research results Research validation
  59. 59. What do program committees look for? The program committee looks for Research questions – a clear statement of the specific problem you solved – the question about software development you answered – an explanation of how the answer will help solve an important software engineering problem You'll devote most of your paper to describing your result, but you should begin by explaining what question you're answering and why the answer matters
  60. 60. Research results Explain precisely – what you have contributed to the store of software engineering knowledge – how this is useful beyond your own project
  61. 61. What do program committees look for? The program committee looks for – interesting, novel, exciting results that significantly enhance our ability • to develop and maintain software • to know the quality of the software we develop • to recognize general principles about software • or to analyze properties of software You should explain your result in such a way that someone else could use your ideas
  62. 62. What do program committees look for? What’s new here? Use verbs that shows RESULTS, not only efforts
  63. 63. Philosophical moment
  64. 64. What has been done before? How is your work different or better? • What existing technology does your research build on? • What existing technology or prior research does your research provide a superior alternative to? • What’s new here compared to your own previous work? • What alternatives have other researchers pursued? • How is your work different or better?
  65. 65. Explain the relation to other work clearly 70
  66. 66. What, precisely, is the result? • Explain what your result is and how it works. Be concrete and specific. Use examples. – Example: system implementation • If the implementation demonstrates an implementation technique, how does it help the reader use the technique in another setting? • If the implementation demonstrates a capability or performance improvement, what concrete evidence does it offer to support the claim? • If the system is itself the result, in what way is it a contribution to knowledge? Does it, for example, show you can do something that no one has done before?
  67. 67. Why should the reader believe your result? Show evidence that your result is valid—that it actually helps to solve the problem you set out to solve
  68. 68. 73
  69. 69. What do program committees look for? Why should the reader believe your result? • If you claim to improve on prior art, compare your result objectively to the prior art • If you used an analysis technique, follow the rules of that analysis technique • If you offer practical experience as evidence for your result, establish the effect your research has. If at all possible, compare similar situations with and without your result • If you performed a controlled experiment, explain the experimental design. What is the hypothesis? What is the treatment? What is being controlled? • If you performed an empirical study, explain what you measured, how you analyzed it, and what you concluded
  70. 70. A couple of words on the abstract of a paper People judge papers by their abstracts and read the abstract in order to decide whether to read the whole paper. It's important for the abstract to tell the whole story Don't assume, though, that simply adding a sentence about analysis or experience to your abstract is sufficient; the paper must deliver what the abstract promises
  71. 71. Example of an abstract structure: 1. Two or three sentences about the current state of the art, identifying a particular problem 2. One or two sentences about what this paper contributes to improving the situation 3. One or two sentences about the specific result of the paper and the main idea behind it 4. A sentence about how the result is demonstrated or defended
  72. 72. Coming back to the initial example… ✓✗ ✓ ✗ ✓ State of the art Overall contribution Specific results Validation
  73. 73. Second try… State of the art Overall contribution Specific results Validation
  74. 74. Homework
  75. 75. Homework ICSE 2014 features a "Future of Software Engineering" track, which provides delegates with a unique opportunity to assess the current status of software engineering and to indicate where the field is heading in the future. FOSE is an invitation-only ICSE track that is held (approx.) every 7 or more years at ICSE An international group of leading experts has been invited to report on different topics, to provide a broad and in-depth view of the evolution of the field. http://2014.icse-conferences.org/fose
  76. 76. Homework GOALS: 1. to have the chance to study a specific area of software engineering that may be of interest to you 2. to be exposed to recurrent and important problems in software engineering TASKS: 1. Pick an article from the FOSE 2014 proceedings 2. Carefully read it and analyse it in terms of: – its research domain, its evolution over time, and its future challenges – [where possible] understand which research strategies have been applied either in the paper or in the research area in general 3. give a presentation (max 25 slides) to the classroom – other post-docs and students will attend the presentations
  77. 77. What this lecture means to you? You now know how to carry on research in SE Don’t focus on the “size” of the problem, but on – the relevance (the practical, but also the theoretical!) – the accuracy in the investigation (problem and evaluation research) When conducting empirical research, don’t make claims you cannot eventually measure Finally, don’t think in black and white only – don’t divide the world in methods, analyses, case study, etc. – don’t be afraid to look also at other disciplines à we are software engineers in any case J
  78. 78. Suggested readings 1. Checking App Behavior Against App Descriptions (Alessandra Gorla, Ilaria Tavecchia, Florian Gross, Andreas Zeller), In Proceedings of the 36th International Conference on Software Engineering, ACM, 2014. 2. Linares-Vásquez, M., Bavota, G., Bernal-Cárdenas, C., Oliveto, R., Di Penta, M., and Poshyvanyk, D., "Mining Energy-Greedy API Usage Patterns in Android Apps: an Empirical Study", in Proceedings of 11th IEEE Working Conference on Mining Software Repositories (MSR'14), Hyderabad, India, May 31- June 1, 2014, pp. 2-11 3. Shaw, M. (2003), Writing Good Software Engineering Research Paper., in Lori A. Clarke; Laurie Dillon & Walter F. Tichy, ed., 'ICSE' , IEEE Computer Society, , pp. 726-737 . 4. Shaw, M. (2002), 'What makes good research in software engineering?', STTT 4 (1) , 1-7 .
  79. 79. References http://link.springer.com/book/10.1007%2F978-3-642-29044-2
  80. 80. Contact Ivano Malavolta | Post-doc researcher Gran Sasso Science Institute iivanoo ivano.malavolta@gssi.infn.it www.ivanomalavolta.com

×