Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Green Lab - [01 C] Empirical software engineering

1,578 views

Published on

This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx

http://www.ivanomalavolta.com

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Green Lab - [01 C] Empirical software engineering

  1. 1. 1 Het begint met een idee Empirical software engineering Ivano Malavolta
  2. 2. Vrije Universiteit Amsterdam Software engineering Empirical software engineering - definition - dimensions - types of studies 2 Ivano Malavolta / S2 group / Empirical software engineering Roadmap
  3. 3. Vrije Universiteit Amsterdam 3 Ivano Malavolta / S2 group / Empirical software engineering Software engineering
  4. 4. Vrije Universiteit Amsterdam Where is software today? 4 Ivano Malavolta / S2 group / Empirical software engineering Software today
  5. 5. Vrije Universiteit Amsterdam How “big” is software today? 5 Ivano Malavolta / S2 group / Empirical software engineering Software today http://www.informationisbeautiful.net/visualizations/million-lines-of-code http://hbr.org/2010/06/why-dinosaurs-will-keep-ruling-the-auto-industry/ar/1
  6. 6. Vrije Universiteit Amsterdam To DESIGN software - software development has to be a systematic activity QUALITY assurance - we have to verify and validate our SW in order to make it something people can rely on - we have to do it as soon as possible ABSTRACTION - the principal instrument for managing complexity 6 Ivano Malavolta / S2 group / Empirical software engineering Needs
  7. 7. Vrije Universiteit Amsterdam The application of engineering to software Field of computer science dealing with software systems that are: – large and complex – built by teams – exist in many versions – last many years – undergo changes Programming skills are not enough “Physicist example” – 7 Ivano Malavolta / S2 group / Empirical software engineering Software engineering (SE) Programmer: complete program Software engineer: software component
  8. 8. Vrije Universiteit Amsterdam 8 Ivano Malavolta / S2 group / Empirical software engineering A more precise definition of SE "Software Engineering means application of a systematic, disciplined, quantifiable approach to development, operation and maintenance of software [1]" 1. “IEEE Standard Glossary of Software Engineering Terminology,” IEEE std 610.12-1990, 1990.
  9. 9. Vrije Universiteit Amsterdam Computer science - Computability, algorithms and complexity, programming languages, data structures, databases, artificial intelligence, etc. Software engineering - The APPLICATION of computer science, mathematics, project management to build high quality software 9 Ivano Malavolta / S2 group / Empirical software engineering Software engineering VS computer science
  10. 10. Vrije Universiteit Amsterdam 10 Ivano Malavolta / S2 group / Empirical software engineering Empirical software engineering
  11. 11. Vrije Universiteit Amsterdam Scientific use of quantitative and qualitative data to - understand and - improve software products and software development processes [Victor Basili] Data is central to address any research question Issues related to validity addressed continuously 11 Ivano Malavolta / S2 group / Empirical software engineering Empirical software engineering
  12. 12. Vrije Universiteit Amsterdam 12 Ivano Malavolta / S2 group / Empirical software engineering Intuition ● It is an application of the scientific method ○ ask yourself a question ○ observe a phenomenon ○ formulate an hypothesis ○ build a prediction ○ setup an experiment ○ perform analysis on your results
  13. 13. Vrije Universiteit Amsterdam Anecdotal evidence or “common-sense” often not good enough - Anecdotes often insufficient to support decisions in the industry - Practitioners need better advice on how and when to use methodologies Evidence important for successful technology transfer - systematic gathering of evidence - wide dissemination of evidence 13 Ivano Malavolta / S2 group / Empirical software engineering Why empirical studies?
  14. 14. Vrije Universiteit Amsterdam 14 Ivano Malavolta / S2 group / Empirical software engineering
  15. 15. Vrije Universiteit Amsterdam ● Tested on a single system (lack of generalization) ● Data analysis not statistical ● No indications on how many runs were performed, error rate, etc. 15 Ivano Malavolta / S2 group / Empirical software engineering How NOT to do experiments
  16. 16. Vrije Universiteit Amsterdam 16 Ivano Malavolta / S2 group / Empirical software engineering How NOT to do experiments
  17. 17. Vrije Universiteit Amsterdam 17 How NOT to do experiments IE9 Chrome FireFox Opera Safari
  18. 18. Vrije Universiteit Amsterdam “In the lab” versus “in the wild” studies Qualitative versus quantitative studies Primary versus secondary studies 18 Ivano Malavolta / S2 group / Empirical software engineering Dimensions of empirical studies
  19. 19. Vrije Universiteit Amsterdam Common “in the lab” methods – Controlled experiments – Literature reviews – Simulations Common “in the wild” methods – Quasi-experiments – Case studies – Survey research – Ethnographies – Action research 19 Ivano Malavolta / S2 group / Empirical software engineering “In the lab” VS “in the wild” studies
  20. 20. Vrije Universiteit Amsterdam 20 Ivano Malavolta / S2 group / Empirical software engineering Examples
  21. 21. Vrije Universiteit Amsterdam Qualitative research studying objects in their natural setting and letting the findings emerge from the observations – inductive process – the subject is the person Quantitative research quantifying a relationship or to compare two or more groups with the aim to identify a cause-effect relationship – fixed implied factors – focus on quantitative data → promotes comparison and statistical analyses 21 Ivano Malavolta / S2 group / Empirical software engineering Qualitative VS quantitative studies
  22. 22. Vrije Universiteit Amsterdam 22 Ivano Malavolta / S2 group / Empirical software engineering Example They are complementary
  23. 23. Vrije Universiteit Amsterdam Primary studies empirical studies in which we directly make measurements or observations about the objects of interest, whether by surveys, experiments, case studies, etc. Secondary studies empirical studies that do not generate any data from direct measurements, but: – analyze a set of primary studies – usually seek to aggregate the results from these in order to provide stronger forms of evidence about a phenomenon 23 Ivano Malavolta / S2 group / Empirical software engineering Primary VS secondary studies
  24. 24. Vrije Universiteit Amsterdam 24 Ivano Malavolta / S2 group / Empirical software engineering Examples
  25. 25. Vrije Universiteit Amsterdam 25 Ivano Malavolta / S2 group / Empirical software engineering … and what about this?
  26. 26. Vrije Universiteit Amsterdam Survey Case study Experiment 26 Ivano Malavolta / S2 group / Empirical software engineering Types of empirical studies
  27. 27. Vrije Universiteit Amsterdam Def: a system for collecting information from or about people to describe, compare or explain their knowledge, attitudes and behavior Often an investigation performed in retrospect Interviews and questionnaires are the primary means of gathering qualitative or quantitative data Surveys are done by taking a sample which is representative from the population to be studied 27 Ivano Malavolta / S2 group / Empirical software engineering Survey
  28. 28. Vrije Universiteit Amsterdam 28 Ivano Malavolta / S2 group / Empirical software engineering Example
  29. 29. Vrije Universiteit Amsterdam 1. ADLs Identification – Identify preliminary set of ADLs – Systematic search 2. Planning the Survey 3. Designing the survey 4. Analyzing the Data – vertical analysis (and coding) + horizontal analysis 29 Ivano Malavolta / S2 group / Empirical software engineering Example
  30. 30. Vrije Universiteit Amsterdam Def: an empirical enquiry to investigate one instance (or a small number of instances) of a contemporary software engineering phenomenon within its real-life context, especially when the boundary between phenomenon and context cannot be clearly specified Observational study Data collected to track a specific attribute or establishing relationships between different attributes Multivariate statistical analysis is often applied 30 Ivano Malavolta / S2 group / Empirical software engineering Case study
  31. 31. Vrije Universiteit Amsterdam 31 Ivano Malavolta / S2 group / Empirical software engineering Example
  32. 32. Vrije Universiteit Amsterdam Def: an empirical enquiry that manipulates one factor or variable of the studied setting 1. Identify and understand the variables that play a role in software development, and the connections between variables 2. Learn cause-effect relationships between the development process and the obtained products 3. Establish laws and theories about software construction that explain development behaviour 32 Experiment
  33. 33. Vrije Universiteit Amsterdam Experiment Models key characteristics of a reality in a controlled environment and manipulating them iteratively to investigate the impact of such variations and get a better understanding of a phenomenon Laboratory Simplified and controllable reality where the phenomenon under study can be manipulated CONTROL - CONTROL - CONTROL 33 Ivano Malavolta / S2 group / Empirical software engineering A more intuitive definition...
  34. 34. Vrije Universiteit Amsterdam 34 Example
  35. 35. Vrije Universiteit Amsterdam 35 Ivano Malavolta / S2 group / Empirical software engineering What to choose?
  36. 36. Vrije Universiteit Amsterdam This is called technology transfer 36 Ivano Malavolta / S2 group / Empirical software engineering How to have an impact in reality?
  37. 37. Vrije Universiteit Amsterdam Software is pervasive Software engineering != computer science You know the basics of empirical SE now Don’t make claims you cannot eventually measure Don’t think in black and white only – don’t divide the world in methods, analyses, case study, etc. – don’t be afraid to look also at other disciplines → we are software engineers in any case 37 Ivano Malavolta / S2 group / Empirical software engineering What this lecture means to you?
  38. 38. Vrije Universiteit Amsterdam 38 Ivano Malavolta / S2 group / Empirical software engineering Readings Chapters 1, 2, 3Chapters 1 and 2 + one article among the ones mentioned in the previous slides

×