Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building and Evaluating Theories 
 in Software Engineering

463 views

Published on

Talk given at a Dagstuhl seminar on research methods training for software engineering. http://dagstuhl.de/19143

Published in: Science

Building and Evaluating Theories 
 in Software Engineering

  1. 1. Daniel Méndez TUM, Germany www.mendezfe.org Building and Evaluating Theories 
 in Software Engineering @mendezfe Software Engineering Research Methods Dagstuhl, April 2019
  2. 2. Simplified view on epistemological setting Principle ways of working Methods and strategies Fundamental theories Philosophy of science
  3. 3. Simplified view on epistemological setting Principle ways of working Methods and strategies Fundamental theories Philosophy of science Epistemology Empiricism Statistics Hypothesis testing Controlled Experimentation Example!
  4. 4. Setting: Empirical Software Engineering Principle ways of working Methods and strategies Fundamental theories Philosophy of science Typically in scope of Empirical Software 
 Engineering
  5. 5. Setting: Empirical Software Engineering Principle ways of working Methods and strategies Fundamental theories Philosophy of science Theory building 
 and evaluation Methods and 
 strategies … supported by… Analogy: Theoretical and 
 Experimental Physics
  6. 6. Goals of this talk Principle ways of working Methods and strategies Fundamental theories Philosophy of science Theory building 
 and evaluation Methods and 
 strategies … supported by… Analogy: Theoretical and 
 Experimental Physics Basic understanding of • Notion of theories in our field • (Philosophy of science) context of research methods and strategies for theory building and evaluation
  7. 7. Ground rule Whenever you have questions / remarks, don’t ask , but share them with the whole group.
  8. 8. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  9. 9. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  10. 10. What is a theory? What do you think?
  11. 11. Theories (generally speaking) Examples: • Vaccinations lead to autism • Global warming is invented to harm the industry • Earth is flat • … A theory is a belief that there is a pattern in phenomena. Further examples: Twitter profiles by right-wing populists
  12. 12. Theories (generally speaking) Examples: • Vaccinations lead to autism • Global warming is invented to harm the industry • Earth is flat • … A theory is a belief that there is a pattern in phenomena. Are these theories scientific? Further examples: Twitter profiles by right-wing populists » Speculations based on imagination or opinions that often cannot be refuted 
 (i.e. logical fallacies)
  13. 13. Scientific Theories 1. Tests • Experiments, simulations, … • Replications 2. Criticism • Peer reviews / acceptance in the community • Corroborations / extensions with further theories A scientific theory is a belief that there is a pattern in phenomena while having survived 1. tests against experiences 2. criticism by critical peers
  14. 14. Scientific Theories 1. Tests • Experiments, simulations, … • Replications 2. Criticism • Peer reviews / acceptance in the community • Corroborations / extensions with further theories A scientific theory is a belief that there is a pattern in phenomena while having survived 1. tests against experiences 2. criticism by critical peers Note: Addresses so-called Demarcation Problem to distinguishing science from non- science (as per introduction by K. Popper)
  15. 15. “There is no universally agreed upon definition of the concept of an empirically-based theory [in Software Engineering], nor is there any uniform terminology for describing theories.” Source: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010. Approach by characteristics
  16. 16. Scientific Theories have… Quality criteria: • Testability • Empirical support / (high) level of confidence • Explanatory power • Usefulness to researchers and / or practitioners • … Adopted from: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010. Analytical Explanatory Predictive Explanatory & Predictive Scope Descriptions and conceptualisation, 
 including taxonomies, classifications, and ontologies
 - What is? Identification of phenomena by identifying causes, mechanisms or reasons
 - Why is? Prediction of what will happen in the future 
 - What will happen? Prediction of what will happen in the future and explanation
 - What will happen and why?
 Further information: https://goo.gl/SQQwxt A purpose:
  17. 17. Scientific Theories have… Quality criteria: • Testability • Empirical support / (high) level of confidence • Explanatory power • Usefulness to researchers and / or practitioners • … Adopted from: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010. Analytical Explanatory Predictive Explanatory & Predictive Scope Descriptions and conceptualisation, 
 including taxonomies, classifications, and ontologies
 - What is? Identification of phenomena by identifying causes, mechanisms or reasons
 - Why is? Prediction of what will happen in the future 
 - What will happen? Prediction of what will happen in the future and explanation
 - What will happen and why?
 Further information: https://goo.gl/SQQwxt A purpose: Note: Many theories in software engineering are so-called “design [science] theories”, i.e. scientific theories about artefacts in a context. [Artefact specification] X [Context assumptions] → [Effects]
  18. 18. Scientific Theories have… Quality criteria: • Testability • Empirical support / (high) level of confidence • Explanatory power • Usefulness to researchers and / or practitioners • … Adopted from: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010. Analytical Explanatory Predictive Explanatory & Predictive Scope Descriptions and conceptualisation, 
 including taxonomies, classifications, and ontologies
 - What is? Identification of phenomena by identifying causes, mechanisms or reasons
 - Why is? Prediction of what will happen in the future 
 - What will happen? Prediction of what will happen in the future and explanation
 - What will happen and why?
 Note: Laws “versus” theories A law is a purely descriptive theory about phenomena (without explanations), i.e. an analytical theory. Further information: https://goo.gl/SQQwxt A purpose: Note: Many theories in software engineering are so-called “design [science] theories”, i.e. scientific theories about artefacts in a context. [Artefact specification] X [Context assumptions] → [Effects]
  19. 19. Exemplary framework for describing theories in Software Engineering •Constructs: What are the basic elements?
 (Actors, technologies, activities, system entities, context factors) •Propositions: How do the constructs interact? •Explanations: Why are the propositions as specified? •Scope: What is the universe of discourse in which the theory is applicable? Source: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010.
  20. 20. Exemplary framework for describing theories in Software Engineering •Constructs: What are the basic elements?
 (Actors, technologies, activities, system entities, context factors) •Propositions: How do the constructs interact? •Explanations: Why are the propositions as specified? •Scope: What is the universe of discourse in which the theory is applicable? Source: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010. Example Source: Wagner, Mendez Fernandez et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, TOSEM 2018. Req Elicitation Technique Interview Scenario Prototyping Facilitated Meetings Observation Req Documentation Technique Structured req list Domain/business process model Use case model Goal model Data model Non‐functional req Textual Semi‐formal Formal Technology Req Test Alignment Approach Req review by tester Coverage by tests Acceptance criteria Test derivation from models Req Change Approach Product backlog update Change requests Trace management  Impact analysis Activity Req Elicitation Req Documentation Req Change Management Req Test Alignment P 1‐5 P 6‐13 P 14‐20 P 21‐24 Actor Req Engineer Test Engineer Req Standard Application Practice Control Tailoring Req Eng Process Standard P 25‐28 Req Standard Defintion Compliance Development Tool support Quality assurance Project management Knowledge transfer Process complexity Communication demand Willigness to change Possibility of standardisation   Req Improvement Means Continuous improvement Strengths/weaknesses Own business unit/role Req Eng Improvement  P 29-44 P 45-49 Proposition: “Structured requirements lists are documented textually in free form or textually with constraints.” Explanation: “Free-form and constraint textual requirements are sufficient for many contexts such as in agile projects where they only act as reminders for further conversations.”
  21. 21. Theories and hypotheses Empirical Approaches Theory / Theories (Tentative) Hypothesis Falsification / 
 Corroboration Theory (Pattern)
 Building Hypothesis
 Building Hypothesis • “[…] a statement that proposes a possible explanation to some phenomenon or event” (L. Given, 2008) • Grounded in theory, testable and falsifiable • Often quantified and written as a conditional statement Scientific theory • “[…] based on hypotheses tested and verified multiple times by detached researchers” (J. Bortz and N. Döring, 2003) If cause/assumption (independent variables) then (=>) consequence (dependent variables) Note: We don’t “test theories”, but their consequences via hypotheses 
 (i.e. testable propositions)
  22. 22. From real world phenomena to theories 
 (and back) - The Empirical Lifecycle Empirical Approaches Theory / Theories (Tentative) Hypothesis Falsification / 
 Corroboration Theory (Pattern)
 Building Units of Analysis Sampling Frame Sampling Hypothesis
 Building Empirical Inquiries Induction Inference of a 
 general rule 
 from a particular case/result 
 (observation) Abduction (Creative) Synthesis of an 
 explanatory case from a general rule 
 and a particular result (observation) Deduction Application of a general rule 
 to a particular case, 
 inferring a specific result Source: Mendez and Passoth. Empirical Software 
 Engineering: from Discipline to Interdiscipline, 2018.
  23. 23. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  24. 24. Example: Goal-oriented RE (“GORE”) Papers published [1]: 966 [2] Mavin, et al. Does Goal-Oriented Requirements Engineering Achieve its Goal?, 2017 [3] Mendez et al. Naming the Pain in Requirements Engineering Initiative - www.re-survey.org [1] Horkoff et al. Goal-Oriented Requirements Engineering: A Systematic Literature Map, 2016 Papers including a case study [1]: 131 Studies involving practitioners [2]: 20 Practitioners actually using GORE [3]: ~ 5%
  25. 25. Example: Goal-oriented RE (“GORE”) Papers published [1]: 966 [2] Mavin, et al. Does Goal-Oriented Requirements Engineering Achieve its Goal?, 2017 [3] Mendez et al. Naming the Pain in Requirements Engineering Initiative - www.re-survey.org [1] Horkoff et al. Goal-Oriented Requirements Engineering: A Systematic Literature Map, 2016 Papers including a case study [1]: 131 Studies involving practitioners [2]: 20 Practitioners actually using GORE [3]: ~ 5% For comparison: Icelanders believing in elves [4]: [4] https://www.nationalgeographic.com/travel/destinations/europe/iceland/believes-elves-exist-mythology/ 54%
  26. 26. “[…] judging a theory by assessing the number, faith, and vocal energy of its supporters […] basic political credo of contemporary religious maniacs” — Imre Lakatos, 1970 Current state of evidence in SE is weak * Addressing the situation in the quantum mechanics research community, an analogy
  27. 27. Current state of evidence in SE is weak Source (levels of evidence): Wohlin. An Evidence Profile for Software Engineering Research and Practice, 2013. In favour / corroboration Against / refutation Strong evidence Evidence Circumstantial evidence Third-party claim First or second party claim Strong evidence Circumstantial evidence Third-party claim Evidence First or second party claim + - In most cases, 
 we are here Available studies often… • … remain isolated • … don’t report negative results • … strengthen confidence on 
 own hopes (and don’t 
 report anything around) • … discuss little (if at all) 
 relation to existing evidence
  28. 28. SE largely dominated by conventional wisdom “Leprechauns”: Folklore turned into facts • Emerge from times where claims by authorities were treated as “facts” • Reasons manifold: – Lack of empirical awareness – Neglecting particularities 
 of practical contexts – Neglecting relation to existing evidence – No proper citations (one side the medal, over-conclusions, etc.) – Lack of data – …
  29. 29. Consequences » Practical relevance and impact? » Potential for transfer into practice and adoption?
  30. 30. The essence Theory building and theory evaluation 
 are crucial in SE research » Reason about the discipline and (e.g. social) phenomena involved » Recognise and understand limits and effects of artefacts (technologies, techniques, processes, models, etc.) in their contexts
  31. 31. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  32. 32. Recap: The Empirical Lifecycle Source: Mendez and Passoth. Empirical Software 
 Engineering: from Discipline to Interdiscipline, 2018.
  33. 33. (Empirical) methods •Each method… – …has a specific purpose – …relies on a specific data type •Purposes – Exploratory – Descriptive – Explanatory – Improving •Data Types – Qualitative – Quantitative
  34. 34. (Empirical) methods •Each method… – …has a specific purpose – …relies on a specific data type •Purposes – Exploratory – Descriptive – Explanatory – Improving •Data Types – Qualitative – Quantitative “Grounded Theory” Qualitative Data Descriptive, Exploratory, or Explanatory Example!
  35. 35. (Empirical) methods - where do they belong? Ethnographic studies, Folklore gathering Case studies /
 Field studies (Confirmatory) Formal analysis / logical reasoning Survey research Case studies /
 Field studies (Exploratory)
  36. 36. (Empirical) methods - where do they belong? Ethnographic studies, Folklore gathering Case studies /
 Field studies (Confirmatory) Formal analysis / logical reasoning Survey research Case studies /
 Field studies (Exploratory)
  37. 37. Which research methods to use 
 in which situations?
  38. 38. Empirical process: an abstract view Planning and Definition Method and Strategy Selection Design and (Method) Execution Conclusion Drawing Packaging and Reporting •Identify and outline problem (area) •Determine research objectives and questions •Select type of study and method(s) •Identify necessary environment 
 (including units of analysis) •Design and validate study protocol (and validity procedures) •Collect & analyse data following detailed processes •Interpret results •Reflect upon potential threats to validity •Package and (ideally) disclose data •Report on results in tune with audience
  39. 39. Empirical process: an abstract view Planning and Definition Method and Strategy Selection Design and (Method) Execution Conclusion Drawing Packaging and Reporting Scope of detailed empirical methods The selection of proper research 
 methods is a recognised problem 
 in the community
  40. 40. There is no such thing as a 
 universal way of scientific practice!
  41. 41. Method selection depends on many 
 non-trivial questions • What is the purpose of the study? – Exploratory? Descriptive? Explanatory? Improving? • What is the nature of the study? – Inductive? Deductive? • What is the relation to existing evidence? – Building a new theory? “Testing” existing theory? • What is the nature of the questions we ask? – What-questions? Why-questions? • What is the nature of the environment? – Controlled environments? Realistic environments? • What is the necessary sample? – Population source? – Units of analysis? Criteria for 
 selecting methods Criteria for 
 environment selection
 (and sampling) Non-exclusive andnon-sequential
  42. 42. Not trivial, but possible: By using checklists More advanced • Chapter 16 + Appendix • http://bit.ly/checklists-design_science Good starting point
  43. 43. How to achieve scientific progress? In step-wise iterations, with multiple methods 
 (aka “research programme”)
  44. 44. Progress via multi-study approaches Large-scale evaluation 5 e.g. Field Study or longitudinal study 1 2 3 4 5 Validation of new technology in artificial setting 3 e.g. Controlled Experiment Evaluation of new technology in realistic setting 4 e.g. Case Study Replication Replication Proposal new / adaptation existing technology 2 Problem analysis 1 e.g. Systematic Mapping Study
 or Survey e.g. RE Improvement 
 Approach
  45. 45. Postulate I Every study has a specific scope of validity only Source: Sjøberg, D., Dybå,T.,Anda, B., Hannay, J. Building Theories in Software Engineering, 2010.
  46. 46. Postulate II Scope of validity ≠ Degree of reality Simulation Field Study 
 Research Case Study 
 Research Survey Research Artificial environment Realistic environment Scope of validity*Controlled (lab) Experiment Replications Replications * Extremely simplified view to orient discussions
  47. 47. Postulate III Case studies and experiments complement each other in scaling up to practice Based on: Wieringa R. Empirical Research Methods for TechnologyValidation: Scaling Up to Practice, 2013. Lab credibility Street credibility Simple model Realistic case Small sample Large sample Focus of 
 case studies Focus of field studies 
 and replications Scaling up to practice Focus of 
 (lab) experiments Similarity to 
 population units Sample size
  48. 48. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  49. 49. What is the difference between qualitative and quantitative research? What do you think?
  50. 50. Warning: EmSE emerges from natural science, thus, qualitative methods are often confronted with prejudice “I prefer working with real data 
 [not with qualitative data]” — Anonymous ISERN member “In contrast [to previous qualitative studies], this study attempts to obtain more scientific evidence in the form of objective, quantitative data.” — Anonymous author “With all due respect, DO NOT make such ridiculous claims that only quantitative studies are "scientific" and “objective" There is NO more objectivity in numbers than there is in qualitative data. ” — Anonymous reviewer … response
  51. 51. The essence Qualitative studies focus primarily on the kind of evidence that will enable you to understand the meaning [and purpose, reasoning, etc] of what is going on. Quantitative studies focus primarily on the kind of evidence that will enable you to understand what is going on. Qualitative and quantitative methods have complementary purposes, strengths, and limitations in theory building.
  52. 52. Two complementary approaches (+1) •Quantitative research – Describing events and finding causes to predict similar events in the future – (Typically) focus on what, how much, or how many – (Typically) in numerical forms – (Typically) descriptive purpose •Qualitative research – Understanding meaning [and purpose, reasoning, etc.] of a phenomenon for those involved – (Typically) focus on why/meaning, and how people interpret their experiences – (Typically) in variety of non-numerical forms, like texts, diagrams, etc. – (Typically) exploratory or explanatory purpose •Mix-method research Adopted from: Da Silva. Tutorial given at the Ibero-American Conference on Software Engineering, 2018 (Bogota, Colombia)
  53. 53. Two complementary approaches (+1) Quantitative data Qualitative data “Why?”-Questions“What?”-Questions Descriptive & predictive purposes Explanatory & exploratory purposes
  54. 54. Two complementary approaches (+1) Quantitative data Qualitative data Case study research Survey research (Quasi-) Controlled 
 experiments Action 
 research Ethnographic studies “Why?”-Questions“What?”-Questions Descriptive & predictive purposes Explanatory & exploratory purposes
  55. 55. Comparison: qualitative “vs” quantitative Qualitative Research Quantitative Research Goals • Understanding, reasoning, explanations, descriptions, meaning (to subjects), discovery • Hypothesis generation (typically) • Description, control, prediction • Hypothesis testing (typically) Design characteristics • Flexible, evolving, emergent • Inductive, constant comparative • Predetermined, structured / fixed • Deductive, statistical Samples • Small, non-random (sometimes even opportunistic), purposeful, theoretical • Large(r), random, representative Data collection • Researcher often primary instrument • Interviews, observations, document analysis, … • Inanimate instruments (tests, surveys, questionnaires, etc.) Findings • Comprehensive, holistic, rich descriptions • Precise and statistical Adopted from: Da Silva. Tutorial given at the Ibero-American Conference on Software Engineering, 2018 (Bogota, Colombia) * For all, you can add a “in tendency”
  56. 56. Researchers’ necessary skills •Being a careful observer with high tolerance to ambiguity and existing belief systems •Ability to ask “proper”, context- specific questions without actively distorting results •Thinking inductively •Patience (and charm) •Comfort with observing 
 (and writing :-) Adopted from: Da Silva. Tutorial given at the Ibero-American Conference on Software Engineering, 2018 (Bogota, Colombia)
  57. 57. Challenges in conducting qualitative studies • Cases/subjects need to offer rich insights (challenging in industry contexts) —> Case selection often opportunistic • Data collection demands skills beyond statistics (e.g. social skills) • Data analysis is often a (pragmatic and) creative task. • Data disclosure particularly important (and challenging for ethical and legal reasons) • Communication and publication often rendered difficult by prejudice Source: Mendez, Wagner. Case Studies in Industry: What we have Learnt, 2016
  58. 58. Further reading: Selected Papers Good starting point Terminological demarcation and key characteristics 
 of methods Epistemological setting
  59. 59. Theory Building in Software Engineering ‣Theories in a Nutshell … where we will briefly talk about the general notion of theories ‣State of Evidence in Software Engineering … where we will see why theory building is so important to our field ‣Research Methods in Software Engineering … where we will put research methods in a larger (philosophical) picture ‣Qualitative “vs” quantitative research … where we will discuss different research approaches
  60. 60. Theory Building in Software Engineering ‣ Theories are the backbone of our discipline allowing us to move forward from paradigmatic stage of an engineering discipline to a scientific one. ‣ Every research method has its place in a larger picture. Thank you! ‣ Qualitative and quantitative research has complementary purposes, strengths, and limitations in building and evaluating theories.

×