Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fundamentals and 
Technische Universität München 
Case Studies in Industry 
Daniel Méndez 
! 
! 
@mendezfe 
Technische Uni...
What is a case study? 
Empirical inquiry that investigates a contemporary phenomenon within its real-life context, 
especi...
In quest for a case study definition 
We opt for realistic environments 
Theoretical space 
(methods, models) 
Theory 
Env...
In quest for a case study definition 
We rely on a mix of qualitative & quantitative data 
Case studies 
Surveys and inter...
In quest for a case study definition 
We rely on a mix of empirical methods 
Data collection procedure 
Case studies 
Acti...
In quest for a case study definition 
Summary of some important characteristics 
! 
! 
Most* case studies: 
• are conducte...
Agenda 
1. Case study research: Why and how? 
2. Examples 
3. Challenges in case study research 
4. Time for questions
Agenda 
1.Case study research: Why and how? 
2. Examples 
3. Challenges in case study research 
4. Time for questions
Why case studies? 
Learn from real life environments 
Disclaimer 
Cases studies do not imply more 
“validity” or “trustwor...
Why case studies? 
Case studies support scaling up to practice 
Lab credibility 
Street credibility 
Focus of case studies...
What is the lifecycle of a case study? 
Case study 
design 
Preparation for 
data collection 
Data collection Data analysi...
Agenda 
1. Case study research: Why and how? 
2.Examples 
3. Challenges in case study research 
4. Time for questions
! 
! 
Example #1
Model-based process enactment 
Develop and evaluate an approach to dynamically test the conformance of 
software project (...
Results 
Problem analysis Evaluation 
Development 
(TAR) 
Case study 
research 
Interviews 
Document 
analysis 
• Moving t...
! 
! 
Example #2
Model-based requirements engineering 
Develop and evaluate a model-based requirements engineering approach for embedded re...
Results 
Problem analysis Evaluation 
Development 
(TAR) 
Case study 
research 
Interviews 
Document 
analysis 
No access ...
! 
! 
Example #3
Model-based requirements engineering (2nd try) 
Evaluate a model-based requirements engineering approach for business info...
Case study 
Assessment 
(Interview) 
TAR 
workshops 
Results 
The new model-based approach supported: 
• more flexibility ...
Conclusion: 
The new model-based approach better supports 
a flexible process with syntactically consistent results. 
What...
Agenda 
1. Case study research: Why and how? 
2. Examples 
3.Challenges in case study research 
4. Time for questions
Industrial research cooperations with 
• up to 3 years duration 
• 3-30 participants 
• multiple case studies 
We distingu...
Challenges in approaching a case study
Challenges in approaching a case study 
#1: Finding contacts 
» Integrate studies into ongoing / planned research projects...
Challenges in approaching a case study 
#2: Convincing contacts to participate 
» Find appropriate terminology (technology...
Challenges in approaching a case study 
#3: Opportunistic case and subject selection 
» Welcome to case study research! 
I...
Challenges in approaching a case study 
#4: Planning a case study 
You need to: 
• schedule meetings / workshops 
• alloca...
Challenges in approaching a case study 
#5: Uncertainty 
• What will the stakeholder characteristics be? 
• Availability 
...
Challenges in conducting a case study
Context variables (and phenomena) 
• hard to determine (or unavailable) 
• hard to measure 
» Context often hard to descri...
Challenges in conducting a case study 
#2: Communication 
Possible problems: 
• No clear communication of case study in or...
Challenges in conducting a case study 
#3: Defining the proper instruments 
Possible problems: 
• Wrong assumptions about ...
Challenges in conducting a case study 
#4: The data collection might take long… 
» Be patient 
» Never push! 
Image source...
Challenges in conducting a case study 
#5: Working with sensitive data 
» Clarify the rules: 
» what you need for the case...
Challenges in conducting a case study 
#6: (Social) skills 
Possible distortions in data collection: 
• biased researchers...
Challenges in conducting a case study 
#7: Analysing qualitative data 
Qualitative data 
• is inherently subjective 
• is ...
Challenges in conducting a case study 
#8: Moving targets 
• Moving targets are something natural 
• The difficult part: 
...
Challenges in conducting a case study 
#9: Tool support 
Unfortunately, 
still the best tools we have…
Transversal challenges
Transversal challenges 
#1: The ivory tower 
Possible problem: 
Support by management…only by management. 
» No backup in ...
Transversal challenges 
#2: Sometimes, size seems to matter… 
» „[Even if] one swallow does not make a summer, one swallow...
Transversal challenges 
#3: Drawing proper conclusions 
» Knowledge is more than statistical significance 
» When analysin...
Transversal challenges 
#4: The quest for universal truth (“generalisation”) 
1. Software engineering is about human being...
Transversal challenges 
#5: Openness and transparency 
Case study data should be disclosed 
• reliability and trustworthin...
Transversal challenges 
#6: Properly reporting the results 
Case study results need to be disseminated 
• to practice 
• t...
KEY TAKE AWAYS
My personal manifesto for case study research 
6. Be unbiased 
7. Be accurate and critical 
8. Be honest 
9. Be patient 
1...
Agenda 
1. Case study research: Why and how? 
2. Examples 
3. Challenges in case study research 
4.Time for questions
Don’t ask , ask us!
! 
! 
! 
Thank you! 
Daniel Méndez 
Daniel.Mendez@tum.de 
@mendezfe 
Further reading 
• Slides will be made available 
(an...
Upcoming SlideShare
Loading in …5
×

Case studies in industry - fundamentals and lessons learnt

1,204 views

Published on

Talk given at the International Advanced School of Empirical Software Engineering.

Published in: Science

Case studies in industry - fundamentals and lessons learnt

  1. 1. Fundamentals and Technische Universität München Case Studies in Industry Daniel Méndez ! ! @mendezfe Technische Universität München Germany ! IASESE 2014 Torino, Italy Based on material from a joint work with: Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering) Stefan Wagner (University of Stuttgart) ! Feedback from Antonio Vetrò (Technische Universität München)
  2. 2. What is a case study? Empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. [Yin 2002]
  3. 3. In quest for a case study definition We opt for realistic environments Theoretical space (methods, models) Theory Environment made irrelevant Artificial (controlled) environment Realism Reality
  4. 4. In quest for a case study definition We rely on a mix of qualitative & quantitative data Case studies Surveys and interviews Controlled experiments Quasi experiments Action research Grounded Theory Quantitative Qualitative
  5. 5. In quest for a case study definition We rely on a mix of empirical methods Data collection procedure Case studies Action research Data analysis procedure Grounded Theory Surveys and interviews * Example * *
  6. 6. In quest for a case study definition Summary of some important characteristics ! ! Most* case studies: • are conducted in socio-economic contexts • are of confirmatory nature (“Let’s see if my method is better than yours”) • rely on qualitative data (e.g. as part of interview transcripts) • rely on a mix of various research methods, often (technical) action research Case study research Setting Purpose Data type Mixed w/ •Realistic environments •Realistic models •Confirmatory •Exploratory •Qualitative •Quantitative * Unsupported claim, don’t sue me. •Action research •Interview research •…
  7. 7. Agenda 1. Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions
  8. 8. Agenda 1.Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions
  9. 9. Why case studies? Learn from real life environments Disclaimer Cases studies do not imply more “validity” or “trustwor thiness” than experiments - they complement each other! Learn from realistic (socio-economic) environments with realistic conditions: • subjects with: opinions, experiences, expectations, fears, beliefs, … • cases with: realistic models, realistic complexity, realistic decisions, … … at the cost of control (and internal validity) ! » Ideally: Explore the unpredictable (e.g. when testing a SE method)
  10. 10. Why case studies? Case studies support scaling up to practice Lab credibility Street credibility Focus of case studies Realistic model Simple model Small sample Large sample Based on: Wieringa Empirical Research Methods for Technology Validation: Scaling Up to Practice (Focus of field studies and replications) Scaling up to practice Focus of (lab) experiments
  11. 11. What is the lifecycle of a case study? Case study design Preparation for data collection Data collection Data analysis Reporting • Research objectives • Research goals • … • Schedules • Protocols • Document analyses • Interviews, Surveys • Action research • … • Descriptive statistics • Grounded theory • … • Packaging • (Publishing) Further reading
  12. 12. Agenda 1. Case study research: Why and how? 2.Examples 3. Challenges in case study research 4. Time for questions
  13. 13. ! ! Example #1
  14. 14. Model-based process enactment Develop and evaluate an approach to dynamically test the conformance of software project (instances) and software process models Problem analysis Evaluation Development (TAR*) Case study research Interviews Document analysis * Technical Action Research
  15. 15. Results Problem analysis Evaluation Development (TAR) Case study research Interviews Document analysis • Moving targets detected when designing the instruments for the interviews • New goals had no scientific relevance (very low theoretical and practical impact)
  16. 16. ! ! Example #2
  17. 17. Model-based requirements engineering Develop and evaluate a model-based requirements engineering approach for embedded reactive systems in the automotive domain Problem analysis Evaluation Development (TAR) Case study research Interviews Document analysis
  18. 18. Results Problem analysis Evaluation Development (TAR) Case study research Interviews Document analysis No access to software project environments (“The ivory tower”)
  19. 19. ! ! Example #3
  20. 20. Model-based requirements engineering (2nd try) Evaluate a model-based requirements engineering approach for business information systems in comparison to previously used approach Case study Assessment (Interview) TAR workshops
  21. 21. Case study Assessment (Interview) TAR workshops Results The new model-based approach supported: • more flexibility in process • higher productivity • higher syntactic quality in artefacts ! Conclusion: The new model-based approach better supports a flexible process with high quality artefacts. See also: Mendez Fernandez et al. A Case Study on the Application of an Artefact-based Requirements Engineering Approach !
  22. 22. Conclusion: The new model-based approach better supports a flexible process with syntactically consistent results. What problems do you see in this case study (and conclusions)? • Action research workshops inherently threaten the internal validity • High degree of subjectivity • Results sensitive to their context (generalisation?) • … ! » Each “successful” case study still offers plethora of problems and challenges!
  23. 23. Agenda 1. Case study research: Why and how? 2. Examples 3.Challenges in case study research 4. Time for questions
  24. 24. Industrial research cooperations with • up to 3 years duration • 3-30 participants • multiple case studies We distinguish three classes of challenges: 1. Approaching a case study 2. Conducting a cases study 3. Transversal challenges (…)
  25. 25. Challenges in approaching a case study
  26. 26. Challenges in approaching a case study #1: Finding contacts » Integrate studies into ongoing / planned research projects » Actively approach – alumni of your university – companies in your area – practitioners at (local) events Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1 !
  27. 27. Challenges in approaching a case study #2: Convincing contacts to participate » Find appropriate terminology (technology transfer) » Find appropriate incentives for project partners » Respond to their local needs » Provide early feedback / results » In funded projects, probability is higher that practitioners collaborate
  28. 28. Challenges in approaching a case study #3: Opportunistic case and subject selection » Welcome to case study research! Image source: xkcd !
  29. 29. Challenges in approaching a case study #4: Planning a case study You need to: • schedule meetings / workshops • allocate resources • (formally) define deliverables • clarify responsibilities • clarify NDA-related issues • … (the list can get long) ! » Plan a case study like you’d plan a software development project: – Time plan and resource allocation – Quality assurance and acceptance of protocols and reports – … Image source: https://c1.staticflickr.com/1/186/467087455_e90b880c92.jpg !
  30. 30. Challenges in approaching a case study #5: Uncertainty • What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • … • What will the data look like? •Quality of the data •Quantity of the data » Take early samples and test them! » Test instruments and data quality via pilots » Get to know your subjects early » Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”
  31. 31. Challenges in conducting a case study
  32. 32. Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure » Context often hard to describe (hence, hard to reproduce by others) Variables described Challenges in conducting a case study in report #1: Properly characterising the context » Orient yourself on Variables you should measure Variables you can measure » other (similar) studies or » on classification schemes from the area of software process models (“Tailoring”) » Focus on whole context, not only on cases for which you want to draw conclusions » Ask yourself: “What information would I need to understand and replicate the study?”
  33. 33. Challenges in conducting a case study #2: Communication Possible problems: • No clear communication of case study in organisation • Case study delegated to s.o. not interested • No clear communication lines and reporting lines » Define clear communication/reporting/escalation lines at the beginning. Take no shortcut! » When communicating with subjects: » Communicate purpose and goals of case study » Find a supporter at project level (with own interests) » Don’t call them subjects ;-) Image source: https://c1.staticflickr.com/1/32/62630024_10a8fa36ce_z.jpg?zz=1 !
  34. 34. Challenges in conducting a case study #3: Defining the proper instruments Possible problems: • Wrong assumptions about population • (Research) questions not thought through » Pre-tests your design, e.g. » Pilot your surveys » Check if planned data analysis techniques apply » A way to calibrate the instruments is to conduct a small experiment with students first Image source: How to lie with statistics !
  35. 35. Challenges in conducting a case study #4: The data collection might take long… » Be patient » Never push! Image source: https://c1.staticflickr.com/9/8185/8375141385_df9bd4f283_z.jpg !
  36. 36. Challenges in conducting a case study #5: Working with sensitive data » Clarify the rules: » what you need for the case study » what can be stored and reported (internally and externally) » Plan review and acceptance cycles for reports and publications » The “easier” way (sometimes): Joint publication
  37. 37. Challenges in conducting a case study #6: (Social) skills Possible distortions in data collection: • biased researchers (trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents (missing trust!) • (…) » Don’t be biased! » Involve other researchers in: » Design of instruments » Data collection (e.g. interviews) » Continuously practice your » Moderation skills » Rhetoric and listening skills » Build trust by showing loyalty and respect
  38. 38. Challenges in conducting a case study #7: Analysing qualitative data Qualitative data • is inherently subjective • is difficult to interpret (we all have different mental models!) » When analysing qualitative data » Work in teams (researcher triangulation) » When making interpretations, document the rationale » Validate the results with the subjects (if possible) » Be careful when drawing conclusions Image source: How to lie with statistics !
  39. 39. Challenges in conducting a case study #8: Moving targets • Moving targets are something natural • The difficult part: • Detecting moving targets • Reacting to them » Continuously provide and get feedback » When planning your study, agree on » key goals and questions » open space left for new research questions (flexible part) » Goals should always meet the interest of both industry and academia
  40. 40. Challenges in conducting a case study #9: Tool support Unfortunately, still the best tools we have…
  41. 41. Transversal challenges
  42. 42. Transversal challenges #1: The ivory tower Possible problem: Support by management…only by management. » No backup in projects (organisational culture) » Make clear study plan including needed subjects & cases (and formally agree on it) » Talk early to subjects » Find active supporter (not sponsor) with own interests in project (= their own incentives)
  43. 43. Transversal challenges #2: Sometimes, size seems to matter… » „[Even if] one swallow does not make a summer, one swallow does... » prove the existence of swallows » provide a great deal of reliable information about swallow anatomy“* » Do not be that kind of reviewer that asks for both realistic models and large sample sizes * Y. Anzai and H.A. Simon. The theory of learning by doing (1979) / H. Sharp. Keynote (EASE 2012)
  44. 44. Transversal challenges #3: Drawing proper conclusions » Knowledge is more than statistical significance » When analysing the results, be accurate and critical! » Stay on the ground » You may interpret, as long as the interpretation is clearly stated as such » Rely on other case studies that might explain your observations Image source: xkcd !
  45. 45. Transversal challenges #4: The quest for universal truth (“generalisation”) 1. Software engineering is about human beings (Subjectivity) 2. Reality cannot be captured in complete system of variables ! » “Truth” will always be something relative to its context » Aim for trustworthiness and transparency rather than for int./…/ext. validity » Support scaling up to practice » Appreciate the value of replication studies » Align your study with similar ones » Make clear relation to existing evidence » Disclose your data (e.g. via promisedata) See also: Mendez et al. In Quest for Requirements Engineering Oracles: Dependent Measurements and Variables for (good) RE
  46. 46. Transversal challenges #5: Openness and transparency Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability? » The success of empirical SE relies on a collaborating community » If possible, make your (anonymised) data accessible to others
  47. 47. Transversal challenges #6: Properly reporting the results Case study results need to be disseminated • to practice • to academia » How to properly report? » Report results according to target group: » Small presentations to management w/ focus on results » Peer-reviewed publication w/ (selected) research scope » (Technical) Reports with full data (and analysis) » … » Rely on established guidelines for reporting on case studies !
  48. 48. KEY TAKE AWAYS
  49. 49. My personal manifesto for case study research 6. Be unbiased 7. Be accurate and critical 8. Be honest 9. Be patient 10. Be flexible (and pragmatic!) 1. Carefully plan your study like a project 2. Make clear goals, purpose, context, threats 3. Train your empirical and social skills 4. Disclose your data (reliability & replicability) 5. Collaborate
  50. 50. Agenda 1. Case study research: Why and how? 2. Examples 3. Challenges in case study research 4.Time for questions
  51. 51. Don’t ask , ask us!
  52. 52. ! ! ! Thank you! Daniel Méndez Daniel.Mendez@tum.de @mendezfe Further reading • Slides will be made available (and probably tweeted) • Ask me if you need material (studies, templates, …)

×