Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Case Studies in Industry - What We Have Learnt

393 views

Published on

Presentation at the 4th Intl. Workshop on Conducting Empirical Studies in Industry CESI 2016 - An ICSE 2016 Workshop‎

Published in: Science
  • Be the first to comment

  • Be the first to like this

Case Studies in Industry - What We Have Learnt

  1. 1. What We Have Technische Universität München Case Studies in Industry Daniel Méndez Technical University of Munich Germany CESI 2016 Austin,Texas @mendezfe Joint work with Stefan Wagner (University of Stuttgart) Based on material from a joint work with: Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering) Stefan Wagner (University of Stuttgart) Feedback from AntonioVetrò (Technical University of Munich) Jonas Eckhardt @binsanoj
  2. 2. Compliments Critique
  3. 3. (…) Background
  4. 4. But before we start…
  5. 5. Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
  6. 6. Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Selective perception “Everything went great!”
  7. 7. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Selective perception “Everything went great!”
  8. 8. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception “Everything went great!” “Well, maybe not everything, but it wasn’t our fault […]”
  9. 9. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial “Everything went great!” “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?”
  10. 10. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial “Everything went great!” Acceptance “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?” “Ok, this was really naive | stupid | biased | not really scientific | …”
  11. 11. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial Active learning “Everything went great!” Acceptance “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?” Learning by doing…it wrong. “Ok, this was really naive | stupid | biased | not really scientific | …”
  12. 12. Finding contacts Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
  13. 13. » Integrate studies into ongoing / planned research projects » Actively approach – alumni of your university – companies in your area – practitioners at (local) events Finding contacts Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
  14. 14. Convincing contacts to participate
  15. 15. Convincing contacts to participate » Find appropriate terminology (technology transfer) » Find appropriate incentives for project partners » Respond to their local needs » Provide early feedback / results » In funded projects, probability is higher that practitioners collaborate
  16. 16. Dealing with uncertainty • What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • … • What will the data look like? • Quality of the data • Quantity of the data
  17. 17. Dealing with uncertainty • What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • … • What will the data look like? • Quality of the data • Quantity of the data » Take early samples and test them! » Test instruments and data quality via pilots » Get to know your subjects early » Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”
  18. 18. Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure » Context often hard to describe 
 (hence, hard to reproduce by others) Variables actually reported Properly characterising the context Variables you 
 should measure Variables you
 can measure
  19. 19. Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure » Context often hard to describe 
 (hence, hard to reproduce by others) Variables actually reported Properly characterising the context » Orient yourself on » other (similar) studies or » on classification schemes from the area of software process models (“Tailoring”) » Focus on whole context, not only on cases for which you want to draw conclusions » Ask yourself:“What information would I myself need to understand and replicate the study under the assumption it would yield same or similar results?” Variables you 
 should measure Variables you
 can measure
  20. 20. Working with sensitive data
  21. 21. Working with sensitive data » Clarify the rules: » what you need for the case study » what can be stored and reported (internally and externally) » Plan review and acceptance cycles for reports and publications » The “easier” way (sometimes): Joint publication
  22. 22. Possible distortions in data collection: • biased researchers 
 (trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents 
 (missing trust!) • (…) Skills matter
  23. 23. Possible distortions in data collection: • biased researchers 
 (trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents 
 (missing trust!) • (…) Skills matter » Don’t be biased! » Involve other researchers in: » Design of instruments » Data collection (e.g. interviews) » Continuously practice your » Moderation skills » Rhetoric and listening skills » Build trust by showing loyalty and respect
  24. 24. Possible problem: 
 Support by management…only by management. » No backup in projects (organisational culture)
 Beware the ivory tower
  25. 25. Possible problem: 
 Support by management…only by management. » No backup in projects (organisational culture)
 » Make clear study plan including needed subjects & cases (and formally agree on it) » Talk early to subjects » Find active supporter (not sponsor) with own interests in project (= their own incentives) Beware the ivory tower
  26. 26. Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?
 » The success of empirical SE relies on a collaborating community (Data) Openness and transparency
  27. 27. Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?
 » The success of empirical SE relies on a collaborating community (Data) Openness and transparency » If possible, make your (anonymised) data 
 accessible to others http://openscience.us/repo/
  28. 28. Properly reporting the results Case study results need to be disseminated • to practice • to academia » How to properly report?
  29. 29. Properly reporting the results Case study results need to be disseminated • to practice • to academia » How to properly report? » Report results according to target group: » Small presentations to management w/ focus on results » Peer-reviewed publication w/ (selected) research scope » (Technical) Reports with full data (and analysis) » … » Rely on established guidelines for reporting on case studies
  30. 30. KEY TAKE AWAYS
  31. 31. Case Studies are a little bit like dating • Finding someone sharing your interest is difficult… 
 getting his attention to get further with you even more. • You need deal with uncertainties • You need to compromise 
 (be pragmatic and, maybe, also a little bit of opportunistic) • Social skills — especially listening skills — matter • Sometimes, it takes very long to get lucky
  32. 32. Case Studies are a little bit like dating • Finding someone sharing your interest is difficult… 
 getting his attention to get further with you even more. • You need deal with uncertainties • You need to compromise 
 (be pragmatic and, maybe, also a little bit of opportunistic) • Social skills — especially listening skills — matter • Sometimes, it takes very long to get lucky But there is one difference • Null results matter • In contrast to your dating experiences, we want to hear about your 
 case study experiences (especially the failed ones)
  33. 33. Thank you! Daniel Méndez Daniel.Mendez@tum.de @mendezfe • Slides will be made available 
 (and probably tweeted) • Approach him if you need material (studies, templates, …)

×