Case Studies in Industry - What We Have Learnt

1. What We Have Technische Universität München Case Studies in Industry Daniel Méndez Technical University of Munich Germany CESI 2016 Austin,Texas @mendezfe Joint work with Stefan Wagner (University of Stuttgart) Based on material from a joint work with: Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering) Stefan Wagner (University of Stuttgart) Feedback from AntonioVetrò (Technical University of Munich) Jonas Eckhardt @binsanoj

2. Compliments Critique

3. (…) Background

4. But before we start…

5. Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”

6. Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Selective perception “Everything went great!”

7. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Selective perception “Everything went great!”

8. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception “Everything went great!” “Well, maybe not everything, but it wasn’t our fault […]”

9. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial “Everything went great!” “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?”

10. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial “Everything went great!” Acceptance “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?” “Ok, this was really naive | stupid | biased | not really scientific | …”

11. Reflection Reviewer Request “ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]” Excuse Selective perception Denial Active learning “Everything went great!” Acceptance “Well, maybe not everything, but it wasn’t our fault […]” “Seriously, how should I have reacted?” Learning by doing…it wrong. “Ok, this was really naive | stupid | biased | not really scientific | …”

12. Finding contacts Image source: https://c1.staticﬂickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

13. » Integrate studies into ongoing / planned research projects » Actively approach – alumni of your university – companies in your area – practitioners at (local) events Finding contacts Image source: https://c1.staticﬂickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1

14. Convincing contacts to participate

15. Convincing contacts to participate » Find appropriate terminology (technology transfer) » Find appropriate incentives for project partners » Respond to their local needs » Provide early feedback / results » In funded projects, probability is higher that practitioners collaborate

16. Dealing with uncertainty • What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • … • What will the data look like? • Quality of the data • Quantity of the data

17. Dealing with uncertainty • What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • … • What will the data look like? • Quality of the data • Quantity of the data » Take early samples and test them! » Test instruments and data quality via pilots » Get to know your subjects early » Be ﬂexible, always ask yourself (and others): “What is the potential of the data I’m getting?”

18. Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure » Context often hard to describe   (hence, hard to reproduce by others) Variables actually reported Properly characterising the context Variables you   should measure Variables you  can measure

19. Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure » Context often hard to describe   (hence, hard to reproduce by others) Variables actually reported Properly characterising the context » Orient yourself on » other (similar) studies or » on classiﬁcation schemes from the area of software process models (“Tailoring”) » Focus on whole context, not only on cases for which you want to draw conclusions » Ask yourself:“What information would I myself need to understand and replicate the study under the assumption it would yield same or similar results?” Variables you   should measure Variables you  can measure

20. Working with sensitive data

21. Working with sensitive data » Clarify the rules: » what you need for the case study » what can be stored and reported (internally and externally) » Plan review and acceptance cycles for reports and publications » The “easier” way (sometimes): Joint publication

22. Possible distortions in data collection: • biased researchers   (trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents   (missing trust!) • (…) Skills matter

23. Possible distortions in data collection: • biased researchers   (trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents   (missing trust!) • (…) Skills matter » Don’t be biased! » Involve other researchers in: » Design of instruments » Data collection (e.g. interviews) » Continuously practice your » Moderation skills » Rhetoric and listening skills » Build trust by showing loyalty and respect

24. Possible problem:   Support by management…only by management. » No backup in projects (organisational culture)  Beware the ivory tower

25. Possible problem:   Support by management…only by management. » No backup in projects (organisational culture)  » Make clear study plan including needed subjects & cases (and formally agree on it) » Talk early to subjects » Find active supporter (not sponsor) with own interests in project (= their own incentives) Beware the ivory tower

26. Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?  » The success of empirical SE relies on a collaborating community (Data) Openness and transparency

27. Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?  » The success of empirical SE relies on a collaborating community (Data) Openness and transparency » If possible, make your (anonymised) data   accessible to others http://openscience.us/repo/

28. Properly reporting the results Case study results need to be disseminated • to practice • to academia » How to properly report?

29. Properly reporting the results Case study results need to be disseminated • to practice • to academia » How to properly report? » Report results according to target group: » Small presentations to management w/ focus on results » Peer-reviewed publication w/ (selected) research scope » (Technical) Reports with full data (and analysis) » … » Rely on established guidelines for reporting on case studies

30. KEY TAKE AWAYS

31. Case Studies are a little bit like dating • Finding someone sharing your interest is difﬁcult…   getting his attention to get further with you even more. • You need deal with uncertainties • You need to compromise   (be pragmatic and, maybe, also a little bit of opportunistic) • Social skills — especially listening skills — matter • Sometimes, it takes very long to get lucky

32. Case Studies are a little bit like dating • Finding someone sharing your interest is difﬁcult…   getting his attention to get further with you even more. • You need deal with uncertainties • You need to compromise   (be pragmatic and, maybe, also a little bit of opportunistic) • Social skills — especially listening skills — matter • Sometimes, it takes very long to get lucky But there is one difference • Null results matter • In contrast to your dating experiences, we want to hear about your   case study experiences (especially the failed ones)

33. Thank you! Daniel Méndez Daniel.Mendez@tum.de @mendezfe • Slides will be made available   (and probably tweeted) • Approach him if you need material (studies, templates, …)

Case Studies in Industry - What We Have Learnt

Case Studies in Industry - What We Have Learnt

Recommended

More Related Content

What's hot

What's hot(19)

Viewers also liked

Viewers also liked(9)

Similar to Case Studies in Industry - What We Have Learnt

Similar to Case Studies in Industry - What We Have Learnt(20)

More from Daniel Mendez

More from Daniel Mendez(20)

Recently uploaded

Recently uploaded(20)

Case Studies in Industry - What We Have Learnt