1. What We Have
Technische Universität München
Case Studies in Industry
Daniel Méndez
Technical University of Munich
Germany
CESI 2016
Austin,Texas
@mendezfe
Joint work with
Stefan Wagner (University of Stuttgart)
Based on material from a joint work with:
Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering)
Stefan Wagner (University of Stuttgart)
Feedback from
AntonioVetrò (Technical University of Munich)
Jonas Eckhardt
@binsanoj
5. Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
6. Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Selective
perception
“Everything went great!”
7. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Selective
perception
“Everything went great!”
8. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”
9. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
10. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
“Ok, this was really naive | stupid | biased | not really scientific | …”
11. Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the
authors [Jonas] reflect on what they [the authors]
learned from the process of reflecting on the experiences
and writing the paper […]”
Excuse
Selective
perception
Denial Active learning
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
Learning by doing…it wrong.
“Ok, this was really naive | stupid | biased | not really scientific | …”
13. » Integrate studies into ongoing / planned research projects
» Actively approach
– alumni of your university
– companies in your area
– practitioners at (local) events
Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
15. Convincing contacts to participate
» Find appropriate terminology (technology transfer)
» Find appropriate incentives for project partners
» Respond to their local needs
» Provide early feedback / results
» In funded projects, probability is higher that practitioners collaborate
16. Dealing with uncertainty
• What will the stakeholder characteristics be?
• Availability
• Skills
• Motivation to participate
• Commitment to goals
• Representativeness
• …
• What will the data look like?
• Quality of the data
• Quantity of the data
17. Dealing with uncertainty
• What will the stakeholder characteristics be?
• Availability
• Skills
• Motivation to participate
• Commitment to goals
• Representativeness
• …
• What will the data look like?
• Quality of the data
• Quantity of the data
» Take early samples and test them!
» Test instruments and data quality via pilots
» Get to know your subjects early
» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”
18. Context variables (and phenomena)
• hard to determine (or unavailable)
• hard to measure
» Context often hard to describe
(hence, hard to reproduce by others)
Variables actually
reported
Properly characterising the context
Variables you
should measure
Variables you
can measure
19. Context variables (and phenomena)
• hard to determine (or unavailable)
• hard to measure
» Context often hard to describe
(hence, hard to reproduce by others)
Variables actually
reported
Properly characterising the context
» Orient yourself on
» other (similar) studies or
» on classification schemes from the area of software process models (“Tailoring”)
» Focus on whole context, not only on cases for which you want to draw conclusions
» Ask yourself:“What information would I myself need to understand and replicate the
study under the assumption it would yield same or similar results?”
Variables you
should measure
Variables you
can measure
21. Working with sensitive data
» Clarify the rules:
» what you need for the case study
» what can be stored and reported (internally and externally)
» Plan review and acceptance cycles for reports and publications
» The “easier” way (sometimes): Joint publication
22. Possible distortions in data collection:
• biased researchers
(trying to sell own PhD topic)
• weak moderation skills
• hidden agendas of respondents
(missing trust!)
• (…)
Skills matter
23. Possible distortions in data collection:
• biased researchers
(trying to sell own PhD topic)
• weak moderation skills
• hidden agendas of respondents
(missing trust!)
• (…)
Skills matter
» Don’t be biased!
» Involve other researchers in:
» Design of instruments
» Data collection (e.g. interviews)
» Continuously practice your
» Moderation skills
» Rhetoric and listening skills
» Build trust by showing loyalty and respect
24. Possible problem:
Support by management…only by management.
» No backup in projects (organisational culture)
Beware the ivory tower
25. Possible problem:
Support by management…only by management.
» No backup in projects (organisational culture)
» Make clear study plan including needed
subjects & cases (and formally agree on it)
» Talk early to subjects
» Find active supporter (not sponsor) with own
interests in project (= their own incentives)
Beware the ivory tower
26. Case study data should be disclosed
• reliability and trustworthiness?
• reproducibility and replicability?
» The success of empirical SE relies on a
collaborating community
(Data) Openness and transparency
27. Case study data should be disclosed
• reliability and trustworthiness?
• reproducibility and replicability?
» The success of empirical SE relies on a
collaborating community
(Data) Openness and transparency
» If possible, make your (anonymised) data
accessible to others
http://openscience.us/repo/
28. Properly reporting the results
Case study results need to be disseminated
• to practice
• to academia
» How to properly report?
29. Properly reporting the results
Case study results need to be disseminated
• to practice
• to academia
» How to properly report?
» Report results according to target group:
» Small presentations to management w/ focus on results
» Peer-reviewed publication w/ (selected) research scope
» (Technical) Reports with full data (and analysis)
» …
» Rely on established guidelines for reporting on case studies
31. Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult…
getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise
(be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky
32. Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult…
getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise
(be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky
But there is one difference
• Null results matter
• In contrast to your dating experiences, we want to hear about your
case study experiences (especially the failed ones)