Discussion session hosted by Leonie van Drooge at the Workshop exploring Qualitative and Mixed Methods in Research Evaluation and Policy 2015 (QMM2015)

1. Case study narratives
Discussion session – Leonie van Drooge

2. 2 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
Societal relevance (SEP 2009-2015):
Self-evaluation report:
• “Group leaders collaborate with more than 31 pharmaceutical,
robotics and technical companies” “prof X founded a
company” “5 patents were filed”…..
“Several group leaders appeared in prominent television (63
times) and radio programs (75 times)”…..
Evaluation report:
• “In general societal impact is guaranteed since health and
disease are central issues in any society; and, in particular,
subfield Y is highly relevant given increase in the number of
elderly people in Western societies.”

3. 3 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
Case studies: why is it an issue?
• Case studies have been introduced in a number of ex post
evaluation schemes (REF, SEP)
• However, the use seems problematic
• Stefan de Jong, Jorrit Smit & Leonie van Drooge (2015):
“read it with scepticism, because I already knew….something is being
imposed on us which did not originate from us. Without any motivation.”
“And the international reviewer has no idea what we mean by it
[knowledge utilization], absolutely no idea.”
• Gabrielle Samuel & Gemma Derrick (2015):
“different types of conceptualizations impact evaluators may bring to
formal assessment frameworks before embarking on assessment”

4. 4 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
Instructions for case studies or impacts
• REF (UK ex post): a clear explanation of the process or
means….
• SEP (NL ex post): how the unit achieved this….
• Broader Impacts (US ex ante): describe the potential of the
proposed activity to benefit society and contribute to the
achievement of specific, desired societal outcomes

5. 5 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
SEP
• The committee assesses the quality, scale and relevance of
contributions targeting specific economic, social or cultural target
groups, of advisory reports for policy, of contributions to public
debates, and so on. The point is to assess contributions in areas
that the research unit has itself designated as target areas.
• The point of the narrative is to describe the most convincing
examples of relevance, impact or added value to society achieved
by means of the scientific work of the research unit.
• The indicators given in the table are only examples. The research
unit may select these or other indicators compatible with its own
profile and mission, within clearly defined boundaries.

6. 6 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
What are the pitfalls (or challenges)?
• Researchers and groups (evaluated) are advised and
supported, however, evaluators are not.
• Narritives that show no relation between research, actions and
impact (“and then…. and then…. and then”)
• Narratives showing too much intention, leading to overly
positivistic or deductive narratives

7. 7 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
What is required – what is our responisibility now
• Do we have a responsibility and are we needed?
• Case study should provide insight
• Context is important – learn from ex ante situations
• We should come up with something helpful, not complicated
• We know a range of schemes (different countries, ex post &
ex ante)

8. 8 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
What do you need?
• ?

9. 9 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives
Thank you
Leonie van Drooge
Rathenau Instituut
l.vandrooge@rathenau.nl