Presentation given by Ingeborg Meijer at the Workshop exploring Qualitative and Mixed Methods in Research Evaluation and Policy 2015 (QMM2015)

1. Involving end users in research proposal
evaluation: A case study with the Dutch
Heart Foundation
Ingeborg Meijer
Centre for Science and Technology Studies, Leiden University
QMM Workshop, Brunel University, London
1-2 October 2015

2. Outline
• Societal quality: Ensuring use of of research
• DHF and the end user panel
• Evaluation in practice - triangulation
• Methods
– Observation of peer review
– Questionnaire
– Interview
• Results
• Conclusion
1

3. Scientific & societal quality
• Economic
returns
• Cultural
returns
• Social
returns
• Scientific
returns
Scientific
interactions
Professional
interactions
Private
interactions
Public
interactions
2

4. Science and society
Evaluation
science
Monitoring
Involving End
users
Programming
Societal
Demand
driven
• The process of value creation is by transferring
knowledge from a research institute into society (private
or public parties, or general public), and includes
(demand-driven, user-inspired) research programming,
and interaction with potential users during the research.

5. Dutch Heart
Foundation
objectives
4

6. Dutch Heart Foundation (DHF)
• Charity Fund > 50% of budget to scientific research
• Mission: 3 times less….
– Less people get a cardiovascular disease;
– Less people suffer from cardiovascular disease;
– Less people die from cardiovascular disease.
• Change in objective:
– Closer connected to donators,
– Faster translation from results science to patients
– More participation from stakeholders
• Goal: to experiment with processes in the core of the
DHF > evaluation of research proposals.
• End User panel (EP) & criteria to evaluate societal
aspects
5

7. Societal quality operationalised
5 Criteria
• Relevance of the health problem
• Contribution of the research to
solving the health problem
• To the next step in the research
(or development) process.
• Focus on activities towards
(eventual) application of results in
healthcare
– CV of applicant
– Objectives, strategy and actions
• Participation of stakeholders
• Relation between criteria
6
Relevance
Activity
towards
actors
Societal quality
Participation

8. Phase 2 Practicing in reality
• Separate calls
– Young investigator personal grant
– Focused topic: women and cardiovascular disease
– Focused topic: congenital heart disease in children
• End user panel evaluation in parallel to scientific
evaluation panel
• Different types of information sharing
• No formal input of evaluation of societal aspects on
decision by the board of DHF (despite formal advice)
• Evaluation of the whole process by CWTS
7

9. Overview of calls and EP input
8
Societal aspects in evaluation process Personal Women Congenital CTMM
Information in call text + + + -
Intermediate advice to scientists - + - +
Final advice to scientists + + + -
Advice to ISC as referent - + - -
Advice to ISC in person - - + -
Interaction with EP on same day as ISC + + NA
Interaction with EP on another day as ISC - - + NA

10. Research questions
• Is the EP able to play a role in evaluation of research
proposals, and what kind?
• Is there a difference in focus between ISC and EP?
• How do scientists value the feedback from external
stakeholders?
• Are scientists aware of societal relevance or quality
aspects of their work?
• What do they think about the aim and set up of the DHF?
9

11. Evaluation in practice: triangulation
10
Questionnaire Interview
EP
Societal
ISC
EP
ISC
Interaction
Observation

12. Observations
11

13. Observations of peer review
• Aim: comparing
distribution of
arguments between
ISC and EP
• 2x ISC and 1x EP
• Session was chaired
• Only arguments were
scored, not final
judgement
12

14. Observations – Scores
• Use of observation score form
• Scoring arguments
– Positive
– Negative
– Person
– Before or after presentation
– Content of remarks
• Counting occurrence of arguments
• Limitation: Validation of scores by independent other;
field knowledge
13

15. Result observation Peer review –
congenital EP vs ISC
0
1
2
3
4
5
6
MA1
MA2
MA3
MA4
MA5
WA1
WA2
WA3
WA4
WA5
AP1
AP2
AP3
AP4
CL1
CL2
CL3
CL4
pos voor
pos na
14
EP most used
“Next step” MA3 > positive
“Activity” MA4 > negative
Often in combination with:
“Feasibility” WA4 positive
“Solution” WA2 negative
Negative goes down
AP and CL arguments used
0
2
4
6
8
10
12
14
MA1
MA2
MA3
MA4
MA5
WA1
WA2
WA3
WA4
WA5
AP1
AP2
AP3
AP4
CL1
CL2
CL3
CL4
pos
neg
ISC most used
“Feasibility” WA4 both in positive and
negative context
All others: very limited

16. Result observation Peer review –
ISC congenital & women
15
Comparison two calls:
Feasibility is dominant argument in discussion
Secondly, the chosen approach (WA2) and workplan (WA5) are
discussed most and it relates to how realistic plan is
Note: ISC has not been asked to discuss societal aspects, and
contribution of EP in person (congenital) has little effect.
0
2
4
6
8
10
12
14
MA1
MA2
MA3
MA4
MA5
WA1
WA2
WA3
WA4
WA5
AP1
AP2
AP3
AP4
CL1
CL2
CL3
CL4
pos
neg
0
1
2
3
4
5
6
7
8
9
MA1
MA2
MA3
MA4
MA5
WA1
WA2
WA3
WA4
WA5
AP1
AP2
AP3
AP4
CL1
CL2
CL3
CL4
pos
neg

17. Preliminary conclusion
• EP and ISC use different arguments, where the EP uses
more scientific argumemts than the ISC societal
arguments.
• Suggests that EP has a broader vision compared to ISC
which has little attention for applicant or context related
aspects.
• EP peer review could have added value
16

18. Questionnaire
17

19. Questionnaire
• Aimed at scientists involved in session with End user
panel (n=20)
• Goal: Establishing perceptions of “societal” and opinion
on capabilities of End User Panel
• Three main questions, worked out through statements to
agree upon (4-point Likert scale)
– Vision of scientists on the concept of societal quality
– Vision of scientists on the role of the End Users and their feedback
– Weighing the contribution of the EP and societal criteria on the outcome
of the evaluation
18

20. 19
Geef bij de stellingen die horen bij onderstaande vraag aan in welke mate u het er mee eens bent (1=
helemaal niet mee eens – 4 is helemaal eens; 1 vakje aankruisen svp)
3. Wat betekent voor u valorisatie en het maatschappelijk nut of de maatschappelijke kwaliteit van
onderzoek?
Stelling: Helemaal
niet mee
eens
Niet
mee
eens
Mee
eens
Helemaal
mee eens
Ik denk dat het proces van valorisatie, van kennis tot
maatschappelijk nut leidt.
☐ ☐ ☐ ☐
Volgens mij heeft maatschappelijk nut sociale,
economische en culturele elementen.
☐ ☐ ☐ ☐
Ik denk dat maatschappelijk nut vooral over
communicatie met gebruikers gaat.
☐ ☐ ☐ ☐
De maatschappelijke kwaliteit van onderzoek heeft
denk ik betrekking op het creëren van waarde door de
interactie van een onderzoeksgroep met de
maatschappelijke omgeving (via kennisproductie,
kennisoverdracht en kennisgebruik).
☐ ☐ ☐ ☐
Volgens mij gaat valorisatie van kennis vooral over
economische innovatie.
☐ ☐ ☐ ☐
Ik vind dat maatschappelijke kwaliteit van onderzoek
samenhangt met gerichte activiteiten naar diverse
gebruikers.
☐ ☐ ☐ ☐
Ik denk dat maatschappelijke kwaliteit van onderzoek
onafhankelijk is van de wetenschappelijke kwaliteit en
interactie met medewetenschappers.
☐ ☐ ☐ ☐
Ik denk dat door in een vroeg stadium aandacht te
besteden aan de maatschappelijke kwaliteit van een
onderzoeksvoorstel, de kans groter wordt dat de
resultaten – indien positief - ook sneller op de juiste
plek (de volgende partij in de keten) terecht komen.
☐ ☐ ☐ ☐
Ik vind het van belang dat onderzoekers ook rekening
houden met de context van de verschillende
gebruikers, en hun overdracht van kennis naar die
partijen daarop afstemmen.
☐ ☐ ☐ ☐
Ik vind het terecht dat collectebusfondsen ook
aandacht vragen voor maatschappelijke kwaliteit en
maatschappelijke nut van onderzoek.
☐ ☐ ☐ ☐

21. Q: What is societal quality?
Yes, agree No, don’t agree
Process of knowledge valorisation leads to
societal benefit.
I think that societal benefit is mainly related to
communication with users.
Societal benefit has social, economic and cultural
aspects
I think that valorisation has mainly to do with
economic innovation (some doubt)
Societal quality of research relates to value
creation through the interaction of the research
group with the societal environment
I think that societal quality of research is
independent of the scientific quality and
interaction with fellow scientists.
Societal quality is related to focused activities
towards diverse users.
Paying attention to societal quality in an early
stage of a research project increases the chance
that results are indeed transferred to the right
actor much quicker.
It is important that scientists take into account the
context of users, and adjust the way they transfer
knowledge to that user.
20

22. Q: Contribution of EP to assessment
Statement All
(median
2)
personal EP
The judgement of my scientific peers on my
proposal is sufficient for me.
2,5 3 2 (2)
I understand that when a proposal is
scientifically ‘state of the art’, it may be
rejected based on insufficient societal
quality.
2,32 1,75 2,63 (3)
I understand that only proposal that have
sufficient societal quality proceed to
scientific evaluation process.
2,21 1,75 2,5 (2)
I think that scientific and societal quality
should count equally
2,18 1,5 2,38 (2)
I find all this attention for societal quality or
benefit unnecessary and too much
1,95 2,75 2,25 (2)
21

23. Preliminary conclusion
• Scientists and EP largely agree on what societal quality is.
• Young researchers have less interest in application of knowledge,
don’t want to pay attention to it in an early stage, and think it has
mainly to do with communication.
• Societal quality cannot be leading in deciding what proposals to fund
(scientists and EP)
• Method wise, also look at distribution of answers since n=low
22

24. Interviews
23

25. Interviews – semi-structured
• N=20
• 3 topics:
– Preparing for the interaction with EP
– The actual interaction with the EP
– Feedback from EP
• Interview usually 30 minutes, after questionnaire
• All comments collected, not counted, broad picture
24

26. Interviews – snapshot of results
25
Presentation
Professionals in
EP
Role in decision
Positive
Next step
Communication
Career/grant
Feedback vague
Not transparent
Personal grants

27. Mixed method
message
26

28. Preliminary conclusion
• The contribution, assessment, and feedback of the EP has
added value, but in general scientists do not hear anything
new. It is a different emphasis.
• Scientsist think that End users have a different view, which is
true in some aspects but not in others.
• Interaction EP with ISC could be valuable
• Young scientists: different approach necessary because of
funding pressure and scientific careers (MM)
• Procedure needs to be transparent
• Feedback from EP more explicit
27

29. Thank you for your attention!
Want to know more.....contact me at
i.meijer@cwts.leidenuniv.nl
28