EVALUATION OF SOCIAL IMPACT productive interactions in health (policy) research

1. EVALUATION OF SOCIAL IMPACT
productive interactions in
health (policy) research
HTAi conference Bilbao, 27-06-2012
Jack Spaapen

2. 2
Three questions (SIAMPI project)
Q1. What is social impact? - Conceptual and practical problems
Q2. What are ‘productive’ interactions? And what do they have to do
with social impact?
Q3. How to evaluate social impact: PI, intermediate impacts, social
impact

3. 3
Q1. WHAT IS SOCIAL IMPACT
- Conceptual problems
- Linear models vs network approaches
- Temporality
- Attribution / contribution
- Overlap between various impacts
- Positive / negative impacts
- Practical problems
- Reward systems
- Indicators
- Data collection

4. 4
where is social impact?

5. 5
research is contextualized
Topsector Health / Life sciences (Netherlands)
New Dutch Science Policy Top sectors: energy, water,
food/agriculture, health/life sciences, creative industry, …
Healthy aging (EU)
Creative industry, new media, (serious) games
Nanotechnology
Green meat production in 2020 in NL
Water management (New Orleans)
Social impact is the outcome of a joint effort in a network of relations: research, industry,
government, NGOs, consumer-organisations  public private partnerships
Which data are relevant? Activities in the network  coalitions, goals, interactions,
intermediate results

6. 6
New coalitions for Research, development and innovation
- Triple helix, golden triangle: research, industry, society (government, NGOs, general
public), new collaborative arrangements: PPP (transdisciplinary research,
transepistemic communities – Knorr-Cetina, science 3.0 - Miedema)
- Interdisciplinary input from research: natural science/technical fields, social sciences,
humanities (technical knowledge, content, use/behavior)
- But also input from other expertise: politics, law, economists, ethics, consumer-
interests, etc.
- Co-creation of ‘ new knowledge’ and practical solutions = innovation, research by
design, iterative process (non-linear) -
- Consensus about long term goals (“healthy aging”, “clean energy”), but in the meantime
shifting coalitions, different partners, different intermediate goals, different interests
- Problems arise from not being used to work together (academics – industry, not
knowing what to formulate as research questions), institutional problems, political and
cultural problems….

7. 7
RESEARCH, DEVELOPMENT and INNOVATION (RDI)
AS A LONG TERM NETWORK EFFORT
The network consists of a variety of
stakeholders working on a common
problem; but goals and people shift
Everybody produces
knowledge, everybody does
research: transdisciplinary
collaboration
The result is to be socially robust knowledge : scientifically reliable, socially valuable

8. 8
Q2. PRODUCTIVE INTERACTIONS
What are ‘productive’ interactions?
- Relations in a network that produce something of value
- Not only of economic value (knowledge-new technology-cash value)
- Also socio-cultural, ethical, intellectual, technological, environmental
And what do they have to do with social impact?
- They show what it takes to achieve impact
- Guideline for indicators of intermediate impact

9. 9
SIAMPI: productive interactions between science and society
network oriented
• Personal interactions : joint projects, advisory, consultancy, double
functions, mobility
• Media interactions :
• Texts : articles, books, catalogues, protocols, new diagnostics
• Artifacts : instruments, exhibitions, models, designs
• Support: contracts, subsidies, patenting, licensing, sharing of people and
facilities
• Instances of social impact

10. 10
DATA AND INDICATORS:
FOCUS ON INTERACTION AND INTERMEDIATE OUTCOME
representing productive interactions
between a variety of stakeholders
= research, policy, industry, society
Data collection:
(i) personal interactions
(i) interaction thru media
(iii) financial or material support
(iv) intermediate output
(v) indications of social impact
(vi) new products, procedures, etc

11. 11
Q3. HOW TO EVALUATE
CONCEPTUAL [self evaluation reports]
• Mission orientation
• Network analysis (thru texts, people, organisations)
• Stakeholder approach (involvement from the start)
PRACTICAL
• Network indicators - indications
• Intermediate indicators – 3-5 years
• Impact indicators – instances of impact

12. 12
HEALTH CASE: NIVEL and LUMC
NIVEL public/private institute for health policy and primary health care
LUMC academic department (public health, gynaecology,
anatomy/embryology)
NIVEL: top down organisation of network activities, stakeholder contacts
actively organised to safeguard financial support and to enhance
chance of implementation of results, include all types of stakeholders
LUMC: bottom up, incentive structure based on WoS, but on the move
from traditional academic department to contextual research (top
sector policy for life sciences and health research)

13. 13
EXAMPLE OF CASE STUDIES
Social impacts related to research at NIVEL or LUMC are often too intricate to be
attributed directly to or identified with specific groups.
LUMC: Anatomy, stem cell projects. Results / impacts are only to be expected in
the very long run (10-15 years)
In this case, we did see some impact in clinical practice of a neighbouring field,
cardiology.
NIVEL: the result of one project in Public Health pointed out that currently
popular health centers specifically for elderly only raised health care demand
but did not improve the health situation of elderly.
Impacts (i) can be positive or negative; and (ii) take shape in various contexts:
the academic group (also other groups!), the hospital, the policy domain
(stem cells!)

14. 14
Type of Example Effects (success)
interaction
Direct Consultation rounds with stakeholders Adaptation in research agenda
User groups, supervising boards (PPP Adaptation of research projects
consortia)
Presentations to health care professionals Sustaining relations with
(conferences, meetings, post-academic stakeholders, knowledge transfer
teaching)
Collaboration in research Mutual adaptation in research
projects
Indirect Annual plans Mutual agreements with funding
agencies over future research agenda
Implementation plans Mutual agreements with funding
agency over future implementation of
project results
Reports & medical guidelines, Knowledge transfer, Response
scientific publications (uptake of knowledge) by a wider
variety of stakeholders than those
involved in direct interactions
Financial Contracts, Licences, Project grants Enabling completion of research
projects
Lump sum grants Enabling independent research

15. 15
SOCIAL IMPACT OF NIVEL: reception of reports per domain
©AD PRINS
Google searches for reports / PDFs of 8 NIVEL domains

16. 16
SOCIAL IMPACT OF LUMC: MEDICAL GUIDELINES FOR GPS
© AD PRINS
Google searches for three guidelines of LUMC Dep. of Public Health and General Practice

17. Table 1: SIAMPI indicators for productive interactions
personal interactions interaction between Financial / material
between stakeholders stakeholders through media interaction between
stakeholders
•face-to-face meetings •academic journals •research contracts, public and
•double functions, other •professional journals private, and mixed, national,
mobility arrangements •non academic journals international
•phone conferences •popular media •facility, instruments sharing
•email •exhibitions • start ups
•social media •artefacts, models •contribution “in kind” (people)
•videoconferencing •films •IPR arrangements, patents,
•public debate •master theses, graduate licenses
•radio, tv, internet projects •Professional training
•etc. •standards, protocols •Other stakeholder interest
•social media •etc.
•etc.

18. 18
What is necessary to evaluate research in context
o A network perspective, mutual learning, extended review
(more than peers), new reward systems, administrators that
dare to care
o Development and use of new type of criteria and indicators
that focus on process and intermediate results
o Continuous feed back between stakeholders, impact through
learning!
o Use / acceptance of these methods at all levels of research
(institute, national, EU)

19. 19
Research nano ict Health care ssh
domain
Country NL, France UK, NL, EU NL Esp, UK
Research type Frontier, basic, Basic, applied, TD strategic, applied, Basic, strategic,
strategic policy applied
Research mode Academic, in Open to partnership of Academic, open to Academic, open to
collaboration with knowledge producers collaboration with collaboration with
industry and users industry, govern- policy, institutions,
ment, patient wider public,
groups, professio- industry
nals
Productive Public Transport use, Consultation, colla- Informal links and
Interactions understanding, security, interaction boration, regula- advice, formal
ethical debates, between citizens and tions, proto-cols, research contract
and collaborative
policy making, government commercial
projects,
products exchanges, PPPs, consultancy,
post academic cultural events
training, patient
organizations
Social Impact Health, safety, Transport use, Diagnostics, Policy tools and
public security, interaction treatments, safety, techniques,
acceptance of between citizens and general health, management
nano tech government policy advice methods, cultural
goods and services