EVALUATION OF SOCIAL IMPACT productive interactions in health (policy) research HTAi conference Bilbao, 27-06-2012 Jack Spaapen
2 Three questions (SIAMPI project)Q1. What is social impact? - Conceptual and practical problemsQ2. What are ‘productive’ interactions? And what do they have to do with social impact?Q3. How to evaluate social impact: PI, intermediate impacts, social impact
3 Q1. WHAT IS SOCIAL IMPACT - Conceptual problems - Linear models vs network approaches - Temporality - Attribution / contribution - Overlap between various impacts - Positive / negative impacts - Practical problems - Reward systems - Indicators - Data collection
5 research is contextualizedTopsector Health / Life sciences (Netherlands)New Dutch Science Policy Top sectors: energy, water, food/agriculture, health/life sciences, creative industry, …Healthy aging (EU)Creative industry, new media, (serious) gamesNanotechnologyGreen meat production in 2020 in NLWater management (New Orleans)Social impact is the outcome of a joint effort in a network of relations: research, industry, government, NGOs, consumer-organisations public private partnershipsWhich data are relevant? Activities in the network coalitions, goals, interactions, intermediate results
6 New coalitions for Research, development and innovation- Triple helix, golden triangle: research, industry, society (government, NGOs, general public), new collaborative arrangements: PPP (transdisciplinary research, transepistemic communities – Knorr-Cetina, science 3.0 - Miedema)- Interdisciplinary input from research: natural science/technical fields, social sciences, humanities (technical knowledge, content, use/behavior)- But also input from other expertise: politics, law, economists, ethics, consumer- interests, etc.- Co-creation of ‘ new knowledge’ and practical solutions = innovation, research by design, iterative process (non-linear) -- Consensus about long term goals (“healthy aging”, “clean energy”), but in the meantime shifting coalitions, different partners, different intermediate goals, different interests- Problems arise from not being used to work together (academics – industry, not knowing what to formulate as research questions), institutional problems, political and cultural problems….
7 RESEARCH, DEVELOPMENT and INNOVATION (RDI) AS A LONG TERM NETWORK EFFORT The network consists of a variety of stakeholders working on a common problem; but goals and people shift Everybody produces knowledge, everybody does research: transdisciplinary collaborationThe result is to be socially robust knowledge : scientifically reliable, socially valuable
8 Q2. PRODUCTIVE INTERACTIONSWhat are ‘productive’ interactions? - Relations in a network that produce something of value - Not only of economic value (knowledge-new technology-cash value) - Also socio-cultural, ethical, intellectual, technological, environmentalAnd what do they have to do with social impact? - They show what it takes to achieve impact - Guideline for indicators of intermediate impact
9 SIAMPI: productive interactions between science and society network oriented• Personal interactions : joint projects, advisory, consultancy, double functions, mobility• Media interactions : • Texts : articles, books, catalogues, protocols, new diagnostics • Artifacts : instruments, exhibitions, models, designs• Support: contracts, subsidies, patenting, licensing, sharing of people and facilities• Instances of social impact
10 DATA AND INDICATORS: FOCUS ON INTERACTION AND INTERMEDIATE OUTCOME representing productive interactions between a variety of stakeholders = research, policy, industry, society Data collection: (i) personal interactions (i) interaction thru media (iii) financial or material support (iv) intermediate output (v) indications of social impact (vi) new products, procedures, etc
11 Q3. HOW TO EVALUATECONCEPTUAL [self evaluation reports] • Mission orientation • Network analysis (thru texts, people, organisations) • Stakeholder approach (involvement from the start)PRACTICAL • Network indicators - indications • Intermediate indicators – 3-5 years • Impact indicators – instances of impact
12 HEALTH CASE: NIVEL and LUMCNIVEL public/private institute for health policy and primary health careLUMC academic department (public health, gynaecology, anatomy/embryology)NIVEL: top down organisation of network activities, stakeholder contacts actively organised to safeguard financial support and to enhance chance of implementation of results, include all types of stakeholdersLUMC: bottom up, incentive structure based on WoS, but on the move from traditional academic department to contextual research (top sector policy for life sciences and health research)
13EXAMPLE OF CASE STUDIESSocial impacts related to research at NIVEL or LUMC are often too intricate to be attributed directly to or identified with specific groups.LUMC: Anatomy, stem cell projects. Results / impacts are only to be expected in the very long run (10-15 years)In this case, we did see some impact in clinical practice of a neighbouring field, cardiology.NIVEL: the result of one project in Public Health pointed out that currently popular health centers specifically for elderly only raised health care demand but did not improve the health situation of elderly.Impacts (i) can be positive or negative; and (ii) take shape in various contexts: the academic group (also other groups!), the hospital, the policy domain (stem cells!)
14 Type of Example Effects (success) interaction Direct Consultation rounds with stakeholders Adaptation in research agenda User groups, supervising boards (PPP Adaptation of research projects consortia) Presentations to health care professionals Sustaining relations with (conferences, meetings, post-academic stakeholders, knowledge transfer teaching) Collaboration in research Mutual adaptation in research projects Indirect Annual plans Mutual agreements with funding agencies over future research agenda Implementation plans Mutual agreements with funding agency over future implementation of project results Reports & medical guidelines, Knowledge transfer, Response scientific publications (uptake of knowledge) by a wider variety of stakeholders than those involved in direct interactions Financial Contracts, Licences, Project grants Enabling completion of research projects Lump sum grants Enabling independent research
Table 1: SIAMPI indicators for productive interactionspersonal interactions interaction between Financial / materialbetween stakeholders stakeholders through media interaction between stakeholders•face-to-face meetings •academic journals •research contracts, public and•double functions, other •professional journals private, and mixed, national,mobility arrangements •non academic journals international•phone conferences •popular media •facility, instruments sharing•email •exhibitions • start ups•social media •artefacts, models •contribution “in kind” (people)•videoconferencing •films •IPR arrangements, patents,•public debate •master theses, graduate licenses•radio, tv, internet projects •Professional training•etc. •standards, protocols •Other stakeholder interest •social media •etc. •etc.
18What is necessary to evaluate research in context o A network perspective, mutual learning, extended review (more than peers), new reward systems, administrators that dare to care o Development and use of new type of criteria and indicators that focus on process and intermediate results o Continuous feed back between stakeholders, impact through learning! o Use / acceptance of these methods at all levels of research (institute, national, EU)
19Research nano ict Health care sshdomainCountry NL, France UK, NL, EU NL Esp, UKResearch type Frontier, basic, Basic, applied, TD strategic, applied, Basic, strategic, strategic policy appliedResearch mode Academic, in Open to partnership of Academic, open to Academic, open to collaboration with knowledge producers collaboration with collaboration with industry and users industry, govern- policy, institutions, ment, patient wider public, groups, professio- industry nalsProductive Public Transport use, Consultation, colla- Informal links andInteractions understanding, security, interaction boration, regula- advice, formal ethical debates, between citizens and tions, proto-cols, research contract and collaborative policy making, government commercial projects, products exchanges, PPPs, consultancy, post academic cultural events training, patient organizationsSocial Impact Health, safety, Transport use, Diagnostics, Policy tools and public security, interaction treatments, safety, techniques, acceptance of between citizens and general health, management nano tech government policy advice methods, cultural goods and services