Erik Arnold and Bea Mahieu Závěrečná konference projektu IPN Metodika / Praha, 15. října 2015

1. Governance, assessment and incentives in the
research and innovation funding system
Erik Arnold and Bea Mahieu
Prague
15 October 2015

2. Research and innovation policy – three contracts or
governance paradigms
• First: Post-War – Endless Frontier – ’hands-off’ approach to science
funding; expectation that welfare would increase in response but in
unpredictable ways
• 1960s, OECD and the start of ‘science policy’ as tuning science to
societal needs (Freeman, Frascati and the resurgence of Bernal ..)
• Second: 1970s on, breakdown in trust in science as a ‘neutral’ force;
politicisation of technology (eg Vietnam); societal demands of S&T
focus on industrial and technological development
• Third: Circa 2000, ‘grand’ (systemic?) challenges; no longer about
industry but fear that we have finally hit the limits to growth (climate,
energy, ageing, disease … )
• In policy these generate layers – the current funding system needs to
balance basic research, innovation-related and societally driven efforts
all at the same time
2

3. 3
Some key observations from studies of national innovation
systems
• Innovation is a non-linear process involving many actors
• ‘Bounded rationality’ causes path-dependency and means both
institutions and learning affect performance
• Institutions are inter-dependent and co-evolve in ways that may be
specific to the national system
• Good systems performance depends upon intelligence and
performance in all sub-systems
• ‘Bottleneck analysis’ and system development are key policy roles
• Innovation system complexity tends to defeat central planning but
distributed intelligence enables a healthy mix of bottom-up and
top-down policy design and implementation (‘subsidiarity’)

4. Principles of good governance
• Multi-layer governance based on
• Distributed intelligence
• Subsidiarity
• Distributed capacity
• Linked to both higher (EU) ad lower (regional) levels
• Able to overview system performance, policy and its effects
• Support the formation of an overall strategy
• Balance or prioritise different elements in the mix
• Able horizontally to group sector interests
• Bringing needed societal stakeholders into the formation and
implementation of strategy
• Under the new paradigm: evolving to tackle the societal challenges
4

5. 5
All countries struggle to govern the state’s role in the NIS
R&D Institutes
Parliament
Government Policy council
Ministry of
Education
Research Councils
and Academies
Universities
Other Sectoral
Ministries
Producers:
Firms, farms,
hospitals, etc
Ministry of
Industry
Technology &
Innovation Agencies
Support Programme
Agencies
Programme Contractors
Instructions, resources
Advice
Results
Horizontal co-ordination and integration
Level 1
High-level cross-
cutting policy
Level 2
Ministry mission-
centred co-ordination
Level 3
Detailed policy
development, co-
ordination
Level 4
Research and
innovation
performers
Key

6. 6
Czech Republic model is hybrid, transitional
AS CR
PrincipalPerformers
GOVERNMENT
R&D&I COUNCIL
Agent
Ministry of
Health
Deptfor
Development
andResearch
InternalGrant
Agency
Ministry of
Education
Dept.Int’l
Cooper./ERA
Ministry of
Industry &
Trade
Deptof
industrial
R&D
InternalGrant
Agency
Private Research
Institutes – Agricultural
R&D
Universities
Industry
Private
Institutes
–
Industrial
R&D
Technology
Agency
Grant
Agency CR
Ministry of
Agriculture
Ministry of
Culture
Dept.forR&D
programmes
Sectoral Public Research Institutes / State Contributory R&D
Organisations
Public Research
Institutes
Ministry of
Defence
Deptfor
Managing
Armament,
R&D
Deptof
Security
R&d&i
Ministry of
Interior
Individual researchers
Institutional
support
Competitive/Targe
ted funding
Other Public Research
Institutes
Centralco-
ordination

7. The Audit found significant weaknesses in the R&D&I
Council
• Under-estimation of importance of consensus-building & open dialogue
with policy implementing bodies, stakeholders & citizens
• Culture of strong top-down steering & control of policy implementation
• Setting up a long-term strategy, essentially top down
• Has to include MEYS for international dimension
• Priority setting governed by the Council
• Agencies to design and implement programmes
• Weak links to sectoral policies
• Strategic intelligence
• Research and analysis outsourced, but not to stakeholders
• Evaluation ‘automated’; provides little information about policy effectiveness
• Focus on resource allocation means members become representatives
• Over-centralisation means there is too much for the Council to do to do it
well
7

8. Desiderata for a Council
• Functions as an open arena for consensus
• Is legitimate in scientific, industrial and political terms
• Collates and publishes strategic intelligence when needed,
within a system of distributed strategic intelligence
• Sets long-term strategic directions, reducing dynamic
inconsistency
• Coordinates vertically, horizontally and over time
• Has a high profile with the government and the public
• Is independent enough to be a change agent
• Has a clear interface to government
8

9. The structure of research funding
9
Education Ministry Institutional funding
Excellence funding
Relevance funding
Research council
Sector authorities
Coordination?
Funding flows
Potential PRFS
parameters
Societal challenge funding

10. The CR needs a developmental assessment system with
many dimensions – not just simplified performance
measures
10
Research Units
EvU and RO
Disciplinary areasFields
Scientific research excellence
National policy makers
National R&D governance bodies
Research performance
Societal relevance
Institutional management &
development potential
Membership of the (world)
research community
Institutional
research strategy
Institutional &
HR management
Positioning at
national level
Positioning at
international
levelModels of good
practice
Positioning at
international level
National research strategy
Alignment with RD&I
priorities
Strengths & weaknesses
Needs for policy
interventions
Needs for policy interventions
Positioning at the international level
Sectoral R&D strategy
Priority areas for performance contracts

11. A performance-based institutional funding system needs
up to three functional elements
• A ‘block grant’
• Providing stability and a degree of ‘Planungssicherheit’
• Slowly changing over time, reflecting change needs and performance
in the research system
• A performance-based component
• Providing rewards for good performance in the short-medium term
• Providing an economic incentive for change
• Encouraging good performance through prestige
• A prospective element, such as a performance contract
• Enabling entry into the system
• Combating the conservative tendencies of block and performance-
based funding by supporting development and capacity-building
11

12. UK Experience
• The RAE is the ‘mother of all PRFS’; allocates most of the money
• Peer review – in more recent times ‘informed’ by bibliometrics
• Driven by massification and a need to justify cuts in the 1980s
• “A complex process whereby the Russell Group gives itself most of
the money”
• Non-linear allocation formula intended to concentrate resources
• Widely acknowledged bias against multidisciplinary and heterodox
research
• Stable outcomes; high correlation with performance in research
council system
• Anecdotally, massive effects on recruitment, promotion, research
management
• High cost: recurring question about greater reliance on metrics
12

13. Czech Republic
• Post-reform system of ‘research intentions’ as basis for funding
abandoned owing to low trust and low governance capability
• ‘Coffee grinder’ 2009-11 wholly metrics based – across fields and
different types of research organisation
• ‘Coffee Grinder points’ devalued by 60% 2009-11
• Included many categories of non-scholarly output – which were
clearly gamed (as were some peer-reviewed publications)
• Combined with erratic allocation of state research budget, the
Coffee Grinder caused instability in institutional funding
• Despite constant fiddling with the parameters, the Coffee Grinder
was dropped as unfit for purpose following the Research Audit in
2012
13

14. Norway
• PRFS introduced following the university ‘quality reform’ of 2002
– at first in the universities, later (separately) in the institutes
• Simple, metrics-based, no field normalisation, includes a
classification of local publication channels
• Reallocates 2% of funding – huge change for little money
• University PRFS
• Quantity but not quality of publications has risen (cp Australia)
• Proportion of faculty publishing has risen – especially in weaker
organisations
• Decline in monetary value of a publication
• Institutes PRFS: effects on publication volume, research
management and HR but not on international income or
cooperation with universities (already quite high)
14

15. About performance-based research systems
• There’s not much evidence yet behind the policy trend to PRFS
• Policy purposes seem rarely to be made explicit. If you dig, you can
find them
• UK: Matthew effect
• NO: Quality of the whole system
• CZ: Overcoming governance failures
• PRFS are high-leverage interventions
• Behaviour change drivers are probably career and status
• Possible to use them without destabilising institutional funding
• Highly prone to gaming and unintended effects
• Longer-term risks include ‘normalisation’ of science and research
(Kuhn), changes in cooperation behaviour and undermining
academia/rest-of-society links
• They provide a tool for policy implementation – they are not a
substitute for strategy, policy or governance
15

16. 16
Thank you
bea.mahieu@technopolis-group.com
erik.arnold@technopolis-group.com
www.technopolis-group.com
technopolis |group| has offices in Amsterdam, Brighton, Brussels,
Frankfurt/Main, Paris, Stockholm, Tallinn and Vienna