/ Erik Arnold, Technopolis Group + více k II. Mezinádorní konferenci IPN Metodika v článku Konference k financování přinesla podnětnou diskuzi.

1. Architecture of the EM
and some key
experience from
abroad
www.metodika.reformy-msmt.cz

2. Road map
• Making sure we all start our discussion in the same place
• The architecture of the new Evaluation Method
• The funding principles
• Scenarios and their potential effects
• The community’s comments
• Overview of comments
• Our reflections based on the comments
• Discussion
• Breakouts
• A. The Evaluation Methodology, its added value and role
• B. Stabilization and motivation roles of institutional funding
• C. How can – and should – the Evaluation and Funding System foster change in
the Czech RD&I system?
• Reports from breakout groups to the plenary
2

3. Policy development and
implementation are at the heart of
the new methodology
3
Assessment
process
Funding
process
Policy
development

4. Some key principles
• Only research-performing research units should be included
• Implicitly, research units are only included if some part of the state is willing to
provide institutional funding
• Central role of ‘informed’ peer/panel review
• Common assessment ‘spine’ with field-specific variations, allowing
aggregation of assessment results (eg at field or institutional level)
• Common assessment criteria covering performance in 5 different
dimensions – no expectation that everyone does well against all criteria
• Translation of performance into funding is based on the societal
function of the research unit
• Institutional funding contains elements that
• Provide a substantial measure of funding stability
• Reward the quality and impact of the research unit
• Encourage strategic development of the research unit 4

5. Building up from the fundamental
unit of assessment: the Research
Unit
5
Research unit
(RU)
Individual
researchers
Evaluated Unit
(EvU)
Research Organisation
Research Institute /
University Faculty
R&D Governance
Bodies & Agencies
Scientific
fields
National R&D
System
(International)
Structure of
Science
Disciplinary
areas

6. Links from assessment to policy
and institutional feedback
6
Research Units
EvU and RO
Disciplinary areasFields
Scientific research excellence
National policy makers
National R&D governance bodies
Research performance
Societal relevance
Institutional management &
development potential
Membership of the (world)
research community
Institutional
research strategy
Institutional &
HR management
Positioning at
national level
Positioning at
international
levelModels of good
practice
Positioning at
international level
National research strategy
Alignment with RD&I
priorities
Strengths & weaknesses
Needs for policy
interventions
Needs for policy interventions
Positioning at the international level
Sectoral R&D strategy
Priority areas for performance contracts

7. The use of peers ties the system to
the field level
7
Research Units
Scientific
excellence
Quality,
relevance & the
research
environment
Performance
of the EvU
and RO
Referees
Subject Panels
Main Panels
Submitted
research
outputs
Disciplinary areas
Fields
Sub-fields
National
performance
in
disciplinary
areas and
scientific
fields
Strategic
information
for R&D
governance
bodies

8. Overview of the funding system
8
RUs
Entry criteria
Assessment
Agreement
Automatic
on renewal
PRFS
Performance
Agreement
Block Grant
Process Funding

9. PRFS at a more detailed level
9
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
Management
& potential
Membership of
research
community
Research
excellence
Research
performance
Information Peer
review
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• -----------
• -----------
• -----------
Societal
relevance
• Star rating
• Strategic
feedback
Money ‘pot’ per
RO type
% % % %
% % % %
% % % %
% % % %
% % % %
Institutiona
l research
funding per
evaluated
RU

10. Where does the money come
from?
10
PRFS
Performance
Agreement
Block Grant
‘Owning’ ministry
Pot1 Pot2 Pot3 Pot4
‘Owning’ ministry
Groups of owning ministries

11. Institutional funding is normally
grounded in the economics of
research
• In economic theory, knowledge is ‘non-rival’ in the sense that it can be
used by many people without being consumed
• If I make a cake and you eat it, I cannot consume it
• But if I make some knowledge, we can both use it
• A rare case where you can have your cake and eat it?
• Knowledge is ‘non-excludable’ in the sense that it is difficult to prevent
people who want it from getting hold of it
• Non-rival, non-excludable goods are ‘public goods’. They cannot be
produced by the market so the state must make them
• In economic reality, however, there is imperfect information, path-
dependency and costs to acquire and use knowledge
• It appears even more costly to absorb science than technology
• Knowledge is useless without know-how
11

12. The rate of subsidy is typically
consistent with the degree of
spillover
12
Basic Applied Experimental
Development
Applications
Engineering …
Market Failure
Spillovers
Risk
Applicability
Subsidy rate in state intervention

13. These ideas about the economics
of knowledge drive funding
behaviour
• Actual rates of institutional funding do vary among countries – but the
broad principle implied by the previous slide is followed
• Public service research performers generally have high rates of
‘subsidy’
• They produce public goods, for which there are few other producers
• They produce information for policy and regulation, for which there are few other
customers than the state
• All types of RO are increasingly becoming ‘hybrid organisations’, so as
ever reality is more complex than theory
• States generally act as ‘intelligent customers’ when they supply
institutional funding – making needs-based buying decisions
themselves, rather than allowing ROs to qualify for such funding
bottom up
13

14. 14
Conventional governance of the research and
innovation system
R&D Institutes
Parliament
Government Policy council
Ministry of
Education
Research
Councils and
Academies
Universities
Other Sectoral
Ministries
Producers:
Firms, farms,
hospitals, etc
Ministry of
Industry
Technology &
Innovation Agencies
Support Programme
Agencies
Programme
Contractors
Instructions, resources
Advice
Results
Horizontal co-ordination and
integration
Level 1
High-level cross-cutting
policy
Level 2
Ministry mission-centred
co-ordination
Level 3
Detailed policy
development, co-
ordination
Level 4
Research and innovation
performers
Key

15. • Governance relies on a ‘sector’ principle – where individual ministries
address the needs of specific ‘sectors’ of society – including ministries’
own research needs
• This system of principals and agents embodies various types of
‘performance agreements’
• Generally these are becoming increasingly explicit
• Based on management by objectives, as opposed to micro-management
• Ministries may choose to share policy implementation (for example, by
using multi-principal agents) but need to allocate their own resources
• Ensuring their own missions are performed
• Requiring the ministries to be ‘intelligent customers’ and to have significant
policymaking capacity
• In general, funding streams and the responsibility for assessment are
clearly separated, especially among different types of RO
15

16. Děkujeme za pozornost!
www.metodika.reformy-msmt.cz