The document discusses the need to reform the Czech Republic's research evaluation methodology to improve performance and competitiveness. It outlines the objectives to develop a new national evaluation methodology that provides strategic information to all levels of the research system and informs institutional funding. The new methodology is based on several key principles: it reflects strategic policy objectives; uses comprehensive peer review of research units; and aims to be fair, cost-effective, and anchored at the international level. The methodology assesses research excellence, performance, societal relevance and institutional potential through peer judgements rather than calculations. Evaluation results will inform funding allocation and provide strategic feedback to research organizations and policymakers.

1. The new Evaluation
Methodology
Erik Arnold & Bea Mahieu -
Technopolis Group
www.metodika.reformy-msmt.cz

2. Why reform the evaluation
methodology?
• Need to improve the performance of the research and innovation
system
• Outputs and competitiveness
• Formative as well as summative information needed
• Cope with a need for competitiveness at the international level
• Train for the Olympics, not the national games
• Need for information at both policy and performer levels
• Address limitations of the Coffee Mill, including
• Unstable outcomes
• Lack of formative information
• Inability to contribute to wider evaluations or policymaking
• Not anchored at the international level
• All numbers, no judgement
1

3. The EM in its historical context
• Evolution after the reform of 2008:
• Progressive restriction of the scope: research outputs only
• Reduction of the complexity of performance to an overly simple category of outputs
• Lack of consideration for disciplinary differences and for the missions of ROs
• Narrowing of the function of evaluation: only as component of the PRFS
• Concept of evaluation as part of a policy cycle providing strategic information is not
perceived; it provides information that is at the best of limited relevance
• Increasing breadth of coverage: from funding bodies to individual researchers
• Metodika 2013-2015: an improvement but still focused exclusively on outputs
• Result:
• Evaluation is perceived as counting outputs = points = funding
• It constitutes the key factor for R&D management throughout the entire system
• It is detached from any discourse on policy and strategy related to the national
R&D system
2

4. Our mandate
• Objectives: to develop an evaluation methodology
• Conducted on a national basis
• Providing strategic information for the actors at all levels in the R&D system
• Evaluation results inform the institutional funding system (PRFS)
• The expectations:
• A peer-review evaluation process
• Fulfil formative and summative functions
• Cover outputs, impacts, and institutional projections of research development
• Take into consideration the different missions of research organisations and the
field specifics
• Evaluation processes resistant to clientelism and conflicts of interests
• Take into account ‘gaming’
• Total costs should not exceed 1% of public institutional support for R&D in a five-
year time period
3

5. The key principles
• The EM
• Reflects the strategic policy objectives for the R&D system
• Functions = to act as source for strategic information & directly inform public
institutional funding for research
• The evaluation
• Is comprehensive: all dimensions of the research activities and its outputs,
outcomes and impact
• Is at the level of field-defined Research Unit (RU) within an Evaluated Unit, i.e. a
research organisation or in the case of the public HEI, a faculty
• Covers all research organisations of a critical size, on a voluntary basis
• Is a process of informed peer review, based on a mix of appropriate quantitative
and qualitative data to support their professional judgement
• Is a fair and egalitarian system: a single framework for assessment while allowing
for a reasonable level of field- and RO typology-specific variations
• The minimum possible cost and burden to deliver a robust and defensible process
4

6. Policy development and
implementation are at the heart of
the new methodology
5
Assessment Funding
Policymaking
Governance

7. Building up from the fundamental
unit of assessment: the Research
Unit
6
Research unit
(RU)
Individual
researchers
Evaluated Unit
(EvU)
Research Organisation
Research Institute /
University Faculty
R&D Governance
Bodies & Agencies
Scientific
fields
National R&D
System
(International)
Structure of
Science
Disciplinary
areas

8. Assessment is based on peer
judgements, not calculation
7
Self assessment
Bibliometrics
Best outputs
• Institutional management
and development potential
• Membership of the (world)
research community
• Scientific research
excellence
• Research performance
• Societal relevance
EvU/RO
Governance
Policymakers
Peer judgements

9. Some of what the ‘users’ get
EvU/RO Governance Policymakers
Institutional and research
strategy
Institutional and HR
management
Positioning at national level
International positioning
Models of good practice
Strengths and weaknesses
Needs for policy interventions
Sectoral R&D strategies
International positioning
Priorities for performance
contracts
National research strategy
Alignment with RDI priorities
International positioning
Needs for policy intervention
8

10. The use of peers ties the system to
the field level
9
Research Units
Scientific
excellence
Quality,
relevance & the
research
environment
Performance
of the EvU
and RO
Referees
Subject Panels
Main Panels
Submitted
research
outputs
Disciplinary areas
Fields
Sub-fields
National
performance
in
disciplinary
areas and
scientific
fields
Strategic
information
for R&D
governance
bodies

11. Overview of the funding system
10
RUs
Entry criteria
Assessment
Agreement
Automatic
on renewal
PRFS
Performance Agreement
Block Grant
Process Funding

12. PRFS at a more detailed level
11
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
• -----------
Management
& potential
Membership of
research
community
Research
excellence
Research
performance
Information Peer
review
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• Star rating
• Strategic
feedback
• -----------
• -----------
• -----------
Societal
relevance
• Star rating
• Strategic
feedback
Money ‘pot’ per RO
type and assessment
criterion
% % % %
% % % %
% % % %
% % % %
% % % %
Institutiona
l research
funding per
evaluated
RU

13. Benefits
• Performance improvements
• Self-evaluation and interaction with the assessment generates a learning process
• Evaluation Method encourages improved research management in the
organisations
• Better positioning for international collaboration, eg Framework Programme
• International inputs to the process
• Providing an ‘Olympic’ benchmark
• Independence and objectivity of peer review
• Information for both policy and organisational levels, to underpin
strategy
• Funding – giving incentives but also a basis for planning
• Some stability
• Rewards for past performance
• Strategic ‘space’ for future investment
12

14. Some issues to tackle:
Governance
• Governance
• Increasing capacities, especially in sector ministries
• Institutionalising the evaluation methodology
• Reducing uncertainties about the research budget as a whole through
multi-year planning
• Process for evolving % allocations to performance and funding
categories, eg rising PRFS and contractual components over time
• Research Organisation categories
• Assessment dimensions
• Balance among the three components
• Stability
• PRFS
• Performance contracting
13

15. Some issues to tackle: Policy
• Periodicity – adjusting frequency as system performance improves
• Some principles
• ASCR and the universities in the same pot?
• Defining FTEs and untangling research from teaching
• Defining ‘Research Organisation’ and eligibility rules
• Widening the scope for applied inputs without reviving ‘gaming’
• Timetable for implementation – moving fast enough to address the
underlying needs
• Minimise the time we have to live with a transitional system
• Exploit the great interest in the debate about assessment and funding within the
research community
• Use the resources and momentum build up in the IPN project
• Carpe diem!!
14

16. Thank you for your
attention!
www.metodika.reformy-msmt.cz