Decisions: Rethinking Review, Allocation and Evaluation. Wilhelm Krull, Secretary General, Volkswagen Foundation

1. Wilhelm Krull
Decisions:
Rethinking Review, Allocation,
and Evaluation
New Frontiers for Research on Research, Session 5
London, 30 September 2019

2. I. Rethinking Peer Review
„The peer review process is deeply influenced by who gets
asked to serve as a panelist and what viewpoints and
intellectual habitus those individuals bring to the table. Biases
are unavoidable.”
M. Lamont, How Professors Think, 2009, p. 242f.
2

3. a) Structural Problems and Conflicting Interests
Expectation of applicants:
 Timely decisions, transparent
processes
 Comprehensive feedback,
acceptable funding rate
 Support for unconventional
ideas
Strategy of the Foundation:
 Funding of interdisciplinary, risky
projects
 Funding beyond mainstream
 Driving force, also concerning
structures and procedures
Current developments at the
Foundation:
 Increasing number of applications
 Changing role and
responsibilities of programme
managers, e.g. internal
preselection
Peer Review System:
 Increasing overload with
declining response rates (4:1),
 Pronounced adaptive behaviour
of reviewers to standardized
procedures, risk of bias
3

4. b) Subjective Influences: Cognitive Bias
https://royalsociety.org/topics-policy/publications/2015/unconscious-bias/
4

5. c) Group or Social Dynamics
 Biases such as group
and/or social dynamics
within review panels.
 Potential lack of
fairness and reliability
of peer review.
“Platzhirsch-Effekt”
(Top-Dog Effect)
Source: Liviatour through Wikimedia commons
5

6. 6
d) Latent Risk Aversion
 Tendency towards
established and already
proven ideas and
methodologies.
 Potential originality-
inhibiting tendency
of peer review.

7. e) A Cultural Problem: Dealing with Failures
In our current academic landscape, only positive results are valued and
failure is stigmatized. However:
7
“Science progresses not
because of simple and
charming serendipity, but
because of bruising accidents
and crashing failures and a lot
of tough repair work.”
Stuart Firestein, Failure: Why
Science is so Successful, 46.
Consequence: Applicants (predominantly early career researchers) are
discouraged to take original and risky approaches, and they are often advised
to rather stick to conventional approaches.

8. f) Predictive Validity of Past Performance in Peer Review
Source: art.com
8

9. 9
 Overburdening of leading researchers
through multiple requests.
 Involvement of second choice
experts in reviews processes.
 Potential decrease of quality of
assessments and reviews.
 Overburdening of female reviewers due
to quota of women for the constitution of
panels.
g) Overburdening of Leading (and Female) Researchers

10. h) Defining the Right Criteria and Assessment Methods
10

11. Experiments and Experiences with Alternative
Assessment Methods – Four Examples
a) Selecting Highly Original and Innovative Projects
11
Application and Selection
Procedure
 3 pages project outline.
 Anonymous application (no
information about applicant and
institution is being forwarded to
the review panel), application
limited to 1.000 +/- 100 words,
incl. optional references, no self
citation.
 1-page self assessment.

12. Self Assessment Sheet
12

13.  Each panel member has one “Joker” (i. e.
wildcard) to vote for an application against
the majority vote.
 So far, on avarage, two “Jokers” per
meeting were used, each panel consisted of
seven to nine panel members.
13
The Funding Joker (“Wildcard”)

14. Randomization to complement peer review
14

15. Motivation for Randomization
 Ensures diversity (only a limited range of subjects/expertise can be represented
in the jury).
 Avoids implicit bias – the lot is “blind”.
 Alternative to “difficult decisions” which cannot be made solely on the basis of
scholarly arguments.
 Face-saving for applicants who will not receive funding.
Moreover – in general:
 Decisions about scientifically similarly ranked applications against the backdrop
of budget constraints.
 Increases chances for minority topics vs. overfunding of established fields and
disciplines.
15

16. b) Selecting the Next Generation of Researchers
16

17. The “Freigeist-Kollegium”
 About 40 international researchers from various disciplines, serving for 3
rounds of applications.
 Each member comparatively reviews 3 to 7 applications according to a
“Quick Assessment” sheet.
 Usually, the “Kollegium” assorts a shortlist of 35 to 40 applications from a
total of 90 to 100.
17

18. 18
Motivation for the Constitution of a Review College
 Needs less reviewers than a process based on individual written
assessments.
 Provides enough feedback to base decisions on scholarly
arguments.
 Enables an interdisciplinary and multi-perspectival review process.
 Ensures that reviewers are familiar with the specific conditions and
the rationale/strategy of the respective funding initiative.
 Can provide a basis for positive group dynamics among the
members of the college.

19. Freigeist-Fellowships – Quick Assessments
 Feedback for rejected applicants.
 Basis for structuring the panel
meeting.
19

20. c) Two-Step Selection Processes
Wherever possible the Volkswagen Foundation employs two-step
selection processes to …
 … concentrate the review process on the most promising
candidates,
 … enable a presentation of the candidate(s) and/or principal
investigator(s) in person as well as a thorough discussion of the
project with the reviewers,
 … speed up the decision-making process for those with no prospect
of success (Secretary General has a mandate to reject applications
which were not selected for the second phase of the process),
 … avoid overburdening of reviewers.
20

21. d) Differentiating Criteria: Projects vs. Funding of People
scholarly solidity, state-of-the-art
significant contribution to the field
and beyond
originality
high risk-high gain, readiness to
take risks
self-reflectivity, perspectival
suppleness
practical relevance
meeting the thematic agendaadequate budget & time
equality of partners
academic environment
personal qualities
project design
conceptualization and concise
presentation
Project Funding Funding of People
21

22. II. Rethinking Allocation
The Volkswagen Foundation’s Approach to Research Funding
 Curious, creative, courageous, concerned, and connected.
 Attentive even to weak signals to anticipate change.
 Convinced that small steps can lead to systemic change.
 Encourage risk-taking and experimenting in unknown territories.
 Act responsibly vis-à-vis our partners, in particular early career
researchers.
 Establish a high trust culture of creativity.
 Set ambitious goals and evaluate externally how we have done
against the objectives.
22

23. Validation Processes for New Funding Initiatives:
General Criteria (I)
 Funding of international cooperations
 Readiness to take risks
 Investment in future potentials
 Initiation of innovative of processes, structures, or
methods beyond mainstream.
23

24. Validation Processes for New Funding Initiatives:
General Criteria (II)
Complementary and anti-cyclical funding in relation to
public funding
 medium-scale projects vs. clusters
 long-term perspectives vs. short-term project based
funding
 focus on creative minds and personalities
 beyond direct application needs/relevance aspects.
24

25. Validation of New Funding Initiatives:
Specific Criteria (I)
 Opportunities and relevance:
Originality and innovativeness, increase of knowledge,
impact on field of research, inter- and transdisciplinarity,
impact on society.
 Potential grant recipients and focus groups:
Size of research community, career perspectives for
young researchers, structural impact on institutions
and/or processes.
25

26. Validation of New Funding Initiatives:
Specific Criteria (II)
 Research funding landscape:
Activities of other national or international research
funders, complementarity and/or potential for
cooperations.
 Strategy and funding portfolio:
Goal achievement, unique selling point, relation to other
initiatives, financial implications.
26

27. III. Rethinking Evaluation
Fundamental Issues of Research Policy Making
 Enormous shift from core funding to project based funding:
 Overburdending of the peer review system.
 Short term and project-based thinking.
 Lack of time and space for creativity and fundamentally new ideas.
 Less breakthroughs (Öquist/Benner 2012, max. 40 % project-based
funding).
 Multiscale and mulitlevel system of evaluations:
 Overburdening of experts.
 Conflicting or even wrong incentives.
 Time and resource consuming processes.
 Lack of mutual trust and creativity.
27

28. Evaluations in Research Funding Organisations
 Evaluations are an instrument of critical self-reflection as well as an
indicator of good governance.
 Research funding organisations in particular should establish
processes of critical (self-)reflection and exchange with their
respective communities.
 External experts and perspectives are essential.
 Evaluation results must not be the sole basis for decision-making
on funding strategies.
 Philanthropic institutions should be subjected to evaluation on a
medium-term basis (8 to 10 years, ‟effective philanthropy”).
28

29. The Evaluation Pyramid
29

30. A Culture of Creativity Should Make a Difference
Creativity begins with …
 Seeing things differently
 Providing a stimulating
environment
 Encouraging risk-taking
 Allowing for failures
 Establishing high-trust
relationships
 It rests on the pillars of
research-friendly
governance structures and
effective management.
La Clairvoyance, Rene Magritte, 1936 © VG Bild-Kunst, Bonn 2017
30
30

31. Thank you very much
for your attention!
Dr. Wilhelm Krull
Secretary General
Volkswagen Foundation
Phone: +49 (0)511 8381 215
Fax: +49 (0)511 8381 235
E-Mail: krull@volkswagenstiftung.de
Internet: www.volkswagenstiftung.de