Insight in funding and
evaluation of
applications in open
competition at Danish
private foundations.
Thomas Sinkjær
Head of Talent Programmes
Grants & Prizes
Copenhagen Bioscience Lecture
5 December 2019
Rasmus Larsen
Programme Manager
Biomedicine& Health Science

1. What we do
Rasmus Larsen
2. Can we do it slightly better?
Thomas Sinkjær
• The use of open competitionfundingand peer review in NNF and in general
• The open competitionevaluationprocessin NNF
• What we knowfrom the literature
• Granting the most deservingresearch applications
• Peering the peer review system
• Let us experiment!
3. Questions, thoughts and discussion

Why open competition and peer review
• Idea(l)
• Traditional and generally accepted process for evaluation of scientific quality, in manuscripts and grants
• Fairness, objectivity, legitimacy
• Ensures to support the best research and researchers
• Strategicbenefits
• ‘Bottom-up’ approach
• The unexpected idea
• Flexible mechanism
• Practical
• Makes it possible to receive and assess a large number of diverse research applications
• Allows to reach a large researcher environment
3

NNF open competition in numbers
4
The Novo Nordisk FoundationGrant Report 2018 (https://novonordiskfonden.dk/en/publications/)
GRANT TYPES at NNF
• Project grants
• Investigator grants
• Fellowships
• Large collaborative grants
• Infrastructure grants
• Conferences and symposia
• Prizes

Current status of peer review
5
Chawla (2019),Nature
Grant reveiw in focus (2019),Publons

Transparency (or lack of same)
6
Obscure negotiations
&
Unfair scheming
All the best ideas in the World The usual suspects are funded
The Black Box of grant assessment
Powell (2010), Nature
”For those who do not have direct experience of panel membership,
gaining insights into the ‘black box’ can be difficult, even mystifying.”
Coveney et al. (2017), Res Integr Peer Rev

The NNF open
competition evaluation
process
- and what we know from the literature
7

The open competition evaluation process in NNF
8
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplicationApplicant(s)

The open competition evaluation process in NNF
9
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplication
External peer review
Applicant(s)
Applicantinterview
Stage2 (full) application

The process
10
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings

The process – Applicant(s) & Applications
11
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Necessary and relevant information
• Idea and qualification
• Budgetary appropriateness
At NNF
• Relatively short grant applications (15-25 pages total)
• Use of purpose-build grant programmes (fx career)
• Intuitive and accessible application system
• Identity of committee members is public
• Public announcement of awarded grants
Challenge(s)
• Avoid unnecessary waste of applicants’ time
• Balancing ”need to know” (enough info) vs ”nice to know” (excess burden)
National Health and Medical Research Council
• 70-120 pages applications
• Only 9 pages was the research plan
• 2009:
• 2983 grant proposals submitted
• Median 22 days preparation time
• 180 years of researcher time spent!
Graves et al (2011), BMJ
BURDEN (write+assess)
SUPERFICIALITY
Big Small
Application detail/size
ASSESSMENT QUALITY

The process – Pre-assessors & Scoring
12
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Individual and independent peer review
Challenges
• Balance of grants per pre-assessor vs burden
• Score variation
• Bias (gender, age, institution)
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval
At NNF
• 3, 4 or all (small rounds) committee members per application
• One individual score per pre-assessor per application
• A written comment per pre-assessor per application
• 1 (best) to 6 (worst)
• 1 / 3 / 5 for Stage 1 of 2 stage-assessment round
• Declaration of conflicts of interest (internationalization)
• No application discussion permitted between members
Funder From To
NIH (US) Decimals (1.0,1.1,1.2,… 5.0) Integers(1, 2, …, 5)
NHMRC (AUS) Large scale (1-7) Small scale (A, B,C)
NNF (stage1 of 2) Normal scale (1-6) Small scale (1, 3, 5)
FEASIBILITY
QUALITY
Few Many
Pre-assessors per application
UNCERTAINTY
BURDEN
Snell (2015), PLOS One

The process – Pre-assessors & Scoring
13
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
3 pre-assessors/appl
40-50 appl each
Best score Worst score
NNF,example
Purpose
• Individual and independent peer review
Challenges
• Balance of grants per pre-assessor vs burden
• Score variation
• Bias (gender, age, institution)
At NNF
• 3, 4 or all (small rounds) committee members per application
• One individual score per pre-assessor per application
• A written comment per pre-assessor per application
• 1 (best) to 6 (worst)
• 1 / 3 / 5 for Stage 1 of 2 stage-assessment round
• Declaration of conflicts of interest (internationalization)
• No application discussion permitted between members

The process – Ranking & Assessment Meetings
14
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Joint peer review and discussion
• Final funding recommendation
At NNF
• The grant assessment
• Rank as basis, but not binding, and all applications are up for discussion
• Pre-assessor(s) present their application(s); all are familiar with all applications
• Lower-ranked applications can be brought up
• Large variations must be discussed
• Discussion until funding agreement and budget exhaustion
• Chair directs discussion and ensures that all are heard
• NNF Secretariat members oversee, advise and help the committee
• The committee must adhere to
• Rules of Eligibility (any ineligible members must step outside, literally; no self-funding)
• Mandate Letter with grant purpose, criteria, budget, etc.
• Rules of Procedure - Confidentiality
• Environment sought after
• Pleasant, friendly; good food, etc.
• International members/committees
• Awareness of unconscious bias
Challenges
• Functional and efficient peer group
• Fatigue and randomness
• Bias (group-think, flaw-focus)
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval

The process – Ranking & Assessment Meetings
15
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
NNF,example
44/189 = 23%
Best score Worst score
Thorough discussion
Some/selecteddiscussion
Little or no discussion

The process – Ranking & Assessment Meetings
16
Martin et al (2010), PLOS One
Pre- vs Post-scoring
R01’s, 2983 applications.
Panel discussion had a practical impact on 19.8% of applications.
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Pier et al (2015), WCER Working Paper
Flaw-seeking
4 meetings, 11.% improved, 41%
unchanged, 48% worsened during
meeting.

The process – Ranking & Assessment Meetings
17
Gallo et al (2016), PLOS One
The critical expert
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Area expertise means harsher judgement.
Worse towards junior than senior applicants.
NNF Grant Report,2018
Age and gender bias

The process – Ranking & Assessment Meetings
18
Derrick(2018),Nature
Panel 1 Panel 2
Thegrantproposal
Quality of writing
Simple scoring system, large panels
Innovation and novelty Translation
Budgets are secondary ”It depends on the budget”
Person centered Project centered
Thegrantassessment
Skeptical about use of external peers
Importance of the ‘spokesperson’ setting the trend
Recognize and awareness of ‘off the record’ conversations
Strong chair person essential – good and bad
Important to stay within expertise Okay to go outside of ‘comfort zone’
“Gaming” the score expected “Gaming” the score openly discussed
Young researchers irrelevant or “budget boost” Capacity building essential
Combative atmosphere with strong personalities Collegial, supporting atmosphere
The National Health and Medical Research Councils (NHMRC)
Coveney et al (2017), Res Integr Peer Rev
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings

The process – Ranking & Assessment Meetings
19
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval
Purpose
• Joint peer review and discussion
• Final funding recommendation
At NNF
• The grant assessment
• Rank as basis, but not binding, and all applications are up for discussion
• Pre-assessor(s) present their application(s); all are familiar with all applications
• Lower-ranked applications can be brought up
• Large variations must be discussed
• Discussion until funding agreement and budget exhaustion
• Chair directs discussion and ensures that all are heard
• NNF Secretariat members oversee, advise and help the committee
• The committee must adhere to
• Rules of Eligibility (any ineligible members must step outside, literally; no self-funding)
• Mandate Letter with grant purpose, criteria, budget, etc.
• Rules of Procedure - Confidentiality
• Environment sought after
• Pleasant, friendly; good food, etc.
• International members/committees
• Awareness of unconscious bias
Challenges
• Functional and efficient peer group
• Fatigue and randomness
• Bias (group-think, flaw-focus)

The process – Feedback (recommendation & offer)
20
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
FEASIBILITY (technical, reviewer)
BURDEN (reviewer)
None
Level of feedback
Rank
Stnd comment
Rank
Full comments
Stnd comment
Rank
Purpose
• Explanation for funding decision
• Can allow for rebuttal
• For potential improvement
At NNF
• Normally no feedback
• Interview as (occasional) form of feedback
Challenges
• Technical challenges
• Resource constraints – peers and secretariat
• Balance level of feedback vs review burden
• Making the feedback useful

The process – Feedback (recommendation & offer)
21
Purpose
• Explanation for funding decision
• Can allow for rebuttal
• For potential improvement
At NNF
• Normally no feedback
• Interview as (occasional) form of feedback
Challenges
• Technical challenges
• Resource constraints – peers and secretariat
• Balance level of feedback vs review burden
• Making the feedback useful
Question
% ‘Most useful’or
‘Veryuseful’
Overall how useful was the reviewer feedback you received on
your last grantsubmission?
38.5%
How useful was the reviewer feedback in improving your
grantsmanship?
29.4%
How useful was the reviewer feedback in improving your future
submissions?
34.1%
How useful was the reviewer feedback in informing your future
scientific endeavors in the proposedresearch area?
26.1%
Gallo et al (2020), In preparation
Gallo et al (2019), Sci Eng Ethics
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
0 10 20 30 40 50 60 70 80 90
Innovation
Risk
PI track record
RESPONDENTS (%)
CRITERIA
What is important in feedback
Applicants Reviewers
Adapted fromGallo et al (2018), Environ Sys Decis
P<0.001
P<0.001
P<0.001

Recap
22

Recap – what we do
23
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplicationApplicant(s)
”Ultimately, peer review is going to be an imperfect process.
But we are not doing a bad job.”
Chairman(anonymous),
in Powell (2010), Nature