SlideShare a Scribd company logo
1 of 23
Download to read offline
Insight in funding and
evaluation of
applications in open
competition at Danish
private foundations.
Thomas Sinkjær
Head of Talent Programmes
Grants & Prizes
Copenhagen Bioscience Lecture
5 December 2019
Rasmus Larsen
Programme Manager
Biomedicine& Health Science
1. What we do
Rasmus Larsen
2. Can we do it slightly better?
Thomas Sinkjær
• The use of open competitionfundingand peer review in NNF and in general
• The open competitionevaluationprocessin NNF
• What we knowfrom the literature
• Granting the most deservingresearch applications
• Peering the peer review system
• Let us experiment!
3. Questions, thoughts and discussion
Why open competition and peer review
• Idea(l)
• Traditional and generally accepted process for evaluation of scientific quality, in manuscripts and grants
• Fairness, objectivity, legitimacy
• Ensures to support the best research and researchers
• Strategicbenefits
• ‘Bottom-up’ approach
• The unexpected idea
• Flexible mechanism
• Practical
• Makes it possible to receive and assess a large number of diverse research applications
• Allows to reach a large researcher environment
3
NNF open competition in numbers
4
The Novo Nordisk FoundationGrant Report 2018 (https://novonordiskfonden.dk/en/publications/)
GRANT TYPES at NNF
• Project grants
• Investigator grants
• Fellowships
• Large collaborative grants
• Infrastructure grants
• Conferences and symposia
• Prizes
Current status of peer review
5
Chawla (2019),Nature
Grant reveiw in focus (2019),Publons
Transparency (or lack of same)
6
Obscure negotiations
&
Unfair scheming
All the best ideas in the World The usual suspects are funded
The Black Box of grant assessment
Powell (2010), Nature
”For those who do not have direct experience of panel membership,
gaining insights into the ‘black box’ can be difficult, even mystifying.”
Coveney et al. (2017), Res Integr Peer Rev
The NNF open
competition evaluation
process
- and what we know from the literature
7
The open competition evaluation process in NNF
8
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplicationApplicant(s)
The open competition evaluation process in NNF
9
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplication
External peer review
Applicant(s)
Applicantinterview
Stage2 (full) application
The process
10
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
The process – Applicant(s) & Applications
11
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Necessary and relevant information
• Idea and qualification
• Budgetary appropriateness
At NNF
• Relatively short grant applications (15-25 pages total)
• Use of purpose-build grant programmes (fx career)
• Intuitive and accessible application system
• Identity of committee members is public
• Public announcement of awarded grants
Challenge(s)
• Avoid unnecessary waste of applicants’ time
• Balancing ”need to know” (enough info) vs ”nice to know” (excess burden)
National Health and Medical Research Council
• 70-120 pages applications
• Only 9 pages was the research plan
• 2009:
• 2983 grant proposals submitted
• Median 22 days preparation time
• 180 years of researcher time spent!
Graves et al (2011), BMJ
BURDEN (write+assess)
SUPERFICIALITY
Big Small
Application detail/size
ASSESSMENT QUALITY
The process – Pre-assessors & Scoring
12
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Individual and independent peer review
Challenges
• Balance of grants per pre-assessor vs burden
• Score variation
• Bias (gender, age, institution)
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval
At NNF
• 3, 4 or all (small rounds) committee members per application
• One individual score per pre-assessor per application
• A written comment per pre-assessor per application
• 1 (best) to 6 (worst)
• 1 / 3 / 5 for Stage 1 of 2 stage-assessment round
• Declaration of conflicts of interest (internationalization)
• No application discussion permitted between members
Funder From To
NIH (US) Decimals (1.0,1.1,1.2,… 5.0) Integers(1, 2, …, 5)
NHMRC (AUS) Large scale (1-7) Small scale (A, B,C)
NNF (stage1 of 2) Normal scale (1-6) Small scale (1, 3, 5)
FEASIBILITY
QUALITY
Few Many
Pre-assessors per application
UNCERTAINTY
BURDEN
Snell (2015), PLOS One
The process – Pre-assessors & Scoring
13
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
3 pre-assessors/appl
40-50 appl each
Best score Worst score
NNF,example
Purpose
• Individual and independent peer review
Challenges
• Balance of grants per pre-assessor vs burden
• Score variation
• Bias (gender, age, institution)
At NNF
• 3, 4 or all (small rounds) committee members per application
• One individual score per pre-assessor per application
• A written comment per pre-assessor per application
• 1 (best) to 6 (worst)
• 1 / 3 / 5 for Stage 1 of 2 stage-assessment round
• Declaration of conflicts of interest (internationalization)
• No application discussion permitted between members
The process – Ranking & Assessment Meetings
14
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Purpose
• Joint peer review and discussion
• Final funding recommendation
At NNF
• The grant assessment
• Rank as basis, but not binding, and all applications are up for discussion
• Pre-assessor(s) present their application(s); all are familiar with all applications
• Lower-ranked applications can be brought up
• Large variations must be discussed
• Discussion until funding agreement and budget exhaustion
• Chair directs discussion and ensures that all are heard
• NNF Secretariat members oversee, advise and help the committee
• The committee must adhere to
• Rules of Eligibility (any ineligible members must step outside, literally; no self-funding)
• Mandate Letter with grant purpose, criteria, budget, etc.
• Rules of Procedure - Confidentiality
• Environment sought after
• Pleasant, friendly; good food, etc.
• International members/committees
• Awareness of unconscious bias
Challenges
• Functional and efficient peer group
• Fatigue and randomness
• Bias (group-think, flaw-focus)
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval
The process – Ranking & Assessment Meetings
15
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
NNF,example
44/189 = 23%
Best score Worst score
Thorough discussion
Some/selecteddiscussion
Little or no discussion
The process – Ranking & Assessment Meetings
16
Martin et al (2010), PLOS One
Pre- vs Post-scoring
R01’s, 2983 applications.
Panel discussion had a practical impact on 19.8% of applications.
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Pier et al (2015), WCER Working Paper
Flaw-seeking
4 meetings, 11.% improved, 41%
unchanged, 48% worsened during
meeting.
The process – Ranking & Assessment Meetings
17
Gallo et al (2016), PLOS One
The critical expert
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Area expertise means harsher judgement.
Worse towards junior than senior applicants.
NNF Grant Report,2018
Age and gender bias
The process – Ranking & Assessment Meetings
18
Derrick(2018),Nature
Panel 1 Panel 2
Thegrantproposal
Quality of writing
Simple scoring system, large panels
Innovation and novelty Translation
Budgets are secondary ”It depends on the budget”
Person centered Project centered
Thegrantassessment
Skeptical about use of external peers
Importance of the ‘spokesperson’ setting the trend
Recognize and awareness of ‘off the record’ conversations
Strong chair person essential – good and bad
Important to stay within expertise Okay to go outside of ‘comfort zone’
“Gaming” the score expected “Gaming” the score openly discussed
Young researchers irrelevant or “budget boost” Capacity building essential
Combative atmosphere with strong personalities Collegial, supporting atmosphere
The National Health and Medical Research Councils (NHMRC)
Coveney et al (2017), Res Integr Peer Rev
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
The process – Ranking & Assessment Meetings
19
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
Individual review Group review
Independence Width
Expertise Concensus
Rules-guided Group dynamics
Van Arensbergenet al. (2014), Res Eval
Purpose
• Joint peer review and discussion
• Final funding recommendation
At NNF
• The grant assessment
• Rank as basis, but not binding, and all applications are up for discussion
• Pre-assessor(s) present their application(s); all are familiar with all applications
• Lower-ranked applications can be brought up
• Large variations must be discussed
• Discussion until funding agreement and budget exhaustion
• Chair directs discussion and ensures that all are heard
• NNF Secretariat members oversee, advise and help the committee
• The committee must adhere to
• Rules of Eligibility (any ineligible members must step outside, literally; no self-funding)
• Mandate Letter with grant purpose, criteria, budget, etc.
• Rules of Procedure - Confidentiality
• Environment sought after
• Pleasant, friendly; good food, etc.
• International members/committees
• Awareness of unconscious bias
Challenges
• Functional and efficient peer group
• Fatigue and randomness
• Bias (group-think, flaw-focus)
The process – Feedback (recommendation & offer)
20
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
FEASIBILITY (technical, reviewer)
BURDEN (reviewer)
None
Level of feedback
Rank
Stnd comment
Rank
Full comments
Stnd comment
Rank
Purpose
• Explanation for funding decision
• Can allow for rebuttal
• For potential improvement
At NNF
• Normally no feedback
• Interview as (occasional) form of feedback
Challenges
• Technical challenges
• Resource constraints – peers and secretariat
• Balance level of feedback vs review burden
• Making the feedback useful
The process – Feedback (recommendation & offer)
21
Purpose
• Explanation for funding decision
• Can allow for rebuttal
• For potential improvement
At NNF
• Normally no feedback
• Interview as (occasional) form of feedback
Challenges
• Technical challenges
• Resource constraints – peers and secretariat
• Balance level of feedback vs review burden
• Making the feedback useful
Question
% ‘Most useful’or
‘Veryuseful’
Overall how useful was the reviewer feedback you received on
your last grantsubmission?
38.5%
How useful was the reviewer feedback in improving your
grantsmanship?
29.4%
How useful was the reviewer feedback in improving your future
submissions?
34.1%
How useful was the reviewer feedback in informing your future
scientific endeavors in the proposedresearch area?
26.1%
Gallo et al (2020), In preparation
Gallo et al (2019), Sci Eng Ethics
Applicant(s)& Applications
Funding recommendation
and offer
Pre-assessors& Scoring
Ranking & Meetings
0 10 20 30 40 50 60 70 80 90
Innovation
Risk
PI track record
RESPONDENTS (%)
CRITERIA
What is important in feedback
Applicants Reviewers
Adapted fromGallo et al (2018), Environ Sys Decis
P<0.001
P<0.001
P<0.001
Recap
22
Recap – what we do
23
Pre-assessors Scoreand Ranking
Committee meeting
Final
recommendation
GrantofferApplicationApplicant(s)
”Ultimately, peer review is going to be an imperfect process.
But we are not doing a bad job.”
Chairman(anonymous),
in Powell (2010), Nature

More Related Content

Recently uploaded

PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCE
PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCEPLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCE
PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCETALAPATI ARUNA CHENNA VYDYANAD
 
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptx
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptxBiochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptx
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptxjayabahari688
 
Continuum emission from within the plunging region of black hole discs
Continuum emission from within the plunging region of black hole discsContinuum emission from within the plunging region of black hole discs
Continuum emission from within the plunging region of black hole discsSérgio Sacani
 
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxPlasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxmuralinath2
 
A Giant Impact Origin for the First Subduction on Earth
A Giant Impact Origin for the First Subduction on EarthA Giant Impact Origin for the First Subduction on Earth
A Giant Impact Origin for the First Subduction on EarthSérgio Sacani
 
TEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfTEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfmarcuskenyatta275
 
Detectability of Solar Panels as a Technosignature
Detectability of Solar Panels as a TechnosignatureDetectability of Solar Panels as a Technosignature
Detectability of Solar Panels as a TechnosignatureSérgio Sacani
 
GBSN - Microbiology Lab (Compound Microscope)
GBSN - Microbiology Lab (Compound Microscope)GBSN - Microbiology Lab (Compound Microscope)
GBSN - Microbiology Lab (Compound Microscope)Areesha Ahmad
 
Lubrication System in forced feed system
Lubrication System in forced feed systemLubrication System in forced feed system
Lubrication System in forced feed systemADB online India
 
In-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptxIn-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptxMAGOTI ERNEST
 
Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!University of Hertfordshire
 
Mining Activity and Investment Opportunity in Myanmar.pptx
Mining Activity and Investment Opportunity in Myanmar.pptxMining Activity and Investment Opportunity in Myanmar.pptx
Mining Activity and Investment Opportunity in Myanmar.pptxKyawThanTint
 
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptAerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptsreddyrahul
 
Factor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary GlandFactor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary GlandRcvets
 
GBSN - Microbiology Lab (Microbiology Lab Safety Procedures)
GBSN -  Microbiology Lab (Microbiology Lab Safety Procedures)GBSN -  Microbiology Lab (Microbiology Lab Safety Procedures)
GBSN - Microbiology Lab (Microbiology Lab Safety Procedures)Areesha Ahmad
 
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Lahari
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana LahariERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Lahari
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Laharimuralinath2
 
Hemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. MuralinathHemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. Muralinathmuralinath2
 
Manganese‐RichSandstonesasanIndicatorofAncientOxic LakeWaterConditionsinGale...
Manganese‐RichSandstonesasanIndicatorofAncientOxic  LakeWaterConditionsinGale...Manganese‐RichSandstonesasanIndicatorofAncientOxic  LakeWaterConditionsinGale...
Manganese‐RichSandstonesasanIndicatorofAncientOxic LakeWaterConditionsinGale...Sérgio Sacani
 
Isolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptxIsolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptxGOWTHAMIM22
 
Erythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C KalyanErythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C Kalyanmuralinath2
 

Recently uploaded (20)

PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCE
PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCEPLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCE
PLANT DISEASE MANAGEMENT PRINCIPLES AND ITS IMPORTANCE
 
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptx
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptxBiochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptx
Biochemistry and Biomolecules - Science - 9th Grade by Slidesgo.pptx
 
Continuum emission from within the plunging region of black hole discs
Continuum emission from within the plunging region of black hole discsContinuum emission from within the plunging region of black hole discs
Continuum emission from within the plunging region of black hole discs
 
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxPlasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
 
A Giant Impact Origin for the First Subduction on Earth
A Giant Impact Origin for the First Subduction on EarthA Giant Impact Origin for the First Subduction on Earth
A Giant Impact Origin for the First Subduction on Earth
 
TEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfTEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdf
 
Detectability of Solar Panels as a Technosignature
Detectability of Solar Panels as a TechnosignatureDetectability of Solar Panels as a Technosignature
Detectability of Solar Panels as a Technosignature
 
GBSN - Microbiology Lab (Compound Microscope)
GBSN - Microbiology Lab (Compound Microscope)GBSN - Microbiology Lab (Compound Microscope)
GBSN - Microbiology Lab (Compound Microscope)
 
Lubrication System in forced feed system
Lubrication System in forced feed systemLubrication System in forced feed system
Lubrication System in forced feed system
 
In-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptxIn-pond Race way systems for Aquaculture (IPRS).pptx
In-pond Race way systems for Aquaculture (IPRS).pptx
 
Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!Quantifying Artificial Intelligence and What Comes Next!
Quantifying Artificial Intelligence and What Comes Next!
 
Mining Activity and Investment Opportunity in Myanmar.pptx
Mining Activity and Investment Opportunity in Myanmar.pptxMining Activity and Investment Opportunity in Myanmar.pptx
Mining Activity and Investment Opportunity in Myanmar.pptx
 
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptAerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
 
Factor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary GlandFactor Causing low production and physiology of mamary Gland
Factor Causing low production and physiology of mamary Gland
 
GBSN - Microbiology Lab (Microbiology Lab Safety Procedures)
GBSN -  Microbiology Lab (Microbiology Lab Safety Procedures)GBSN -  Microbiology Lab (Microbiology Lab Safety Procedures)
GBSN - Microbiology Lab (Microbiology Lab Safety Procedures)
 
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Lahari
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana LahariERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Lahari
ERTHROPOIESIS: Dr. E. Muralinath & R. Gnana Lahari
 
Hemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. MuralinathHemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. Muralinath
 
Manganese‐RichSandstonesasanIndicatorofAncientOxic LakeWaterConditionsinGale...
Manganese‐RichSandstonesasanIndicatorofAncientOxic  LakeWaterConditionsinGale...Manganese‐RichSandstonesasanIndicatorofAncientOxic  LakeWaterConditionsinGale...
Manganese‐RichSandstonesasanIndicatorofAncientOxic LakeWaterConditionsinGale...
 
Isolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptxIsolation of AMF by wet sieving and decantation method pptx
Isolation of AMF by wet sieving and decantation method pptx
 
Erythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C KalyanErythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C Kalyan
 

Featured

PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 

Featured (20)

PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 

Insight in funding and evaluation of applications in open competition at Danish private foundations - Rasmus Larsen

  • 1. Insight in funding and evaluation of applications in open competition at Danish private foundations. Thomas Sinkjær Head of Talent Programmes Grants & Prizes Copenhagen Bioscience Lecture 5 December 2019 Rasmus Larsen Programme Manager Biomedicine& Health Science
  • 2. 1. What we do Rasmus Larsen 2. Can we do it slightly better? Thomas Sinkjær • The use of open competitionfundingand peer review in NNF and in general • The open competitionevaluationprocessin NNF • What we knowfrom the literature • Granting the most deservingresearch applications • Peering the peer review system • Let us experiment! 3. Questions, thoughts and discussion
  • 3. Why open competition and peer review • Idea(l) • Traditional and generally accepted process for evaluation of scientific quality, in manuscripts and grants • Fairness, objectivity, legitimacy • Ensures to support the best research and researchers • Strategicbenefits • ‘Bottom-up’ approach • The unexpected idea • Flexible mechanism • Practical • Makes it possible to receive and assess a large number of diverse research applications • Allows to reach a large researcher environment 3
  • 4. NNF open competition in numbers 4 The Novo Nordisk FoundationGrant Report 2018 (https://novonordiskfonden.dk/en/publications/) GRANT TYPES at NNF • Project grants • Investigator grants • Fellowships • Large collaborative grants • Infrastructure grants • Conferences and symposia • Prizes
  • 5. Current status of peer review 5 Chawla (2019),Nature Grant reveiw in focus (2019),Publons
  • 6. Transparency (or lack of same) 6 Obscure negotiations & Unfair scheming All the best ideas in the World The usual suspects are funded The Black Box of grant assessment Powell (2010), Nature ”For those who do not have direct experience of panel membership, gaining insights into the ‘black box’ can be difficult, even mystifying.” Coveney et al. (2017), Res Integr Peer Rev
  • 7. The NNF open competition evaluation process - and what we know from the literature 7
  • 8. The open competition evaluation process in NNF 8 Pre-assessors Scoreand Ranking Committee meeting Final recommendation GrantofferApplicationApplicant(s)
  • 9. The open competition evaluation process in NNF 9 Pre-assessors Scoreand Ranking Committee meeting Final recommendation GrantofferApplication External peer review Applicant(s) Applicantinterview Stage2 (full) application
  • 10. The process 10 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings
  • 11. The process – Applicant(s) & Applications 11 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Purpose • Necessary and relevant information • Idea and qualification • Budgetary appropriateness At NNF • Relatively short grant applications (15-25 pages total) • Use of purpose-build grant programmes (fx career) • Intuitive and accessible application system • Identity of committee members is public • Public announcement of awarded grants Challenge(s) • Avoid unnecessary waste of applicants’ time • Balancing ”need to know” (enough info) vs ”nice to know” (excess burden) National Health and Medical Research Council • 70-120 pages applications • Only 9 pages was the research plan • 2009: • 2983 grant proposals submitted • Median 22 days preparation time • 180 years of researcher time spent! Graves et al (2011), BMJ BURDEN (write+assess) SUPERFICIALITY Big Small Application detail/size ASSESSMENT QUALITY
  • 12. The process – Pre-assessors & Scoring 12 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Purpose • Individual and independent peer review Challenges • Balance of grants per pre-assessor vs burden • Score variation • Bias (gender, age, institution) Individual review Group review Independence Width Expertise Concensus Rules-guided Group dynamics Van Arensbergenet al. (2014), Res Eval At NNF • 3, 4 or all (small rounds) committee members per application • One individual score per pre-assessor per application • A written comment per pre-assessor per application • 1 (best) to 6 (worst) • 1 / 3 / 5 for Stage 1 of 2 stage-assessment round • Declaration of conflicts of interest (internationalization) • No application discussion permitted between members Funder From To NIH (US) Decimals (1.0,1.1,1.2,… 5.0) Integers(1, 2, …, 5) NHMRC (AUS) Large scale (1-7) Small scale (A, B,C) NNF (stage1 of 2) Normal scale (1-6) Small scale (1, 3, 5) FEASIBILITY QUALITY Few Many Pre-assessors per application UNCERTAINTY BURDEN Snell (2015), PLOS One
  • 13. The process – Pre-assessors & Scoring 13 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings 3 pre-assessors/appl 40-50 appl each Best score Worst score NNF,example Purpose • Individual and independent peer review Challenges • Balance of grants per pre-assessor vs burden • Score variation • Bias (gender, age, institution) At NNF • 3, 4 or all (small rounds) committee members per application • One individual score per pre-assessor per application • A written comment per pre-assessor per application • 1 (best) to 6 (worst) • 1 / 3 / 5 for Stage 1 of 2 stage-assessment round • Declaration of conflicts of interest (internationalization) • No application discussion permitted between members
  • 14. The process – Ranking & Assessment Meetings 14 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Purpose • Joint peer review and discussion • Final funding recommendation At NNF • The grant assessment • Rank as basis, but not binding, and all applications are up for discussion • Pre-assessor(s) present their application(s); all are familiar with all applications • Lower-ranked applications can be brought up • Large variations must be discussed • Discussion until funding agreement and budget exhaustion • Chair directs discussion and ensures that all are heard • NNF Secretariat members oversee, advise and help the committee • The committee must adhere to • Rules of Eligibility (any ineligible members must step outside, literally; no self-funding) • Mandate Letter with grant purpose, criteria, budget, etc. • Rules of Procedure - Confidentiality • Environment sought after • Pleasant, friendly; good food, etc. • International members/committees • Awareness of unconscious bias Challenges • Functional and efficient peer group • Fatigue and randomness • Bias (group-think, flaw-focus) Individual review Group review Independence Width Expertise Concensus Rules-guided Group dynamics Van Arensbergenet al. (2014), Res Eval
  • 15. The process – Ranking & Assessment Meetings 15 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings NNF,example 44/189 = 23% Best score Worst score Thorough discussion Some/selecteddiscussion Little or no discussion
  • 16. The process – Ranking & Assessment Meetings 16 Martin et al (2010), PLOS One Pre- vs Post-scoring R01’s, 2983 applications. Panel discussion had a practical impact on 19.8% of applications. Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Pier et al (2015), WCER Working Paper Flaw-seeking 4 meetings, 11.% improved, 41% unchanged, 48% worsened during meeting.
  • 17. The process – Ranking & Assessment Meetings 17 Gallo et al (2016), PLOS One The critical expert Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Area expertise means harsher judgement. Worse towards junior than senior applicants. NNF Grant Report,2018 Age and gender bias
  • 18. The process – Ranking & Assessment Meetings 18 Derrick(2018),Nature Panel 1 Panel 2 Thegrantproposal Quality of writing Simple scoring system, large panels Innovation and novelty Translation Budgets are secondary ”It depends on the budget” Person centered Project centered Thegrantassessment Skeptical about use of external peers Importance of the ‘spokesperson’ setting the trend Recognize and awareness of ‘off the record’ conversations Strong chair person essential – good and bad Important to stay within expertise Okay to go outside of ‘comfort zone’ “Gaming” the score expected “Gaming” the score openly discussed Young researchers irrelevant or “budget boost” Capacity building essential Combative atmosphere with strong personalities Collegial, supporting atmosphere The National Health and Medical Research Councils (NHMRC) Coveney et al (2017), Res Integr Peer Rev Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings
  • 19. The process – Ranking & Assessment Meetings 19 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings Individual review Group review Independence Width Expertise Concensus Rules-guided Group dynamics Van Arensbergenet al. (2014), Res Eval Purpose • Joint peer review and discussion • Final funding recommendation At NNF • The grant assessment • Rank as basis, but not binding, and all applications are up for discussion • Pre-assessor(s) present their application(s); all are familiar with all applications • Lower-ranked applications can be brought up • Large variations must be discussed • Discussion until funding agreement and budget exhaustion • Chair directs discussion and ensures that all are heard • NNF Secretariat members oversee, advise and help the committee • The committee must adhere to • Rules of Eligibility (any ineligible members must step outside, literally; no self-funding) • Mandate Letter with grant purpose, criteria, budget, etc. • Rules of Procedure - Confidentiality • Environment sought after • Pleasant, friendly; good food, etc. • International members/committees • Awareness of unconscious bias Challenges • Functional and efficient peer group • Fatigue and randomness • Bias (group-think, flaw-focus)
  • 20. The process – Feedback (recommendation & offer) 20 Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings FEASIBILITY (technical, reviewer) BURDEN (reviewer) None Level of feedback Rank Stnd comment Rank Full comments Stnd comment Rank Purpose • Explanation for funding decision • Can allow for rebuttal • For potential improvement At NNF • Normally no feedback • Interview as (occasional) form of feedback Challenges • Technical challenges • Resource constraints – peers and secretariat • Balance level of feedback vs review burden • Making the feedback useful
  • 21. The process – Feedback (recommendation & offer) 21 Purpose • Explanation for funding decision • Can allow for rebuttal • For potential improvement At NNF • Normally no feedback • Interview as (occasional) form of feedback Challenges • Technical challenges • Resource constraints – peers and secretariat • Balance level of feedback vs review burden • Making the feedback useful Question % ‘Most useful’or ‘Veryuseful’ Overall how useful was the reviewer feedback you received on your last grantsubmission? 38.5% How useful was the reviewer feedback in improving your grantsmanship? 29.4% How useful was the reviewer feedback in improving your future submissions? 34.1% How useful was the reviewer feedback in informing your future scientific endeavors in the proposedresearch area? 26.1% Gallo et al (2020), In preparation Gallo et al (2019), Sci Eng Ethics Applicant(s)& Applications Funding recommendation and offer Pre-assessors& Scoring Ranking & Meetings 0 10 20 30 40 50 60 70 80 90 Innovation Risk PI track record RESPONDENTS (%) CRITERIA What is important in feedback Applicants Reviewers Adapted fromGallo et al (2018), Environ Sys Decis P<0.001 P<0.001 P<0.001
  • 23. Recap – what we do 23 Pre-assessors Scoreand Ranking Committee meeting Final recommendation GrantofferApplicationApplicant(s) ”Ultimately, peer review is going to be an imperfect process. But we are not doing a bad job.” Chairman(anonymous), in Powell (2010), Nature