Slides from my 2020 NSF-GRFP course on the mysterious topic of Broader Impacts. The National Science Foundation’s Graduate Research Fellowship Program (https://www.nsfgrfp.org) offers fellowships to new and incoming graduate students in the sciences. Applications deadlines are typically in October. The views presented here are solely my own, not those of my employer or the National Science Foundation.

1. AN [ ANONYMOUS]
REVIEWER’S
PERSPECTIVE ON NSF
GRFP APPLICATIONS
H O W T O M A K E Y O U R R E A D E R H A P P I E R
A N D
* * N O T * * W A N T T O I N C I N E R AT E Y O U R A P P L I C AT I O N
Slides from:

2. WHO ARE THE REVIEWERS?
Academics (usually tier 2-4 universities and colleges)
Industrial scientists
Government scientists
Don’t expect them to be experts in your field!
• How experienced?
– 30% are repeat offenders
– The rest are newbies
• ALL OF THEM ARE BUSY PEOPLE
– Your application will likely only be read ONCE. It
must flow and all points must be made up front.
MUST DELIVER PUNCH ON ONE READ

3. HOW ARE THEY TRAINED TO
REVIEW?
• NSF has 2 on-line Skype-like sessions
(reviewer ed is still an in-progress project for NSF)
– Reviewers go over instructions and discuss to the two merit criteria (Intellectual Merit,
Broader Impacts)
– Reviewers practice reading and scoring four representative applications of differing
Levels and Quality Groups
– On-line discussion of reviews by NSF bureaucrats with reviewer Q & A
PROBLEM:Too short a time with too little instruction from expert reviewers => uneven
reviews

4. HOW ARE THEY NOT
TRAINED/INSTRUCTED?
• What the “Ideal Recipient is”
• Specific types of content
• Specific points beyond Merit Criteria
• How to weigh Merit Criteria
– Some: 50-50 (me), others: however they please
FROM MY EXPERIENCE:
Broader impacts make a bigger difference because the science is harder to
differentiate (larger range of rankings in BI vs. IM)- Reviewers come from wide
variety of backgrounds

5. THINGS THAT REVIEWERS DO
WRONG
• *Incompletely read application
– (Fix: BE LOUD AND CLEARWITH STRENGTHS-REPEAT IF
NECESSARY-PUT IMPORTANT STUFF UP FRONT)
• SPECIFIC BIASES (research area, demographics, school)
• Unreasonable expectations for Level
• Overweight on Intellectual Merit or Broader Impacts over the
other
• Disregard Instructions (can’t do much about that)

6. PROCEDURE
• Reviewers have access to applications for ~ 3-4 weeks FINALS
WEEK AND WINTER BREAK
– 30 applications/reviewer, randomly distributed
– Typically read/score time is 15-20 minutesTOTAL
• Scores entered into an on-line form
• Scores are collated and normalized by NSF to give Z-score (# SD
from reviewer’s mean)
– Ranking, Large discrepancies of CHECK BOX (more than 2 levels) are
“resolved” by 3 reviewers
– Reranking allowed only in those cases
– Global discussion of ranking (with new format not much happens here)
– Final ranking agreed upon
• NSF awards top half of awards as per reviews, and the next equal
portion using internal criteria (~12.5% overall).

7. HOW ARE APPLICANTS SCORED?
Intellectual Merit (this is the “check box”)
• Excellent •Very Good •Good •Fair •Poor
100 character minimum comment
Broader Impacts
• Excellent •Very Good •Good •Fair •Poor
100 character minimum comment
Score (0-50) (not seen by you)

8. KISSES OF DEATH
– Incomplete application, not following instructions
• Not properly addressing Broader Impacts (you need more than a
paragraph heading!!!!)
– Any equivocation in LOR (must be GOLDEN)
– Level 2 and 3: Lack of LOR from Institution and Proposal Adviser
– Poor English, sloppy or difficult-to-read/understand sentences
– Personal Statement
• Egotistical
• Overly personal and irrelevant to the application (“I always wanted to be
a scientist since I watched my #### go down the toilet”)
• Lack of short and long term goals
– Research Plan
• Lack of a coherent hypothesis
• Lack of coherent contextualization (significance)
• Excessive experimental detail at the expense of the bigger picture
• BORING S.O.S.
• TOTALLY UNREALISTIC
• FOR LEVELS 2 and 3, NOT CONNECTEDTO RESEARCH BEING
DONE AT INSTITUTION (!!!!)
• Unfilled “holes” in plan (Delete extraneous stuff)

9. SUREFIRE WINNERS
– **Easy-to-read applications**
– BE SPECIFIC about as much as possible
– Truly transformative/creative research plan (VERY
RARE!!!)
– STELLAR Letters from BigWheels
– Coherent plan for Broader Impacts
• Include SPECIFIC plans for activities
• Have demonstrated track record of actually
doing these things (BI LORVERY HELPFUL)
• If possible, make connections to research
project
– Demonstrated previous commitment to outreach
– URM-involved activities

10. EXCELLENT REVIEW (AIM FOR THIS!)
2019 Plant Biology Proposal

11. VERY GOOD REVIEW - NOTES PUB IN
PREP
2017 Molecular Cell Bio Proposal

12. VERY GOOD BI REVIEW – BUT COULD
BE STRONGER

13. UNFAIR REVIEW

14. 44
Excellent-
23
Your academic record, multidisciplinary background,
impressive publication record and obvious love of
science speak to a highly promising and interactive
research career. Your proposal to block leader peptide
binding in RiPPs is interesting- although it seems to be
somewhat technically demanding. Is this a realistic
project? But nonetheless a nice proposal.
Excellent-
21
Your commitment and
demostrated participation
in outreach, chemical
education and increasing
access to computing is
exemplary. Your outreach
plans, besides those already
in the works, to bridge the
social gulf between science
and religion is laudable and
well-concieved.
5 pubs, 2 J Chem Ed, 1 1st, 2
2nd, 3.98 GPA- great letters,
highest promise, interactive,
interested in education, jst
gret. Multidisciplinary-
computers, chem, biochem.
Great outreach experience:
three different schools/URM
programs and enhancing
access to computing-born
out in letter. Great outreach
plans, religious background
wants to address the
compatibility of science with
religion.
ANONYMOUS REVIEWER’S REVIEWS
INTELLECTUAL MERIT BROADER IMPACT

15. 7 Fair-6
Although you are clearly a bright
and motivated young scientist with
a likely successful research career
ahead, you also need to fulfill the
requirements of the GRFP
application. While this reader
appreciates the difficulty of
formulating a research proposal so
early in your studies, nonetheless it
is a valuable exercise that gives the
reviewers a view of how well a
student can define a problem, pose
a hypothesis, and test it using the
tools of the trade. You research
description could have further
benefitted from more focus on the
big picture rather than on details.
Poor-1
You needed to address the
broader impacts of your
research and program.
Although you apparently have
an interest in teaching and
education, you needed to
formulate more specific ideas
and plans about how GRFP
resources would be used to
carry them out.
Clearly did not read the
directions. Research
Plan was overly
technical and did not
demonstrate
understanding.
ANONYMOUS REVIEWER’S REVIEWS
INTELLECTUAL MERIT
BROADER IMPACT

16. Excellent-
21
You recommendation letters, well-
articulated essays on physics, math and
biophysics indicate that you have the
intellectual tools to relate different
forms of data and make great
contributions to science. Your proposal
to study defined arrangements of
motors and cargo in order to understand
the more complex behaviors within cells
is ambitious and interesting. What this
reader would have appreciated was a
more focused and defined problem with
a testable hypothesis, but nonetheless it
seems like important work.
Good-
11
Like a good engineer, you
clearly know how to
collaborate, and your
discussions of your activities
at Princeton and as a
software engineer suggest
that you are thoughtful about
the meaning of your work.
You do propose outreach in
the form of a blog. The
proposal would have been
greatly strengthened by a
more concrete description of
the topics, scope and
audience you plan to reach
this way- sort of like a market
research study.
U Princeton (Physics-Ast, 3.6
GPA)/G Rice U Bioengineering.
Software Eng/Tech serv at
Adku/Groupon. Left to pursue
science-was jaded by sales
types. At PU, formed a trio of
software types and got some
contracts. No papers or pubs.
Actually discusses future
outreach plans: blogs digital
media to communicate
science (but no specific
program). Previous ECA:
Outdoor adventure leader, RA
in summer pre-program and
helped found The Roundtable
a group that discusses
contemporary issues. Very
academic approach- very
math-bright. Letters all say so.
INTELLECTUAL MERIT BROADER IMPACT

17. 37
Very good-
16
Although your academic record is
less competitive, your letters,
articulate research descriptions
and laboratory productivity
indicate that you are a motivated
and talented researcher with a
promising career ahead. Your
proposal about studying DHFR
variation is interesting. It might
have benefitted from a broader
view- many protein variants have
been created and it was unclear as
to how this study would add to
that knowledge, especially with a
course genetic analysis. Placing
the experimental outputs in a
larger context would have
strengthened the proposal.
Excellent-
21
Your personal story is
compelling Your clear
willingness to step out and
be a leader and a mover,
starting at an early age is
commendable. Your previous
outreach works provide
convincing evidence that you
will continue to carry such
plans out. However, specific
future plans would have
been a beneficial addition.
Burundian refugee
immigrant (5 yrs in
US)-UCSF grades not
so hot- but letters
suggest otherwise-
creative, energetic
and productive. In
MARC and Hughes
programs. No papers
in press- nor a
mention of one. S0-
so proposal but
probably did it
herself with no help.
INTELLECTUAL MERIT BROADER IMPACT