My part of the "Digital Science Webinar: Articulating Research Impact – Strategies from Around the Globe" (http://www.digital-science.com/events/digital-science-webinar-articulating-research-impact-strategies-from-around-the-globe/) Daniel S. Katz will discuss how reviewers at the National Science Foundation (USA) consider the “intellectual merit” and “broader impacts” criteria for funding and in particular how metrics might help applicants understand their impacts in these areas.Dan will also talk about how reviewers might use qualitative and quantitative altmetrics data to inform their peer reviews for grant applications. He will address many of the salient questions around this use of metrics, for example, do reviewers take metrics seriously and what types of metrics are of most value to them?

1. US University Research Funding,
Peer Reviews, and Metrics
Daniel S. Katz
Computation Institute,
University of Chicago & Argonne National Laboratory
@danielskatz d.katz@ieee.org

2. US University Funding
• Large number of universities
– Public (state-funded, not federally-funded),
private, for profit
• No direct national funding
• Indirect national funding of education through
students
• Indirect national funding of research through
projects
Research assessment is specific to projects, usually as part of proposals

3. National Research Funding Agencies
• National Science Foundation
– Mission: support for all fields of fundamental science and
engineering, except for medical sciences
• National Institutes of Health
– Mission: to seek fundamental knowledge about the nature and
behavior of living systems and the application of that knowledge
to enhance health, lengthen life, and reduce illness and
disability
• Department of Energy
– Mission: to ... [address] energy, environmental and nuclear
challenges through transformative science and technology
solutions
• Department of Defense
– Mission: ... to protect the security of our country
• Many other agencies
– Each with a specific mission
Research assessment is specific to agencies, usually as part of proposals

4. NSF Standard Merit Review Criteria
Intellectual Merit Broader Impacts
What is the potential for the proposed activity to
advance knowledge and understanding within its
own field or across different fields?
What is the potential for the proposed activity to
benefit society or advance desired societal
outcomes?
To what extent do the proposed activities suggest
and explore creative, original, or potentially
transformative concepts?
To what extent do the proposed activities suggest
and explore creative, original, or potentially
transformative concepts?
Is the plan for carrying out the proposed activities
well-reasoned, well-organized, and based on a
sound rationale? Does the plan incorporate a
mechanism to assess success?
Is the plan for carrying out the proposed activities
well-reasoned, well-organized, and based on a
sound rationale? Does the plan incorporate a
mechanism to assess success?
How well qualified is the individual, team, or
organization to conduct the proposed activities?
How well qualified is the individual, team, or
organization to conduct the proposed activities?
Are there adequate resources available to the PI
(either at the home organization or through
collaborations) to carry out the proposed
activities?
Are there adequate resources available to the PI
(either at the home organization or through
collaborations) to carry out the proposed
activities?
http://www.nsf.gov/bfa/dias/policy/merit_review/
Intellectual Merit
What is the potential for the proposed activity to
advance knowledge and understanding within its
own field or across different fields?
To what extent do the proposed activities suggest
and explore creative, original, or potentially
transformative concepts?
Is the plan for carrying out the proposed activities
well-reasoned, well-organized, and based on a
sound rationale? Does the plan incorporate a
mechanism to assess success?
How well qualified is the individual, team, or
organization to conduct the proposed activities?
Are there adequate resources available to the PI
(either at the home organization or through
collaborations) to carry out the proposed
activities?

5. Additional Review Criteria
• NSF Software Infrastructure for Sustained
Innovation (SI2), for example, includes:
– Does the proposal discuss how the proposed
software will fill a recognized need and advance
research capability within a significant area (or areas)
of science and engineering?
– Are tangible metrics described to measure the
success of any software that may be developed?
– Does the software engineering and development plan
include and/or enable the integration of relevant
research activities to ensure the software is
responsive to new computing developments?
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf14520

6. Agencies Expectations for Societal
Benefit
• NSF
– 2 non-scored criteria (one is broader impacts), plus maybe
more, put together qualitatively, one final rating
• NIH
– Scored criteria put together quantitatively, plus non-scored
criteria
– No specific request for description of societal benefit, but need
is strongly implied in all programs
• DOE and other mission agencies, impact is measured in
how research contributes to their missions
– Non-scored review processes in general
– At a high level, succeeding in this missions already includes the
concept of societal benefit

7. Determining What NSF Funds
• Peer review of new project is primary factor
– Also includes proposers information about
previous projects
• What is the role of data?
• As input to peer review, not to replace it
• So, the question is what data will give the
peer-reviewers new information and
potentially lead them to different decisions
– Answers may be project specific

8. Guidance to Proposers - Metrics
• Decide what you want to measure to make a
convincing case for your project
– Quality
– Impact
– Attention
• Then decide how to measure it
– Technical metrics
– Biobliometrics
– Altmetrics