More Related Content Similar to Gap analysis investments in research prof. norman braveman Similar to Gap analysis investments in research prof. norman braveman (20) Gap analysis investments in research prof. norman braveman1. Gap Analysis: Guiding Government
Investments In Research Using Data Driven
Scientific Program and Policy Development
Norman S. Braveman, Ph.D.
President
Braveman BioMed Consultants
Elsevier Reed Conference
Accelerate Research Excellence: Evaluate, Benchmark and
Strategize
September 23, 2001
New Delhi, India
Braveman BioMed Consultants - "Working with you for excellence in research." 1
2. Braveman, N.S. What science indicators can tell policymakers. In Margaret O.
Meredith, Stephen D. Nelson and Albert H. Teich (Eds.) Science and Technology
Yearbook: 1991. Washington, D.C.: American Association for the Advancement of
Science, 267-278.
Braveman, N.S. Evaluation at NIH: The policy issues. In Susan L. Sauer (Ed.) R & D
in FY 1989: Looking Ahead in an Election Year. Washington, D.C.: American
Association for the Advancement of Science, 1988.“Evaluation at NIH: The policy
issues”
Policies based on data are better than those that are not
The primary driver or source of data for development of
science policy and programs should be science itself
But we need to be cognizant of other drivers and sources of
information
AS A GRANT MAKER YOU HAVE AN EXTREMELY IMPORTANT
IMPACT ON SCIENCE
2
© Braveman BioMed Consultants - "Working with you for excellence in research."
3. So what prevents decision making based on science from
happening?
Limited and/or uninformative data –> need for additional and
new metrics (demographic/bibliometric vs. content)
Lack of access to or difficulty in accessing data –> getting data
is not easy and time consuming for first level users
Austere times –> tendency to not develop
Prevailing attitudes about formal planning in science
3
© Braveman BioMed Consultants - "Working with you for excellence in research."
4. Prevailing attitudes toward PP&D in science
It‟s not possible or advisable to plan science -> can lock you into
unproductive pathways
It‟s important to be flexible and able to change directions quickly
to follow leads/new directions
„Peer review‟/publication of findings are the only endpoints that
matter in determining future directions
Serendipity or chance historically lead to great findings –> the
double helix; clinical applications of iPS
© Braveman BioMed Consultants - "Working with you for excellence in research." 4
5. "In the fields of observation, chance favors only the prepared mind."
Louis Pasteur
A „prepared mind‟ (in science) is one that has considered
alternative paths, even in an informal way, particularly when
results don‟t turn out as expected.
“A bad is experiment is one that will provide answers only if the
hypothesis is supported or disproven. A good experiment is one
that will provide answers no matter what the outcome of the
experiment.”
Anonymous Peer Reviewer
© Braveman BioMed Consultants - "Working with you for excellence in research." 5
6. The context
Some sources of support for research in the US
Current interests of grant makers
An approach to Data Driven Scientific Program and Policy
Development using PP&D
Logic Analysis
Strategic and Implementation Planning
Gap Analysis
Scientific fingerprinting – a way to collect data for PP&D of science
policies and programs
What is scientific fingerprinting
Examples of current use in PP&D
6
© Braveman BioMed Consultants - "Working with you for excellence in research."
7. 2011 Agency Budget Request
Funding Agency (% for R&D)
National Institutes of Health $32.1 billion (85)
National Science Foundation $7.4 billion (80)
Department of Energy $28.4 billion (18)
NASA $19.0 billion (33)
Department of Defense $708.0 billion (10)
US Department of Agriculture $146.0 billion (2)
Various State Funding Programs
(stem cells, tobacco –related --
diseases)
7
© Braveman BioMed Consultants - "Working with you for excellence in research."
8. There are approximately 4000 private grant makers
in the US
Health Research Alliance:
not-for-profit, non-governmental funders of biomedical
research and training from biomedical science to
applications that advance health
47 member organizations
HRA member organizations provide over $1.5 billion in
research awards each year, funding over 5500 researchers
annually
8
© Braveman BioMed Consultants - "Working with you for excellence in research."
9. Many of these sources of funding relies on
PP&D in order to
stay current in their area of science
be ahead of the curve in planning for future
scientific developments
Insure that their resources are used appropriately
and in the most effective way
possible…particularly in austere times
9
© Braveman BioMed Consultants - "Working with you for excellence in research."
10. Scientifically meritorious research
Innovative research that will move the field ahead
Development of the research workforce of the future
Basic scientists
Clinician and clinical scientists
Interdisciplinary scientists
Diversity in racial and ethnic backgrounds
Topic of the day
Personalized medicine
Translational research
Systems approach to biomedicine
Stem cell research
Disease of the day
Obesity
Cancer
Heart disease
10
© Braveman BioMed Consultants - "Working with you for excellence in research."
11. The foundation of PP&D is evaluation…
a systematic method for collecting, analyzing, and using data
and information to answer questions about projects, policies
and scientific programs
„Data „ is not the plural of „anecdote‟. Unknown
© Braveman BioMed Consultants - "Working with you for excellence in research." 11
12. Demographic/Quantitative
Who is receiving funding
When did they receive it
Where are they located
How much did they receive; how much in total is
being spent on the area of science
How many publications/inventions have resulted
from the research (return on investment)
How many publications have resulted from grants
© Braveman BioMed Consultants - "Working with you for excellence in research." 12
13. Demographic/Categorical
Content Based/Qualitative
Captures the content of the research in the portfolio ->
What is the content of the research that is being addressed?
What isn‟t being addressed…and why?
Systematic, comprehensive, reliable and valid
Can link a current portfolio to portfolios that exist in other
organizations (e.g., funding agencies)
Can add important scientific information in addressing
some of the quantitative questions
© Braveman BioMed Consultants - "Working with you for excellence in research." 13
14. What are the goals of PP&D for organizations that
fund research?
To maximize the impact of funding by advancing science
To determine whether what has been done has met the
specific goals and objectives of the organization
To chart the future priorities and directions of the
organization
© Braveman BioMed Consultants - "Working with you for excellence in research." 14
15. Logic Model
Clear statement of mission and goals of the organization
Identification of inputs and outputs
Identification of expectations and measures of success
Strategic Planning
Statement of values, mission, priorities and goals
Implementation Planning
Timelines, benchmarks, mechanisms, sequence of events
Gap Analysis
© Braveman BioMed Consultants - "Working with you for excellence in research." 15
16. A Major Component of PP&D
Answers the questions –
Where are we?
Where do we want to be?
Are we making the best use of our resources?
“It‟s the difference between what an organization is
doing or has been doing and what the
organization should be doing or wants to do…”
Source Unknown
© Braveman BioMed Consultants - "Working with you for excellence in research." 16
17. Applied from a business/economics environment to organizations
involved in funding research
Identifies the gap between the optimized allocation and
integration of the inputs, and the current or future level of
allocation
What are the inputs in the world of science? -> money, people,
facilities/equipment, knowledge
What do we mean by “optimization” or “integration ” when we‟re talking
about funding of science? -> a unique combination of inputs that
yields best performance on outcome measures
© Braveman BioMed Consultants - "Working with you for excellence in research." 17
18. Applied from a business/economics environment to organizations
involved in funding research
Identifies the gap between the optimized allocation and
integration of the inputs, and the current or future level of
allocation
Used to compare a program‟s actual performance with its
potential performance
How do we determine a funding organizations potential performance?
from strategic plan, implementation plan and logic model
How do we measure a funding organization‟s actual performance?
use outcome measures developed from logic model
© Braveman BioMed Consultants - "Working with you for excellence in research." 18
19. Also addresses specific questions about the current
state-of-the-science as well as about pathways to
scientific advancement
What do we need to do to move ahead?
What are the barriers and where are the
opportunities?
Gaps are then defined in terms of the current
portfolio of the organization and the near-
and long-term state-of-the-science
© Braveman BioMed Consultants - "Working with you for excellence in research." 19
20. Strategic Plan
Background & Outcome
Stakeholders Measures
Goals
Logic Model
Gap
Analysis
Expert Panel
Current
•Future of area portfolio vs. Data
•Barriers “ideal” Collection
•Possible priorities portfolio
Organization Data-Driven Decision Process
© Braveman BioMed Consultants - "Working with you for excellence in research." 20
21. Background (stakeholder input; source documents)
What is the organization‟s role in the big picture?
What are our present goals?
Where are we going?
How have we gotten to where we are?
What has changed that we need an updated assessment?
Who are the other participants and what is their role?
What is/has been our relationship with them?
© Braveman BioMed Consultants - "Working with you for excellence in research." 21
22. Background
Portfolio Analysis (data collection)
What has the organization accomplished toward current
goals?
What science have we been funding? What science have we not
been funding?
What is/are the scientific content of the same or a similar
portfolio(s) of other organizations?
What science areas are covered by applications that have been
submitted to our organization?
Of those submitted to our organization, which ones are being
funded? Which ones are not being funded…and why?
© Braveman BioMed Consultants - "Working with you for excellence in research." 22
23. Background
Data collection
Gap Analysis
Compare content of current portfolio with „ideal‟ portfolio
Assess content of current support by all organizations for
duplication, overlap and collaboration
Compare the content of current portfolio(s ) with the future of
science
Compare content of all submitted applications with program goals
and future of science
Compare content of successful applications with program goals
and future of science
Compare content of unsuccessful applications with the future of
science and program goals
© Braveman BioMed Consultants - "Working with you for excellence in research." 23
24. Background
Data collection
Gap Analysis
Expert Panel (thought leaders)
Where is the field going in the short-/long-term?
What are the scientific barriers to getting there?
Assessment of current content of programs by all organizations
for duplication, overlap and collaboration
Where are the gaps between the content of current portfolios and
the future of science…with „ideal‟ portfolio
Recommendations/priorities about where the organization being
analyzed can make significant and unique contributions/impacts
© Braveman BioMed Consultants - "Working with you for excellence in research." 24
25. These are the drivers of the
development of science policies and
funding programs
But they aren‟t the only ones…the
final step -> organizational decision
making
© Braveman BioMed Consultants - "Working with you for excellence in research." 25
26. Background
Data collection
Gap Analysis
Expert Panel
Organizational Decision Making (stakeholders and
leadership)
What unique scientific contributions do we want to be
known for?
Resources available – people, fiscal
Possibilities for collaboration/leveraging with other
organizations
Political considerations
Determine which gaps are going to be addressed
Develop timelines and benchmarks
© Braveman BioMed Consultants - "Working with you for excellence in research." 26
27. © Braveman BioMed Consultants - "Working with you for excellence in research." 27
28. A representation of a document generated with the
Elsevier Fingerprint Engine
A precise abstract representation of a text which
allows us to “look into the text” (content) rather
than only looking at the meta data (demographics).
© Braveman BioMed Consultants - "Working with you for excellence in research." 28
29. © Braveman BioMed Consultants - "Working with you for excellence in research." 29
30. © Braveman BioMed Consultants - "Working with you for excellence in research." 30
31. NIH Research, Condition, and Disease
Categorization Process (RCDC)
Introduced in 2009
A computerized process used at the end of each fiscal year
to sort and report the amount funded in each of 229
historically reported categories of disease, condition, or
research area.
The Florida Biomedical Research Program –
scientific program management and development
National Heart, Lung and Blood Institute (NIH) –
identifying ideas and tracking progress
© Braveman BioMed Consultants - "Working with you for excellence in research." 31
32. Florida Biomedical Research
Program:
Integration of PP&D Within
An Ongoing Scientific
Program
© Braveman BioMed Consultants - "Working with you for excellence in research." 32
33. Logic Model Strategic Plan
Portfolio Analysis • Program Goals
• Outcome
Program Planning
Measures Implementation Plan
Gap Analysis
Recommendations
Expert Science Panel
Council Review
Program Development
Call Development Logic Model & Gap Analysis
• Priority science areas • Program Goals & Objectives
• Funding mechanisms • Outcome Measures
• Application configuration • Gap Analysis
Biomed Experts &
Reviewer Finder
Issue RFA(s)
Grantsmanship Seminars
(Answer questions about RFA)
Applications
Applications
Summary Statements
Grants Management & Peer Review
To Applicants
Reviewer Finder Systems
Council
Review & Funding Notice of Award
Recommendation
Grant Monitoring
Progress Reports
Grants Management &
Annual Report
Program Review
•Grant Outcomes
Financial Reports •Success Stories
•Program Outcomes
© Braveman BioMed Consultants - "Working with you for excellence in research."
34. National Heart, Lung and Blood
Institute (NIH)
Identifying ideas and tracking
their implementation through
program development
© Braveman BioMed Consultants - "Working with you for excellence in research." 34
35. SCIENTIFIC
WORKSHOPS I
PROFESSIONAL
C NATIONAL
GROUPS
L ADVISORY
DHHS LEADERSHIP
E
G
COUNCIL
NIH OFFICES I OR BOARD
S IC Staff
L
Ad Hoc Advisory
Groups A
PATIENT
Opportunity T
E Decision
GROUPS D Process
CONGRESS M
I
OTHER FEDERAL S IC NIH Early
AGENCIES S ADJUSTMENTS Notification
I
EXTRAMURAL O
System
SCIENTISTS N
INDUSTRY/
PRIVATE SECTOR
ISSUE RFA OR PA
© Braveman BioMed Consultants - "Working with you for excellence in research." 35
37.
37
© Braveman BioMed Consultants - "Working with you for excellence in research."
39. The effective use of data driven PP&D in developing
science policy and funding programs is dependent
on metrics that reflect the scientific content of the
policy or programs as well as demographic
information.
The metrics must be:
reliable, valid, and meaningful to the user
accessible, interpretable, and readily analyzable by the user
And the system used to compile and summarize the metrics
must be adaptable to a variety of circumstances
© Braveman BioMed Consultants - "Working with you for excellence in research." 39