Gap Analysis: Guiding GovernmentInvestments In Research Using Data DrivenScientific Program and Policy DevelopmentNorman S...
 Braveman, N.S. What science indicators can tell policymakers. In Margaret O.   Meredith, Stephen D. Nelson and Albert H....
 So what prevents decision making based on science from  happening?  Limited and/or uninformative data –> need for addit...
Prevailing attitudes toward PP&D in science  It‟s not possible or advisable to plan science -> can lock you into   unprod...
"In the fields of observation, chance favors only the prepared mind."                                                     ...
   The context          Some sources of support for research in the US          Current interests of grant makers   An...
2011 Agency Budget Request         Funding Agency                                 (% for R&D)National Institutes of Health...
   There are approximately 4000 private grant makers    in the US   Health Research Alliance:     not-for-profit, non-g...
Many of these sources of funding relies onPP&D in order to stay current in their area of science be ahead of the curve i...
   Scientifically meritorious research   Innovative research that will move the field ahead   Development of the resear...
The foundation of PP&D is evaluation… a systematic method for collecting, analyzing, and using data and information to ans...
 Demographic/Quantitative Who is receiving funding When did they receive it Where are they located How much did they ...
 Demographic/Categorical Content       Based/Qualitative     Captures the content of the research in the portfolio ->  ...
What are the goals of PP&D for organizations that fund research?  To maximize the impact of funding by advancing science ...
Logic Model  Clear statement of mission and goals of the organization  Identification of inputs and outputs  Identific...
   A Major Component of PP&D   Answers the questions –    Where are we?     Where do we want to be?     Are we making...
Applied from a business/economics environment to organizations  involved in funding research   Identifies the gap between...
Applied from a business/economics environment to organizations involved in funding research  Identifies the gap between t...
Also addresses specific questions about the currentstate-of-the-science as well as about pathways toscientific advancement...
Strategic Plan                                Background &                                                                ...
   Background (stakeholder input; source documents)     What is the organization‟s role in the big picture?     What ar...
   Background   Portfolio Analysis (data collection)     What has the organization accomplished toward current      goa...
   Background   Data collection   Gap Analysis     Compare content of current portfolio with „ideal‟ portfolio     As...
   Background   Data collection   Gap Analysis   Expert Panel (thought leaders)     Where is the field going in the s...
These are the drivers of the development of science policies and         funding programsBut they aren‟t the only ones…th...
   Background   Data collection   Gap Analysis   Expert Panel   Organizational Decision Making (stakeholders and    l...
© Braveman BioMed Consultants - "Working with you for excellence in research." 27
   A representation of a document generated with the    Elsevier Fingerprint Engine   A precise abstract representation ...
© Braveman BioMed Consultants - "Working with you for excellence in research."   29
© Braveman BioMed Consultants - "Working with you for excellence in research." 30
   NIH Research, Condition, and Disease    Categorization Process (RCDC)     Introduced in 2009     A computerized proc...
Florida Biomedical Research          Program: Integration of PP&D Within    An Ongoing Scientific         Program         ...
Logic Model               Strategic Plan                       Portfolio Analysis        •   Program Goals                ...
National Heart, Lung and Blood         Institute (NIH)Identifying ideas and trackingtheir implementation through    progra...
SCIENTIFICWORKSHOPS                        IPROFESSIONAL                                 C                                ...
36© Braveman BioMed Consultants - "Working with you for excellence in research."
                                                                                     37    © Braveman BioMed Consultants ...
38© Braveman BioMed Consultants - "Working with you for excellence in research."
The effective use of data driven PP&D in developing science policy and funding programs is dependent on metrics that refle...
Upcoming SlideShare
Loading in...5
×

Gap analysis investments in research prof. norman braveman

222
-1

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
222
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Gap analysis investments in research prof. norman braveman

  1. 1. Gap Analysis: Guiding GovernmentInvestments In Research Using Data DrivenScientific Program and Policy DevelopmentNorman S. Braveman, Ph.D.PresidentBraveman BioMed ConsultantsElsevier Reed ConferenceAccelerate Research Excellence: Evaluate, Benchmark andStrategizeSeptember 23, 2001New Delhi, India Braveman BioMed Consultants - "Working with you for excellence in research." 1
  2. 2.  Braveman, N.S. What science indicators can tell policymakers. In Margaret O. Meredith, Stephen D. Nelson and Albert H. Teich (Eds.) Science and Technology Yearbook: 1991. Washington, D.C.: American Association for the Advancement of Science, 267-278.  Braveman, N.S. Evaluation at NIH: The policy issues. In Susan L. Sauer (Ed.) R & D in FY 1989: Looking Ahead in an Election Year. Washington, D.C.: American Association for the Advancement of Science, 1988.“Evaluation at NIH: The policy issues” Policies based on data are better than those that are not The primary driver or source of data for development of science policy and programs should be science itself But we need to be cognizant of other drivers and sources of information AS A GRANT MAKER YOU HAVE AN EXTREMELY IMPORTANT IMPACT ON SCIENCE 2 © Braveman BioMed Consultants - "Working with you for excellence in research."
  3. 3.  So what prevents decision making based on science from happening?  Limited and/or uninformative data –> need for additional and new metrics (demographic/bibliometric vs. content)  Lack of access to or difficulty in accessing data –> getting data is not easy and time consuming for first level users  Austere times –> tendency to not develop  Prevailing attitudes about formal planning in science 3 © Braveman BioMed Consultants - "Working with you for excellence in research."
  4. 4. Prevailing attitudes toward PP&D in science  It‟s not possible or advisable to plan science -> can lock you into unproductive pathways  It‟s important to be flexible and able to change directions quickly to follow leads/new directions  „Peer review‟/publication of findings are the only endpoints that matter in determining future directions  Serendipity or chance historically lead to great findings –> the double helix; clinical applications of iPS © Braveman BioMed Consultants - "Working with you for excellence in research." 4
  5. 5. "In the fields of observation, chance favors only the prepared mind." Louis PasteurA „prepared mind‟ (in science) is one that has consideredalternative paths, even in an informal way, particularly whenresults don‟t turn out as expected. “A bad is experiment is one that will provide answers only if the hypothesis is supported or disproven. A good experiment is one that will provide answers no matter what the outcome of the experiment.” Anonymous Peer Reviewer © Braveman BioMed Consultants - "Working with you for excellence in research." 5
  6. 6.  The context  Some sources of support for research in the US  Current interests of grant makers An approach to Data Driven Scientific Program and Policy Development using PP&D  Logic Analysis  Strategic and Implementation Planning  Gap Analysis Scientific fingerprinting – a way to collect data for PP&D of science policies and programs  What is scientific fingerprinting  Examples of current use in PP&D 6 © Braveman BioMed Consultants - "Working with you for excellence in research."
  7. 7. 2011 Agency Budget Request Funding Agency (% for R&D)National Institutes of Health $32.1 billion (85)National Science Foundation $7.4 billion (80)Department of Energy $28.4 billion (18)NASA $19.0 billion (33)Department of Defense $708.0 billion (10)US Department of Agriculture $146.0 billion (2)Various State Funding Programs(stem cells, tobacco –related --diseases) 7 © Braveman BioMed Consultants - "Working with you for excellence in research."
  8. 8.  There are approximately 4000 private grant makers in the US Health Research Alliance:  not-for-profit, non-governmental funders of biomedical research and training from biomedical science to applications that advance health  47 member organizations  HRA member organizations provide over $1.5 billion in research awards each year, funding over 5500 researchers annually 8 © Braveman BioMed Consultants - "Working with you for excellence in research."
  9. 9. Many of these sources of funding relies onPP&D in order to stay current in their area of science be ahead of the curve in planning for future scientific developments Insure that their resources are used appropriately and in the most effective way possible…particularly in austere times 9 © Braveman BioMed Consultants - "Working with you for excellence in research."
  10. 10.  Scientifically meritorious research Innovative research that will move the field ahead Development of the research workforce of the future  Basic scientists  Clinician and clinical scientists  Interdisciplinary scientists  Diversity in racial and ethnic backgrounds Topic of the day  Personalized medicine  Translational research  Systems approach to biomedicine  Stem cell research Disease of the day  Obesity  Cancer  Heart disease 10 © Braveman BioMed Consultants - "Working with you for excellence in research."
  11. 11. The foundation of PP&D is evaluation… a systematic method for collecting, analyzing, and using data and information to answer questions about projects, policies and scientific programs „Data „ is not the plural of „anecdote‟. Unknown © Braveman BioMed Consultants - "Working with you for excellence in research." 11
  12. 12.  Demographic/Quantitative Who is receiving funding When did they receive it Where are they located How much did they receive; how much in total is being spent on the area of science How many publications/inventions have resulted from the research (return on investment) How many publications have resulted from grants © Braveman BioMed Consultants - "Working with you for excellence in research." 12
  13. 13.  Demographic/Categorical Content Based/Qualitative  Captures the content of the research in the portfolio -> What is the content of the research that is being addressed? What isn‟t being addressed…and why?  Systematic, comprehensive, reliable and valid  Can link a current portfolio to portfolios that exist in other organizations (e.g., funding agencies)  Can add important scientific information in addressing some of the quantitative questions © Braveman BioMed Consultants - "Working with you for excellence in research." 13
  14. 14. What are the goals of PP&D for organizations that fund research?  To maximize the impact of funding by advancing science  To determine whether what has been done has met the specific goals and objectives of the organization  To chart the future priorities and directions of the organization © Braveman BioMed Consultants - "Working with you for excellence in research." 14
  15. 15. Logic Model  Clear statement of mission and goals of the organization  Identification of inputs and outputs  Identification of expectations and measures of successStrategic Planning  Statement of values, mission, priorities and goalsImplementation Planning  Timelines, benchmarks, mechanisms, sequence of eventsGap Analysis © Braveman BioMed Consultants - "Working with you for excellence in research." 15
  16. 16.  A Major Component of PP&D Answers the questions – Where are we?  Where do we want to be?  Are we making the best use of our resources? “It‟s the difference between what an organization is doing or has been doing and what the organization should be doing or wants to do…” Source Unknown © Braveman BioMed Consultants - "Working with you for excellence in research." 16
  17. 17. Applied from a business/economics environment to organizations involved in funding research  Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation  What are the inputs in the world of science? -> money, people, facilities/equipment, knowledge  What do we mean by “optimization” or “integration ” when we‟re talking about funding of science? -> a unique combination of inputs that yields best performance on outcome measures © Braveman BioMed Consultants - "Working with you for excellence in research." 17
  18. 18. Applied from a business/economics environment to organizations involved in funding research  Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation  Used to compare a program‟s actual performance with its potential performance  How do we determine a funding organizations potential performance?  from strategic plan, implementation plan and logic model  How do we measure a funding organization‟s actual performance?  use outcome measures developed from logic model © Braveman BioMed Consultants - "Working with you for excellence in research." 18
  19. 19. Also addresses specific questions about the currentstate-of-the-science as well as about pathways toscientific advancement What do we need to do to move ahead? What are the barriers and where are the opportunities?Gaps are then defined in terms of the currentportfolio of the organization and the near-and long-term state-of-the-science © Braveman BioMed Consultants - "Working with you for excellence in research." 19
  20. 20. Strategic Plan Background & OutcomeStakeholders Measures Goals Logic Model Gap Analysis Expert Panel Current •Future of area portfolio vs. Data •Barriers “ideal” Collection •Possible priorities portfolio Organization Data-Driven Decision Process © Braveman BioMed Consultants - "Working with you for excellence in research." 20
  21. 21.  Background (stakeholder input; source documents)  What is the organization‟s role in the big picture?  What are our present goals?  Where are we going?  How have we gotten to where we are?  What has changed that we need an updated assessment?  Who are the other participants and what is their role?  What is/has been our relationship with them? © Braveman BioMed Consultants - "Working with you for excellence in research." 21
  22. 22.  Background Portfolio Analysis (data collection)  What has the organization accomplished toward current goals?  What science have we been funding? What science have we not been funding?  What is/are the scientific content of the same or a similar portfolio(s) of other organizations?  What science areas are covered by applications that have been submitted to our organization?  Of those submitted to our organization, which ones are being funded? Which ones are not being funded…and why? © Braveman BioMed Consultants - "Working with you for excellence in research." 22
  23. 23.  Background Data collection Gap Analysis  Compare content of current portfolio with „ideal‟ portfolio  Assess content of current support by all organizations for duplication, overlap and collaboration  Compare the content of current portfolio(s ) with the future of science  Compare content of all submitted applications with program goals and future of science  Compare content of successful applications with program goals and future of science  Compare content of unsuccessful applications with the future of science and program goals © Braveman BioMed Consultants - "Working with you for excellence in research." 23
  24. 24.  Background Data collection Gap Analysis Expert Panel (thought leaders)  Where is the field going in the short-/long-term?  What are the scientific barriers to getting there?  Assessment of current content of programs by all organizations for duplication, overlap and collaboration  Where are the gaps between the content of current portfolios and the future of science…with „ideal‟ portfolio  Recommendations/priorities about where the organization being analyzed can make significant and unique contributions/impacts © Braveman BioMed Consultants - "Working with you for excellence in research." 24
  25. 25. These are the drivers of the development of science policies and funding programsBut they aren‟t the only ones…the final step -> organizational decision making © Braveman BioMed Consultants - "Working with you for excellence in research." 25
  26. 26.  Background Data collection Gap Analysis Expert Panel Organizational Decision Making (stakeholders and leadership)  What unique scientific contributions do we want to be known for?  Resources available – people, fiscal  Possibilities for collaboration/leveraging with other organizations  Political considerations  Determine which gaps are going to be addressed  Develop timelines and benchmarks © Braveman BioMed Consultants - "Working with you for excellence in research." 26
  27. 27. © Braveman BioMed Consultants - "Working with you for excellence in research." 27
  28. 28.  A representation of a document generated with the Elsevier Fingerprint Engine A precise abstract representation of a text which allows us to “look into the text” (content) rather than only looking at the meta data (demographics). © Braveman BioMed Consultants - "Working with you for excellence in research." 28
  29. 29. © Braveman BioMed Consultants - "Working with you for excellence in research." 29
  30. 30. © Braveman BioMed Consultants - "Working with you for excellence in research." 30
  31. 31.  NIH Research, Condition, and Disease Categorization Process (RCDC)  Introduced in 2009  A computerized process used at the end of each fiscal year to sort and report the amount funded in each of 229 historically reported categories of disease, condition, or research area. The Florida Biomedical Research Program – scientific program management and development National Heart, Lung and Blood Institute (NIH) – identifying ideas and tracking progress © Braveman BioMed Consultants - "Working with you for excellence in research." 31
  32. 32. Florida Biomedical Research Program: Integration of PP&D Within An Ongoing Scientific Program © Braveman BioMed Consultants - "Working with you for excellence in research." 32
  33. 33. Logic Model Strategic Plan Portfolio Analysis • Program Goals • OutcomeProgram Planning Measures Implementation Plan Gap Analysis Recommendations Expert Science Panel Council Review Program Development Call Development Logic Model & Gap Analysis • Priority science areas • Program Goals & Objectives • Funding mechanisms • Outcome Measures • Application configuration • Gap Analysis Biomed Experts & Reviewer Finder Issue RFA(s) Grantsmanship Seminars (Answer questions about RFA) Applications Applications Summary Statements Grants Management & Peer Review To Applicants Reviewer Finder Systems Council Review & Funding Notice of Award Recommendation Grant Monitoring Progress Reports Grants Management & Annual Report Program Review •Grant Outcomes Financial Reports •Success Stories •Program Outcomes © Braveman BioMed Consultants - "Working with you for excellence in research."
  34. 34. National Heart, Lung and Blood Institute (NIH)Identifying ideas and trackingtheir implementation through program development © Braveman BioMed Consultants - "Working with you for excellence in research." 34
  35. 35. SCIENTIFICWORKSHOPS IPROFESSIONAL C NATIONALGROUPS L ADVISORYDHHS LEADERSHIP E G COUNCILNIH OFFICES I OR BOARD S IC Staff LAd Hoc AdvisoryGroups APATIENT Opportunity T E DecisionGROUPS D ProcessCONGRESS M IOTHER FEDERAL S IC NIH EarlyAGENCIES S ADJUSTMENTS Notification IEXTRAMURAL O SystemSCIENTISTS NINDUSTRY/PRIVATE SECTOR ISSUE RFA OR PA © Braveman BioMed Consultants - "Working with you for excellence in research." 35
  36. 36. 36© Braveman BioMed Consultants - "Working with you for excellence in research."
  37. 37.  37 © Braveman BioMed Consultants - "Working with you for excellence in research."
  38. 38. 38© Braveman BioMed Consultants - "Working with you for excellence in research."
  39. 39. The effective use of data driven PP&D in developing science policy and funding programs is dependent on metrics that reflect the scientific content of the policy or programs as well as demographic information.The metrics must be:  reliable, valid, and meaningful to the user  accessible, interpretable, and readily analyzable by the userAnd the system used to compile and summarize the metricsmust be adaptable to a variety of circumstances © Braveman BioMed Consultants - "Working with you for excellence in research." 39
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×