to the NIH
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,206
On Slideshare
1,206
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. CSR Where we are and where we would like to be Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa PRAC, August 2006
  • 2. Changes in CSR Operations Our Vision for Peer Review
  • 3. This is CSR
  • 4.
    • Increased Communication and Transparency
    • Increase Uniformity
    • Increase Efficiency
    Changes in CSR Operations
  • 5. Changes in CSR Operations
    • 1. Increased Communication and Transparency
    • Within CSR
      • Biweekly meetings with SRAs and other staff
      • New comprehensive communications plan
    • With NIH and other Agencies
    • With the Scientific Community
      • Expanded Peer Review Notes newsletter
      • Six Open House Meetings with leaders of scientific societies and disease groups
  • 6. Changes in CSR Operations
    • 2. Increase Uniformity
    • Summary Statements
      • Post all within 1 month of meeting (97.3%)
      • Post new investigator summary statements within 1 week
      • Produce complete and structured resumes
    • Unscoring
      • Unscore 50% (49.7%)
    • Best Practices Committee Assessments
      • Appropriate Rosters
      • Types of Meetings
      • Structure of Summary Statements
  • 7. Changes in CSR Operations
    • 3. Increase Efficiency
    • Electronic Submission
    • Text Fingerprinting, Artificial Intelligence Software
      • Assigning applications to Integrated Review Groups or Study Sections
    • Major pilot to directly assign applications to Integrated
    • Review Groups and/or Study Sections will begin in
    • October, with implementation by February 2007.
  • 8. Scoring – Current Situation
    • Compression of scoring
    • “Approach is King” and “Significance” has been downgraded
    • Reasons for scoring often not clear in summary statements
    • Scoring of A2 applications may be inappropriately influenced by reviewers knowing A1 score
  • 9. IRG and Study Sections Realignment
    • Increasing concern from applicants, societies and diseases groups about the variance in the breadth of science covered by study sections.
  • 10. Monitoring IRGs and Study Sections
    • Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders.
    • Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff.
    • Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee.
    • Most study sections visited by CSR’s Director and senior staff at least once a year.
    • All retiring study section chairs called by CSR’s Director to learn about problems and possible improvements.
    • All summary statements read by CSR’s Director.
    • Multiple outreach efforts now gather fresh input.
  • 11. IRG Review Schedule Renal and Urological Sciences (RUS) Respiratory Sciences (RES) Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) Oncological Sciences (ONC) Musculoskeletal, Oral and Skin Sciences (MOSS) Cell Biology (CB) Biobehavioral and Behavioral Processes (BBBP) Infectious Diseases and Microbiology (IDM) Biology of Development and Aging (BDA) Scheduled 2007 -- 9 IRGs Health of the Population (HOP) Immunology (IMM) Hematology (HEME) Molecular, Cellular and Developmental Neuroscience (MDCN) Integrative, Functional and Cognitive Neuroscience (IFCN) Brain Disorders and Clinical Neuroscience (BDCN) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) Digestive Sciences (DIG) Genes, Genomes and Genetics (GGG) Risk, Prevention, and Health Behavior (RPHB) AIDS and Related Research (AARR) Bioengineering Sciences and Technologies (BST) Cardiovascular Science (CVS) Biological Chemistry and Macromolecular Biophysics (BCMB) Scheduled 2006 --14 IRGs
  • 12. Monitoring IRGs and Study Sections
    • Every IRG is being assessed by all senior CSR staff every 2 years in addition to the current 5-year assessments involving all stakeholders.
    • Emergent problems addressed by working groups with leaders from the extramural community, and senior NIH/CSR staff.
    • Substantive issues/changes reviewed by the NIH Peer Review Advisory Committee.
    • Most study sections visited by CSR’s Director, Deputy Director, and senior staff at least once a year.
    • All retiring study section chairs called to learn about problems and possible improvements.
    • Multiple outreach efforts now gather fresh input.
    • Open house meetings to be held.
  • 13. Broad Scientific Areas
    • Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG)
    • Integrated Biological (5): Immunology (IMM); Hematology (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA)
    • Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR)
    • Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC);  Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB)
    • Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN)
    • Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB)
  • 14. Our Agenda
    • Shorten the review cycle
    • Address concern that clinical research is not properly evaluated
    • Improve the assessment of innovative, high- risk/high-reward research
    • Do more to recruit and retain more high-quality reviewers
  • 15. Our Agenda
    • Shorten the review cycle
  • 16. Shortening the NIH Review Cycle, Initial Steps
    • Pilot study with ~600 new investigators in 40 study sections who may revise and resubmit for the very next review cycle 4 months earlier than before
    • (Effective Feb 06)
    • Summary statements posted within one month after the study section meeting
  • 17. Short Review Cycle Pilot of New Investigator R01 Applications 86 631 Number 14% Amended and Submitted for the Next Round 100% Reviewed in Pilot Percent New Investigator R01 Applications
  • 18. R01 A1 Resubmissions in the Next Round
  • 19. Our Agenda
    • Shorten the review cycle
    • Address concern that clinical research is not properly evaluated
  • 20. 17.56 22.07 M. Martin, CSR/NIH/DHHS
  • 21. Our Agenda
    • Shorten the review cycle
    • Address concern that clinical research is not properly evaluated
    • Improve the assessment of innovative, high- risk/high-reward research
  • 22. Our Agenda
    • Shorten the review cycle
    • Address concern that clinical research is not properly evaluated
    • Improve the assessment of innovative, high- risk/high-reward research
    • Do more to recruit and retain more high-quality reviewers
  • 23. Applications Received for all of NIH and Applications Referred for CSR Review, FY 1998 - 2006 NIH Applications Applications for CSR Review
  • 24. Number of Research Grant Applications/Applicant
  • 25. Growth of R01 Applications Reviewed at CSR vs. Other ICs
  • 26. Growth of R21 Applications Reviewed at CSR vs. Other ICs
  • 27. Institutes and Centers Use R21s Differently
    • R21s fund ~14 types of efforts, such as—
      • Exploratory/developmental research
      • Junior investigators
      • Phase I/II clinical trials
      • International research planning
      • High-risk/payoff research
    • R21s have broad parameters—
      • Award period: 1-3 years
      • Funding allowed: $100K-$450K
      • Research Plan: 10-20 pages
  • 28. Average Number of Applications Per Reviewer October Council Only
  • 29. Academic Rank of All CSR Reviewers Standing and Ad Hoc Combined
  • 30. Reviewers – Current Situation
    • Far too many reviewers on study sections
      • Broader science
      • Decrease in reviewer load
      • Unnecessary
    • Too many ad hoc reviewers
    • Different processes for enlisting chartered and ad hoc members
    • Often too many telephone reviewers
  • 31. Challenges to Recruiting Reviewers
      • Manage the need for 20,000 reviewers
      • Decrease the number of reviewers and increase the quality
      • Increase the number of applications reviewed without extra workload
      • Recruit and retain the best reviewers
      • Have smaller study sections with fewer ad hoc reviewers
  • 32. Possible Approaches for Recruiting and Retaining the Best Reviewers
    • Require less travel by using electronic review modes
  • 33. Expanding Peer Review Platforms
    • Electronic Reviews
    • Telephone Enhanced Discussions
    • Video Enhanced Discussions
    • Asynchronous Electronic Discussions
    Our Goal: 10% of all reviews to be electronic in 2007
  • 34. Electronic Review Methods Number of Applications Reviewed by Year
  • 35. Possible Approaches for Recruiting and Retaining the Best Reviewers
    • Require less travel by using electronic review modes
    • Have shorter meetings
    • Use Various Review Platforms
    • Unscore 40% of the F32s (postdoctoral fellowships)
    • Shorten Applications
  • 36. Size of Grant Applications
    • RO1
    • Will increase number of applications reviewed by reviewers
    • Will decrease the number of reviewers in a study section
    • May be combined with a change in format of the application, more consonant to review criteria
    • Strong support by councils and scientific leadership
    • May result in better focus on significance (and innovation)
  • 37. Trans-NIH Committee to Shorten the Application Chairs Robert Finkelstein, Ph.D. (Co-Chair), NINDS Donald Schneider, Ph.D. (Co-Chair), CSR Members Mary Custer, Ph.D. CSR Ann A. Hagan, Ph.D. NIGMS Craig A. Jordan, Ph.D. NIDCD Sherry Mills, M.D., M.P.H. OER Philip F. Smith, Ph.D. NIDDK Barbara Spalholz NCI Betsy Wilder, Ph.D. NIDDK Alan L. Willard, Ph.D. NINDS
  • 38. Special Thanks