preliminary evaluation of the program
Upcoming SlideShare
Loading in...5
×
 

preliminary evaluation of the program

on

  • 273 views

 

Statistics

Views

Total Views
273
Views on SlideShare
273
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

preliminary evaluation of the program preliminary evaluation of the program Presentation Transcript

  • INTERIM REPORT EVALUATION OF PILOT TO SHORTEN THE REVIEW CYCLE Presentation to Peer Review Advisory Committee 22 May 2006 Bettie J. Graham, Ph.D. for the Committee
  • THE LONG-TERM GOAL
    • TO ALLOW AMENDED APPLICATIONS TO BE SUBMITTED FOR THE VERY NEXT REVIEW CYCLE
  • RECOMMENDATIONS (Amended Applications)
    • PILOT R01* APPLICATIONS IN A FEW CSR STUDY SECTIONS (3 cycles)
    • EVALUATE
    • MAKE DECISION TO/WHEN TO MOVE TO NEXT PHASE
    • EXPAND TO ALL R01* APPLICATIONS IN ALL CSR STUDY SECTIONS
    • EVALUATE
    • MAKE DECISION TO/WHEN TO MOVE TO NEXT PHASE
    • EXPAND TO ALL APPLICATIONS IN ALL
    • CSR STUDY SECTIONS
    • EVALUATE
  • OVERARCHING PRINCIPLE
    • MAINTAIN CORE VALUES OF NIH PEER REVIEW
    • Expertise appropriate to review applications
    • Fairness/Objectivity in reviewing applications
  • Timeline for Shortening the Review Cycle 11/20 R01* Amended 7/20 R01* Amended Cycle 1 (Feb – Oct) Cycle 2 (Jun – Feb) Cycle 3 (Oct – Jun) Oct Nov Dec Jan Feb Mar Apr May Jun 10/1 Type 1 Receipt 1/15 - 2/15 Meetings May - Jun Councils 12/15 - 1/15 Mailouts 2/20 - 2/25 SS for R01* Applications Released and in Commons 11/25 - Last appl to IRG/ end of appls to IRG 11/1 Type 2 Receipt/all amendeds R01* 11/15 Deadline for Type 1 and Type 2 in IRG
  • Oct 2006 Applications 2,466 New investigator R01s received 10,025 Total R01 applications received 18,958 Total applications received
  • PURPOSE OF EVALUATION
    • TO PROVIDE INFORMATION TO NIH DECISION MAKERS WHO WILL DETERMINE IF/WHEN TO PROCEED TO THE NEXT PHASE.
  • DESCRIPTION OF PILOT EVALUATION PROCESS
    • 40 STUDY SECTIONS FROM 10 IRGS WERE SELECTED
    • STAKEHOLDERS INCLUDE:
      • REFERRAL OFFICERS
      • SCIENTIFIC REVIEW ADMINISTRATORS
      • REVIEWERS
      • PROGRAM ADMINISTRATORS
      • APPLICANTS
    • QUALITATIVE AND QUANTITATIVE INFORMATION WILL BE COLLECTED
  • EVALUATION OF PILOT
    • QUALITATIVE---SURVEYS
    • QUANTITATIVE---IMPAC II DATA
  • PARTICIPATING IRGS [NIH(25 ICs); FDA (1); CDC (4)] Surgical Sciences, Biomedical Imaging, and Bioengineering (4) Health of the Population (6) Respiratory Sciences (3) Digestive Sciences (5) Oncological Sciences (2) Infectious Diseases and Microbiology (3) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (3) Genes, Genomes and Genetics (6) Biological Chemistry and Macromolecular Biophysics (6) Bioengineering Sciences and Technologies (2)
  • IMPLEMENTATION OF PILOT
    • APPLICATIONS REFERRED TO STUDY SECTIONS EARLIER.
    • REVIEWERS HAVE APPROXIMATELY 4 WEEKS TO EVALUATE APPLICATIONS.
    • STUDY SECTION MEETINGS HELD EARLIER.
    • SUMMARY STATEMENTS WRITTEN WITHIN ONE WEEK FOR R01* AND 30 DAYS FOR ALL OTHER APPLICATIONS.
    • ADMINISTRATIVE NOTE ON R01* SUMMARY STATEMENTS ABOUT PILOT
    • APPLICANTS DISCUSS FEASIBILITY OF EARLY RESUBMISSION WITH PROGRAM DIRECTORS, IF APPLICABLE
    • AMENDED APPLICATIONS MAY BE SUBMITTED 20 DAYS AFTER RECEIPT DATE FOR TYPE 2/AMENDED APPLICATIONS.
  • COMMUNICATIONS TRIANGLE
    • Applicant
    • Program Review
  • COMMUNICATIONS TRIANGLE (cont’d)
    • SCIENTIFIC REVIEW ADMINISTRATORS
    • Eileen Bradley is meeting with CSR SRAs.
    • PROGRAM STAFF
    • Philip Smith (DK) is meeting with IC Program Staff and is providing feedback to CSR Staff.
    • APPLICANTS
      • Notice has been placed in the NIH Guide to Grants and Contracts
      • Addendum will be added to R01* Summary Statements
  • EVALUATION OF REFERRAL PROCESS
    • WHAT % OF APPLICATIONS WERE REFERRED 6 WEEKS AFTER THE TYPE 1 RECEIPT DATE?
    • WHAT % OF APPLICATIONS WERE SCANNED AND IN IAR 6 WEEKS AFTER THE TYPE 1 RECEIPT DATE?
  • EVALUATION OF REFERRAL PROCESS
    • WHAT WORKED?
    • WHAT DID NOT WORK?
    • HOW DID THIS PILOT AFFECT YOU PROFESSIONALLY AND PERSONALLY?
    • WHAT IMPROVEMENTS WOULD YOU RECOMMEND?
  • EVALUATION OF REFERRAL PROCESS
  •  
  •  
  • REVIEWER SURVEY
    • DO YOU KNOW WHAT THE GOALS OF THE PILOT ARE?
    • DID YOU RECEIVE YOUR APPLICATIONS AT LEAST 4 WEEKS PRIOR TO THE MEETING?
    • DID YOU HAVE SUFFICIENT TIME TO REVIEW YOUR APPLICATIONS?
    • DID YOU CHANGE ANY OF YOUR USUAL PRACTICES TO ACCOMMODATE THIS PILOT?
    • WHAT IMPROVEMENTS WOULD YOU RECOMMEND?
  • FINAL REVIEWER SURVEY
  • FINAL REVIEWER SURVEY
  • DATA COLLECTION ON APPLICANTS
    • Data will be collected from all chartered study sections in the pilot that review R01 grant applications.
    • Comparison groups will be:
      • Pilot new PIs who submit early
      • Pilot new PIs who do not submit early
      • Non-pilot new PIs.
  • DATA COLLECTION ON APPLICANTS
    • RESUBMISSIONS
      • When did applicants have access to their summary statements?
      • How many resubmitted early?
      • How many resubmitted later and when?
      • How many did not resubmit ?
  • DATA COLLECTION ON APPLICANTS
    • OUTCOMES
      • Differences in the priority scores
      • Differences in the funding rates
      • Differences in the time between first review and funding
  • WHAT NEXT?
    • ANALYZE DATA AS THEY BECOME AVAILABLE
    • DEVELOP INSTRUMENTS FOR OTHER STAKEHOLDERS (Scientific Review Administrator survey next)
    • PROVIDE FEEDBACK TO NIH LEADERSHIP
    • KEEP NIH INFORMED
  • THE COMMITTEE Bettie J. Graham, Ph.D. (Co-Chair) National Human Genome Research Institute Robert Hammond, Ph.D. Consultant Teresa Levitin, Ph.D. National Institute on Drug Abuse Jane Steinberg, Ph.D. National Institute of Mental Health Susan Streufert, Ph.D. National Institute of Child Health and Human Development Consultant Georgine Pion, Ph.D. Vanderbilt University Center for Scientific Review Eileen Bradley, D.Sc. (Co-Chair) Chuck Dumais, M.S. Teresa Lindquist , M.S. Elliot Postow, Ph.D. Terra L. Vinson, M.P.A.