Changes in peer review v3


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Changes in peer review v3

  1. 1. Office of the Vice President for Research and Division of Sponsored Programs April 7, 2010
  2. 2. <ul><li>New Investigators and Early Stage Investigators </li></ul><ul><li>Enhanced Review Criteria </li></ul><ul><li>Template-Based Critiques </li></ul><ul><li>Scoring </li></ul><ul><li>Order of Review </li></ul><ul><li>Summary Statements </li></ul>
  3. 3. <ul><li>New Investigator (NI) - PD/PI who has not yet competed successfully for a substantial NIH research grant (for multiple PIs, all must meet requirement for NI status). </li></ul><ul><li>Early Stage Investigator (ESI) - PD/PI who qualifies as a New Investigator AND is within 10 years of completing their terminal research degree or within 10 years of completing their medical residency (or equivalent). </li></ul><ul><li>Generally only applies to R01 applications. </li></ul>
  4. 4. <ul><li>Overall Impact - Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved (think paradigm shifting). </li></ul><ul><li>New Core Criteria Order </li></ul><ul><ul><li>Significance </li></ul></ul><ul><ul><li>Investigator(s) </li></ul></ul><ul><ul><li>Innovation </li></ul></ul><ul><ul><li>Approach </li></ul></ul><ul><ul><li>Environment </li></ul></ul>
  5. 5. <ul><li>Impact </li></ul><ul><li>Overall Impact is not a sixth review criterion and is not necessarily the arithmetic mean of the scores for the scored review criteria. Overall Impact is the synthesis/integration of the five core review criteria that are scored individually along with the additional review criteria which are not scored separately. </li></ul><ul><li>Significance </li></ul><ul><ul><li>Does the project address an important problem or critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? </li></ul></ul>
  6. 6. <ul><li>Provide concise, explicit and evaluative statements </li></ul><ul><li>Identify strengths and weaknesses for each criterion </li></ul><ul><li>Provide criterion scores </li></ul>1. Significance Limit text to ¼ page <ul><li>Strengths </li></ul><ul><li>Weaknesses </li></ul>
  7. 7. <ul><li>Scoring scale 1-9, whole numbers only </li></ul><ul><li>Impact score is NOT average of criterion scores </li></ul><ul><li>Applications reviewed in order of preliminary overall impact scores </li></ul><ul><li>Applications with initial scores between 7-9 are unlikely to be discussed </li></ul><ul><li>All applications receive criterion scores </li></ul><ul><li>Criterion scores are not discussed at review </li></ul>
  8. 8. Score Descriptor Additional Guidance on Strengths/Weaknesses 1 Exceptional Exceptionally strong with essentially no weaknesses 2 Outstanding Extremely strong with negligible weaknesses 3 Excellent Very strong with only some minor weaknesses 4 Very Good Strong but with numerous minor weaknesses 5 Good Strong but with at least one moderate weakness 6 Satisfactory Some strengths but also some moderate weaknesses 7 Fair Some strengths but with at least one major weakness 8 Marginal A few strengths and a few major weaknesses 9 Poor Very few strengths and numerous major weaknesses
  9. 9. <ul><li>Applications with best preliminary scores are reviewed first </li></ul><ul><li>NI/ESI R01s clustered at beginning of review </li></ul><ul><li>Other types of grants may be clustered if included in the same review meeting (R03, R15, R21, etc) </li></ul><ul><li>Final scores of discussed applications may differ from preliminary scores (happens >50% of the time) </li></ul>
  10. 10. <ul><li>Only 50-60% of applications at review meeting will be discussed </li></ul><ul><li>Reviewers can request that any other application also be discussed </li></ul><ul><li>The remaining applications will NOT be discussed (applications receive criterion scores only). These applications are considered to have low impact. </li></ul>
  11. 11. <ul><li>Assigned reviewers (typically 3 primary) define the range of scores for each specific application </li></ul><ul><li>At the time of the review, scores are provided by all present. Any reviewer who plans to score outside the range established by the assigned (primary) reviewers needs to declare their intention, which is based on their determination of overall impact. </li></ul>
  12. 12. <ul><li>Discussed applications receive an overall score from each eligible (non-conflicted) panel member. Scores are averaged to one decimal place and multiplied by 10. Hence, there are 81 possible priority scores that range from 10 to 90. </li></ul><ul><li>Percentiles are reported as whole numbers </li></ul>
  13. 13. <ul><li>Summary statement is short and focused </li></ul><ul><li>Only discussed applications receive a summary of the panel’s deliberation </li></ul><ul><li>Applications not discussed receive only criterion scores and primary reviewers’ critiques </li></ul>
  14. 14. <ul><li>Critiques: The Critiques below were prepared by the reviewers assigned to this application. These commentaries do not necessarily reflect the position of the authors at the close of the group discussion, nor the final majority opinion of the group , although reviewers are asked to amend their critiques if their position changed during the discussion. The resume and other initial sections of the summary statement are the authoritative representation of the final outcome of group discussion. If there is any discrepancy between the peer reviewers’ commentaries and the action statement for a given component, or the overall score, the action statement for the component, the overall critique, and the overall numerical score on the face page of this summary statement should be considered the most accurate representation of the final outcome of the group discussion . </li></ul>
  15. 15. <ul><li>Read RFA and explicitly follow directions </li></ul><ul><li>Write to the review criteria </li></ul><ul><li>New page limitations require that you </li></ul><ul><ul><li>focus your writing </li></ul></ul><ul><ul><li>limit jargon </li></ul></ul><ul><ul><li>clearly define significance and overall impact </li></ul></ul><ul><ul><li>articulate the big picture in lieu of minutiae </li></ul></ul>