Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Final phase ii scientific reviewer training


Published on

Published in: Health & Medicine
  • Be the first to comment

Final phase ii scientific reviewer training

  1. 1. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTEPa#ent-­‐Centered  Outcomes  Research  Ins#tute  (PCORI)  Phase  II  Scien4fic  Reviewer  Training   October  2012  
  2. 2. Housekeeping  –  Presenta4on  Mode  Ø  A9endee  phone  lines  are  muted    Ø  Ques#ons  may  be  submi9ed  via  Chat  in   the  lower  right  hand  side  of  your  screen   à      Ø  Please  send  ques#ons  as  they  occur  to   you.  They  will  be  answered  at  the  end  of   1. Type your the  session,  as  #me  permits   question here.  Ø  Press  “0”  on  the  phone  for  a  private  help   session  with  the  operator   2. Click Send
  3. 3. Agenda               1.  Introduc4on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  4. 4. Our  SROs   Assessment  of  Preven4on,  Diagnosis,  and   Communica4on  and  Dissemina4on   Treatment  Op4ons     Research   Howard  Underwood,  MD,   Marianne  H.   Kimberly  A.  Marschhauser,   MBA,  MS   Alcia#,  Ph.D.   Ph.D.   Jessica  Nadler,  Ph.D.   Improving  Healthcare  Systems   Addressing  Dispari4es   Lev  Nevo,  MD   Sabina  I.  Robinson,  Ph.D.   Parag  Aggarwal,  Ph.D.  
  5. 5. Announcements   Open  session  to  any  per4nent  announcements     Key  Dates   Phase  II  Assignments  Released  –  Oct  12th     Preliminary  Scores  Due  –  November  2nd  at  5:00  pm     Op#onal  Dinner  –  Nov  14th  in  Washington,  DC  from  6:00  pm  –  9:30  pm     Phase  II  Panels  –  Nov  15th  in  Washington,  DC  from  7:00  am  –  3:00  pm  
  6. 6. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  7. 7. PCORI  Mission,  Vision  and  PCOR  Ø  PCORI  is  a  non-­‐governmental,  non-­‐profit   organiza#on  founded  by  the  Pa#ent   Protec#on  and  Affordable  Healthcare  Act  of   PCORI  Mission  Statement   2010    ( h9p:// PCORI  helps  people  make  informed  healthcare   PCORI_EstablishingLeg.pdf)   decisions  and  improves  healthcare  delivery  and     outcomes  by  producing  and  promo#ng  high  integrity,  Ø  PCORI  aims  to  fund  pa#ent-­‐centered   evidence-­‐based  informa#on  that  comes  from   research  that  will  improve  healthcare   research  guided  by  pa#ents,  caregivers,  and  the   outcomes  for  pa#ents,  their  caregivers,  and   broader  health  care  community.   other  stakeholders      Ø  Pa4ent-­‐centered  outcomes  research   (PCOR)  helps  people  and  their  caregivers   Vision   communicate  and  make  informed   Pa#ents  and  the  public  have  the  informa#on  they   healthcare  decisions,  allowing  their  voices   need  to  make  decisions  that  reflect  their  desired   to  be  heard  in  assessing  the  value  of   health  outcomes.   healthcare  op#ons     For  more  informa#on  on  PCOR,  please  reference  the  PCORI  Methodology  report  at:   h9p://­‐Comment.pdf    
  8. 8. PCORI’s  Na4onal  Priori4es   Purpose   Methodologies   Research  Agenda   Provide  informa#on  to  PCORI   Support  iden#fica#on  of   Support  the  collec#on  of  preliminary   that  informs  future  itera#ons  of   research  methodologies   data  to  advance  the  field  of  pa#ent-­‐ na#onal  research  priori#es  for   that  advance  pa#ent-­‐ centered  outcomes  research,  providing   pa#ent-­‐centered  outcomes   centered  outcomes   the  plagorm  for  an  evolving  PCORI   research.     research   research  agenda.  PCORI’s  ini#al   Research  Agenda:   The  Na#onal  Priori#es  are:   1.  Comparisons  of  Preven#on,   1.  Compara#ve  Assessments  of   Diagnosis,  and  Treatment  Op#ons   Preven#on,  Diagnosis,  and   2.  Improving  Healthcare  Systems   Treatment  Op#ons   3.  Communica#on  &  Dissemina#on   2.  Improving  Healthcare   4.  Addressing  Dispari#es   Systems   5.  Accelera#ng  Pa#ent-­‐Centered  and   3.  Communica#on  and   Methodological  Research   Dissemina#on   4.  Addressing  Dispari#es   5.  Accelera#ng  Pa#ent-­‐ Centered  and   Methodological  Research  
  9. 9.    Why  PCORI  is  Unique   PCORI  is  unique  because:  (a)  it  requires  stakeholders  included  as  part  of  the  research   team,  and  (b)  research  must  be  focused  on  pa4ent-­‐centered  outcomes     •  Projects  must  include  stakeholders  as  partners  with  significant  involvement  at  all  appropriate   stages  of  the  research  project   •  Tangible,  meaningful  outcomes  are  the  ul#mate  goal  of  all  funded  research   Who  are  Stakeholders?   •  Pa#ents  and  caregivers   •  Payers   •  Pa#ent  and  caregiver  organiza#ons   •  Industry   •  Clinician  and  clinician  organiza#ons   •  Researchers   •  Organiza#onal  Providers   •  Policymakers   •  Purchasers   •  Training  ins#tu#ons     •  Others  who  can  bring  insight  
  10. 10. Stakeholder  Engagement  PCORI  is  seizing  the  opportunity  to  engage  stakeholders  in  unprecedented  ways:   •  Partners  in  the  research  project  enterprise   Why Engage •  Inclusion  as  equal  partners  in  research  review   Stakeholders? •  Leverage  their  value,  including  wisdom  and  unique   To  create  more  relevant   decision-­‐making  tools  to  assure   exper#se   be9er  pa#ent  outcomes   •  Increase  the  relevance  and  impact  of  research  by   integra#ng  mul#ple  stakeholders  into  the  process   •  Foster  environments  that  facilitate  cross-­‐fer#liza#on  and  novel  collabora#ons    
  11. 11. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  12. 12. PCORI  Funding  Announcements  PCORI  Funding  Announcements  (PFAs)  are  the  mechanisms  by  which  PCORI  gives  out  research  funding  The  current  funding  cycle  has  four  issued  PFAs:   And  coming  this  fall  2012:   Assessment  of   Improving   Accelerang  Paent-­‐ Preven4on,   Healthcare   Centered  Outcomes   Diagnosis,  and   Systems   Research  and   Treatment  Op4ons     Methodological     Research   Communica4on   and   Addressing   Dissemina4on   Dispari4es   Research      
  13. 13. PFAs  Assessment  of  Preven4on,  Diagnosis,  and  Treatment  Op4ons   Projects  that  address  cri4cal  decisions  that  pa4ents,  their  caregivers,  and  clinicians  face   with  too  lible  informa4on   In  this  PFA  we  seek  to  fund  projects  that:     Available  funds:    $48  Million   •  Address  crical  decisions  that  face  paents,  their   caregivers,  and  clinicians  every  day  and  with  too   Expected  awards:    54  awards   li?le  informaon   Maximum  project  period:  3   years   •  Address  consequenal  decisions  now  occurring   without  key  evidence  about  the  comparave   Ini#al  funding  period:   effecveness  of  two  or  more  opons   December  2012  –  January   2013   •  Benefit  paents/caregivers  with  new  knowledge  in   ways  that  are  clear  and  important    
  14. 14. PFAs  Improving  Healthcare  Systems   Projects  that  address  cri4cal  decisions  that  face  healthcare  systems,  the  pa4ents  and   caregivers  who  rely  on  them,  and  the  clinicians  who  work  within  them   In  this  PFA  we  seek  to  fund  projects  that:     Available  funds:    $24  Million   •  Address  crical  decisions  that  face  healthcare   system  leaders  and  policymakers,  clinicians,  and   Expected  awards:    27  awards   the  paents  and  caregivers  who  rely  on  them   Maximum  project  period:  3   years   •  Offer  substanal  potenal  that  paents/caregivers   will  benefit  from  the  new  knowledge  in  ways  that   Ini#al  funding  period:   are  important   December  2012  –  January     2013  
  15. 15. PFAs   Communica4on  and   Dissemina4on  Research   Projects  that  address  cri4cal  elements  in  the  communica4on  and  dissemina4on  process   among  pa4ents,  their  caregivers  and  clinicians   In  this  PFA  we  seek  to  fund  projects  that:     Available  funds:    $12  Million   •  Address  crical  knowledge  gaps  in  the   Expected  awards:    14  awards   communicaon  and  disseminaon  process   Maximum  project  period:  3   •  Gaps  to  consider:     years   Ø  The  communicaon  and  disseminaon  of   research  results  to  paents,  their  caregivers,   Ini#al  funding  period:   and  clinicians     December  2012  –  January   Ø  The  communicaon  between  paents,   2013   caregivers,  and  clinicians  in  the  service  of   enabling  paents  and  caregivers  to  make  the   best  possible  decisions  in  choosing  among   available  opons  for  care  and  treatment  
  16. 16. PFAs   Addressing  Dispari4es   Projects  that  will  inform  the  choice  of  strategies  to  eliminate  dispari4es   In  this  PFA  we  seek  to  fund  projects  that:     •  Will  inform  the  choice  of  strategies  to  eliminate   Available  funds:    $12  Million   disparies   Ø  We  are  not  interested  in  studies  that   Expected  awards:    14  awards   describe  disparies;  instead,  we  want   studies  that  will  idenfy  best  opons  for   Maximum  project  period:  3   eliminang  disparies   years   Ini#al  funding  period:   •  Focus  on  areas  of  importance  to  paents  and   December  2012  –  January   their  caregivers,  where  there  are  crical   2013   disparies  that  disadvantage  members  of  a   parcular  group  and  limit  their  ability  to  achieve   opmal,  paent-­‐centered  outcomes  
  17. 17. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica4on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  18. 18. Applica4on  and  Review  Process  Summary   Merit   Applica4on   Approval   Review   •  LOI  submission  via   •  Phase  I:  Scien#fic/   •  PCORI  Business   PCORI  Online   Technical  Review   Review  and   Balance  Analysis   •  Applica#on   •  Phase  II:  In-­‐person   submission  via   panel  –  Impact   •  Board  of   PCORI  Online   Review   Governors   Approval   •  Internal  quality   control  
  19. 19. The  Merit  Review  Process  The  process  by  which  applica4ons  for  research  funding  are  evaluated  –     Phase  I   •  Each  applica#on  is  assigned  to  a  pre-­‐ determined,  set  number  of  reviewers   Phase  II   •  Phase  I  Reviewers  have  scien#fic   •  Scien#st  and  Pa#ents/Stakeholders   exper#se,  and  assess  the  applica#on  for   assess  Phase  I  cri#que  and  assign  one  (1)   scien#fic  rigor  and  research  approach   preliminary  score  and  cri#que   •  Reviewers  assign  an  ini#al  priority  score   •  Panel  of  chairs  plus  two  scien#fic,  one   of  1  to  9  based  on  all  eight  PCORI  merit   stakeholder,  and  one  pa#ent  reviewer   review  criteria   convene  in-­‐person  for  discussion  and  re-­‐ •  Scores  are  compiled     score   •  Top  scoring  applica#ons  proceed  to     Phase  II    
  20. 20.    Merit  Review  Phase  II:  Overview  As  a  Phase  II  Reviewer,  you  are  responsible  for  assessing  and  appropriately  scoring  your  assigned  applica4ons   Assign   In-­‐Person   Access  Assigned   Final  Impact   COI   Preliminary  Score   Review   Cri4ques  &  Scores   Scoring   &  Cri4que   Panels   Key Tasks 1.  Access  Phase  II  scores  and  cri#ques  in  PCORI  Online   2.  Conflict  of  Interest  (COI):  Ensure  no  conflict  exits   3.  Assign  preliminary  numerical  preliminary  impact  score  (1-­‐9)  and  provide  cri#que   4.  Panels  convene  and  discuss   5.  Assign  final  impact  scores   Raise  issues,  risks,  and  request  support  as  needed  
  21. 21.  Conflicts  of  Interest   What  is  a  conflict  of  interest?   As  defined  by  PCORI’s  establishing  legisla#on,  a  conflict  of  interest  is  any  “associa#on,  including  a   financial  or  personal  associa#on;  that  has  the  poten#al  to  bias  or  have  the  appearance  of  biasing   an  individual’s  decisions  in  ma9ers  related  to  the  Ins#tute  or  the  conduct  of  ac#vi#es”.         Conflicts  of  interest  will  be  considered  and  prohibited  throughout  every  step  of  the  review  and   selec#on  process,  including  but  not  limited  to:  the  technical  and  programma#c  reviews,  the   selec#on  and  assignment  of  scien#fic  and  stakeholder  reviewers,  Board  of  Governors   delibera#ons,  and  post-­‐award  nego#a#ons  and  monitoring.   More  informa4on  is  included  in  the  PCORI  Online  confiden4ality  and  conflict  of  interest  document   that  you  must  agree  to  in  order  to  access  your  applica4ons.    
  22. 22.    Your  Role   Some  addi4onal  guidance  about  your  role  and  ac4vi4es  as  a  Phase   II  Scien4fic  Reviewer:   Before  the  in-­‐person  review  panels  on  November  15th:   •  Access  your  assigned  applica#ons  in  PCORI  Online   •  Score  and  provide  wri9en  comments   •  Be  prepared  to  substan#vely  qualify  and  discuss  your  score  and  comments  during   the  in-­‐person  review  panel     During  the  in-­‐person  review  panel:   •  Reviewers  assigned  to  each  applica#on  will  briefly  discuss  their  preliminary  score   and  provide  feedback   •  Open  to  panel-­‐wide  discussion   •  All  reviewers  assign  a  final  score  in  real-­‐#me,  on  personal  laptops  via  PCORI   Online   •  Scores  are  compiled  and  averaged  by  PCORI   •  Top  scoring  applica#ons  will  be  forwarded  and  receive  recommenda#on  for   funding  
  23. 23. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien4fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  24. 24. Merit  Review  Criteria   •  Impact  of  the   Condi4on   •  Innova#on/     Criterion  1:  Impact  of  the  Condi?on     on  the  Health  of  Individuals  and  Popula?ons   Poten#al  for   Improvement   §  Does  the  applica#on  specify  the  burden  of  the  disease  or  area  under   •  Impact  on   considera#on,  with  a  preference  for  the  U.S.  popula#on,  including:   Healthcare   Ø  The  frequency  of  the  disease/condi#on,     Performance   Ø  Expected  mortality  and  burden  of  suffering  from  symptoms,       •  Pa#ent-­‐ Ø  Complica#ons  or  other  consequences  of  the  disease/condi#on,       Centeredness   Ø  The  frequency  with  which  the  interven#on  or  treatment  would   •  Rigorous  Research   apply,   Methods   Ø  Costs  to  the  US  popula#on  (healthcare  services  u#liza#on),  and   to  individual  pa#ents  (out-­‐of-­‐pocket  and  intangible  costs).   •  Inclusiveness  of   Different   §  Primary  emphasis  is  on  chronic  condi#ons,  as  well  as  preven#on  and   Popula#ons   treatment  of  common  acute  events  that  may  have  long-­‐term   •  Team  and   consequences.   Environment   §  Studies  that  are  relevant  to  pa#ents  with  two  or  more  condi#ons  are   •  Efficient  Use  of   also  of  interest.  Also  of  interest  are  rare  diseases.     Resources  
  25. 25. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova4on/     Poten4al  for   Criterion  2:  Innova?on  and  Poten?al  for  Improvement  Through   Improvement   Research   •  Impact  on   §  How  will  the  research  influence  current  prac#ce  and  lead  to  meaningful   Healthcare   improvement  in  pa#ent  health,  well-­‐being,  or  quality  of  care?   Performance   §  Does  the  research  involve  a  novel  interven#on  or  employ  an  innova#ve   •  Pa#ent-­‐ approach  in  terms  of  analy#cs,  study  popula#on,  or  research  team  that   Centeredness   makes  it  more  likely  to  change  prac#ce?     •  Rigorous  Research   §  Does  preliminary  data  suggest  that  the  comparison  will  show  large   Methods   differences  in  effec#veness?   •  Inclusiveness  of   §  Does  the  research  ques#on  address  a  cri#cal  gap  in  current  knowledge?   Different   Has  it  been  iden#fied  as  important  by  pa#ent,  caregiver,  or  clinician   Popula#ons   groups?  Have  other  agencies  iden#fied  this  topic  as  a  priority?   •  Team  and   §  How  quickly  could  posi#ve  findings  be  disseminated  to  affect  changes  in   Environment   current  prac#ce?  How  will  the  research  findings  support  improved   •  Efficient  Use  of   decision-­‐making  for  pa#ents?   Resources  
  26. 26. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for   Improvement   Criterion  3:  Impact  on  Healthcare  Performance   •  Impact  on   Healthcare   §  What  is  the  impact  of  the  proposed  research  on  the  efficiency  of   Performance   pa#ent  care,  for  individual  pa#ents  or  for  pa#ent  popula#ons?   •  Pa#ent-­‐ §  For  example,  do  the  findings  lead  to  be9er  outcomes  for  a  given   Centeredness   investment  of  #me,  personnel,  or  other  resources?  Or  does  the   research  promise  poten#al  improvements  in  convenience  or   •  Rigorous  Research   elimina#on  of  wasted  resources,  while  maintaining  or  improving   Methods   pa#ent  outcomes?   •  Inclusiveness  of   Different   Popula#ons   •  Team  and   Environment   •  Efficient  Use  of   Resources  
  27. 27. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for   Criterion  4:  Pa?ent-­‐Centeredness   Improvement   §  Is  the  proposed  research  focused  on  ques#ons  and  outcomes  of   •  Impact  on   specific  interest  to  pa#ents  and  their  caregivers?  Pa4ent-­‐ Healthcare   centeredness  is  a  perspec4ve  on  health  that  is  derived  from  and   Performance   directly  relevant  to  the  pa4ent’s  experience  of  illness  and  of  care.     •  Pa4ent-­‐ Centeredness   §  Does  the  research  address  one  or  more  of  the  key  ques#ons   men#oned  in  PCORI’s  defini#on  of  pa#ent-­‐centered  outcomes   •  Rigorous  Research   research?     Methods   •  Inclusiveness  of   §  Are  the  outcomes  proposed  of  importance  to  pa#ents?  Is  the   Different   absence  of  any  par#cularly  important  outcomes  discussed?   Popula#ons   §  Pa#ent  engagement  in  the  research  team  is  dis#nct  and  discussed  in   •  Team  and   Criterion  7,  Team  and  Environment.     Environment   •  Efficient  Use  of   Resources  
  28. 28. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for   Criterion  5:  Rigorous  Research  Methods   Improvement   §  Does  the  research  use  appropriate  and  rigorous  research  methods   •  Impact  on   to  generate  pa#ent-­‐centered  evidence?   Healthcare   Performance   Ø  Applicants  are  encouraged  to  refer  to  the  contents  of  the   first  dras  of  the  PCORI  Methodology  Report,  at   •  Pa#ent-­‐ h9p://­‐we-­‐do/methodology,  in   Centeredness   developing  their  research  plan.  Because  the  dras  report  will   •  Rigorous  Research   not  have  been  finalized  with  the  benefit  of  public  comment   Methods   before  the  July  31st,  2012  applica#on  deadline,  adherence  to   •  Inclusiveness  of   the  Report’s  standards  will  not  be  a  required  element  of   Different   applica#ons  for  this  funding  cycle.     Popula#ons   Ø  How  likely  is  it  that  the  proposed  study  popula#on,  study   •  Team  and   design,  and  available  sample  size  will  yield  generalizable   Environment   informa#on  with  sufficient  precision  to  be  useful  and  reliable   for  pa#ents,  their  caregivers,  and  clinicians?   •  Efficient  Use  of   Resources      
  29. 29. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for   Criterion  6:  Inclusiveness  of  Different  Popula?ons   Improvement   •  Impact  on   §  Does  the  research  include  diverse  popula#ons  with  respect  to   Healthcare   age,  gender,  race,  ethnicity,  geography,  or  previously   Performance   understudied  popula#ons  for  whom  effec#veness  informa#on  is   par#cularly  needed?  Is  the  study  popula#on  representa#ve  of  the   •  Pa#ent-­‐ full  popula#on  of  interest?   Centeredness   •  Rigorous  Research   §  How  does  the  proposed  research  enable  a  more  personalized   Methods   approach  to  decision-­‐making  based  on  a  pa#ent’s  unique   biological,  clinical,  or  socio-­‐demographic  characteris#cs?   •  Inclusiveness  of   Different   §  Does  the  study  provide  sample  size  calcula#ons  that  will  describe   Popula4ons   the  power  available  to  evaluate  possible  differences  in   •  Team  and   effec#veness  in  different  groups,  or  the  precision  available  for   Environment   es#ma#ng  effec#veness  in  a  specific  previously  understudied   popula#on?   •  Efficient  Use  of   Resources  
  30. 30. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for    Criterion  7:  Team  and  Environment   Improvement   •  Impact  on   §  Are  the  inves#gators  appropriately  trained  and  experienced  to   Healthcare   carry  out  the  planned  studies?  Is  the  work  proposed  appropriate   Performance   to  the  experience  level  of  the  principal  inves#gator?     •  Pa#ent-­‐ §  Does  the  study  team  have  complementary  and  integrated   Centeredness   exper#se;  is  their  leadership  approach,  governance,  and   •  Rigorous  Research   organiza#onal  structure  appropriate  for  the  project?     Methods   §  Are  relevant  pa#ents  and  other  key  stakeholders  of  the  study   •  Inclusiveness  of   informa#on  appropriately  included  on  the  team?     Different   Popula#ons   §  Do  the  experiments  proposed  take  advantage  of  unique  features   of  the  scien#fic  environment  or  employ  useful  collabora#ve   •  Team  and   arrangements?     Environment   •  Efficient  Use  of   §  Is  there  evidence  of  ins#tu#onal  or  other  support?     Resources    
  31. 31. Merit  Review  Criteria   •  Impact  of  the   Condi#on   •  Innova#on/     Poten#al  for    Criterion  8:  Efficient  Use  of  Research  Resources   Improvement   •  Impact  on   §  Does  the  budget  appear  to  be  reasonable  in  rela#on  to  the   Healthcare   poten#al  contribu#on  of  the  research?     Performance   §  Does  the  jus#fica#on  address  the  efficiency  with  which  PCORI   •  Pa#ent-­‐ resources  would  be  used?  Are  there  opportuni#es  to  make  the   Centeredness   study  more  efficient?   •  Rigorous  Research   §  Are  there  addi#onal  benefits  to  a  PCORI  investment  in  this  study   Methods   through  the  crea#on  of  common  data  or  infrastructure  that  could   •  Inclusiveness  of   support  future  research?   Different   Popula#ons   •  Team  and   Environment   •  Efficient  Use  of   Resources  
  32. 32. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  33. 33.    The  Focus  on  Impact   PCORI  Defini4on   Reviewers  will  provide  an  overall  impact   score  that  considers  the  following:     •  Does  the  project  have  poten4al  to   The  assessment  of  impact  is  par4cularly   change  clinical  prac4ce  or  pa4ent   informed  by  three  of  the  eight  PCORI   behavior  in  ways  that  will  create  and   Merit  Review  Criteria.     sustain  improvement  in  outcomes     and  the  health  of  pa4ents?   Use  criteria  2,  4,  and  7  to  evaluate  an   applica4on’s  answer  to  these  ques4ons   •  How  quickly  can  the  results  of  the   project  be  disseminated  and  applied   (from  the  assessment  of   dissemina4on  and  implementa4on   poten4al)?  
  34. 34.    The  8  Merit  Review  Criteria:  Phase  II  Focus   Phase  II  is  centered  around  impact,  focusing     on  the  following  of  the  Merit  Review  Criteria:   The  8  Merit  Review  Criteria:   Innova4on/   1.  Impact  of  the  Condi#on   Poten4al  for   Pa4ent   Team  and   Improvement   Centeredness   Environment   2.  Innova4on/Poten4al  for         •  Innova#on  –  in   •  Focus  on   •  Inves#gators   Improvement   ways  that  are   ques#ons  and   trained   3.  Impact  on  Healthcare   likely  to  change   outcomes  of   •  Study  team   prac#ce?   specific  interest   exper#se   Performance   •  Poten#al  for   to  pa#ents  and   •  Plan  for  leadership   improvement   their  caregivers   and  governance   4.  Pa4ent-­‐Centeredness   (will  findings   •  Robust  pa#ent   5.  Rigorous  Research  Methods   improve  pa#ent   and  stakeholder   well-­‐being  or   engagement  plan   6.  Inclusiveness  of  Different   quality  of  care?)   •  Inclusiveness  of   different   Popula#ons   popula#ons   7.  Team  and  Environment   •  Ins#tu#onal  or   other  relevant   8.  Efficient  Use  of  Resources   organiza#onal   support  
  35. 35.    Phase  II  Key  Focus  Areas   Criterion  2:  Innova?on  and  Poten?al  for   Improvement  Through  Research     •  “Is  there  uncertainty?”   –  Varia#on  in  prac#ce,  systema#c  reviews  have  iden#fied  as  such,  or  pa#ent/clinician  groups  have   specifically  called  for  this  informa#on   •  How  will  the  research  influence  current  prac#ce  and  lead  to  meaningful  improvement  in  pa#ent  health,   well-­‐being,  or  quality  of  care?     •  Does  the  research  involve  a  novel  interven#on  or  employ  an  innova#ve  approach  in  terms  of  analy#cs,   study  popula#on,  or  research  team  that  makes  it  more  likely  to  change  prac#ce?       •  Does  preliminary  data  suggest  that  the  comparison  will  show  large  differences  in  effec#veness?     •  Does  the  research  ques#on  address  a  cri#cal  gap  in  current  knowledge?  Has  it  been  iden#fied  as  important   by  pa#ent,  caregiver,  or  clinician  groups?  Have  other  agencies  iden#fied  this  topic  as  a  priority?     •  How  quickly  could  posi#ve  findings  be  disseminated  to  affect  changes  in  current  prac#ce?  How  will  the   research  findings  support  improved  decision-­‐making  for  pa#ents?     –  PCORI  is  interested  in  funding  studies  with  a  high  likelihood  that  results  will  be  disseminated  and   incorporated  into  prac#ce  immediately  or  within  a  short  period  of  #me  (3-­‐  5  years).    Please  refer  to   the  dissemina#on  and  implementa#on  assessment  in  the  applica#on  for  detail  and  clarifica#on,  if   necessary.  
  36. 36.    Phase  II  Key  Focus  Areas   Criterion  4:  Pa?ent  Centeredness   •  Is  the  proposed  research  focused  on  ques#ons  and  comparisons  that  have  relevance  and   specific  interest  to  pa#ents  and  their  caregivers?  Pa4ent-­‐centeredness  is  a  perspec4ve  on   health  that  is  derived  from  and  directly  relevant  to  the  pa4ent’s  experience  of  illness  and  of   care.   •  Does  the  research  fit  with  one  or  more  of  the  key  ques#ons  men#oned  in  PCORI’s  defini#on  of   pa#ent-­‐centered  outcomes  research?   •  Are  the  outcomes  proposed  of  importance  to  pa#ents?  Is  the  absence  of  any  par#cularly   important  outcomes  discussed?   •  Note:  Pa#ent  engagement  in  the  research  team  is  dis#nct  and  discussed  in  Criterion  7,  Team   and  Environment.  
  37. 37.    Phase  II  Key  Focus  Areas    Criterion  7:  Team  and  Environment   •  Are  the  inves#gators  appropriately  trained  and  experienced  to  carry  out  the  planned  studies?   Is  the  work  proposed  appropriate  to  the  experience  level  of  the  principal  inves#gator?       •  Does  the  study  team  have  complementary  and  integrated  exper#se;  is  their  leadership   approach,  governance,  and  organiza#onal  structure  appropriate  for  the  project?       •  Are  relevant  pa#ents  and  other  key  stakeholders  in  the  study  informa#on  appropriately   included  on  the  team?       •  Do  the  proposed  experiments  take  advantage  of  unique  features  of  the  scien#fic  environment,   or  employ  useful  collabora#ve  arrangements?       •  Is  there  evidence  of  ins#tu#onal  support?  
  38. 38.    Phase  II  Preliminary  Scoring   Assignments  Released   Preliminary  Scoring   •  Assignments  are  released  in   •  Use  Phase  I  cri#ques  to  assign   October   preliminary  score  of  1-­‐9     –  You  will  have  access  to  the  full   •  In  PCORI  Online   applica#on,  but  please  use  only  to   reference   •  Login  to  access  your  applica#ons     –  Ensure  no  conflicts  of  interest,   •  Provide  substan#ve,  produc#ve   and  your  qualifica#on  to  review   comments  as  well  as  specific   strengths  and  weaknesses  to   •  All  applica#ons  re-­‐released  and   ul#mately  help  answer  the  ques#on:   assigned  to  Phase  II  Reviewers   –  “How,  and  to  what  extent,  will  the   were  top  scorers  in  Phase  I   proposed  research  plan  impact   –  Have  been  ve9ed  for  basic,  hard   paents  in  the  next  3  to  5  years?”   science  and  programma#c   review   •  Enter  comments  and  numerical  score   in  PCORI  Online  
  39. 39.    Reviewer  Guidance  &  Scoring  Chart   For  the  preliminary  Phase  II  impact  score,  the  far  right  column  in  the  scoring  chart  below   provides  a  descrip4ve  guide  of  how  strengths  and  weaknesses  are  considered  in  a  ra4ng:       Impact Score Descriptor 1 Exceptional High 2 Outstanding 3 Excellent 4 Very  Good Medium 5 Good 6 Satisfactory 7 Fair Low 8 Marginal 9 Poor
  40. 40. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  41. 41.    General  Logis4cs  for  Phase  II  Panel  Reviewers   Date/Loca4on   Details   Date  of  Phase  II  Panels:   Thursday,  November  15,  2012   Loca4on:   Hya9  Regency  Washington  on  Capitol  Hill,  Washington  D.C.   Time:     8:00am  to  3:00pm   Number  of  Panels:     Five     Ø  Panel  par#cipants  will  reflect  the  fact  that  each  applica#on  has  2  scien#sts,  one   stakeholder,  and  one  pa#ent  assigned  to  provide  commentary  and  preliminary  scores  
  42. 42.    Phase  II  Panel  Process     Introduction Presentation Overview and Triage Co-Chair Presents Application Synopsis •  Chair  Introduc#on   •  SRO  captures  panel  discussion     ‒  Stress  confiden#ality  and  focus  on  impact   •  Individual  review  and  scoring  on  PCORI  Online  on   •  Triage  process  to  eliminate  lowest-­‐ranked   personal  laptops   applica#ons  from  panel  discussion   Review Scoring Discussion of Application Open to Discussion •  Provides  their  preliminary  impact  score  and   •  All  panelists  free  to  discuss,  Chair  moderates  if   assessment    and  its  poten#al  for  significant   needed   outcomes/impact   •  If  no  discussion  –  move  to  final  vote.  Reviewers   score  individually  in  PCORI  Online  
  43. 43.    Triage  Process   During  Phase  II,  a  triage  process  will  take  place:   Ø  Applica#ons  are  ordered  according  to  ranking  (highest  to  lowest  scores)   Ø  Lowest  scoring  applica#ons  will  be  eliminated  all  at  once  from  in-­‐person  panel   discussion     Ø  If  you  as  a  Reviewer  want  to  specifically  discuss  an  applica#on,  please  come  to   the  panels  prepared  to  do  so  
  44. 44.    Roles  &  Responsibili4es     Each  panel  will  be  lead  by  a  Chair,  Co-­‐Chair,  and  SRO  in   tradi4onal  advisory  roles   Scien4fic  Reviewers  (2)   Pa4ent  Reviewer   Stakeholder  Reviewer   Role   Provide  addi#onal  depth  for   Provides  addi#onal  depth  for    up  to  10  applica#ons  for  all           up  to  10  applica#ons  for  all   par#cipants   par#cipants   Key   Provides  their  preliminary  impact  score  and  assessment    and  its  poten#al  for  significant  outcomes/ Responsibili4es   impact   PCORI  review  is  different  because  there  is  no  primary/secondary/ter#ary  reviewer  structure   •  Also  u#lizes  different,  PCORI-­‐unique  merit  review  criteria   Each  reviewer  must  provide  substan#ve  strengths  and  weaknesses  for  his/her  assigned   applica#on(s)     •  Reviewers  must  be  prepared  to  speak  to  these  comments  and  scoring  during  the  in-­‐person   review  panels  
  45. 45.    Time  Breakdown  per  Applica4on  NOTE:  Panels  will  spend  no  more  than  10-­‐15  minutes  per    applica4on.  An  example:     Up  to…   Descrip4on   1  minute   Co-­‐chair  briefly  introduces  applica#on   2  minutes   Scien#fic  Reviewer  #1:  overview  and  score   2  minutes   Stakeholder  reviewer:  overview  and  score   2  minutes   Scien#fic  Reviewer  #2:  overview  and  score   2  minutes   Pa#ent  Reviewer:  overview  and  score   4  minutes   General  discussion,  if  any     2  minutes   Take  vote  and  enter  scores  in  PCORI  Online   Ø  Some  applica#ons  may  be  reviewed  in  less  than  the  15  minutes  allocated.   Ø  It  is  important  to  understand  the  #me  constraints  and  keep  conversa#on  focused,  pointed,  and   succinct  throughout  the  day  to  ensure  fair  and  proper  scoring  of  all  applica#ons  
  46. 46. Agenda               1.  Introduc#on  and  Announcements   2.      Background   3.      Program  Funding  Announcements     4. The  Applica#on  and  Review  Process   5. Merit  Review:  Phase  I  –  Scien#fic  Review   6.      Merit  Review:  Phase  II  –  Impact  Review   7.      Merit  Review:  Phase  II  –  In-­‐Person  Panel   8.      Phase  II:  PCORI  Online  –  Process  and  Procedures  
  47. 47.    PCORI  Online:  Confiden4ality  Agreement   The  first  thing  you  will  have  to  do  upon  log-­‐in  is  agree  to  the  Confiden4ality  Agreement.  This  applies   to  both  preliminary  scoring,  and  any/all  scores  and  discussion  that  take  place  during  the  in-­‐person   review  panel.    
  48. 48.    Accessing  Your  Assigned  Applica4ons   Next,  you  will  be  able  to  see  your  list  of  assigned  applica4ons  by  selec4ng  “Review  Assignments”  in   the  side  bar  on  the  lep  side  of  your  screen.  The  list  will  appear  similar  to  below:    
  49. 49.    Note  any  Conflicts  of  Interest   Use  the  drop-­‐down  box  to  note  any  COIs.   If  there  is  a  COI,  use  the  second  drop-­‐down  box  to  indicate  type.  
  50. 50.    Accessing  and  Scoring  Once  you  have  confirmed  there  are  no  COIs,  three  new  icons  will  appear  to  the  right  of  an  assigned  applica4on:   Ø  The  first  symbol  (farthest  to  the  lep),  a  PDF  form,  is  the  complete  applica4on   Ø  The  second  symbol  (in  from  lep)  opens  the  applica4on  abstracts   Ø  The  third,  orange  symbol  is  an  applica4on’s  Phase  I  Reviews  (scores  and  wriben  cri4ques)   Ø  The  last  symbol  on  the  far  right  opens  your  Cri4que  Form  
  51. 51.    Preliminary  Scoring  in  PCORI  Online   The  PCORI  Online  scoring  screen  for  preliminary  impact  scoring:    
  52. 52.    Preliminary  Scoring:  Review  and  Submit   Final  screen  once  preliminary  scores  are  submibed:    
  53. 53. Q&A  
  54. 54. Wrap-­‐Up               This  concludes  today’s  session.  We  hope  you  found  this  training  helpful  and  informa#ve.     Thank  you  again  for  your  commitment  to  PCORI.       If  any  quesons  remain  unanswered  at  this  point,  please  email  them  to