PCORI Merit Review: Learning
from Patients, Scientists, and
other Stakeholders
October 30, 2013
Introductions: PCORI Moderators
 Lori Frank, PhD
Director, Research Integration and Evaluation

 Laura Forsythe, PhD, MP...
Introductions: Panelists
 Vernal Branch
Patient Advocate
Public Policy Advisor, Virginia Breast Cancer
Foundation

 Juli...
Agenda
Describe PCORI and PCORI’s unique
Merit Review Process
Learning from past reviewers: surveys,
group interviews, rev...
Background: PCORI and
PCORI Merit Review

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

5
About PCORI

An independent non-profit research organization
authorized by Congress as part of the 2010 Patient
Protection...
Why PCORI?
Patients have
questions that
research can answer
People want to know
which treatment is right
for them
Patients...
Our Mission
PCORI helps people make informed healthcare
decisions, and improves healthcare delivery and
outcomes, by produ...
Our Growing Research Portfolio
Total number of research
projects awarded to date:
197
Total funds committed to
date:
$273....
Patient Engagement as a Path to Rigorous
Research

Tell us what PCORI
should study

Help determine what
we fund

Engagemen...
Building an Inclusive Merit Review
PCORI created three categories of reviewers to bring various
perspectives to the review...
Why Be Inclusive in Merit Review?
Consistent with PCORI’s mission for research
guided by patients and other stakeholders
C...
Application Submission and Merit Review
Process Overview

PCORI Merit Review: Learning from
Patients, Scientists and other...
Merit Review Criteria

1. Impact of the condition
on the health of
individuals and
populations

2. Potential for the study...
Learning from Past
Reviewers

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

15
Learning from PCORI Reviewers
Reviewer surveys
 Closed-ended questions
 Open-ended questions

Group interviews with revi...
Cycle III (April – August 2013)

440
applications
received

PCORI Merit Review: Learning from
Patients, Scientists and oth...
Use of PCORI Criteria

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

18
How important was each of the criteria to your
final scores? … Impact of the condition
100%
90%
80%
70%
60%
50%
40%

Very ...
How important was each of the criteria to your final
scores? … Potential for improving care and
outcomes
100%
90%
80%
68%
...
How important was each of the criteria to your
final scores? … Patient-centeredness
98%

100%
90%

83%

80%
71%

70%
60%

...
How important was each of the criteria to your
final scores? … Rigorous research methods
100%
90%

88%

80%
70%
60%

57%

...
How important was each of the criteria to your
final scores?…Research team and environment
100%
90%
80%
70%
60%
50%

50%

...
Challenges for PCORI Merit Review:
Qualitative Findings
Need for more guidance about the meaning of
criteria and use of a ...
In-Person Panel

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

25
The chair(s) ensured that different points
of view were heard
100%
90%
79%

80%
70%

74%
67%

Strongly Agree
Somewhat Agre...
The scientific reviewers provided valuable
input during the discussion
100%
90%
80%

83%

80%

76%

70%
Strongly Agree
Som...
The patient and stakeholder reviewers
provided valuable input during the
discussion
100%
90%

83%

81%

80%
70%
60%

Stron...
Overall, scientific reviewers were receptive
to input from patient and stakeholder
reviewers
100%
90%
80%

73%
69%

70%

6...
Overall, patient and stakeholder reviewers
were receptive to input from scientific
reviewers
100%
90%

83%

80%

80%
70%
6...
PCORI In-Person Review: Qualitative
Findings
Mix of reviewers is unique and valued
 “It was very rewarding to be part of ...
Challenges for PCORI In-Person Review:
Qualitative Findings
Continued need to promote equality
 “I think there was some a...
My final scores were influenced by input
from other reviewers
100%
90%
80%
70%
60%
50%

Strongly Agree
Somewhat Agree
Neut...
Change in Overall Scores: Pre- to PostDiscussion

Total

Scientists

Patients

Stakeholders

p

Change by
>1 point

51%

4...
Overall Impressions of
PCORI Merit Review

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

...
Using a review panel comprised of researchers and non-researchers
helped PCORI ensure that selected research proposals wer...
Are you interested in participating as a
PCORI reviewer again in the future?
100%

98%

95%

94%

90%
80%
70%
60%
50%

Yes...
Conclusions
PCORI has a unique approach to reviewing
research applications
 Inclusion of patients and other stakeholders
...
Opportunities for
Improvement

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

39
Summary of Improvements – One
Streamline criteria
Streamline written critique format
Improve reviewer and chair training
I...
Summary of Improvements – Two
More clearly define reviewer roles
Improve communications with reviewers and
applicants
Enha...
Panel Discussion

PCORI Merit Review: Learning from
Patients, Scientists and other Stakeholders

42
Panel Discussion
What are the benefits of PCORI’s approach
to merit review?
In what ways should PCORI’s merit review be
im...
Become a Reviewer of Funding
Applications
PCORI invites professional
and lay audiences to be
reviewers of research
applica...
Stay Informed
Training Webinars for Applicants, Dec 2013:
http://www.pcori.org/funding-opportunities/fundingannouncements/...
Thank you!
Acknowledgements
 Scientist, patient, and stakeholder reviewers
 Reviewer panelists

Stay current with email ...
Upcoming SlideShare
Loading in …5
×

PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders

405 views

Published on

Slide presentation from the October 30, 2013 webinar which described the process and rationale for PCORI’s unique approach to reviewing research proposals for funding.

Published in: Health & Medicine, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
405
On SlideShare
0
From Embeds
0
Number of Embeds
67
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders

  1. 1. PCORI Merit Review: Learning from Patients, Scientists, and other Stakeholders October 30, 2013
  2. 2. Introductions: PCORI Moderators  Lori Frank, PhD Director, Research Integration and Evaluation  Laura Forsythe, PhD, MPH Program Officer, Research Integration and Evaluation  James Hulbert Pre-Award Manager, Contracts  Suzanne Schrandt, JD Deputy Director, Patient Engagement PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 2
  3. 3. Introductions: Panelists  Vernal Branch Patient Advocate Public Policy Advisor, Virginia Breast Cancer Foundation  Julie Panepinto, MD, MSPH Professor of Pediatrics Hematology, Director, Center for Clinical Effectiveness Research, Vice Chair of Value, Department of Pediatrics, Medical College of Wisconsin/Children’s Hospital of Wisconsin  Gregory Sawchyn, MD, MBA President, Caduceus Capital, LLC and Senior Director, Clinical Guidance Councils for OhioHealth PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 3
  4. 4. Agenda Describe PCORI and PCORI’s unique Merit Review Process Learning from past reviewers: surveys, group interviews, review scores Panel Discussion with past reviewers  Question and Answer session PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 4
  5. 5. Background: PCORI and PCORI Merit Review PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 5
  6. 6. About PCORI An independent non-profit research organization authorized by Congress as part of the 2010 Patient Protection and Affordable Care Act (ACA). Committed to continuously seeking input from patients and a broad range of stakeholders to guide its work. PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 6
  7. 7. Why PCORI? Patients have questions that research can answer People want to know which treatment is right for them Patients need information they can understand and use PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 7
  8. 8. Our Mission PCORI helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader healthcare community. Pictured: PCORI Board of Governors (March 2012) PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 8
  9. 9. Our Growing Research Portfolio Total number of research projects awarded to date: 197 Total funds committed to date: $273.5 million Number of states where we are funding research: 36 states (including the District of Columbia) PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 9
  10. 10. Patient Engagement as a Path to Rigorous Research Tell us what PCORI should study Help determine what we fund Engagement Tell us how we are doing PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Help us share research findings 10
  11. 11. Building an Inclusive Merit Review PCORI created three categories of reviewers to bring various perspectives to the review process. The reviewer committee will adhere to a 2:1:1 ratio meaning 2 scientists, 1 patient, and 1 stakeholder will be represented in each group. Patients PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholders Scientists 11
  12. 12. Why Be Inclusive in Merit Review? Consistent with PCORI’s mission for research guided by patients and other stakeholders Consistent with the goal of funding research that is relevant to patients and their caregivers More relevant research PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders More informed health decision making 12 Improved health outcomes
  13. 13. Application Submission and Merit Review Process Overview PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 13
  14. 14. Merit Review Criteria 1. Impact of the condition on the health of individuals and populations 2. Potential for the study to improve healthcare and outcomes 3. Technical Merit 4. Patient-centeredness 5. Patient and stakeholder engagement PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 14
  15. 15. Learning from Past Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 15
  16. 16. Learning from PCORI Reviewers Reviewer surveys  Closed-ended questions  Open-ended questions Group interviews with reviewers  Discuss survey findings  Hear more about concerns and suggestions for improvement Review of merit review scores pre- and postdiscussion PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 16
  17. 17. Cycle III (April – August 2013) 440 applications received PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 173 reviewers participated 17 52 projects funded for $96.2 million
  18. 18. Use of PCORI Criteria PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 18
  19. 19. How important was each of the criteria to your final scores? … Impact of the condition 100% 90% 80% 70% 60% 50% 40% Very Important Moderately important Somewhat important Not at all important 51% 50% 47% 43% 42% 36% 30% 20% 10% 14% 7% 7% 6% 0% 0% Scientific Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholder Reviewers 0% Patient Reviewers 19
  20. 20. How important was each of the criteria to your final scores? … Potential for improving care and outcomes 100% 90% 80% 68% 70% 60% 63% 57% Very Important Moderately important Somewhat important Not at all important 50% 40% 31% 30% 27% 20% 10% 0% 5% 8% 6% 5% 0% Scientific Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% Stakeholder Reviewers 0% Patient Reviewers 20
  21. 21. How important was each of the criteria to your final scores? … Patient-centeredness 98% 100% 90% 83% 80% 71% 70% 60% Very Important Moderately important Somewhat important Not at all important 50% 40% 30% 28% 20% 14% 10% 0% 1% 0% Scientific Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% 2% 0% Stakeholder Reviewers 0% 3% Patient Reviewers 21
  22. 22. How important was each of the criteria to your final scores? … Rigorous research methods 100% 90% 88% 80% 70% 60% 57% Very Important Moderately important Somewhat important Not at all important 51% 50% 40% 37% 32% 30% 20% 10% 0% 11% 10% 1% 0% Scientific Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 11% 0% Stakeholder Reviewers 0% Patient Reviewers 22
  23. 23. How important was each of the criteria to your final scores?…Research team and environment 100% 90% 80% 70% 60% 50% 50% 48% 46% 43% 43% 40% Very Important Moderately important Somewhat important Not at all important 36% 30% 20% 16% 11% 10% 0% 7% 0% Scientific Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% Stakeholder Reviewers 0% Patient Reviewers 23
  24. 24. Challenges for PCORI Merit Review: Qualitative Findings Need for more guidance about the meaning of criteria and use of a scoring scale  “More emphasis needs to be placed on how to choose a score during the training.” Time burden for review  Too many applications to review  Written critiques have redundant components PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 24
  25. 25. In-Person Panel PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 25
  26. 26. The chair(s) ensured that different points of view were heard 100% 90% 79% 80% 70% 74% 67% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 60% 50% 40% 30% 21% 21% 20% 11% 10% 6% 5% 6% 1% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% 0% 0% Stakeholder Reviewers 9% 0% Patient Reviewers 26
  27. 27. The scientific reviewers provided valuable input during the discussion 100% 90% 80% 83% 80% 76% 70% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 60% 50% 40% 30% 20% 18% 17% 11% 10% 0% 5% 0% 1% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% 0% 0% Stakeholder Reviewers 0% 3% 6% Patient Reviewers 27
  28. 28. The patient and stakeholder reviewers provided valuable input during the discussion 100% 90% 83% 81% 80% 70% 60% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 56% 50% 40% 30% 20% 10% 22% 19% 13% 7% 9% 1% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 0% 0% 0% Stakeholder Reviewers 0% 3% 6% Patient Reviewers 28
  29. 29. Overall, scientific reviewers were receptive to input from patient and stakeholder reviewers 100% 90% 80% 73% 69% 70% 66% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 60% 50% 40% 30% 21% 10% 0% 20% 17% 20% 7% 5% 5% 1% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 14% 2% Stakeholder Reviewers 0% 0% Patient Reviewers 29
  30. 30. Overall, patient and stakeholder reviewers were receptive to input from scientific reviewers 100% 90% 83% 80% 80% 70% 60% 58% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 50% 40% 30% 23% 20% 10% 14% 11% 7% 6% 6% 0% 2% 0% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholder Reviewers 0% 3% 6% Patient Reviewers 30
  31. 31. PCORI In-Person Review: Qualitative Findings Mix of reviewers is unique and valued  “It was very rewarding to be part of the process, and I truly believe it resulted in a more substantial and ultimately meritorious consideration of the applications.”  “As a scientist, I have really learned from the other reviewers, both scientists and patient/stakeholders.” Panels are well moderated by chairs for collegial, collaborative, respectful dialogue  “This was the most positive, collaborative review process that I've participated (in).” PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 31
  32. 32. Challenges for PCORI In-Person Review: Qualitative Findings Continued need to promote equality  “I think there was some amount of—even selfcensorship on the part of patients, because they just felt like they didn’t have a right to their opinion, the scientists did.” Differences between scientific reviewers and patient/stakeholder reviewers  Perceptions that scientists, patients, and other stakeholders score the same proposals differently  Perceptions that criteria are valued differently by scientists, patients, and other stakeholders PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 32
  33. 33. My final scores were influenced by input from other reviewers 100% 90% 80% 70% 60% 50% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 55% 49% 49% 40% 30% 29% 28% 23% 20% 10% 14% 9% 7% 7% 7% 6% 9% 9% 2% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholder Reviewers Patient Reviewers 33
  34. 34. Change in Overall Scores: Pre- to PostDiscussion Total Scientists Patients Stakeholders p Change by >1 point 51% 48% 50% 57% 0.311 Change by >2 points 22% 16% 26% 31% 0.001 PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 34
  35. 35. Overall Impressions of PCORI Merit Review PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 35
  36. 36. Using a review panel comprised of researchers and non-researchers helped PCORI ensure that selected research proposals were both methodologically rigorous and important to patients and other stakeholders 100% 90% 83% 80% 74% 70% 60% Strongly Agree Somewhat Agree Neutral Somewhat Disagree Strongly Disagree 53% 50% 40% 30% 20% 10% 24% 17% 12% 8% 11% 4% 0% 0% 0% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholder Reviewers 9% 3% 3% Patient Reviewers 36
  37. 37. Are you interested in participating as a PCORI reviewer again in the future? 100% 98% 95% 94% 90% 80% 70% 60% 50% Yes No 40% 30% 20% 10% 2% 6% 5% 0% Scientist Reviewers PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders Stakeholder Reviewers 37 Patient Reviewers
  38. 38. Conclusions PCORI has a unique approach to reviewing research applications  Inclusion of patients and other stakeholders intended to obtain and value a range of perspectives  Merit review criteria created to ensure research is methodologically rigorous and important to patients and other stakeholders Challenges with review criteria and with reviewer process are the focus of intense process improvement PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 38
  39. 39. Opportunities for Improvement PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 39
  40. 40. Summary of Improvements – One Streamline criteria Streamline written critique format Improve reviewer and chair training Initiate standing panels PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 40
  41. 41. Summary of Improvements – Two More clearly define reviewer roles Improve communications with reviewers and applicants Enhance mentor program PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 41
  42. 42. Panel Discussion PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 42
  43. 43. Panel Discussion What are the benefits of PCORI’s approach to merit review? In what ways should PCORI’s merit review be improved? Are different reviewer perspectives elicited fairly through the process? What are some of your most meaningful experiences as a PCORI reviewer? PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 43
  44. 44. Become a Reviewer of Funding Applications PCORI invites professional and lay audiences to be reviewers of research applications Help us support research that will be both scientifically rigorous and truly patientcentered Learn more and apply online: www.pcori.org/getinvolved/reviewers 44
  45. 45. Stay Informed Training Webinars for Applicants, Dec 2013: http://www.pcori.org/funding-opportunities/fundingannouncements/applicant-trainings/ Winter 2014 Applications Due: Jan 21, 2014 Spring 2014 LOIs Due: March 7, 2014 Opening a Pipeline to Patient-Centered Research Proposals webinar November 13, 2013, 1:00– 2:00 PM (ET) http://www.pcori.org/events/opening-a-pipeline-to-patientcentered-research-proposals/ PCORI Merit Review: earning from Patients, Scientists and other Stakeholders 45
  46. 46. Thank you! Acknowledgements  Scientist, patient, and stakeholder reviewers  Reviewer panelists Stay current with email alerts at http://www.pcori.org/home/signup and follow us on Twitter @PCORI Please send questions or comments to: Lori Frank, PhD Director of Research Integration and Evaluation lfrank@pcori.org PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 46

×