Board of Governors Meeting, South Sea Tea Washington

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
September 19, 2011
Patient-Centered Outcomes Research Institute
Communications, Outreach and
Engagement Committee (COEC)
Report

COEC Members
2
• Sharon Levine, MD (Chair)
• Debra Barksdale, PhD, RN
• Robert Jesse, MD, PhD
• Grayson Norquist, MD, MSPH
• Ellen Sigal, PhD
• Harlan Weisman, MD

COEC Charter
3
The Communication, Outreach and Engagement Committee shall advise and
assist the Board of Governors of PCORI, and provide recommendations to the
Board, regarding:
1. The Institute’s communications and branding work,
2. Strategies to engage all stakeholders in the work of PCORI, and
3. Methodologically sound approaches to disseminating and implementing the
research results and ensuring their utility to patients
and clinicians

COEC Report to the Board
4
I. Recent Opportunities for Public Input (Update)
II. Promoting PCORI RFPs (Update)
III. Redesigned PCORI Website (Update)
IV. Outreach Plan for Pilot Project Funding Announcement
(Recommendation for Decision)
V. Stakeholder Outreach (Update)
VI. Speakers Bureau (Update)
VII.PCORI Communications Staff (Update)

Recent Opportunities for Public Input
5
• Working definition of “patient-centered outcomes research”
– Nearly 600 responses were received (July 20-September 2,
2011)
– All responses received through the website or emailed PDFs will
be posted on pcori.org
– PCORI issued an RFP to solicit proposals for the analysis and
summarization of the input received and for recommendations
on methodologies to receive further input from patient
audiences

Analysis and Summary of Input on Working
Definition of PCOR
6
Issue RFP (and reach out to potential applicants) August 24, 2011
Deadline for applications September 9, 2011
Complete review of applications September 16, 2011
Complete contract with chosen contractor September 19, 2011
Submit Phase I findings and draft questions for focus
groups
September 30, 2011
PCORI completes revision of PCOR Definition October 26, 2011
Revise questions for focus groups November 9, 2011
Focus Groups (or other way of seeking patient input) November 18, 2011
Report due to PCORI December 16, 2011 (with
preliminary report in time for
December 8 meeting)
PCORI completes further revisions of PCOR Definition January 15, 2012
PCORI Board considers revised PCOR Definition January 18, 2012

Working Definition of PCOR:
A Foundation for PCORI’s Work
7
• At the time of PCORI’s creation, “patient-centered outcomes
research” (PCOR) was not a commonly used term, nor had it been
formally defined
• Defining PCOR helps clarify PCORI’s focus and scope of work
• Through iterative and transparent processes that elicit feedback
from all stakeholders, including patients, we will continue to revise
the definition, ensuring it:
– Emphasizes the “patient-centered” focus of PCORI’s mission
– Is consistent with the intent of the statute that established PCORI
– Is broad enough to support the range of research PCORI should fund

Recent Opportunities for Public Input
8
• Initial topics for PCORI pilot
projects.
– More than 150 responses
were received
(August 1-30, 2011)
– Responses provided to the
PDC Working Group to
review and update topics
as appropriate.
• 89.26% Self Identified
• 10.74% Anonymous
• 59.76% Responded as Individual
• 40.27% On behalf of an Organization
Respondents
• 40.94% Academia
• 23.49% Provider
• 17.45% Patient
• 3.36% Industry
• 0.67% Government
• 11.41% Non-Identified
Organizational Respondents

Promoting PCORI RFPs
9
• To date PCORI has issued 4 Requests for Proposals (RFP)
– Analysis of Input Received on Working Definition of “Patient-Centered Outcomes Research”
– Review and Synthesis of Evidence for Eliciting the Patient's Perspective in Patient-Centered
Outcomes Research (Literature Review)
– Expert Stakeholder Interviews to Identify Evidence for Eliciting the Patient's Perspective in
Patient-Centered Outcomes Research (Interviews)
– Methods for Setting Priorities in Research (White Papers)
• For each RFP, PCORI performed outreach to: email list subscribers (620+),
supplemental lists of clinical and translational research centers (60+), academic
research institutions (30+), and provider and advocacy organizations (30+)
• PCORI is an independent, non-profit, non-governmental organization. All PCORI
funding announcements will be posted on pcori.org. PCORI funding announcements
will not be published in the Federal Register

Redesigned PCORI Website
10

Redesigned PCORI Website
11
• Includes visual changes based on logo and color palette approved
at July Board Meeting
• Designed to be more user friendly
• Will support the expansion of PCORI content and opportunities for
interaction as work progresses
• Expanded features include:
– Easier mechanisms for joining PCORI mailing list and providing
general input
– An “Executive Director’s Corner” that provides regular updates
on PCORI’s latest developments

Outreach Plan for Pilot Project Funding
Announcement
12
• Post announcement on pcori.org
• Direct email to stakeholders and potential applicants
• Conduct proactive media relations
• Promote the opportunity though grassroots outreach
• Integrate the announcement with speakers bureau presentations
• Host a Q&A teleconference for applicants

Stakeholder Outreach
13
PCORI’s engagement with stakeholders around Board meetings
• Stakeholder discussion forum
March
St. Louis, MO
• Stakeholder discussion forum
May
New York, NY
• Two small group meetings with patients and caregivers
• Eight small group meetings with 43 stakeholder organizations
July
Washington, DC
• Invited presentations from Northwest Pacific stakeholders
September
Seattle, WA
• Site visits to clinical care facilities
November
New Orleans, LA

Seattle Stakeholder Engagement Forum
14
• PCORI has invited individuals and organizations from the Pacific
Northwest to make brief presentations to the Board about their
work or perspective and how it can inform PCORI’s work
• Native American/Alaska Native perspectives
• Complementary and alternative medicine researchers and
providers
• Outcomes and comparative effectiveness researchers
• Monday, September 19, 7:00-9:00 p.m. PT
• Open to the public

Speakers Bureau
15
PCORI has presented at 24 meetings since March
Recent Speaking Engagements
• 7/25 – IOM CER Innovation Collaborative Meeting
• 7/25 – Center for Medical Technology Policy CER Institute
• 8/12 – Consumers United for Evidence Based Health Care Annual Meeting
• 8/25 – Florida Chiropractic Association National Convention & Expo
• 9/9 – University of Michigan School of Nursing
• 9/16 – Oregon Institute for Patient-Centered Comparative Effectiveness Annual Research Intensive
Upcoming Opportunities
• 9/27 -- AdvaMed 2011 MedTech Conference
• 9/28 – Partnership to Improve Patient Care Annual Membership Forum
• 10/11 – NIH National Center for Research Resources Clinical and Translational Science Awards Program
• 10/12 – Comparative Effectiveness Research Summit
• 10/12 – American Academy of Nursing, Council of Advancement of Nursing Science 2011 Special Topics
Conference
• 10/19 – Health Industry Forum
• 10/20 – Pfizer Health Advocates Leadership Breakfast

PCORI Communications Staff
16
• Seeking applicants for three positions:
– Director of Communications
– Director of Patient Engagement
– Director of Stakeholder Engagement
• Staff will develop and implement strategic communications plan
• Distinct engagement officers for patients and other stakeholders,
recognizing their need for different engagement mechanisms

September 19-20, 2011
Seattle, WA
Executive Director Update
Board of Governors
PATIENT-CENTERED OUTCOMES
RESEARCH INSTITUTE

Objectives for today
 Highlight PCORI Activities since July 2011 BoG Meeting
 Discuss Growing the PCORI Staff
 Consider Critical Timelines/Milestones for Next 9 Months

PCORI Activities – (Staff, Board, MC together)
 Establishing Policies and Practice
 HR Policies
 Purchasing/RFP policies
 COI policies
 Research Policies
 Re-designing Website
 Outreach
 Developing IT Infrastructure
 Growing Staff
 Finding Long-term Space

4
• $48 per sq. foot
• Attractive, not
• extravagant
• 13,000 sq feet
• Close to Metro
• Green building
• Move-in: February
PCORI’s Long-Term Home
1828 L Street, DC

Urgent Issues Coming Out of July BoG Meeting:
 Address the need for greater clarity in decision-making and take
steps to find more time for substantive discussions among board,
with MC
 Patient Engagement! - Officer, Advisory Panel
 Get started on the National Priorities and Research Agenda!
 Support the PDC in preparing PCORI Pilot Projects RFP and review
 Support the MC in getting the RFPs out for the Methodology Report

Patient Engagement
6
 Patient-Engagement Working Group formed;
 Job Description for Director of Patient Engagement drafted
 Directors of Stakeholder Engagement and Director of Communications Added
 All three positions are posted and a search is underway
Patient
Engagement
Communi-
cations
Stakeholder
Engagement
External
Engagement

Program Development Committee
National Priorities and Research Agenda
• C Clancy and A Epstein, co-chairs
• Environmental scan on past CER analysis and developed an initial
framework for the National Priorities which will serve as the
foundation for developing specific priorities
• Recognized the close link between the National Priorities and
Research Agenda (H Krumholz and L Hole-Curry, co-chairs)
• Developed a timeline and notional process for engaging
stakeholders as part of the National Priorities development timeline

Developing PCORI’s National Priorities
and Research Agenda
8
Collect
Stakeholder
Input
Develop Candidate Priorities
Public
Comment
Finalize
Priorities
Develop
Candidate
Priorities
Underway Sept - Nov Dec-Feb March
Collect
Stakeholder
Input
Public
Comment
Finalize
Research
Agenda
April
Feb - Mar
Sept - Jan
PRIORITIES:
RESEARCH AGENDA:

PCORI Pilot Projects
 C Goertz and G Hunt co-chairing this process.
 Public Input has been reviewed and incorporated into revised “areas
of interest” in the PCORI Funding Announcement (PFA)
 An application form and instructions have been developed and
revised
 Working closely with NIH staff on defining each step in the
application and review process

Supporting Methodology Committee
 Interim researcher process established; at least one
already working
 RFPs (2) for Patient-centeredness Working Group posted
 RFP (1) for Research Prioritization Working Group posted
 Review Process in Development, including COI policies for
review

Definition of PCOR
 Input on the working definition of “patient-centered outcomes research”
gathered (n=600); to be posted shortly after anonymization
 RFP posted for qualitative analysis and summarization of the input
received and for recommendations on methodologies to receive further
input from patient audiences posted on PCORI website
 12 responses received and reviewed
 First PCORI research contract issued on 9/16 to National Opinion
Research Center
II
MC
I
9/16 9/30 11/15 12/31
Synthesis
& Question
Generation
Revision of
Definition
Testing of
Revised
Definition
Preparation of
MS for Publication
Led by MC

Three Engagement
Directors
Applications
Arrive
LOI’s
Arrive
Growing PCORI Staff:
09/18/11 –
09/20/11
We are here
11/1/11 2/1/12 05/1/12
04/1/12
1/1/12
COO
Begins
Mark
In!
Director
Finance
Begins
First
Scientists
Additional
Scientists
Additional
Admin
Ass’ts
Director of
Staff
12/1/11 3/1/12
Funding
Decision
Made
Funding
Issued
Program
Staff Hiring
Begins
Finance/Grant
Staff

Growing PCORI Staff
13
Research Scientist
Patient engagement
Research
Research Scientist
Stakeholder Engagement
Research Scientist
Communicaiton/Disseminat
ion Research
Research Scientist(s)
Comparative Clinical
Effectiveness Research
Director
Patient Engagement
Director*
Stakeholder Eng.
Director
Communications
Chief Financial Officer
(CFO)
Director
HR
Director
Director
Grants Management
Research Scientist(s)
Evidence Synthesis
External
Engagement
Executive Director
Chief Operating Officer
Chief Science Officer

Critical Issues: September 2011 – June 2012
14
 Gathering Stakeholder Input on National Priorities and Research Agenda
 Thorough, balanced, incorporated
September – November:
November- December:
 Finalizing First Draft of Priorities (Prior to Public Comment)
 How are Priorities expressed, what’s in, is anything out?
January – February:
 Finalizing First Draft of Research Agenda (Prior to Public Comment)
 What (specifically) is in, out – what will the first RFAs fund?
 Data Infrastructure Grants?
 Translation/Implementation Research?
 Systems research?
 Pharmacogenetics?
 Large clinical trials?

Finance & Administration Committee
September 20, 2011, Seattle, WA
RESEARCH INSTITUTE

Bylaws Revisions
• GAO currently reviewing
• Committee Name Change: Finance, Administration & Audit
• Adoption Proposed at November BOG Meeting

Audit Update
• Released this week
• Bidders selected by annual revenues
• Proposals by end of October
• Recommendation to BOG
• Field work in November/December
• Completed audit by March
• Provided to GAO for report to Congress due April 1st

Financial Statements – August 31, 2011
• Statement of Financial Position: Total Assets of $46.3M; $5M Cash
on Hand
• Expenditures:
– Budget: $7.5M
– Actual: $3.7M
– Positive variances in Board compensation, management fees,
professional services, travel
– Negative variance in conferences

Conflicts of Interest Policy
• Working group created
Members: Larry Becker, Chair
Debra Barksdale
Arnie Epstein
Sherine Gabriel
Bob Jesse
• Group is gathering key documents

1
September 19, 2011
PCORI Methodology Committee
Progress Report
Sherine E. Gabriel, MD, MSc
Sharon-Lise T. Normand, PhD
PCORI Methodology Committee

2
• Establishment
• Overview
• Timeline
• Structure
• Mission
• Scope of Work
• Progress to Date
• Next Steps
• Questions
1. Methodology Committee
2. Methodology
Committee Workgroups
3. Wrap Up
• Next Steps
Agenda

3
Methodology Committee Charge
Establishes priorities to
address gaps in
research methods and
their application
The Methodology
Committee shall make
recommendations
regarding methods for
patient-centered
outcomes research
Provides guidance
about the appropriate
use of methods

4
Methodology Committee Structure
Methodology Committee
Co Chairs: Sherine Gabriel, MD, Mayo Clinic & Sharon-Lise Normand, PHD, Harvard Medical School
Patient-Centeredness
Research Prioritization
Methods to incorporate the patient
perspective into all phases of PCOR
Methods to inform prioritization of new
research studies
Methods for using data, design, and
statistical analyses to conduct PCOR
Research Methods
Report
Assimilation

5
Methodology Committee Activity Timeline
Research
Methods
Aug „11 Oct „11 Nov „11 Dec „11
Patient-
Centeredness
Jan „12 Feb „12 Mar „12 Apr „12 May „12
Workshop
Final
Report
Workshop
Identify
Gaps/
Issues
Translation
Table/Tool
Workshop Final
Papers
Due
Final
Draft
Final
Report
Final
Report
Compile
First
Draft
Legend
Research
Prioritization
Report
Assimilation
Feb – Jul „11
Revise Report Outline & Edit Chapters
Final
Report
Final
Papers
Due
Electronic Data Systems Landscape Review
Identify & Implement Voting
Standards Process
Solicitations: White Papers/Identify Experts
Solicitations: Literature Review & Interviews
MC
Charter
MC
Work
Plan
Interim
Researchers
Staffed
PCOR
Definition
Solicitation
Commission White Papers
Sep „11

6
Methodology Committee Discussion Points
• Importance of methods generated by Patient Centeredness Work
Group for incorporating patient engagement
• Importance of methods generated by Research Prioritization
Work Group vis-à-vis prioritization framework set by the board
• Generation of tools/standards for Methods Report is a multiyear
process
• Importance of continuing to interact with Board to ensure
coordination of efforts
• Board/Methodology Committee teleconference calls were a
productive forum to receive input from the Board regarding their
needs and priorities

7
• Establishment
• Overview
• Structure
• Timeline
• Mission
• Scope of Work
• Next Steps
• Questions
2. Methodology
3. Wrap Up
• Next Steps

8
Patient-Centeredness Workgroup
Ethan Basch, MD
Memorial Sloan-Kettering
Cancer Center
Workgroup Chair
Mary Tinetti, MD
Yale University School of
Medicine
Workgroup Co-Chair
Naomi Aronson, PhD
Blue Cross and Blue Shield
Workgroup Member
Brian Mittmann, MD
VA Center for Implementation
Practice & Research Support
Workgroup Member
David Flum, MD, MPH
University of Washington
Report Assimilation Group Liaison

9
-
Patient Centeredness Mission/Scope of Work
Development and
prioritization of research
questions
Identify methodological standards for incorporating
the patient perspective into three key areas:
Design of study
components, including
selection of interventions,
comparators, and outcomes
(including patient-reported
outcomes)
Processes of clinical
decision-making/care
delivery

10
Patient Centeredness Progress to Date
Released RFP
for Literature
Review
Released RFP
for Interviews
“Review and Synthesis of
Evidence for Eliciting the
Patient‟s Perspective in
Patient-Centered
Outcomes Research”
“Expert Stakeholder
Interviews to Identify
Evidence for Eliciting the
Patient‟s Perspective in
Patient-Centered Outcomes
Research”

11
Proposals Due
Awardee Announced
Final Report Due
Oct 6, 2011
Oct 13, 2011
March 1, 2012
Issue Open RFI Plan Workshop
Develop Sections
for Methodology
Report
Patient Centeredness Next Steps

12
Patient Centeredness Question
How does this work
complement ongoing and
planned activities of the
Board?

13
Research Prioritization Workgroup
David Meltzer, MD, PHD
University of Chicago
Department of Medicine
Workgroup Chair
John Ioannidis, MD, DSc
Stanford Prevention
Research Center
Workgroup Member
Jean Slutsky, PA, MSPH
Agency for Healthcare
Research and Quality
Workgroup Member
Clyde Yancy, MD
Northwestern University
Feinberg School of Medicine
Workgroup Member
Alfred Berg, MD, MPH
Department of Family Medicine

14
Research Prioritization Mission
Mission
To provide guidance concerning the use of methods to inform the
establishment of research prioritization approaches that best
fulfill PCORI’s mission.
• To the PCORI Board to aid in development and future
refinement of Research Priorities]
• To the broader multi-stakeholder communities to enhance
understanding of PCORI’s approach to research prioritization
and to encourage their engagement in PCORI’s research
prioritization process

15
-
Commission White
Papers
To prepare a section for the May 2012 Methodology Committee Report
that addresses methods to inform the establishment of research
prioritization
Synthesize White Papers
during January 2012
Workshop
Research Prioritization Scope of Work
Topic
Generation
Gap Analysis in
Systematic
Reviews
Value of
Information
Analysis
Peer Review
Suggest strategies to
identify possible future
priority areas
Suggest standards for how
PCORI systematic reviews
should be performed and
used to generate research
topics
Provide PCORI Board and/or
grant applicants with tools
to quantify expected
benefits of research to
inform priorities
Provide data to inform how
PCORI might design,
evaluate and continually
improve its peer review
process

16
Research Prioritization Progress to Date
Identified
Methodological
Areas of Interest
in Research
Prioritization
Identified
Interim
Researcher to
Assist
Workgroup
Released RFA
for White
Papers on
September 9,
2011

17
White Paper Proposals Due
Review and select among White Paper
proposals
Awardee Announced
Sept 30, 2011
Oct 1- 13, 2011
Oct. 14 2011
Explore the role of research prioritization methods vis-à-vis the broader
PCORI research prioritization process
Engage researchers as needed to fill in expected gaps from White Papers
Research Prioritization Next Steps

18
Research Prioritization Question
How can we best work with you to
understand your evolving plans for
research prioritization so the
methods we are studying can be of
most use to you?

19
Research Methods Workgroup
Steven Goodman, MD, MHS, PHD
The John Hopkins University
School of Medicine and Public Health
Workgroup Chair
Mike Lauer, MD
National Institute of Health
National Heart, Lung, and
Blood Institute
Workgroup Member
Robin Newhouse, PHD RN
University of Maryland
School of Nursing
Workgroup Member
Sebastian Schneeweiss,
MD, ScD
Harvard Medical School
Workgroup Member
Mark Helfand, MD MPH MS
Oregon Health & Science University
School of Medicine
Sharon-Lise Normand, PHD
Methodology Committee Chair

20
-
Develop Translation
Table/Tool (from research
question to design and
analysis)
Identify Standards for the use of data, design, and statistical analyses to
conduct patient centered outcomes research through:
Review current
methodological “state of
the art” statements issued
by expert bodies to guide
PCORI research
Research Methods Mission/Scope of Work
Review existing data
systems designed to
permit exploration of
causal questions from data
gathered in the course of
clinical practice. Identify
PCORI role in this arena.

21
Research Methods Progress to Date
Established initial
specifications for
dimensions to be used
in translation
instrument, with
options for
implementation
Developing
prototype for
standards
documents that
combines expert
statements,
published examples
and PCORI MC input
Interviewing leading
researchers of
electronic data, from
which research and
methods
recommendations
will be developed

22
Research Methods Progress to Date (Detailed)
Framework for
Methods Standards /
Recommendations
1. Method
2. Key sources
3. Major recommendations
4. PCORI MC commentary
5. Published examples
6. Tools for researchers
Translation Table
Dimensions
Intrinsic Factors
• Internal validity (aka bias)
• External validity (aka
generalizability, transportability)
• Precision
• Heterogeneity in risk or benefit
(aka “personalized” evidence)
• Ethical dimensions
Extrinsic Factors
• Timeliness (Rapidly changing
technology, policy urgency)
• Logistical burden (e.g. study
size, complexity, cost)
• Constraints (Data availability,
randomization possible?)
Question: condition, population, treatment & comparator, outcomes, setting

23
Finalize translation table dimensions and
determine categories of development
Finalize “standards” document format and
determine categories of development
Finish and summarize electronic data
systems environmental scan
Research Methods Next Steps
(e.g. diagnostic tests, missing
data, systematic reviews, etc.)
(e.g. drug safety, therapeutic
efficacy, etc.)
Jan. 2012 Workshop

24
Research Methods Question
Can the Board provide feedback on
the preliminary formats developed
(i.e., translation table dimensions
and methods standards format)?

25
Report Assimilation Workgroup
Mark Helfand, MD
School of Medicine
Workgroup Chair
Alfred Berg, MD, MPH
Workgroup Member
David Flum, MD, MPH
Workgroup Member
Howard Balshem, MS
Interim Researcher
Sherine Gabriel, MD
Mayo Clinic
Workgroup Member
Sharon-Lise Normand, PHD
Workgroup Member

26
• Establishment
• Overview
• Structure
• Timeline
• Mission
• Scope of Work
• Next Steps
• Questions
2. Methodology
3. Wrap Up
• Next Steps

27
Solicitation Responses and Questions Received to Date
Methodology Committee Next Steps
Letters of Intent
Received**
Questions
Patient Centeredness WG --
Stakeholder Interviews
25 8*
Patient Centeredness WG -
Literature Review
26 4*
Research Prioritization WG -
White Papers
11 0
**as of 9/16 at 8:30 a.m.
*This number of questions
includes a document
containing 13 individual
sub-questions.

28
Administrative
Review
Content
Review
Programmatic
Review
Executive
Decision
1. Clear, transparent and reproducible review process based on best
practices (eg. NIH)
2. Aligned with PCORI Vision, Mission, Principles and Policies (eg CoI)
3. Timely (~Applicants notified of award two weeks after submission)
Will be Established prior to receipt of first response (September 30, 2011)
Solicitation Review Process and Principles

29
Research
Methods
Aug „11 Sep „11 Oct „11 Nov „11 Dec „11
Patient-
Centeredness
Jan „12 Feb „12 Mar „12 Apr „12 May „12
Workshop
Final
Report
Workshop
Identify
Gaps/
Issues
Translation
Table/Tool
Workshop Final
Papers
Due
Final
Draft
Final
Report
Final
Report
Compile
First
Draft
Legend
Research
Prioritization
Report
Assimilation
Feb – Jul „11
Revise Report Outline & Edit Chapters
Final
Report
Final
Papers
Due
Electronic Data Systems Landscape Review
Identify & Implement Voting
Standards Process
Solicitations: White Papers/Identify Experts
Solicitations: Literature Review & Interviews
MC
Charter
MC
Work
Plan
Interim
Researchers
Staffed
PCOR
Definition
Solicitation
Commission White Papers

30
Thank You!

Presentation to PCORI Board of Governors
September 20, 2011
RESEARCH INSTITUTE:

Committee Members
• Carolyn Clancy
• Francis Collins
• Arnold Epstein
• Christine Goertz
• Leah Hole-Curry
• Gail Hunt
• Harlan Krumholz
• Richard Kuntz
Methodology Committee
Representatives
• Sherine Gabriel
• Sharon-Lise Normand

Report Outline
 PCORI Pilot Project Grant Program
 National Priorities and Research Agenda
 Landscape Review Program

September 20, 2011
Christine Goertz, DC, PhD
PCORI Pilot Projects

Report Outline:
• Purpose of PCORI Pilot Projects
• PCORI Funding Announcement - Key Highlights
• Areas of Interest
• Overview of the Four-Stage Review Process
• Application Review Criteria
• Grant Application & Instructions
• Webinar & Process to Handle Inquiries
• Timeline, Key Dates, and Activities

Purpose of PCORI Pilot Projects
The purpose of the PCORI Pilot Projects Grant Program is threefold:
1. National Priorities. Pilot projects will provide information to
PCORI that informs future iterations of national research priorities
for patient centered outcomes research.
2. Research Agenda. The program will support the collection of
preliminary data that can be used to advance the field of patient-
centered outcomes research, providing the platform for an
evolving PCORI research agenda.
3. Methodologies. The program will support identification of
research methodologies that advance patient-centered outcomes
research

PCORI Funding Announcement:
Key Highlights
 Applicants must address at least one of the eight areas of interest.
 Stakeholder involvement is required unless the application can sufficiently
explain why it is not feasible.
 Expect to commit approximately $13 million in FY 2012 to support
approximately 40 projects.
 Applicants can propose project periods of up to two years. Second year
funding is subject to noncompetitive review by PCORI staff.
 Direct costs are limited to $250,000 per year with an additional 40% of
salary and fringe allowable as indirect costs.
 Letters of intent are due on November 1 with applications due on
December 1.
 NIH will conduct the merit review process with all other aspects of grant
awards and grants management handled by PCORI.

Areas of Interest: Development
Areas of Interest
Board
Members
Methodology
Committee
Stakeholder /
Public Input
Projects focused on developing,
testing, and/or evaluating
methods or approaches that:
 Inform the evolving
National Priorities
 Bring together different
stakeholders
 Translate research into
practice
 Identify gaps in relation to
issues for disadvantaged
populations
 Identify predictors of
patient outcomes

Areas of Interest
Developing, testing, refining, and/or evaluating new or existing methods (qualitative and quantitative) and
approaches that can inform the process of establishing and updating national priorities for the conduct of
patient-centered outcomes research (PCOR). This may include research prioritization approaches (such as
Value of Information (VOI), burden of illness, peer review/expert opinion/Delphi approaches) or methods for
incorporating the perspectives of patients or other stakeholders into the development of national priorities.
Developing, testing, and/or refining existing methods for bringing together patients, caregivers, clinicians
including non-traditional partners, and other stakeholders in all stages of a multi-stakeholder research
process, from the generation and prioritization of research questions to the conduct and analysis of a study to
dissemination of study results – including methods for training participants in participatory research and the
potential use of new technologies to facilitate engagement.
Developing, refining, testing, and/or evaluating patient-centered approaches, including decision-support
tools, for translating evidence-based care into health care practice in ways that account for individual patient
preferences for various outcomes. This may include developing or comparing conceptual models of
translation or dissemination of CER research findings from the patient perspective.
Developing, refining, testing, and/or evaluating methods to identify gaps in CE knowledge such as tools for
the ongoing collection and assessment of gaps as perceived by patients and providers. Of special interest are
gaps that are particularly relevant to vulnerable populations, including but not limited to, low-income
populations; minorities; children; elderly; women; people with disabilities, chronic, rare and/or multiple
medical conditions.

Areas of Interest
Identifying, testing, and/or evaluating patient-centered outcomes instruments. This may include predictive
tools (eg: instruments that measure or predict outcomes of interest to patients) or identifying standards for
measurement properties of patient-reported outcomes for use in comparative effectiveness research, across a
variety of interventions and patient populations.
Identifying, testing, and evaluating methods that can be used to assess the patient perspective when
researching behaviors, lifestyles, and choices within the patient’s control that may influence their outcomes.
Identifying, testing, refining and/or evaluating methods for studying the patient care team interaction in
situations where multiple options for wellness, prevention, diagnosis or treatment exist. Of special interest
are strategies that respect patient autonomy and promote informed decision-making, incorporating the best
health care knowledge into the application of care.
Advancing analytical methods for CER. Examples include but are not limited to the incorporation of mixed
methods research designs (qualitative/quantitative), identifying existing methodology to statistically
accommodate irregularly spaced multivariate longitudinal data, the use of instrumental variables; and
potential solutions for assessing treatment heterogeneity in observational and randomized CER studies.

Overview of the 4-Step Process
Preliminary
Check
Merit Review
Deliberation
Business
Determination that the application meets the minimum technical
requirements enumerated in the PFA and fits within programmatic
priorities.
PCORI Responsive/eligible
applications will be
forwarded for
review.
Remaining applications are assigned to 3 reviewers with
appropriate expertise who will critique independently and then
meet to discuss and designate a final score.
NIH A rank-ordered list
of applications and
summary sheets
The Pilot Projects Review Committee considers the merit review
scores and programmatic balance across several categories to
develop a recommended award slate for consideration by the BOG.
PCORI Funding decisions
are made.
A final review for suitability to manage an award, adherence to
human subjects requirements, receipt of IRB approval, budget
review, etc.
PCORI Negotiations,
where needed
Description Responsible Result

Preliminary Check
Technical
Requirements
• Receipt of Letter of
Intent
• Eligible applicant
• Stakeholder
requirements
• Page limit and
formatting
requirements
• Deadline met and
application properly
submitted
Programmatic
Requirements
• Does the application
fit within PCORI
program requirements
listed in the PFA?
Transfer to NIH
• Applications meeting
both technical and
programmatic
requirements will be
forwarded to NIH for
the Merit Review.
Conducted by PCORI staff

Merit Review
Prior to the Review
Meeting
• Each application is
assigned to 3 reviewers
• Reviewers include
scientists and other
stakeholders
• Stakeholders without
previous review
experience will be trained
• Reviewers assign an initial
priority score of 1 to 9
based on PCORI-provided
review criteria
• Provides basis for
discussion at in person
meeting
In-Person Review
• In-person meeting of
reviewers to discuss the
most promising
applications
• Review and discuss
applications based on
impact, stakeholder
involvement, innovation,
and significance of the
science
• Provide a final priority
score of 1 to 9
• Scientific Review Officer
compiles a summary
statement with reviewer
critiques for each
application
Summary & Rankings
• The review process
results in a ranked, scored
summary of applications
for PCORI consideration
Conducted by NIH

PCORI Deliberations
PCORI Staff
• Analyzes the applications
using priority score and
classification categories.
• Creates funding scenario
options based on
analytics.
• Supports the PCORI
Review Committee and
Board of Governors by
providing revised options,
as needed.
PCORI Review
Committee
• Meets to review materials
• Considers the balance of
priority-scored
applications across
classification categories,
requesting additional
analysis and options from
staff, as needed
• Prepares a recommended
slate of selected projects
for funding consideration
Board of Governors
• Meets to consider the
recommended slate
• Reviews the slate based
on priorities and balance
to ensure appropriate
distribution
• Requests additional
options, if needed
• Approves a final slate of
selected projects for
funding
Classification categories may include: Applicability of the 8 areas of interest, geographic
distribution, research methodology, and identification of target populations.

Recommended Funding
Slate Analysis
Following the Review
Committee meeting, analysts
will prepare the recommended
slate along with accompanying
analysis for consideration by
the Board of Governors.

Merit Review Criteria
 Significance. Does the project address an important problem or a critical barrier to patient-
centered outcomes research? Does the project address one of the key questions outlined in
the PCOR definition described above? Is the project focused on one of the areas of interest
identified in this PFA? Will the results produce new knowledge that can advance PCOR
methods or infrastructure? Does the investigator demonstrate thorough knowledge of
previous and ongoing work related to their proposed topic?
 Patient/Stakeholder Engagement. Will the research make a unique contribution to learning
about engagement of patients and non-traditional stakeholders in PCOR research efforts?
Does the research team demonstrate authentic, feasible, sustainable, novel partnerships with
patients, families and care givers, providers, and other appropriate non-traditional
stakeholders? Is there evidence that non-traditional stakeholders were involved in the
preparation of the research proposal?
 Investigator(s). Is the research team well suited to the project? Is it multi-disciplinary? Is
there appropriate scientific expertise? Does at least one member of the study team have
experience in patient and other stakeholder engagement? Is there a high level of confidence
that the Principle Investigator and rest of the study team will be able to achieve the study
aims as described? Does the study team have complementary and integrated expertise; are
their leadership approach, governance and organizational structure appropriate for the
project?

Merit Review Criteria
(continued)
 Innovation. Does the project either address a new method or approach or apply a proven
method or approach in a novel way to the field of PCOR? Is a refinement, improvement, or
new application of theoretical concepts, approaches or methodologies, instrumentation, or
interventions proposed?
 Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate
to accomplish the specific aims of the project? Are potential problems, alternative strategies,
and benchmarks for success presented? Will the strategy establish feasibility and will
particularly risky aspects be managed? Is the proposed budget and timeframe appropriate for
the research plan? Are there appropriate plans for dissemination among key PCOR
stakeholders in education, practice and policy?
 Environment. Will the environment in which the work will be done contribute to the
probability of success? Are the institutional support, equipment and other physical resources
available to the investigators adequate for the project proposed? Will the project benefit from
unique features of the research environment, community involvement, patient populations,
or non-traditional stakeholder collaborative arrangements?

Grant Application & Instructions
• Application is based on the NIH PHS 398 form
with changes to ease responses and to meet the
needs of PCORI.
• Applicants will be bound by the Human Subjects
policies of NIH and will include appropriate
plans within the Research Strategy section.
• Applicants will provide information related to
classification categories to enable the Review
Committee and Board to consider this
information when developing a balanced slate
for funding.
Self-Reported
Classification Categories:
• Applicability of the “4
questions” to the research
• 8 areas of interest
• Identification of specific
populations including:
― Underserved or
disadvantaged
populations
― Specific ethic or cultural
populations
― Rural or urban
populations
― Disabled populations

Webinar & Inquiry Process
• Webinar for Applicants: October 12, 2011 at 2PM EST.
• Policy for responding to inquiries:
– A set of FAQs will be developed and issued along with the PFA with
frequent updates posted on the PCORI website. Questions and
answers will be framed in terms of general applicability.
– Inquiries should be directed to the PCORI website at pcori.org. PCORI
staff and appropriate consultants will receive, triage, route, and ensure
answers are provided and shared generally through the website.

Timeline, Key Dates, and Activities
2011
09 /11
PFA Posted
11/01/11
Letters of
Intent Due
12/1/11
Applications
Due
11-12/11
Identification of
Merit Reviewers
2012
12/11
Completeness/
Compliance/
Eligibility Check
2/12
Merit Review
3/12
Slate of Projects
to Board &
Selection Made
05/12/12
Award
Notification &
Funding
PCORI Applicant NIH/PCORI Applicant
PCORI NIH PCOR PCORI / Applicant

Arnold Epstein, MD
September 20, 2011
PCORI National Priorities
Development

National Priorities are the critical beginning
and will be dynamic
Increasing
Specificity
National
Priorities
Research
Agenda
Individual
PFAs

Develop
Candidate
Framework
Engage
Stakeholders
Public
Comment
Finalize National
Priorities
Currently
Underway
September-
November
December-
February
March
Proposed Timeline for National Priorities

Environmental
scan of existing
priorities and
criteria
Candidate
priorities and
criteria
identified
Framework to
inter-relate
Priorities and
Criteria
Reviewed initial
stakeholder
input advising us
to not “reinvent
the wheel.”
Reviewed prior
CER frameworks
(e.g., IOM,
FCCCER,
National
Priorities
Partnership,
NQF)
Identified broad
priorities and
criteria that
were used often
in prior
frameworks and
fit PCOR.
Framework to be
used for refining
priorities, and
determining
Research
Agenda and
PFAs.
Initial
Stakeholder
feedback
Development of a National Priorities Framework

Previous priorities have varied in granularity
Granularity
IOM 2009
(e. g., Compare the effectiveness of upper endoscopy
utilization and frequency for patients with
gastroesophageal reflux disease on morbidity, quality of
life, and diagnosis of esophageal adenocarcinoma.)
National Priorities
Partnership
(e. g., Engage patients and
families in managing their health
and making decisions about their
care.)
National Prevention
Council
(e. g., Elimination of
health disparities)
FCCCER
(e.g., Expanding high-impact patient
registries)
National Quality
Forum
(e. g., Infrastructure –
Information Technology)
Least Most
National Quality
Strategy
(e. g., Promoting
effective communication
and coordination of care)

Ten candidate priorities
consistent with prior efforts
Source
Prevention
Acute
Care
Chronic
Disease
Care
Palliative
Care
Care
Coordination
Patient
Engagement
Safety
Appropriate
Use
HIT
to
Improve
Patient
Experience
Impact
of
New
Technology
IOM 2009: Priorities for
CER
√ √ √ √ √ √ √ √ √
Federal Coordinating
Committee for CER
√ √ √ √ √
AHRQ National Quality
Strategy
√ √ √ √
AHRQ Effective Health
Care Program
√ √ √ √ √ √
National Quality Forum √ √ √ √ √ √ √ √
National Prevention
Council
√ √
National Priorities
Partnership
√ √ √ √ √ √

Proposed criteria include
those in statute and
additional criteria
Used in Prior
Efforts
Criterion
Required in
statute
√ Impact on Health √
√ Improvability via Research √
√ Inclusiveness of Subpopulations √
√ Impact on Health System Performance √
√ Gaps in Knowledge √
√ Variation or Disparities in Delivery or Outcomes √
√ Potential to Inform Decision-Making at Point of Care √
√ Responsiveness to Expressed Needs √
Advances CER Methods
Fits the Definition

•Impact on Health of Individuals and Populations
•Probability of Improvability via Research
•Inclusiveness of Different Sub-populations
•Current Gaps in Knowledge / Variation in Care
•Impact on Health System Performance
•Current Health Disparities
•Potential to Influence Decision-Making at Point of Care
•Responsiveness to Expressed Needs
•Advances CER Methods
•Fits the Definition of PCOR
Framework to Inspire Drafting of PCOR-
Specific National Priorities
Health Information
to Improve Patient
Experience
Prevention &
Screening
Acute Care
Impact of New
Technology
PCORI National Priorities
(Illustrative Examples)
Prevention & Screening
Develop information that will guide
patients and providers to improve
outcomes in chronic conditions.
Develop information to help patients
coordinate their care across multiple
providers and care settings.
Develop evidence to help patients
improve the safety of their health
care.
Appropriate Use
PCORI Criteria
Proposed PCORI Criteria
Existing
Priority Areas
In
Statute
Chronic Disease
Care
Palliative Care &
Pain Management
Care Coordination
Patient
Engagement
Safety

Stakeholder Engagement
Guiding Principles
Balanced
Representation
Transparency
Facilitate
Participation
Enable an open engagement process that makes clear how
participants can get involved and how their input will be
incorporated
Obtain feedback from a diverse and representative range of
patients, health care stakeholders and the general public
Provide easy, accessible forums for participants to provide
feedback equitably

Stakeholder
Engagement Overview
Stakeholder
Engagement
Plan
Identify
Engage
Track
Assess
Report

PATIENTS PROFESSIONAL ORGANIZATIONS
Individuals with Disease
Families
Caregivers
Patient Advocacy Groups
General Public
Providers & Practitioners
Employers
Payers
Research Societies
Academia
Health Information Exchanges
Federal/State/Local
Government
Congress
Life Sciences Industry
Who are PCORI’s
stakeholders?
Key Stakeholder Groups

Proposed Stakeholder Engagement Timeline
Create
Engagement
Plan
and Identify
Stakeholders
Decide on
Content,
Methods
of Dialogue
Conduct Dialogue with
Stakeholders
Review,
incorporate
and Report
on
Feedback
September October November December
Week
1
Week
2
Week
3
Week
4
Week
5
Week
6
Week
7
Week
8
Week
9
Week
10
Week
11
Week
12
Week
13
Week
14
Week
15
Week
16
* *
* Board Meetings
*

Richard Kuntz, MD
September 20, 2011
Landscape Review Program

Goals of Landscape Reviews
• Identify existing knowledge and gaps in existing knowledge related
to PCOR and CER
• Avoid funding patient-centered outcomes research that duplicates
ongoing or existing research
• Provide background information to spur innovation in research and
methodology, and contribute to the infrastructure for conducting
patient-centered outcomes research
• Create shared understanding about the state of the art of
dissemination and patient engagement strategies and contribute
to best practices
• Provide input into National Priorities and Research Agenda

Chronology of Landscape Review Development
Initial Phase:
Environmental Scan of
Comparative
Effectiveness Research
Discovery Phase:
Learning about
Lewin Group
Inventory of CER
Refining the Scope of
Landscape Reviews &
Methodology
Committee
Requirements
Coordination of
Landscape
Reviews across
PCORI: Landscape
Review Program
Landscape Review Program:
• Process to initiate and manage landscape reviews
• Templates to develop contract specifications and budget
requirements
• Resource library to share findings

Landscape Review Process Overview
Confirm questions
to be answered
Document known
information & gaps
Assess relevance /
usefulness of
information
Engage with NIH,
AHRQ, others to
leverage work of
interest to PCORI
Identify additional
information
needed
Prepare RFI as
needed to acquire
information
PCORI RESOURCE LIBRARY

Landscape Reviews/Summaries Completed or Underway
Report Status
“Existing and Ongoing NIH Work Related to Dissemination of Research: Overview
of NIH Briefing Book”
Completed 06/04/2011
“Existing and Ongoing Work Related to Patient Involvement in Peer Review:
Overview of NIH Briefing Book”
Completed 06/04/2011
“Overview of Lewin CER Inventory Project” Completed 07/07/2011
“Organizations that Conduct CER” Completed 08/09/2011
Review and Synthesis of Evidence for Eliciting the Patient’s Perspective in Patient-
Centered Outcome Research (Literature Review)
RFP issued 8/31/2011
Expert Stakeholder Interviews to Identify Evidence for Eliciting the Patient’s
Perspective in Patient-Centered Outcome Research (Interviews)
RFP issued 8/31/2011
Landscape review of existing methodological standards Under development by MC
Landscape review of patient engagement Under consideration by COEC

Board of Governors Meeting, South Sea Tea Washington

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Board of Governors Meeting, South Sea Tea Washington

Similar to Board of Governors Meeting, South Sea Tea Washington (20)

More from Patient-Centered Outcomes Research Institute

More from Patient-Centered Outcomes Research Institute (20)

Recently uploaded

Recently uploaded (20)

Board of Governors Meeting, South Sea Tea Washington