The webinar provided an overview of the PCORI Methodology Standards and the process used to develop them. It discussed how the standards aim to improve the quality, transparency, and rigor of patient-centered outcomes research by reflecting best practices. The webinar faculty described how some standards apply broadly to most projects while others are specific to certain study designs or data sources. They emphasized applying the standards to formulating important research questions, using patient-centered approaches, and fully documenting methods and results. Looking ahead, PCORI plans to provide more guidance on implementing the standards while continuing to refine them based on experience.

1. Using the PCORI Methodology
Standards to Generate Robust,
Relevant, and Timely Evidence for
Patient-Centered Outcomes Research
Joe V. Selby, David Hickam, Brian Mittman, and
Sean Tunis
June 4, 2013

2. The audio and slide presentation will
be delivered directly to your
computer
 Speakers or headphones are required to hear the
audio portion of the webinar.
 If you do not hear any audio now, check your
computer’s speaker settings and volume.
 If you need an alternate method of accessing audio,
please submit a question through the Q&A pod.

3. Technical Assistance
 Live technical assistance:
– Call Adobe Connect at (800) 422-3623
 Refer to the ‘Technical Assistance’ box
in the bottom left corner for tips to
resolve common technical difficulties.

4. To submit a question:
1. Click in the Q&A
box on the left side
of your screen
2. Type your question
into the dialog box
and click the Send
button
Questions may be submitted at
any time during the presentation

5. Welcome
 Erin Holve, Ph.D., M.P.H.,
M.P.P.
– Senior Director of Research
& Education, AcademyHealth

6. A Collaborative Effort to
Improve Training in PCOR

7. PCORI Mini-Course
 Two morning panels
– Developing a Program of Comparative Effectiveness
Research at the Patient Centered Outcomes Research
Institute” (9:45-11:15am)
• Chair: Joe Selby, PCORI
• Faculty: Chad Boult, Rachael Fleurence, Romana Hasnain-
Wynia, and David Hickam, PCORI
– “PCORI Methods Pilot Projects: Learning from Engagement”
(11:30am-1:00pm)
• Chair: Sue Sheridan, PCORI
• Faculty: Katherne Bevans, Children’s Hospital of Philadelphia;
Renee Robinson, Southcentral Foundation; and Tiffany
Haynes, University of Arkansas for Medical Sciences

8. PCORI Mini-Course
 Afternoon workshop (June 25, 2:00-6:00pm ET)
– Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins
University School of Medicine
– Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
– Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of
Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome

9. Learning Objectives
 Provide an overview of the PCORI Methodology
Standards and the process used to develop the
standards;
 Provide general guidance on using the standards
when responding to PCORI funding announcements;
and
 Review plans for implementing the standards and
updating them in the future.

10. Webinar Faculty
Joe V. Selby, MD,
MPH
David Hickam, MD,
MPH
Brian Mittman,
PhD
Sean Tunis, MD,
MSc

11. Academy Health Webinar
June 4, 2013
Joe V. Selby
Executive Director, PCORI
Using the PCORI Methodology
Standards to Generate Robust,
Relevant, and Timely Evidence
for Patient-Centered Outcomes
Research

12. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
About PCORI
• An independent, non-profit organization authorized by
Congress.
• Mission is to fund research that will provide patients, their
caregivers and clinicians with the evidence-based
information needed to make better-informed health care
decisions.
• Committed to continuously seeking input from patients
and a broad range of stakeholders to guide its work.

13. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
What is Patient Centered Outcomes
Research?
Patient-Centered Outcomes Research (PCOR) helps people and their
caregivers communicate and make informed health care decisions, allowing
their voices to be heard in assessing the value of health care options.
PCOR has the following characteristics:
• Actively engages patients and key stakeholders throughout the research
process.
• Compares important clinical management options.
• Evaluates the outcomes that are most important to patients.
• Addresses implementation of the research finings in clinical care
environments.

14. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Approximate PCORI Funding
* 20% of this amount goes to AHRQ/HHS each year for dissemination and capacity-building

15. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Current Funding Opportunities and Events
http://www.pcori.org/funding-opportunities/funding-announcements/funding-center/
Announcement Letter of Intent Due
Assessment of Prevention, Diagnosis, and Treatment Options 06/15/13
Improving Healthcare Systems 06/15/13
Communication and Dissemination Research 06/15/13
Addressing Disparities 06/15/13
Improving Methods for Conducting Patient-Centered Outcomes
Research
06/15/13
Improving Infrastructure for Conducting Patient Centered Outcomes
Research: Phase One —Clinical Data Research Networks
06/19/13
Improving Infrastructure for Conducting Patient Centered Outcomes
Research: Phase One —Patient-Powered Research Networks
06/19/13

16. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
David Hickam, MD, MPH, PCORI

17. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Developing the PCORI Methodology
Standards
• Congressional Requirements- Patient Protection and Affordable
Care Act, Subtitle D, Paragraph (6)(C)(i)

18. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
PCORI Methodology Committee
• Sherine Gabriel, MD, MSc (chair),
Mayo Clinic
• Robin Newhouse, PhD, RN (vice
chair), University of Maryland
• Naomi Aronson, PhD, BlueCross
BlueShield
• Ethan Basch, MD, MSc, Memorial
Sloan-Kettering Cancer Center
• Alfred O. Berg, MD, MPH, University
of Washington
• David Flum, MD, MPH, University of
Washington
• Steven Goodman, MD, MHS, PhD,
Johns Hopkins School of Public
Health
• Mark Helfand, MD, MS, MPH, Dept.
of Veterans Affairs
• John Ioannidis, MD, DSc, Stanford
University
• Michael S Lauer, MD, National Heart,
Lung, and Blood Institute
• David O. Meltzer MD, PhD,
University of Chicago
• Brian S. Mittman, PhD, Dept. of
Veterans Affairs
• Sebastian Schneeweiss, MD, ScD,
Harvard Medical School
• Jean R. Slutsky, Agency for
Healthcare Research and Quality
• Mary Tinetti, MD, Yale University
School of Medicine
• Clyde Yancy, MD, MSc, Northwestern
University Feinberg School of
Medicine

19. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
PCORI Methodology Standards
http://www.pcori.org/research-we-support/methodology/
PCORI
Methodology
Committee
Formed
(2011)
Draft
Methodology
Report
Posted (July
2012)
Public
Comment
Period (July-
September
2012)
Proposed
Revisions to
Methodology
Standards
(November
2012)
Revised
PCORI
Methodology
Standards
(December
2012)
Final
Methodology
Report
(Spring 2013)

20. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
11 Categories of Methodology Standards
47 Individual Standards
*Bold indicates topics to be discussed in detail at the June 25 in-person workshop at the AcademyHealth Annual Research
• Formulating Research
Questions
• Patient-Centeredness
• Systematic Reviews
• General and Cross-cutting
Methods for all PCOR
• Causal Inference
• Heterogeneity of
Treatment Effects
• Preventing/Handling Missing
Data
• Data Networks
• Adaptive and Bayesian
Trial Designs
• Data Registries
• Studies of Diagnostic Tests

21. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Characteristics of the Methodology
Standards
• Are minimal standards for performing comparative effectiveness
research.
• Are intended to provide helpful guidance to researchers and those
who use research results.
• Reflect generally accepted best practices.
• Provide guidance for both project protocols and reporting of
results.
• Are used to assess the scientific rigor of funding applications.
• Context of research should drive use of the standards.

22. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Standards that Apply to Most Projects
• Formulating research questions (6 standards)
• Patient-centeredness (4 standards)
• General and cross-cutting (6 standards)
• Missing data (5 standards)

23. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Formulating Research Questions
 Identify affected populations and health decision(s), and how
study results will inform the health decision (RQ-3)
 Identify and assess participant subgroups (RQ-4)
 Select appropriate interventions and comparators and make
explicit how each were selected (RQ-5)

24. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Patient-Centeredness
 Engage the population of interest and other relevant stakeholders
as appropriate and necessary in a given research context (PC-1)
 Identify, select, recruit, and retain study participants
representative of the population of interest (PC-2)
 Use patient-reported outcomes when patients or people at risk of a
condition are the best sources of information. (PC-3)
 Support dissemination and implementation of study results (PC-4)

25. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
General and Cross-cutting Methods
• Assess the suitability of the data source (GM-1)
• Document data analysis plans for major aims (GM-3)
• Document validated scales and tests
• Include characteristics of the scales and psychometric
properties (GM-4)

26. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Missing Data
• Use Validated Methods to Deal with Missing Data that
Properly Account for Statistical Uncertainty Due to
Missingness (MD-3)
• Record and Report All Reasons for Dropout and Missing
Data, and Account for All Patients in Reports (MD-4)

27. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Standards that Apply to Specific Data
Sources and Study Designs
• Systematic reviews (1 standard)
• Causal inference (6 standards)
• Heterogeneity of treatment effects (4 standards)
• Data networks (2 standards)
• Adaptive trials (5 standards)
• Registries (3 standards)
• Diagnostic tests (5 standards)

28. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Causal Inference
• Precisely Define the Timing of the Outcome Assessment
Relative to the Initiation and Duration of Exposure (CI-3)
• Measure Confounders before Start of Exposure. Report data
on confounders with study results (CI-4)

29. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Diagnostic Tests
• Assess the Effect of Factors Known to Affect Diagnostic
Performance and Outcomes (DT-3)

30. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Brian Mittman, PhD, Department of Veterans Affairs

31. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
How to Use the Standards
• The standards represent “best practice” recommendations for
improving the quality and value of PCOR and CER.
• The standards aim to achieve greater transparency, rigor and other
improvements to strengthen the science and methods of PCOR and
CER.

32. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Implementing the standards: improving PCOR
research practices
• “Research practices” include decisions and actions by numerous
stakeholders (funders, reviewers, researchers, etc.)
• Changing these practices is challenging
– Researchers value independence and autonomy
– Researcher actions are guided by knowledge, experience,
judgment
• Analogue: the challenge of improving research practices through
standards is comparable to the challenge of improving clinical
practices through evidence-based clinical practice guidelines.

33. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Implementing research standards
• Professional practices are influenced by multiple factors operating at
multiple levels
• Implementation gaps must be thoroughly diagnosed to identify “root
causes”
• Implementation campaigns must include multiple, coordinated
components targeting the full spectrum of determinants of gaps
• Influences on research practices – and points of leverage to change
practices – should be depicted in a causal, conceptual model
There are “no magic bullets”: examples of dramatic change in research
practices (e.g., trial registration) are rare and exceptional.

34. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Conceptual model of the research process
• Research question identification and prioritization (researchers,
funding agencies)
• Study design and methods selection (researcher judgment guided
by training, professional norms, published standards)
• Peer review (reviewer judgment guided by training, professional
norms, published standards)
• Study governance and conduct (researcher judgment, possibly
guided by reporting requirements, IRB review, DSMB review)
• Documentation and reporting (researcher judgment guided by
reporting guidelines, journal requirements, peer review)

35. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Dissemination and Future Directions
• Initial set of standards released in December 2012.
• PCORI Methodology Report will be released in Summer 2013.
• No change in the set of standards.
• Provides further guidance on interpretation and use.
• Responding to future PCORI Funding Announcements (PFAs).
• All future submissions must demonstrate adherence
to/adoption of/consideration of the standards.
• PCORI will continue to develop tools and resources for
researchers, reviewers and other stakeholders.

36. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Sean Tunis MD, MSc - CMTP

37. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Purpose of Standards: PMC draft report 7/12
“Departures from basic good research practices are partially
responsible for the mismatch between the quality and relevance of
the information research provides and the information patients need
to make informed clinical decisions.”
“The initial range of topics was chosen to reflect areas in which the
Committee believed that there were either substantial deficiencies or
inconsistencies in how available methods were applied in practice, or
for which there was specialized knowledge in how best to conduct
research that had not been effectively disseminated.”

38. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Required by Law! (ACA)
“Methodological standards shall provide specific criteria for”:
• Internal validity (robust)
• Generalizability (relevant)
• Timeliness (timely)
• Feasibility

39. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Where to Focus
• Craft an important, relevant research question
• Informed by gaps in systematic reviews and meaningful
input from patients and other decision makers
• Patient-centered care requires attention to subgroups and PROs
• Apply best practices in application of selected methods
• Provide full transparency
Clear, detailed protocol – explain deviations from standards
Follow existing reporting standards

40. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Asking the Right Question
“To produce information that is meaningful and useful to people when
making specific health decisions, research proposals and protocols
should describe: 1) the specific health decision the research is
intended to inform; 2) the specific population for whom the health
decision is pertinent; and 3) how study results will inform the health
decision.”
• Requires dialogue with decision makers (more is better)
• Methods should serve research question; not vice versa

41. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Use of Patient Reported Outcomes
“Select outcomes based on input directly elicited from patient
informants, people representative of the population of interest, either
in previous studies or in the proposed research.”
• Use existing core outcomes measures sets when available
• COMET (Core Outcomes Measures in Effectiveness Trials)
• Effectiveness Guidance Documents (Green Park Collaborative)

42. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Feasibility: An Unresolved Issue
Standard for Studies of Dx Tests:
“Studies of clinical outcomes after diagnostic testing should use a
prospective randomized study design when possible. If a non-
randomized design is proposed, the reason for using an observational
study (or modeling and simulation) should be addressed and efforts
to minimize confounding documented.”

43. PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Looking Forward
• Specifics of implementation of standards not yet clear
• Initial emphasis of methods standards on patients and
caregivers as decision makers. Intent to expand scope to
clinicians, payers/purchasers, policy makers, others.
• PCORI thinking continues to evolve based on experience and
public feedback from researchers and other stakeholders
• How best to achieve optimal balance of internal validity,
generalizability, timeliness and feasibility remains an
important challenge

44. To submit a question:
1. Click in the Q&A
box on the left side
of your screen
2. Type your question
into the dialog box
and click the Send
button
Submitting Questions

45. Expert Q&A
 Erin Holve (facilitator)
 Dave Hickam
 Brian Mittman
 Sean Tunis

46. Still time to sign up for the PCORI Mini-Course
• In-person events at AcademyHealth’s Annual Research Meeting in Baltimore, MD (June 25)
Two morning panels
• “Developing a Program of Comparative Effectiveness Research at the Patient Centered
Outcomes Research Institute” (9:45-11:15am)
• “PCORI Methods Pilot Projects: Learning from Engagement” (11:30am-1:00pm)
• Afternoon workshop (June 25, 2:00-6:00pm ET)
• Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins University School of
Medicine
• Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
• Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome
• Click here to register: $50 for AcademyHealth members, $100 for nonmembers

47. Thank You
Please take a moment to fill out the
brief evaluation which will appear in your browser.