Your SlideShare is downloading. ×
0
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
PCORI at Academy Health
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

PCORI at Academy Health

288

Published on

PCORI's slide presentation at the AcademyHealth Annual Research Meeting on June 23-25, 2013.

PCORI's slide presentation at the AcademyHealth Annual Research Meeting on June 23-25, 2013.

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
288
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. K ATHE RI NE B . B E VAN S AMY K RATC HMAN C HRI STOP HE R B . FORRE ST C HI LDRE N’S HOSP I TAL OF P HI LADE LP HI A UN I VE RSI TY OF P E N N SY LVAN I A SUCCESSES AND CHALLENGES IN ENHANCING PATIENT ENGAGEMENT DEVELOPMENT OF METHODS FOR IDENTIFYING CHILD AND PARENT HEALTH OUTCOME PRIORITIES
  • 2. ADVANCES IN PATIENT-REPORTED OUTCOME MEASUREMENT AND ADOPTION • NIH Initiatives: Patient-Reported Outcome (PRO) Measurement Information System® (PROMIS), Neuro- QoL, NIH Toolbox • Provides qualitative and quantitative standards for demonstrating the reliability, validity, sensitivity, and efficiency of PRO measurement tools. • Food and Drug Administration guidance • Integration into EPIC • Increasing adoption in clinical research and practice
  • 3. A MISSING PIECE… • Usually, outcomes are selected without sufficient patient input • Precludes assessment of the full spectrum of patient concerns • May lead to identification of “best practice” interventions that have little or no effect on the outcomes that matter most to children and parents • Core sets = the “minimal standards” for the assessment and reporting of outcomes in clinical studies and encounters • Usually comprised of outcomes identified by researchers and clinicians • Some attempts to obtain patient input through qualitative means (e.g., focus groups)
  • 4. CHILDREN WITH AUTISM SPECTRUM DISORDERS: COMMONLY EVALUATED PRO’S VS. PARENT- IDENTIFIED OUTCOME PRIORITIES Behavioral and emotional inhibition and control Sensory regulation Restricted interests, flexibility, and adaptation Stereotyped behavior Energy level Externalizing behavior problems Interpersonal communication Anxiety PRO Parent-Identified
  • 5. PURPOSE OF PCORI PILOT PROJECT • To develop and evaluate novel methods for eliciting and prioritizing children’s health outcomes that are: • Comprehensible to both patients and clinicians • Measureable • Ready for integration into comparative effectiveness research • Focused on six of the most common childhood conditions: ASD, ADHD, epilepsy, JIA, asthma, obesity
  • 6. OUR TEAM Patient Advisors (6) Scientific Advisors (3) Researchers Bevans +3 Patient-Scientists Kratchman +3 Harmonization Concept Validation Patient Understanding Prioritization CORES: Co-directed by a researcher and a patient-scientist Patient Advocacy and Support Groups
  • 7. CHALLENGES AND LESSONS LEARNED: DEVELOPING THE TEAM • Inclusion of “representative” patients • Recognize and respect varying levels of patient engagement • Related to patients’ readiness to engage as a patient-scientist • Reasons for readiness = personal journey, experience, confidence… • Researchers’ readiness to adopt the PCOR model • Power and team structure • Communication: Learning the language vs. using a translator • Consider researchers’ and stakeholders’ unique and complementary expertise in assigning roles and responsibilities
  • 8. AIM 1: DEVELOP A TYPOLOGY OF HEALTH OUTCOMES ICF-CY, PRO concepts, NCS Intervention Research Outcomes (Systematic Literature Review) Clinician-Identified Outcomes Child- and Parent- Identified Outcomes Focus on six common childhood conditions: ASD, ADHD, epilepsy, asthma, JIA, obesity
  • 9. AIM 1: DEVELOP A TYPOLOGY OF HEALTH OUTCOMES ICF-CY, PRO concepts, NCS Intervention Research Outcomes (Systematic Literature Review) Clinician-Identified Outcomes Child- and Parent- Identified Outcomes Focus on six common childhood conditions: ASD, ADHD, epilepsy, asthma, JIA, obesity Researcher- Driven
  • 10. Develop Controlled Vocabulary Condition-Specific AND Measurement AND Self/Proxy- Report Limited to Children Execute Search PubMed 3189 articles Identify Unique Article Remove Duplicates 2865 articles Abstraction of Articles In-Scope 201 ASD articles (19%) 309 ADHD articles (39%) 292 epilepsy articles (29%) Identification of Tools 40 ASD tools 36 ADHD tools 28 epilepsy tools Obtain Tools 32 ASD tools 31 ADHD tools 24 epilepsy tools Examine and Classify Items until Concept Saturation is Achieved 1592 ASD items 698 ADHD items Epilepsy items (TBD) IDENTIFYING COMMONLY ASSESSED OUTCOMES SYSTEMATIC LITERATURE REVIEW
  • 11. TYPOLOGY MAPPING: VINELAND SOCIALIZATION SCALE Most commonly assessed ICF-CY categories: b126: temperament and personality functions d125: disposition and interpersonal functions b125: emotional functions d710: basic interpersonal interactions d720: complex interpersonal interactions d750: informal social relationships d330: speaking d880: engagement in play d920: recreation and leisure
  • 12. AIM 1: DEVELOP A TYPOLOGY OF HEALTH OUTCOMES ICF-CY, PRO concepts, NCS Intervention Research Outcomes (Systematic Literature Review) Clinician-Identified Outcomes Child- and Parent- Identified Outcomes Focus on six common childhood conditions: ASD, ADHD, epilepsy, asthma, JIA, obesity Patient-Scientist Driven
  • 13. ASSESSING CHILD AND PARENT PERSPECTIVES What problems would you like your child’s treatment to address? How would you like your child’s treatment to improve his/her life? A lot of research involving children with asthma focus on symptoms like difficulty breathing, but asthma impacts kids in other ways too. How does your child’s asthma impact his/her life? How would you like that to change? (Probe specific life domains: at school, with friends, how he/she communicates, etc.) • Preference for positive focus • “Treatment” facilitates focus on symptomatic outcomes • Need to be specific about condition • Consider impact on family, not just the child Participants: Youth aged 12+ with six conditions and their parents; two in-person and two online data-collection modalities
  • 14. AIM 1 PRODUCT Outcome typology derived from patients (parents and youth), clinicians, and a review of the intervention research
  • 15. CHALLENGES AND LESSONS LEARNED: AIM 1 • Empower patient-scientists to lead development of methodology • Methods for accessing and engaging patient partners to elicit patient- centered outcome priorities Data Quality Access Feasibility Acceptability Network Barriers: Geography Sample Bias Cost
  • 16. AIM 2: ENSURE THE COMPREHENSIBILITY AND CONTENT VALIDITY OF THE TYPOLOGY e-Delphi with clinicians Round 1: Open-ended elicitation of outcome concept definitions for clinicians and patients Rounds 2 and 3: Rate each outcome definition: • Clinician understanding • Patient understanding Suggest revisions 2x Cognitive Interviews with youth and parents Development of clinician and patient outcome definitions Revisions Test patient-centered outcome definitions using a cognitive interview approach Iteratively refine and re-test until definitions are well- understood Outcomes defined in BOTH clinician- oriented and lay language
  • 17. AIM 2 PRODUCT Typology with outcomes defined in clinician-oriented and lay language. Uses: • To translate and standardize the dialects that health professionals and patients use to describe health experiences • As a substrate for the outcome prioritization process
  • 18. CHALLENGES AND LESSONS LEARNED: AIM 2 • Communication: Learning the language vs. using a translator • Use the expertise of patient-scientists • Capitalize on clinicians’ experience in communicating with patients • Commitment to developing a replicable and scalable approach to identifying patient-centered outcomes (rather than a product)
  • 19. AIM 3: DEVELOP AND TEST METHODS FOR OBTAINING PATIENT-OUTCOME PRIORITIES • Test two technology-enabled methods for prioritizing patient outcomes • Method 1: Individually administered • Participants (youth with ASD and parents of children with ASD) are asked to choose between two outcomes (with patient definitions). • Outcome ratings are determined by analyzing results of the pair-wise comparisons using analytic hierarchy processes. • Method 2: Administered in groups of similar participants • Participants rate the importance of each outcome using an audience response system. • After each question, response frequencies are shown in real- time to the group to facilitate discussion.
  • 20. AIM 3 PRODUCT A scalable approach to identifying patient-prioritized outcomes Uses: • To establish “patient-centered” core sets for use in clinical research involving specific sub-groups (e.g., diagnostic, capacity, age) • For further testing in clinical settings as a means for identifying individual patient’s outcome priorities
  • 21. AIM 3: ANTICIPATED CHALLENGES AND FUTURE DIRECTIONS • Selecting an outcome prioritization approach that is understandable, is feasible, and yields meaningful results • Promote approaches for developing “patient- centered core sets” • Patient prioritization of outcomes included in PRO development recommendations/standards • Sustain the patient-scientist network • Involve patient-scientists in interpretation of results and dissemination efforts • Strengthen and deepen engagement
  • 22. Contact: Katherine Bevans, PhD Assistant Professor of Pediatrics CHOP/Penn Medicine bevans@email.chop.edu 267-426-2967
  • 23. PCORI PILOT PROJECTS Aims 1. To develop a typology of health outcomes that is relevant to the experiences and preferences of children. 2. To ensure the comprehensibility of the health outcome typology for children and their parents. 3. To develop and test the feasibility and practicality of methods that use the typology to prioritize health outcomes from the perspectives of children and their families, using Autism Spectrum Disorder as a proof-of-principle condition.
  • 24. Patient and Clinician Perceptions of Engagement in Research 1 Laura Forsythe, PhD, MPH AcademyHealth Annual Research Meeting June 25, 2013
  • 25.  Improved research recruitment and retention rates (Edwards et al. 2011)  Enhanced trust between researchers and participants (Decker et al. 2010; Edwards et al. 2011; Staniszewska et al . 2007)  Improved content and construct validity of measures (Cashman et al. 2008; Cotterell 2008)  Improved patient understanding of results (Chalmers 1995; McCauley et al. 2001; Doyle 2010)  Increased relevance of research results to patients (summarized in Nass et al. 2012) Patient Engagement in Research
  • 26. Objectives Describe and compare patient and clinician attitudes about engagement in research Identify perceived barriers to and facilitators of engagement in research 3
  • 27. Study Methods
  • 28. Methods: Survey Development 5 Identify Existing Survey Items Develop New Survey Items • Perceived value of engagement • Interest in engagement • Barriers and facilitators for engagement Partner with Patients and Clinicians for Feedback • Survey concepts • Item wording • Survey layout • Dissemination
  • 29. Methods: Crowdsourced Survey Recruitment from existing opt-in panels based on pre-supplied profile information Web based survey Rapid data collection Limited generalizability 6
  • 30. Methods: Item Format Example 7
  • 31. Respondents: Patients (N=900) 8 80% 20% Disease group Chronic disease patients Rare disease patients 89% 11% Primary Language English Spanish
  • 32. Respondents: Primary Care Clinicians (N=750) 9 53% 27% 12% 8% Type of Provider Physicians Nurse Practitioners Nurses Physician Assistants 7% 23% 35% 25% 10% Years in Practice < 3 Years 3 to 9 Years 10 to 19 Years 20 to 29 Years >30 Years
  • 33. Results 10
  • 34. Perceived Value of Research that Measures Things Patients Care About 11 87% 87% 0% 20% 40% 60% 80% 100% Patients Clinicians %Very/ModeratelyImportant Survey Group p>0.05
  • 35. 72% 77% 0% 20% 40% 60% 80% 100% Patients Clinicians %Strongly/SomewhatAgree Survey Group *p <0.05 Perceptions that Research Helps Patients Make Better Treatment Decisions 12
  • 36. Perceived Value of Engaging Patients in Research 13 86% 72% 83% 0% 20% 40% 60% 80% 100% CliniciansCliniciansPatients %Strongly/SomewhatAgree Survey Group *p<0.001 “Patients working directly with researchers can improve the value of medical research”
  • 37. Perceived Value of Engaging Clinicians in Research 14 86% 72% 83% 0% 20% 40% 60% 80% 100% CliniciansCliniciansPatients %Strongly/SomewhatAgree Survey Group *p<0.001 “Patients working directly with researchers can improve the value of medical research” “Providers working directly with researchers can improve the value of medical research”
  • 38. Interest in Engaging in Research 15 66% 55% 0% 20% 40% 60% 80% 100% Patients Clinicians %Strongly/SomewhatAgree Survey Group *p<0.001
  • 39. Barriers and Facilitators of Engagement 16 Barriers Facilitators Patients • Lack of time (43%) • Concerns about privacy (36%) • Work, school or caregiving commitments (33%) • Helping others with their medical condition (68%) • Learning about their health (63%) • Helping the next generation (57%) • Getting paid (56%) • Making research more meaningful to patients (49%) Clinicians • Lack of time (79%) • Lack of payment (47%) • Lack of research training (35%) • Helping patients receive better care (79%) • Getting paid (78%) • Contributing to scientific knowledge (61%) • Making research more meaningful for patients (61%) • Improving professional satisfaction (52%) • Helping researchers decide what to study (43%)
  • 40. Strengths and Limitations Strengths  Exploration of understudied topic areas  Inclusiveness of understudied populations: Spanish speakers, rare-disease patients Limitations  Generalizability  Self-reported data  New survey items testing complex constructs 17
  • 41. Conclusions Most patients and clinicians agreed that engagement can improve the value of health research. Many patients and clinicians reported interest in engaging in research themselves. Strategies to facilitate both patient and clinician engagement:  Establish link between engagement and patient care.  Provide financial compensation.  Minimize time burden. 18
  • 42. Implications Find ways to help those who are interested in engaging in research to be involved in meaningful ways. Develop evidence on the value of engagement in research, particularly for improving patient outcomes. Address barriers to engaging patients and primary care clinicians in research. 19
  • 43. Next Steps 20 Foundational Elements • Awareness of methods for PCOR • Valuing patient perspective • Interest in PCOR • Ways for patients and researchers to partner • Resources and infrastructure • Policies/governance
  • 44. Thank you! Acknowledgements  Patient, caregiver, and clinician partners  Collaborators: Lori Frank, Kara Odom Walker, Diane Hayes, Sue Levine, Ayodola Anise, Natalie Wegener, Gail Hunt, Anne Beal, Harlan Weisman, Freda Lewis Hall Laura Forsythe, PhD, MPH lforsythe@pcori.org www.pcori.org 21
  • 45. Developing Conceptual Frameworks to Inform and Guide PCOR 25 June 2013
  • 46. Understanding the choices patients face Aligning research questions and methods with patient needs Providing patients and providers with information for better decisions Patient Engagement Patient-Driven Research Dissemination Taking Patient-Centeredness Seriously
  • 47. Research engagement  Who determines  The research questions?  The database to use?  The variables to examine outcomes, comparators, and covariates?  The analytic methods?  Who is on the research team?  Patients or patient advocates?  Other stakeholders?  What will the research team do with the information?  How will results apply to health decisions?  Is there a plan for interaction between researchers and the community?
  • 48. Frameworks for PCOR Identify PCOR concepts and relationships between them Identify key components of PCOR and role of engagement Theoretical and evaluative frameworks:  One conceptual model for PCOR  A distinct measurement model
  • 49. PCORI Conceptual Model of PCOR Process for development informed by:  Evaluative framework of research engagement (AcademyHealth)  Review of PCOR literature Refined and vetted through:  Discussion with AcademyHealth  Input from PCORI’s Patient Engagement Advisory Panel
  • 50. PCOR conceptual model The following key changes were made based on discussion with the patient engagement advisory panel:  TBD  TBD Final structure: PCOR conceptual model  Use as basis for deriving testable measurement models.
  • 51. Foundational Elements • Awareness of methods for PCOR • Valuing of the patient perspective • Interest in PCOR • Ways for patients and researchers to partner • Resources and infrastructure • Policies/governance Actions • Initiate and maintain partnerships between researchers and stakeholders • Facilitate cross-communication among research stakeholders • Capture, use, and optimize patient perspective across phases of research • Ensure meaningful influence on research • Train for partnering • Share and use learnings Outcomes PCOR Principles Trust, Co-learning, Transparency, Respect • Culture of patient-centeredness in research • Meaningful & effective partnerships • Optimal health outcomes • Research relevant to patients/other stakeholders • Use of research results in health decisions • Quality health decisions • Satisfaction with health care experiences Conceptual Model for PCOR Outcomes Near-term Long-term Intermedia
  • 52. AcademyHealth and PCORI Partnership Contract awarded in 2012 to:  Conduct literature scan on stakeholder engagement in research (manuscript in progress)  Develop framework for patient engagement in research  Establish learning network for 50 PCORI pilot projects  Monitor progress and experiences of PCORI pilot projects in key areas 8www.pcori.org
  • 53. PCORI Pilot Projects Program 50 projects in 24 states and Washington, DC $31 million (over two years) www.pcori.org 9
  • 54. AcademyHealth Approach Framework for patient and consumer engagement in research:  Literature scan  Initial framework draft (input from PCORI staff)  Three iterations of input from Consumer Patient Researcher Roundtable  Revision and augmentation ongoing
  • 55. Evaluative Framework for Patient Engagement in Research Key Attributes  Intended for use as evaluative tool  Based on Donabedian model  Agnostic with respect to entity “sponsoring” research effort  Inclusive definition of the term “patient”  Acknowledges how much remains to be learned about how to conduct PCOR
  • 56. Evaluative Framework for Patient Engagement in Research Key Components:  Structure: refers to existence of assets, policies, and other infrastructure deemed necessary for conduct of PCOR  Process: examines how patient engagement is incorporated or integrated into research process  Outcomes: near and longer-term end points of patient engagement in research (such as improvement in research design, health outcomes)
  • 57. Engagement Infrastructure Engagement infrastructure Patient/researcher matching and outreach Patient-to- researcher mapping and selection Research stage(s) of engagement Continuity and frequency Structure Process
  • 58. Increased quality and/or relevance of research More informed decision making and uptake of research Improvements to dissemination and access to research Policy deliberations/changes Improvements in patient activation and empowerment Improvements in health outcomes and health status Outcomes Attitudes, perceptions, and activation Modifications to research process Interim Outcomes Longer-term Outcomes
  • 59. Next Steps Anticipated:  Still a work in progress, so welcome questions, suggestions, and other feedback  Developing manuscript for peer review Hoped:  Would like the framework to be used as a guidance tool for those engaged in PCOR • Not a prescriptive “how to”
  • 60. Thank You! Consumer Patient Researcher Roundtable Contributors: Bryan Dowd, Jason Goldwater, Mark Gorman, Alice Leiter, Musa Mayer, Eva Powell, John Santa, Shoshanna Sofaer, and Mike Stoto AcademyHealth and PCORI Staff: Rochelle Bent, Laura Esmail, Laura Forsythe, Lori Frank, Emily Moore, Raj Sabharwal, Veronica Thomas, Rachel Witsaman, and Natalie Wegener
  • 61. Improving the Impact of Patient-Engaged Research: Recommendations for Evaluation Lori Frank, PhD and Sue Sheridan, MIM, MBA AcademyHealth Annual Research Meeting June 25, 2013
  • 62. October 2012 Transforming Patient-Centered Research: Building Partnerships and Promising Models Workshop purpose: Begin a dialogue on how to conduct patient-centered research. Input received was used by PCORI staff to develop policy and programmatic recommendations. 2
  • 63. Workshop Structure The two-day event included 142 representatives from a range of communities.  75% of participants were patient representatives.  Other participants: clinicians, providers, researchers Small group brainstorming sessions 3
  • 64. Breakout Session Topics 4 Identifying and Selecting Research Questions How should PCORI identify and select specific research questions that are patient-centered for funding ? Reviewing Research Proposals for Funding How can PCORI effectively engage and use the real-world experience of patients to help evaluate research proposals we receive? Matching Patients and Stakeholders with Researchers How can PCORI connect patients and stakeholders with researchers for collaborative work that ensures studies reflect patient perspectives? Disseminating Research to the Community How do we ensure that patients and those who care for them can access and use PCORI’s research to make more- informed decisions? Evaluating PCORI’s Patient and Stakeholder Engagement Programs How can PCORI measure the effectiveness of its programs to involve patients and stakeholders throughout its work?
  • 65. Evaluating Engagement Session: Questions for Consideration Participants were asked to consider:  How can PCORI best measure the effectiveness of patient and stakeholder engagement in research?  What novel methods can patients and patient advocates propose for evaluation of research engagement that would capitalize on the growing networks of patients engaged in research? 5
  • 66. Evaluating Engagement in Research 6
  • 67. Nominations for Principles of Engagement Transparency Empathy Infrastructure change to achieve parity between researchers and patients Incorporating learning in an ongoing way Aiming for excellence in engagement Evaluating the meaningfulness, quality, and impact of the engagement
  • 68. Recommendations for Evaluating Engagement in Research • The objective for engaging patients and researchers as partners in research may be study- specific, but must be pre-defined to permit adequate evaluation of the partnership. Define Success • Topic generation will require different forms of engagement than will establishing research hypotheses or analyzing data and reporting findings. Determine goals of engagement by research phase • Use to ensure that all research partners can provide input on the quality and value of the research engagement, using established metrics based on definitions of success. Establish feedback channels • Including compensation for patient time and co-authorship Address patient and researcher parity • Assessments should happen before and after engagement. Assess patient and researcher perceptions of the value and appropriateness of engagement 8
  • 69. Items for Evaluating Engagement in Research  Stakeholders: How did engagement in the research change the project?  Has your PCORI-funded project been used by other research groups?  Was there a successful outcome for your study? Does it relate to quality of patient engagement?  Are people voting with their feet?  Was the research idea collaborative?  How quickly are results being used in further research and in clinical care? Additional metrics should include both subjective and objective measures of quality of engagement. 9
  • 70. Process Recommendations for Evaluation Funding applicants should be asked to build an evaluation of patient engagement into their proposals for the conclusion of their studies. Dissemination metrics: Go beyond peer-review. Evaluate quality and impact of engagement during and after the study. Evaluate diversity of project participants. Consider linking funding to evaluation results, 10
  • 71. Recommendations for Strengthening Engagement in Research Incentivize co-learning between researchers and patient partners. Organizational structures of health systems and research organizations should include patient research experts. “Seal of Approval” recognition of optimal research engagement methods Require it: Evaluate engagement in all PCORI funded projects. 11
  • 72. Conclusions Patients and other stakeholders provided concrete and feasible recommendations for evaluating quality of engagement in research. Metrics for success should be linked to quality of research and quality of engagement. Different metrics are needed by research phase. Organizational structures and funding requirements can be powerful levers for ensuring engagement quality. 12
  • 73. Implications for Policy, Delivery, and Practice Meaningfulness and impact of the research should be measured and link to engagement quality made overt. Research organizational structures can be modified to support success. 13
  • 74. Thank You! 14 Patient, caregiver, and clinician partners Contact: Lori Frank, PhD lfrank@pcori.org Sue Sheridan, MIM, MBA ssheridan@pcori.org
  • 75. Engaging Stakeholders to Improve Depression Management in a Tribal Health System 1 Renee Robinson (PI) Jennifer Shaw Vanessa Hiratsuka Sara Norman Julia Smith Denise Dillard Southcentral Foundation Helene Starks University of Washington
  • 76. Presentation Objectives and PCORI Pilot Project Overview • Familiarize audience with… • Organization, population, and stakeholders • Depression management at our organization • Decision-support tools • Stakeholder engagement • Approval versus engagement • Impact of engagement on process/project • Barriers encountered/proposed solutions 2
  • 77. Southcentral Foundation (SCF) • Mission – Working together with the Native Community to achieve wellness through health and related services • Vision – A Native Community that enjoys physical, mental, emotional, and spiritual wellness 3
  • 78. Population • SCF provides services to… – ~60,000 AN/AI people – Across 107,413 square miles – 231 federally recognized tribes – 60 villages 4
  • 79. Our Stakeholders • AAIRB • Tribal Leaders (EC, BOD) • Customer-owners • Providers • Steering Committee • Leaders 5 Customer- Owners ProvidersLeaders
  • 80. Depression Management at SCF • In 2001, SCF integrated annual depression screening into all primary care clinics due to: – High prevalence of symptoms – Low utilization of behavioral health services 6
  • 81. To make a good healthcare decision, you need to… • Clarify the decision… – What is the decision you need to make? – What is your reason for making this decision? – Look at timeline for the decision. • Explore the decision… – What healthcare options are available to you? – What are positive and negative effects of options? – What is the value of the positive and negative effects to you? (Explore your decision.) 7
  • 82. Decision-Support Tools • Interactive tool to aid clinical decision-making • Describe treatment options, risks, benefits, and efficacy • Help clarify preferences and values for treatment options • Communication between customer-owners and providers • Facilitate improved depression management – Educational – Interactive – Selective 8
  • 83. Pilot Project • Background and Rationale… – Many people don’t pursue depression treatment • Negative effects of untreated illness (e.g., absenteeism) – Many people prematurely discontinue treatment • Wasted resources of healthcare system (e.g., unused Rx) – Few people prefer holistic depression treatment • Under-utilized services 9
  • 84. Project Aims and Overview 10 Aim 1 • Identify stakeholder priorities, preferences, and needs for depression management • Key informant interviews with customer- owners, providers, and leaders Aim 2 • Develop and evaluate decision-support tool • Steering committee guidance and feedback • Pilot test in one primary-care clinic Aim 3 • Evaluate health and economic outcomes • Test refined tool in multiple clinics • Evaluate disease management, economic, and health outcomes
  • 85. Stakeholder Engagement: Aim 1 • Data Collection – 38 stakeholder interviews conducted Dec to Feb • 19 Customer-Owners, 9 Leaders, 10 Providers – Semi-structured interview guide • Depression screening and treatment resources • Decision-tool content, design, and deployment • Analysis – Transcripts summarized – Themes identified – Synthesized information presented to Steering Committee • Used to guide tool development 11
  • 86. Deciding on Drug Therapy 12 I didn't even realize that I was depressed for a long time. I thought everyone felt this way; at least, everyone in my family seemed to. I probably would have just gone on like that if my doctor hadn't asked one day if I had ever thought about taking an antidepressant. I was relieved to find out that it isn't normal to feel like I do and that a lot of people are helped by medicines. I know it might take a while to find the right one, but I'm in no hurry; I've spent my whole life feeling sad. I guess I'm just not comfortable with taking medicine for my depression. I feel like I ought to be able to manage this on my own without needing medicine. It seems too much like taking the easy route. But maybe I just don't feel bad enough yet. I recently began going to counseling. I know that if I took an antidepressant, I might feel better sooner, but I don't like the sound of the side effects I could have. My therapist and I have set some goals for me to work on, and we agreed to revisit my decision in three months. I want to wait and see how the counseling goes before I take medicine.
  • 87. Example Screens of First Draft of Decision-Support Tool Approval versus Engagement 13
  • 88. Get the facts • What is depression? – A very common, highly treatable, medical illness that involves the body, mood, and thoughts. Physical, mental, emotional, and spiritual well-being. – It affects how people think about things, feel about themselves, the way a person functions socially, at work and in relationships, as well as everyday activities like eating and sleeping. – It is more than feeling blue, down in the dumps or sad about a particular issue or situation.(Link to Table with Symptoms of Depression, See next slide) Link to Video about learn more about depression http://www.youtube.com/watch?v=IeZCmqePLzM&playnext=1&list=PL3E95B34CC364B7D0&feature=results_main 14
  • 89. What healthcare options are available to you to manage your depression… • Lifestyle Changes (Link to page expanding each) – Healthy eating – Exercise – Relationship building – Stress management • Behavioral Health Services - Therapy – Screening – Psychotherapy • Medications • Traditional Healing • Other
  • 90. Modifications to Decision-Support Tool Impact of Engagement on Process Barrier and Proposed Solutions 16
  • 91. Stakeholder Recommendations: Decision Tool Content • Provide context for the tool – Who will see the information and how will it be used • Give people information about depression – De-stigmatize/normalize (common life experience) – High level of depression in Alaska due to multiple factors (seasonal, trauma, situational) – Different types (situational versus chronic/organic) • Assess/offer resources – SCF and non-SCF resources, including Customer-Owner personal resources (social support) – Include non-medication resources (dietician, traditional healing, individual counseling, and groups) – Explain potential benefits and burdens (side effects, wait times) – Follow-up/make multiple contacts • Use story/culturally consonant communication – Include Native faces/voices; include testimonials of success Context Information Resources Story 17
  • 92. • Adapt for multiple audiences – iPad generally endorsed for providing screener and general information, especially with younger people – Elders and others may prefer personal or written option • Strategize location and timing – Could deploy screen in waiting area or provide general info on closed-circuit TV in waiting area and use iPad for screen in examination or talking room • Maximize customer choice – Make tool voluntary, not mandatory – Provide staff assistance with tool (e.g., RN or BHC) – Allow time for customers to explore options, including time in examination room and in follow-up appointments • Align with other initiatives – Minimize duplication and multiple requests for information Stakeholder Recommendations: Decision Tool Design and Deployment Alignment Audience Strategy Choice 18
  • 93. Factors Related to Workflow • Administration of tool – BHC – PCP • Role of tool – Facilitate discussion – Provide information – Support Decision • Time 19
  • 94. Example Screens of Resultant Decision-Support Tool Impact of Engagement on Process 20
  • 95. The Signs and Symptoms that Bother Me Most Are … Feel Hopeless Loss of Interest in Daily Activities Weight and/or Appetite Changes Sleep Changes Anger or Irritability Decreased Energy Self Loathing Trouble Concentrating Pain Exit
  • 96. What can I do about my depression? Healthy Eating Exercise Stress Reduction Support Groups Clergy Counseling Traditional Healing Drugs Herbal Remedies Exit
  • 97. Engagement Barriers and Proposed Solutions • Barriers – Cultural – Time – Communication – Information • Solution – Cultural-adaptation – Technology/resources – Decision-support tool – Decision-support tool and extended tool 23
  • 98. Other Example Screen Shots of Old and New Tools 24
  • 99. Every day people face complex health decisions… However, most healthcare decisions have no clear best choice. 0% 10% 20% 30% 40% 50% Beneficial Outcomes Probably Beneficial Need to Weigh Benefits versus Risk Probably Not Beneficial Insufficient Evidence of Usefulness Healthcare Intervention Classifications 25
  • 100. Balance, Health and Wellness Exit
  • 101. What causes depression? Life Experiences Biology Exit Money problems/ unemployment Grief and loss Violence Drugs/ Alcohol Abuse Genetics Neurotransmitters (brain chemicals) Hormones Other Medical Conditions
  • 102. AcademyHealth ARM PCORI Engagement Panel ADDRESSING MENTAL HEALTH NEEDS OF RURAL AFRICAN AMERICANS: LESSONS FROM ENGAGEMENT
  • 103. Acknowledgements Funding: • UAMS Translational Research Institute (UL1RR029884; KL2RR029883) • Patient-Centered Outcomes Research Institute (PCORI)
  • 104. This Presentation Will: 1. Describe the process of engaging stakeholders in our pilot project 2. Identify lessons about engaging stakeholders 3. Briefly describe the current status of the project
  • 105. The setting: Arkansas Delta • Primarily rural with agriculture- based economy • Characterized by: • Poor economic conditions • Higher prevalence of chronic health conditions • Increased risk of early mortality • Poor access to quality health services • Highest percentage of cities with predominately African-American populations
  • 106. Mental Health in the Delta • Poorer self-reported mental health • Increased levels of frequent mental distress (BFRSS) • Poorer mental health outcomes • Increased burden of disability • Increased mortality • Poorer management of chronic physical health • Underuse of mental health services • Limited access • Stigma
  • 107. UAMS Psychiatric Research Institute • Division of Health Services Research • Improve the lives of individuals with mental health or substance use disorders • Focus on rural areas • Focus on partnership with key stakeholders What can we do to improve mental health service use in rural Arkansas?
  • 108. LESSONS FROM ENGAGEMENT Decide who you need to partner with: • What type of expertise do you need to answer your question? • Ground • People affected by the condition • Grassroots • People and organizations who are near the ground • Caregivers, support system, and service organization • Grass tops • Policymakers
  • 109. The Partners: Tri County Rural Health Network • Improves the health of individuals in the Delta • Uses community health worker model • Connects community members to health resources in rural counties • Works with UAMS College of Public Health to create “research friendly communities” • Connects with community members and policymakers
  • 110. Mental Health in the Delta I’m worried about our minds. About our ability to cope with life. Especially with our young people. I used to didn’t hear about depression or suicide. That was never something you heard about with our people. Now, I have had to sit with several families of young Blacks who’ve taken their own lives. I have people who are depressed in my congregation. Something has to be done. - A pastor from the Arkansas Delta
  • 111. LESSONS FROM ENGAGEMENT Determine if you are ready to engage: 1. Do you have similar values/missions? • Both focused on improving health in rural areas 2. Do you have the capacity to work together? • Tri County experience with research • DHSR experience with community engagement 3. Will this partnership be mutually beneficial?
  • 112. What’s the Solution? • Focus on improving access to and use of mental health services • Adapt community-connector program • Build on both partners’ expertise • Start with identifying the mental health needs of rural African Americans • Important to hear the voice of the community DHSR’s Idea Tri County’s Idea Tri County was originally created to provide job training in the Delta. But the people said, ‘We ain’t got no jobs here, so we don’t need no job training. What we do need is better healthcare.’ Let’s hear the community before deciding where to start.
  • 113. Our Idea • Conduct formative research to: • Better understand the mental health needs of rural African Americans • Inform the development of culturally appropriate interventions to improve mental health in rural African American communities
  • 114. Writing the Application • Focus Groups • Researcher identifies stakeholders • Researcher creates interview guide • More traditional research method • Deliberative Democracy • Allows community to self-identify as stakeholders • Community creates the frame DHSR Methods Tri County’s Methods The energy is important. Forums create energy around an idea. What would happen if we implemented an intervention in an area that has this energy already buzzing?
  • 115. Writing the Application • New Questions: • What is the best way to gather stakeholder input? • Does one method lead to more activation? • Is activation important in intervention implementation?
  • 116. Specific Aims • Compare two ways of gathering stakeholder input: deliberative democracy and focus groups • Themes • Empowerment • Gather information to inform the development of an intervention to improve mental health in rural African American communities
  • 117. LESSONS FROM ENGAGEMENT Engagement before Funding 1. Helps decide what are the most relevant research questions • Whole idea changed as a result of partner input 2. More responsive questions= Better applications
  • 118. Starting the Research • Community Advisory Board • One provider • Two mental health consumers • One connector • One college student • One clergy • One lay community member • Advises on consent procedures, measurement tools, recruitment, and data analysis
  • 119. Mental Health versus Emotional Wellness No one is going to talk to you about mental health. When people hear mental health they think of crazy. Ain’t nobody going to talk to you about being crazy. I ain’t crazy. I don’t know how to help you help crazy people If you want my expertise, you have to ask me about things I know about. CAB member
  • 120. Mental Health versus Emotional Wellness • Based on CAB feedback: • Application written using the medical frame • Wellness framework is more culturally acceptable • Interview guides and framing sessions used term “emotional wellness” instead of “mental health”
  • 121. LESSONS FROM ENGAGEMENT Respect your partner’s expertise 1. Partner is a part of the research team • Not just approval 2. Listen and incorporate ideas when possible • Using wellness frame versus illness frame • Changing wording of measurement tools • Not paying unless they finish 3. Respect builds trust • Trust builds better partnerships
  • 122. LESSONS FROM ENGAGEMENT Prepare for some differences of opinions • Audio recording versus note taking But remember: • Your core values; what are you working towards • Respect each other’s point of view and expertise
  • 123. Where are we now? • Completed six focus groups (n=50) • Faith community • College students and administrators • Patients • Providers • Currently completing community forums (n=86) • Four forums completed (two more planned) • Lay community members • Service organization leaders • Political leaders
  • 124. Where are we now? • Preliminary analysis • Stigma and low mental health literacy major barriers to care • Importance of “reaching people where they are” • Community-based services versus clinic-based services • Importance of community support in prevention, treatment, and recovery • Provide education and support • Address contextual causes that affect emotional wellness
  • 125. Conclusions Engagement can lead to: 1. More culturally acceptable research questions 2. Applications that are more responsive to community needs When engaging stakeholders in research: 1. Make sure you are ready to engage in research 2. Engage stakeholders before funding 3. Respect your partners’ expertise 4. Prepare for differences of opinions
  • 126. Any real attempt to address the problems that are plaguing our community will take everyone coming to the table; the grassroots, the grass-tops, and all the grass in between. Everyone talks about how difficult it is to bring everyone together, but if you focus on solving the problem, the mountains become just bumps in the road. Mary Olson, Tri County Rural Health
  • 127. Academic Team Greer Sullivan, MD MSPH (PI) Mary Olson, D. Min (Community PI) Geoff Curran, PhD Ann Cheney, PhD Keneshia Bryant, PhD FNP Christina Reaves, MPH Naomi Cottoms Elise Allee Faye Smith
  • 128. Community Advisory Board Rev. George Barnes Linda Cole Melva Trask Gloria Scott Edlun Marshall Pamela Barnes Earnest Virgil Ward

×