Downloaded 80 times
More Related Content
Similar to Implementation Research: A Primer
Implementation Research: A Primer
- 1. Implementation Research: APrimer Stefan Baral, MD, MPH, CCFP, FRCPC (CM)
- 2. Overview Definitions Characteristics Conceptual Frameworks Evidence and Outcomes Methods Qualitative Quantitative Descriptive Analytic Experimental Conclusions 2
- 3. Definitions Implementation Theuse of strategies to introduce or change evidence-based health interventions (policies, programs, individual practices) within specific settings Implementation Science in HIV Implementation science is a multi-disciplinary field that seeks generalizable knowledge about the behaviour of stakeholders, organizations, communities, and individuals in order to understand the magnitude, reasons for and strategies to close the gap between evidence and routine practice for health in real world contexts Key Themes Multidisciplinary Generalizable Multiple stakeholders Closing gap between evidence and practice Real world contexts 3Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; Odeny, Padian, Doherty, Baral, Beyrer, Ford, Geng, Definitions of implementation science used in the HIV/AIDS literature: a synthetic review. The Lancet Infectious Diseases, In Press, 2015
- 4. Implementation Research andOther PH Study Designs 4Source: Olakunle Alonge, Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; https://catalyst.harvard.edu/pathfinder/t2detail.html
- 5. Characteristics of ImplementationResearch 5 Findings are Warranted to Inform Policy/Program There is “sufficient evidence” to support the conclusions of the work What is sufficient evidence? Transparency of Methods Support Critical Assessment of the Study Whether processes are adequate Conclusions justified Repeatability Don’t be afraid of “failure” A well done study is still a success in terms of generating generalizable knowledge
- 6.
- 7. Differences with IR Competencieson a IR team: Research Methodologist Qual, Quant, Mixed Methods Ministry, Government, Agencies Either as members of team or study oversight committee Health Professionals Involvement of health professionals from study settings Communications Public Health Professionals Health Commissioner/Associate Health Commissioner Public Health Inspector/Public Health Nurse Privacy Expert Stakeholder Assessment Community 7http://www.who.int/tdr/publications/year/2014/ir-toolkit-manual/en/
- 8. Traditional IR Approaches 8http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml ActiveConsultation Active Consultation Appropriate Team, Active Consultation Active Consultation Active Consultation Active Consultation
- 9. IR Specific Objectives:Three Broad Areas Three Broad Areas of IR Specific Objectives 1. Describe Health Situation or Interventions 2. Provide Data to Evaluate Ongoing Interventions or Information Needed to Adjust Interventions 3. Analyze missed targets and potential solutions 9
- 10. Describe Health Situationand Intervention Magnitude of the problem Distribution of health needs of the population Risk factors for some problems People’s awareness of the problem Utilization patterns of services Cost-effectiveness of available and potential other interventions 10
- 11. Evaluate Interventions Coverageof priority health needs Coverage of target groups Acceptability of the services Quality of services Cost-effectiveness of the intervention Impact of the program on health 11
- 12. Analyze Missed Targets Availability Acceptability Affordability Service delivery problems Fidelity 12
- 13. Evidence and Outcomesin Implementation Research Conceptual Frameworks Outcomes Evidence
- 14. Conceptual Frameworks Commonlyused in IR RE-AIM Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance Stages of implementation National Implementation Research Network Exploration and Adoption, Program Installation (Prep), Initial Implementation (pilot/adapt), Full Implementation (>50% coverage), Sustainability Consolidated Framework for Implementation Research Intervention Characteristics, Inner Setting, Outer setting, Individuals in the Intervention, Implementation process Many others…. 14 Source: Glasgow et al 1999, National Implementation Research Network, 2005, Damschroder, 2009
- 15. Outcomes in ImplementationResearch 15 Clients Outcome Satisfaction Symptomatology Function Population- Based Incidence of diseases Morbidity Mortality DALYs Health Outcomes Efficiency Coverage Equity Responsiveness Services Outcomes Acceptability Adoption Appropriateness Costs Feasibility Fidelity Penetration Sustainability Implementation Outcomes Source: Olakunle Alonge, Proctor et al 2011
- 16. Implementation Outcomes 16 Implementation Outcome Working Definition*Related terms** Acceptability Perception among stakeholders that an intervention is agreeable Related factors: (e.g. Comfort, Relative advantage, Credibility) Adoption Intention, initial decision, or action to try to employ a new intervention Uptake, Utilization, Intention to try, Appropriateness Perceived fit or relevance of the intervention in a particular setting or for a particular target audience (e.g. provider or consumer) or issue Relevance, Perceived fit, Compatibility, Perceived usefulness or suitability Feasibility The extent to which an intervention can be carried out in a particular setting or organization Practicality, Actual fit, Utility, Trialability Fidelity The degree to which an intervention was implemented as it was designed in an original protocol, plan, or policy Adherence, Delivery as intended, Integrity, Quality of programme delivery, Intensity or dosage of delivery Implementation cost Incremental cost of the implementation strategy Marginal cost, Total cost*** Coverage Degree to which the population that is eligible to benefit from an intervention actually receives it. Reach, Access, Service Spread or Effective Coverage, Penetration Sustainability The extent to which an intervention is maintained or institutionalized in a given setting. Maintenance, Continuation, Routinization Institutionalization, Incorporation Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013
- 17. What is SufficientEvidence? Evidence-based medicine is a global standard Double-Blinded (DB) RCT is gold standard Evidence-based PH interventions should also be a global standard Often limited evidence, PH decision still needs to be made Precautionary Principle for PH? When there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation To develop guidelines Need to characterize Efficacious Effective Sustainable and Scalable programs 17
- 18. Tension Between Internaland External Validity Challenges for Evidence Combination Prevention Internal Validity Minimal study biases suggesting confidence in ultimate conclusion of the study External Validity Generalizability of ultimate findings to broader population Traditional Question for Clinicians/Programmers Does it work? What is effect size? Should I use it? Implementation Questions How, when, why, and where does it work? What factors influence effectiveness? Should I use it? How should I use it?
- 19. Traditional Research Pathway Effectiveness Research (and guideline development) generally happens prior to implementation research Are there more time-effective approaches to integrate implementation research with effectiveness/efficacy research Assess barriers/facilitators to intervention uptake acceptance/adoption/routinization Diagnose quality gaps Fidelity Characterize Sustainability Maintenance, Cost-Effectiveness
- 20. Methods in ImplementationResearch
- 21. Research Objectives QualitativeMethods Exploration Using inductive methods to explore concepts, constructs, phenomena, situations to develop hypotheses Quantitative Methods Explanation Characterizing the Relationship between concepts and phenomena Reasons for occurrences of events Prediction Use knowledge to forecast events Experiment Intervene/manipulate different settings or variables to produce a desired outcome Mixed Methods Description Identify and describing the precursors/antecedents, nature, or etiology of an outcome/phenomena
- 22. Qualitative Methods inImplementation Research To explore a health problem Evidence-based approach to characterizing individual, network, and structural determinants of the health issue To identify variables that can later be measured Measuring the “right” implementation outcomes To develop a complex and detailed understanding of an issue Formative work to design the implementation plan for an effective intervention Maximizing Acceptability, Appropriateness To understand the contexts or settings surrounding an experience or phenomenon Case-Study of a program or a policy To understand the meaning behind an issue or experience Explore the “why” or “how” an intervention works or why not Fidelity
- 23. Quantitative Research Designs Observational Categories Descriptive Analytic Cross-Sectional Prospective Quasi-Experimental Studies no randomization of individuals/communities/institutions Experimental Rapid Innovation in Implementation Design
- 24. Cohorts In ImplementationResearch Method Traditional “Active” Cohort includes Enrollment of participants and active follow-up of individuals Expensive to ensure retention “Passive” Cohorts (registries, EHR Repositories, etc) Rapidly becoming the standard in IR, but limitations include quality and extent of data collection What is Different from Traditional Cohort Studies? Traditional Cohorts Incidence, Prevalence, Relationship between Exposure and Outcome Cohorts in IR focus on measuring traditional IR Outcomes Acceptability, adoption, appropriateness, feasibility, fidelity of interventions, implementation costs (cost-effectiveness), Determinants of Coverage, Sustainability/Maintenance
- 25. Implementation Outcomes Outcome ofInterest Definition Data Source Proportion of HIV-positive men who enter care At least one clinic visit attended (where they see a clinician) after positive test result within 3, 6 and 12-months of study visit Clinic records, NHLS documentation of lab tests completed Proportion of men who receive CD4 results Receive POC CD4 results at study visit or get CD4 tests at clinic and return to receive results Study visit CRFs, Clinic records, SMS surveys Proportion of ART-eligible men who initiate ART ART initiated by 12 months Study visit CRFs, Clinic records Time to ART initiation for ART-eligible men Length of time between receiving positive HIV test result and CD4 <350 to initiating ART Study visit CRFs, Clinic records Proportion of ART-eligible men who initiate ART and are retained in care Attend 6-month clinic visit (see a clinician) Attend 12-month clinic visit (see a clinician) Attend 2+ clinic visits at least 3 months after within a 12- month period Clinic records, NHLS documentation lab tests completed (CD4, viral load, others) Proportion of treatment ineligible HIV- infected men who receive a CD4 test within 6 months following their study visit CD4 Receipt of repeat CD4 test Clinic records, NHLS documentation lab tests completed (CD4, viral load, others)
- 26. Implementation Outcomes Outcome ofInterest Data Source Acceptability of outreach/CBO-based testing intervention Questions on study visit CRFs, surveys about why participants chose to test Relative advantages of non-clinic-based ART initiation and retention package compared to standard of care CRFs, questions in surveys, qualitative data from both participants and providers Perceived credibility of CBOs to initiate ART as compared to standard ART clinics Survey indicators, qualitative data Adoption of experimental interventions including intention of use of decentralized NIMART-trained nurse initiated ART and peer navigator based support by participants CRFs, survey indicators, qualitative data Implementation costs associated with experimental condition Review of clinic-based budgets and ultimate costs to assess marginal and total costs of interventions as compared to standard of care Maintenance and routinization of using clinic- based approaches and peer-navigators for retention as indicators to describe potential sustainability of the interventions Provider and participant qualitative data, survey indicators
- 27. Experimental Studies Explanatory(Traditional Gold Standard) Understand and explain benefit of an intervention under controlled conditions Maximize internal validity Pragmatic Trials Focus on the intervention in routine practice Intentional maximization of variability in how study is implemented Variability of research settings (communities, practice settings, types of providers, patients) Maximize external validity Adaptive Designs Emerging area of implementation research that attempts to balance internal and external validity
- 28. Pragmatic Trials Testinga new intervention while maximizing external validity Formative Period Qualitative work, some descriptive or analytic observational work Consider different types of outcomes of effectiveness Directly Measured Health Outcomes Service Outcomes Implementation Outcomes Resources Institutional Human Financial Cost-effectiveness, Cost-Utility, Cost- Minimization, Cost-Benefit, etc Indirect Assessment/Modeled Benefits Increasing use of Mathematical Models to Scale Results for potential longer term outcomes, etc.
- 29. Effectiveness-Implementation Hybrid Trials Goal Measure markers of implementation and impact in the same study Three Broad Designs Differentiated by prioritization of data collection Type 1 1st priority - Impact of health intervention 2nd Priority - gathering measures of implementation Feasibility/Acceptability using qualitative/mixed methods Type 2 Equal priority to impact and implementation Type 3 1st priority – Implementation Fidelity of intervention, measures of adoption 2nd Priority Impact of Health Intervention
- 30. Stepped Wedge ClusterRandomized Designs Method Assess baseline situation in all communities, but randomly phase in intervention activities in steps, evaluating impact of intervention time on outcomes http://www.biomedcentral.com/1471-2288/6/54
- 31. Stepped Wedge Designs Pros Differences in exposure time allow each community site to receive the intervention Ethics Mixed views on the ethics of stepped wedge Some believe more ethical to give intervention to all and more feasible to implement, others believe trial not warranted if success of intervention is certain and standard of care can be justified so why not assess more cleanly with parallel design Cons Analysis concerns around when an intervention starts – e.g. if a community starts receiving the intervention today but takes 2 months to scale up and reach a substantial number of people, exposure time will be diluted as everyone starts receiving the exposure at the same time within the cluster If the outcome cannot be expected to happen over the time period of one step, stepped wedge designs will be underpowered
- 32. Adaptive Designs ForImplementation Studies Adaptive Intervention/Adaptive Implementation Strategy Specific decision rules for the implementation of an interventions based on individual/community needs Trying to maximize both internal and external validity Trial Design Sequential, multiple assignment, randomized trials (SMART) Use outcome data to inform the implementation of the intervention being evaluated Can be at multiple steps http://methodology.psu.edu/ra/adap-inter
- 33. SMART Study Example ResearchQuestion: Among clinics not responding to Replicating effective interventions/REP, how much does external or internal facilitation help improve mood disorders program Source: Kilbourne, Almirall, Implementation Science, 2014 Adaptive Implementation Strategy A priori decisions about intervention based on response Randomize to control for confounders but being done in real world setting Improved balance of Internal/External Validity Measure Implementation Outcomes Throughout
- 34. Power and SampleSize Calculations P&S Calculations for IS studies are complicated Powered to assess the “preponderance of evidence” of the benefit of interventions Most realistic, but murky Powered for at least primary outcome (Eg. Viral suppression) Cleaner, but is this really implementation research? Powered on Outcome and Implementation Outcomes Limited resources, etc.
- 35. De-Implementation Science Studies The science of dissemination and implementation confronts two problems Getting wider uptake of evidence-based interventions in clinical or public health practice Elimination from clinical or public health practice of tests and interventions that use resources without enhancing patient outcomes As a field, we focus more on increasing interventions than we do reducing unnecessary ones More incentive to discover new tools than to try and more politically sensitive to try and remove services for folks De-Implementation Science Methods Use many of the same experimental methods (CRCT, SW, etc) but in reverse
- 36. Conclusions Implementation Research Seeks Generalizable Information intending to Close Gap Between Evidence and Practice Tension Between Internal and External Validity What is most important for you and your stakeholders Rapid Evolution of Experimental Approaches in Implementation Research 36
Editor's Notes
- #13 .