Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Implementation Research: A Primer
Stefan Baral, MD, MPH, CCFP, FRCPC (CM)
Overview
 Definitions
 Characteristics
 Conceptual Frameworks
 Evidence and Outcomes
 Methods
 Qualitative
 Quantit...
Definitions
 Implementation
 The use of strategies to introduce or change evidence-based health
interventions (policies,...
Implementation Research and Other PH Study Designs
4Source: Olakunle Alonge, Lobb and Coldtiz, Implementation Science and ...
Characteristics of Implementation Research
5
 Findings are Warranted to Inform Policy/Program
 There is “sufficient evid...
Traditional Scientific Method
6http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml
Differences with IR
Competencies on a IR team:
 Research Methodologist
 Qual, Quant, Mixed Methods
 Ministry, Governmen...
Traditional IR Approaches
8http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml
Active Cons...
IR Specific Objectives: Three Broad Areas
Three Broad Areas of IR Specific Objectives
1. Describe Health Situation or Inte...
Describe Health Situation and Intervention
 Magnitude of the problem
 Distribution of health needs of the population
 R...
Evaluate Interventions
 Coverage of priority health needs
 Coverage of target groups
 Acceptability of the services
 Q...
Analyze Missed Targets
 Availability
 Acceptability
 Affordability
 Service delivery problems
 Fidelity
12
Evidence and Outcomes in
Implementation Research
Conceptual Frameworks
Outcomes
Evidence
Conceptual Frameworks Commonly used in IR
 RE-AIM
 Reach, Efficacy/Effectiveness, Adoption,
Implementation, Maintenance
...
Outcomes in Implementation Research
15
Clients
Outcome
Satisfaction
Symptomatology
Function
Population-
Based
Incidence of...
Implementation Outcomes
16
Implementation
Outcome
Working Definition* Related terms**
Acceptability Perception among stake...
What is Sufficient Evidence?
 Evidence-based medicine is a global standard
 Double-Blinded (DB) RCT is gold standard
 E...
Tension Between Internal and External Validity Challenges for Evidence
Combination Prevention
 Internal Validity
 Minima...
Traditional Research Pathway
 Effectiveness Research (and guideline development)
generally happens prior to implementatio...
Methods in Implementation Research
Research Objectives
 Qualitative Methods
 Exploration
 Using inductive methods to explore concepts, constructs,
phenome...
Qualitative Methods in Implementation Research
 To explore a health problem
 Evidence-based approach to characterizing i...
Quantitative Research Designs
 Observational
 Categories
 Descriptive
 Analytic
 Cross-Sectional
 Prospective
 Quas...
Cohorts In Implementation Research
 Method
 Traditional “Active” Cohort includes
 Enrollment of participants and active...
Implementation Outcomes
Outcome of Interest Definition Data Source
Proportion of HIV-positive men who
enter care
At least ...
Implementation Outcomes
Outcome of Interest Data Source
Acceptability of outreach/CBO-based testing
intervention
Questions...
Experimental Studies
 Explanatory (Traditional Gold Standard)
 Understand and explain benefit of an intervention
under c...
Pragmatic Trials
 Testing a new intervention while maximizing external validity
 Formative Period
 Qualitative work, so...
Effectiveness-Implementation Hybrid Trials
 Goal
 Measure markers of implementation and impact in the same
study
 Three...
Stepped Wedge Cluster Randomized Designs
 Method
 Assess baseline situation in all communities, but randomly phase
in in...
Stepped Wedge Designs
 Pros
 Differences in exposure time allow each community site to
receive the intervention
 Ethics...
Adaptive Designs For Implementation Studies
 Adaptive Intervention/Adaptive Implementation Strategy
 Specific decision r...
SMART Study Example
Research Question: Among clinics not responding to Replicating
effective interventions/REP, how much d...
Power and Sample Size Calculations
 P&S Calculations for IS studies are complicated
 Powered to assess the “preponderanc...
De-Implementation Science Studies
 The science of dissemination and implementation
confronts two problems
 Getting wider...
Conclusions
 Implementation Research
 Seeks Generalizable Information intending to Close
Gap Between Evidence and Practi...
Upcoming SlideShare
Loading in …5
×

Implementation Research: A Primer

1,219 views

Published on

Implementation Research: A Primer

Published in: Education
  • Be the first to comment

Implementation Research: A Primer

  1. 1. Implementation Research: A Primer Stefan Baral, MD, MPH, CCFP, FRCPC (CM)
  2. 2. Overview  Definitions  Characteristics  Conceptual Frameworks  Evidence and Outcomes  Methods  Qualitative  Quantitative  Descriptive  Analytic  Experimental  Conclusions 2
  3. 3. Definitions  Implementation  The use of strategies to introduce or change evidence-based health interventions (policies, programs, individual practices) within specific settings  Implementation Science in HIV  Implementation science is a multi-disciplinary field that seeks generalizable knowledge about the behaviour of stakeholders, organizations, communities, and individuals in order to understand the magnitude, reasons for and strategies to close the gap between evidence and routine practice for health in real world contexts  Key Themes  Multidisciplinary  Generalizable  Multiple stakeholders  Closing gap between evidence and practice  Real world contexts 3Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; Odeny, Padian, Doherty, Baral, Beyrer, Ford, Geng, Definitions of implementation science used in the HIV/AIDS literature: a synthetic review. The Lancet Infectious Diseases, In Press, 2015
  4. 4. Implementation Research and Other PH Study Designs 4Source: Olakunle Alonge, Lobb and Coldtiz, Implementation Science and Its Application to Population Health Annual Review of Public Health, 2013; https://catalyst.harvard.edu/pathfinder/t2detail.html
  5. 5. Characteristics of Implementation Research 5  Findings are Warranted to Inform Policy/Program  There is “sufficient evidence” to support the conclusions of the work  What is sufficient evidence?  Transparency of Methods  Support Critical Assessment of the Study  Whether processes are adequate  Conclusions justified  Repeatability  Don’t be afraid of “failure”  A well done study is still a success in terms of generating generalizable knowledge
  6. 6. Traditional Scientific Method 6http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml
  7. 7. Differences with IR Competencies on a IR team:  Research Methodologist  Qual, Quant, Mixed Methods  Ministry, Government, Agencies  Either as members of team or study oversight committee  Health Professionals  Involvement of health professionals from study settings  Communications  Public Health Professionals  Health Commissioner/Associate Health Commissioner  Public Health Inspector/Public Health Nurse  Privacy Expert  Stakeholder Assessment  Community 7http://www.who.int/tdr/publications/year/2014/ir-toolkit-manual/en/
  8. 8. Traditional IR Approaches 8http://www.sciencebuddies.org/science-fair-projects/project_scientific_method.shtml Active Consultation Active Consultation Appropriate Team, Active Consultation Active Consultation Active Consultation Active Consultation
  9. 9. IR Specific Objectives: Three Broad Areas Three Broad Areas of IR Specific Objectives 1. Describe Health Situation or Interventions 2. Provide Data to Evaluate Ongoing Interventions or Information Needed to Adjust Interventions 3. Analyze missed targets and potential solutions 9
  10. 10. Describe Health Situation and Intervention  Magnitude of the problem  Distribution of health needs of the population  Risk factors for some problems  People’s awareness of the problem  Utilization patterns of services  Cost-effectiveness of available and potential other interventions 10
  11. 11. Evaluate Interventions  Coverage of priority health needs  Coverage of target groups  Acceptability of the services  Quality of services  Cost-effectiveness of the intervention  Impact of the program on health 11
  12. 12. Analyze Missed Targets  Availability  Acceptability  Affordability  Service delivery problems  Fidelity 12
  13. 13. Evidence and Outcomes in Implementation Research Conceptual Frameworks Outcomes Evidence
  14. 14. Conceptual Frameworks Commonly used in IR  RE-AIM  Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance  Stages of implementation  National Implementation Research Network  Exploration and Adoption, Program Installation (Prep), Initial Implementation (pilot/adapt), Full Implementation (>50% coverage), Sustainability  Consolidated Framework for Implementation Research  Intervention Characteristics, Inner Setting, Outer setting, Individuals in the Intervention, Implementation process  Many others…. 14 Source: Glasgow et al 1999, National Implementation Research Network, 2005, Damschroder, 2009
  15. 15. Outcomes in Implementation Research 15 Clients Outcome Satisfaction Symptomatology Function Population- Based Incidence of diseases Morbidity Mortality DALYs Health Outcomes Efficiency Coverage Equity Responsiveness Services Outcomes Acceptability Adoption Appropriateness Costs Feasibility Fidelity Penetration Sustainability Implementation Outcomes Source: Olakunle Alonge, Proctor et al 2011
  16. 16. Implementation Outcomes 16 Implementation Outcome Working Definition* Related terms** Acceptability Perception among stakeholders that an intervention is agreeable Related factors: (e.g. Comfort, Relative advantage, Credibility) Adoption Intention, initial decision, or action to try to employ a new intervention Uptake, Utilization, Intention to try, Appropriateness Perceived fit or relevance of the intervention in a particular setting or for a particular target audience (e.g. provider or consumer) or issue Relevance, Perceived fit, Compatibility, Perceived usefulness or suitability Feasibility The extent to which an intervention can be carried out in a particular setting or organization Practicality, Actual fit, Utility, Trialability Fidelity The degree to which an intervention was implemented as it was designed in an original protocol, plan, or policy Adherence, Delivery as intended, Integrity, Quality of programme delivery, Intensity or dosage of delivery Implementation cost Incremental cost of the implementation strategy Marginal cost, Total cost*** Coverage Degree to which the population that is eligible to benefit from an intervention actually receives it. Reach, Access, Service Spread or Effective Coverage, Penetration Sustainability The extent to which an intervention is maintained or institutionalized in a given setting. Maintenance, Continuation, Routinization Institutionalization, Incorporation Source: Proctor et al 2011; Peters, Adams, Alonge et al 2013
  17. 17. What is Sufficient Evidence?  Evidence-based medicine is a global standard  Double-Blinded (DB) RCT is gold standard  Evidence-based PH interventions should also be a global standard  Often limited evidence, PH decision still needs to be made  Precautionary Principle for PH?  When there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation  To develop guidelines  Need to characterize  Efficacious  Effective  Sustainable and Scalable programs 17
  18. 18. Tension Between Internal and External Validity Challenges for Evidence Combination Prevention  Internal Validity  Minimal study biases suggesting confidence in ultimate conclusion of the study  External Validity  Generalizability of ultimate findings to broader population  Traditional Question for Clinicians/Programmers  Does it work? What is effect size?  Should I use it?  Implementation Questions  How, when, why, and where does it work?  What factors influence effectiveness?  Should I use it? How should I use it?
  19. 19. Traditional Research Pathway  Effectiveness Research (and guideline development) generally happens prior to implementation research  Are there more time-effective approaches to integrate implementation research with effectiveness/efficacy research  Assess barriers/facilitators to intervention uptake  acceptance/adoption/routinization  Diagnose quality gaps  Fidelity  Characterize Sustainability  Maintenance, Cost-Effectiveness
  20. 20. Methods in Implementation Research
  21. 21. Research Objectives  Qualitative Methods  Exploration  Using inductive methods to explore concepts, constructs, phenomena, situations to develop hypotheses  Quantitative Methods  Explanation  Characterizing the  Relationship between concepts and phenomena  Reasons for occurrences of events  Prediction  Use knowledge to forecast events  Experiment  Intervene/manipulate different settings or variables to produce a desired outcome  Mixed Methods  Description  Identify and describing the precursors/antecedents, nature, or etiology of an outcome/phenomena
  22. 22. Qualitative Methods in Implementation Research  To explore a health problem  Evidence-based approach to characterizing individual, network, and structural determinants of the health issue  To identify variables that can later be measured  Measuring the “right” implementation outcomes  To develop a complex and detailed understanding of an issue  Formative work to design the implementation plan for an effective intervention  Maximizing Acceptability, Appropriateness  To understand the contexts or settings surrounding an experience or phenomenon  Case-Study of a program or a policy  To understand the meaning behind an issue or experience  Explore the “why” or “how” an intervention works or why not  Fidelity
  23. 23. Quantitative Research Designs  Observational  Categories  Descriptive  Analytic  Cross-Sectional  Prospective  Quasi-Experimental Studies  no randomization of individuals/communities/institutions  Experimental  Rapid Innovation in Implementation Design
  24. 24. Cohorts In Implementation Research  Method  Traditional “Active” Cohort includes  Enrollment of participants and active follow-up of individuals  Expensive to ensure retention  “Passive” Cohorts (registries, EHR Repositories, etc)  Rapidly becoming the standard in IR, but limitations include quality and extent of data collection  What is Different from Traditional Cohort Studies?  Traditional Cohorts  Incidence, Prevalence, Relationship between Exposure and Outcome  Cohorts in IR focus on measuring traditional IR Outcomes  Acceptability, adoption, appropriateness, feasibility, fidelity of interventions, implementation costs (cost-effectiveness), Determinants of Coverage, Sustainability/Maintenance
  25. 25. Implementation Outcomes Outcome of Interest Definition Data Source Proportion of HIV-positive men who enter care At least one clinic visit attended (where they see a clinician) after positive test result within 3, 6 and 12-months of study visit Clinic records, NHLS documentation of lab tests completed Proportion of men who receive CD4 results Receive POC CD4 results at study visit or get CD4 tests at clinic and return to receive results Study visit CRFs, Clinic records, SMS surveys Proportion of ART-eligible men who initiate ART ART initiated by 12 months Study visit CRFs, Clinic records Time to ART initiation for ART-eligible men Length of time between receiving positive HIV test result and CD4 <350 to initiating ART Study visit CRFs, Clinic records Proportion of ART-eligible men who initiate ART and are retained in care  Attend 6-month clinic visit (see a clinician)  Attend 12-month clinic visit (see a clinician)  Attend 2+ clinic visits at least 3 months after within a 12- month period Clinic records, NHLS documentation lab tests completed (CD4, viral load, others) Proportion of treatment ineligible HIV- infected men who receive a CD4 test within 6 months following their study visit CD4 Receipt of repeat CD4 test Clinic records, NHLS documentation lab tests completed (CD4, viral load, others)
  26. 26. Implementation Outcomes Outcome of Interest Data Source Acceptability of outreach/CBO-based testing intervention Questions on study visit CRFs, surveys about why participants chose to test Relative advantages of non-clinic-based ART initiation and retention package compared to standard of care CRFs, questions in surveys, qualitative data from both participants and providers Perceived credibility of CBOs to initiate ART as compared to standard ART clinics Survey indicators, qualitative data Adoption of experimental interventions including intention of use of decentralized NIMART-trained nurse initiated ART and peer navigator based support by participants CRFs, survey indicators, qualitative data Implementation costs associated with experimental condition Review of clinic-based budgets and ultimate costs to assess marginal and total costs of interventions as compared to standard of care Maintenance and routinization of using clinic- based approaches and peer-navigators for retention as indicators to describe potential sustainability of the interventions Provider and participant qualitative data, survey indicators
  27. 27. Experimental Studies  Explanatory (Traditional Gold Standard)  Understand and explain benefit of an intervention under controlled conditions  Maximize internal validity  Pragmatic Trials  Focus on the intervention in routine practice  Intentional maximization of variability in how study is implemented  Variability of research settings (communities, practice settings, types of providers, patients)  Maximize external validity  Adaptive Designs  Emerging area of implementation research that attempts to balance internal and external validity
  28. 28. Pragmatic Trials  Testing a new intervention while maximizing external validity  Formative Period  Qualitative work, some descriptive or analytic observational work  Consider different types of outcomes of effectiveness  Directly Measured  Health Outcomes  Service Outcomes  Implementation Outcomes  Resources  Institutional  Human  Financial  Cost-effectiveness, Cost-Utility, Cost- Minimization, Cost-Benefit, etc  Indirect Assessment/Modeled Benefits  Increasing use of Mathematical Models to Scale Results for potential longer term outcomes, etc.
  29. 29. Effectiveness-Implementation Hybrid Trials  Goal  Measure markers of implementation and impact in the same study  Three Broad Designs  Differentiated by prioritization of data collection  Type 1  1st priority - Impact of health intervention  2nd Priority - gathering measures of implementation  Feasibility/Acceptability using qualitative/mixed methods  Type 2  Equal priority to impact and implementation  Type 3  1st priority – Implementation  Fidelity of intervention, measures of adoption  2nd Priority  Impact of Health Intervention
  30. 30. Stepped Wedge Cluster Randomized Designs  Method  Assess baseline situation in all communities, but randomly phase in intervention activities in steps, evaluating impact of intervention time on outcomes http://www.biomedcentral.com/1471-2288/6/54
  31. 31. Stepped Wedge Designs  Pros  Differences in exposure time allow each community site to receive the intervention  Ethics  Mixed views on the ethics of stepped wedge  Some believe more ethical to give intervention to all and more feasible to implement, others believe trial not warranted if success of intervention is certain and standard of care can be justified so why not assess more cleanly with parallel design  Cons  Analysis concerns around when an intervention starts – e.g. if a community starts receiving the intervention today but takes 2 months to scale up and reach a substantial number of people, exposure time will be diluted as everyone starts receiving the exposure at the same time within the cluster  If the outcome cannot be expected to happen over the time period of one step, stepped wedge designs will be underpowered
  32. 32. Adaptive Designs For Implementation Studies  Adaptive Intervention/Adaptive Implementation Strategy  Specific decision rules for the implementation of an interventions based on individual/community needs  Trying to maximize both internal and external validity  Trial Design  Sequential, multiple assignment, randomized trials (SMART)  Use outcome data to inform the implementation of the intervention being evaluated  Can be at multiple steps http://methodology.psu.edu/ra/adap-inter
  33. 33. SMART Study Example Research Question: Among clinics not responding to Replicating effective interventions/REP, how much does external or internal facilitation help improve mood disorders program Source: Kilbourne, Almirall, Implementation Science, 2014  Adaptive Implementation Strategy  A priori decisions about intervention based on response  Randomize to control for confounders but being done in real world setting  Improved balance of Internal/External Validity  Measure Implementation Outcomes Throughout
  34. 34. Power and Sample Size Calculations  P&S Calculations for IS studies are complicated  Powered to assess the “preponderance of evidence” of the benefit of interventions  Most realistic, but murky  Powered for at least primary outcome (Eg. Viral suppression)  Cleaner, but is this really implementation research?  Powered on Outcome and Implementation Outcomes  Limited resources, etc.
  35. 35. De-Implementation Science Studies  The science of dissemination and implementation confronts two problems  Getting wider uptake of evidence-based interventions in clinical or public health practice  Elimination from clinical or public health practice of tests and interventions that use resources without enhancing patient outcomes  As a field, we focus more on increasing interventions than we do reducing unnecessary ones  More incentive to discover new tools than to try and more politically sensitive to try and remove services for folks  De-Implementation Science Methods  Use many of the same experimental methods (CRCT, SW, etc) but in reverse
  36. 36. Conclusions  Implementation Research  Seeks Generalizable Information intending to Close Gap Between Evidence and Practice  Tension Between Internal and External Validity  What is most important for you and your stakeholders  Rapid Evolution of Experimental Approaches in Implementation Research 36

×