Key Workshop Facilitators: Florence Nyangara, PhD: CSHGP/MCHIP/ICF Kristina Gryboski, PhD: CSHGP/USAID Emily Peca, MA, MPH: TRAction/URC Others from the CSHGP and TRACtion projects will be commenting Jim Sherry, M.D., PhD: TRAction/URC
Before we provide our definitions of OR and IR as applied to our programs, first I will like to know what your definitions of these terms are?
Implementation Science encompasses all aspects of research relevant to the scientific study of methods to promote the uptake of research findings into routine settings in clinical, community and policy contexts Implementation science is the scientific study of methods to promote the integration of research findings and evidence-based interventions into healthcare policy and practice and hence, to improve the quality and effectiveness of health services and care
Unfortunately, there continues to be great variation in how these terms are used depending on who is using them and the purpose. However for the purpose of this workshop this is what we mean:
Before we go into discussions on how these OR studies are done, let us hear about IR by TRACtion
Understanding Why, When, and What it Will Take to do Operations and/or Implementation Research in a Program_ Florence Nyangara
Understanding Why, When, and What it willtake to do Operations and/orImplementation Research in a ProgramExperiences from CSHGP and TRAction ProjectsFlorence Nyangara, PhD, MCHIP/ICFPre-CORE Spring Meeting on OR/IRApril 22nd, 2013
Objectives of this OR/IR Workshop1. Establish a common understanding of terms used inprogram-based research (learning by doing) - OR/IR2. Share knowledge and skills on why and how toidentify & formulate OR/IR research questions, anddecide on research approaches that fit the questions3. Recognize what it will take to undertake OR/IR withinyour program4. Understand major issues surrounding OR/IR withincommunity-based health programs i.e. by NGOs
Caveat! The 4-hour workshop is NOTintended to make you OR/IRexperts! BUT, we hope you will leave herefeeling that OR/IR can be done inyour program
Workshop FormatInteractive and participatory with: Limited presentations just to overviews on taxonomyof OR/IR, and about CSHGP & TRAction projectc Discussions, demonstrations, and illustrative examplesfrom existing implementations (CSHGP/TRAction) Open discussions of common issues, unknowns, andfuture directions of OR/IR types of research
Brainstorming Activity 1 What is Operations Research (OR)? What is Implementation Research (IR)?Instructions: Members of each table take 5 minutes todefine what they understand by OR and IR terms;each group identifies 1 person to present thedefinitions to the audience!!
Terms Used in Program Research – Are You Confused?Monitoring and Evaluation - routineImplementation Research (IR) - FieldImplementation Science (IS)Process EvaluationOperations Research (OR)Impact EvaluationWithin Routine settings: Clinical, community, PolicyOthers e.g., Case studies
Overlap in Terminologies in Public Health Programs’ResearchOperations ResearchOTHER (IS): Health SystemsResearch (HSR), QIImplementationResearchImplementationScience
Another Way of Understanding OR versus IR!Operations Research Implementation ResearchFocus Addresses specificproblems within aprogram by testing afeasible solution (problemmust be what managerscan address (control)More general research inquiryfocusing on "what is happeningand why" in all program aspects-design, implementation,administration, operation, services,& outcomesApproach Uses scientific techniques-wide range - descriptiveto experimental forimpact assessments - i.e.evaluative studiesUses scientific techniques: -------- -- Describes program experiences- Assesses and explains, what/why- No experimental studies, i.e. notdesigned to compare “what ishappening Vs. what would havehappenedPurpose Effectiveness, efficiency, Identify common implementation
Present examples on OR and IR OR – CSHGP – Africare example IR – TRAction
Overview of the Child Survival & Health OR Grants 30 CSHGP/OR studies imbedded within diverse programs implementedby iNGOs including MNCH, Nutrition, family planning, Malaria, etc tosupport national (i.e. MOH) efforts to reduce maternal & childmortality in low resourced countries CSHGP mainly focus at community level health systems, remote areas,and vulnerable populations to address inequities and improve overallMNCH outcomes but links all levels of the HS (nationalcommunity) CSHGP develops new partnerships with local communities, CBOs,research institutions, and national stakeholders to design andimplement responsive programs Strong partnerships between iNGOs, North/South Universities, MOH,several CBOs, and a few donor agencies i.e. UNICEF and WFP10
Key Guiding Principles for Successful OR?1. Contextual relevance (i.e. aligned withMOH policy/strategy/plan)2. Use-focused results (i.e., how results willinform/influence/change)3. Diverse and strategic partnerships4. Stakeholder engagement and learning
Examples of Programming Barriers Identified for CSHGP/OR Inadequate human capacity, i.e. limited availability ofskilled and motivated CHWs Limited accessibility to services i.e.: Physical inaccessibility (i.e. distance to facility) Financial – inability to pay Cultural, etc Poor quality of available services i.e. lack of coordinationof services and continuum of care Poor information and monitoring systems i.e.: Limited availability and use of evidence-based data fordecision-making
Examples of Solutions tested through CSHGP/OR Maternity Waiting Homes (MWH) Incentivize, motivate and retain community-based humancapacity e.g. Community Health Workers (CHW) Integrate service delivery within the health sector and/oracross other relevant sectors (i.e. Agriculture, education) System Integration or coordination for continuum of care Strengthen local community structures and build their capacity(i.e. commitment, resources, skills) Strengthen Community-based Health Information Systems forinformed decision making
Illustrative CSHGP/OR ExampleMaternity Waiting Homes (MWH) byAfricare, in Liberia (2010 Cohort)Key Partners: University of Michigan,MOHSW, County Health Teams
Programming Barrier addressed by Africare/ORProblem: physicalinaccessibility (longdistance, poor roads tofacility) to get Maternal andNewborn services (i.e.Skilled Birth)Policy: MOH’s national policydirective promotes onlyfacility based skilled births(no home delivery!)
Africare, Liberia --------ContinuedSolution: establish Maternity WaitingHomes (MWH) structures near/closeto a health facility for pregnantwomen to stay 2-3 weeks beforedelivery due date and post-delivery ifthey choose to rest/need furtherobservation)Relevance: aligned with MOH’s nationalpolicy directive for only facility basedskilled births. Therefore findings likelyto be used to inform theimplementation of this policyResults: increase facility-based birthsand access to other MNC services andimproved outcomes
OR Steps and Processes: CSHGP/SCALEProblem Analysis inconsultation with keystakeholdersDevelopSolution(Developobjectives &questions)ConductFormativeResearchRefine theIntervention mixi.e. concepts,messages, surveytools, trainingmaterials, etcIncorporate findings toprogram operationsand monitorOngoing Analysis andUse of InformationImplement the studydesign & conductongoing monitoring &documentationEvaluative /Summative PhaseFinal Evaluation of Study- Reflect on Findings withStakeholdersScale-up successfulinterventions
Problem Analysis and Contextualization Identify and define the operational problem(barrier) hindering uptake of interventions Identify subsystems affected by the problem Analyze and narrow the problem (i.e. lit. review,experience, context, observations, etc) Prioritize the specific aspects of the problem forOR that if addressed might bring the mostimpact
Solution/Intervention Development What are the objectives of proposed solution? What is the rationale for selecting this solution? Identify & prioritize specify decision variables i.e.Training methods, amount of ORS packets Specify constraints on the solution i.e. SES Specify the key intervention mix (package) Specify research questions Develop a conceptual model -- important
Describe the OR Interventions in Detail Intervention components must be described indetail for replication or scale-up Without details we are often left knowing verylittle about the details of an intervention or thefunctional relationship between the componentsof the intervention and outcomes
Question?Instructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be included in MWHpackage?1.2.3.
Some ResponseInstructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be included in MWHpackage?1.Structure (building)2.Community outreach3.Training (CHWs, facility staff, etc)
Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deliverPPFP at household levelQ 1.1: What are main barriers to promotionand acceptance of PPFP, and how can theybe addressed?Q 1.2: What is optimal timing and contentof PP home visits by community midwives?Objectives and Research questions for theFormative Research concern the specificintervention content
Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deliverPPFP at household levelQ 1.1: What are main barriers to promotionand acceptance of PPFP, and how can theybe addressed?Q 1.2: What is optimal timing and contentof PP home visits by community midwives?Objective #2: Tomeasure quality andimpact of interventionQ 2.1: Can adequate supply ofcontraceptives be maintained atcommunity level?Q 2.2: What is quality of counseling by CM?Q 2.3: What is effect of intervention onLAM and modern contraceptive use?
Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus groupsQ 1.1: Why do people have wrong beliefs aboutfamily planning?Q 1.2: Why do people think that babies needextra water in addition to breast milk? Too specificQuestions anticipate acertain answerAssumption thatcommunity is wrong “To conduct FR” can’tbe an objective Methods shouldn’t bein the objectives
Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus groupsQ 1.1: Why do people have wrong beliefs aboutfamily planning?Q 1.2: Why do people think that babies needextra water in addition to breast milk?Objective #2: Toprove that theintervention ishighly effectiveQ 2.1: What is the contraceptive prevalence rate?Q 2.2: What is the mean age of women who usecontraceptives?Q 2.3: How much lower is the maternal mortalityrate in the intervention group compared to thecomparison group, from the household survey?Q 2.4: What is the cost-effectiveness ofcontraceptive distribution by midwives?
Objectives and research questions: BADResearch questionsQ 2.1: What is the contraceptiveprevalence rate?Q 2.2: What is the mean age of womenwho use contraceptives?Q 2.3: How much lower is the maternalmortality rate in the intervention groupcompared to the comparison group,from the household survey?Q 2.4: What is the cost-effectiveness ofcontraceptive distribution by midwives?Questions toospecificInsufficientsample size formaternalmortalityImpossible todetermine giventhe methods
Formative Research: Objectives Define the intervention content Design and pre-test specific interventioncomponents: Communication materials Counseling guidelines Packaging/instructions for delivering the services Pilot test entire package on small scale e.g. a fewvillages to refine the intervention
Formative Research: MethodsWomen Men, mothers-in-lawCommunitymidwivesIn-depthinterviews or FGto identifyfacilitators/barriersPretesting ofmaterials andcounselingguidelinesPilot test of PPhome visits
Evaluative Research: Objectives Assess the quality of intervention Measure coverage of intervention Measure study outcomes Implement, Evaluate, and Scale-up the Solution Ongoing evaluation and modifications if needed Scale-up – Integrate the solution with the largersystem
Rigor/Study design: How believable is the evidence?Non-experimental (Case study)• Exploratory to see how to implement the intervention• Ongoing documentation on how bets it can be implemented• No comparison group Quasi-experimental (most common for CSHGP/OR)• Two groups (intervention and comparison)• Non-random selection to groups Experimental• Two groups (intervention and Control)• Random selection of subjects (1 - CSHGP/OR using clusters)
A generic study design diagramSource: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs
A generic diagram: Information to includeWho was yourpopulation? Site andstudy participantsHow did youdecide whoreceived theintervention?What did the interventionand comparison groupsreceive?What methods are youusing to compareoutcomes in interventionand comparisongroups?What analysis willyou conduct tocompareoutcomes inintervention andcomparisongroups?
Reporting results of a study: What happened?Source:http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=53;issue=4;sp
CSHGP OR Study Design: Quasi-Experimental Quasi-experimental (most common for OR)• Two groups (intervention and comparison)• Non-random selection to groups
Challenges of designing a rigorous OR design Difficult for program staff to design research study (limited skill &time) Non-random placement of program participants to - interventionareas and control areas often not comparable Suitable control areas may not exist – other programs in controlareas or cross-over of interventions to control areas Need to control for other factors beyond the program that mightaffect outcomes Interventions are often multi-faceted – causality difficult to judge
Challenges of designing a rigorous OR design Timing – projects are already underway and it ishard to incorporate a strong evaluation design Scale – many projects are too small to expect to beable to demonstrate impact Pressure for rapid results before projects ends Ethical considerations
Cross-cutting issues and concluding thoughtsOpen discussions of common issues, unknowns,and future directions of OR/IR types of research1.What are the common issues?2. What else should we know – unknowns3. Suggestions for Future directions of OR/IR typesof researchInstructions: Divide into 3 groups and choose 1, 2,or 3, discuss among yourselves and withaudience?
Web-based Tools and Resources www.MCHIPNGO.net www.coregroup.org www.MCHIP.net43
Reference Materials on Operations andImplementation Research