Understanding Why, When, and What it willtake to do Operations and/orImplementation Research in a ProgramExperiences from ...
Objectives of this OR/IR Workshop1. Establish a common understanding of terms used inprogram-based research (learning by d...
Caveat! The 4-hour workshop is NOTintended to make you OR/IRexperts! BUT, we hope you will leave herefeeling that OR/IR ...
Workshop FormatInteractive and participatory with: Limited presentations just to overviews on taxonomyof OR/IR, and about...
Brainstorming Activity 1 What is Operations Research (OR)? What is Implementation Research (IR)?Instructions: Members of...
Terms Used in Program Research – Are You Confused?Monitoring and Evaluation - routineImplementation Research (IR) - FieldI...
Overlap in Terminologies in Public Health Programs’ResearchOperations ResearchOTHER (IS): Health SystemsResearch (HSR), QI...
Another Way of Understanding OR versus IR!Operations Research Implementation ResearchFocus Addresses specificproblems with...
Present examples on OR and IR OR – CSHGP – Africare example IR – TRAction
Overview of the Child Survival & Health OR Grants 30 CSHGP/OR studies imbedded within diverse programs implementedby iNGO...
Key Guiding Principles for Successful OR?1. Contextual relevance (i.e. aligned withMOH policy/strategy/plan)2. Use-focused...
Examples of Programming Barriers Identified for CSHGP/OR Inadequate human capacity, i.e. limited availability ofskilled a...
Examples of Solutions tested through CSHGP/OR Maternity Waiting Homes (MWH) Incentivize, motivate and retain community-b...
Illustrative CSHGP/OR ExampleMaternity Waiting Homes (MWH) byAfricare, in Liberia (2010 Cohort)Key Partners: University of...
Programming Barrier addressed by Africare/ORProblem: physicalinaccessibility (longdistance, poor roads tofacility) to get ...
Africare, Liberia --------ContinuedSolution: establish Maternity WaitingHomes (MWH) structures near/closeto a health facil...
IR by TRAction Project
How OR is done?
OR Steps and Processes: CSHGP/SCALEProblem Analysis inconsultation with keystakeholdersDevelopSolution(Developobjectives &...
Problem Analysis and Contextualization Identify and define the operational problem(barrier) hindering uptake of intervent...
Solution/Intervention Development What are the objectives of proposed solution? What is the rationale for selecting this...
Describe the OR Interventions in Detail Intervention components must be described indetail for replication or scale-up W...
Question?Instructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be inclu...
Some ResponseInstructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be i...
Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deli...
Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deli...
Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus gr...
Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus gr...
Objectives and research questions: BADResearch questionsQ 2.1: What is the contraceptiveprevalence rate?Q 2.2: What is the...
Formative Research: Objectives Define the intervention content Design and pre-test specific interventioncomponents: Com...
Formative Research: MethodsWomen Men, mothers-in-lawCommunitymidwivesIn-depthinterviews or FGto identifyfacilitators/barri...
Evaluative Research: Objectives Assess the quality of intervention Measure coverage of intervention Measure study outco...
Rigor/Study design: How believable is the evidence?Non-experimental (Case study)• Exploratory to see how to implement the ...
A generic study design diagramSource: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs
A generic diagram: Information to includeWho was yourpopulation? Site andstudy participantsHow did youdecide whoreceived t...
Reporting results of a study: What happened?Source:http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=5...
Reporting results of a study: What happened?
CSHGP OR Study Design: Quasi-Experimental Quasi-experimental (most common for OR)• Two groups (intervention and compariso...
Challenges of designing a rigorous OR design Difficult for program staff to design research study (limited skill &time) ...
Challenges of designing a rigorous OR design Timing – projects are already underway and it ishard to incorporate a strong...
Cross-cutting issues and concluding thoughtsOpen discussions of common issues, unknowns,and future directions of OR/IR typ...
Web-based Tools and Resources www.MCHIPNGO.net www.coregroup.org www.MCHIP.net43
Reference Materials on Operations andImplementation Research
Understanding Why, When, and What it Will Take to do Operations and/or Implementation Research in a Program_ Florence Nyan...
Upcoming SlideShare
Loading in …5
×

Understanding Why, When, and What it Will Take to do Operations and/or Implementation Research in a Program_ Florence Nyangara

631 views

Published on

Published in: Technology, Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
631
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Key Workshop Facilitators: Florence Nyangara, PhD: CSHGP/MCHIP/ICF Kristina Gryboski, PhD: CSHGP/USAID Emily Peca, MA, MPH: TRAction/URC Others from the CSHGP and TRACtion projects will be commenting Jim Sherry, M.D., PhD: TRAction/URC
  • Before we provide our definitions of OR and IR as applied to our programs, first I will like to know what your definitions of these terms are?
  • Implementation Science encompasses all aspects of research relevant to the scientific study of methods to promote the uptake of research findings into routine settings in clinical, community and policy contexts Implementation science is the scientific study of methods to promote the integration of research findings and evidence-based interventions into healthcare policy and practice and hence, to improve the quality and effectiveness of health services and care
  • Unfortunately, there continues to be great variation in how these terms are used depending on who is using them and the purpose. However for the purpose of this workshop this is what we mean:
  • Before we go into discussions on how these OR studies are done, let us hear about IR by TRACtion
  • Understanding Why, When, and What it Will Take to do Operations and/or Implementation Research in a Program_ Florence Nyangara

    1. 1. Understanding Why, When, and What it willtake to do Operations and/orImplementation Research in a ProgramExperiences from CSHGP and TRAction ProjectsFlorence Nyangara, PhD, MCHIP/ICFPre-CORE Spring Meeting on OR/IRApril 22nd, 2013
    2. 2. Objectives of this OR/IR Workshop1. Establish a common understanding of terms used inprogram-based research (learning by doing) - OR/IR2. Share knowledge and skills on why and how toidentify & formulate OR/IR research questions, anddecide on research approaches that fit the questions3. Recognize what it will take to undertake OR/IR withinyour program4. Understand major issues surrounding OR/IR withincommunity-based health programs i.e. by NGOs
    3. 3. Caveat! The 4-hour workshop is NOTintended to make you OR/IRexperts! BUT, we hope you will leave herefeeling that OR/IR can be done inyour program
    4. 4. Workshop FormatInteractive and participatory with: Limited presentations just to overviews on taxonomyof OR/IR, and about CSHGP & TRAction projectc Discussions, demonstrations, and illustrative examplesfrom existing implementations (CSHGP/TRAction) Open discussions of common issues, unknowns, andfuture directions of OR/IR types of research
    5. 5. Brainstorming Activity 1 What is Operations Research (OR)? What is Implementation Research (IR)?Instructions: Members of each table take 5 minutes todefine what they understand by OR and IR terms;each group identifies 1 person to present thedefinitions to the audience!!
    6. 6. Terms Used in Program Research – Are You Confused?Monitoring and Evaluation - routineImplementation Research (IR) - FieldImplementation Science (IS)Process EvaluationOperations Research (OR)Impact EvaluationWithin Routine settings: Clinical, community, PolicyOthers e.g., Case studies
    7. 7. Overlap in Terminologies in Public Health Programs’ResearchOperations ResearchOTHER (IS): Health SystemsResearch (HSR), QIImplementationResearchImplementationScience
    8. 8. Another Way of Understanding OR versus IR!Operations Research Implementation ResearchFocus Addresses specificproblems within aprogram by testing afeasible solution (problemmust be what managerscan address (control)More general research inquiryfocusing on "what is happeningand why" in all program aspects-design, implementation,administration, operation, services,& outcomesApproach Uses scientific techniques-wide range - descriptiveto experimental forimpact assessments - i.e.evaluative studiesUses scientific techniques: -------- -- Describes program experiences- Assesses and explains, what/why- No experimental studies, i.e. notdesigned to compare “what ishappening Vs. what would havehappenedPurpose Effectiveness, efficiency, Identify common implementation
    9. 9. Present examples on OR and IR OR – CSHGP – Africare example IR – TRAction
    10. 10. Overview of the Child Survival & Health OR Grants 30 CSHGP/OR studies imbedded within diverse programs implementedby iNGOs including MNCH, Nutrition, family planning, Malaria, etc tosupport national (i.e. MOH) efforts to reduce maternal & childmortality in low resourced countries CSHGP mainly focus at community level health systems, remote areas,and vulnerable populations to address inequities and improve overallMNCH outcomes but links all levels of the HS (nationalcommunity) CSHGP develops new partnerships with local communities, CBOs,research institutions, and national stakeholders to design andimplement responsive programs Strong partnerships between iNGOs, North/South Universities, MOH,several CBOs, and a few donor agencies i.e. UNICEF and WFP10
    11. 11. Key Guiding Principles for Successful OR?1. Contextual relevance (i.e. aligned withMOH policy/strategy/plan)2. Use-focused results (i.e., how results willinform/influence/change)3. Diverse and strategic partnerships4. Stakeholder engagement and learning
    12. 12. Examples of Programming Barriers Identified for CSHGP/OR Inadequate human capacity, i.e. limited availability ofskilled and motivated CHWs Limited accessibility to services i.e.: Physical inaccessibility (i.e. distance to facility) Financial – inability to pay Cultural, etc Poor quality of available services i.e. lack of coordinationof services and continuum of care Poor information and monitoring systems i.e.: Limited availability and use of evidence-based data fordecision-making
    13. 13. Examples of Solutions tested through CSHGP/OR Maternity Waiting Homes (MWH) Incentivize, motivate and retain community-based humancapacity e.g. Community Health Workers (CHW) Integrate service delivery within the health sector and/oracross other relevant sectors (i.e. Agriculture, education) System Integration or coordination for continuum of care Strengthen local community structures and build their capacity(i.e. commitment, resources, skills) Strengthen Community-based Health Information Systems forinformed decision making
    14. 14. Illustrative CSHGP/OR ExampleMaternity Waiting Homes (MWH) byAfricare, in Liberia (2010 Cohort)Key Partners: University of Michigan,MOHSW, County Health Teams
    15. 15. Programming Barrier addressed by Africare/ORProblem: physicalinaccessibility (longdistance, poor roads tofacility) to get Maternal andNewborn services (i.e.Skilled Birth)Policy: MOH’s national policydirective promotes onlyfacility based skilled births(no home delivery!)
    16. 16. Africare, Liberia --------ContinuedSolution: establish Maternity WaitingHomes (MWH) structures near/closeto a health facility for pregnantwomen to stay 2-3 weeks beforedelivery due date and post-delivery ifthey choose to rest/need furtherobservation)Relevance: aligned with MOH’s nationalpolicy directive for only facility basedskilled births. Therefore findings likelyto be used to inform theimplementation of this policyResults: increase facility-based birthsand access to other MNC services andimproved outcomes
    17. 17. IR by TRAction Project
    18. 18. How OR is done?
    19. 19. OR Steps and Processes: CSHGP/SCALEProblem Analysis inconsultation with keystakeholdersDevelopSolution(Developobjectives &questions)ConductFormativeResearchRefine theIntervention mixi.e. concepts,messages, surveytools, trainingmaterials, etcIncorporate findings toprogram operationsand monitorOngoing Analysis andUse of InformationImplement the studydesign & conductongoing monitoring &documentationEvaluative /Summative PhaseFinal Evaluation of Study- Reflect on Findings withStakeholdersScale-up successfulinterventions
    20. 20. Problem Analysis and Contextualization Identify and define the operational problem(barrier) hindering uptake of interventions Identify subsystems affected by the problem Analyze and narrow the problem (i.e. lit. review,experience, context, observations, etc) Prioritize the specific aspects of the problem forOR that if addressed might bring the mostimpact
    21. 21. Solution/Intervention Development What are the objectives of proposed solution? What is the rationale for selecting this solution? Identify & prioritize specify decision variables i.e.Training methods, amount of ORS packets Specify constraints on the solution i.e. SES Specify the key intervention mix (package) Specify research questions Develop a conceptual model -- important
    22. 22. Describe the OR Interventions in Detail Intervention components must be described indetail for replication or scale-up Without details we are often left knowing verylittle about the details of an intervention or thefunctional relationship between the componentsof the intervention and outcomes
    23. 23. Question?Instructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be included in MWHpackage?1.2.3.
    24. 24. Some ResponseInstructions: Using the MWH intervention byAFRicare as an example, please list 3 keycomponents that must be included in MWHpackage?1.Structure (building)2.Community outreach3.Training (CHWs, facility staff, etc)
    25. 25. Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deliverPPFP at household levelQ 1.1: What are main barriers to promotionand acceptance of PPFP, and how can theybe addressed?Q 1.2: What is optimal timing and contentof PP home visits by community midwives?Objectives and Research questions for theFormative Research concern the specificintervention content
    26. 26. Objectives and research questions: GOODObjectives Research questionsObjective #1: Todevelop content ofintervention to deliverPPFP at household levelQ 1.1: What are main barriers to promotionand acceptance of PPFP, and how can theybe addressed?Q 1.2: What is optimal timing and contentof PP home visits by community midwives?Objective #2: Tomeasure quality andimpact of interventionQ 2.1: Can adequate supply ofcontraceptives be maintained atcommunity level?Q 2.2: What is quality of counseling by CM?Q 2.3: What is effect of intervention onLAM and modern contraceptive use?
    27. 27. Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus groupsQ 1.1: Why do people have wrong beliefs aboutfamily planning?Q 1.2: Why do people think that babies needextra water in addition to breast milk? Too specificQuestions anticipate acertain answerAssumption thatcommunity is wrong “To conduct FR” can’tbe an objective Methods shouldn’t bein the objectives
    28. 28. Objectives and research questions: BADObjectives Research questionsObjective #1: Toconduct formativeresearch usingfocus groupsQ 1.1: Why do people have wrong beliefs aboutfamily planning?Q 1.2: Why do people think that babies needextra water in addition to breast milk?Objective #2: Toprove that theintervention ishighly effectiveQ 2.1: What is the contraceptive prevalence rate?Q 2.2: What is the mean age of women who usecontraceptives?Q 2.3: How much lower is the maternal mortalityrate in the intervention group compared to thecomparison group, from the household survey?Q 2.4: What is the cost-effectiveness ofcontraceptive distribution by midwives?
    29. 29. Objectives and research questions: BADResearch questionsQ 2.1: What is the contraceptiveprevalence rate?Q 2.2: What is the mean age of womenwho use contraceptives?Q 2.3: How much lower is the maternalmortality rate in the intervention groupcompared to the comparison group,from the household survey?Q 2.4: What is the cost-effectiveness ofcontraceptive distribution by midwives?Questions toospecificInsufficientsample size formaternalmortalityImpossible todetermine giventhe methods
    30. 30. Formative Research: Objectives Define the intervention content Design and pre-test specific interventioncomponents: Communication materials Counseling guidelines Packaging/instructions for delivering the services Pilot test entire package on small scale e.g. a fewvillages to refine the intervention
    31. 31. Formative Research: MethodsWomen Men, mothers-in-lawCommunitymidwivesIn-depthinterviews or FGto identifyfacilitators/barriersPretesting ofmaterials andcounselingguidelinesPilot test of PPhome visits
    32. 32. Evaluative Research: Objectives Assess the quality of intervention Measure coverage of intervention Measure study outcomes Implement, Evaluate, and Scale-up the Solution Ongoing evaluation and modifications if needed Scale-up – Integrate the solution with the largersystem
    33. 33. Rigor/Study design: How believable is the evidence?Non-experimental (Case study)• Exploratory to see how to implement the intervention• Ongoing documentation on how bets it can be implemented• No comparison group Quasi-experimental (most common for CSHGP/OR)• Two groups (intervention and comparison)• Non-random selection to groups Experimental• Two groups (intervention and Control)• Random selection of subjects (1 - CSHGP/OR using clusters)
    34. 34. A generic study design diagramSource: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs
    35. 35. A generic diagram: Information to includeWho was yourpopulation? Site andstudy participantsHow did youdecide whoreceived theintervention?What did the interventionand comparison groupsreceive?What methods are youusing to compareoutcomes in interventionand comparisongroups?What analysis willyou conduct tocompareoutcomes inintervention andcomparisongroups?
    36. 36. Reporting results of a study: What happened?Source:http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=53;issue=4;sp
    37. 37. Reporting results of a study: What happened?
    38. 38. CSHGP OR Study Design: Quasi-Experimental Quasi-experimental (most common for OR)• Two groups (intervention and comparison)• Non-random selection to groups
    39. 39. Challenges of designing a rigorous OR design Difficult for program staff to design research study (limited skill &time) Non-random placement of program participants to - interventionareas and control areas often not comparable Suitable control areas may not exist – other programs in controlareas or cross-over of interventions to control areas Need to control for other factors beyond the program that mightaffect outcomes Interventions are often multi-faceted – causality difficult to judge
    40. 40. Challenges of designing a rigorous OR design Timing – projects are already underway and it ishard to incorporate a strong evaluation design Scale – many projects are too small to expect to beable to demonstrate impact Pressure for rapid results before projects ends Ethical considerations
    41. 41. Cross-cutting issues and concluding thoughtsOpen discussions of common issues, unknowns,and future directions of OR/IR types of research1.What are the common issues?2. What else should we know – unknowns3. Suggestions for Future directions of OR/IR typesof researchInstructions: Divide into 3 groups and choose 1, 2,or 3, discuss among yourselves and withaudience?
    42. 42. Web-based Tools and Resources www.MCHIPNGO.net www.coregroup.org www.MCHIP.net43
    43. 43. Reference Materials on Operations andImplementation Research

    ×