Now, we will walk through two examples of program planning, monitoring and evaluation. As we present the examples, we will highlight the resources that are available within the toolkit.
The first example involves evaluating the impact of a family planning program in a rural village in West Africa.The goal is to increase knowledge of contraceptives, fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity) and modern contraceptive use.
We follow the sequence of questions identical to the ones described in the Toolkit. These questions describe the steps that are used when conducting an evaluation plan.For our particular question of interest, if there is a recent demographic and health survey from the country, data from the rural area could show that there is low contraceptive prevalence, knowledge, and attitudes (as well as limited exposure to media and other potential intervention strategies)The Toolkit outlines examples of other potential population based surveys and their strengths and limitations.For our example, Data from a pre-existing formative research would also show that there is low prevalence of contraceptive use.
OR you could conduct a needs assessment in the village of interest by surveying village leaders and community members.In this example, the results would show-A greater need for access to a large range of modern contraceptive methodsThe toolkit describes how to conduct a needs assessment for your question of interest
Design a community outreach program where health workers visit homes and address the contraceptive needs of familiesBefore carrying out the full-fledged intervention, the intervention is carried out in a neighborhood of the village as a pilot project. The lessons learned by getting feedback from the health workers and community members is used to improve upon the project design and is expanded to the entire village.
In order to track our activities, we can develop and use several tools.A logic model will outline all the steps between inputs and outcomes.The toolkit describes logic models and provides examples as well.We can also create an M&E plan that is helpful as it includes a timeline of program activities.The Toolkit also gives a good outline of an M&E plan and also has some program examples.Furthermore, we could produce a PMP which is useful as this document can be shared with donors and other stakeholders to give them an update on proposed activities. It also has a list of indicators that will be collected and analyzed.The toolkit has a great compendium of good FP indicators.In all these steps and processes, it is important to engage stakeholders in every step of the program planning and implementation as well as monitoring and evaluation.
Once our program activities are underway, we monitor the various components of the program.We monitor the use of inputs such as Finances, staff, training materials, contraceptives and transportationSome indicators that can be used to monitor our FP program activities includeThe number of health workers trained to work in the communityThe number of health workers who actually visit the community periodically and distribute contraceptivesThe frequency of meetings between program officers and village leadersMonitoring of inputs and processes ensures that the quality of the program activities are maintained.
In order to determine if we are implementing the program as planned, we collect indicators of output and outcome monitoring.Indicators of output monitoring for our FP program of interest include:Number of first visits madeNumber of follow-up visitsTypes and numbers of contraceptives distributedAn indicator for outcome monitoring is change in percentage of contraceptive users over time.The toolkit has some excellent indicator guides for FP programs.Indicators: fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity)
In order to determine if the FP intervention we conducted actually made a difference in contraceptive use, we conduct an outcome evaluation. One possible study design we use is the quasi-experimental design where we choose a control village early on. The villagers in this control village have similar sociodemographic characteristics than those in the intervention village. We collect data on contraceptive use, knowledge and attitudes before the intervention and after the intervention from both villages and determine if contraceptive use significantly went up in the intervention village. The toolkit also has examples of other evaluation designs and resources.Impact evaluationShould be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
Once the evaluation results show that our FP program was successful, we can make the FP program sustainable by ensuring that the local government continues the community outreach program. With the aid of the same health workers, we can also build the community’s capacity to continue accessing voluntary family planning at local pharmacies and health care centers.Furthermore, the program can be scaled up and expanded to other villages and regions within the country.The toolkit describes how to make programs sustainable and scalable and provides several successful examples as well.
Once the program has been implemented and evaluated, it is important to share the findings and best practices with all the stakeholders that were involved including the village leaders and community members of the village, the local government and health department, the funding agency and other higher levels of government and health leadership. The findings can also be shared with others through mass media, research literature and the internet.
The second example we will discuss today involves the evaluation of a malaria prevention program in a district in East Africa.The main goal of the program is to ensure that all pregnant women and children <14 years are sleeping under ITNs in all the villages within a district in East Africa.
In order to determine the extent of the malaria problem in this district, we review recent population based surveys such as the DHS and MICS. The results will show that there is a high prevalence of malaria and low use of ITNs.We can also conduct a needs assessment by selecting a few homes in a village and interviewing some community members within the selected homes. The results of this needs assessment would show that there is a limited use of ITNs due to lack of knowledge and high cost of ITNs.The toolkit outlines examples of other potential data sources that can be used to identify a health problem within a specified region and also states the strengths and limitations of each of the data sources.
In order to determine what interventions could possibly work in this situation, we review other programs, collaborate with more experienced programs and staff, even maybe plan a pilot project.The toolkit has some great resources to learn about existing successful malaria interventions, such as on the Roll Back Malaria website referenced in our toolkit.In this example, we could design an intervention in our district of interest. With the help of handheld GPS devices, we create clusters of 200 households within the district. We visit all homes, identify pregnant women and children <14 years and provide them with vouchers for ITNs. Then, we have program staff do follow-up visits to ensure that the ITN is installed correctly and teach villagers on how to reapply the insecticide.
Once we have decided on the intervention we will roll out, we develop a logic model that outlines all the steps between inputs and outcomes. We can also develop a PMP to help guide the activities.It is important to engage stakeholders at every step of the program. These stakeholders includeCommunity members and village leadersLocal government and district officialsExperts in the fieldDonors and policy makersThe toolkit describes a logic model and has a checklist of the important components within a PMP plan for malaria program prevention.
In order to determine whether we are implementing the program as planned, we monitor the various components of the program. This also ensure that the quality of the program is maintained.The toolkit has several resources for monitoring malaria program, such as Indicator guides for malaria programs, References on how to conduct routine monitoring as well as impact evaluation.
In order to conduct an outcome evaluation for our malaria program, we can use a time series design. Generally, Time-series designs look for changes over time to determine trends. Evaluators observe the intervention group multiple times before and after the intervention and analyze trends before and after.Impact evaluationCan be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
Once the program has been rolled out successfully, it is important to ensure that the program remains sustainable. This can be done by ensuring that the local government continues the voucher program. Also, the community members are empowered to access ITNs and use them effectively.The program can also be scaled up to other districts and regions within the country thus benefiting a larger audience.The toolkit has descriptions and examples of how programs can be sustained and scaled up to new areas.
Thank you for joining us today. We’ve appreciated your participation. We are about to begin a Q&A session for this webinar but I want to let you know about a 3-day online forum that is has begun today. It will1. build capacity in monitoring and evaluation 2. advocate the importance of investing in monitoring and evaluation (M&E)3. and provide a more in-depth overview of MLE’s Measuring Success ToolkitOn your screen, please see the instructions to join the forum. Please join us in the Q&A being moderated by Gretchen.
Two Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning,Monitoring and Evaluation Laili Irani
Example 1: Evaluate a Family Planning Program Objective: – To evaluate the impact of a family planning program in a rural village in West Africa Main goal: – To increase contraceptive knowledge, fertility preferences and contraceptive use
Q.1) What is the problem and why does it exist? Study recent DHS for evidence on rural area of interest – Results show: low contraceptive prevalence, knowledge and attitudes See Toolkit for examples of population based surveys Review data from formative research – Results shows: low prevalence of contraceptive use
Q.1) What is the problem and why does it exist? (cont.) Conduct a needs assessment – Survey village leaders and community members – In this example, results show: • Greater need for modern contraceptive methods • Easier access to wider range of methods See Toolkit on how to conduct a needs assessment
Q. 2) What interventions can work? Design an intervention – A community outreach program in which health workers visit homes and address the contraceptive needs of families Plan a pilot project – Carry out the project in one neighborhood of the village – Expand to the entire village
Q. 3) What are we doing? Develop a logic model – Inputs outcomes See Toolkit for description and examples of logic models Create an M&E plan – Include a timeline of program activities See Toolkit for sample outline & program examples Produce a Performance Monitoring Plan (PMP) – Share proposed activities with stakeholders and donors – Identify indicators to be collected and analyzed See Toolkit for compendium of FP indictors Engage stakeholders in every step of the program
Q. 3) What are we doing? (cont.) Monitor the various components of the program – Inputs • Finances, staff, training materials, contraceptives and transportation – Processes • Health workers trained to work in the community • Health workers visit community periodically and distribute contraceptives • Program officers meet with village leaders often Ensure quality of program is maintained
Q. 4) Are we implementing the program as planned? Output monitoring – Number of first visits made – Number of follow-up visits – Types and numbers of contraceptives distributed Outcome monitoring – Change in percentage of contraceptive users over time See Toolkit for indicator guides for FP programs
Q. 5) Are the interventions working / making a difference? Outcome evaluation See Toolkit for examples of evaluation designs and resources
Q. 6) Is the program sustainable and scalable? Sustainability – Ensure local government continues the community outreach program with the aid of health workers – Build the community’s capacity to encourage voluntary contraceptive use among families Scalability – Expand the program to other villages and regions within the country See Toolkit for means to measure sustainability and scalability
Next Steps Share findings with all the stakeholders, including – Village leaders and community members – Local government and health department – Funding agency – Higher levels of government and health leadership Disseminate findings widely including through mass media, research literature and the internet
Example 2: Evaluate a Malaria Prevention Program Objective: – To evaluate the impact of a malaria prevention program in a district in East Africa Main goal: – To ensure all pregnant women and children <14 years are sleeping under insecticide treated nets (ITNs) in all the villages of a district in East Africa
Q.1) What is the problem & why does it exist? Study recent population based surveys – DHS, Multiple Indicator Cluster Survey – Results show: High malaria prevalence and low ITNs use Conduct a needs assessment – Visit selected homes; interview selected community members – In this example, results show: Use of ITNs is low due to lack of knowledge and cost See Toolkit for various data sources and assessment designs
Q. 2) What interventions can work? Review other programs Collaborate with more experienced programs Plan a pilot project Design an intervention – Use handheld GPS devices to create clusters – Visit all homes, identify pregnant women and children <14 years and provide them with vouchers for ITN – Program staff visit to ensure ITN installed correctly and teach villagers how to reapply insecticide See Toolkit for resources, i.e., Roll Back Malaria website
Q. 3) What are we doing? Develop a logic model – Inputs outcomes Develop a PMP Engage stakeholders in every step of the program – Community members and village leaders – Local government and district officials – Experts in the field – Donors and policy makers See Toolkit for a logic model and draft checklist for developing PMP plan for malaria program
Q. 4) Are we implementing the program as planned? Monitor the various components of the program – Ensure quality of program is maintained See Toolkit for: – Indicator guides for malaria programs – References on how to conduct routine monitoring – Impact evaluation references
Q. 5) Are the interventions working / making a difference? Outcome evaluation See Toolkit for alternative study designs
Q. 6) Is the program sustainable and scalable? Sustainability – Ensure local government continues the voucher program – Empower community leaders to encourage community to access ITNs and to use them effectively Scalability – Expand the program to other districts and regions within the country See Toolkit for descriptions and examples of how programs can be sustainable and scalable