MEASURE Evaluation works to improve collection, analysis and presentation of data to promote better use of data in planning, policymaking, managing, monitoring and evaluating population, health and nutrition programs.
Sep. 26, 2012•0 likes•70,404 views
1 of 19
Two Examples of Program Planning, Monitoring and Evaluation
Sep. 26, 2012•0 likes•70,404 views
Download to read offline
Report
Technology
Health & Medicine
Presented by Laili Irani, Senior Policy Analyst for the Population Reference Bureau, as part of the Measuring Success Toolkit webinar in September 2012.
MEASURE Evaluation works to improve collection, analysis and presentation of data to promote better use of data in planning, policymaking, managing, monitoring and evaluating population, health and nutrition programs.
Two Examples of Program Planning, Monitoring and Evaluation
1. Two Examples of
Program Planning,
Monitoring and Evaluation
Laili Irani
2. Example 1: Evaluate a Family Planning
Program
Objective:
– To evaluate the impact of a family planning program
in a rural village in West Africa
Main goal:
– To increase contraceptive knowledge, fertility
preferences and contraceptive use
3. Q.1) What is the problem and why does it
exist?
Study recent DHS for evidence on rural area
of interest
– Results show: low contraceptive prevalence,
knowledge and attitudes
See Toolkit for examples of population based surveys
Review data from formative research
– Results shows: low prevalence of contraceptive
use
4. Q.1) What is the problem and why does it
exist? (cont.)
Conduct a needs assessment
– Survey village leaders and community members
– In this example, results show:
• Greater need for modern contraceptive methods
• Easier access to wider range of methods
See Toolkit on how to conduct a needs assessment
5. Q. 2) What interventions can work?
Design an intervention
– A community outreach program in which health
workers visit homes and address the contraceptive
needs of families
Plan a pilot project
– Carry out the project in one neighborhood of the
village
– Expand to the entire village
6. Q. 3) What are we doing?
Develop a logic model
– Inputs outcomes
See Toolkit for description and examples of logic models
Create an M&E plan
– Include a timeline of program activities
See Toolkit for sample outline & program examples
Produce a Performance Monitoring Plan (PMP)
– Share proposed activities with stakeholders and donors
– Identify indicators to be collected and analyzed
See Toolkit for compendium of FP indictors
Engage stakeholders in every step of the program
7. Q. 3) What are we doing? (cont.)
Monitor the various components of the program
– Inputs
• Finances, staff, training materials, contraceptives
and transportation
– Processes
• Health workers trained to work in the community
• Health workers visit community periodically and
distribute contraceptives
• Program officers meet with village leaders often
Ensure quality of program is maintained
8. Q. 4) Are we implementing the program as
planned?
Output monitoring
– Number of first visits made
– Number of follow-up visits
– Types and numbers of contraceptives distributed
Outcome monitoring
– Change in percentage of contraceptive users over
time
See Toolkit for indicator guides for FP programs
9. Q. 5) Are the interventions working /
making a difference?
Outcome evaluation
See Toolkit for examples of evaluation designs and resources
10. Q. 6) Is the program sustainable
and scalable?
Sustainability
– Ensure local government continues the community
outreach program with the aid of health workers
– Build the community’s capacity to encourage voluntary
contraceptive use among families
Scalability
– Expand the program to other villages and regions within
the country
See Toolkit for means to measure sustainability and scalability
11. Next Steps
Share findings with all the stakeholders, including
– Village leaders and community members
– Local government and health department
– Funding agency
– Higher levels of government and health leadership
Disseminate findings widely including through
mass media, research literature and the internet
12. Example 2: Evaluate a Malaria Prevention
Program
Objective:
– To evaluate the impact of a malaria prevention
program in a district in East Africa
Main goal:
– To ensure all pregnant women and children <14 years
are sleeping under insecticide treated nets (ITNs) in
all the villages of a district in East Africa
13. Q.1) What is the problem & why does it exist?
Study recent population based surveys
– DHS, Multiple Indicator Cluster Survey
– Results show: High malaria prevalence and low ITNs
use
Conduct a needs assessment
– Visit selected homes; interview selected community
members
– In this example, results show: Use of ITNs is low due to
lack of knowledge and cost
See Toolkit for various data sources and assessment designs
14. Q. 2) What interventions can work?
Review other programs
Collaborate with more experienced programs
Plan a pilot project
Design an intervention
– Use handheld GPS devices to create clusters
– Visit all homes, identify pregnant women and children
<14 years and provide them with vouchers for ITN
– Program staff visit to ensure ITN installed correctly and
teach villagers how to reapply insecticide
See Toolkit for resources, i.e., Roll Back Malaria website
15. Q. 3) What are we doing?
Develop a logic model
– Inputs outcomes
Develop a PMP
Engage stakeholders in every step of the
program
– Community members and village leaders
– Local government and district officials
– Experts in the field
– Donors and policy makers
See Toolkit for a logic model and draft checklist for developing
PMP plan for malaria program
16. Q. 4) Are we implementing the program as
planned?
Monitor the various components of the
program
– Ensure quality of program is maintained
See Toolkit for:
– Indicator guides for malaria programs
– References on how to conduct routine monitoring
– Impact evaluation references
17. Q. 5) Are the interventions working /
making a difference?
Outcome evaluation
See Toolkit for alternative study designs
18. Q. 6) Is the program sustainable
and scalable?
Sustainability
– Ensure local government continues the voucher program
– Empower community leaders to encourage community
to access ITNs and to use them effectively
Scalability
– Expand the program to other districts and regions within
the country
See Toolkit for descriptions and examples of how programs can be
sustainable and scalable
Now, we will walk through two examples of program planning, monitoring and evaluation. As we present the examples, we will highlight the resources that are available within the toolkit.
The first example involves evaluating the impact of a family planning program in a rural village in West Africa.The goal is to increase knowledge of contraceptives, fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity) and modern contraceptive use.
We follow the sequence of questions identical to the ones described in the Toolkit. These questions describe the steps that are used when conducting an evaluation plan.For our particular question of interest, if there is a recent demographic and health survey from the country, data from the rural area could show that there is low contraceptive prevalence, knowledge, and attitudes (as well as limited exposure to media and other potential intervention strategies)The Toolkit outlines examples of other potential population based surveys and their strengths and limitations.For our example, Data from a pre-existing formative research would also show that there is low prevalence of contraceptive use.
OR you could conduct a needs assessment in the village of interest by surveying village leaders and community members.In this example, the results would show-A greater need for access to a large range of modern contraceptive methodsThe toolkit describes how to conduct a needs assessment for your question of interest
Design a community outreach program where health workers visit homes and address the contraceptive needs of familiesBefore carrying out the full-fledged intervention, the intervention is carried out in a neighborhood of the village as a pilot project. The lessons learned by getting feedback from the health workers and community members is used to improve upon the project design and is expanded to the entire village.
In order to track our activities, we can develop and use several tools.A logic model will outline all the steps between inputs and outcomes.The toolkit describes logic models and provides examples as well.We can also create an M&E plan that is helpful as it includes a timeline of program activities.The Toolkit also gives a good outline of an M&E plan and also has some program examples.Furthermore, we could produce a PMP which is useful as this document can be shared with donors and other stakeholders to give them an update on proposed activities. It also has a list of indicators that will be collected and analyzed.The toolkit has a great compendium of good FP indicators.In all these steps and processes, it is important to engage stakeholders in every step of the program planning and implementation as well as monitoring and evaluation.
Once our program activities are underway, we monitor the various components of the program.We monitor the use of inputs such as Finances, staff, training materials, contraceptives and transportationSome indicators that can be used to monitor our FP program activities includeThe number of health workers trained to work in the communityThe number of health workers who actually visit the community periodically and distribute contraceptivesThe frequency of meetings between program officers and village leadersMonitoring of inputs and processes ensures that the quality of the program activities are maintained.
In order to determine if we are implementing the program as planned, we collect indicators of output and outcome monitoring.Indicators of output monitoring for our FP program of interest include:Number of first visits madeNumber of follow-up visitsTypes and numbers of contraceptives distributedAn indicator for outcome monitoring is change in percentage of contraceptive users over time.The toolkit has some excellent indicator guides for FP programs.Indicators: fertility preferences (preferences for limiting and spacing births, modern contraceptive use and parity)
In order to determine if the FP intervention we conducted actually made a difference in contraceptive use, we conduct an outcome evaluation. One possible study design we use is the quasi-experimental design where we choose a control village early on. The villagers in this control village have similar sociodemographic characteristics than those in the intervention village. We collect data on contraceptive use, knowledge and attitudes before the intervention and after the intervention from both villages and determine if contraceptive use significantly went up in the intervention village. The toolkit also has examples of other evaluation designs and resources.Impact evaluationShould be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
Once the evaluation results show that our FP program was successful, we can make the FP program sustainable by ensuring that the local government continues the community outreach program. With the aid of the same health workers, we can also build the community’s capacity to continue accessing voluntary family planning at local pharmacies and health care centers.Furthermore, the program can be scaled up and expanded to other villages and regions within the country.The toolkit describes how to make programs sustainable and scalable and provides several successful examples as well.
Once the program has been implemented and evaluated, it is important to share the findings and best practices with all the stakeholders that were involved including the village leaders and community members of the village, the local government and health department, the funding agency and other higher levels of government and health leadership. The findings can also be shared with others through mass media, research literature and the internet.
The second example we will discuss today involves the evaluation of a malaria prevention program in a district in East Africa.The main goal of the program is to ensure that all pregnant women and children <14 years are sleeping under ITNs in all the villages within a district in East Africa.
In order to determine the extent of the malaria problem in this district, we review recent population based surveys such as the DHS and MICS. The results will show that there is a high prevalence of malaria and low use of ITNs.We can also conduct a needs assessment by selecting a few homes in a village and interviewing some community members within the selected homes. The results of this needs assessment would show that there is a limited use of ITNs due to lack of knowledge and high cost of ITNs.The toolkit outlines examples of other potential data sources that can be used to identify a health problem within a specified region and also states the strengths and limitations of each of the data sources.
In order to determine what interventions could possibly work in this situation, we review other programs, collaborate with more experienced programs and staff, even maybe plan a pilot project.The toolkit has some great resources to learn about existing successful malaria interventions, such as on the Roll Back Malaria website referenced in our toolkit.In this example, we could design an intervention in our district of interest. With the help of handheld GPS devices, we create clusters of 200 households within the district. We visit all homes, identify pregnant women and children <14 years and provide them with vouchers for ITNs. Then, we have program staff do follow-up visits to ensure that the ITN is installed correctly and teach villagers on how to reapply the insecticide.
Once we have decided on the intervention we will roll out, we develop a logic model that outlines all the steps between inputs and outcomes. We can also develop a PMP to help guide the activities.It is important to engage stakeholders at every step of the program. These stakeholders includeCommunity members and village leadersLocal government and district officialsExperts in the fieldDonors and policy makersThe toolkit describes a logic model and has a checklist of the important components within a PMP plan for malaria program prevention.
In order to determine whether we are implementing the program as planned, we monitor the various components of the program. This also ensure that the quality of the program is maintained.The toolkit has several resources for monitoring malaria program, such as Indicator guides for malaria programs, References on how to conduct routine monitoring as well as impact evaluation.
In order to conduct an outcome evaluation for our malaria program, we can use a time series design. Generally, Time-series designs look for changes over time to determine trends. Evaluators observe the intervention group multiple times before and after the intervention and analyze trends before and after.Impact evaluationCan be conducted a few years after the end of the programNeed to measure attribution of change to the intervention taking into account all other changes/interventions that might have occurred during the same periodCan use existing data sources, such as large surveys
Once the program has been rolled out successfully, it is important to ensure that the program remains sustainable. This can be done by ensuring that the local government continues the voucher program. Also, the community members are empowered to access ITNs and use them effectively.The program can also be scaled up to other districts and regions within the country thus benefiting a larger audience.The toolkit has descriptions and examples of how programs can be sustained and scaled up to new areas.
Thank you for joining us today. We’ve appreciated your participation. We are about to begin a Q&A session for this webinar but I want to let you know about a 3-day online forum that is has begun today. It will1. build capacity in monitoring and evaluation 2. advocate the importance of investing in monitoring and evaluation (M&E)3. and provide a more in-depth overview of MLE’s Measuring Success ToolkitOn your screen, please see the instructions to join the forum. Please join us in the Q&A being moderated by Gretchen.