Monitoring & Evaluation Notes

1. Monitoring and Evaluation
Charles Katende PhD.
Director of Monitoring, Evaluation and Research
JHPIEGO (An affiliate of John Hopkins University)

2. Session Objective
• To increase participants understanding of
the concepts used in designing M&E
Frameworks and Plans
• To build participants competence in
designing Program M&E Plans

3. Expected Results
• At the end of the sessions participants will know
about Program frameworks, M&E frameworks
and the difference between the two frameworks
• Participants will be able to identify and select
appropriate indicators for a program.
• Participants will be able to produce a program
monitoring and evaluation framework.

4. Introduction…
• Write the health problem addressed by a major
public health program in your country
• Write at up to three specific objectives of a
public health program that addresses the above
mentioned health problem.
• Write down two indicators the program
mentioned above uses to monitor it progress or
performance towards its objectives.

5. What is Program Monitoring, Evaluation?
Monitoring is the
routine process of
data collection and
measurement of
progress toward
program objectives.
Evaluation is the use of
social research
methods to
systematically
investigate a
achievement of a
program’s results

6. Key Questions
• What is the purpose of carrying out M&E
• Who needs, uses M&E Information
• Who carries out M&E?
• How is M&E carried out?
• When should M&E be carried out?

7. What is the purpose….?
• Improve program implementation
– Data on program progress and implementation
– Improve program management and decision making
• Inform future programming
• Inform stakeholders
– Accountability (donors, beneficiaries)
– Advocacy

8. Who needs, uses M&E Information?
To Improve program
implementation…
To Inform and improve
future programs
Inform stakeholders
• Managers
• Donors
• Governments
• Technocrats
• Donors
• Governments
• Communities
• Beneficiaries

9. Who conducts M&E….?
Program implementer
Stakeholders
Beneficiary
Remember ..
M&E Technical skills
Participatory process

10. How to carry out M&E…?
Key Features
1. Program Framework: Analyze and systematically lay out
program elements
2. Identify key elements to monitor and evaluate.
3. Determine and describe the measures to be used for
monitoring and evaluation
4. Develop M&E Framework and action plans, including
data collection and analysis, reporting and dissemination
of findings.

11. Program Framework
What do you know about your program….?

12. Program Framework
• Systematic lay out of the program
elements and path showing what
the program plans to:
do ……………..achieve!

13. Program Framework
• Based on a theoretical, empirical model, or
general understanding
Public health Problem
Population, system level factors that cause the public
health problem
Action/interventions that can change the factors and ultimately
alleviate/eliminate the problem

14. Results Framework
Improved Health Status
Improved (Sustained) Use of Key Health Services
and Practices/Behaviors
Strategic Objective:
Intermediate Result:
Increased quality
of…
Increased
availability/ access
to…
Strategies (Sub IR):
Strategies: Strategies:
Improved social /
policy
environment…
Strategies:
Impact:

15. Example: Result Framework for a
Family Planning Project
GOAL: REDUCED FERTILITY
SO: Increased FP use and improved FP/RH practices
Increased knowledge of,
improved attitudes toward,
and acceptance of key
services and behavior
Increased quality of FP
counseling and services
for
Increased
availability/access to
FP/RH
Strategies:
Increase availability of
educational materials at
clinic and community
level
• Community
mobilization (using
PRA and PDI)
including men
• Implement mass media
strategy
• Mobilize opinion
leaders at national and
local level
Design/ implement
supportive supervision
System
• Train service providers
(in-service and
preservice in FP
counseling and
management of side
effects
• Remodel clinic to allow
for privacy
• Design and implement
quality improvement
program
Strengthen logistics
management
• Mobilize private sector
providers
• Mobilize CHWs/CBDs
• Encourage socially
marketed pills
Improved social and
policy environment for
FP
Advocate for community
based distribution of
pills
Promote addition of
Depo injections to EPI
outreach strategy
Pilot social marketing of
pills

16. Basic Logic Model

17. 26
VCT Program Implementation Logic Model
VCT Program Implementation Logic Model
Train nurses &
lab techs in VCT
ACTIVITIES OUTPUTS
IMMEDIATE
OUTCOMES
INTERMEDIATE
OUTCOMES
IMPACTS
INPUTS
Risk behaviors
decreased
HIV treatment
increased
Nurses
Lab techs
Provide pre-test
counseling
Conduct HIV test
Provide post-test
counseling to all
clients tested
Refer pregnant
HIV+ women to
PMTCT svcs
Govt. funds
GAP funds
Other donor
funds
Counseling
protocol
HIV test kits
Referral
system for
prevention &
Tx services
Refer HIV+
clients to Tx
services
Clients are
counseled for
HIV testing
Clients are
tested
Clients receive
results and post-
test counseling
Pregnant HIV+
women referred
to PMTCT svcs
HIV+ clients
referred to ARV,
support & HBC
Quality of VCT
increased
Access to VCT
increased
Knowledge of
HIV status
increased
Knowledge
about & access
to prevention
resources
increased
Access to HIV
treatment
resources
increased
HIV transmission
rates decreased
HIV prevalence
decreased
HIV morbidity &
mortality
decreased
Nurses & lab
techs trained

18. PROJECT DESIGN FRAMEWORK LABELS
Labels Used By Various Organizations
Project Design Level USAID Others
Impact Impact Goal
Strategic Objectives Purpose
Intermediate Results (IR)
Specific Objectives
Startegies (Sub-IR)
Outputs
Activities
Outputs
Activity Clusters
Activities
Inputs
Level A:
Improvement
in Health
Status
Level B:
Use of
Services
Level C:
Demand for
Services
Capacity to
Deliver
Services
Level D:
Interventions

19. Case 1: To decrease maternal mortality, a 10-year program plan to
improve to train midwifes to Delivery and ANC services at health
facilities, and to train and deploy CHWs to increase the community’s
awareness about, and use of the improved services at the health
facilities.
Case 2: To reduce high fertility, a 5-year program plans to work with
the Government to change policies in order to allow and promote
use of modern family planning methods, train family planning
providers to provide better FP services, and to launch public
campaigns that promote family planning methods.
Case 3: To reduce HIV infection among adolescents, a five-year
program plans to implement income generation activities for the
youth, provide and promote universal secondary education, and
build adolescent-friendly reproductive health service delivery points.

20. Exercise
• Identify and state is the Public Health problem
implied in the case study.
• What are population level factors will the
program target to change in order to alleviate
the public health problem
• Prepare a Program Framework for the scase
study

21. Monitoring and Evaluations
Framework

22. M&E Questions
• Monitoring questions
– What is being done?
– By whom?
– Target population?
– When?
– How much?
– How often?
– Additional outputs?
– Resources used? (Staff,
funds, materials, etc.)

23. M&E Questions
• Evaluation Questions?
– Is the content of the
intervention or the activity
being delivered as
planned?
– Does the content of the
intervention or the activity
reflect the requisite
standards?
– Have the intervention
achieved the expected
results?

24. What do we need to answer these questions…?
INDICATORS …to take measurements.

25. Indicators: Definition
• Markers that help to measure change by
showing progress towards meeting
objectives
• Observable, measurable, and agreed upon
as valid markers of a less well-defined
concept or objective
• Indicators differ from objectives in that they
address specific criteria that will be used to
judge the success of the project or program.
See comment for examples

26. Type and Level of Each Indicator
• Type
–Input/Process (Monitoring)
–Outcome / Impact (Evaluation)
• Level
–Global level
–Country level
–Program level

27. Exercise: Group work
• Use your case study and identify at least
two indicators for program monitoring and
two indicators for program evaluation.

28. What Is a Good Indicator?
• Valid: Measures the effect it is supposed to measure
• Reliable: Gives same result if measured in the same way
• Precise: Is operationally defined so people are clear about what
they are measuring
• Timely: Can be measured at an interval that is appropriate to the
level of change expected
• Comparable: Can be compared across different target groups or
project approaches

29. Criteria for Indicator Selection
• Consistent with project design—measure the desired
result
• Useful—contributes to project design, management, and
evaluation
• Available
• Affordable

30. Standard Indicators
Where possible, a project should select
standard indicators.
• They have been tested for validity and
reliability.
• They allow comparison between projects
or sites.
• They tend to be available for SOs and
some IRs.

31. How Many Indicators?
• Choose at least one or two indicators per intermediate
result, as well as the SO for evaluation purposes.
• Choose one or two indicators per result for program
monitoring.
• Choose indicators that may be able to “cover” more than
one element.
• For management, think about basic activities that you
need to monitor to judge if you are implementing
activities as planned; include indicators that help you
make decisions.

32. Exercise: Group work
Refer to the indicators you selected..
• Were the good indicators ?
• Did you select a minimum number
recommended given the type and size of
your program?

33. M&E Framework

34. Sample M&E Framework
Result Indicator Definition Data
source
Collection
Method
Freq-
uency
Respons-
ible Party
Active
Management
of the Third
Stage of
labor
increased
Proportion
of trained
clinicians
performing
AMTSL to
standard
# of
trained
midwives
performin
g all steps
of AMTSL
on all
patients/
AMTSL
observa-
tion
checklist
Clinical
observation
Annual Zambia
JHPIEGO
staff
Preventing Post Partum Hemorrhage :
Increase Active Management of the Third Stage of Labor
See comment for examples

35. M&E Plan
• The plan is a managerial tool that specifies the schedule, resources,
responsibilities, for your M&E activities (data collection, data quality
control, reporting, dissemination and use of data)
• Note:
– The plan should specify the time points when evaluations will be carried
out, for example: Midterm, and End term.
– Outcome/Impact evaluation is reserved for large longer term programs
that can make impact at public health status level.
– Your plan should include activities to monitor and evaluate the
implementation, as planned, of the M&E plan.

36. Strategic Planning for M&E: Setting Realistic
Expectations
Most Some Few *
All
Monitoring and Evaluation Pipeline
Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001
Input/Output
Monitoring
Process
Evaluation
Outcome
Monitoring/
Evaluation
Impact Monitoring/
Evaluation
Levels of Monitoring & Evaluation Effort
Number of
Projects
* Supplemented with impact
indicators from surveillance data.
Source: CDC. Global AIDS program monitoring
and evaluation (M&E) field guide

37. Question
If funding for your case study program was
cut off and the program closed in two years.
What changes would you make to you M&E
Plan?

38. Sources of Information
• http://www.cpc.unc.edu/measure
• http://www.unaids.org/DocOrder/OrderFor
m.aspx
• http://www.fhi.org/en/Publications/index.ht
m

39. THE END