Presenter notes for module 8 of a course on project cycle management.

1. Development Project
Planning 4
Monitoring and Evaluation
In the Project Cycle
Need – flipcharts for watsan
logframe indicators, Presenter notes.
Handouts – Prepared handout,
Project Cycle Mgmt Toolkit Handout
10 & 13 Pages 54, 55, 58, 59,
Watsan Logframe
M&E and the Project Cycle
 What is M&E
 Monitoring
o Types of monitoring
o Working with Indicators
o Data sources
 Evaluation
 Choosing Indicators
Why ?
The purpose of monitoring and
evaluation is to measure program
effectiveness
.. And to demonstrate it to donors
and others
Uses Of M & E
Monitoring and evaluation helps:
 make decisions on implementation
 ensure the most effective and efficient use
of resources
 determine if the project is on track and make
any needed corrections
 evaluate the impact of the project
What is Monitoring ?
Monitoring is collecting routine data to measure the
progress of a project or activity.
It is used to track project performance over time, to
make informed decisions about the effectiveness of
projects and the efficient use of resources.
Monitoring is also called process evaluation because it
looks at the implementation process and asks:
o How well has the project been implemented?
o How much does implementation vary from site to site?
o Did the project benefit the intended people? At what
cost?
What is Evaluation ?
Evaluation measures how well the program
activities have achieved the objectives,
and
how much the changes in outcomes are due
to the project.
The difference in the outcome between having
the project or not having the project is
known as its “impact,” and measuring this
difference is referred to as “impact
evaluation.”
Some Other Words
Impact: any effect caused by project activity
including human health and safety, plants,
animals, soil, air, water, climate, landscape,
structures, cultural heritage or socio-economic
conditions.
Metric: a unit of measure
Benchmark: a chosen level of an indicator
Some Other Words
Objective: factual, real; can be measured
scientific, repeatable result
e.g. how many people have TB?
Subjective: from one person’s view
Can be different for different people
e.g. are we a democratic country?
M&E Plan
The project proposal must include an M&E plan
 M&E requires resources – time, staff, money
 The indicators must relate to the project aim
 The data must be reliable
 Managers must be willing to use and learn from
the results, and follow up
 Dissemination – share information and lessons
learned

2. M&E Plans should include..
 assumptions regarding context, activities, and goals
 relationships between activities, targets, and outcomes
 description of measures and operational definitions
(indicators and metrics), with baseline values,
monitoring schedule, data sources, and M&E resource
estimates
 partnerships and collaborations required to achieve
results
 specific attention to periodic evaluation, with resources
allocated at least midterm and at project end.
M&E
 M&E looks at progress against the
INDICATORS in the logframe
 So the first step for good M&E is choice of
indicators
This is done at the planning stage,
so if the project proposal is already
accepted then these are set, but
than can be changed (with
agreement).
What is an Indicator ?
a variable …
that measures one aspect of a project
that is directly related to the program’s
objectives.
No detail – go on to next 3 slides
What is an Indicator ?
An indicator is
 a variable whose value changes from the
baseline level (at the time the program
began) to a new value after the program
and its activities have made their impact
felt.
 Then the variable, or indicator, is
calculated again.
What is an Indicator ?
 Secondly, an indicator is a measurement.
It measures the value of the change in
meaningful units that can be compared to
past and future units.
This is usually expressed as a percentage or
a number.
What is an Indicator ?
 Finally, an indicator focuses on a single
aspect of a program or project.
This aspect may be an input, an output or an
overarching objective, but it should be
narrowly defined in a way that captures this
one aspect as precisely as possible.
How many Indicators?
 an appropriate set of indicators will include
at least one for each significant element of
the project (i.e. at least one per box in a
logframe)
 a reasonable guideline recommends one or
two indicators per result, at least one
indicator for each activity, but no more than
10-15 indicators per area of significant
program focus.
A good indicator should:
 produce the same results when used
repeatedly to measure the same condition
or event;
 measure only the condition or event it is
intended to measure;
 show changes in the state or condition over
time;
 have reasonable measurement costs; and
 be defined in clear and unambiguous terms.
This is often expressed as.. (go to
next slide)

3. Good Indicators
 Valid
 Reliable
 Precise
 Independent
 Timely
 Comparable
Good Indicators
Validity
 The indicator measures what it is intended
to measure

4. Good Indicators
Reliability
 The indicator minimises measurement error
repeatable
Types of measurement error
 Sampling Error - caused by observing a
sample instead of the whole population
 Non-Sampling Error – all other errors
 Subjective Measurement - bias
Good Indicators
Precise Definition
 Is defined in clear terms
o What you measure and how
Good Indicators
Independence
 Not directional – can be positive OR
negative
 One dimensional – up and down a line
 describe a discrete result at a single point in
time
Directional – if you ask only ‘what is
the increase’ you may miss a
decrease
Dimensional – ‘what colour shirt’
gives an array of answers, ‘how
bright is the shirt’ gives a linear
progression
Discrete – scientific repeatable type
of measure
Good Indicators
Timeliness
 Provides a measurement over the periods of
time that matter (e.g. the project life)
 with data available for all appropriate
intervals
Good Indicators
Comparability
 With other similar situations
 Assists in understanding results across
different population groups and project
approaches
Eg a national census figure would be
no good if it takes place only every 5
years
Additional Factors Influencing
Indicator Selection
 Data availability
 Resources
 Program needs
 Donor requirements
Take a break
Types of Indicators
Quantitative - an actual number of some output
Qualitative - descriptive observations that can supplement the
numbers and percentages provided by quantitative indicators.
They add to quantitative indicators a richness of information about
the context in which the program has been operating.
Examples include “availability of a clear, strategic organizational
mission statement” and “existence of a multi-year procurement
plan for each product offered.”
“For a quantitative indicator you would collect numbers, and for a
qualitative indicator you would collect facts or opinions.”
Types of Indicators
 Confusion exists in what are qualitative and quantitative indicators.
 It is clear that quantitative indicators measure changes that can be
counted. It is not clear what is a qualitative indicator.
 some say qualitative indicators relate to the quality of the change
being measured (DFID, 1995) - e.g. women’s political representation: a
quantitative indicator could be the percentage of parliament seats
occupied by women, while a qualitative indicator would describe the
quality of women’s political participation.
 some say qualitative indicators describe a subjective opinion on an
issue or project impact.
 But - most do not define indicators as qualitative or quantitative -
they assume that all indicators are by definition quantitative.
Next slide for example

5. Qualitative is Quantitative?
For example:
A qualitative indicator could be “most village
women feel they have a voice at meetings”
[ but how do you know this? ]
So this could be quantitative:
“the percentage of women surveyed who say
they have a voice at meetings”
Indicators of What ?
Utilisation Shows if a service is being used, e.g.
• Number of people attending a nutrition course
• Number of people requesting VCT
• Applications received for a training course
Utililisation
Shows what a service has done or provided, e.g.
• Number of brochures produced
• Number of condoms distributed
• Number of home care visits
• Number of clients counselled and tested
Activity or
Output
Shows what goes in to providing a service - the resources
used, e.g.
• Amount spent on travel per week
• Home care supplies purchased per month
• Wages, allowances and incentives paid
• Production costs for brochures and posters
Inputs
Indicators of What ?
Performance Shows how well something was done, e.g.
• Number of people reporting they are ‘satisfied’ with a
training workshop
• Number of reported cases of STIs
• Proportion of VCT clients returning to collect their HIV test
results
• Number of orphans supported in the community
Performance
Coverage Shows what proportion of people / groups in need
receive a service, e.g.
• Proportion of all orphans receiving visits
• Proportion of schools with an AIDS awareness club
• Proportion of commercial farms with peer education
programmes
Coverage
Exercise
Water and Sanitation Project
1. Read the Concept Note
2. Using the Logframe, choose the Indicators
Setting up an Indicator
 You must identify exactly how a given concept or
behaviour will be measured – the Metric.
 The Metric is the precise calculation or formula on
which the indicator is based.
Calculation of the metric establishes the indicator’s
objective value at a point in time.
Even if the factor itself is subjective or qualitative, (eg
attitudes of a target population), the indicator metric
calculates its value at a given time objectively
 This can be called “Operationalising” an Indicator
Setting up an Indicator
You need to be careful about exactly how you
define the metric
e.g. “the percentage of HIV+ mothers who have
prepartum AZT therapy”
is it – ‘% of those births attended by the
health care system’ or ‘% of all births’
is it – ‘% of recorded diagnosed HIV+ women’
or ‘% of all HIV+ women’
If “all births” how do you measure
that – are they reported
If all HIV+ how do you get the figure
if they are not all diagnosed?

7. Setting up an Indicator
In many cases, indicators need to have definitions
of the terms used.
 For instance, let’s look at the indicator: ‘number of antenatal
care (ANC) providers trained’. If this indicator is used by a
program, definitions need to be included.
 Providers would need to be defined, e.g
‘any worker providing direct clinical services to clients seeking
ANC at a public health facility’.
For this indicator then, providers would not include those
working in private facilities.
 Trained would also need to be defined, perhaps as ‘those staff
who attended every day of a five-day training course and
passed the final exam with a score of at least 85%’.