1.
Development Project Planning 6 Monitoring and Evaluation In the Project Cycle

2.
Ground Rules <ul><li>Suggested Rules: </li></ul><ul><li>Come on time </li></ul><ul><li>No Phones </li></ul><ul><li>No come and go </li></ul><ul><li>No chat </li></ul>

3.
M&E and the Project Cycle <ul><li>What is M&E </li></ul><ul><li>Monitoring </li></ul><ul><ul><li>Types of monitoring </li></ul></ul><ul><ul><li>Working with Indicators </li></ul></ul><ul><ul><li>Data sources </li></ul></ul><ul><li>Evaluation </li></ul><ul><li>Choosing Indicators </li></ul>

4.
Why ? <ul><li>The purpose of monitoring and evaluation is to measure program effectiveness </li></ul>

5.
Uses Of M & E <ul><li>Monitoring and evaluation helps: </li></ul><ul><li>make decisions on implementation </li></ul><ul><li>ensure the most effective and efficient use of resources </li></ul><ul><li>determine if the project is on track and make any needed corrections </li></ul><ul><li>evaluate the impact of the project </li></ul>

6.
What is Monitoring ? <ul><li>Monitoring is collecting routine data to measure the progress of a project or activity. </li></ul><ul><li>It is used to track proje c t performance over time, to make informed decisions about the effectiveness of projects and the efficient use of resources. </li></ul><ul><li>Monitoring is also called process evaluation because it looks at the implementation process and asks: </li></ul><ul><ul><li>How well has the project been implemented? </li></ul></ul><ul><ul><li>How much does implementation vary from site to site? </li></ul></ul><ul><ul><li>Did the project benefit the intended people? At what cost? </li></ul></ul>

7.
What is Evaluation ? <ul><li>Evaluation measures how well the program activities have achieved the objectives, </li></ul><ul><li>and </li></ul><ul><li>how much the changes in outcomes are due to the project. </li></ul><ul><li>The difference in the outcome between having the project or not having the project is known as its “impact,” and measuring this difference is referred to as “impact evaluation.” </li></ul>

8.
Some Other Words <ul><li>Impact: any effect caused by project activity </li></ul><ul><li>including human health and safety, plants, animals, soil, air, water, climate, landscape, structures, cultural heritage or socio-economic conditions . </li></ul><ul><li>Metric: a unit of measure </li></ul><ul><li>Benchmark: a chosen level of an indicator </li></ul>

9.
Some Other Words <ul><li>Objective: factual, real; can be measured </li></ul><ul><li>scientific, repeatable result </li></ul><ul><li>e.g. how many people have TB? </li></ul><ul><li>S u b j e c t i v e : from one person’s view </li></ul><ul><li>Can be different for different people </li></ul><ul><li>e.g. are we a democratic country? </li></ul>

10.
M&E Plan <ul><li>The project proposal must include an M&E plan </li></ul><ul><li>M&E requires resources – time, staff, money </li></ul><ul><li>The indicators must relate to the project aim </li></ul><ul><li>The data must be reliable </li></ul><ul><li>Managers must be willing to use and learn from the results, and follow up </li></ul><ul><li>Dissemination – share information and lessons learned </li></ul>

11.
M&E Plans should include.. <ul><li>assumptions regarding context, activities, and goals </li></ul><ul><li>relationships between activities, targets, and outcomes </li></ul><ul><li>description of measures and operational definitions (indicators and metrics), with baseline values, monitoring schedule, data sources, and M&E resource estimates </li></ul><ul><li>partnerships and collaborations required to achieve results </li></ul><ul><li>specific attention to periodic evaluation , with resources allocated at least midterm and at project end. </li></ul>

12.
M&E <ul><li>M&E looks at progress against the INDICATORS in the logframe </li></ul><ul><li>So the first step for good M&E is choice of indicators </li></ul>

13.
What is an Indicator ? <ul><li>a variable … </li></ul><ul><li>that measures one aspect of a project </li></ul><ul><li>that is directly related to the program’s objectives. </li></ul>

14.
What is an Indicator ? <ul><li>An indicator is </li></ul><ul><li>a variable whose value changes from the baseline level (at the time the program began) to a new value after the program and its activities have made their impact felt. </li></ul><ul><li>Then the variable, or indicator, is calculated again. </li></ul>

15.
What is an Indicator ? <ul><li>Secondly, an indicator is a measurement. </li></ul><ul><li>It measures the value of the change in meaningful units that can be compared to past and future units. </li></ul><ul><li>This is usually expressed as a percentage or a number. </li></ul>

16.
What is an Indicator ? <ul><li>Finally, an indicator focuses on a single aspect of a program or project. </li></ul><ul><li>This aspect may be an input, an output or an overarching objective, but it should be narrowly defined in a way that captures this one aspect as precisely as possible. </li></ul>

17.
How many Indicators? <ul><li>an appropriate set of indicators will include at least one for each significant element of the project (i.e. at least one per box in a logframe) </li></ul><ul><li>a reasonable guideline recommends one or two indicators per result, at least one indicator for each activity, but no more than 10-15 indicators per area of significant program focus. </li></ul>

18.
A good indicator should: <ul><li>produce the same results when used repeatedly to measure the same condition or event; </li></ul><ul><li>measure only the condition or event it is intended to measure; </li></ul><ul><li>show changes in the state or condition over time; </li></ul><ul><li>have reasonable measurement costs; and </li></ul><ul><li>be defined in clear and unambiguous terms. </li></ul>

19.
Good Indicators <ul><li>Valid </li></ul><ul><li>Reliable </li></ul><ul><li>Precise </li></ul><ul><li>Independent </li></ul><ul><li>Timely </li></ul><ul><li>Comparable </li></ul>

20.
Good Indicators Validity <ul><li>The indicator measures what it is intended to measure </li></ul>

21.
Good Indicators Reliability <ul><li>The indicator minimises measurement error </li></ul><ul><li>repeatable </li></ul><ul><li>Types of measurement error </li></ul><ul><li>Sampling Error - caused by observing a sample instead of the whole population </li></ul><ul><li>Non-Sampling Error – all other errors </li></ul><ul><li>Subjective Measurement - bias </li></ul>

22.
Good Indicators Precise Definition <ul><li>Is defined in clear terms </li></ul><ul><ul><li>What you measure and how </li></ul></ul>

23.
Good Indicators Independence <ul><li>Not directional – can be positive OR negative </li></ul><ul><li>One dimensional – up and down a line </li></ul><ul><li>Describe a discrete result at a single point in time </li></ul>

24.
Good Indicators Timeliness <ul><li>Provides a measurement over the periods of time that matter (e.g. the project life) </li></ul><ul><li>with data available for all appropriate intervals </li></ul>

25.
Good Indicators Comparability <ul><li>Compares with other similar situations </li></ul><ul><li>Assists in understanding results across different population groups and project approaches </li></ul>

26.
Additional Factors Influencing Indicator Selection <ul><li>Data availability </li></ul><ul><li>Resources </li></ul><ul><li>Program needs </li></ul><ul><li>Donor requirements </li></ul>

27.
<ul><li>Take a break </li></ul>

28.
Types of Indicators <ul><li>Quantitative - an actual number of some output </li></ul><ul><li>Qualitative - descriptive observations that can supplement the numbers and percentages provided by quantitative indicators. </li></ul><ul><li>They add to quantitative indicators a richness of information about the context in which the program has been operating. </li></ul><ul><li>Examples include “availability of a clear, strategic organisational mission statement” and “existence of a multi-year procurement plan for each product offered.” </li></ul><ul><li>“ For a quantitative indicator you would collect numbers , and for a qualitative indicator you would collect facts or opinions .” </li></ul>

29.
Types of Indicators <ul><li>Confusion exists in what are qualitative and quantitative indicators. </li></ul><ul><li>It is clear that quantitative indicators measure changes that can be counted. It is not clear what is a qualitative indicator. </li></ul><ul><li>some say qualitative indicators relate to the quality of the change being measured (DFID, 1995) - e.g. women’s political representation: a quantitative indicator could be the percentage of parliament seats occupied by women, while a qualitative indicator would describe the quality of women’s political participation. </li></ul><ul><li>some say qualitative indicators describe a subjective opinion on an issue or project impact. </li></ul>

30.
Qualitative is Quantitative? <ul><li>Most do not define indicators as qualitative or quantitative - they assume that all indicators are by definition quantitative. </li></ul><ul><li>For example: </li></ul><ul><li>A qualitative indicator could be “most village women feel they have a voice at meetings” </li></ul><ul><li>[ but how do you know this? ] </li></ul><ul><li>So this could be quantitative: </li></ul><ul><li>“ the percentage of women surveyed who say they have a voice at meetings” </li></ul>

31.
Indicators of What ? Utilisation Shows if a service is being used, e.g. • Number of people attending a nutrition course • Number of people requesting VCT • Applications received for a training course Utililisation Shows what a service has done or provided, e.g. • Number of brochures produced • Number of condoms distributed • Number of home care visits • Number of clients counselled and tested Activity or Output Shows what goes in to providing a service - the resources used, e.g. • Amount spent on travel per week • Home care supplies purchased per month • Wages, allowances and incentives paid • Production costs for brochures and posters Inputs

32.
Indicators of What ? Performance Shows how well something was done, e.g. • Number of people reporting they are ‘satisfied’ with a training workshop • Number of reported cases of STIs • Proportion of VCT clients returning to collect their HIV test results • Number of orphans supported in the community Performance Coverage Shows what proportion of people / groups in need receive a service, e.g. • Proportion of all orphans receiving visits • Proportion of schools with an AIDS awareness club • Proportion of commercial farms with peer education programmes Coverage

33.
Exercise <ul><li>Water and Sanitation Project </li></ul><ul><li>Read the Concept Note </li></ul><ul><li>Using the Logframe, choose the Indicators </li></ul>

34.
Setting up an Indicator <ul><li>You must identify exactly how a given concept or behaviour will be measured – the Metric. </li></ul><ul><li>The Metric is the precise calculation or formula on which the indicator is based. </li></ul><ul><li>Calculation of the metric establishes the indicator’s objective value at a point in time. </li></ul><ul><li>Even if the factor itself is subjective or qualitative, (eg attitudes of a target population), the indicator metric calculates its value at a given time objectively </li></ul><ul><li>This can be called “Operationalising” an Indicator </li></ul>

35.
Setting up an Indicator <ul><li>You need to be careful about exactly how you define the metric </li></ul><ul><li>e.g. “the percentage of HIV+ mothers who have prepartum AZT therapy” </li></ul><ul><li>is it – ‘% of those births attended by the health care system’ or ‘% of all births’ </li></ul><ul><li>is it – ‘% of recorded diagnosed HIV+ women’ or ‘% of all HIV+ women’ </li></ul>

36.
S e t t i n g u p a n I n d i c a t o r <ul><li>In many cases, indicators need to have definitions of the terms used. </li></ul><ul><li>For instance, let’s look at the indicator: ‘number of antenatal care (ANC) providers trained’. If this indicator is used by a program, definitions need to be included. </li></ul><ul><li>Providers would need to be defined, e.g </li></ul><ul><li>‘ any worker providing direct clinical services to clients seeking ANC at a public health facility’. </li></ul><ul><li>For this indicator then, providers would not include those working in private facilities. </li></ul><ul><li>Trained would also need to be defined, perhaps as ‘ those staff who attended every day of a five-day training course and passed the final exam with a score of at least 85%’. </li></ul>

37.
Thankyou <ul><li>Please keep papers for next session </li></ul>

38.
Produced by Tony Hobbs Health Unlimited, Ratanakiri, Cambodia www.healthunlimited.org With the support of Australian Volunteers International www.australianvolunteers.com © 2009 HU. Use with Acknowledgement