This document outlines the key steps to conduct an impact evaluation of a school feeding program in Mali in 7-8 steps. It involves engaging stakeholders, defining relevant evaluation questions, building a theory of change, defining indicators, designing the evaluation using a randomized controlled trial across treatment and control groups, determining an appropriate sample size, conducting a household survey, and analyzing the collected data. The goal is to evaluate the program's impact on education, nutrition, local agriculture, and welfare outcomes.

1. Impact evaluation in 7 or 8 steps

2. Step 1
Engage with the stakeholders

3. June mission in Mali
• Government interested in a field experiment
(RCT) using project and control villages
• Government interested in three main issues:
o Good governance: how to make sure that children
are fed?
o Education: what is the impact on attendance
rates?
o Local economy: how the project benefits small
farmers?

4. Step 2
Define relevant evaluation questions

5. Impact on education and nutrition
• Impact on enrolment and achievements, and
what is the role played by school quality?
• Does the programme improve attention and
cognition?
• Does the programme improve nutritional status?
Is there catching-up growth?
• What is the extent of substitution effects within
the household?
• What is the impact of the programme on the diet
of the poor?

6. Impact on agriculture
• What is the impact on small farmers in the
short term (incomes) and in the long term
(farm investments)?
• What is the impact of the intervention on
prices and therefore on consumers?
• What is the impact on the wider economy at
the village and regional levels?

7. Step 3
Build a theory of change

8. Overall programme theory

9. Agriculture pathways

10. Education pathways

11. Nutrition pathways

12. Step 4
Define the indicators

13. Welfare outcomes
• Welfare outcomes are the MDGs metrics

14. Intermediate welfare outcomes
• Sometimes welfare outcomes cannot be
observed because:
• Occur in the very long term (example increase in
employment)
• Are not observable (example maternal mortality)
• Intermediate outcomes are used in these
cases: proxy indicators of the final outcomes
along the causal chain

15. Education: outcome indicators
• Enrolment, attendance rates and drop-outs
• Achievement tests (test scores on maths and
language)
• Attention and cognition

16. Nutrition: outcome indicators
• Anthropometric measurement.
• Measures of diet composition

17. Food security: outcome indicators
• Full income questionnaire for farmers will
provide data on:
 marketed surplus
 Farm profits
 Technology and capitalisation
 Input use

18. Step 5
The evaluation design

19. The Mali evaluation design
• The government is expanding the intervention
to 60 of the most vulnerable communes
• A commune is an administrative unit
comprising 5 to 15 villages and a similar
number of schools
• The groups considered by the study are:
 Control group (no intervention)
 Standard school feeding
 Home grown school feeding

20. Level 1 comparison: school
feeding-control group
• MOE needs to know the impact of the
intervention on educational indicators
• First comparison is between any school
feeding and a control group with no
intervention
• Outcomes of interests are:
 Enrolment rates
 Learning achievements
 Attention and cognition
 Nutritional outcomes

21. Level 2 comparison: school feeding
/home-grown school feeding
• The second comparison is between the
conventional government programme and the
‘home grown’ programme
• Outcomes of interest:
 Small farmers’ incomes
 Overall programme performance

22. Selection of schools and
communes
• In each of the 60 communes Mayors will
select two school for the intervention of which
one will be randomly assigned to the
programme (pair-matching design or
stratification by commune). A protocol is
designed to avoid contamination.
• Of the 60 communes assigned to the
programme, 30 will be randomly assigned to
the home grown component

23. Step 6
Set the sample size

24. Sample size
• We collected data from 30 households in each
village:
 20 households with children aged 5 to12
 10 farmer households (with or without children)
• Sample size is:
 1,200 farmer households
 3,600 farmer and non-farmer households
 6,000 to 7,000 children of primary school age

25. Calculate the sample size
• The size needs to be sufficiently large to
detect the expected effect of the programme
• Detecting sample size is guesswork and the
goal is to produce lower and upper bounds
rather than exact samples
• There is statistical software which is designed
to do this

26. Power
• You need a powerful sample to detect impact
• The power of your sample will be a function of
• The expected programme impact (increasing)
• The variance of the outcome of interest
(decreasing)
• The homogeneity within clusters (decreasing)
• The desired level of Type I error (decreasing)

27. Step 7
Run a household survey

28. main issues
• Choose the unit of
observation
• Find existing datasets and
surveys
• Establish the timing of data
collection
• Establish whether collecting
cross-section or panel data
• Establish the number of
surveys
• Design the questionnaires
• Administer the survey

29. Choose the unit of observation
• The preferred level of observation is the
‘individual’ or the ‘household’
• Note that individuals are difficult to interview (ex:
consumption data or panel)
• Household is the most frequent unit of observation
• Observations can also be made at cluster level
(village, school, clinic or other)
• Note that sample size will be small
• Data is difficult to collect (who is interviewed?)

30. Data scoping
• Before starting collecting any data you should
first investigate what data and surveys are
available:
• Census data can be used to frame the sample or to
extract control variables
• Existing household surveys (LSMS or DHS) can be used
to form control groups in matching techniques
• Project monitoring data can be used to observe trends
• Survey maybe underway in the same areas. This is
rarely the case, but piggybacking is theoretically
possible

31. Timing of data collection
• 3 main issues to consider:
• How many surveys will be run?
• Baseline, midterm and end-line
• When is the survey starting and for how long?
• Seasonality issues need to be considered
• What is the recall period adopted in the
questionnaire?
• Short recall is more reliable but loses information

32. Cross-section or panel data?
• A cross-section survey collects data from a
population at a point in time
• A panel survey collects data from a population
repeated times
• Panel data are preferable because they
simplify the analysis
• But panel data are not always feasible
• But attrition can be large and there can be
differential attrition

33. Questionnaire design
• Identify the modules that are needed. For
example: a household roster, an education
module, a consumption module etc.
• Look at existing questionnaire designed by
other researchers in similar context
• Examples can be taken from:
• LSMS surveys
• DHS surveys
• Other resources

34. Running a survey in practice
• Hire a firm with the desired capacity
• Ensure enumerators are properly trained and
manuals are available
• Test the questionnaires many times
• Ensure supervision of enumerators in the field
• Ensure households collaborate
• Obtain ethical approval

35. Step 8
Analyse the data