What have we learnt from randomized control trials


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

What have we learnt from randomized control trials

  1. 1. Impact evaluation in 7 or 8 steps
  2. 2. Step 1Engage with the stakeholders
  3. 3. June mission in Mali• Government interested in a field experiment (RCT) using project and control villages• Government interested in three main issues: o Good governance: how to make sure that children are fed? o Education: what is the impact on attendance rates? o Local economy: how the project benefits small farmers?
  4. 4. Step 2Define relevant evaluation questions
  5. 5. Impact on education and nutrition• Impact on enrolment and achievements, and what is the role played by school quality?• Does the programme improve attention and cognition?• Does the programme improve nutritional status? Is there catching-up growth?• What is the extent of substitution effects within the household?• What is the impact of the programme on the diet of the poor?
  6. 6. Impact on agriculture• What is the impact on small farmers in the short term (incomes) and in the long term (farm investments)?• What is the impact of the intervention on prices and therefore on consumers?• What is the impact on the wider economy at the village and regional levels?
  7. 7. Step 3Build a theory of change
  8. 8. Overall programme theory
  9. 9. Agriculture pathways
  10. 10. Education pathways
  11. 11. Nutrition pathways
  12. 12. Step 4Define the indicators
  13. 13. Welfare outcomes• Welfare outcomes are the MDGs metrics
  14. 14. Intermediate welfare outcomes• Sometimes welfare outcomes cannot be observed because: • Occur in the very long term (example increase in employment) • Are not observable (example maternal mortality)• Intermediate outcomes are used in these cases: proxy indicators of the final outcomes along the causal chain
  15. 15. Education: outcome indicators• Enrolment, attendance rates and drop-outs• Achievement tests (test scores on maths and language)• Attention and cognition
  16. 16. Nutrition: outcome indicators• Anthropometric measurement.• Measures of diet composition
  17. 17. Food security: outcome indicators• Full income questionnaire for farmers will provide data on:  marketed surplus  Farm profits  Technology and capitalisation  Input use
  18. 18. Step 5The evaluation design
  19. 19. The Mali evaluation design• The government is expanding the intervention to 60 of the most vulnerable communes• A commune is an administrative unit comprising 5 to 15 villages and a similar number of schools• The groups considered by the study are:  Control group (no intervention)  Standard school feeding  Home grown school feeding
  20. 20. Level 1 comparison: school feeding-control group• MOE needs to know the impact of the intervention on educational indicators• First comparison is between any school feeding and a control group with no intervention• Outcomes of interests are:  Enrolment rates  Learning achievements  Attention and cognition  Nutritional outcomes
  21. 21. Level 2 comparison: school feeding /home-grown school feeding• The second comparison is between the conventional government programme and the ‘home grown’ programme• Outcomes of interest:  Small farmers’ incomes  Overall programme performance
  22. 22. Selection of schools and communes• In each of the 60 communes Mayors will select two school for the intervention of which one will be randomly assigned to the programme (pair-matching design or stratification by commune). A protocol is designed to avoid contamination.• Of the 60 communes assigned to the programme, 30 will be randomly assigned to the home grown component
  23. 23. Step 6Set the sample size
  24. 24. Sample size• We collected data from 30 households in each village:  20 households with children aged 5 to12  10 farmer households (with or without children)• Sample size is:  1,200 farmer households  3,600 farmer and non-farmer households  6,000 to 7,000 children of primary school age
  25. 25. Calculate the sample size• The size needs to be sufficiently large to detect the expected effect of the programme• Detecting sample size is guesswork and the goal is to produce lower and upper bounds rather than exact samples• There is statistical software which is designed to do this
  26. 26. Power• You need a powerful sample to detect impact• The power of your sample will be a function of • The expected programme impact (increasing) • The variance of the outcome of interest (decreasing) • The homogeneity within clusters (decreasing) • The desired level of Type I error (decreasing)
  27. 27. Step 7Run a household survey
  28. 28. main issues • Choose the unit of observation • Find existing datasets and surveys • Establish the timing of data collection • Establish whether collecting cross-section or panel data • Establish the number of surveys • Design the questionnaires • Administer the survey
  29. 29. Choose the unit of observation• The preferred level of observation is the ‘individual’ or the ‘household’ • Note that individuals are difficult to interview (ex: consumption data or panel) • Household is the most frequent unit of observation• Observations can also be made at cluster level (village, school, clinic or other) • Note that sample size will be small • Data is difficult to collect (who is interviewed?)
  30. 30. Data scoping• Before starting collecting any data you should first investigate what data and surveys are available: • Census data can be used to frame the sample or to extract control variables • Existing household surveys (LSMS or DHS) can be used to form control groups in matching techniques • Project monitoring data can be used to observe trends • Survey maybe underway in the same areas. This is rarely the case, but piggybacking is theoretically possible
  31. 31. Timing of data collection• 3 main issues to consider: • How many surveys will be run? • Baseline, midterm and end-line • When is the survey starting and for how long? • Seasonality issues need to be considered • What is the recall period adopted in the questionnaire? • Short recall is more reliable but loses information
  32. 32. Cross-section or panel data?• A cross-section survey collects data from a population at a point in time• A panel survey collects data from a population repeated times• Panel data are preferable because they simplify the analysis • But panel data are not always feasible • But attrition can be large and there can be differential attrition
  33. 33. Questionnaire design• Identify the modules that are needed. For example: a household roster, an education module, a consumption module etc.• Look at existing questionnaire designed by other researchers in similar context• Examples can be taken from: • LSMS surveys • DHS surveys • Other resources
  34. 34. Running a survey in practice• Hire a firm with the desired capacity• Ensure enumerators are properly trained and manuals are available• Test the questionnaires many times• Ensure supervision of enumerators in the field• Ensure households collaborate• Obtain ethical approval
  35. 35. Step 8Analyse the data