What have we learnt from randomized control trials
Upcoming SlideShare
Loading in...5
×
 

What have we learnt from randomized control trials

on

  • 10,561 views

 

Statistics

Views

Total Views
10,561
Views on SlideShare
8,953
Embed Views
1,608

Actions

Likes
0
Downloads
16
Comments
0

33 Embeds 1,608

http://ifad-un.blogspot.com.ar 395
http://ifad-un.blogspot.com 377
http://ifad-un.blogspot.it 363
http://unjobs.org 137
http://betterevaluation.org 83
http://ifad-un.blogspot.co.uk 42
http://ifad-un.blogspot.ca 28
http://ifad-un.blogspot.in 26
http://ifad-un.blogspot.nl 22
http://ifad-un.blogspot.fr 17
http://ifad-un.blogspot.se 14
http://ifad-un.blogspot.ch 12
http://ifad-un.blogspot.com.es 10
http://ifad-un.blogspot.co.at 10
http://ifad-un.blogspot.be 9
http://ifad-un.blogspot.com.au 9
http://ifad-un.blogspot.com.br 8
http://ifad-un.blogspot.mx 8
http://ifad-un.blogspot.de 8
http://beta.betterevaluation.org 7
http://ifad-un.blogspot.co.nz 3
http://ifad-un.blogspot.kr 3
http://ifad-un.blogspot.ie 3
http://ifad-un.blogspot.dk 2
http://ifad-un.blogspot.hk 2
http://www.betterevaluation.org 2
http://ifad-un.blogspot.sg 2
http://ifad-un.blogspot.ru 1
http://ifad-un.blogspot.hu 1
http://ifad-un.blogspot.cz 1
http://ifad-un.blogspot.gr 1
http://ifad-un.blogspot.ro 1
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

What have we learnt from randomized control trials What have we learnt from randomized control trials Presentation Transcript

  • Impact evaluation in 7 or 8 steps
  • Step 1Engage with the stakeholders
  • June mission in Mali• Government interested in a field experiment (RCT) using project and control villages• Government interested in three main issues: o Good governance: how to make sure that children are fed? o Education: what is the impact on attendance rates? o Local economy: how the project benefits small farmers? View slide
  • Step 2Define relevant evaluation questions View slide
  • Impact on education and nutrition• Impact on enrolment and achievements, and what is the role played by school quality?• Does the programme improve attention and cognition?• Does the programme improve nutritional status? Is there catching-up growth?• What is the extent of substitution effects within the household?• What is the impact of the programme on the diet of the poor?
  • Impact on agriculture• What is the impact on small farmers in the short term (incomes) and in the long term (farm investments)?• What is the impact of the intervention on prices and therefore on consumers?• What is the impact on the wider economy at the village and regional levels?
  • Step 3Build a theory of change
  • Overall programme theory
  • Agriculture pathways
  • Education pathways
  • Nutrition pathways
  • Step 4Define the indicators
  • Welfare outcomes• Welfare outcomes are the MDGs metrics
  • Intermediate welfare outcomes• Sometimes welfare outcomes cannot be observed because: • Occur in the very long term (example increase in employment) • Are not observable (example maternal mortality)• Intermediate outcomes are used in these cases: proxy indicators of the final outcomes along the causal chain
  • Education: outcome indicators• Enrolment, attendance rates and drop-outs• Achievement tests (test scores on maths and language)• Attention and cognition
  • Nutrition: outcome indicators• Anthropometric measurement.• Measures of diet composition
  • Food security: outcome indicators• Full income questionnaire for farmers will provide data on:  marketed surplus  Farm profits  Technology and capitalisation  Input use
  • Step 5The evaluation design
  • The Mali evaluation design• The government is expanding the intervention to 60 of the most vulnerable communes• A commune is an administrative unit comprising 5 to 15 villages and a similar number of schools• The groups considered by the study are:  Control group (no intervention)  Standard school feeding  Home grown school feeding
  • Level 1 comparison: school feeding-control group• MOE needs to know the impact of the intervention on educational indicators• First comparison is between any school feeding and a control group with no intervention• Outcomes of interests are:  Enrolment rates  Learning achievements  Attention and cognition  Nutritional outcomes
  • Level 2 comparison: school feeding /home-grown school feeding• The second comparison is between the conventional government programme and the ‘home grown’ programme• Outcomes of interest:  Small farmers’ incomes  Overall programme performance
  • Selection of schools and communes• In each of the 60 communes Mayors will select two school for the intervention of which one will be randomly assigned to the programme (pair-matching design or stratification by commune). A protocol is designed to avoid contamination.• Of the 60 communes assigned to the programme, 30 will be randomly assigned to the home grown component
  • Step 6Set the sample size
  • Sample size• We collected data from 30 households in each village:  20 households with children aged 5 to12  10 farmer households (with or without children)• Sample size is:  1,200 farmer households  3,600 farmer and non-farmer households  6,000 to 7,000 children of primary school age
  • Calculate the sample size• The size needs to be sufficiently large to detect the expected effect of the programme• Detecting sample size is guesswork and the goal is to produce lower and upper bounds rather than exact samples• There is statistical software which is designed to do this
  • Power• You need a powerful sample to detect impact• The power of your sample will be a function of • The expected programme impact (increasing) • The variance of the outcome of interest (decreasing) • The homogeneity within clusters (decreasing) • The desired level of Type I error (decreasing)
  • Step 7Run a household survey
  • main issues • Choose the unit of observation • Find existing datasets and surveys • Establish the timing of data collection • Establish whether collecting cross-section or panel data • Establish the number of surveys • Design the questionnaires • Administer the survey
  • Choose the unit of observation• The preferred level of observation is the ‘individual’ or the ‘household’ • Note that individuals are difficult to interview (ex: consumption data or panel) • Household is the most frequent unit of observation• Observations can also be made at cluster level (village, school, clinic or other) • Note that sample size will be small • Data is difficult to collect (who is interviewed?)
  • Data scoping• Before starting collecting any data you should first investigate what data and surveys are available: • Census data can be used to frame the sample or to extract control variables • Existing household surveys (LSMS or DHS) can be used to form control groups in matching techniques • Project monitoring data can be used to observe trends • Survey maybe underway in the same areas. This is rarely the case, but piggybacking is theoretically possible
  • Timing of data collection• 3 main issues to consider: • How many surveys will be run? • Baseline, midterm and end-line • When is the survey starting and for how long? • Seasonality issues need to be considered • What is the recall period adopted in the questionnaire? • Short recall is more reliable but loses information
  • Cross-section or panel data?• A cross-section survey collects data from a population at a point in time• A panel survey collects data from a population repeated times• Panel data are preferable because they simplify the analysis • But panel data are not always feasible • But attrition can be large and there can be differential attrition
  • Questionnaire design• Identify the modules that are needed. For example: a household roster, an education module, a consumption module etc.• Look at existing questionnaire designed by other researchers in similar context• Examples can be taken from: • LSMS surveys • DHS surveys • Other resources
  • Running a survey in practice• Hire a firm with the desired capacity• Ensure enumerators are properly trained and manuals are available• Test the questionnaires many times• Ensure supervision of enumerators in the field• Ensure households collaborate• Obtain ethical approval
  • Step 8Analyse the data