Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Module 3: Randomization
SHAHID KHANDKER
INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
Setting the Counterfactual
Counterfactual is what would have happened to participants had they not
participated
The same...
Equivalent Group Comparison
(Y2-Y1) is the program effect
Y0 is income level for treated and not-treated before program ...
Statistical Design of Randomization
But finding “equivalent” groups is difficult in practice
Treatment effects using suc...
Calculating Treatment Effects

Ti 1 if treated

Ti  0 if control

Yi(1) = outcome if treated

Yi(0) = outcome if...
ATE vs. TOT
 ATE - can be determined by ensuring external and internal validity of
the randomization design (two-stage pr...
ATE and Counterfactual
ATE  E[Yi(1) Yi(0)]

TOT  E[Yi(1)Yi(0)Ti 1]
Problem: only observe

E[Yi(1)Ti  1]

E[Yi(...
TOT and Counterfactual
ATE  E[Yi(1) Yi(0)]

TOT  E[Yi(1)Yi(0)Ti 1]
Problem: only observe

E[Yi(1)Ti  1]

E[Yi(...
TOT and Intent to Treat (ITT)
However, if there is selection bias in TOT (due to
partial randomization), we obtain the In...
Approaches to Randomization
Pure randomization
Partial randomization
Yi   Ti i
Can be estimated by OLS; estimates ...
Partial Randomization
Conditional exogeneity assumption:

E(i
T
| X,T  t)  E(i
C
| X,T  t)  0
• In practice, apply...
Different methods of Randomization
1. Oversubscription
2. Randomized phase-in
Limited resources; program allocated among
...
Different methods of Randomization
3. Within-group randomization
4. Encouragement design
Even with phase-in, those waitin...
Concerns with Randomization
1. Ethical issues
May be difficult to convince policymakers to
completely randomize an interv...
Concerns with Randomization
2. External validity satisfied? If not, how generalizable are
the results?
• External validity...
Concerns with Randomization
3. Compliance and spillovers
Level of randomization matters - if randomized at
local level, i...
Concerns with Randomization
4. Heterogeneity in program impacts
One implication of ITT is that ultimate program
impact ca...
Concerns with Randomization
4. Heterogeneity in program impacts (cont’d.)
• Solutions: instrumental variables; interaction...
Value of a Baseline
With a baseline, have the ability to examine interactions
between initial conditions and the impact o...
Conclusions
Randomized experiments have advantage of avoiding
selection bias at level of randomization
Following lecture...
Case Studies:
RANDOMIZATION
Statistical Design of Randomization
Problem of counterfactual is what would have happened to the
participants had they no...
Case Study 1:
PROGRESA/Oportunidades
Conditional cash transfer program in Mexico
Overview
Goal: conditional cash transfer program targeting a
number of health and educational outcomes including
malnutri...
Allocation of Funds
2. Support also directly provided to schools and
health centers
Partial participation possible, i.e.,...
Randomization Setup
• Only extreme poor regions were targeted, with a phased-in
randomized strategy across targeted locali...
Randomization Setup
Two-stage setup: regional/village-level, and
household level
• Within localities, households chosen b...
Studies on PROGRESA
• Gertler, 2004: improved child health - also controlled
for socioeconomic characteristics
• Behrman e...
Studies on PROGRESA
• Schultz, 2004: (+) effects on enrollment - check
randomization with pre-program comparisons;
robustn...
Case Study 2:
School Vouchers in Colombia: PACES (Plan
for Increasing Secondary Education
Coverage) Program
Overview
Vouchers (about US$190) covered about half the
cost; renewable based on performance
Program provided 125,000 pu...
How to Measure Schooling Impact?
Angrist et. al., 2002: lottery randomized, so surveyed
lottery winners and losers from t...
How to Measure Schooling Impact?
However, only about 90% of lottery winners ever
used the voucher or any other funding so...
Case Study 3:
Primary School Deworming Project,
Kenya
Deworming in Kenya
Miguel and Kremer, 2004: medical treatment and
health education for intestinal worms in children
75 p...
Randomization Approach
Randomly divided the schools into three equal
sized groups; phased in program over three years
(19...
Problem with Randomization:
Spillover effects
Randomization occurred at school level, but individual
(child) outcomes wou...
Approach
• Examined impacts at the school level, since
deworming program was randomized across schools,
and treatment and ...
Results
• Treated schools: significantly (about 25%) lower
absenteeism rates, although academic test scores did
not improv...
Results
Including externality benefits find that the cost per
additional year of school participation is just US$3.50
Dew...
Ethiopia Case: Cluster-Based Program
Two-stage randomization: (1) Randomly select
clusters for treatment and control amon...
Ethiopia Case: Cluster-Based Program
Two-stage randomization: (1) Randomly select
clusters for treatment and control amon...
Upcoming SlideShare
Loading in …5
×

Randomization (Module 3)

125 views

Published on

The goal of this course is to provide policy analysts and project managers with the tools for evaluating the impact of a project, program or policy. This course provides information on the methods that can be used to measure the impact of a project, program or policy on the well-being of individuals and households. The course addresses the ways in which the results of an impact evaluation may be put to use – such as, to improve the design of projects and programs, as an input into cost-benefit analysis, and as a basis for policy decisions.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Randomization (Module 3)

  1. 1. Module 3: Randomization SHAHID KHANDKER INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
  2. 2. Setting the Counterfactual Counterfactual is what would have happened to participants had they not participated The same person cannot be observed at the same time Challenge is to formulate counterfactual in practice Consider a random distribution of two groups of individuals – one group is treated and other not Both are “equivalent” in that both groups are similar prior to treatment So the difference after treatment is what caused by treatment
  3. 3. Equivalent Group Comparison (Y2-Y1) is the program effect Y0 is income level for treated and not-treated before program intervention Participants Time Income Y2 Y1 Y0 Program Control Impact = (Y2 -Y1)
  4. 4. Statistical Design of Randomization But finding “equivalent” groups is difficult in practice Treatment effects using such a method not generalizable Also effects themselves are not a function of only program itself Statisticians therefore propose two-stage randomization approach 1st stage– select randomly a sample of potential participants from the defined population (representative population) that ensures external validity 2nd stage—randomly assign to treatment and control from this sample, ensuring internal validity
  5. 5. Calculating Treatment Effects  Ti 1 if treated  Ti  0 if control  Yi(1) = outcome if treated  Yi(0) = outcome if control  ATE  E[Yi(1) Yi(0)] Average Treatment Effect (ATE):  TOT  E[Yi(1)Yi(0)Ti 1] But typically can only get Treatment Effect on Treated (TOT):
  6. 6. ATE vs. TOT  ATE - can be determined by ensuring external and internal validity of the randomization design (two-stage process) Stage 1  External validity: sample represents the population Stage 2  Internal validity: measured impact due to program and not other confounding factors  If just internal validity holds (program targeted on a subpopulation): can only estimate TOT consistently
  7. 7. ATE and Counterfactual ATE  E[Yi(1) Yi(0)]  TOT  E[Yi(1)Yi(0)Ti 1] Problem: only observe  E[Yi(1)Ti  1]  E[Yi(0)Ti  0]and E[Yi(1)Ti 1] E[Yi(0)Ti  0] ATE if E[Yi(1)Ti 1]  E[Yi(1)] only in large samples and two-stage randomizationE[Yi(0)Ti  0]  E[Yi(0)]
  8. 8. TOT and Counterfactual ATE  E[Yi(1) Yi(0)]  TOT  E[Yi(1)Yi(0)Ti 1] Problem: only observe  E[Yi(1)Ti  1]  E[Yi(0)Ti  0]and E[Yi(1)Ti 1] E[Yi(0)Ti  0] TOT if (E[Yi(0)Ti 1] E[Yi(0)Ti  0])  0  E[Yi(1)Ti 1] E[Yi(0)Ti 1]  E[Yi(1)Yi(0)Ti 1] TOT Selection Bias
  9. 9. TOT and Intent to Treat (ITT) However, if there is selection bias in TOT (due to partial randomization), we obtain the Intent to Treat (ITT) effect. So we compare between pure randomization and partial randomization Pure randomization: If randomization is done following two step procedure, treated and non-treated units have the same expected outcomes in the absence of treatment, i.e., E(Y(0)|T=1) = E(Y(0)|T=0).
  10. 10. Approaches to Randomization Pure randomization Partial randomization Yi   Ti i Can be estimated by OLS; estimates difference in outcomes of treated and control. OLSˆ  Treatment and control samples chosen randomly, but conditional on some observable characteristics X (e.g., landholding or income). Assuming conditional exogeneity of program placement: is unbiased.  ˆOLS
  11. 11. Partial Randomization Conditional exogeneity assumption:  E(i T | X,T  t)  E(i C | X,T  t)  0 • In practice, apply common-impact model: Then the treatment effect is T  C . T C . Yi T  T  T Xi  i T Treatment (Ti = 1)  Yi C  C  C Xi  i C Control (Ti = 0) Pool two samples:
  12. 12. Different methods of Randomization 1. Oversubscription 2. Randomized phase-in Limited resources; program allocated among random subset of eligible units Controls are eligible areas waiting to get program Example: Lottery for private school vouchers in Colombia (PACES program) Example: PROGRESA/Oportunidades CCT program in Mexico
  13. 13. Different methods of Randomization 3. Within-group randomization 4. Encouragement design Even with phase-in, those waiting may not cooperate - so provide program to subgroups in each eligible area Rather than randomize treatment, randomize annoucement of program  then observe takeup Example: Pratham in India - all schools received program, but random subset asked to apply program in grade 3/others in grade 4 Example: Different interest rate offers by lender in South Africa (Karlan and Zinman, 2007)
  14. 14. Concerns with Randomization 1. Ethical issues May be difficult to convince policymakers to completely randomize an intervention Example: Policies during economic crisis One argument in favor of randomization is that it would help determine which policies work and which don’t, through a scientific approach
  15. 15. Concerns with Randomization 2. External validity satisfied? If not, how generalizable are the results? • External validity = results obtained could be generalizable to other groups or settings  E[Yi(0)Ti  1] E[Yi(0)Ti  0]  E[Yi(1)Ti  1] E[Yi(1)Ti  0] • However, two-stage process almost never applied; is often the best we can obtain TOT  E[Yi(1)Yi(0)Ti 1]
  16. 16. Concerns with Randomization 3. Compliance and spillovers Level of randomization matters - if randomized at local level, individual takeup may still exhibit selection bias (compliance, attrition) • Spillovers: controls may move to project areas, and their outcomes change from exposure to the program • Solution: instrumenting actual program participation by the randomized assignment strategy (“Intent to Treat” estimate, or ITT)
  17. 17. Concerns with Randomization 4. Heterogeneity in program impacts One implication of ITT is that ultimate program impact cannot necessarily be measured as a binary variable (that is, T=1 for an individual participant in a treatment area, and T=0 for an individual in a control area). While a program may be randomized at an aggregate level, there may still be individual selection in the response to treatment.
  18. 18. Concerns with Randomization 4. Heterogeneity in program impacts (cont’d.) • Solutions: instrumental variables; interactions in the regression model between the targeting criteria (education, household expenditures, poverty indices) and the treatment indicator T • Quantile treatment effects can also help examine distributional impacts (across income, for example)
  19. 19. Value of a Baseline With a baseline, have the ability to examine interactions between initial conditions and the impact of the program Useful for testing external validity Treated might have access to similar programs prior to new initiative Opportunity to check that randomization conducted without spillovers Problem: does conducting baseline itself alter the counterfactual? (Giné, Karlan and Zinman, 2008)
  20. 20. Conclusions Randomized experiments have advantage of avoiding selection bias at level of randomization Following lectures: cover non-experimental studies (PSM, DD, IV, RD), which try to replicate randomization as much as possible. Typically, however, only ever observe TOT, and in particular ITT
  21. 21. Case Studies: RANDOMIZATION
  22. 22. Statistical Design of Randomization Problem of counterfactual is what would have happened to the participants had they not participated? Randomization solves this problem through randomization in such a way that participants and nonparticipants have the same probability of participation in a program Statisticians propose two-stage randomization approach 1st stage– select randomly a sample of potential participants from the defined population (representative population) that ensures external validity 2nd stage—randomly assign to treatment and control from this sample, ensuring internal validity
  23. 23. Case Study 1: PROGRESA/Oportunidades Conditional cash transfer program in Mexico
  24. 24. Overview Goal: conditional cash transfer program targeting a number of health and educational outcomes including malnutrition, high infant mortality, high fertility, and school attendance By 2004, 5 million families/25 million individuals; budget ≈ US$2.5 billion or 0.3% of Mexico’s GDP One of the largest randomized interventions by a single country Began in mid-1997 following 1994-95 crisis, targeting rural and marginal urban areas
  25. 25. Allocation of Funds 2. Support also directly provided to schools and health centers Partial participation possible, i.e., in school subsidy initiative if a household sent only a proportion of their children to school 1. CCT to mothers, based on regular school attendance for children and visits to health centers Average benefit received by households ≈ 20% of value of consumption expenditure before program
  26. 26. Randomization Setup • Only extreme poor regions were targeted, with a phased-in randomized strategy across targeted localities Two-stage setup: regional/village-level, and household level Regional level 1/3 of randomly targeted eligible communities delayed entry by 18 months, and remaining 2/3 received program at inception.
  27. 27. Randomization Setup Two-stage setup: regional/village-level, and household level • Within localities, households chosen based on discriminant analysis that used their socieconomic characteristics (obtained from census data) to classify households as poor or nonpoor Household level On average, 78% of households in selected localities were considered eligible, and ≈ 93% eligible households enrolled
  28. 28. Studies on PROGRESA • Gertler, 2004: improved child health - also controlled for socioeconomic characteristics • Behrman et. al., 2006: control for attrition of youth in examining labor outcomes; re-weight post-program observations to have same distribution of X as prior to the attrition Health Labor market outcomes • Skoufias and di Maro, 2007: before-after comparisons of controls and treated over time, to examine effect of CCT on work incentives
  29. 29. Studies on PROGRESA • Schultz, 2004: (+) effects on enrollment - check randomization with pre-program comparisons; robustness checks with probit at child level Schooling Nutrition • Behrman and Hoddinott, 2005: PROGRESA also distributed food supplements to children; some selectivity in distribution in one area, so conducted child-specific fixed effects regressions. Program effect went from insignificant to (+)
  30. 30. Case Study 2: School Vouchers in Colombia: PACES (Plan for Increasing Secondary Education Coverage) Program
  31. 31. Overview Vouchers (about US$190) covered about half the cost; renewable based on performance Program provided 125,000 pupils from poor neighborhoods with vouchers for private secondary school Oversubscription: many vouchers allocated by lottery because number of eligibles exceeded number of available vouchers
  32. 32. How to Measure Schooling Impact? Angrist et. al., 2002: lottery randomized, so surveyed lottery winners and losers from three groups of applicants. Administered a test to subset of surveyed students Marginally significant difference in outcomes for lottery winners
  33. 33. How to Measure Schooling Impact? However, only about 90% of lottery winners ever used the voucher or any other funding source, while 24% of non-winners received other scholarships. Could therefore use lottery as an instrument/ intent to treat impact 50% greater effects on grade completion and reduced repetitions than in simple comparison.
  34. 34. Case Study 3: Primary School Deworming Project, Kenya
  35. 35. Deworming in Kenya Miguel and Kremer, 2004: medical treatment and health education for intestinal worms in children 75 primary schools in one rural district of Kenya, 1998-2002
  36. 36. Randomization Approach Randomly divided the schools into three equal sized groups; phased in program over three years (1998-2000)
  37. 37. Problem with Randomization: Spillover effects Randomization occurred at school level, but individual (child) outcomes would also be of interest Since infections spread easily, strong treatment externalities across treated and nontreated children. Not accounting for such externalities would bias program impact, and randomizing the program within schools therefore not possible.
  38. 38. Approach • Examined impacts at the school level, since deworming program was randomized across schools, and treatment and comparison schools were located sufficiently far away School-level likelihood of spillovers across schools much smaller • Measured size of externality by comparing untreated children in treated schools with control Child-level
  39. 39. Results • Treated schools: significantly (about 25%) lower absenteeism rates, although academic test scores did not improve relative to control schools. • Substantial treatment externalities: untreated children in treated schools had significantly improved health and school participation relative to children in control schools Individual level/externalities School-level
  40. 40. Results Including externality benefits find that the cost per additional year of school participation is just US$3.50 Deworming more cost-effective than subsidies in reducing absenteeism
  41. 41. Ethiopia Case: Cluster-Based Program Two-stage randomization: (1) Randomly select clusters for treatment and control among potential clusters (control will receive treatment later because of limited resources); (2) Randomly select farmers from selected clusters willing to participate in contract farming Oversubscription: Farmers allocated for treatment by lottery because number of willing to accept contract farming may exceed possible numbers to treat with limited resources
  42. 42. Ethiopia Case: Cluster-Based Program Two-stage randomization: (1) Randomly select clusters for treatment and control among potential clusters (control will receive treatment later because of limited resources); (2) Randomly select farmers from selected clusters willing to participate in contract farming Oversubscription: Farmers allocated for treatment by lottery because number of willing to accpt contract farming may exceed possible numbers to treat with limited resources

×