Successfully reported this slideshow.
Upcoming SlideShare
×

# Randomization (Module 3)

125 views

Published on

The goal of this course is to provide policy analysts and project managers with the tools for evaluating the impact of a project, program or policy. This course provides information on the methods that can be used to measure the impact of a project, program or policy on the well-being of individuals and households. The course addresses the ways in which the results of an impact evaluation may be put to use – such as, to improve the design of projects and programs, as an input into cost-benefit analysis, and as a basis for policy decisions.

Published in: Government & Nonprofit
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Randomization (Module 3)

1. 1. Module 3: Randomization SHAHID KHANDKER INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
2. 2. Setting the Counterfactual Counterfactual is what would have happened to participants had they not participated The same person cannot be observed at the same time Challenge is to formulate counterfactual in practice Consider a random distribution of two groups of individuals – one group is treated and other not Both are “equivalent” in that both groups are similar prior to treatment So the difference after treatment is what caused by treatment
3. 3. Equivalent Group Comparison (Y2-Y1) is the program effect Y0 is income level for treated and not-treated before program intervention Participants Time Income Y2 Y1 Y0 Program Control Impact = (Y2 -Y1)
4. 4. Statistical Design of Randomization But finding “equivalent” groups is difficult in practice Treatment effects using such a method not generalizable Also effects themselves are not a function of only program itself Statisticians therefore propose two-stage randomization approach 1st stage– select randomly a sample of potential participants from the defined population (representative population) that ensures external validity 2nd stage—randomly assign to treatment and control from this sample, ensuring internal validity
5. 5. Calculating Treatment Effects  Ti 1 if treated  Ti  0 if control  Yi(1) = outcome if treated  Yi(0) = outcome if control  ATE  E[Yi(1) Yi(0)] Average Treatment Effect (ATE):  TOT  E[Yi(1)Yi(0)Ti 1] But typically can only get Treatment Effect on Treated (TOT):
6. 6. ATE vs. TOT  ATE - can be determined by ensuring external and internal validity of the randomization design (two-stage process) Stage 1  External validity: sample represents the population Stage 2  Internal validity: measured impact due to program and not other confounding factors  If just internal validity holds (program targeted on a subpopulation): can only estimate TOT consistently
7. 7. ATE and Counterfactual ATE  E[Yi(1) Yi(0)]  TOT  E[Yi(1)Yi(0)Ti 1] Problem: only observe  E[Yi(1)Ti  1]  E[Yi(0)Ti  0]and E[Yi(1)Ti 1] E[Yi(0)Ti  0] ATE if E[Yi(1)Ti 1]  E[Yi(1)] only in large samples and two-stage randomizationE[Yi(0)Ti  0]  E[Yi(0)]
8. 8. TOT and Counterfactual ATE  E[Yi(1) Yi(0)]  TOT  E[Yi(1)Yi(0)Ti 1] Problem: only observe  E[Yi(1)Ti  1]  E[Yi(0)Ti  0]and E[Yi(1)Ti 1] E[Yi(0)Ti  0] TOT if (E[Yi(0)Ti 1] E[Yi(0)Ti  0])  0  E[Yi(1)Ti 1] E[Yi(0)Ti 1]  E[Yi(1)Yi(0)Ti 1] TOT Selection Bias
9. 9. TOT and Intent to Treat (ITT) However, if there is selection bias in TOT (due to partial randomization), we obtain the Intent to Treat (ITT) effect. So we compare between pure randomization and partial randomization Pure randomization: If randomization is done following two step procedure, treated and non-treated units have the same expected outcomes in the absence of treatment, i.e., E(Y(0)|T=1) = E(Y(0)|T=0).
10. 10. Approaches to Randomization Pure randomization Partial randomization Yi   Ti i Can be estimated by OLS; estimates difference in outcomes of treated and control. OLSˆ  Treatment and control samples chosen randomly, but conditional on some observable characteristics X (e.g., landholding or income). Assuming conditional exogeneity of program placement: is unbiased.  ˆOLS
11. 11. Partial Randomization Conditional exogeneity assumption:  E(i T | X,T  t)  E(i C | X,T  t)  0 • In practice, apply common-impact model: Then the treatment effect is T  C . T C . Yi T  T  T Xi  i T Treatment (Ti = 1)  Yi C  C  C Xi  i C Control (Ti = 0) Pool two samples:
12. 12. Different methods of Randomization 1. Oversubscription 2. Randomized phase-in Limited resources; program allocated among random subset of eligible units Controls are eligible areas waiting to get program Example: Lottery for private school vouchers in Colombia (PACES program) Example: PROGRESA/Oportunidades CCT program in Mexico
13. 13. Different methods of Randomization 3. Within-group randomization 4. Encouragement design Even with phase-in, those waiting may not cooperate - so provide program to subgroups in each eligible area Rather than randomize treatment, randomize annoucement of program  then observe takeup Example: Pratham in India - all schools received program, but random subset asked to apply program in grade 3/others in grade 4 Example: Different interest rate offers by lender in South Africa (Karlan and Zinman, 2007)
14. 14. Concerns with Randomization 1. Ethical issues May be difficult to convince policymakers to completely randomize an intervention Example: Policies during economic crisis One argument in favor of randomization is that it would help determine which policies work and which don’t, through a scientific approach
15. 15. Concerns with Randomization 2. External validity satisfied? If not, how generalizable are the results? • External validity = results obtained could be generalizable to other groups or settings  E[Yi(0)Ti  1] E[Yi(0)Ti  0]  E[Yi(1)Ti  1] E[Yi(1)Ti  0] • However, two-stage process almost never applied; is often the best we can obtain TOT  E[Yi(1)Yi(0)Ti 1]
16. 16. Concerns with Randomization 3. Compliance and spillovers Level of randomization matters - if randomized at local level, individual takeup may still exhibit selection bias (compliance, attrition) • Spillovers: controls may move to project areas, and their outcomes change from exposure to the program • Solution: instrumenting actual program participation by the randomized assignment strategy (“Intent to Treat” estimate, or ITT)
17. 17. Concerns with Randomization 4. Heterogeneity in program impacts One implication of ITT is that ultimate program impact cannot necessarily be measured as a binary variable (that is, T=1 for an individual participant in a treatment area, and T=0 for an individual in a control area). While a program may be randomized at an aggregate level, there may still be individual selection in the response to treatment.
18. 18. Concerns with Randomization 4. Heterogeneity in program impacts (cont’d.) • Solutions: instrumental variables; interactions in the regression model between the targeting criteria (education, household expenditures, poverty indices) and the treatment indicator T • Quantile treatment effects can also help examine distributional impacts (across income, for example)
19. 19. Value of a Baseline With a baseline, have the ability to examine interactions between initial conditions and the impact of the program Useful for testing external validity Treated might have access to similar programs prior to new initiative Opportunity to check that randomization conducted without spillovers Problem: does conducting baseline itself alter the counterfactual? (Giné, Karlan and Zinman, 2008)
20. 20. Conclusions Randomized experiments have advantage of avoiding selection bias at level of randomization Following lectures: cover non-experimental studies (PSM, DD, IV, RD), which try to replicate randomization as much as possible. Typically, however, only ever observe TOT, and in particular ITT
21. 21. Case Studies: RANDOMIZATION
22. 22. Statistical Design of Randomization Problem of counterfactual is what would have happened to the participants had they not participated? Randomization solves this problem through randomization in such a way that participants and nonparticipants have the same probability of participation in a program Statisticians propose two-stage randomization approach 1st stage– select randomly a sample of potential participants from the defined population (representative population) that ensures external validity 2nd stage—randomly assign to treatment and control from this sample, ensuring internal validity
23. 23. Case Study 1: PROGRESA/Oportunidades Conditional cash transfer program in Mexico
24. 24. Overview Goal: conditional cash transfer program targeting a number of health and educational outcomes including malnutrition, high infant mortality, high fertility, and school attendance By 2004, 5 million families/25 million individuals; budget ≈ US\$2.5 billion or 0.3% of Mexico’s GDP One of the largest randomized interventions by a single country Began in mid-1997 following 1994-95 crisis, targeting rural and marginal urban areas
25. 25. Allocation of Funds 2. Support also directly provided to schools and health centers Partial participation possible, i.e., in school subsidy initiative if a household sent only a proportion of their children to school 1. CCT to mothers, based on regular school attendance for children and visits to health centers Average benefit received by households ≈ 20% of value of consumption expenditure before program
26. 26. Randomization Setup • Only extreme poor regions were targeted, with a phased-in randomized strategy across targeted localities Two-stage setup: regional/village-level, and household level Regional level 1/3 of randomly targeted eligible communities delayed entry by 18 months, and remaining 2/3 received program at inception.
27. 27. Randomization Setup Two-stage setup: regional/village-level, and household level • Within localities, households chosen based on discriminant analysis that used their socieconomic characteristics (obtained from census data) to classify households as poor or nonpoor Household level On average, 78% of households in selected localities were considered eligible, and ≈ 93% eligible households enrolled
28. 28. Studies on PROGRESA • Gertler, 2004: improved child health - also controlled for socioeconomic characteristics • Behrman et. al., 2006: control for attrition of youth in examining labor outcomes; re-weight post-program observations to have same distribution of X as prior to the attrition Health Labor market outcomes • Skoufias and di Maro, 2007: before-after comparisons of controls and treated over time, to examine effect of CCT on work incentives
29. 29. Studies on PROGRESA • Schultz, 2004: (+) effects on enrollment - check randomization with pre-program comparisons; robustness checks with probit at child level Schooling Nutrition • Behrman and Hoddinott, 2005: PROGRESA also distributed food supplements to children; some selectivity in distribution in one area, so conducted child-specific fixed effects regressions. Program effect went from insignificant to (+)
30. 30. Case Study 2: School Vouchers in Colombia: PACES (Plan for Increasing Secondary Education Coverage) Program
31. 31. Overview Vouchers (about US\$190) covered about half the cost; renewable based on performance Program provided 125,000 pupils from poor neighborhoods with vouchers for private secondary school Oversubscription: many vouchers allocated by lottery because number of eligibles exceeded number of available vouchers
32. 32. How to Measure Schooling Impact? Angrist et. al., 2002: lottery randomized, so surveyed lottery winners and losers from three groups of applicants. Administered a test to subset of surveyed students Marginally significant difference in outcomes for lottery winners
33. 33. How to Measure Schooling Impact? However, only about 90% of lottery winners ever used the voucher or any other funding source, while 24% of non-winners received other scholarships. Could therefore use lottery as an instrument/ intent to treat impact 50% greater effects on grade completion and reduced repetitions than in simple comparison.
34. 34. Case Study 3: Primary School Deworming Project, Kenya
35. 35. Deworming in Kenya Miguel and Kremer, 2004: medical treatment and health education for intestinal worms in children 75 primary schools in one rural district of Kenya, 1998-2002
36. 36. Randomization Approach Randomly divided the schools into three equal sized groups; phased in program over three years (1998-2000)
37. 37. Problem with Randomization: Spillover effects Randomization occurred at school level, but individual (child) outcomes would also be of interest Since infections spread easily, strong treatment externalities across treated and nontreated children. Not accounting for such externalities would bias program impact, and randomizing the program within schools therefore not possible.
38. 38. Approach • Examined impacts at the school level, since deworming program was randomized across schools, and treatment and comparison schools were located sufficiently far away School-level likelihood of spillovers across schools much smaller • Measured size of externality by comparing untreated children in treated schools with control Child-level
39. 39. Results • Treated schools: significantly (about 25%) lower absenteeism rates, although academic test scores did not improve relative to control schools. • Substantial treatment externalities: untreated children in treated schools had significantly improved health and school participation relative to children in control schools Individual level/externalities School-level
40. 40. Results Including externality benefits find that the cost per additional year of school participation is just US\$3.50 Deworming more cost-effective than subsidies in reducing absenteeism
41. 41. Ethiopia Case: Cluster-Based Program Two-stage randomization: (1) Randomly select clusters for treatment and control among potential clusters (control will receive treatment later because of limited resources); (2) Randomly select farmers from selected clusters willing to participate in contract farming Oversubscription: Farmers allocated for treatment by lottery because number of willing to accept contract farming may exceed possible numbers to treat with limited resources
42. 42. Ethiopia Case: Cluster-Based Program Two-stage randomization: (1) Randomly select clusters for treatment and control among potential clusters (control will receive treatment later because of limited resources); (2) Randomly select farmers from selected clusters willing to participate in contract farming Oversubscription: Farmers allocated for treatment by lottery because number of willing to accpt contract farming may exceed possible numbers to treat with limited resources