Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Module 5
DOUBLE DIFFERENCE (DD) METHODS
SHAHID KHANDKER
INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
DD and the Missing Data Problem
Resolves problem of counterfactual by observing
participants and non-participants in pre-...
DD and the Missing Data Problem
Resolves problem of counterfactual by observing
participants and non-participants in pre-...
DD with Randomization
Iron Supplements in Indonesia
(Thomas et. al., 2004):
Randomized iron supplements to individuals in...
Rural Roads in Bangladesh
(Khandker et. al., 2008):
Two rural road paving projects (not randomized)
Quasi-experimental h...
Two specifications
DD - Regression Framework

(1) Yit   Ti1t  Ti1  t it,   DD

(2) Yit = Tit Xit ii...
Two specifications
DD - Regression Framework
DD E(Y1
T
Y0
T
|T1 1)  E(Y1
C
Y0
C
|T1  0)

 [(      )(  ...
Two specifications
DD - Regression Framework
 Yit = Tit Xit  it

Yit = Tit Xit iit,   DD(2)
Also kno...
• Relaxes the assumption of conditional exogeneity; tractable
approach
Pros and Cons of DD
Advantages
• Is time-invariant ...
Time-invariant bias a valid assumption?
t=0 t=1
Participants
Time
Income
Y2
Y1
Y0
Control
Bias
 Parallel trend assumption...
Comparability across project and control
Initial (pre-program) area conditions may bias targeting
Poor-Area Program in Ch...
Comparability Across Project and Control
DD can also be biased if project and control areas are
not comparable for other ...
PSM and DD

DDi  (Y1i
T
 Y0i
T
) Wijj
 (Y1j
C
 Y0j
C
)
= weight given to the jth control area matched
to treatment ...
Equivalently, regression-based DD-PSM estimator
(Hirano and Imbens, 2003):

Yit   Ti it ,   DD
Weights equal = ...
Key assumption of PSM-DD: selection bias only function of
observed covariates in baseline
PSM and DD
If unobserved pre-p...
What if a baseline is not available?
Triple Difference
Example: a program set up quickly during an economic
crisis.
Step 1: entirely separate control groups after program
intervention (i.e., a set of untreated observations in
treated and ...
Conclusions
DD, as compared to PSM, assume the presence of
unobserved heterogeneity in participation, but that such
facto...
Case Studies:
DOUBLE DIFFERENCE (DD) ESTIMATION
How does DD work?
DD assumes that observed changes for non-participants
provide the counterfactual for participants
Thus...
Case Study 1:
Southwest Poverty Reduction
Program in China
Overview
National level: Central Government in China introduced
large poverty reduction initiative in 1986 for 327
‘‘nati...
(1) Household data for sample of 4 contiguous
provinces in southern and southwest China
targeted by program, 1985–90
Data
...
Unconditional comparison of areas covered by
program and those not covered
Initial Results - Simple DD
Program was able ...
DD and Initial Conditions
Jalan and Ravallion test this prediction through a
GMM-time series model for household consumpt...
Results
Program effects are indeed influenced by initial
household and community wealth
Dropping initial area conditions ...
Conclusions
Failing to control for spatial differences in growth
process can lead to a substantial underestimation of the...
Case Study 2:
Rural Road Development
in Asia
Overview
Khandker et. al., 2008
an de Walle and Mu, 2008
• Two rural road paving projects (RDP and RRMIMP) in
Bangladesh -...
• RDP: 26 project and 12 control villages = 1,075
households each year. RRMIMP: 12 project and 2
control villages = 872 ho...
Accounting for Initial Area Characteristics
Khandker et. al., 2008
Initial area characteristics included:
• # banks, schoo...
Accounting for Initial Area Characteristics
van de Walle and Mu, 2008
• Since access to large number of project and contro...
Results
Both studies found significant impact of rural roads on
short-run outcomes: schooling enrollment, increased
agric...
Results - Initial Conditions
Both studies found that project impact strengthened for
most outcomes, after accounting for ...
Results
Note that both studies assume program placement is
correlated with observed initial area characteristics
Case Study 3:
Revisiting Trabajar in Argentina
Overview
Trabajar: workfare program set up during 1997 crisis
No baseline; only data after program intervention
Ravalli...
Triple-Difference (DDD)
Without baseline, how does one calculate treatment
effect?
Ravallion et. al. therefore calculate...
Triple-Difference: Need for Third
Comparison Group
Without matched group of non-participants, simple DD
between stayers a...
Triple-Difference: Need for Third
Comparison Group
However, stayers may be less likely to have better earnings
opportunit...
Triple-Difference: Steps
DDD = DD for stayers - DD for leavers
Steps:
3. Calculate A-B
1. : DD for stayers (relative
to no...
Triple-Difference: Underlying Assumptions
(1) no selection bias in leaving program
(2) no current income gains to ex-parti...
Upcoming SlideShare
Loading in …5
×

Double Difference Methods (Module 5)

116 views

Published on

The goal of this course is to provide policy analysts and project managers with the tools for evaluating the impact of a project, program or policy. This course provides information on the methods that can be used to measure the impact of a project, program or policy on the well-being of individuals and households. The course addresses the ways in which the results of an impact evaluation may be put to use – such as, to improve the design of projects and programs, as an input into cost-benefit analysis, and as a basis for policy decisions.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Double Difference Methods (Module 5)

  1. 1. Module 5 DOUBLE DIFFERENCE (DD) METHODS SHAHID KHANDKER INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
  2. 2. DD and the Missing Data Problem Resolves problem of counterfactual by observing participants and non-participants in pre- and post- intervention periods DD  E(Y1 T  Y0 T T1  1) E(Y1 C  Y0 C T1  0) Can be used with panel or cross-section data; all that is needed are four groups (two from treatment, two from control)
  3. 3. DD and the Missing Data Problem Resolves problem of counterfactual by observing participants and non-participants in pre- and post- intervention periods Impact Participants Time Income Y4 Y2 Y0 Program Control Y1 Y3 DD= (Y4 – Y0) – ( Y3 – Y1) Main assumption to identify program effect: time invariant unobserved heterogeneity
  4. 4. DD with Randomization Iron Supplements in Indonesia (Thomas et. al., 2004): Randomized iron supplements to individuals in mostly agricultural households; half respondents receiving treatment, and controls received placebo Baseline conducted before the intervention Baseline also useful in examining compliance with the intervention -- comparing changes in outcomes among treatment subjects relative to controls
  5. 5. Rural Roads in Bangladesh (Khandker et. al., 2008): Two rural road paving projects (not randomized) Quasi-experimental household panel dataset surveying project and control villages before and after program implementation Project and control areas shared similar socioeconomic and community-level characteristics prior to program implementation; control areas also targets for future rounds of program DD without Randomization
  6. 6. Two specifications DD - Regression Framework  (1) Yit   Ti1t  Ti1  t it,   DD  (2) Yit = Tit Xit iit,   DD With two time periods, (1) and (2) are equivalent
  7. 7. Two specifications DD - Regression Framework DD E(Y1 T Y0 T |T1 1)  E(Y1 C Y0 C |T1  0)   [(      )(  )]  [(  )] If only participant, before & after:  (  ) If participant & control, but no baseline:  (  )     Yit   Ti1t  Ti1  t it (1)
  8. 8. Two specifications DD - Regression Framework  Yit = Tit Xit  it  Yit = Tit Xit iit,   DD(2) Also known as panel fixed-effects; time-invariant heterogeneity differenced out
  9. 9. • Relaxes the assumption of conditional exogeneity; tractable approach Pros and Cons of DD Advantages • Is time-invariant bias a valid assumption? • Comparability of control and project areas pre-intervention: program placement correlated with initial area characteristics Concerns
  10. 10. Time-invariant bias a valid assumption? t=0 t=1 Participants Time Income Y2 Y1 Y0 Control Bias  Parallel trend assumption  DD overestimates impact  DD underestimates impact DD impact Participants Time Income Y2 Y1 Y0 Program DD Impact: = (Y2 -Y0)- (Y1-Y0) = (Y2 -Y1) Control DD assumes unobserved bias is time invariant (parallel trend assumption) Otherwise, DD over/under-estimates impact
  11. 11. Comparability across project and control Initial (pre-program) area conditions may bias targeting Poor-Area Program in China (Jalan and Ravallion, 1998): • Changes over time  function of initial conditions that also influence program placement. • Corrected bias in DD by controlling for area characteristics that initially attracted program. • Found significant longer-term impacts while none had been evident in the standard DD estimator.
  12. 12. Comparability Across Project and Control DD can also be biased if project and control areas are not comparable for other reasons Example: schooling enrollment program - if control areas selected initially much further away from local schools than targeted areas, DD would overestimate the program impact on participating localities One solution: run PSM on base year, then apply DD on units that remain in the common support
  13. 13. PSM and DD  DDi  (Y1i T  Y0i T ) Wijj  (Y1j C  Y0j C ) = weight given to the jth control area matched to treatment area i  Wij DD PS-matched estimator:
  14. 14. Equivalently, regression-based DD-PSM estimator (Hirano and Imbens, 2003):  Yit   Ti it ,   DD Weights equal = 1 for treated and for the comparison communes.  P ^ (X)/(1 P ^ (X)) PSM and DD
  15. 15. Key assumption of PSM-DD: selection bias only function of observed covariates in baseline PSM and DD If unobserved pre-program characteristics affect changes in outcomes as well as program placement, even PSM-DD biased
  16. 16. What if a baseline is not available? Triple Difference Example: a program set up quickly during an economic crisis.
  17. 17. Step 1: entirely separate control groups after program intervention (i.e., a set of untreated observations in treated and non-treated areas) Triple Difference Step 2: additional difference from the first experiment would then be taken from the change in this control sample Method would therefore require data on multiple years after program intervention, even though baseline data were not available.
  18. 18. Conclusions DD, as compared to PSM, assume the presence of unobserved heterogeneity in participation, but that such factors are time-invariant. Where simple DD may be biased: • DD with initial conditions • DD-PSM • Triple Difference
  19. 19. Case Studies: DOUBLE DIFFERENCE (DD) ESTIMATION
  20. 20. How does DD work? DD assumes that observed changes for non-participants provide the counterfactual for participants Thus, DD resolves problem of counterfactual by observing participants and non-participants in pre- and post-intervention periods The critical assumption with DD is, however, that unobserved heterogeneity is fixed over time
  21. 21. Case Study 1: Southwest Poverty Reduction Program in China
  22. 22. Overview National level: Central Government in China introduced large poverty reduction initiative in 1986 for 327 ‘‘national-poor counties’’ Province level: additional counties identified as ‘‘provincial- poor’’ on relative poverty criteria Aid included subsidized credit for village projects, ‘‘food for work’’ programs, raising county governments’ budgets
  23. 23. (1) Household data for sample of 4 contiguous provinces in southern and southwest China targeted by program, 1985–90 Data (2) Field interviews with local program officials and households, 1994–95 Total sample: balanced panel, 6651 households surveyed over each of 6 years Jalan and Ravallion (1998) sought to determine program impact using panel data:
  24. 24. Unconditional comparison of areas covered by program and those not covered Initial Results - Simple DD Program was able to select poorer areas, but did not improve their position relative to untargeted areas
  25. 25. DD and Initial Conditions Jalan and Ravallion test this prediction through a GMM-time series model for household consumption growth, including initial area conditions on the right hand side. Use second and higher lags of consumption as instruments for lagged consumption to obtain consistent estimates.
  26. 26. Results Program effects are indeed influenced by initial household and community wealth Dropping initial area conditions led to national program effect losing significance completely; provincial program effects changed sign and became slightly negative. Overall, found significant longer-term impacts than those obtained using simple fixed-effects methods.
  27. 27. Conclusions Failing to control for spatial differences in growth process can lead to a substantial underestimation of the welfare gains from program.
  28. 28. Case Study 2: Rural Road Development in Asia
  29. 29. Overview Khandker et. al., 2008 an de Walle and Mu, 2008 • Two rural road paving projects (RDP and RRMIMP) in Bangladesh - household-level • Rural road rehabilitation in Vietnam - commune-level Both studies: • Surveyed project and control areas • Not randomized • Program correlated w/ initial area conditions
  30. 30. • RDP: 26 project and 12 control villages = 1,075 households each year. RRMIMP: 12 project and 2 control villages = 872 households each year. Accounting for initial area characteristics Khandker et. al., 2008 • Panel survey covering two periods: before project (mid- 1990s) and 5 years later, after project • OLS-first difference model over the two periods, including initial area characteristics on right hand side.
  31. 31. Accounting for Initial Area Characteristics Khandker et. al., 2008 Initial area characteristics included: • # banks, schools and hospitals serving village • Distance from village to nearest paved road • Average short-term interest rate in village • # active microfinance institutions in village
  32. 32. Accounting for Initial Area Characteristics van de Walle and Mu, 2008 • Since access to large number of project and control communes  PSM-DD approach • 94 project and 95 control communes over 3 periods: baseline in 1997, as well as in 2001 and 2003 • Project impact then estimated by comparing change in outcomes for project communes with that for matched comparison communes
  33. 33. Results Both studies found significant impact of rural roads on short-run outcomes: schooling enrollment, increased agricultural production van de Walle and Mu also examined long-term effects: outcomes like market expansion and availability of non- food goods took longer to emerge Markets, for example, developed in about 10% more project than control communes after 7 years
  34. 34. Results - Initial Conditions Both studies found that project impact strengthened for most outcomes, after accounting for initial area heterogeneity Khandker et. al.: sign of effect didn’t change on household expenditure, labor market outcomes, schooling; but magnitude of effect greater van de Walle and Mu: market impacts greater if commune is initially poorly developed
  35. 35. Results Note that both studies assume program placement is correlated with observed initial area characteristics
  36. 36. Case Study 3: Revisiting Trabajar in Argentina
  37. 37. Overview Trabajar: workfare program set up during 1997 crisis No baseline; only data after program intervention Ravallion et. al. (2005): examine impacts on income for “stayers” versus “leavers” in program
  38. 38. Triple-Difference (DDD) Without baseline, how does one calculate treatment effect? Ravallion et. al. therefore calculate the difference in incomes for participants leaving the program with those still participating, after netting out aggregate economy- wide changes by using matched group of non- participants.
  39. 39. Triple-Difference: Need for Third Comparison Group Without matched group of non-participants, simple DD between stayers and leavers = unbiased if each group’s income opportunities, had program not existed, is same for each. namely, the counterfactual
  40. 40. Triple-Difference: Need for Third Comparison Group However, stayers may be less likely to have better earnings opportunities outside the program than those who dropped out. • As a result, a DD estimate comparing just these two groups will underestimate the program impact.
  41. 41. Triple-Difference: Steps DDD = DD for stayers - DD for leavers Steps: 3. Calculate A-B 1. : DD for stayers (relative to non-participants from national survey)  A  (Y 2 T Y1 T )(Y 2 C Y1 C ) Di2 1  2. : DD for leaversB  (Y 2 T Y1 T )(Y 2 C Y1 C ) Di2 0  Let Dit=participation in time t. Di2=1 if still participating in period 2 (stayer); Di2=0 if leaver
  42. 42. Triple-Difference: Underlying Assumptions (1) no selection bias in leaving program (2) no current income gains to ex-participants To interpret the triple-difference estimate as a measure of average gains to participants, need to assume:

×