Successfully reported this slideshow.
Upcoming SlideShare
×

# Instrumental Variables, Regression Discontinuity Models, Pipeline Methods (Module 6)

127 views

Published on

The goal of this course is to provide policy analysts and project managers with the tools for evaluating the impact of a project, program or policy. This course provides information on the methods that can be used to measure the impact of a project, program or policy on the well-being of individuals and households. The course addresses the ways in which the results of an impact evaluation may be put to use – such as, to improve the design of projects and programs, as an input into cost-benefit analysis, and as a basis for policy decisions.

Published in: Government & Nonprofit
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Instrumental Variables, Regression Discontinuity Models, Pipeline Methods (Module 6)

1. 1. Module 6 INSTRUMENTAL VARIABLES REGRESSION DISCONTINUITY MODELS PIPELINE METHODS SHAHID KHANDKER INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)
2. 2. 1. Instrumental Variables
3. 3. Why use IV? Two sources of bias: program placement and households self-selection; Examples: bias in formation of income generating/credit networks; bias from targeting programs towards more entrepreneurial class or poor areas; such bias can also be time varying IV allows for unobserved bias in individual participation and / or program placement  Also accounts for errors in measuring participation as well as accounts for time varying endogeneity
4. 4. IV Approach Find a variable (or instrument, “Z”) highly correlated with program placement or participation, but not correlated with unobserved characteristics affecting outcomes. Can have multiple instrumental variables for the treatment variable T
5. 5. Model Step 1: Regress T on instrument vector Z (“First stage regression”):  Ti  Zi  Xi  ui Step 2: Apply predicted values in place of the endogenous variable in outcome equation Yi  (  Zi   Xi  ui ) Xi i   .IV program effect is
6. 6. IV - Main Assumptions (1) corr (Z , T) ≠ 0  Ti   Zi  Xi  ui Yi (  Zi    Xi  ui)  Xi i (2) corr (Z, ) = 0: the exclusion restriction  
7. 7. Sources of IVs: Randomization • Savings program phased in randomly across individuals • Take-up not enforced across targeted sample • Where unobserved bias affected take-up, targeting used as instrument Commitment Savings Product in Philippines(Karlan, Ashraf and Yin, 2006):
8. 8. Sources of IVs: Eligibility Rules Grameen Bank in Bangladesh (Pitt and Khandker, 1998): • Program not allocated randomly • But program required that only households with 50 decimals of land could participate • Program participation is subject to gender-specific
9. 9. • External shocks affecting program placement or participation, but not affecting outcomes Sources of IVs: External Shocks Rainfall and Civil Conflict in Africa (Miguel, Satyanath and Sergenti, 2004): • Examine impact of economic growth on civil conflict across 41 countries in Africa, 1998-99 • Use exogenous variation in rainfall to instrument for economic growth
10. 10. Interpretation of IV Estimate Z does not perfectly capture participation/program targeting will only capture the program effect for a subset of participants    IV impact is therefore similar to ITT; also known as Local Average Treatment Effect (LATE)   
11. 11. Testing for Endogeneity Best way: detailed information on program implementation and participation Examples: baseline survey, discussions with program officials (qualitative information helps as well) Can also help in designing instruments
12. 12. Testing for Endogeneity  Can also do Wu-Hausman test 1. Isolate unobserved heterogeneity affecting T: Regress T on Z and the other exogenous X, and obtain the residuals. 2. Regress Y on X, Z, and residuals. If coefficient on residuals is significant, reject the null that T is exogenous.
13. 13. Problems with IV: Weak Instruments Corr (Z, ) ≠ 0    Biased estimate of program effect If more than one instrument exists (equation is “overidentified,”) can test for this: 1. Estimate the structural equation by 2SLS, and obtain the residuals 2. Regress residuals on X and Z, obtain R2 and calculate test statistic q= (# instruments - # endogenous vars). nR2 ~ q 2 , 3. If reject corr (Z, ) = 0 .nR2  q 2 , 
14. 14. Problems with IV: Weak Instruments Larger standard errors, potentially inconsistent estimates Corr (Z ,T) is low Yi   Ti i  cov(Yi,Zi)  cov[( Ti i),Zi]cov(Ti,Zi)  cov(Yi ,Zi ) cov(Ti ,Zi )  
15. 15. IV with Panel Data  Yit Qit i vit, t 1,...,T, Correlation between unobserved effect and treatment variable T in addressed by differencing  i  Qit Instruments introduced to allow for correlation between some of the regressors in (such as T) and error itZ  Qit vit
16. 16. Conclusions IV relaxes assumption of time-invariant heterogeneity Instruments should be chosen carefully from detailed data on program design and other exogenous factors affecting participation
17. 17. 2. Regression Discontinuity and Pipeline Methods
18. 18. Approaches Discontinuity design (RD): similar to IV - exogenous eligibility rules used to compare participants and nonparticipants Pipeline approach: exploit variation in the timing of program implementation, using as comparison group eligible units that have not yet received the program (e.g., PROGRESA) Examine outcomes for units just above and below cutoff point (so units are comparable)
19. 19. Combining RD and Pipeline Where there is a treatment is allocated based on some exogenous criteria (RD), and potential participants (perhaps for a related program) are awaiting the intervention (Pipeline)
20. 20. Application - RD Score = Si Yi (pre-intervention) + + ++ ++ ++ + + ++ + + + ++ + + * * * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * * * * * * * * * * * * * * * * * * * * * ** ** ** ** ** ** * * * * * * * * * ** ** ** ** * * * * * * * ** * Poor Non-poor + ++ ++ ++ + + ++ + + + ++ + ++ ++ ++ + + ++ + + + ++ + ++ ++ ++ + + ++ + + + ++ + ++ ++ ++ + + ++ + + + ++ + + + + + + + + + + + + + + + + + + + + + + ++ ++ ++ ++ ++ + + + + + + + + + + + ++ ++ ++ + + ++ + + + ++ + + + + s* Outcomes before intervention We need a variable S determining eligibility (such as age, asset holdings, etc.), with an eligibility cutoff of s*. Score = Si Yi (post-intervention) + + ++ ++ ++ + + ++ + + + ++ + * * * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * ** ** ** * * * * * * * ** * * * * * * * * * * * * * * * * * * * * * ** ** ** ** ** ** * * * * * * * * * ** ** ** ** * * * * * * * ** * Poor Non-poor ++ ++ ++ + + ++ + + + ++ + + + + ++ ++ ++ + + ++ + + + ++ + ++ ++ ++ + + ++ + + + ++ + ++ ++ ++ + + ++ + + + ++ + + + + + + + + + + + + + + + + + + + + + + ++ ++ ++ ++ ++ + + + + + + + + + + + ++ ++ ++ + + ++ + + + ++ + + + s* Outcomes after intervention Treatment effect
21. 21. Specification Checks for RD Graphing the predicted treatment effect Plotting the density of the variable determining eligibility around the threshold Plotting the average values of the covariates around the threshold also can provide an indication of specification problems
22. 22. Pipeline Methods Pipeline approach: can be randomized (e.g., PROGRESA) or non-experimental (Jefes y Jefas Program in Argentina) In non-experimental setting, can use matched double- difference approach as discussed in last lecture
23. 23. RD and Pipeline Methods Eligibility criteria combined with phased-in approach Example: pension program awaiting budget expansion Example: local investment, such as road, is the source of additional market improvements, so that individuals around the road boundary would benefit from the future interventions Variation in potential exposure as a function of distance from road can be exploited as a source of identification
24. 24. Overview Advantage of RD: unbiased estimate of treatment effect at the discontinuity Potential problems: • Interpretation, as with IV, is only for subset of relevant sample • Fewer observations to work with • Results are sensitive to functional form, including nonlinear relationships and interactions
25. 25. Case Studies: IV AND REGRESSION DISCONTINUITY
26. 26. How Do IV, RD and Pipeline Methods work? IV exploits exogenous eligibility condition that affects only the treatment but not the outcomes of interest; it allows for both time-invariant and time-varying heterogeneity causing sample selection bias; RD exploits exogenous eligibility rules to compare participants and non-participants around the cut-off point. Pipeline method constructs a comparison group from subjects who are eligible for the program but have not yet received it. Pitt and Khandker (1998) study the impact of group-based microfinance programs in Bangladesh
27. 27. Case Study 1: Grameen Bank and Microcredit in Bangladesh
28. 28. Overview Pitt and Khandker (1998) study the impact of group- based microfinance programs in Bangladesh Objective: to see if gender of participants matters for program impact on household per capita expenditure, school enrollment of children, household labor supply, assets Credit programs examined: BRAC, BRDB, Grameen Bank
29. 29. Identifying Program Impacts Relied on exogenous eligibility conditions based on landholding (specifically, eligibility cut-off of maximum 1/2 acre of land owned) Also relied on fact that only men can participate in men’s credit groups, and likewise for women - easily enforceable, even where land criteria may not be Credit provision can therefore be measured by comparing returns to owning land
30. 30. Data Quasi-experimental dataset from 1991-92 of about 1,800 households across random sample of 29 thanas (87 villages) About 1,540 targeted households from 24 thanas with at least one of the three programs; remaining 5 thanas had none Of targeted households, about 60% were participating in credit programs
31. 31. Data  Household-level: eligible and ineligible households from both types of villages Thus, enough variation in targeting and eligibility to identify impacts:  Village-level: villages with and without program  Individual-level: program participants and nonparticipants across eligible households, in villages with program
32. 32. Instruments and Fixed Effects • Instrument: land-based eligibility rule interacted with household and village-level characteristics • Gender constraint on groups can also identify program impacts across gender • Use village fixed-effects (for example, to account for why some villages have just men-only groups, and other villages have only female groups) Household-level endogeneity Village-level endogeneity
33. 33. Initial Results Using 1991 cross-section, Pitt and Khandker find that when women are the program participants, program credit has a larger impact on household outcomes However, land criteria as an instrument is not strictly enforceable, so instrument efficiency was reduced but tested and found not too inefficient  Increase in annual household expenditure of 18 taka, compared to 11 taka for men  Average poverty reduction of about 5 percentage points
34. 34. Testing Sensitivity of Results with Panel Data Khandker (2005) uses the 1998-99 follow-up survey to the 1991/92 survey to assess sensitivity of the earlier findings Identification strategy same as before, only that participation is now a dynamic variable  Average returns to female borrowing are about 21% in 1998/99  Average poverty reduction lower (about 2%): perhaps diminishing returns to borrowing
35. 35. Case Study 2: Impact of Social Protection in Argentina: A Study Using Pipeline Method
36. 36. Argentina Social Protection Program Glasso and Ravallion (2004) examined the effect of a large social protection program called, Jefes Y Jefas, in Argentina introduced during the 2001 economic crisis. Public safety net provided income to families with dependents for whom main source of earnings lost. However, not all the applicants received government support around the same time; there was a waiting period, which provides a case for applying pipeline method
37. 37. Argentina Social Protection Program Glasso and Ravallion exploited the program design to construct the counterfactual; comparison units were constructed from a subset of applicants who had not yet received the program Participants were matched to comparison observations on the basis of propensity score matching method; Differences in outcomes between these groups provides an estimate of program effect However, the data revealed that the program’s eligibility criteria was not strictly enforced and that 80% of eligibles did not receive the program