DID.pdf

5. Difference-in Difference (DID): Baics
Difference-in-Differences (DID, or DD) is a statistical
technique which attempts to mimic an experimental
research design using observational study data.
It calculates the effect of a treatment on an outcome by
comparing the average change over time in the outcome
variable for the treatment group to the average change
over time for the control group.

Consider the case that a NGO select some schools (in a
non random manner) to offer textbooks in a less
developed region, and we want to estimate the impact of
the treatment on test scores.
At date t = 0, a baseline (or pre-treatment) survey is
conducted in a set of different schools.
Just after this baseline survey, some of the schools are
selected non-randomly for being treated (i.e: for getting
access to textbooks).
At date t = 1, a post-treatment survey is conducted in
both the set of treated and untreated schools.

The baseline survey allows to get information on E(YC
i0|T)
and on E(YC
i0|C), the average test score in period 0 in
schools that are destined to be treated and in schools
that will remain untreated respectively.
The post-treatment survey allows to get information on
E(YT
i1|T) and on E(YC
i1|C), the average test score in
period 1 in schools that have been treated right after
period 0 and in schools that belong to the control group
respectively.

5. Difference-in Difference (DID): Illustration

DID; Estimator
The DID estimator is given by:
DD = [E(YT
i1|T) − E(YC
i1|C)] − [E(YC
i0|T) − E(YC
i0|C)]
= [E(YT
i1|T) − E(YC
i0|T)] − [E(YC
i1|C) − E(YC
i0|C)]
= (T1 − T0) − (C1 − C0)
= (CTVCT + TE) − CTVCC,
where CTVCT and CTVCC stand for ”changes in
time-varying characteristics” in the treatment and in the
control group respectively, and TE stands for the
”treatment effect”.

Parallel Trend Assumption: Illustration

DD Assumption
It is critical to keep in mind that, although the treatment
is not orthogonal to individual characteristics when it is
introduced, in our example, the schools that will receive
the treatment differ from those who will stay in the
control group, this treatment should not be expected by
the schools at date t = 0.
Put differently, the treatment should not already affect
schools test scores in the control and in the treatment
group in the pre-treatment period.

DD Regression
The regression counterpart to obtain DD is given by:
Yi = α + βTreati + γPost + δ(Treati ∗ Post) + X′
iσ + ui
where
Treat is equal to 1 if the school belongs to the
treatment group, and is equal to 0 if they belong to the
control group;
Post is equal to 1 if the period is post-treatment, and is
equal to 0 if the period is pre-treatment.
Xi is a vector of other covariates, and σ is the vector of
corresponding coefficients.
ui is the error term.
The OLS estimation of δ captures DD (proof on next
slide).

DD Regression

DD Regression
Relying on a regression analysis has a few advantages:
It is easy to calculate standard errors.
We can control for other variables which may reduce the
residual variance (lead to smaller standard errors).
It is easy to include multiple periods.
We can study treatments with different treatment
intensity.

Sensitivity Analysis
Perform a ”placebo” DD, i.e. use a ”fake” treatment
group.
Using pre-treatment periods (e.g. period -2, -1 if
treatment in period 0).
Using as a treatment group a population you know was
NOT affected.
If the DD estimate is different from 0, the trends are not
parallel, and our original DD is likely to be biased.
Use a different comparison group.
The two DDs should give the same estimates.
Use an outcome variable Y2 which you know is NOT
affected by the intervention.
Using the same comparison group and treatment year.
If the DD estimate is different from zero, we have a
problem.

Difference-in-Differences: Example
Card & Krueger (1994)
Prior to April 1992, the minimum wage in New Jersey
(NJ) and Pennsylvania (PA) was $4.25/hour.
On Apr 1, 1992, NJ raised the state minimum wage from
$4.25 to $5.05.
Whereas in the bordering state of PA, the minimum wage
stayed at $4.25 throughout this period
They surveyed about 400 fast food stores both in NJ and
in PA both before and after the minimum wage increase
in NJ.

Graph - Observed Data

Graph - DD

DD Regression Results

Findings
Card and Krueger found that rising the minimum wage
increased employment in some of their comparisons but in
no case caused an employment reduction.
How credible are Card and Krueger’s underlying
assumptions?
This article originated much economic and political
debate.
DID estimation has become a very popular method of
obtaining causal effects, especially in the US, where the
federal structure provides cross state variation in
legislation.

Key Assumption of DD Strategy: Common Trends
The key assumption for any DD strategy is that the
outcome in treatment and control group would follow the
same time trend in the absence of the treatment.
This does not mean that they have to have the same
mean of the outcome!
Common trend assumption is difficult to verify but one
often uses pre- treatment data to show that the trends
are the same.
Even if pre-trends are the same one still has to worry
about other policies changing at the same time.

RCT with DID
Paul Glewwe, Albert Park, Meng Zhao. 2016. “A better vision
for development: Eyeglasses and academic performance in
rural primary schools in China.” Journal of Development
Economics 122.
Simply enrolling children may have little effect on
educational outcomes if children do not acquire academic
skills when they are in school.
About 10% of primary school students in developing
countries have poor vision, but very few of them wear
glasses.
Eyeglasses can be purchased for $10-15, but many in
China believe that wearing glasses causes vision to
deteriorate faster.

Background
Almost no research examines the impact of poor vision on
school performance, and simple OLS estimates are likely
to be biased as studying harder often adversely affects
one’s vision.
The lack of evidence on the impact of offering eyeglasses
to students in developing countries led to the Gansu
Vision Intervention Project (GVIP).
Gansu province ranked 30th out of 31 provinces in per
capita disposable income (National Bureau of Statistics,
2005).
Examine the impact of providing eyeglasses to students in
grades 4-6 with poor vision.

Experiment Design
In 2004, a team of Chinese and international researchers
implemented a randomized trial to examine the impact of
providing glasses to students with poor vision in two
counties (hereafter, County 1 and County 2).
County 1 has 22 townships; 19 of them, with 101 primary
schools, participated in the program.
10 of these 19 townships were randomly assigned to the
program, and the other 9 became the control group.
County 2 has 23 townships; 18 of these, with 155 primary
schools, participated.
9 of these 18 were randomly assigned to the program,
and the other 9 were the control group.

Experiment Design
Random assignment was conducted in 2004 as follows:
In each county, all participating townships were ranked
by 2003 per capita income.
Starting with the two wealthiest, one was randomly
assigned to be a treated township and the other to the
control group.
In County 1, the 19th township (the poorest) was not
paired with another township; it was randomly assigned
to the treatment group.
In each township, primary schools were either all
assigned to the treated group or all to the control group.

Experiment Design
Baseline data were collected in June of 2004 on student
characteristics, exam scores, and visual acuity.
Treatment school students slated to enter grades 4–6 in
fall of 2004 and those who had poor vision were offered
free glasses.
During summer, an optometrist visited all townships to
conduct formal eye exam, and if poor vision was
confirmed, they were prescribed appropriate lenses.
The 2004 fall semester began in late August; most
students who accepted the offer received glasses by
mid-September.
At the end of the 2004–05 school year (late June/early
July of 2005), fall and spring semester exam scores were
collected.

Empirical Method
The simplest estimate of the program’s impact on
students with poor vision is a t-test that compares the
mean test scores of students with poor vision in the
treatment schools with the same mean for their
counterparts in the control schools.
This estimates the impact of offering eyeglasses (intent to
treat effect), not the impact of receiving them.
This can be done by regressing the (standardized) test
score (Tis) on a constant and a binary variable for
enrolling in a program school (Ps ):
Tis = α + βPs + uis
for student i in school s, where the residual uis is
uncorrelated with Ps due to random program assignment.

Empirical Method

Balance Test

Result

Why No Impact in County 2?
It is possible that the program and/or data collection
were not properly implemented in County 2.
First, in County 2 data were collected in a decentralized
way, with Excel files sent to schools to be filled in by
teachers and returned to the research center. This may
have reduced data quality, as teachers received little
training and lacked incentives to collect data carefully.
In contrast, in County 1 all data were collected by the
center’s professional staff trained by the authors, and
their work was monitored.
Indeed, the initial data files had many more problems for
County 2.
Second, program implementation may also have been
superior in County 1; County 2 is more populous, which
made monitoring more difficult in that county.

Regression Discontinuity Design (RDD)
Basic Idea
Often people use rules to assign individuals to
“treatments” which can be exploited for estimating causal
effects.
One of those rules is the threshold rule that is based on
some ex-ante variable, typically, correlated with the
expected effectiveness of the treatment
Score in entry exams
Grade for scholarship
Income for subsidy eligibility
Age eligibility for old-age pension
Age limit for alcohol consumption or driving
This ex-ante variable is called the running (forcing,
assignment) variable.
Selected threshold of the running variable divides
individuals into “treated” and “not treated.”

Basic Idea
The idea in Regression Discontinuity Design (RDD) is to
exploit the randomness of this threshold.
Compare individuals below (non-treated) and above
(treated) the threshold
Two conditions for credible and precise RDD estimates:
1 Variation in the treatment status near the threshold is as
good as random.
How likely is this? Were there a way to anticipate the
threshold and to manipulate the running variable?
2 Sufficient number of observations around the threshold.

Regression Discontinuity Design (RDD) A visual
depiction

RDD Visual Depiction

Key Assumption for the RDD
Selection process is completely known and can be
modeled through a regression line of the assignment and
outcome variables.
It is like an experiment around the cutoff and thus,
untreated portion serves as a counterfactual.
All other unobserved determinants of Y are continuously
related to the running variable X. This allows us to use
average outcomes of units just below the cutoff as a
valid counterfactual for units right above the cutoff.
This assumption cannot be directly tested. But there
are some tests which give suggestive evidence whether
the assumption is satisfied .

Rule of Treatment
The treatment of the population depends on whether an
observed variable X, exceeds a critical value c.
This variable X, is called the assignment variable or the
forcing variable.
The treatment rule is given by:

Examples
Effect of the Minimum Legal Drinking Age (MLDA) on
death rates (Carpenter and Dobkin 2009):
1 outcome variable yi: death rate
2 treatment Di: legal drinking status
3 running variable xi: age
4 cutoff: MLDA transforms 21-year-olds from underage
minors to legal alcohol consumers.

Examples

Simple Linear RD Setup
Assuming that the relationship between y and X is
otherwise linear, a simple way of estimating the treatment
effect is by fitting the linear regression:
Y = α + DτXβ + ϵ
where ϵ is the error term.
This case is depicted in the figure on the following page,
which shows both the true underlying function and
numerous realization of ϵ.

Measure the Causal Impact: Identification
Two conditions must be satisfied to consider � as the
causal impact:
First and obviously, the treatment of the population
must depend on whether an observed variable exceeds a
critical value denoted c.
Second, individuals do not have a precise control on the
assignment (or forcing) variable.

No Precise Control: An Intuitive Explanation
”No precise control” means that among those scoring
near the threshold, it is a matter of ”luck” as to which
side of the threshold they land.
Those who marginally fail (characterized by the running
variable just below the cutoff) and those who marginally
pass (characterized by a running variable just above the
cutoff) are identical.
If individuals have a precise control on the assignment
variable, this means that individuals of different types
(characterized by different sets of observed and
unobserved characteristics) will reach distinct outcomes.
Therefore in this case it is not possible to attribute the
jump in y to the impact of the merit award only.

Formulation of the RD Design

RD example: Malamud and Pop-Eleches (2011)
Example: “Home Computer Use and the Development of
Human Capital”, by Ofer Malamud and Cristian
Pop-Eleches (QJE 2011)
Setting of example: Evaluate effects of a 2008
government program in Romania which provided vouchers
for computers for low-income children
Data: Conducted a detailed household survey of
families(in 2009) who participated from 7 counties in
Romania

Program was targeted to low-income children enrolled in
school (administered through the school)
To be eligible to apply: at least one child under 26 years
of age, enrolled in grade 1 to 12, or university - monthly
income per household member of less than 150 RON
(about $65)
In 2008, 52,212 families applied
Assignment mechanism:
Applicants ranked based on monthly income per
household member
due to limited funds, only 35,484 applicants with lowest
incomes were awarded vouchers • resulting cut-off:
62.58 RON ($27)

Propensity Score Matching(PSM)
Brief Introduction
Idea: Compare individuals that are similar in observable
characteristics
Implementation of matching
1 Divide workers into different categories on the basis of
the observable characteristics.
2 Compare means in outcomes over these different
categories.
Propensity score matching
1 Estimate the propensity of the treatment using rich set
of observational characteristics (propensity score).
2 Compare means within cells defined on the basis of the
propensity score.

Example: Smoking and Mortality
Cochran 1968, Biometrics
How should we interpret this descriptive evidence?

Non-smokers and smokers differ in age.
How should we interpret this descriptive evidence?

We could compare death rates with in age
groups(matching by age).
This way,we neutralize any imbalances in the observed
sample related with age.
Matching
Divide the sample into several age groups.
Compute death rates for smokers and non-smokers by
age group.
Compare smokers and non-smokers by age group and
calculate the average effect using some weight.

Adjusted average death rates:
Cigarette smokers had relatively low death rates only
because in the sample they were younger on average.
Perhaps the these groups are unbalanced in another
variable... (any idea?)

Matching Method
In an experimental design, randomization ensures that all
the relevant characteristics, either observable or
unobservable, of the studied units are balanced between
treatment and control groups and, because of this, the
difference in mean outcomes correctly estimates the
impact of the intervention.
However, in a non-experimental setting, the “treatment”
and “control” groups are likely to differ not only in their
treatment status, but also in their values of X.
In the context of non-experimental designs (or flawed
experimental designs), it is necessary to account for
differences between treated and untreated groups in order
to properly estimate the impact of the program.

Matching Method
Matching essentially uses statistical techniques to allow
you to construct a comparison group that has the most
similar characteristics to a treatment group when the
assignment to the treatment is done on the basis of
observable variables.
Note: Matching STILL doesn’t allow to control for
selection bias that arises when the assignment to the
treatment is done on the basis of non-observables.
The method assumes there are no remaining unobservable
differences between treatment and comparison groups.
Intuition: the comparison group needs to be as similar
as possible to the treatment group, in terms of the
observables before the start of the treatment.

Matching Method
Finding a good match for each program participant
requires approximating as closely as possible variables that
explain that individual’s decision to enroll in the program.
However, it is not easy to match, if the list of relevant
observed characteristics is very large, or if each
characteristic takes on many values.

The curse of dimensionality can be easily solved using a
method called, Propensity Score Matching (PSM), the
most commonly used matching method.
Instead of attempting to create a match for each
participant with exactly the same value of X, PSM
matchs on the estimated probability of of being treated
(propensity score).
The propensity score is P(X) = Pr(d = 1|X), where d = 1
indicates participation in the project.

Using PSM, the probability of participation summarizes
all the relevant information contained in the � variables.
The major advantage is the reduction of dimensionality, as
it allows for matching on a single variable (the propensity
score) instead of on the entire set of covariates.
In effect, the propensity score is a balancing score for �,
assuring that for a given value of the propensity score,
the distribution of � will be the same for treated and
comparison units.

Assumptions
To ensure that the matching estimators consistently
estimate the treatment effects, two assumptions are
needed:
1 Unconfoundedness: assignment to the treatment is
independent of the outcomes, conditional on covariates.
2 Overlap or common support condition: the
probability of assignment is bounded away from zero and
one.

Assumption 1: Unconfoundedness
The unconfoundeness condition (or conditional
independence assumption, CIA) states that the decision
to take the treatment is purely random for individuals
with similar values of the pretreatment variables.
In other words, given a set of observable covariates �
which are not affected by treatment, potential outcomes
are independent of treatment assignment.
There is a set � of regressors, observable to the
researcher, such that after controlling for these regressors,
the potential outcomes are independent of the treatment
status (i.e., the treatment assignment is as good as
random).

Assumption 2: Common Support
Common support condition: 0 < P(d = 1|X) < 1.
The overlap or common support condition ensures that
persons with the same � values have a positive probability
of being both treated and untreated.
In other words, the proportion of treated and untreated
individuals must be greater than zero for every possible
value of �. This assumption is also called a “common
support condition”.
It rules out the phenomenon of perfect predictability of �
given �.
Forexample,if P(d = 1|x) = 1,i.e.,everyone with values of
� is treated, then we cannot have a proper comparison
group.

Common Support

Similar?
A researcher’s goal is to control for all the variables that
are suspected of influencing selection into treatment.
The researcher should have access to a large number of
variables (for A1) as well as observations (for A2).
“Matching” methods find a non-treated unit that is
similar to a treated (participating) unit, allowing an
estimate of the intervention’s impact as the difference
between a participant and the matched comparison case.
One of the critical issues in implementing matching
techniques is to define clearly (and justify) what “similar”
means.

Similar?
Only variables that influence simultaneously the treatment
status and the outcome variable should be included.
Hence, economic theory, a sound knowledge of previous
research and also information about the institutional
settings should guide the researcher in building up the
model.
Those variables should not be affected by treatment. To
ensure this, variables should either be fixed over time or
measured before participation. In the latter case, it must
be guaranteed that the variable has not been influenced
by the anticipation of participation.

How Do We Match on Propensity Score
Taken literally, should match on exactly propensity score.
In practice it is hard to do so.
Thus the practical strategy is to match treated units to
comparison units whose p-scores are sufficiently close.
How close

Choosing a Matching Algorithm
Nearest Neighbour Matching
Calliper Matching
Kernel Matching
Weighting by the propensity score

Match individual treatment and control units one-on -one
For each treated units i find a non treated unit j who is
nearest on the distribution of the probability p(x)
Find control j that minimizes the distance between
p(Xi) and p(xj)

Steps in PSM
Need representative and comparable data for both
treatment and comparison groups.
Use logit (or other discrete choice model such as probit)
to estimate program participation as a function of
observable pre-treatment covariates.
Use predicted values from logit to generate propensity
score for all treatment and comparison group members.
Restrict a sample to common support.
Match treated units using a matching algorithm, e.g.
nearest neighbour (NN).
Check the balancing.
Once matches are made, we can calculate the treatment
effect by comparing the means of outcomes across
participants and their matched pairs.

A (very) Simple Example of PSM
Consider the following simple example where Y is income,
and X1 and X2 are education and work experience,
respectively. T is a treatment such as an employee
training program.

Consider the following simple example where � is income,
and �1 and �2 are education and work experience,
respectively. � is a treatment such as an employee training
program.

Use logit model to obtain estimated propensity score
P(x). logit T x1 x2 predict px

Find the best match for each observation based on the
propensity score.

Find counterfactual (comparison) outcomes Yicfor each
non-treated and Y[0c]for each treated.

Compute the impact Yic − Y for each non-treated, and
Y − Y0c for each treated.

Propensity score matching with Stata

Example

When Shall We Use PSM?
Typically used when neither randomization, RD or other
quasi experimental options are not possible.
Matching helps control only for OBSERVABLE
differences, not unobservable differences.
Matching becomes much better in combination with
other techniques, such as exploiting baseline data for
matching and then using difference-in-difference strategy.

DID.pdf

More Related Content

What's hot

Similar to DID.pdf

More from natan82253

Recently uploaded

DID.pdf