Within Models

Peter M. Lance, PhD
MEASURE Evaluation
University of North Carolina at
Chapel Hill
December 15, 2016
Within Models

Global, five-year, $180M cooperative agreement
Strategic objective:
To strengthen health information systems – the
capacity to gather, interpret, and use data – so
countries can make better decisions and sustain good
health outcomes over time.
Project overview

Improved country capacity to manage health
information systems, resources, and staff
Strengthened collection, analysis, and use of
routine health data
Methods, tools, and approaches improved and
applied to address health information challenges
and gaps
Increased capacity for rigorous evaluation
Phase IV Results Framework

Global footprint (more than 25 countries)

• The program impact evaluation challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables

𝑌0 = 𝛽0 + 𝛽2 ∙ 𝑥 + 𝜀
𝑌1
= 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥 + 𝜀

𝑌1 − 𝑌0
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 + 𝜖
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 − 𝜖
= 𝛽1

𝑌1 − 𝑌0
= 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥 + 𝜀
− 𝛽0 + 𝛽2 ∙ 𝑥 + 𝜀
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 − 𝜖
= 𝛽1

𝑌 = 𝑃 ∙ 𝑌1
+ 1 − 𝑃 ∙ 𝑌0
= 𝑃 ∗ 𝛽0 + 𝛽1 + 𝜖 + 1 − 𝑃 ∗ 𝛽0 + 𝜖
= 𝑃 ∗ 𝛽0 + 𝑃 ∗ 𝛽1 + 𝑃 ∗ 𝜖
+𝛽0 + 𝜖 − 𝑃 ∗ 𝛽0 − 𝑃 ∗ 𝜖
= 𝛽0 + 𝑃 ∗ 𝛽1 + 𝜖

𝑌 = 𝑃 ∙ 𝑌1
+ 1 − 𝑃 ∙ 𝑌0
= 𝑃 ∙ 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥 + 𝜀
+ 1 − 𝑃 ∙ 𝛽0 + 𝛽2 ∙ 𝑥 + 𝜀
= 𝑃 ∗ 𝛽0 + 𝑃 ∗ 𝛽1 + 𝑃 ∗ 𝜖
+𝛽0 + 𝜖 − 𝑃 ∗ 𝛽0 − 𝑃 ∗ 𝜖

𝑌 = 𝛽0 + 𝛽1 ∙ 𝑃 + 𝛽2 ∙ 𝑥 + 𝜀

Cost of Participation
𝐶 = 𝜌0 + 𝜌1 ∙ 𝑥

Benefit-Cost>0
𝑌1
− 𝑌0
− C > 0
𝛽1 − 𝐶 > 0
𝛽1 − 𝛾0 + 𝛾1 ∗ 𝑥 > 0

Benefit-Cost>0
𝑌1 − 𝑌0 − C > 0
𝛽1 − 𝐶 > 0
𝛽1 − 𝛾0 + 𝛾1 ∗ 𝑥 > 0

Benefit-Cost>0
𝑌1 − 𝑌0 − C > 0
𝛽1 − 𝐶 > 0
𝛽1 − 𝜌0 + 𝜌1 ∙ 𝑥 > 0

Benefit-Cost>0
𝛽1 − 𝜌0 + 𝜌1 ∙ 𝑥 > 0
𝜀

True model:
𝑌 = 𝛽0 + 𝛽1 ∙ 𝑃 + 𝛽2 ∙ 𝑥 + 𝜀
We actually attempt to estimate:
𝑌 = 𝜏0 + 𝜏1 ∙ 𝑃 + 𝜖
Error term now contains:
𝛽2 ∙ 𝑥

𝐸( 𝜏1) = 𝐸
𝑖=1
𝑛
𝑃𝑖 − 𝑃 ∙ 𝑌𝑖
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2

𝐸 𝜏1
= 𝐸
𝑖=1
𝑛
𝑃𝑖 − 𝑃 ∙ 𝛽0 + 𝛽1 ∙ 𝑃𝑖 + 𝛽2 ∙ 𝑥𝑖 + 𝜀𝑖
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2

𝐸 𝜏1
= 𝛽1
+𝐸
𝑖=1
𝑛
𝛽2 ∙ 𝑥𝑖 ∙ 𝑃𝑖 − 𝑃
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2

𝐸 𝜏1
= 𝛽1
+𝛽2 ∙ 𝐸
𝑖=1
𝑛
𝑥𝑖 ∙ 𝑃𝑖 − 𝑃
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2

𝐸 𝜏1
= 𝛽1
+𝛽2 ∙ 𝐸
𝑖=1
𝑛
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2
𝑥𝑖= 𝛾0 + 𝛾1 ∙ 𝑃𝑖 + 𝜗𝑖

𝐸 𝜏1
= 𝛽1
+𝛽2 ∙ 𝛾1
𝑖=1
𝑛
𝑥
𝑖=1
𝑛
𝑥
𝑥𝑖= 𝛾0 + 𝛾1 ∙ 𝑃𝑖 + 𝜗𝑖

𝐸 𝜏1 = 𝛽1 + 𝛽2 ∙ 𝐸
𝑖=1
𝑛
𝑖=1
𝑛
𝑃𝑖 − 𝑃 2

𝐸 𝜏1 = 𝛽1 + 𝛽2 ∙ 𝛾1

𝐸 𝜏1 = 𝛽1 + 𝛽2 ∙ 𝛾1
The actual
causal effect of
P on y

𝐸 𝜏1 = 𝛽1 + 𝛽2 ∙ 𝛾1
The actual
causal effect of
P on y
The actual causal effect of
the omitted variable X on
Y

𝐸 𝜏1 = 𝛽1 + 𝛽2 ∙ 𝛾1
The actual
causal effect of
P on y
Th”Effect” of P on x:
𝑥𝑖= 𝛾0 + 𝛾1 ∙ 𝑃𝑖 + 𝜗𝑖
The actual causal effect of
the omitted variable X on
Y

True model:
𝑌 = 𝛽0 + 𝛽1 ∙ 𝑃 + 𝛽2 ∙ 𝑥 + 𝜀
𝑌 = 𝜏0 + 𝜏1 ∙ 𝑃 + 𝜖
𝑬 𝝉 𝟏 ≠ 𝜷

True model:
𝑌 = 𝛽0 + 𝛽1 ∙ 𝑃 + 𝛽2 ∙ 𝑥 + 𝛽3 ∙ 𝜇 + 𝜀
𝑌 = 𝜏0 + 𝜏1 ∙ 𝑃 + 𝜏2 ∙ 𝑥 + 𝜖
𝜇

Cost of articipation
𝐶 = 𝜌0 + 𝜌1 ∙ 𝑥 + 𝜌2 ∙ 𝜇

Benefit-Cost>0
𝑌1 − 𝑌0 − C > 0
𝛽1 − 𝐶 > 0
𝛽1 − 𝜌0 + 𝜌1 ∙ 𝑥 + 𝜌2 ∙ 𝜇 > 0

(Ben)
This webinar is…
simply terrible
(John)

(John)
(Ben)
𝑃 = 1
𝑌𝐵𝑒𝑛 = 𝑌𝐵𝑒𝑛
1
𝑊𝑒𝑎𝑙𝑡ℎ𝑦
𝑀𝑜𝑡𝑖𝑣𝑎𝑡𝑒𝑑
𝑃 = 0
𝑌𝐽𝑜ℎ𝑛 = 𝑌𝐽𝑜ℎ𝑛
0
𝑃𝑜𝑜𝑟
𝑈𝑛 − 𝑀𝑜𝑡𝑖𝑣𝑎𝑡𝑒𝑑

(John)
(Ben)
𝑃 = 1
1
𝑃 = 0
0
𝑃𝑜𝑜𝑟
𝒀 𝑩𝒆𝒏 − 𝒀 𝑱𝒐𝒉𝒏

(John)
(Ben)
𝑃 = 1
1
𝑃 = 0
0
𝑃𝑜𝑜𝑟
𝒀 𝑩𝒆𝒏 − 𝒀 𝑱𝒐𝒉𝒏
?

True model:
𝑌 = 𝛽0 + 𝛽1 ∙ 𝑃 + 𝛽2 ∙ 𝑥 + 𝛽3 ∙ 𝜇 + 𝜀
𝑌 = 𝜏0 + 𝜏1 ∙ 𝑃 + 𝜖
𝛽2 ∙ 𝑥 + 𝛽3 ∙ 𝜇

𝑃𝐵𝑒𝑛,1 𝑌𝐵𝑒𝑛,1𝜇 𝐵𝑒𝑛
𝑡 = 1
𝑡 = 0

𝑃𝐵𝑒𝑛,1 − 𝑃𝐵𝑒𝑛,0

∆𝑃𝐵𝑒𝑛

∆𝑃𝐵𝑒𝑛 ∆𝑌𝐵𝑒𝑛

∆𝑃𝐵𝑒𝑛 ∆𝑌𝐵𝑒𝑛𝜇 𝐵𝑒𝑛 − 𝜇 𝐵𝑒𝑛

∆𝑃𝐵𝑒𝑛 ∆𝑌𝐵𝑒𝑛0

𝑃𝐵𝑒𝑛,𝑡, 𝑌𝐵𝑒𝑛,𝑡𝜇 𝐵𝑒𝑛
∆𝑃𝐵𝑒𝑛, ∆𝑌𝐵𝑒𝑛
𝜇 𝐵𝑒𝑛

Later folks!!!
It was
awesome!!
I…deeply regret my role
in this webinar

𝑌𝑖𝑡
0
= 𝛽0 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
𝑌𝑖𝑡
1
= 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡

𝑌𝑖𝑡
1
− 𝑌𝑖𝑡
0
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 − 𝜖
= 𝛽1

𝑌𝑖𝑡
1
− 𝑌𝑖𝑡
0
= 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
− 𝛽0 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 − 𝜖
= 𝛽1

𝑌𝑖𝑡
1
− 𝑌𝑖𝑡
0
= 𝛽1
= 𝛽0 + 𝛽1 + 𝜖 − 𝛽0 − 𝜖
= 𝛽1

𝑌𝑖𝑡 = 𝑃𝑖𝑡 ∙ 𝑌𝑖𝑡
1
+ 1 − 𝑃𝑖𝑡 ∙ 𝑌𝑖𝑡
0
= 𝑃𝑖𝑡 ∙ 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
+ 1 − 𝑃𝑖𝑡 ∙ 𝛽0 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
= 𝛽0 + 𝑃 ∗ 𝛽1 + 𝜖

𝑌𝑖𝑡
= 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡

Cost of Participation
𝐶𝑖𝑡 = 𝜌0 + 𝜌1 ∙ 𝑥𝑖𝑡 + 𝜌2 ∙ 𝜇𝑖

Benefitit-Costit>0
𝑌𝑖𝑡
1
− 𝑌𝑖𝑡
0
− 𝐶𝑖𝑡 > 0
𝛽1 − 𝐶𝑖𝑡 > 0
𝛽1 − 𝜌0 + 𝜌1 ∙ 𝑥𝑖𝑡 + 𝜌2 ∙ 𝜇𝑖 > 0

(The Truth)
𝑌𝑖𝑡 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
(What we can actually estimate)
𝑌𝑖𝑡 = 𝛾0 + 𝛾1 ∙ 𝑃𝑖𝑡 + 𝛾2 ∙ 𝑥𝑖𝑡 + 𝜖𝑖𝑡
(The $60,000 Question)
𝐸 𝛾1 = 𝛽1
?

(The Truth)
(What we can actually estimate)
𝑌𝑖𝑡 = 𝛾0 + 𝛾1 ∙ 𝑃𝑖𝑡 + 𝛾2 ∙ 𝑥𝑖𝑡 + 𝜖𝑖𝑡
(The $60,000 Question)
𝐸 𝛾1 ≠ 𝛽1

Time
(t)
t=0 t=1

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
Time
(t)
t=0 t=1

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
Time
(t)
t=0 t=1

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
𝑌𝑖1 − 𝑌𝑖0

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽0 − 𝛽0

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 0

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + 𝛽3 ∙ 𝜇𝑖 − 𝜇𝑖

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + 𝛽3 ∙ 0

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖
𝜇𝑖

𝑌𝑖𝑡
0
= 𝛽0 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
𝑌𝑖𝑡
1
= 𝛽0 + 𝛽1 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖𝑡
𝛽00 ∙ 𝑡

∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖
Time
(t)

∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖
Time
(t)
t=1 t=2 ∙∙∙∙∙∙∙ t=T-1 t=T

𝑌𝑖𝑡 = 𝑌𝑖𝑡 − 𝑌𝑖
𝑌𝑖 =
𝑡=1
𝑇
𝑌𝑖𝑡
𝑇
𝑌𝑖𝑡=

𝑌𝑖𝑡 = 𝑌𝑖𝑡 − 𝑌𝑖
𝑌𝑖 =
𝑡=1
𝑇
𝑌𝑖
𝑇
𝑌𝑖𝑡= 0

𝑃𝑖𝑡 = 𝑃𝑖𝑡 − 𝑃𝑖
𝑃𝑖 =
𝑡=1
𝑇
𝑃𝑖
𝑇
𝑌𝑖𝑡= 𝛽1 ∙ 𝑃𝑖𝑡

𝑥𝑖𝑡 = 𝑥𝑖𝑡 − 𝑥𝑖
𝑥𝑖 =
𝑡=1
𝑇
𝑥𝑖
𝑇
𝑌𝑖𝑡= 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡

𝜇𝑖𝑡 = 𝜇𝑖 − 𝜇𝑖 = 𝜇𝑖 − 𝜇𝑖 = 0
𝜇𝑖 =
𝑡=1
𝑇
𝜇𝑖
𝑇
= 𝜇𝑖
𝑌𝑖𝑡= 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 0

𝜀𝑖𝑡 = 𝜀𝑖𝑡 − 𝜀𝑖
𝜀𝑖 =
𝑡=1
𝑇
𝜀𝑖
𝑇
𝑌𝑖𝑡= 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝜀𝑖𝑡

𝑌𝑖𝑡 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙
𝑖=1
𝑁
𝑑𝑖 ∙ 𝜇𝑖 + 𝜀𝑖𝑡

𝑌𝑖𝑡
= 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 + 𝛽3 ∙
𝑗=1
𝑁
𝑑𝑗 ∙ 𝜇 𝑗 + 𝜀𝑖𝑡

𝑌𝑖𝑡
= 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 +
𝑗=1
𝑁
𝛽3 ∙ 𝑑𝑗 ∙ 𝜇 𝑗 + 𝜀𝑖𝑡

𝑌𝑖𝑡 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 +
𝑗=1
𝑁
𝑑𝑗 ∙ 𝛽3∙ 𝜇 𝑗 + 𝜀𝑖𝑡

𝑌𝑖𝑡 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 +
𝑗=1
𝑁
𝑑𝑗 ∙ 𝜑𝑗 + 𝜀𝑖𝑡
where 𝜑𝑗 = 𝛽3 ∙ 𝜇 𝑗

𝑌𝑖𝑡 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖𝑡 + 𝛽2 ∙ 𝑥𝑖𝑡 +
𝑗=1
𝑁−1
𝑑𝑗 ∙ 𝜑𝑗 + 𝜀𝑖𝑡
where 𝜑𝑗 = 𝛽3 ∙ 𝜇 𝑗

Big Caveats/Limitations/Drawbacks
1.Loss of information
2.Makes measurement error bias worse
3.Very limited options for limited dependent
variables
4.Rooted in a weird kind of paradox

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖
(1)
(2)

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖1 = 𝛾0 + 𝛾1 ∙ 𝑃𝑖1 + 𝛾2 ∙ 𝑥𝑖1 + 𝛾3 ∙ 𝜇𝑖 + 𝜖𝑖1
where 𝑃𝑖1 = 𝑃𝑖1 + 𝜏𝑖1
𝐸 𝛾1 ≠ 𝛽1

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖1 = 𝛾0 + 𝛾1 ∙ 𝑃𝑖1 + 𝛾2 ∙ 𝑥𝑖1 + 𝛾3 ∙ 𝜇𝑖 + 𝜖𝑖1
where 𝑃𝑖1 = 𝑃𝑖1 + 𝜏𝑖1
𝐸 𝛾1 < 𝛽1

𝑌𝑖1 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖1 + 𝛽2 ∙ 𝑥𝑖1 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖1
𝑌𝑖0 = 𝛽0 + 𝛽1 ∙ 𝑃𝑖0 + 𝛽2 ∙ 𝑥𝑖0 + 𝛽3 ∙ 𝜇𝑖 + 𝜀𝑖0
∆𝑌𝑖 = 𝛽1 ∙ ∆𝑃𝑖 + 𝛽2 ∙ ∆𝑥𝑖 + ∆𝜀𝑖
𝑃𝑖
∆𝑃𝑖

t=2000 t=2002
True age 35 37
Measured age 33 39
Error -2 +2
Error
𝐓𝐫𝐮𝐞 𝐚𝐠𝐞
.0571 .054
Age2002-Age2000
True age difference 2
Measured age
difference
6
Error 4
Error
𝐓𝐫𝐮𝐞 difference
1.5

Village 1 Village 2
t=0
time
(t)t=0

Village 1 Village 2
t=0
time
(t)t=0
P=0 P=1

Village 1 Village 2
Village 1 Village 2
t=0
t=1
time
(t)t=0 t=1

Village 1 Village 2
Village 1 Village 2
t=0
t=1
time
(t)t=0 t=1
P=0 P=1

Y
tt=0 t=1
E(Y|P=0,t=0)
E(Y|P=0,t=1)

Y
tt=0 t=1
E(Y|P=0,t=0)
E(Y|P=0,t=1)
A
A=E(Y|P=0,t=1)-E(Y|P=0,t=0)

Y
tt=0 t=1
E(Y|P=0,t=0)
E(Y|P=0,t=1)
E(Y|P=1,t=0)
A

Y
tt=0 t=1
E(Y|P=0,t=0)
E(Y|P=0,t=1)
E(Y|P=1,t=0)
E(Y|P=1,t=1)
A

𝑌𝑖𝑡 = 𝜔0 + 𝜔1 ∙ 𝑃𝑖 + 𝜔2 ∙ 𝑡 + 𝜔3 ∙ 𝑃𝑖 ∙ 𝑡 + 𝜖𝑖𝑡

Controls for
fixed (ie underlying,
not time-varying)
differences between
program participants
and
non-participants

Controls for
not time-varying)
differences between
and
non-participants
Controls for
underlying time trend
common to
and
non-participants

Controls for
not time-varying)
differences between
and
non-participants
Controls for
underlying time trend
common to
and
non-participants
Program impact

𝑌𝑖𝑡
= 𝜔0 + 𝜔1 ∙ 𝑃𝑖 + 𝜔2 ∙ 𝑡 + 𝜔3 ∙ 𝑃𝑖 ∙ 𝑡 + 𝜔4 ∙ 𝑋𝑖𝑡 + 𝜖𝑖𝑡
Controls for
other time-varying
characteristics

𝑌𝑖𝑡
𝑐𝑜𝑟𝑟 𝜖𝑖𝑡, 𝜖𝑖𝑡+𝑗 ≠ 0

Bertrand, Duflo and Mullainathan
Basic Experiment:
Take a dataset (current population survey) with labor
market outcomes (ln(earnings)) for many women-years
(900,000) and “make up” a fake program. Then try to
evaluate the impact of these fake programs with a DID
regression.
The Result:
The null hypothesis that the policy had no effect at the 5
percent level rejected a stunning 50-70 percent of the time,
depending on the econometric approach.

𝑌𝑖𝑡
The “group mean”
fixed effect

𝑌𝑖𝑡
= 𝜔0 + 𝜔1 ∙
𝑖=1
𝑁
𝑑𝑖 ∙ 𝜇𝑖
+𝜔2 ∙ 𝑡 + 𝜔3 ∙ 𝑃𝑖 ∙ 𝑡
+𝜔4 ∙ 𝑋𝑖𝑡 + 𝜖𝑖𝑡
Dummy variable individual
fixed effect

𝑌𝑖𝑡
= 𝜔0 +
𝑖=1
𝑁
𝑑𝑖 ∙ 𝜔1 ∙ 𝜇𝑖
+𝜔2 ∙ 𝑡 + 𝜔3 ∙ 𝑃𝑖 ∙ 𝑡

𝑌𝑖𝑡
= 𝜔0 +
𝑖=1
𝑁
𝑑𝑖 ∙ 𝜔1𝑖
+𝜔2 ∙ 𝑡 + 𝜔3 ∙ 𝑃𝑖 ∙ 𝑡
where 𝜔1𝑖 = 𝜔1 ∙ 𝜇𝑖

Links:
The manual:
http://www.measureevaluation.org/resources/publications/ms-
14-87-en
The webinar introducing the manual:
http://www.measureevaluation.org/resources/webinars/metho
ds-for-program-impact-evaluation
My email:
pmlance@email.unc.edu

MEASURE Evaluation is funded by the U.S. Agency
for International Development (USAID) under terms
of Cooperative Agreement AID-OAA-L-14-00004 and
implemented by the Carolina Population Center, University
of North Carolina at Chapel Hill in partnership with ICF
International, John Snow, Inc., Management Sciences for
Health, Palladium Group, and Tulane University. The views
expressed in this presentation do not necessarily reflect
the views of USAID or the United States government.
www.measureevaluation.org

Within Models

More Related Content

Viewers also liked

Similar to Within Models

More from removed_62798267384a091db5c693ad7f1cc5ac

Recently uploaded

Within Models