Webinar presentation by Peter Lance

1. Peter M. Lance, PhD
MEASURE Evaluation
University of North Carolina at
Chapel Hill
JUNE 29, 2016
Randomization and Its
Discontents

2. Global, five-year, $180M cooperative agreement
Strategic objective:
To strengthen health information systems – the
capacity to gather, interpret, and use data – so
countries can make better decisions and sustain good
health outcomes over time.
Project overview

3. Improved country capacity to manage health
information systems, resources, and staff
Strengthened collection, analysis, and use of
routine health data
Methods, tools, and approaches improved and
applied to address health information challenges
and gaps
Increased capacity for rigorous evaluation
Phase IV Results Framework

4. Global footprint (more than 25 countries)

5. • The program impact evaluation challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables

6. • The program impact evaluation challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables

9. Did the program make a
difference?

10. Did the program cause a change in an
outcome of interest Y ?
(Causality)

11. What happens
if the individual
participates
{Causal} Program impact
𝑌𝑖
1
− 𝑌𝑖
0
= Program impact
What happens
if the individual
does not
participate

12. Average treatment effect (ATE)
𝐸 𝑌1 − 𝑌0
Average effect of treatment on the treated (ATT)
𝐸 𝑌1 − 𝑌0|𝑃 = 1
Treatment effects

13. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome

14. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome

15. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome

16. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome

17. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome
Fundamental Identification
Problem of Program Impact
Evaluation

18. 𝑌𝑖
1
, 𝑌𝑖
0
Observed outcome
Fundamental identification
problem of program impact
evaluation

19. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

20. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

21. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

25. Impact evaluation
𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1

28. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
Average Y
across sample
of participants
−
Average Y
Across Sample
of Non−Participants

29. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
Average Y
across sample
of participants
−
Average Y
across sample
of non−participants

30. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
Average Y
across sample
of participants
−
Average Y
across sample
of non−participants

31. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
Average Y
across sample
of participants
−
Average Y
across sample
of non−participants

34. Impact evaluation
𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
𝐸 𝑌0|𝑃 = 0 = 𝐸(𝑌0|𝑃 = 1)

35. 𝐸 𝑌0
|𝑃 = 0 ≠ 𝐸 𝑌0
𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0
𝐸 𝑌0
|𝑃 = 0 ≠ 𝐸 𝑌0
|𝑃 = 1

36. 𝐸 𝑌0
|𝑃 = 0 ≠ 𝐸 𝑌0
𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0
𝐸 𝑌0
|𝑃 = 0 ≠ 𝐸 𝑌0
|𝑃 = 1

37. X
Y
P

38. X
Y
P
𝑌 = 𝑃 ∙ 𝑌1
+ 1 − 𝑃 ∙ 𝑌0

39. X
Y
P

40. X
Y
P

41. 𝐸 𝑌1
− 𝑌0
|𝑃 = 1
= 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
Average Y
across sample
of participants
−
Average Y
across sample
of non−participants

42. The big idea (part un)
So, the basic idea of the
randomization/experimental
approach to impact evaluation is
that by randomizing program
participation we insure that
participants and non-participants are
alike on average in terms of their
characteristics

43. The big idea (part deux)
If this is the case, then any
differences in average outcomes
between the two groups can be
ascribed to the one way in which
they do differ: program participation

44. 1.Participants in the experiment are randomly
selected from the population of interest and
randomly assigned to their program participation
status;
2.All participants in the trial comply with the program
participation status to which they are assigned;
3.The experiment lasts long enough to replicate the
program under consideration and to influence
outcomes;
4.There are no social interactions that may make a
full-scale program inherently different from a smaller
scale intervention

46. Source: Washington Policy Center

47. Source: EH.net

48. 𝐸 𝑌1
− 𝑌0
= 𝐸 𝑌1 − 𝐸 𝑌0
𝑌 = 𝑃 ∙ 𝑌1 + 1 − 𝑃 ∙ 𝑌0

49. 𝐸 𝑌1
− 𝑌0
= 𝐸 𝑌1 − 𝐸 𝑌0
Average Y
Across Sample
of Participants
−
Average Y
Across Sample
of Non−Participants

50. 𝐸 𝑌1
− 𝑌0
= 𝐸 𝑌1 − 𝐸 𝑌0
Average Y
across sample
of participants
−
Average Y
Across Sample
of Non−Participants

51. 𝐸 𝑌1
− 𝑌0
= 𝐸 𝑌1 − 𝐸 𝑌0
Average Y
across sample
of participants
−
Average Y
across sample
of non−participants

52. X
Y
P

53. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋|𝑃 = 0

54. X
Y
P

55. 𝐸 𝑌1|𝑃 = 1 > 𝐸 𝑌1 > 𝐸 𝑌1|𝑃 = 0

56. 𝐸 𝑌0|𝑃 = 0 < 𝐸 𝑌0 < 𝐸 𝑌0|𝑃 = 1

57. 𝐸 𝑌1
− 𝑌0
=
𝐸 𝑌1 − 𝐸 𝑌0
(Overestimated) (Underestimated)

58. 𝐸 𝑌1
− 𝑌0
=
𝐸 𝑌1 − 𝐸 𝑌0
(Overestimated) (Underestimated)

59. 𝐸 𝑌1
− 𝑌0
=
𝐸 𝑌1 − 𝐸 𝑌0
(Overestimated) (Underestimated)

60. X
Y
P

61. X
Y
P

64. Refusal rates by plan
Plan Refusal rate (%)
Free 8
25 and 50% coinsurance 11
95% coinsurance 25
Source: Newhouse et al. (1993)

73. X
Y
P

74. X
Y
P

75. X
Y
P

76. If such and such is true of the real world
processes that gave rise to program
participation and outcomes in our observed
non-experimental sample, then the estimates
of program impact generated by this quasi-
experimental estimator provide the causal
impact of program participation on outcomes
of interest.

77. Lalonde’s critique of non-
experimental estimators

78. Lalonde’s critique of non-
experimental estimators

80. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs carried out in a limited or pilot setting
can mislead about the impact of the “scaled up” program
-Randomization is often not straightforward

81. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

82. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

83. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant
group cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

84. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

85. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

86. 𝐸 𝑌1
− 𝑌0
= 𝐸 𝑌1 − 𝐸 𝑌0

87. 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1
− 𝑌0
≠ 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1 − 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌0

88. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

89. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

90. -Individuals cannot be forced to participate in a program
-Individuals cannot be forced to accept their random
experimental assignment
-Individuals assigned to the control/non-participant group
cannot be prevented from seeking alternatives
-Individuals cannot be forced to stay in an experiment
-Experiments/RCTs cannot estimate many important
parameters of interest
-Experiments/RCTs in a limited, pilot setting can mislead
about the impact of the “scaled up” program
-Randomization is often not straightforward

91. 1.Participants in the experiment are randomly
selected from the population of interest and
randomly assigned to their program participation
status;
2.All participants in the trial comply with the program
participation status to which they are assigned;
3.The experiment lasts long enough to replicate the
program under consideration and to influence
outcomes;
4.There are no social interactions that may make a
full-scale program inherently different from a smaller
scale intervention

92. Time
Approval/
acceptance

93. Time
Approval/
acceptance
Phase 1

94. Time
Approval/
acceptance
Phase 2

95. Time
Approval/
acceptance
Phase 3

96. Time
Approval/
acceptance
Phase 4

97. Time
Approval/
acceptance
Phase 5

98. Time
Approval/
acceptance
Phase 2

99. Time
Approval/
acceptance
Phase 3

103. Conclusion

104. Conclusion

105. Conclusion

106. Conclusion

107. Links:
The manual:
http://www.measureevaluation.org/resources/publications/ms-
14-87-en
The webinar introducing the manual:
http://www.measureevaluation.org/resources/webinars/metho
ds-for-program-impact-evaluation
My email:
pmlance@email.unc.edu

108. MEASURE Evaluation is funded by the U.S. Agency
for International Development (USAID) under terms
of Cooperative Agreement AID-OAA-L-14-00004 and
implemented by the Carolina Population Center, University
of North Carolina at Chapel Hill in partnership with ICF
International, John Snow, Inc., Management Sciences for
Health, Palladium Group, and Tulane University. The views
expressed in this presentation do not necessarily reflect
the views of USAID or the United States government.
www.measureevaluation.org