1. An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation
Peter M. Lance
MEASURE Evaluation
September 25, 2014

2. Where it all began….

5. Approachable
But Basic
Sophisticated
But
Unapproachable

6. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

7. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

8. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

9. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

10. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

11. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

12. Simple but Limited
1. Heavy reliance on verbal explanations
-Does not convey much nuance
-Does not provide much insight about data structure
-Do not typically relate different IE methods well
2. Little guidance on how actually to estimate the
models
3. Tend to focus on basics, with little recent thinking

13. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

14. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

15. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

16. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

17. A Tale of Two Manuals:
Manual 1
Manual 2
푌1
푌1
The Outcome if Individual i participates
푃
푇
Program Participation/Treatment Exposure of individual i
푇
푡
Time
푁
푛
Number of observations
푃
푁
Size of the population
Δ푥
푥
Change in 푥

18. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

19. Obviously leads to
풏≈ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ ퟏ+ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ 푵
Obviously…. Totally
푃푟−푑≤푝−푃≤푑|푁≥1−훼
The Inequality

20. A B C D E F G

21. A B C D E F G

22. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

23. Sophisticated and Cutting Edge, but…
1. Reliance on dense mathematics
-Assume a lot of background knowledge
-A lot of arcane, inconsistent notation
-Little guidance on the mathematical journey
2. Still provide little practical illustration of how actually to estimate models or practical performance
3. Often Lopsided

25. Key Guiding Features:
1. Consistent Notation
2. Little Prior Knowledge Assumed
3. Explicit Derivation of All Results in the Manual with verbal/narrative explanation
4. Reliance on a single “behavioral model” woven through the discussion
4. Explicit discussion of the practical estimation

26. 5. Equal coverage of major IE traditions
6. Practical examples of how to estimate the IE models and their applied performance in STATA and

27. 5. Equal coverage of major IE traditions
6. Practical examples of how to estimate the IE models and their applied performance in STATA and …. All Statistical Programs for them Released with Manual!!!

28. Topics Covered
1. Introduction to the basic challenge of IE
2. RCTs
3. Selection on Observables
4. “Within” Estimators
5. Instrumental Variables (including Regression Discontinuity)

29. http://www.measureevaluation.org/publications/ms-14-87-en

40. Homer Simpson

41. Homer Simpson
homer@email.deepthinkingu.edu

42. Homer Simpson
homer@email.deepthinkingu.edu
The University of Deep Thinking

43. Homer Simpson
homer@email.deepthinkingu.edu
The University of Deep Thinking
Dear Authors,
Wow this manual is boring. Too much blah blah blah and not enough
cartoons. Yours, Homer

44. Homer Simpson
homer@email.deepthinkingu.edu
The University of Deep Thinking
Dear Authors,
Wow this manual is boring. Too much blah blah blah and not enough
cartoons. Yours, Homer

45. The Future
1. Continued revisions to the first edition of the first volume
2. A second edition of the first volume:
-Expanded Empirical Examples (adding in new STATA treatment effects commands, examples in R, etc.);
-Separate chapter for Regression Discontinuity;
-Extending behavioral model to RCTs;
-What else?

46. The Future
3. A Potential Second Volume
-Hybrid models;
-Deeply structural models;
-Sample selection/control function rooted models;
-Aggregate data (e.g. time series basics, synthetic cohort models, etc.);
-What else?

47. MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) and implemented by the
Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF International, John Snow, Inc., Management Sciences for
Health, and Tulane University. Views expressed in this
presentation do not necessarily reflect the views of USAID or the
U.S. government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.