Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation

2,680 views

Published on

Presented by Peter M. Lance in a September 2014 webinar.

Published in: Education
  • Be the first to comment

An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation

  1. 1. An Introduction to the Manual: How Do We Know if a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation Peter M. Lance MEASURE Evaluation September 25, 2014
  2. 2. Where it all began….
  3. 3. Approachable But Basic Sophisticated But Unapproachable
  4. 4. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  5. 5. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  6. 6. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  7. 7. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  8. 8. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  9. 9. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  10. 10. Simple but Limited 1. Heavy reliance on verbal explanations -Does not convey much nuance -Does not provide much insight about data structure -Do not typically relate different IE methods well 2. Little guidance on how actually to estimate the models 3. Tend to focus on basics, with little recent thinking
  11. 11. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  12. 12. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  13. 13. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  14. 14. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  15. 15. A Tale of Two Manuals: Manual 1 Manual 2 푌1 푌1 The Outcome if Individual i participates 푃 푇 Program Participation/Treatment Exposure of individual i 푇 푡 Time 푁 푛 Number of observations 푃 푁 Size of the population Δ푥 푥 Change in 푥
  16. 16. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  17. 17. Obviously leads to 풏≈ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ ퟏ+ 풁ퟏ−휶 ퟐ ퟐ ∙풑∙ퟏ−풑 풅ퟐ 푵 Obviously…. Totally 푃푟−푑≤푝−푃≤푑|푁≥1−훼 The Inequality
  18. 18. A B C D E F G
  19. 19. A B C D E F G
  20. 20. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  21. 21. Sophisticated and Cutting Edge, but… 1. Reliance on dense mathematics -Assume a lot of background knowledge -A lot of arcane, inconsistent notation -Little guidance on the mathematical journey 2. Still provide little practical illustration of how actually to estimate models or practical performance 3. Often Lopsided
  22. 22. Key Guiding Features: 1. Consistent Notation 2. Little Prior Knowledge Assumed 3. Explicit Derivation of All Results in the Manual with verbal/narrative explanation 4. Reliance on a single “behavioral model” woven through the discussion 4. Explicit discussion of the practical estimation
  23. 23. 5. Equal coverage of major IE traditions 6. Practical examples of how to estimate the IE models and their applied performance in STATA and
  24. 24. 5. Equal coverage of major IE traditions 6. Practical examples of how to estimate the IE models and their applied performance in STATA and …. All Statistical Programs for them Released with Manual!!!
  25. 25. Topics Covered 1. Introduction to the basic challenge of IE 2. RCTs 3. Selection on Observables 4. “Within” Estimators 5. Instrumental Variables (including Regression Discontinuity)
  26. 26. http://www.measureevaluation.org/publications/ms-14-87-en
  27. 27. Homer Simpson
  28. 28. Homer Simpson homer@email.deepthinkingu.edu
  29. 29. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking
  30. 30. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking Dear Authors, Wow this manual is boring. Too much blah blah blah and not enough cartoons. Yours, Homer
  31. 31. Homer Simpson homer@email.deepthinkingu.edu The University of Deep Thinking Dear Authors, Wow this manual is boring. Too much blah blah blah and not enough cartoons. Yours, Homer
  32. 32. The Future 1. Continued revisions to the first edition of the first volume 2. A second edition of the first volume: -Expanded Empirical Examples (adding in new STATA treatment effects commands, examples in R, etc.); -Separate chapter for Regression Discontinuity; -Extending behavioral model to RCTs; -What else?
  33. 33. The Future 3. A Potential Second Volume -Hybrid models; -Deeply structural models; -Sample selection/control function rooted models; -Aggregate data (e.g. time series basics, synthetic cohort models, etc.); -What else?
  34. 34. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group International, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. government. MEASURE Evaluation is the USAID Global Health Bureau's primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.

×