Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fundamentals of Program Impact Evaluation

2,445 views

Published on

This webinar, led by MEASURE Evaluation's Peter Lance, addressed the basic challenges of program impact estimation.

Published in: Education

Fundamentals of Program Impact Evaluation

  1. 1. Peter M. Lance, PhD MEASURE Evaluation University of North Carolina at Chapel Hill MARCH 31, 2016 Fundamentals of Program Impact Evaluation
  2. 2. Global, five-year, $180M cooperative agreement Strategic objective: To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time. Project overview
  3. 3. Improved country capacity to manage health information systems, resources, and staff Strengthened collection, analysis, and use of routine health data Methods, tools, and approaches improved and applied to address health information challenges and gaps Increased capacity for rigorous evaluation Phase IV Results Framework
  4. 4. Global footprint (more than 25 countries)
  5. 5. How Do We Know IfAProgram MadeADifference? ABrief Helicopter Tour of Methods for Estimating Program Impact
  6. 6. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  7. 7. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  8. 8. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  9. 9. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  10. 10. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  11. 11. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  12. 12. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  13. 13. Newton’s “Laws” of Motion 𝐹𝑜𝑟𝑐𝑒 = 𝑀𝑎𝑠𝑠 ∙ 𝐴𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛
  14. 14. Did the program make a difference?
  15. 15. Did the program cause a change in an outcome of interest Y ? (Causality)
  16. 16. Our outcome of Interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌 : 𝑌0 : 𝑌1 :
  17. 17. Our outcome of interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌𝑖 : 𝑌𝑖 0 : 𝑌𝑖 1 :
  18. 18. Our outcome of interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌𝑖 : 𝑌𝑖 0 : 𝑌𝑖 1 :
  19. 19. Our outcome of interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌𝑖 : 𝑌𝑖 0 : 𝑌𝑖 1 :
  20. 20. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate
  21. 21. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate
  22. 22. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate
  23. 23. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate
  24. 24. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate
  25. 25. 𝑃𝑖 = 1ifindividual 𝑖 participates 0if individual 𝑖 does not participate Program Participation
  26. 26. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome
  27. 27. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1
  28. 28. 𝑌𝑖 = 1 ∙ 𝑌𝑖 1 + 1 − 1 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1
  29. 29. 𝑌𝑖 = 𝑌𝑖 1 + 0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1
  30. 30. 𝑌𝑖 = 𝑌𝑖 1 Observed Outcome 𝑃𝑖 = 1
  31. 31. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome
  32. 32. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0
  33. 33. 𝑌𝑖 = 0 ∙ 𝑌𝑖 1 + 1 − 0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0
  34. 34. 𝑌𝑖 = 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0
  35. 35. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome
  36. 36. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome
  37. 37. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome
  38. 38. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome
  39. 39. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome Fundamental Identification Problem of Program Impact Evaluation
  40. 40. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome Fundamental Identification Problem of Program Impact Evaluation
  41. 41. Individual Population
  42. 42. Individual Population Hi. They call me individual i
  43. 43. Individual Population ?!?
  44. 44. 𝑌𝑖 1 , 𝑌𝑖 0
  45. 45. 𝑌𝑖 1 , 𝑌𝑖 0
  46. 46. An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value
  47. 47. An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value
  48. 48. An expected value is the true average of a random variable across a population 𝐸 𝑋 = sometruevalue Expected Value
  49. 49. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties
  50. 50. 𝑬 𝒄 = 𝒄 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties
  51. 51. 𝐸 𝑐 = 𝑐 𝑬 𝒄 ∙ 𝑾 = 𝒄 ∙ 𝑬 𝑾 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties
  52. 52. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝑬 𝑾 + 𝒁 = 𝑬 𝑾 + 𝑬 𝒁 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties
  53. 53. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝑬 𝑾 − 𝒁 = 𝑬 𝑾 − 𝑬 𝒁 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties
  54. 54. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝑬 𝒂 ∙ 𝑾 ± 𝒃 ∙ 𝒁 = 𝒂 ∙ 𝑬 𝑾 ± 𝒃 ∙ 𝑬 𝒁 Expectations: Properties
  55. 55. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties
  56. 56. 𝑬 𝑾 ∙ 𝒁 ≠ 𝑬 𝑾 ∙ 𝑬 𝒁 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties
  57. 57. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝑬 𝑾 𝒁 ≠ 𝑬 𝑾 𝑬 𝒁 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties
  58. 58. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝑬 𝒇 𝑾 ≠ 𝒇 𝑬 𝑾 Expectations: Properties
  59. 59. 𝑌𝑖 1 − 𝑌𝑖 0 Average Treatment Effect (ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Hi there Individual Impact
  60. 60. 𝑌𝑖 1 − 𝑌𝑖 0
  61. 61. 𝐸 𝑌𝑖 1 − 𝑌𝑖 0
  62. 62. Average Treatment Effect (ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Treatment Effects
  63. 63. Suppose that we have a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  64. 64. Suppose that we have a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  65. 65. Remember, however, a key property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  66. 66. Remember, however, a key property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  67. 67. For instance, suppose that in our sample we have: 𝑛 𝑃 participants(𝑃𝑖 = 1) and 𝑛 𝑁 non-participants(𝑃𝑖 = 0) (hence 𝑛 𝑃 + 𝑛 𝑁 = 𝑛) So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  68. 68. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  69. 69. Then an estimator of 𝐸 𝑌1 is 𝒀 𝟏 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  70. 70. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝒏 𝑷 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  71. 71. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  72. 72. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝒏 𝑷 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  73. 73. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝟏 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  74. 74. Similarly, an estimator of 𝐸 𝑌0 is 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 calculated with the 𝑛 𝑁 non-participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  75. 75. So then an estimate of 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 is 𝑌1 − 𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??
  76. 76. But is it a good estimate??
  77. 77. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁
  78. 78. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁
  79. 79. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁
  80. 80. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁
  81. 81. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁
  82. 82. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 𝒀 𝟏 𝑬 𝒀 𝟏
  83. 83. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷
  84. 84. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷
  85. 85. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏
  86. 86. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏
  87. 87. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊
  88. 88. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊
  89. 89. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊
  90. 90. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊
  91. 91. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿
  92. 92. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿
  93. 93. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏
  94. 94. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏
  95. 95. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏
  96. 96. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏
  97. 97. 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏
  98. 98. 𝑬 𝒀 𝟏
  99. 99. 𝑃 = 0 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝒀 𝟏
  100. 100. 𝑃 = 0 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 0 𝒀 𝟏
  101. 101. 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝒀 𝟏
  102. 102. Z W “Z Causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾
  103. 103. Z W “Z causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾
  104. 104. Z W “Z causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾
  105. 105. X Y1
  106. 106. X Y P
  107. 107. X Y P 0
  108. 108. X Y P
  109. 109. X Y P
  110. 110. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1
  111. 111. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1
  112. 112. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1
  113. 113. X Y P
  114. 114. 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1
  115. 115. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0
  116. 116. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0
  117. 117. The estimator 𝑌1 − 𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 of 𝐸 𝑌1 − 𝑌0 would be biased if some individuals occurred only among participants or non-participants Or more often among one of the two groups
  118. 118. X Y P
  119. 119. X Y P
  120. 120. Sir Austin Bradford Hill
  121. 121. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  122. 122. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  123. 123. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  124. 124. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  125. 125. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  126. 126. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  127. 127. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  128. 128. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  129. 129. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  130. 130. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  131. 131. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria
  132. 132. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏
  133. 133. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.
  134. 134. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.
  135. 135. Conclusion
  136. 136. Links: The manual: http://www.measureevaluation.org/resources/publications/ms- 14-87-en The webinar introducing the manual: http://www.measureevaluation.org/resources/webinars/metho ds-for-program-impact-evaluation My email: pmlance@email.unc.edu
  137. 137. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 and implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org

×