Fundamentals of Program Impact Evaluation

1.
Peter M. Lance,PhD MEASURE Evaluation University of North Carolina at Chapel Hill MARCH 31, 2016 Fundamentals of Program Impact Evaluation

2.
Global, five-year, $180Mcooperative agreement Strategic objective: To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time. Project overview

3.
Improved country capacityto manage health information systems, resources, and staff Strengthened collection, analysis, and use of routine health data Methods, tools, and approaches improved and applied to address health information challenges and gaps Increased capacity for rigorous evaluation Phase IV Results Framework

4.
Global footprint (morethan 25 countries)

6.
How Do WeKnow IfAProgram MadeADifference? ABrief Helicopter Tour of Methods for Estimating Program Impact

7.
• The ProgramImpact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables

8.

9.

10.

11.

12.

13.

14.
Newton’s “Laws” ofMotion 𝐹𝑜𝑟𝑐𝑒 = 𝑀𝑎𝑠𝑠 ∙ 𝐴𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛

17.
Did the programmake a difference?

18.
Did the programcause a change in an outcome of interest Y ? (Causality)

19.
Our outcome ofInterest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌 : 𝑌0 : 𝑌1 :

20.
Our outcome ofinterest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌𝑖 : 𝑌𝑖 0 : 𝑌𝑖 1 :

21.

22.

23.
What happens if theindividual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate

24.

25.

26.

27.

28.
𝑃𝑖 = 1ifindividual 𝑖participates 0if individual 𝑖 does not participate Program Participation

29.
𝑌𝑖 = 𝑃𝑖∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome

30.
𝑌𝑖 = 𝑃𝑖∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

31.
𝑌𝑖 = 1∙ 𝑌𝑖 1 + 1 − 1 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

32.
𝑌𝑖 = 𝑌𝑖 1 +0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

33.
𝑌𝑖 = 𝑌𝑖 1 ObservedOutcome 𝑃𝑖 = 1

34.
𝑌𝑖 = 𝑃𝑖∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome

35.
𝑌𝑖 = 𝑃𝑖∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0

36.
𝑌𝑖 = 0∙ 𝑌𝑖 1 + 1 − 0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0

37.
𝑌𝑖 = 𝑌𝑖 0 ObservedOutcome 𝑃𝑖 = 0

38.
𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome

39.

40.

41.

42.
𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome FundamentalIdentification Problem of Program Impact Evaluation

43.
𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome FundamentalIdentification Problem of Program Impact Evaluation

44.
Individual Population

45.
Individual Population Hi. Theycall me individual i

46.
Individual Population ?!?

48.
𝑌𝑖 1 , 𝑌𝑖 0

49.
𝑌𝑖 1 , 𝑌𝑖 0

50.
An expected valuefor a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value

51.
An expected valuefor a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value

52.
An expected valueis the true average of a random variable across a population 𝐸 𝑋 = sometruevalue Expected Value

53.
𝐸 𝑐 =𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

54.
𝑬 𝒄 =𝒄 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

55.
𝐸 𝑐 =𝑐 𝑬 𝒄 ∙ 𝑾 = 𝒄 ∙ 𝑬 𝑾 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

56.
𝐸 𝑐 =𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝑬 𝑾 + 𝒁 = 𝑬 𝑾 + 𝑬 𝒁 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

57.
𝐸 𝑐 =𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝑬 𝑾 − 𝒁 = 𝑬 𝑾 − 𝑬 𝒁 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

58.
𝐸 𝑐 =𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝑬 𝒂 ∙ 𝑾 ± 𝒃 ∙ 𝒁 = 𝒂 ∙ 𝑬 𝑾 ± 𝒃 ∙ 𝑬 𝒁 Expectations: Properties

59.
𝐸 𝑊 ∙𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

60.
𝑬 𝑾 ∙𝒁 ≠ 𝑬 𝑾 ∙ 𝑬 𝒁 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

61.
𝐸 𝑊 ∙𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝑬 𝑾 𝒁 ≠ 𝑬 𝑾 𝑬 𝒁 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

62.
𝐸 𝑊 ∙𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝑬 𝒇 𝑾 ≠ 𝒇 𝑬 𝑾 Expectations: Properties

63.
𝑌𝑖 1 − 𝑌𝑖 0 Average TreatmentEffect (ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Hi there Individual Impact

64.
𝑌𝑖 1 − 𝑌𝑖 0

65.
𝐸 𝑌𝑖 1 − 𝑌𝑖 0

66.
Average Treatment Effect(ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Treatment Effects

69.
Suppose that wehave a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

70.
Suppose that wehave a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

71.
Remember, however, akey property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

72.
Remember, however, akey property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

73.
For instance, supposethat in our sample we have: 𝑛 𝑃 participants(𝑃𝑖 = 1) and 𝑛 𝑁 non-participants(𝑃𝑖 = 0) (hence 𝑛 𝑃 + 𝑛 𝑁 = 𝑛) So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

74.
Then an estimatorof 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

75.
Then an estimatorof 𝐸 𝑌1 is 𝒀 𝟏 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

76.
Then an estimatorof 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝒏 𝑷 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

77.
Then an estimatorof 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

78.
Then an estimatorof 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝒏 𝑷 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

79.
Then an estimatorof 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝟏 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

80.
Similarly, an estimatorof 𝐸 𝑌0 is 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 calculated with the 𝑛 𝑁 non-participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

81.
So then anestimate of 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 is 𝑌1 − 𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

82.
But is ita good estimate??

86.
So we havetwo samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁

87.

88.

89.

90.

91.
So we havetwo samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 𝒀 𝟏 𝑬 𝒀 𝟏

102.
𝒀 𝟏 = 𝒋=𝟏 𝒏𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷

103.
𝒀 𝟏 = 𝒋=𝟏 𝒏𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷

109.
𝒀 𝟏 = 𝒋=𝟏 𝒏𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏

110.
𝒀 𝟏 = 𝒋=𝟏 𝒏𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏

113.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

114.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

115.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

116.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

117.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿

118.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿

119.
𝑬 𝒀 𝟏= 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏

120.
𝑬 𝒀 𝟏= 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

121.
𝑬 𝒀 𝟏= 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

122.
𝑬 𝒀 𝟏= 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

123.
𝑬 𝒀𝒋 𝟏 = 𝑬𝒀 𝟏

126.
𝑬 𝒀 𝟏

127.
𝑃 = 0 𝑃= 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝒀 𝟏

128.
𝑃 = 0 𝑃= 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 0 𝒀 𝟏

129.
𝑃 = 1 𝑃= 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝒀 𝟏

130.
Z W “Z CausesW” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

131.
Z W “Z causesW” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

132.
Z W “Z causesW” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

133.
X Y1

134.
X Y P

135.
X Y P 0

136.
X Y P

137.
X Y P

138.
𝐸 𝑋|𝑃 =1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

139.
𝐸 𝑋|𝑃 =1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

140.
𝐸 𝑋|𝑃 =1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

141.
X Y P

142.
𝑃 = 1 𝑃= 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1

144.
𝐸 𝑌0 |𝑃 =0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0

145.
𝐸 𝑌0 |𝑃 =0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0

146.
The estimator 𝑌1 −𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 of 𝐸 𝑌1 − 𝑌0 would be biased if some individuals occurred only among participants or non-participants Or more often among one of the two groups

147.
X Y P

148.
X Y P

149.
Sir Austin BradfordHill

150.
Strength: How strongis the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria

151.

152.

153.

154.

155.

156.

157.

158.

159.

160.

161.
We are presentedwith data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏

162.
We are presentedwith data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.

163.
We are presentedwith data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.

164.
Conclusion

165.
Links: The manual: http://www.measureevaluation.org/resources/publications/ms- 14-87-en The webinarintroducing the manual: http://www.measureevaluation.org/resources/webinars/metho ds-for-program-impact-evaluation My email: pmlance@email.unc.edu

166.
MEASURE Evaluation isfunded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 and implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org

Fundamentals of Program Impact Evaluation

More Related Content

What's hot

Viewers also liked

Similar to Fundamentals of Program Impact Evaluation

More from removed_62798267384a091db5c693ad7f1cc5ac

Recently uploaded

Fundamentals of Program Impact Evaluation