The document discusses post-treatment bias in social science research. It begins by listing several big social science questions that are difficult to answer, such as whether democratization reduces conflict or ethnic diversity causes civil war. It then explains that while experiments are useful, most research is observational. It discusses how causal inference works by comparing equivalent treatment and control groups, but that omitted variable bias is a major challenge if confounding factors differ between the groups. Post-treatment bias occurs when researchers control for consequences of the treatment or have the wrong causal ordering of predictors.

1. A Hard Unsolved Problem?
Post-Treatment Bias in Big Social Science Questions
Gary King
Institute for Quantitative Social Science
Harvard University
Talk at the “Hard Problems in Social Science” Symposium,
Harvard University 4/10/2010
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

2. A Few Big Social Science Questions We Can’t Answer
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

3. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

4. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

5. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Does trade openness reduce the risk of state failure?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

6. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Does trade openness reduce the risk of state failure?
Do strong international institutions lead to or result from
international cooperation?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

7. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Does trade openness reduce the risk of state failure?
Do strong international institutions lead to or result from
international cooperation?
How to distinguish between (1) the effect of your health on others
and (2) the effect of interventions to improve your health on others?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

8. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Does trade openness reduce the risk of state failure?
Do strong international institutions lead to or result from
international cooperation?
How to distinguish between (1) the effect of your health on others
and (2) the effect of interventions to improve your health on others?
Is individual student achievement caused by (1) students’
socioeconomic characteristics or (2) their peers’ average achievement?
(I.e., does tutoring a subgroup help others and then feedback?)
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

9. A Few Big Social Science Questions We Can’t Answer
Does democratization reduce international conflict?
Does ethnic diversity in developing countries cause civil war?
Does trade openness reduce the risk of state failure?
Do strong international institutions lead to or result from
international cooperation?
How to distinguish between (1) the effect of your health on others
and (2) the effect of interventions to improve your health on others?
Is individual student achievement caused by (1) students’
socioeconomic characteristics or (2) their peers’ average achievement?
(I.e., does tutoring a subgroup help others and then feedback?)
Thousands of other examples in most areas of the social sciences.
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

10. Experiments are terrific, but that’s not the issue
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

11. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

12. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

13. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

14. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

15. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

16. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

17. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Most knowledge learned from experiments is observational
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

18. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Most knowledge learned from experiments is observational
So yes
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

19. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Most knowledge learned from experiments is observational
So yes
experiments are terrific
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

20. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Most knowledge learned from experiments is observational
So yes
experiments are terrific
we can’t run them that often
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

21. Experiments are terrific, but that’s not the issue
Experimental research: requires random treatment assignment
(Perhaps the most important methodological idea in the last century)
Random assignment: occasionally possible, usually not
The vast majority of research is observational
The vast majority of knowledge learned is from observation
Most knowledge learned from experiments is observational
So yes
experiments are terrific
we can’t run them that often
but we can often learn plenty without experiments
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

22. Progress in Causal Inference
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

23. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

24. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

25. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

26. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

27. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

28. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

29. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

30. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
What if those who get the medicine are healthier than those who don’t?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

31. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
What if those who get the medicine are healthier than those who don’t?
What if those who participate in a job training program are less
educated?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

32. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
What if those who get the medicine are healthier than those who don’t?
What if those who participate in a job training program are less
educated?
Well-known solutions to omitted variable bias:
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

33. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
What if those who get the medicine are healthier than those who don’t?
What if those who participate in a job training program are less
educated?
Well-known solutions to omitted variable bias:
Randomize treatment (all confounders are unrelated to treatment)
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

34. Progress in Causal Inference
Inference: using facts you have to learn about facts you don’t have
Causal effect: Your outcome with the treatment minus your outcome
if you had not received the treatment
Causal inference: estimating the causal effect
Get equivalent treatment and control groups
Apply or observe the treatment
average(outcome for treateds) − average(outcome for controls)
Biggest problem we know about: Omitted variable bias
What if those who get the medicine are healthier than those who don’t?
What if those who participate in a job training program are less
educated?
Well-known solutions to omitted variable bias:
Randomize treatment (all confounders are unrelated to treatment)
Control (physically or statistically) for the potential confounders
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

35. So What’s Post-Treatment Bias?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

36. So What’s Post-Treatment Bias?
It occurs:
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

37. So What’s Post-Treatment Bias?
It occurs:
when controlling away for the consequences of treatment
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

38. So What’s Post-Treatment Bias?
It occurs:
when controlling away for the consequences of treatment
when causal ordering among predictors is ambiguous or wrong
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

39. So What’s Post-Treatment Bias?
It occurs:
when controlling away for the consequences of treatment
when causal ordering among predictors is ambiguous or wrong
Resulting biases can go in either direction
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

40. So What’s Post-Treatment Bias?
It occurs:
when controlling away for the consequences of treatment
when causal ordering among predictors is ambiguous or wrong
Resulting biases can go in either direction
The problem is obvious once you think about it
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

41. So What’s Post-Treatment Bias?
It occurs:
when controlling away for the consequences of treatment
when causal ordering among predictors is ambiguous or wrong
Resulting biases can go in either direction
The problem is obvious once you think about it
For many big social science questions, we have no solution
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

42. Examples of Avoidable Post-Treatment Bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

43. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

44. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

45. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

46. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

47. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

48. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Don’t control for position in the firm
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

49. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Don’t control for position in the firm
Causal effect of medicine on health
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

50. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Don’t control for position in the firm
Causal effect of medicine on health
Do control for health prior to the treatment decision
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

51. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Don’t control for position in the firm
Causal effect of medicine on health
Do control for health prior to the treatment decision
Do not control for side effects or other medicines taken later
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

52. Examples of Avoidable Post-Treatment Bias
Causal effect of Party ID on the Vote
Do control for race
Do not control for voting intentions five minutes before voting
Causal effect of Race on Salary in a firm
Do control for qualifications
Don’t control for position in the firm
Causal effect of medicine on health
Do control for health prior to the treatment decision
Do not control for side effects or other medicines taken later
Post-treatment bias in these examples: easy to see & avoid
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

53. Examples of Unavoidable Post-Treatment Bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

54. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

55. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

56. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

57. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

58. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Yes, since trade openness→legislative effectiveness, we must control to
avoid omitted variable bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

59. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Yes, since trade openness→legislative effectiveness, we must control to
avoid omitted variable bias
No, since legislative effectiveness→trade openness, we would have
post-treatment bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

60. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Yes, since trade openness→legislative effectiveness, we must control to
avoid omitted variable bias
No, since legislative effectiveness→trade openness, we would have
post-treatment bias
Causal effect of tutoring on individual test scores; do we control for
average test scores?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

61. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Yes, since trade openness→legislative effectiveness, we must control to
avoid omitted variable bias
No, since legislative effectiveness→trade openness, we would have
post-treatment bias
Causal effect of tutoring on individual test scores; do we control for
average test scores?
Yes, since peers→individual scores, and so must control to avoid
omitted variable bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

62. Examples of Unavoidable Post-Treatment Bias
Causal effect of democratization on civil war; do we control for GDP?
Yes, since GDP→democratization we must control to avoid omitted
variable bias
No, since democratization→GDP, we would have post-treatment bias
Causal effect of legislative effectiveness on state failure; do we control
for trade openness?
Yes, since trade openness→legislative effectiveness, we must control to
avoid omitted variable bias
No, since legislative effectiveness→trade openness, we would have
post-treatment bias
Causal effect of tutoring on individual test scores; do we control for
average test scores?
Yes, since peers→individual scores, and so must control to avoid
omitted variable bias
No, since individual scores→averages, we would have post-treatment
bias
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

63. Solutions for Post-Treatment Bias?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

64. Solutions for Post-Treatment Bias?
Is there a statistical fix?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

65. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

66. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

67. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

68. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

69. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

70. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

71. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

72. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

73. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues? See Kosuke Imai, Adam Glynn,
Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler
VanderWeele, and others.
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

74. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues? See Kosuke Imai, Adam Glynn,
Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler
VanderWeele, and others.
Are there areas of social science scholarship not affected?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

75. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues? See Kosuke Imai, Adam Glynn,
Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler
VanderWeele, and others.
Are there areas of social science scholarship not affected? Yes, but
the bigger the question the more likely you’ll find the problem
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

76. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues? See Kosuke Imai, Adam Glynn,
Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler
VanderWeele, and others.
Are there areas of social science scholarship not affected? Yes, but
the bigger the question the more likely you’ll find the problem
Is there hope?
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

77. Solutions for Post-Treatment Bias?
Is there a statistical fix? Nope.
What about trying it both ways (include and exclude)? Nope. The
estimates do not bound the truth!
Is there identifying information in the ideal case? In many situations,
no; in others we don’t know.
Can we redesign data collection to avoid the problem? Usually not.
Is there progress on related issues? See Kosuke Imai, Adam Glynn,
Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler
VanderWeele, and others.
Are there areas of social science scholarship not affected? Yes, but
the bigger the question the more likely you’ll find the problem
Is there hope? There’s always hope; just no answers!
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9

78. For more information
http://GKing.Harvard.edu
Talk at the “Hard Problems in S
Gary King (Harvard IQSS) Post-Treatment Bias /9