Gary King- Hard Unsolved Problem in Social Science:Post Treatment Bias

  • 1,357 views
Uploaded on

Presentation by Gary King on April 10th Symposium at Harvard on the hardest problems in the Social Sciences. …

Presentation by Gary King on April 10th Symposium at Harvard on the hardest problems in the Social Sciences.
Discuss, propose and vote on what you think are the hardest unsolved problems in the social sciences on our Facebook Fan Page: http://www.facebook.com/pages/Hard-Problems-in-Social-Science/111085732253765

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Check out the Hardest unsolved problems in the Social Sciences Facebook Fan page to discuss, propose and vote over the next 45 days on what you think are the hardest problems.
    http://www.facebook.com/pages/Hard-Problems-in-Social-Science/111085732253765
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,357
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
17
Comments
1
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. A Hard Unsolved Problem? Post-Treatment Bias in Big Social Science Questions Gary King Institute for Quantitative Social Science Harvard University Talk at the “Hard Problems in Social Science” Symposium, Harvard University 4/10/2010 Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 2. A Few Big Social Science Questions We Can’t Answer Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 3. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 4. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 5. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Does trade openness reduce the risk of state failure? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 6. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Does trade openness reduce the risk of state failure? Do strong international institutions lead to or result from international cooperation? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 7. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Does trade openness reduce the risk of state failure? Do strong international institutions lead to or result from international cooperation? How to distinguish between (1) the effect of your health on others and (2) the effect of interventions to improve your health on others? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 8. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Does trade openness reduce the risk of state failure? Do strong international institutions lead to or result from international cooperation? How to distinguish between (1) the effect of your health on others and (2) the effect of interventions to improve your health on others? Is individual student achievement caused by (1) students’ socioeconomic characteristics or (2) their peers’ average achievement? (I.e., does tutoring a subgroup help others and then feedback?) Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 9. A Few Big Social Science Questions We Can’t Answer Does democratization reduce international conflict? Does ethnic diversity in developing countries cause civil war? Does trade openness reduce the risk of state failure? Do strong international institutions lead to or result from international cooperation? How to distinguish between (1) the effect of your health on others and (2) the effect of interventions to improve your health on others? Is individual student achievement caused by (1) students’ socioeconomic characteristics or (2) their peers’ average achievement? (I.e., does tutoring a subgroup help others and then feedback?) Thousands of other examples in most areas of the social sciences. Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 10. Experiments are terrific, but that’s not the issue Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 11. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 12. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 13. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 14. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 15. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 16. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 17. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Most knowledge learned from experiments is observational Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 18. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Most knowledge learned from experiments is observational So yes Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 19. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Most knowledge learned from experiments is observational So yes experiments are terrific Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 20. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Most knowledge learned from experiments is observational So yes experiments are terrific we can’t run them that often Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 21. Experiments are terrific, but that’s not the issue Experimental research: requires random treatment assignment (Perhaps the most important methodological idea in the last century) Random assignment: occasionally possible, usually not The vast majority of research is observational The vast majority of knowledge learned is from observation Most knowledge learned from experiments is observational So yes experiments are terrific we can’t run them that often but we can often learn plenty without experiments Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 22. Progress in Causal Inference Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 23. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 24. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 25. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 26. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 27. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 28. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 29. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 30. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias What if those who get the medicine are healthier than those who don’t? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 31. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias What if those who get the medicine are healthier than those who don’t? What if those who participate in a job training program are less educated? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 32. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias What if those who get the medicine are healthier than those who don’t? What if those who participate in a job training program are less educated? Well-known solutions to omitted variable bias: Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 33. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias What if those who get the medicine are healthier than those who don’t? What if those who participate in a job training program are less educated? Well-known solutions to omitted variable bias: Randomize treatment (all confounders are unrelated to treatment) Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 34. Progress in Causal Inference Inference: using facts you have to learn about facts you don’t have Causal effect: Your outcome with the treatment minus your outcome if you had not received the treatment Causal inference: estimating the causal effect Get equivalent treatment and control groups Apply or observe the treatment average(outcome for treateds) − average(outcome for controls) Biggest problem we know about: Omitted variable bias What if those who get the medicine are healthier than those who don’t? What if those who participate in a job training program are less educated? Well-known solutions to omitted variable bias: Randomize treatment (all confounders are unrelated to treatment) Control (physically or statistically) for the potential confounders Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 35. So What’s Post-Treatment Bias? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 36. So What’s Post-Treatment Bias? It occurs: Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 37. So What’s Post-Treatment Bias? It occurs: when controlling away for the consequences of treatment Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 38. So What’s Post-Treatment Bias? It occurs: when controlling away for the consequences of treatment when causal ordering among predictors is ambiguous or wrong Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 39. So What’s Post-Treatment Bias? It occurs: when controlling away for the consequences of treatment when causal ordering among predictors is ambiguous or wrong Resulting biases can go in either direction Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 40. So What’s Post-Treatment Bias? It occurs: when controlling away for the consequences of treatment when causal ordering among predictors is ambiguous or wrong Resulting biases can go in either direction The problem is obvious once you think about it Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 41. So What’s Post-Treatment Bias? It occurs: when controlling away for the consequences of treatment when causal ordering among predictors is ambiguous or wrong Resulting biases can go in either direction The problem is obvious once you think about it For many big social science questions, we have no solution Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 42. Examples of Avoidable Post-Treatment Bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 43. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 44. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 45. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 46. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 47. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 48. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Don’t control for position in the firm Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 49. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Don’t control for position in the firm Causal effect of medicine on health Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 50. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Don’t control for position in the firm Causal effect of medicine on health Do control for health prior to the treatment decision Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 51. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Don’t control for position in the firm Causal effect of medicine on health Do control for health prior to the treatment decision Do not control for side effects or other medicines taken later Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 52. Examples of Avoidable Post-Treatment Bias Causal effect of Party ID on the Vote Do control for race Do not control for voting intentions five minutes before voting Causal effect of Race on Salary in a firm Do control for qualifications Don’t control for position in the firm Causal effect of medicine on health Do control for health prior to the treatment decision Do not control for side effects or other medicines taken later Post-treatment bias in these examples: easy to see & avoid Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 53. Examples of Unavoidable Post-Treatment Bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 54. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 55. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 56. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 57. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 58. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Yes, since trade openness→legislative effectiveness, we must control to avoid omitted variable bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 59. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Yes, since trade openness→legislative effectiveness, we must control to avoid omitted variable bias No, since legislative effectiveness→trade openness, we would have post-treatment bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 60. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Yes, since trade openness→legislative effectiveness, we must control to avoid omitted variable bias No, since legislative effectiveness→trade openness, we would have post-treatment bias Causal effect of tutoring on individual test scores; do we control for average test scores? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 61. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Yes, since trade openness→legislative effectiveness, we must control to avoid omitted variable bias No, since legislative effectiveness→trade openness, we would have post-treatment bias Causal effect of tutoring on individual test scores; do we control for average test scores? Yes, since peers→individual scores, and so must control to avoid omitted variable bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 62. Examples of Unavoidable Post-Treatment Bias Causal effect of democratization on civil war; do we control for GDP? Yes, since GDP→democratization we must control to avoid omitted variable bias No, since democratization→GDP, we would have post-treatment bias Causal effect of legislative effectiveness on state failure; do we control for trade openness? Yes, since trade openness→legislative effectiveness, we must control to avoid omitted variable bias No, since legislative effectiveness→trade openness, we would have post-treatment bias Causal effect of tutoring on individual test scores; do we control for average test scores? Yes, since peers→individual scores, and so must control to avoid omitted variable bias No, since individual scores→averages, we would have post-treatment bias Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 63. Solutions for Post-Treatment Bias? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 64. Solutions for Post-Treatment Bias? Is there a statistical fix? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 65. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 66. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 67. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 68. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 69. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 70. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 71. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 72. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 73. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? See Kosuke Imai, Adam Glynn, Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler VanderWeele, and others. Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 74. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? See Kosuke Imai, Adam Glynn, Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler VanderWeele, and others. Are there areas of social science scholarship not affected? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 75. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? See Kosuke Imai, Adam Glynn, Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler VanderWeele, and others. Are there areas of social science scholarship not affected? Yes, but the bigger the question the more likely you’ll find the problem Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 76. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? See Kosuke Imai, Adam Glynn, Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler VanderWeele, and others. Are there areas of social science scholarship not affected? Yes, but the bigger the question the more likely you’ll find the problem Is there hope? Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 77. Solutions for Post-Treatment Bias? Is there a statistical fix? Nope. What about trying it both ways (include and exclude)? Nope. The estimates do not bound the truth! Is there identifying information in the ideal case? In many situations, no; in others we don’t know. Can we redesign data collection to avoid the problem? Usually not. Is there progress on related issues? See Kosuke Imai, Adam Glynn, Chuck Manski, Jamie Robins, Paul Rosenbaum, Don Rubin, Tyler VanderWeele, and others. Are there areas of social science scholarship not affected? Yes, but the bigger the question the more likely you’ll find the problem Is there hope? There’s always hope; just no answers! Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9
  • 78. For more information http://GKing.Harvard.edu Talk at the “Hard Problems in S Gary King (Harvard IQSS) Post-Treatment Bias /9