Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Randomization and Its Discontents

881 views

Published on

Webinar presentation by Peter Lance

Published in: Education

Randomization and Its Discontents

  1. 1. Peter M. Lance, PhD MEASURE Evaluation University of North Carolina at Chapel Hill JUNE 29, 2016 Randomization and Its Discontents
  2. 2. Global, five-year, $180M cooperative agreement Strategic objective: To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time. Project overview
  3. 3. Improved country capacity to manage health information systems, resources, and staff Strengthened collection, analysis, and use of routine health data Methods, tools, and approaches improved and applied to address health information challenges and gaps Increased capacity for rigorous evaluation Phase IV Results Framework
  4. 4. Global footprint (more than 25 countries)
  5. 5. • The program impact evaluation challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  6. 6. • The program impact evaluation challenge • Randomization • Selection on observables • Within estimators • Instrumental variables
  7. 7. Did the program make a difference?
  8. 8. Did the program cause a change in an outcome of interest Y ? (Causality)
  9. 9. What happens if the individual participates {Causal} Program impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program impact What happens if the individual does not participate
  10. 10. Average treatment effect (ATE) 𝐸 𝑌1 − 𝑌0 Average effect of treatment on the treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Treatment effects
  11. 11. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome
  12. 12. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome
  13. 13. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome
  14. 14. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome
  15. 15. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome Fundamental Identification Problem of Program Impact Evaluation
  16. 16. 𝑌𝑖 1 , 𝑌𝑖 0 Observed outcome Fundamental identification problem of program impact evaluation
  17. 17. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
  18. 18. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
  19. 19. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
  20. 20. Impact evaluation 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1
  21. 21. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y Across Sample of Non−Participants
  22. 22. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y across sample of non−participants
  23. 23. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y across sample of non−participants
  24. 24. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y across sample of non−participants
  25. 25. Impact evaluation 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 𝐸 𝑌0|𝑃 = 0 = 𝐸(𝑌0|𝑃 = 1)
  26. 26. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 1
  27. 27. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0|𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 1
  28. 28. X Y P
  29. 29. X Y P 𝑌 = 𝑃 ∙ 𝑌1 + 1 − 𝑃 ∙ 𝑌0
  30. 30. X Y P
  31. 31. X Y P
  32. 32. 𝐸 𝑌1 − 𝑌0 |𝑃 = 1 = 𝐸 𝑌1|𝑃 = 1 − 𝐸 𝑌0|𝑃 = 1 Average Y across sample of participants − Average Y across sample of non−participants
  33. 33. The big idea (part un) So, the basic idea of the randomization/experimental approach to impact evaluation is that by randomizing program participation we insure that participants and non-participants are alike on average in terms of their characteristics
  34. 34. The big idea (part deux) If this is the case, then any differences in average outcomes between the two groups can be ascribed to the one way in which they do differ: program participation
  35. 35. 1.Participants in the experiment are randomly selected from the population of interest and randomly assigned to their program participation status; 2.All participants in the trial comply with the program participation status to which they are assigned; 3.The experiment lasts long enough to replicate the program under consideration and to influence outcomes; 4.There are no social interactions that may make a full-scale program inherently different from a smaller scale intervention
  36. 36. Source: Washington Policy Center
  37. 37. Source: EH.net
  38. 38. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 𝑌 = 𝑃 ∙ 𝑌1 + 1 − 𝑃 ∙ 𝑌0
  39. 39. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y Across Sample of Participants − Average Y Across Sample of Non−Participants
  40. 40. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y across sample of participants − Average Y Across Sample of Non−Participants
  41. 41. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 Average Y across sample of participants − Average Y across sample of non−participants
  42. 42. X Y P
  43. 43. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋|𝑃 = 0
  44. 44. X Y P
  45. 45. 𝐸 𝑌1|𝑃 = 1 > 𝐸 𝑌1 > 𝐸 𝑌1|𝑃 = 0
  46. 46. 𝐸 𝑌0|𝑃 = 0 < 𝐸 𝑌0 < 𝐸 𝑌0|𝑃 = 1
  47. 47. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 (Overestimated) (Underestimated)
  48. 48. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 (Overestimated) (Underestimated)
  49. 49. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 (Overestimated) (Underestimated)
  50. 50. X Y P
  51. 51. X Y P
  52. 52. Refusal rates by plan Plan Refusal rate (%) Free 8 25 and 50% coinsurance 11 95% coinsurance 25 Source: Newhouse et al. (1993)
  53. 53. X Y P
  54. 54. X Y P
  55. 55. X Y P
  56. 56. If such and such is true of the real world processes that gave rise to program participation and outcomes in our observed non-experimental sample, then the estimates of program impact generated by this quasi- experimental estimator provide the causal impact of program participation on outcomes of interest.
  57. 57. Lalonde’s critique of non- experimental estimators
  58. 58. Lalonde’s critique of non- experimental estimators
  59. 59. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs carried out in a limited or pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  60. 60. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  61. 61. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  62. 62. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  63. 63. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  64. 64. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  65. 65. 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0
  66. 66. 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1 − 𝑌0 ≠ 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌1 − 𝑚𝑒𝑑𝑖𝑎𝑛 𝑌0
  67. 67. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  68. 68. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  69. 69. -Individuals cannot be forced to participate in a program -Individuals cannot be forced to accept their random experimental assignment -Individuals assigned to the control/non-participant group cannot be prevented from seeking alternatives -Individuals cannot be forced to stay in an experiment -Experiments/RCTs cannot estimate many important parameters of interest -Experiments/RCTs in a limited, pilot setting can mislead about the impact of the “scaled up” program -Randomization is often not straightforward
  70. 70. 1.Participants in the experiment are randomly selected from the population of interest and randomly assigned to their program participation status; 2.All participants in the trial comply with the program participation status to which they are assigned; 3.The experiment lasts long enough to replicate the program under consideration and to influence outcomes; 4.There are no social interactions that may make a full-scale program inherently different from a smaller scale intervention
  71. 71. Time Approval/ acceptance
  72. 72. Time Approval/ acceptance Phase 1
  73. 73. Time Approval/ acceptance Phase 2
  74. 74. Time Approval/ acceptance Phase 3
  75. 75. Time Approval/ acceptance Phase 4
  76. 76. Time Approval/ acceptance Phase 5
  77. 77. Time Approval/ acceptance Phase 2
  78. 78. Time Approval/ acceptance Phase 3
  79. 79. Conclusion
  80. 80. Conclusion
  81. 81. Conclusion
  82. 82. Conclusion
  83. 83. Links: The manual: http://www.measureevaluation.org/resources/publications/ms- 14-87-en The webinar introducing the manual: http://www.measureevaluation.org/resources/webinars/metho ds-for-program-impact-evaluation My email: pmlance@email.unc.edu
  84. 84. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 and implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org

×