Plsc 503 spring-2013-lecture 12
Upcoming SlideShare
Loading in...5
×
 

Plsc 503 spring-2013-lecture 12

on

  • 1,586 views

 

Statistics

Views

Total Views
1,586
Views on SlideShare
139
Embed Views
1,447

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 1,447

http://www.thaddunning.com 1447

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Plsc 503 spring-2013-lecture 12 Plsc 503 spring-2013-lecture 12 Presentation Transcript

  • Standard natural experimentsRD designsPLSC 503: Quantitative Methods, Week 12Thad DunningDepartment of Political ScienceYale UniversityDesign-based inference; natural experiments; RDDLecture Notes, Week 12 1/ 55
  • Standard natural experimentsRD designsA growing emphasis on research designResearch design has always been important in teaching andwriting about methods.Lecture Notes, Week 12 2/ 55
  • Standard natural experimentsRD designsA growing emphasis on research designResearch design has always been important in teaching andwriting about methods.Yet, some researchers are increasingly convinced thatstatistical adjustment can do little to bolster valid causalinference, in the presence of less-than-ideal research designs.Lecture Notes, Week 12 2/ 55
  • Standard natural experimentsRD designsA growing emphasis on research designResearch design has always been important in teaching andwriting about methods.Yet, some researchers are increasingly convinced thatstatistical adjustment can do little to bolster valid causalinference, in the presence of less-than-ideal research designs.As Sekhon (2009: 487) puts it,“Without an experiment, natural experiment, a discontinuity, orsome other strong design, no amount of econometric orstatistical modeling can make the move from correlation tocausation persuasive.”Lecture Notes, Week 12 2/ 55
  • Standard natural experimentsRD designsGrowth of experiments0  2  4  6  8  10  12  14  Number  of  Ar+cles  Source:  Jamie  Druckman,  Donald  P.  Green,  James  H.  Kuklinski,  and  Arthur  Lupia.  2006.  “The  Growth  and  Development  of  Experimental  Research  PoliLcal  Science.”  American  Poli-cal  Science  Review  100:  627-­‐635.  Lecture Notes, Week 12 3/ 55
  • Standard natural experimentsRD designsGrowth of natural experiments0  10  20  30  40  50  60  70  80  90  1960-­‐1989   1990-­‐1999   2000-­‐2009  Number  of  Published  Ar1cles  Political Science EconomicsArticles published in major political science and economics journals with “natural experiment” in the title orabstract (as tracked in the online archive JSTOR).Natural Experiments in Political Science and EconomicsLecture Notes, Week 12 4/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.The analysis and interpretation of regression-discontinuitydesigns (RDD)Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.The analysis and interpretation of regression-discontinuitydesigns (RDD)Then we’ll ask the questions,Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.The analysis and interpretation of regression-discontinuitydesigns (RDD)Then we’ll ask the questions,What makes for a strong research design, and what are thestrengths and limitations of design-based research?Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.The analysis and interpretation of regression-discontinuitydesigns (RDD)Then we’ll ask the questions,What makes for a strong research design, and what are thestrengths and limitations of design-based research?What causal and statistical models are good starting points,and how can design assumptions be validated?Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsOur objectivesThis week we focus on research design and its relationship todata analysis, considering several topics:Natural experiments with as-if random assignment.The analysis and interpretation of regression-discontinuitydesigns (RDD)Then we’ll ask the questions,What makes for a strong research design, and what are thestrengths and limitations of design-based research?What causal and statistical models are good starting points,and how can design assumptions be validated?The distinction between “design-based” and “model-based”analysis is useful, though not absolute.Lecture Notes, Week 12 5/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining randomized controlled experimentsThree hallmarks of experiments:Lecture Notes, Week 12 6/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining randomized controlled experimentsThree hallmarks of experiments:Treatment and control groupsLecture Notes, Week 12 6/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining randomized controlled experimentsThree hallmarks of experiments:Treatment and control groupsRandom assignmentLecture Notes, Week 12 6/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining randomized controlled experimentsThree hallmarks of experiments:Treatment and control groupsRandom assignmentManipulationLecture Notes, Week 12 6/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining natural experimentsHallmarks of natural experiments:Treatment and control groupsLecture Notes, Week 12 7/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefining natural experimentsHallmarks of natural experiments:Treatment and control groupsRandom or “as-if” random assignmentNo manipulation by the researcherLecture Notes, Week 12 7/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefinition of a natural experiment:An observational study (that is, a study without an experimentalmanipulation) in which assignment to treatment and control groupsis done at random – or “as if” at randomLecture Notes, Week 12 8/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefinition of a natural experiment:An observational study (that is, a study without an experimentalmanipulation) in which assignment to treatment and control groupsis done at random – or “as if” at random— We’ve considered examples of randomized natural experiments(the Vietnam draft lottery, the Hajj pilgrimage to Mecca).Lecture Notes, Week 12 8/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDefinition of a natural experiment:An observational study (that is, a study without an experimentalmanipulation) in which assignment to treatment and control groupsis done at random – or “as if” at random— We’ve considered examples of randomized natural experiments(the Vietnam draft lottery, the Hajj pilgrimage to Mecca).— Let’s look at two examples in which assignment is posited to beas-if random.Lecture Notes, Week 12 8/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonLecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonAgainst predominant theories, John Snow hypothesized thatcholera was a water- or waste-born diseaseLecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonAgainst predominant theories, John Snow hypothesized thatcholera was a water- or waste-born diseaseA series of “causal-process observations” supported hishypothesisLecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonAgainst predominant theories, John Snow hypothesized thatcholera was a water- or waste-born diseaseA series of “causal-process observations” supported hishypothesisHis most powerful evidence came from a natural experiment:Lecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonAgainst predominant theories, John Snow hypothesized thatcholera was a water- or waste-born diseaseA series of “causal-process observations” supported hishypothesisHis most powerful evidence came from a natural experiment:In 1852, the Lambeth water company moved its intake pipeupstream on the Thames, to a purer water sourceLecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaJohn Snow’s natural experimentCholera epidemics in mid-nineteenth century LondonAgainst predominant theories, John Snow hypothesized thatcholera was a water- or waste-born diseaseA series of “causal-process observations” supported hishypothesisHis most powerful evidence came from a natural experiment:In 1852, the Lambeth water company moved its intake pipeupstream on the Thames, to a purer water sourceThe Southwark and Vauxhall company left its intake pipe inplace.Lecture Notes, Week 12 9/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaSnow in his own words:“The pipes of each Company go down all the streets...A fewhouses are supplied by one Company and a few by the other,according to the decision of the owner or occupier at that timewhen the Water Companies were in active competition. Inmany cases a single house has a supply different from that oneither side. Each company supplies both rich and poor, bothlarge houses and small; there is no difference either in thecondition or occupation of the persons receiving the water ofeither company...”“It is obvious no experiment could have been designed whichwould more thoroughly test the effect of water supply on theprogress of cholera than this.” – John Snow (1855: 74-75)Lecture Notes, Week 12 10/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaDeath rates from cholera, by water supply sourceNo. of Houses Cholera Deaths Death Rateper 10,000Southwark 40,046 1,263 315and VauxhallLambeth 26,107 98 37Rest of 256,423 1,422 59LondonSource: Snow (1855, Table IX, p. 86)Lecture Notes, Week 12 11/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaSnow’s study as a natural experimentSnow informally showed balance across treatment and controlgroups on pre-treatment covariates—a necessary, if notsufficient, condition for a natural experiment. (We’ll talk moreabout tests of design).Lecture Notes, Week 12 12/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaSnow’s study as a natural experimentSnow informally showed balance across treatment and controlgroups on pre-treatment covariates—a necessary, if notsufficient, condition for a natural experiment. (We’ll talk moreabout tests of design).A priori reasoning also suggests as-if random; for instance,subjects did not self-select into treatment and controlLecture Notes, Week 12 12/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaSnow’s study as a natural experimentSnow informally showed balance across treatment and controlgroups on pre-treatment covariates—a necessary, if notsufficient, condition for a natural experiment. (We’ll talk moreabout tests of design).A priori reasoning also suggests as-if random; for instance,subjects did not self-select into treatment and controlMore than three hundred thousand people“divided into two groups without their choice, and, in mostcases, without their knowledge; one group being supplied withwater containing the sewage of London, and, amongst it,whatever might have come from the cholera patients, the othergroup having water quite free from such impurity” (Snow 1855:75).Lecture Notes, Week 12 12/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaSnow’s study as a natural experimentSnow informally showed balance across treatment and controlgroups on pre-treatment covariates—a necessary, if notsufficient, condition for a natural experiment. (We’ll talk moreabout tests of design).A priori reasoning also suggests as-if random; for instance,subjects did not self-select into treatment and controlMore than three hundred thousand people“divided into two groups without their choice, and, in mostcases, without their knowledge; one group being supplied withwater containing the sewage of London, and, amongst it,whatever might have come from the cholera patients, the othergroup having water quite free from such impurity” (Snow 1855:75).The data analysis is very simple. Conviction comes from thestrength of the design and the size of the effect.Lecture Notes, Week 12 12/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaHow do property rights affect the poor?A well-known hypothesis: establishing property rights for thepoor would raise incomes, increase investment, and allowaccess to capital (De Soto)Lecture Notes, Week 12 13/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaHow do property rights affect the poor?A well-known hypothesis: establishing property rights for thepoor would raise incomes, increase investment, and allowaccess to capital (De Soto)Yet where property rights are extended, they may be extendedto all similarly-situated peopleLecture Notes, Week 12 13/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaHow do property rights affect the poor?A well-known hypothesis: establishing property rights for thepoor would raise incomes, increase investment, and allowaccess to capital (De Soto)Yet where property rights are extended, they may be extendedto all similarly-situated peopleOr, their assignment may reflect confounding characteristicsassociated with particular poor citizensLecture Notes, Week 12 13/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaHow do property rights affect the poor?A well-known hypothesis: establishing property rights for thepoor would raise incomes, increase investment, and allowaccess to capital (De Soto)Yet where property rights are extended, they may be extendedto all similarly-situated peopleOr, their assignment may reflect confounding characteristicsassociated with particular poor citizensLecture Notes, Week 12 13/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaHow do property rights affect the poor?A well-known hypothesis: establishing property rights for thepoor would raise incomes, increase investment, and allowaccess to capital (De Soto)Yet where property rights are extended, they may be extendedto all similarly-situated peopleOr, their assignment may reflect confounding characteristicsassociated with particular poor citizensGaliani and Schargrodsky (2006, 2007) claim to exploit anatural experiment in Argentina, in which assignment toproperty rights is as-if randomLecture Notes, Week 12 13/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaA natural experiment on property rightsIn 1981, squatters organized by the Catholic church occupiedan urban wasteland in the province of Buenos Aires, dividingthe land into similar parcelsLecture Notes, Week 12 14/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaA natural experiment on property rightsIn 1981, squatters organized by the Catholic church occupiedan urban wasteland in the province of Buenos Aires, dividingthe land into similar parcelsAfter the return to democracy, a 1984 law expropriated theland, with the intention of transferring title to the squattersLecture Notes, Week 12 14/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaA natural experiment on property rightsIn 1981, squatters organized by the Catholic church occupiedan urban wasteland in the province of Buenos Aires, dividingthe land into similar parcelsAfter the return to democracy, a 1984 law expropriated theland, with the intention of transferring title to the squattersSome of the original owners challenged the expropriation incourt, leading to long delays, while other titles were cededand transferred to squattersLecture Notes, Week 12 14/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaA natural experiment on property rightsIn 1981, squatters organized by the Catholic church occupiedan urban wasteland in the province of Buenos Aires, dividingthe land into similar parcelsAfter the return to democracy, a 1984 law expropriated theland, with the intention of transferring title to the squattersSome of the original owners challenged the expropriation incourt, leading to long delays, while other titles were cededand transferred to squattersThe legal action created a treatment group—squatters towhom titles were ceded—and a control group—squatterswhose titles were not cededLecture Notes, Week 12 14/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaA natural experiment on property rightsIn 1981, squatters organized by the Catholic church occupiedan urban wasteland in the province of Buenos Aires, dividingthe land into similar parcelsAfter the return to democracy, a 1984 law expropriated theland, with the intention of transferring title to the squattersSome of the original owners challenged the expropriation incourt, leading to long delays, while other titles were cededand transferred to squattersThe legal action created a treatment group—squatters towhom titles were ceded—and a control group—squatterswhose titles were not cededThe authors administered surveys to both groupsLecture Notes, Week 12 14/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Lecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Titled and untitled parcels sit side-by-side (reminiscent ofSnow)Lecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Titled and untitled parcels sit side-by-side (reminiscent ofSnow)Pre-treatment characteristics of squatters (age, sex, etc.) donot predict whether they received titleLecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Titled and untitled parcels sit side-by-side (reminiscent ofSnow)Pre-treatment characteristics of squatters (age, sex, etc.) donot predict whether they received titlePre-treatment characteristics of the parcels (such as distancefrom polluted creeks) are similar in both groupsLecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Titled and untitled parcels sit side-by-side (reminiscent ofSnow)Pre-treatment characteristics of squatters (age, sex, etc.) donot predict whether they received titlePre-treatment characteristics of the parcels (such as distancefrom polluted creeks) are similar in both groupsThe compensation offered by the government (in square meterterms) was very similar across parcelsLecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaWas the ceding of titles as-if random?The pre-treatment equivalence of treated and untreated units:Titled and untitled parcels sit side-by-side (reminiscent ofSnow)Pre-treatment characteristics of squatters (age, sex, etc.) donot predict whether they received titlePre-treatment characteristics of the parcels (such as distancefrom polluted creeks) are similar in both groupsThe compensation offered by the government (in square meterterms) was very similar across parcelsAnother important source of validation of the naturalexperiment is qualitative evidence on the process by whichthe squatting took placeLecture Notes, Week 12 15/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaThe effects of property rightsGaliani and Schargrodsky (2006, 2007) find significant effectson housing investment, household structure, and educationalattainment of children (but not on access to credit markets)Lecture Notes, Week 12 16/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaThe effects of property rightsGaliani and Schargrodsky (2006, 2007) find significant effectson housing investment, household structure, and educationalattainment of children (but not on access to credit markets)In Di Tella et al. (2007), they find a positive effect of propertyright on beliefs in individual efficacy (!)Lecture Notes, Week 12 16/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaThe effects of property rightsGaliani and Schargrodsky (2006, 2007) find significant effectson housing investment, household structure, and educationalattainment of children (but not on access to credit markets)In Di Tella et al. (2007), they find a positive effect of propertyright on beliefs in individual efficacy (!)The research design recalls Snow’s natural experiment oncholera...Lecture Notes, Week 12 16/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaThe effects of property rightsGaliani and Schargrodsky (2006, 2007) find significant effectson housing investment, household structure, and educationalattainment of children (but not on access to credit markets)In Di Tella et al. (2007), they find a positive effect of propertyright on beliefs in individual efficacy (!)The research design recalls Snow’s natural experiment oncholera...Here, too, the analysis can be quite simple: adifference-of-means test may be just the right tool.Lecture Notes, Week 12 16/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaEffects of property rightsThe  Effects  of  Land  Titles  on  Children’s  Health  Source:  Galiani  and  Schargrodsky  (2004).    No;ce  that  this  is  inten;on-­‐to-­‐treat  analysis.  In  the  first  two  rows,  data  for  children  ages  0-­‐11  are  shown;  in  the  third  row,  data  for  teenage  girls  aged  14-­‐17  are  shown.    The  number  of  observa;ons  is  in  brackets.  Lecture Notes, Week 12 17/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaStrengths and limitations of natural experimentsThe examples show the potential strengths of naturalexperiments with as-if random—but they also suggest thelimitations.Lecture Notes, Week 12 18/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaStrengths and limitations of natural experimentsThe examples show the potential strengths of naturalexperiments with as-if random—but they also suggest thelimitations.Validating the claim of as-if random is very far fromstraightforward. We’ll look at tests of design later.Lecture Notes, Week 12 18/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaStrengths and limitations of natural experimentsThe examples show the potential strengths of naturalexperiments with as-if random—but they also suggest thelimitations.Validating the claim of as-if random is very far fromstraightforward. We’ll look at tests of design later.Other questions come up: what are the right causal andstatistical models? What is the theoretical or substantiverelevance of the natural-experimental intervention?Lecture Notes, Week 12 18/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaStrengths and limitations of natural experimentsThe examples show the potential strengths of naturalexperiments with as-if random—but they also suggest thelimitations.Validating the claim of as-if random is very far fromstraightforward. We’ll look at tests of design later.Other questions come up: what are the right causal andstatistical models? What is the theoretical or substantiverelevance of the natural-experimental intervention?We’ll return to these issues later.Lecture Notes, Week 12 18/ 55
  • Standard natural experimentsRD designsSnow on choleraLand titles in ArgentinaStrengths and limitations of natural experimentsThe examples show the potential strengths of naturalexperiments with as-if random—but they also suggest thelimitations.Validating the claim of as-if random is very far fromstraightforward. We’ll look at tests of design later.Other questions come up: what are the right causal andstatistical models? What is the theoretical or substantiverelevance of the natural-experimental intervention?We’ll return to these issues later.It’s useful first to consider another type of natural experimentwith as-if random assignment: the regression-discontinuity(RD) design.Lecture Notes, Week 12 18/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Lecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Lecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Statistics: some work by Rubin (1977) and SpiegelmanLecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Statistics: some work by Rubin (1977) and SpiegelmanEconomics: early discussion by Goldberger (1972) but recentexplosion of applications (Angrist and Lavy)Lecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Statistics: some work by Rubin (1977) and SpiegelmanEconomics: early discussion by Goldberger (1972) but recentexplosion of applications (Angrist and Lavy)Political science: growing number of recent applicationsLecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Statistics: some work by Rubin (1977) and SpiegelmanEconomics: early discussion by Goldberger (1972) but recentexplosion of applications (Angrist and Lavy)Political science: growing number of recent applicationsThe scope of potential applications is quite broadLecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHistory of RDDBorn in psychology and education (Thistlewaite and Campbell1960)Sociology: criminal justice applications (Berk)Statistics: some work by Rubin (1977) and SpiegelmanEconomics: early discussion by Goldberger (1972) but recentexplosion of applications (Angrist and Lavy)Political science: growing number of recent applicationsThe scope of potential applications is quite broadFor more history, see Campbell’s introduction to Trochim(1984), or Cook (2007) in Journal of Econometrics specialissueLecture Notes, Week 12 19/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsAn RD design (Thistlewaite and Campbell 1960)Figure  4.2:  A  Regression-­‐Discon4nuity  Design  The  figure  plots  an  outcome  variable  (such  as  measured  interest  in  an  intellectual  career)  against  student  scores  on  a  qualifying  exam.    Students  who  received  a  score  of  11  and  above  received  public  recogni@on  of  their  scholas@c  achievement,  in  the  form  of  Cer@ficates  of  Merit,  while  those  below  the  key  threshold  merely  receive  commenda@ons  (see  Thistlewaite  and  Campbell  1960).    The  large  diamonds  mark  the  average  outcome  within  each  unit  bin  of  test  scores  (e.g.,  10  to  11,  11  to  12,  and  so  forth).      12  14  16  18  20  22  24  26  28  30  32  3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18  Outcome  Variable  Student  Test  Scores  (Arbitrary  Units)  Lecture Notes, Week 12 20/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThis is sharp RD0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1  3   5   7   9   11   13   15   17  Probability  of  treatment  receipt  Student  Test  Scores  (Arbitrary  Units)  Lecture Notes, Week 12 21/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsObserved outcomes as a function of pre-testFigure  2.1:  Examples  of  Regression-­‐Discon:nui:es  0  5  10  15  20  25  30  35  3   5   7   9   11   13   15   17  Percentage  Receiving  Scholarships  Student  Test  Scores  (Arbitrary  Units)  Series  A  Series  B  Series  C  Lecture Notes, Week 12 22/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSome sources of RD designsSource of RD Design Units TreatmentsEntrance Exams Students, Others Public recognition of scholasticachievementPopulation Thresholds(Municipalpopulation)Municipalities,CitizensVoting technologies; Federal funds;Conditional cash transfers; Electoralrules; Politicians’ salariesOther Size-BasedThresholds: Number ofVoters, SchoolEnrollments, Firm SizesVoters, Students,FirmsVoting by mail, Class size, Anti-biaslawsPoverty rankings,Criminality rankingsMunicipalities,Citizens, PrisonersAnti-poverty programsHigh-security incarcerationAge-based Thresholds(Voting Age, Quarter-of-Birth)Voters, Students Past voting, Years of educationClose elections Candidates/Parties; FirmsIncumbency; Control of the media;Returns to campaign donations  Lecture Notes, Week 12 23/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsElectronic voting in Brazil (Danny Hidalgo 2011)Module 15, Session 1 Module 15, Session 2 Module 15, Session 3 Module 17, Session 3Treatment Status1996 Electoratecount02004006008000 20,000 40,000 60,000 80,000 100,000YearElectronicPaperLecture Notes, Week 12 24/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDigital Democratization (Hidalgo 2011)Module 15, Session 1 Module 15, Session 2 Module 15, Session 3 Module 17, Session 3Results: Invalid VotesElectorate (1996)1998BlankandInvalidVotes(%ofTotalVotes)10203040506020000 40000 60000 80000Lecture Notes, Week 12 25/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Near-winners versus near-losers of elections (Lee 2008)Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Near-winners versus near-losers of elections (Lee 2008)Rules allocating features of elections as a function of municipalsize (see Dunning 2012, Ch. 3 for a survey)Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Near-winners versus near-losers of elections (Lee 2008)Rules allocating features of elections as a function of municipalsize (see Dunning 2012, Ch. 3 for a survey)In each case, the assumption of as-if random near thethreshold may be more or less plausible—so validating thisclaim, at least partially, is critical.Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Near-winners versus near-losers of elections (Lee 2008)Rules allocating features of elections as a function of municipalsize (see Dunning 2012, Ch. 3 for a survey)In each case, the assumption of as-if random near thethreshold may be more or less plausible—so validating thisclaim, at least partially, is critical.In some RD designs, the threshold merely determinedeligibility for treatment:Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsOther examplesThe scope of recent applications is quite broad:Near-winners versus near-losers of elections (Lee 2008)Rules allocating features of elections as a function of municipalsize (see Dunning 2012, Ch. 3 for a survey)In each case, the assumption of as-if random near thethreshold may be more or less plausible—so validating thisclaim, at least partially, is critical.In some RD designs, the threshold merely determinedeligibility for treatment:E.g., a poverty index for individuals or municipalities makesunits below a threshold eligible for an intervention.Lecture Notes, Week 12 26/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1  3   5   7   9   11   13   15   17  Probability  of  treatment  receipt  Student  Test  Scores  (Arbitrary  Units)  Lecture Notes, Week 12 27/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.Lecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.This can be seen as a classic noncompliance problem: someunits do not “comply” with the treatment to which they areassigned by location in relation to the thresholdLecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.This can be seen as a classic noncompliance problem: someunits do not “comply” with the treatment to which they areassigned by location in relation to the thresholdThen, treatment assignment serves an an instrumentalvariable for treatment receiptLecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.This can be seen as a classic noncompliance problem: someunits do not “comply” with the treatment to which they areassigned by location in relation to the thresholdThen, treatment assignment serves an an instrumentalvariable for treatment receiptAs before, location in relation to the threshold is as-if random(treatment receipt is not)Lecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.This can be seen as a classic noncompliance problem: someunits do not “comply” with the treatment to which they areassigned by location in relation to the thresholdThen, treatment assignment serves an an instrumentalvariable for treatment receiptAs before, location in relation to the threshold is as-if random(treatment receipt is not)There is an implicit exclusion restriction: treatment assignmentaffects outcomes only through its effect on treatment receipt.Lecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD is IVNote that here, treatment assignment (location of units inrelation to the threshold) is correlated with treatment receipt atsomething less than 1.This can be seen as a classic noncompliance problem: someunits do not “comply” with the treatment to which they areassigned by location in relation to the thresholdThen, treatment assignment serves an an instrumentalvariable for treatment receiptAs before, location in relation to the threshold is as-if random(treatment receipt is not)There is an implicit exclusion restriction: treatment assignmentaffects outcomes only through its effect on treatment receipt.IV analysis estimates the effect of treatment on Compliers.Lecture Notes, Week 12 28/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsFuzzy RD: causal model with noncompliance€Ti€Ti ……€Ti€Ti€Ci€Ci€Ti€Ci€Ti€Ci€Ci€Ci…€Ti€TiThe assigned-to-treatment groupThe assigned-to-control groupTi Ti Ti CiCi CiCi CiTi Ti Ti CiCi CiCompliersor Always-TreatsNever-TreatsCi CiCompliersor Never-TreatsAlways-TreatsTi Ti……………………Lecture Notes, Week 12 29/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe IV estimatorUnder this model, the usual IV estimator consistentlyestimates the average causal effect for Compliers:YT− YCXT − XC, (1)where YTis the mean outcome in the assigned-to-treatmentgroup, and XTis the proportion of this group that receivestreatment; YCis the mean outcome in the assigned-to-controlgroup, and XCis the proportion of this group that receivestreatment.Lecture Notes, Week 12 30/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe IV estimatorUnder this model, the usual IV estimator consistentlyestimates the average causal effect for Compliers:YT− YCXT − XC, (1)where YTis the mean outcome in the assigned-to-treatmentgroup, and XTis the proportion of this group that receivestreatment; YCis the mean outcome in the assigned-to-controlgroup, and XCis the proportion of this group that receivestreatment.Our usual assumptions must hold (what are they)?Lecture Notes, Week 12 30/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe IV estimatorUnder this model, the usual IV estimator consistentlyestimates the average causal effect for Compliers:YT− YCXT − XC, (1)where YTis the mean outcome in the assigned-to-treatmentgroup, and XTis the proportion of this group that receivestreatment; YCis the mean outcome in the assigned-to-controlgroup, and XCis the proportion of this group that receivestreatment.Our usual assumptions must hold (what are they)?Why is this estimator consistent but not unbiased?Lecture Notes, Week 12 30/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe IV estimatorUnder this model, the usual IV estimator consistentlyestimates the average causal effect for Compliers:YT− YCXT − XC, (1)where YTis the mean outcome in the assigned-to-treatmentgroup, and XTis the proportion of this group that receivestreatment; YCis the mean outcome in the assigned-to-controlgroup, and XCis the proportion of this group that receivestreatment.Our usual assumptions must hold (what are they)?Why is this estimator consistent but not unbiased?N.B. We can always do intention-to-treat analysis.Lecture Notes, Week 12 30/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRD as a natural experimentCampbell discussed RDD as a “tie-breaking experiment”Lecture Notes, Week 12 31/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRD as a natural experimentCampbell discussed RDD as a “tie-breaking experiment”There is an element of “chance” in assignment to one side oranother of the cut-off value (e.g., test scores, population, orpoverty)Lecture Notes, Week 12 31/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRD as a natural experimentCampbell discussed RDD as a “tie-breaking experiment”There is an element of “chance” in assignment to one side oranother of the cut-off value (e.g., test scores, population, orpoverty)Near the key threshold, assignment may be as good asrandomLecture Notes, Week 12 31/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRD as a natural experimentCampbell discussed RDD as a “tie-breaking experiment”There is an element of “chance” in assignment to one side oranother of the cut-off value (e.g., test scores, population, orpoverty)Near the key threshold, assignment may be as good asrandomThis implies statistical independence of potential outcomesand treatment assignmentLecture Notes, Week 12 31/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRDD as a natural experiment: A model………€Yi(1)Treatment group Control group€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)Study groupLecture Notes, Week 12 32/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRDD as a natural experiment: A model………€Yi(1)Treatment group Control group€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)Study group€1N[Yi (1) −i=1N∑ Yi(0)]The estimand:Lecture Notes, Week 12 33/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsRDD as a natural experiment: A model………€Yi(1)Treatment group Control group€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(1)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)€Yi(0)Study group€1N[Yi (1) −i=1N∑ Yi(0)]The estimand:An unbiased estimator:€1m[Yi |Ti =1i=1m∑ ] −1N − m[Yi |Ti = 0i=m+1N∑ ]where is an indicator for treatment assignment. Under thismodel, a random subset of size m<N units is assigned to treatment.The units assigned to the treatment group are indexed from 1 to m.€TiLecture Notes, Week 12 34/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedLecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedAdvantages: the difference-of-means estimator is simple andtransparentLecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedAdvantages: the difference-of-means estimator is simple andtransparentNote that the “regression-discontinuity” design does not implythe use of (multivariate) regression! (“Cut-off design” might bebetter, but we’re probably stuck with the label).Lecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedAdvantages: the difference-of-means estimator is simple andtransparentNote that the “regression-discontinuity” design does not implythe use of (multivariate) regression! (“Cut-off design” might bebetter, but we’re probably stuck with the label).A major difficulty: definition of the study group for which thisbox model holdsLecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedAdvantages: the difference-of-means estimator is simple andtransparentNote that the “regression-discontinuity” design does not implythe use of (multivariate) regression! (“Cut-off design” might bebetter, but we’re probably stuck with the label).A major difficulty: definition of the study group for which thisbox model holdsDefining the study group in RD amounts to bandwidthselection: how “near” the key threshold units must beLecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference-of-means estimatorUnder this interpretation of RDD, the difference-of-meansestimator is unbiasedAdvantages: the difference-of-means estimator is simple andtransparentNote that the “regression-discontinuity” design does not implythe use of (multivariate) regression! (“Cut-off design” might bebetter, but we’re probably stuck with the label).A major difficulty: definition of the study group for which thisbox model holdsDefining the study group in RD amounts to bandwidthselection: how “near” the key threshold units must beIs assignment to treatment really as good as random for thoseunits—as implied by this model?Lecture Notes, Week 12 35/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe RD estimand as a limit quantittyEven without as-if randomization, potential outcomes may becontinuous at the key thresholdLecture Notes, Week 12 36/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsPotential and Observed Outcomes in RD designsFigure  4.3:  Poten0al  and  Observed  Outcomes    in  a  Regression-­‐Discon0nuity  Design  The  figure  plots  the  average  poten1al  outcomes  for  all  test-­‐takers,  at  each  unit  bin  of  student  test  scores.  Squares  indicate  the  poten1al  outcomes  under  treatment,  and  circles  indicate  poten1al  outcomes  under  control.    Darkened  markers  indicate  outcomes  that  are  observed.    Only  students  who  score  higher  than  11  receive  the  treatment:  public  recogni1on  in  the  form  of  Cer1ficates  of  Merit.      12  14  16  18  20  22  24  26  28  30  32  3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18  Average  Poten0al  Outcome  Student  Test  Scores  (Arbitrary  Units)  C_i  (Observed)   T_i  (Unobserved)   T_i  (Observed)   C_i  (Unobserved)  Lecture Notes, Week 12 37/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe RD estimand as a limitHere, the two conditional expectations of the potentialoutcomes are continuous at x=11, but the conditionalexpectation of the observed outcome jumps at this point.Lecture Notes, Week 12 38/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe RD estimand as a limitHere, the two conditional expectations of the potentialoutcomes are continuous at x=11, but the conditionalexpectation of the observed outcome jumps at this point.Here the average causal effect of the treatment at thediscontinuity point is defined byE[Yi(1) − Yi(0)|Xi = c], (2)where X is a pre-treatment covariate and c is the cut-off valueof X that determines treatment assignment.Lecture Notes, Week 12 38/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsThe RD estimand as a limitHere, the two conditional expectations of the potentialoutcomes are continuous at x=11, but the conditionalexpectation of the observed outcome jumps at this point.Here the average causal effect of the treatment at thediscontinuity point is defined byE[Yi(1) − Yi(0)|Xi = c], (2)where X is a pre-treatment covariate and c is the cut-off valueof X that determines treatment assignment.Then, the causal effect is interpreted as a limit:limx⇓cE[Yi(1)|Xi = x] − limx⇑cE[Yi(0)|Xi = x] (3)Lecture Notes, Week 12 38/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsEstimating limitsDefining the causal effect as the limit of a regression functionleads to the definition of bias in the difference-of-meansestimator: the limit must be approximated by the derivative ofobserved outcomes at a boundary point, i.e., the point ofdiscontinuity.Lecture Notes, Week 12 39/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsEstimating limitsDefining the causal effect as the limit of a regression functionleads to the definition of bias in the difference-of-meansestimator: the limit must be approximated by the derivative ofobserved outcomes at a boundary point, i.e., the point ofdiscontinuity.If this derivative is non-zero, there is asymptotic bias in thedifference of means estimator; the size of the bias is a linearfunction of the size of the bandwidth h that is chosen aroundthe RD threshold (see Dunning, Appendix 4.2)Lecture Notes, Week 12 39/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsEstimating limitsDefining the causal effect as the limit of a regression functionleads to the definition of bias in the difference-of-meansestimator: the limit must be approximated by the derivative ofobserved outcomes at a boundary point, i.e., the point ofdiscontinuity.If this derivative is non-zero, there is asymptotic bias in thedifference of means estimator; the size of the bias is a linearfunction of the size of the bandwidth h that is chosen aroundthe RD threshold (see Dunning, Appendix 4.2)Thus, consideration of the shape of potential outcomefunctions on either side of the key threshold is the centralissue in evaluating the argument that the difference-of-meansestimator is biased in RD designs.Lecture Notes, Week 12 39/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsShape of the potential outcomes regression functionAccording to Imbens and Lemieuxs (2007),Lecture Notes, Week 12 40/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsShape of the potential outcomes regression functionAccording to Imbens and Lemieuxs (2007),“We typically do expect the (potential outcomes) regressionfunction to have a non-zero derivative, even in cases wherethe treatment has no effect. In many applications the eligibilitycriterion is based on a covariate that does have somecorrelation with the outcome, so that, for example, those withpoorest prospects in the absence of the program are in theeligible group.”Lecture Notes, Week 12 40/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsBias in the difference of means?Yet, why should units eligible for a program on the basis ofhaving a qualification score that is, say, just above the RDthreshold respond systematically differently to treatment thanthose just below the threshold?Lecture Notes, Week 12 41/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsBias in the difference of means?Yet, why should units eligible for a program on the basis ofhaving a qualification score that is, say, just above the RDthreshold respond systematically differently to treatment thanthose just below the threshold?If the conditional expectation of potential outcomes on eitherside of the discontinuity is much different, for units included inthe study group, we would have to say that the naturalexperiment has failed—for it has not in fact generated as-ifrandom assignment to treatment conditions.Lecture Notes, Week 12 41/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsBias in the difference of means?Yet, why should units eligible for a program on the basis ofhaving a qualification score that is, say, just above the RDthreshold respond systematically differently to treatment thanthose just below the threshold?If the conditional expectation of potential outcomes on eitherside of the discontinuity is much different, for units included inthe study group, we would have to say that the naturalexperiment has failed—for it has not in fact generated as-ifrandom assignment to treatment conditions.In this case, the assigned-to-control group is not a validcounterfactual for the assigned-to-treatment group.Lecture Notes, Week 12 41/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsBias in the difference of means?If, on the other hand, the natural experiment is successful—inthat it generates groups just on either side of the RD thresholdthat are valid counterfactuals for one another—then the graphof the conditional expectations of the potential outcomesshould be approximately flat in the neighborhood of thethresholdLecture Notes, Week 12 42/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsPotential and Observed Outcomes in RD designsFigure  4.3:  Poten0al  and  Observed  Outcomes    in  a  Regression-­‐Discon0nuity  Design  The  figure  plots  the  average  poten1al  outcomes  for  all  test-­‐takers,  at  each  unit  bin  of  student  test  scores.  Squares  indicate  the  poten1al  outcomes  under  treatment,  and  circles  indicate  poten1al  outcomes  under  control.    Darkened  markers  indicate  outcomes  that  are  observed.    Only  students  who  score  higher  than  11  receive  the  treatment:  public  recogni1on  in  the  form  of  Cer1ficates  of  Merit.      12  14  16  18  20  22  24  26  28  30  32  3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18  Average  Poten0al  Outcome  Student  Test  Scores  (Arbitrary  Units)  C_i  (Observed)   T_i  (Unobserved)   T_i  (Observed)   C_i  (Unobserved)  Lecture Notes, Week 12 43/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsAlternatives for estimating limitsLocal-linear regression: fit a linear regression within abandwidth h on either side of the threshold, and estimate thecausal effect as the difference of intercepts in these tworegressions.Lecture Notes, Week 12 44/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsAlternatives for estimating limitsLocal-linear regression: fit a linear regression within abandwidth h on either side of the threshold, and estimate thecausal effect as the difference of intercepts in these tworegressions.Weighted kernel regression, in which weights decrease as thedistance to the cutoff point increases.Lecture Notes, Week 12 44/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsAlternatives for estimating limitsLocal-linear regression: fit a linear regression within abandwidth h on either side of the threshold, and estimate thecausal effect as the difference of intercepts in these tworegressions.Weighted kernel regression, in which weights decrease as thedistance to the cutoff point increases.Global polynomial regressions–typically using data furtherfrom the threshold, and fitting higher-order polynomials of theassignment covariate (and sometimes other covariates).Lecture Notes, Week 12 44/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsAlternatives for estimating limitsLocal-linear regression: fit a linear regression within abandwidth h on either side of the threshold, and estimate thecausal effect as the difference of intercepts in these tworegressions.Weighted kernel regression, in which weights decrease as thedistance to the cutoff point increases.Global polynomial regressions–typically using data furtherfrom the threshold, and fitting higher-order polynomials of theassignment covariate (and sometimes other covariates).Here, the regression draws power from data points far from thethreshold. The model-dependence is not really in the spirit ofdesign-based inference.Lecture Notes, Week 12 44/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference of means vs. the alternativesSo what should one do? Probably, present the simpledifference of means, perhaps with a choice of bandwidths(we’ll discuss this in a bit), along with any alternativeestimators you like.Lecture Notes, Week 12 45/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference of means vs. the alternativesSo what should one do? Probably, present the simpledifference of means, perhaps with a choice of bandwidths(we’ll discuss this in a bit), along with any alternativeestimators you like.If using different weights from a weighted kernel, local-linearregression, or global polynomial does make a difference, itlikely suggests that the results are highly sensitive to thechoice of bandwidth—then caveat emptor.Lecture Notes, Week 12 45/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDifference of means vs. the alternativesSo what should one do? Probably, present the simpledifference of means, perhaps with a choice of bandwidths(we’ll discuss this in a bit), along with any alternativeestimators you like.If using different weights from a weighted kernel, local-linearregression, or global polynomial does make a difference, itlikely suggests that the results are highly sensitive to thechoice of bandwidth—then caveat emptor.E.g., Imbens and Lemieux say, “So the only case where moresophisticated kernels may make a difference is when theestimates are not very credible anyway because of too muchsensitivity to the choice of bandwidth. From a practical point ofview, one may just focus on the simple difference-of-means,but verify the robustness of the results to different choices ofbandwidth.”Lecture Notes, Week 12 45/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsGraphical analysisAlso note that whatever the formal estimand and estimator,graphical analysis is helpful and crucial for assessing theplausibility of a causal effect of treatment.Lecture Notes, Week 12 46/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsGraphical analysisAlso note that whatever the formal estimand and estimator,graphical analysis is helpful and crucial for assessing theplausibility of a causal effect of treatment.As Imbens and Lemiuex put it, “The question is whetheraround the threshold . . . there is any evidence of a jump in theconditional mean of the outcome. The formal statisticalanalyses . . . are essentially just sophisticated versions of this,and if the basic plot does not show any evidence of adiscontinuity, there is relatively little chance that the moresophisticated analyses will lead to robust and credibleestimates with statistically and substantially significantmagnitudes.”Lecture Notes, Week 12 46/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsGraphical analysisAlso note that whatever the formal estimand and estimator,graphical analysis is helpful and crucial for assessing theplausibility of a causal effect of treatment.As Imbens and Lemiuex put it, “The question is whetheraround the threshold . . . there is any evidence of a jump in theconditional mean of the outcome. The formal statisticalanalyses . . . are essentially just sophisticated versions of this,and if the basic plot does not show any evidence of adiscontinuity, there is relatively little chance that the moresophisticated analyses will lead to robust and credibleestimates with statistically and substantially significantmagnitudes.”(In a bit, we’ll also discuss diagnostic graphical analysis, e.g.checks for covariate balance).Lecture Notes, Week 12 46/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDigital Democratization (Hidalgo 2011)Module 15, Session 1 Module 15, Session 2 Module 15, Session 3 Module 17, Session 3Results: Invalid VotesElectorate (1996)1998BlankandInvalidVotes(%ofTotalVotes)10203040506020000 40000 60000 80000Lecture Notes, Week 12 47/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceLecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceIf we don’t have this, we don’t have a natural experiment, andthe rationale for RD is weakerLecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceIf we don’t have this, we don’t have a natural experiment, andthe rationale for RD is weakerIf we do have this, then the simple of difference-of-meansestimator may be the right tool.Lecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceIf we don’t have this, we don’t have a natural experiment, andthe rationale for RD is weakerIf we do have this, then the simple of difference-of-meansestimator may be the right tool.The difference of means is unbiased under the Neyman urnmodel and has the advantage of simplicity and transparency.Lecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceIf we don’t have this, we don’t have a natural experiment, andthe rationale for RD is weakerIf we do have this, then the simple of difference-of-meansestimator may be the right tool.The difference of means is unbiased under the Neyman urnmodel and has the advantage of simplicity and transparency.For standard errors, the “conservative variance estimator” is agood approach. With small natural experiment, we may userandomization inference for hypothesis testing.Lecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSummary: estimands and estimators in RD analysisWithout credible as-if random assignment—for at least someunits very near the threshold—RDD loses its charm as adesign-based approach to causal inferenceIf we don’t have this, we don’t have a natural experiment, andthe rationale for RD is weakerIf we do have this, then the simple of difference-of-meansestimator may be the right tool.The difference of means is unbiased under the Neyman urnmodel and has the advantage of simplicity and transparency.For standard errors, the “conservative variance estimator” is agood approach. With small natural experiment, we may userandomization inference for hypothesis testing.This leaves open the crucial question of selecting a bandwidthand validating as-if randomLecture Notes, Week 12 48/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.Lecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.The basic problem is easy to see:Lecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.The basic problem is easy to see:Go too far from the threshold, and we risk bias: treatmentassignment is no longer as-if random, and assignment may beassociated with potential outcomes. The bias is fromunobservables.Lecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.The basic problem is easy to see:Go too far from the threshold, and we risk bias: treatmentassignment is no longer as-if random, and assignment may beassociated with potential outcomes. The bias is fromunobservables.However, data may be sparse near the threshold, so treatmenteffect estimators may be imprecise if the bandwidth is verynarrow.Lecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.The basic problem is easy to see:Go too far from the threshold, and we risk bias: treatmentassignment is no longer as-if random, and assignment may beassociated with potential outcomes. The bias is fromunobservables.However, data may be sparse near the threshold, so treatmenteffect estimators may be imprecise if the bandwidth is verynarrow.Thus, there is typically a tradeoff between bias and variancereduction.Lecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsChoosing bandwidthsThis is probably the most difficult and unsatisfactory aspect ofRD analysis.The basic problem is easy to see:Go too far from the threshold, and we risk bias: treatmentassignment is no longer as-if random, and assignment may beassociated with potential outcomes. The bias is fromunobservables.However, data may be sparse near the threshold, so treatmenteffect estimators may be imprecise if the bandwidth is verynarrow.Thus, there is typically a tradeoff between bias and variancereduction.Of course, if data are sparse near the threshold, the RDdesign may not be very usefulLecture Notes, Week 12 49/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsValidating bandwidth choiceSome authors have proposed cross-validation techniques andother algorithms for selecting “optimal” bandwidths.Lecture Notes, Week 12 50/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsValidating bandwidth choiceSome authors have proposed cross-validation techniques andother algorithms for selecting “optimal” bandwidths.Yet, a key question remains: does the chosen bandwidthproduce plausible as-if random assignment?Lecture Notes, Week 12 50/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsValidating bandwidth choiceSome authors have proposed cross-validation techniques andother algorithms for selecting “optimal” bandwidths.Yet, a key question remains: does the chosen bandwidthproduce plausible as-if random assignment?Let’s discuss some techniques for validating as-if randomempirically (at least partially).Lecture Notes, Week 12 50/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsValidating bandwidth choiceSome authors have proposed cross-validation techniques andother algorithms for selecting “optimal” bandwidths.Yet, a key question remains: does the chosen bandwidthproduce plausible as-if random assignment?Let’s discuss some techniques for validating as-if randomempirically (at least partially).Ultimately, this is a substantive question and can’t be fullysolved through algorithms.Lecture Notes, Week 12 50/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Without true randomization, always hard to be sure. As inother natural experiments, the onus is on the researcher here.Failure to reject the null hypothesis of balance is not the sameas proving the null.Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Without true randomization, always hard to be sure. As inother natural experiments, the onus is on the researcher here.Failure to reject the null hypothesis of balance is not the sameas proving the null.Still, formal tests of design are essential, e.g:Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Without true randomization, always hard to be sure. As inother natural experiments, the onus is on the researcher here.Failure to reject the null hypothesis of balance is not the sameas proving the null.Still, formal tests of design are essential, e.g:t-tests for the equality of means on pre-treatment covariates,across treatment and control groups;Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Without true randomization, always hard to be sure. As inother natural experiments, the onus is on the researcher here.Failure to reject the null hypothesis of balance is not the sameas proving the null.Still, formal tests of design are essential, e.g:t-tests for the equality of means on pre-treatment covariates,across treatment and control groups;F-tests or log-likelihood tests, in which a regression ofassignment on all pre-treatment covariates is compared toregression of assignment on intercept alone.Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsDiagnostics: Covariate balanceIs assignment at the threshold as good as random?Without true randomization, always hard to be sure. As inother natural experiments, the onus is on the researcher here.Failure to reject the null hypothesis of balance is not the sameas proving the null.Still, formal tests of design are essential, e.g:t-tests for the equality of means on pre-treatment covariates,across treatment and control groups;F-tests or log-likelihood tests, in which a regression ofassignment on all pre-treatment covariates is compared toregression of assignment on intercept alone.Graphical plots of pre-treatment covariates against theassignment covariate are very helpful here as well.Lecture Notes, Week 12 51/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsHidalgo’s (2011) study of BrazilModule 15, Session 1 Module 15, Session 2 Module 15, Session 3 Module 17, Session 3Balance1996 Electorate1996BlankandInvalidVotes(%)0510152025300 20000 40000 60000 80000Lecture Notes, Week 12 52/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSorting/Conditional Density TestsIt is a good idea to plot the number of observations on eachside of the threshold against the assignment covariate/forcingvariable.Lecture Notes, Week 12 53/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSorting/Conditional Density TestsIt is a good idea to plot the number of observations on eachside of the threshold against the assignment covariate/forcingvariable.Any discontinuity would raise the question of whether the valueof this covariate was manipulated, or whether sorting aroundthe threshold occurred.Lecture Notes, Week 12 53/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSorting/Conditional Density TestsIt is a good idea to plot the number of observations on eachside of the threshold against the assignment covariate/forcingvariable.Any discontinuity would raise the question of whether the valueof this covariate was manipulated, or whether sorting aroundthe threshold occurred.If the assignment variable is number of employees, and ananti-discrimination law kicks in for firms with 15 employees, wemay find lots of firms with 14 employees and few with 15 or 16.Lecture Notes, Week 12 53/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSorting/Conditional Density TestsIt is a good idea to plot the number of observations on eachside of the threshold against the assignment covariate/forcingvariable.Any discontinuity would raise the question of whether the valueof this covariate was manipulated, or whether sorting aroundthe threshold occurred.If the assignment variable is number of employees, and ananti-discrimination law kicks in for firms with 15 employees, wemay find lots of firms with 14 employees and few with 15 or 16.Such a finding would undermine the claim of as-if randomassignment, and would likely imply dependence betweentreatment assignment and potential outcomes.Lecture Notes, Week 12 53/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSorting/Conditional Density TestsIt is a good idea to plot the number of observations on eachside of the threshold against the assignment covariate/forcingvariable.Any discontinuity would raise the question of whether the valueof this covariate was manipulated, or whether sorting aroundthe threshold occurred.If the assignment variable is number of employees, and ananti-discrimination law kicks in for firms with 15 employees, wemay find lots of firms with 14 employees and few with 15 or 16.Such a finding would undermine the claim of as-if randomassignment, and would likely imply dependence betweentreatment assignment and potential outcomes.This would lead to bias in the estimators of the average causaleffect of treatment for the study group.This idea has a long history in the RD literature (see the Cook2007 review); McCrary (2007) proposes a formal test.Lecture Notes, Week 12 53/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnostics“Placebo” TestsTests for treatment effects where none should exist aresometimes called placebo tests.Lecture Notes, Week 12 54/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnostics“Placebo” TestsTests for treatment effects where none should exist aresometimes called placebo tests.In the RD context, this involves estimating jumps in outcomeat points away from the discontinuity, where treatmentassignment status does not change and thus there should beno jumps in outcomes.Lecture Notes, Week 12 54/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnostics“Placebo” TestsTests for treatment effects where none should exist aresometimes called placebo tests.In the RD context, this involves estimating jumps in outcomeat points away from the discontinuity, where treatmentassignment status does not change and thus there should beno jumps in outcomes.One suggestion sometimes made is to test for jumps at themedian of the two subsamples on either side of the cutoffvalue.Lecture Notes, Week 12 54/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnostics“Placebo” TestsTests for treatment effects where none should exist aresometimes called placebo tests.In the RD context, this involves estimating jumps in outcomeat points away from the discontinuity, where treatmentassignment status does not change and thus there should beno jumps in outcomes.One suggestion sometimes made is to test for jumps at themedian of the two subsamples on either side of the cutoffvalue.(Some say this leads to the most powerful test—the test withthe greatest power to reject the null hypothesis, conditional onit being false—though this depends on the distribution of theforcing variable. The modes might be best).Lecture Notes, Week 12 54/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSensitivity to bandwidth choiceHowever the bandwidth is chosen, one might investigate thesensitivity of the inferences to this choice.Lecture Notes, Week 12 55/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSensitivity to bandwidth choiceHowever the bandwidth is chosen, one might investigate thesensitivity of the inferences to this choice.Imbens and Lemieux recommend “including results forbandwidths twice (or four times) and half (or a quarter of) thesize of the originally chosen bandwidth . . . if the results arecritically dependent on a particular bandwidth choice, they areclearly less credible than if they are robust to such variation inbandwidths.”Lecture Notes, Week 12 55/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSensitivity to bandwidth choiceHowever the bandwidth is chosen, one might investigate thesensitivity of the inferences to this choice.Imbens and Lemieux recommend “including results forbandwidths twice (or four times) and half (or a quarter of) thesize of the originally chosen bandwidth . . . if the results arecritically dependent on a particular bandwidth choice, they areclearly less credible than if they are robust to such variation inbandwidths.”Adjustments for multiple comparisons may be in order herewhen doing hypothesis testing—e.g., Bonferroni correctionsdivide the nominal level of the test by the number ofcomparisons.Lecture Notes, Week 12 55/ 55
  • Standard natural experimentsRD designsExamples of RD designsFuzzy RDInterpreting the RD estimandBandwidth and diagnosticsSensitivity to bandwidth choiceHowever the bandwidth is chosen, one might investigate thesensitivity of the inferences to this choice.Imbens and Lemieux recommend “including results forbandwidths twice (or four times) and half (or a quarter of) thesize of the originally chosen bandwidth . . . if the results arecritically dependent on a particular bandwidth choice, they areclearly less credible than if they are robust to such variation inbandwidths.”Adjustments for multiple comparisons may be in order herewhen doing hypothesis testing—e.g., Bonferroni correctionsdivide the nominal level of the test by the number ofcomparisons.So if two different bandwidths are tried, the nominal p-valuemay be adjusted by 0.052= 0.025; that is, to reject at the truelevel of 0.05 on any one of the tests, a nominal p-value of0.025 is required. (But N.B., these aren’t independent tests).Lecture Notes, Week 12 55/ 55