1. Comparative Effectiveness
Research (CER) Methods
Patrick Richard, PhD
Adjunct Assistant Professor of Health Economics
GWU Health Policy and Economics Program
The Department of Health Policy

2. Outline
 Major limitations of Randomized Clinical Trials
(RCTs)
 Advantages and challenges of observational
studies
 Sample selection bias or confounding
 Methods to address limitations of observational
studies
 Propensity Score (PS)
 Instrumental Variables (IV)
The Department of Health Policy 2

3. Instrumental Variables (IV)
 IV techniques have the potential to address the
problem of omitted or residual bias due to
unobserved confounding in CER
 A carefully chosen observed variable that should
be:
 (1) Correlated with treatment
 (2) But uncorrelated with the error term in the
outcome equation
The Department of Health Policy 3

4. Instrumental Variables (IV)
 IV is relatively easily to implement in CER if the
outcome equation is linear such as test scores,
biometric measurements, or nonzero spending
amounts
 The IV is constructed by estimating a logistic
regression (or probit) model to predict the
probability of treatment for each observation
 Substitute the predicted probability for the treatment
variable in the outcome equation
The Department of Health Policy 4

5. Instrumental Variables (IV)
 Challenging if outcome is non-linear such as
mortality, hospitalization, heart attacks, strokes,
recurrence of cancer, or other binary or count
variables
 Use the approach developed by Newey et al.
(1999) and others
 Estimate a predicted value for the error term and
include it explicitly in the outcome equation (2
stage residual inclusion-2SRI)
The Department of Health Policy 5

6. Instrumental Variables (IV)
 Both linear and non-linear IV models are two-step
procedures estimated simultaneously, not
sequentially
 Sequential estimations produce incorrect standard
errors
 Linear or non-linear, the following issues still apply:
 The strength of instruments
 Overidentifying restrictions
 Local average treatment effects (LATE)
The Department of Health Policy 6

7. IV-Strength of the instrument
 An IV should be strongly related to treatment
because weak instruments present several
problems:
 Magnify any potential bias (Bound et al. 1995)
 Lead to substantial inconsistency in the IV
estimator
 Yield highly variable estimates, which make it
difficult to detect small effects, even in very large
studies
The Department of Health Policy 7

8. IV-Strength of the instrument
 Staiger and Stock (1997) suggest that a joint test of
the hypothesis that all coefficients on the
instruments equal zero should generate an F-
statistic of 10 or more
 As a rule of thumb, F statistics less than 10 are
thought to be problematic
 Models with stronger instruments (those that
‘‘move’’ more people) produce more generalizable
results
The Department of Health Policy 8

9. IV- Overidentifying restrictions
tests
 Used to test for the correlation between
instruments and the error term in outcome
equation (Davidson and MacKinnon 1993)—The so
called “exclusion criterion”
 Can only be used if more than one IV indicator
 No test is possible if there is only one IV indicator
 Hence, the model is described as ‘‘exactly identified’’
The Department of Health Policy 9

10. Interpretation of the IV results
 The Wald estimator yields the average effect of
treatment among the “compliers”:
 Patients who would always take their assigned
treatment
 Take active therapy if assigned to it
 Take placebo if assigned placebo
 In other words, only patients whose treatment
status is influenced by the IV-Local Average
Treatment Effect (LATE)
The Department of Health Policy 10

11. IV-Good practice
 Justify Need for and Role of IV in the Study
 IV methods are inefficient and should not be used
as a primary analysis unless there is strong
evidence of unmeasured confounding
 Discuss why substantial unmeasured confounding
is expected
The Department of Health Policy 11

12. IV-Good practice
 Describe Theoretical/Empirical Basis for the Choice
of IV
 A good IV should have a theoretical motivation
 Why it is expected to influence treatment, but
unrelated to outcome?
 Is it supported by empirical evidence?
 For example, do patients chose hospitals without
knowledge of their formulary?
 Thus, formulary status may be effectively randomly
assigned
The Department of Health Policy 12

13. IV-Good practice
 Report the first-stage F statistic and the partial R2
attributable to the inclusion of the IV
 F-statistic >10 is desirable
 The partial R2 is the proportion of the variance
explained by the addition of the IV to the model
 Discuss issues related to interpretation of the
estimator
 The IV effect only generalizes to patients whose
treatment status depends on the instrument
The Department of Health Policy 13

14. Examples of IVs
 Preference-based: Defined at the level of the
geographic region, hospital, dialysis center, or
individual physician
 Local variations in physicians’ practice patterns
 Institutional factors such as formulary design
differences, program eligibility rules,
implementation timelines, or provider network
policy changes
The Department of Health Policy 14

15. PS & IV- An example
 Stukel et al. (2007) used four different methods to
assess the effects of cardiac catheterization on
elderly patients hospitalized for acute myocardial
infarction
 Methodological concern: patients in poorer health
were less likely to receive invasive care,
potentially making the effects of treatment look
better than they actually were
The Department of Health Policy 15

16. PS& IV-An Example
 The authors compared multivariate risk
adjustment, propensity score risk adjustment,
propensity score matching, and instrumental
variables results
 IV: Regional cardiac catheterization rate
 The results were substantially different depending
upon method
The Department of Health Policy 16

17. PS & IV –An Example
 Multivariable risk adjustment, propensity score risk
adjustment, and propensity score matching show
reductions in mortality risk between 46 and 49
percentage points
 IV estimates show 16 percentage points reductions
in mortality risk, comparable to estimates from
RCTs, which ranged from 8 to 21 points
 Evidence of sample selection bias in observational
data
The Department of Health Policy 17

18. Conclusions
 Very difficult to find “good” instruments
 Heterogeneity issues
 In a recent report submitted to the President and
Congress, the Federal Coordinating Council on
Comparative Effectiveness Research states:
“Clinicians and patients need to know not only
that a treatment works on average but also which
interventions work best for specific types of
patients (e.g. the elderly, racial and ethnic
minorities)” (FCC Report, June 30, 2009).
The Department of Health Policy 18