Understanding Comparative Effectiveness Research: RCT vs CER Nicholas P. Jewell Departments of Statistics & School of Public Health (Biostatistics) University of California, Berkeley MedicReS Vienna February 13-14, 2017

1. Understanding Comparative
Effectiveness Research: RCT vs CER
Nicholas P. Jewell
Departments of Statistics &
School of Public Health (Biostatistics)
University of California, Berkeley
MedicReS
Vienna
February 13-14, 2017
1

2. Natural History Schematic
Illness-Death Model
Time%
Origin%
(t"="0)%
Disease%
Ini2a2on%
Disease%
Expression%
Death%
Can be many stages between Initiation and Expression
2

3. • Factors associated with
• Cancer prevention
• Early detection
• Improved treatment to cure or delay
expression
• Want evidence-based choices
• Randomized Controlled Trials (RCTs)
• Not always possible
• Many options to be considered
• Optimal treatment strategies over time
• Effects in regular practice with
heterogeneous populations
Cancer Research
3

4. • Sometimes called “patient-centered outcomes
research”
• Diverse data sources include
• Electronic health records
• Patient registries
• Administrative records including
prescription databases
• Many treatment/intervention possibilities
• Observational data
• Concerns about confounding
Comparative Effectiveness Research
(CER)
4

5. • Why are these different? Think about condoms
and the risk of HIV infection!
• Efficacy (RCTs)
• Effect of intervention in ideal controlled
circumstances
• Treatment compared to placebo/standard of
care
• Enroll homogeneous patients to reduce
variability form other sources
• Effectiveness (Observational)
• Typical clinical conditions (not controlled)
• Many treatment “regimes”
• Multiple outcomes
• Heterogeneous patients/larger subgroups
Efficacy vs Effectiveness
5

6. • Usually focused on efficacy
• Controls confounding (well, almost . . .)
• Reduces variation through controlling eligibility etc.
• May have limited generalizability
• Not usually designed for sub-group information
Randomized Clinical Trials
6

7. ! Usually pragmatic trials that reflect “real world” practice
! Often uses active control arms
! More heterogeneous patient populations
• allows possibility of sub-group comparisons if larger
sample sizes are used
• Blinding may be difficult
Effectiveness Randomized Trials:
Pragamatic Trials
7

8. ! Randomization occurs at the group/practice level
! Assess impact of organization and delivery metrics
! Avoids contamination of treatment within clinic (were individual
randomization to be used)
• e.g. effect of patient navigation services for individuals with
abnormal breast screening test
! Smaller number of randomization units
• Imbalance and residual confounding
! Blinding often impossible
! Must account for within cluster correlation (larger sample sizes
needed)
Effectiveness Randomized Trials:
Cluster-Randomized Trials
8

9. ! Evaluation of patient navigation system designed to reduce time
between cancer abnormality and definitive diagnosis—primary
outcome (Wells et al., 2012)
! Cluster randomized trial of 11 primary care clinics clinics (6
receiving intervention, 5 control clinics)
! Breast and colorectal abnormalities
! Linear mixed effects models
! Is this approach effective with such small number of clusters?
Cluster-Randomized Trials: Example
9

10. ! Trial design changes based on current information on
recruited patients
• Patient characteristics
• Treatment alternatives
• Treatment allocations
! Often results unblinded during trial to make adaptive
changes
! Complex statistically
Effectiveness Randomized Trials:
Adaptive Trials
10

11. ! Large diverse populations
! Many treatment regimens/practices
! Sub-group comparisons possible
! No randomization: therefore possible confounding
! e.g. confounding by indication
! Many variables need to be collected to control
confounding effects
Observational Studies
11

12. • “Found” Data Resources/registry information
• Cross-sectional studies (prevalence surveys)
• Cohort Studies (incidence or prevalence or
mixture)
• Case-Control Studies
• Hybrid Studies
All can suffer from various selection effects
Observational Study Designs
12

13. • Most basic cross-sectional study
• Identifies presence/absence of condition
and exposure at time of sampling
• No further information on timing of event if
it has already happened
• Temporality is lost (did exposure occur before
or after onset of disease)
Cross-Sectional Studies
13

14. Cross-Sectional Sample Selection Biases
Sampling at (i) fixed chronological time (vertical line),
and (ii) at fixed duration (horizontal line)
Time%
Dura*on%
10%
20%
30%
40%
50%
60%
14

15. • Follow up of two groups: exposed/unexposed
• Expensive (cheaper if data collected
retrospectively)
• Impractical for rare outcomes
• Confounding remains an issue
Cohort Studies
15

16. Sample Selection Biases
Time%
Age%
2010%2000%1980% 1990%1970%1960%
10%
20%
30%
40%
50%
60%
(i) (left) truncation, (ii) (right) censoring,
(iii) interval censoring
Follow-up Interval between lines
16

17. • North Carolina-Louisiana Prostate Cancer Project
• Population-based sample of incident prostate
cancer cases
• Effect of race on screening, care, cancer
outcomes, etc
• Cases identifies from the Rapid Case
Ascertainment Core Facility (NC) and the
Louisiana Tumor Registry
• African Americans oversampled
Cohort Studies: Example
17

18. • Cross-sectional sampling
• Retain those with prevalent diseases and
follow in time until expression (right
censored usually)
• No further information on timing of event if
it has already happened
• Sometimes (randomized) treatment is begun at
sampling on prevalent cases
• Using time from sampling (treatment
initiation) to expression in a survival
regression model is biased (as compared to
using time from incidence to expression)
Prevalent Cohort Studies
18

19. • Example—assessment of natural history of hip
replacements—how long do artificial hips last
before they need to be replaced (i.e. hip
revision)?
• Elderly populations
• Deaths may occur during follow up
(unrelated to hip issues)
Handling Deaths in Cohort Follow-Up
19

20. • Treat death as censoring
• Death unlikely to be entirely independent of time to
event
• Estimates of survival/failure properties (eg Kaplan-
Meier estimator) in a world where death does not exist
• transportable across different populations
• Treat death as a competing risk
• Estimates a different parameter—what is the probability
of a revision within a certain time assuming I am alive
throughout that period?
• More interest to the individual subject
• But not transportable across populations
• Same issues arise in general longitudinal studies (not just
time to event analyses)
Handling Deaths in Cohort Follow-Up
20

21. • Impact of intermediate event on natural history
• Example 1
• Does using statins delay the need for insulin treatment
for diabetics?
• Statin use determined by start of new prescription after
diabetes diagnosis or entry into the cohort
• Example 2
• Mortality experience of individuals with single primary
melanoma to those who develop multiple primary
melanomas
• Time measured from onset of first primary melanoma
• Time from origin to onset of intermediate event occurrence is
immortal time since such an individual cannot have the event
(otherwise they would be in a different group!)
Immortal Time Bias
Use time-dependent time-to event analysis techniques 21

22. • Can’t measure absolute risks
• Recall bias? (Exposure “measured” after
outcome)
• Efficient for rare outcomes
• Confounding remains an issue
Case-Control Studies
22

23. • Carolina Breast Cancer Study
• Cases identified from the North Carolina
Central Cancer Registry (originally only
invasive but then in situ also)
• Controls from Drivers’ license and Medicare
beneficiary lists
• Frequency matched by age and race
• Logistic regression
• Different risk factors for different sub-types
(parity, age at first full-term pregnancy,
breastfeeding, BMI, etc)
Case-Control Studies: Example
23

24. • Casual diagrams (DAGs) and counterfactuals
• Mediation
• Instrumental variables
• Causal parameters
• Regression methodology (needs a good
understanding of DAGs): outcome mechanism
• G-computation
• Propensity score methods: treatment/intervention
assignment mechanism
• Matching
• stratification
• Weighting
• Longitudinal studies
Methods for Confounding Control
24

25. • Determination of an individual’s “state” at a
given point in time depends on a screening
test
• Screening tests can often detect that an
event has happened (e.g. infection, heart
damage) without being able to pinpoint
when it happened
• Screening tests are subject to
misclassification and we need to know their
sensitivity and specificity
• Comprehensive screening programs may
disrupt natural history processes
• Population natural history summaries
may differ because of different
screening practices
Screening Tests
25

26. • Screening tests may become more sensitive
over time
• Moves detectable pre-clinical period
forward in time
• Gives the appearance of lengthening the
time between onset and expression
(although nothing may have changed)
• Lead time bias (akin to Feinstein’s Stage
Migration bias in cancer studies)
• Very difficult to assess the effects of screening
programs in improving prognosis
• Often use case-control designs
Efficacy of Cancer Screening
26

27. • Nicholas P. Jewell, “Natural history of diseases: Statistical designs and issues,”
2016, Clinical Pharmacology & Therapeutics, 100, 2016, 353-361.
• J. Pearl, M. Glymour and N. P. Jewell. Causal Inference in Statistics: A Primer,
2016, John Wiley.
• A-M Meyer et al., “An overview of methods for comparative effectiveness
research,” Seminars in Radiation Oncology, 24, 2014, 5-13.
• L. M. Kunz et al., “An overview of statistical approaches for comparative
effectiveness research,” In Methods in Comparative Effectiveness Research,
Eds: C. Gatsonis & S.C. Morton, Chapman & Hall/CRC, 2017.
• K. J. Wells et al., “A cluster randomized trial evaluating the efficacy of patient navigation in
improving quality of diagnostic care for patients with breast or colorectal cancer
abnormalities,” Cancer Epidemiol. Biomarkers Prev., 21. 2012. 1664-1672.
• J. C. Schroeder et al., “The North Carolina-Louisiana prostate cancer project (PCaP):
Methods and design of a multidisciplinary population-based cohort study of racial
differences in prostate cancer outcomes,” The Prostate, 66, 2006, 1162-1176.
• R. C. Millikan et al., “Epidemiology of basal-like breast cancer,” Breast Cancer Res. Treat.,
109, 2008, 123-139.
• K. A. Cronin et al., “Case-control studies of cancer screening, JNCI, 90, 1998, 498-504.
• IARC Handbooks of Cancer Prevention, Cervix Cancer Screening, 2005 -- Chapter 5:
Effectiveness of screening in populations.
References