The document presents a detailed discussion on designing causal inference studies using real-world data, emphasizing best practices and methodologies. It includes an overview of the BestMed study, which analyzes second-line diabetes therapies in patients with moderate cardiovascular risk, and a case study evaluating the impact of medication adherence on outcomes for COPD patients. The document also highlights the significance of careful study design to minimize biases and improve the robustness of conclusions drawn from health-related data.

1. Copyright 2022. All Rights Reserved. Contact Presenter for Permission
Designing Causal Inference
Studies Using Real-World
Data
Alexander Turchin, MD, MS
Director of Informatics Research
Division of Endocrinology
Brigham and Women’s Hospital
Shivani Pandya, MS
Associate Research Director
HEOR/Pharmacy Economics
HealthCore, Inc.
Michael Grabner, PhD
Principal Scientist
Scientific Affairs
HealthCore, Inc.

2. Designing Causal Inference
Studies Using Real-World Data
Michael Grabner, PhD
Principal Scientist
HealthCore, Inc.
healthcore.com @HealthCoreRWE
Alexander Turchin, MD, MS
Director of Informatics Research
Brigham and Women's Hospital
Shivani Pandya, MS
Associate Research Director
HealthCore, Inc.

3. Agenda
Part 1
Causal Study Design: Introduction and
Best Practices
Michael Grabner, PhD
Part 2
The BESTMED Study
Alexander Turchin, MD, MS
Part 3
A PBM Case Study
Shivani Pandya, MS

4. Disclosures and Acknowledgements
Michael Grabner and Shivani Pandya are employees of HealthCore, Inc. (a wholly owned subsidiary of
Elevance Health), which conducts health services research with both internal and external funding,
including a variety of private and public entities. Michael Grabner is a shareholder of Elevance Health.
Alexander Turchin receives funding from Astra Zeneca, Edwards, Eli Lilly, Novo Nordisk for projects
unrelated to the content of this webinar. Alexander Turchin is a shareholder of Brio Systems and a
consultant to Proteomics International.
The comments stated herein are the opinions of the authors. HealthCore makes no representations or
warranties, express or implied, with respect to the use or reliance on the opinions stated herein.
The HealthCore presenters are indebted to our colleagues from the following teams: Causality
Workgroup, Pharmacoeconomics, Scientific Affairs, Marketing and Communications.

5. Causal Study Design:
Introduction and Best Practices
Michael Grabner, PhD
Principal Scientist, Scientific Affairs
HealthCore, Inc.

6. HealthCore’s Research Environment
Ecosystem
Direct
Patient
Data
Social
Determinants
of Health
+
+
Eligibility
+
Clinical
Data
Lab
Results
+
Medical
& Pharmacy
Claims
+
Multiple integrated data assets
Providing a more complete picture of health care
80M identifiable lives
in our data ecosystem since 2006
Ability to link
to external data sources such as
NDI, registries, EMR, claims, etc.
Mortality
Data
+
Acronyms: NDI, National Death Index; EMR, Electronic Medical Records

7. HealthCore’s Patient-Centered Research Areas

8. Why We (Should) Care about Causality
Am J Public Health. 2018;108:616–619.
Association
vs.
Causality
Value Health. 2019; 22(5):587–592.
Design
vs.
Statistical
Techniques
Observational
Data for
Decision-
Making
https://www.fda.gov/media/120060/download

9. A Step-By-Step Guide to Causal Study Design
Define
Estimand
Acronyms: GEE, generalized estimating equations; IPC/TW, inverse probability of censoring/treatment weighting;
ITR, individual treatment response; MSM, marginal structural model; TE, treatment effect
1
Define Research Question
Association
Most biases
disregarded by
definition
Causal Effect
Move to Step 2
4
Measure of Effect?
Contrast: Difference or ratio?
Outcome: Risk, rate, hazard, odds, cost…?
8
Plan QC
& Sensitivity Analyses
• Test if model assumptions are fulfilled
• Use different estimand or estimator
• Quantitative bias analysis
7
Explore the Land of Solutions
• “Target trial” thinking
• New user design with active comparator
• Choose estimator and missing data rules
• Confounder adjustment
o Time-invariant (“baseline”)
§ Matching or weighting
§ Best with propensity scores
o Time-varying
§ Survival analysis with time-varying
covariates
§ Mixed models, GEE
§ MSM with IPTW (if confounders are
affected by prior treatment)
• Evaluate confounder balance
• IPCW to account for loss-to-follow-up/
censoring
6
Navigate the Land of Biases
• Measured confounding
• Unmeasured confounding
• Collider bias
• Selection bias
• Immortal time bias
• Protopathic bias (reverse causality)
• Healthy adherer effect
• Prevalent user bias
• Dependent/informed censoring
• Misclassification
• Effect modification
• Generalizability & Transportability
Etc.
5
Create Directed Acyclic Graph
(DAG)
Exposure Outcome
Confounder
Collider
Mediator
Effect in Whom?
Target Population For Counterfactual Contrast
• Average treatment effect (ATE)
• ATE in the (un)treated (ATU or ATT)
• Conditional ATE (subgroups)
• Individual TE (ITR)
2
Intention-to-Treat (ITT)
Per-protocol
As-treated
Which Kind of Effect?
3

10. The BESTMED Study
Alexander Turchin, MD, MS
Director of Informatics Research, Endocrinology
Brigham and Women's Hospital

11. oBservational Evaluation of Second line
Therapy MEdications in Diabetes – BESTMED
• Metformin is the accepted 1st line therapy in type 2 diabetes. There is
no consensus about 2nd line therapy.
• A number of diabetes medications (GLP1 agonists, SGLT2 inhibitors)
have been shown to improve outcomes in patients at high
cardiovascular risk. What about patients at moderate cardiovascular
risk?
• BESTMED will study optimal 2nd line therapy in patients with type 2
diabetes at moderate cardiovascular risk.
• Funded by Patient-Centered Outcomes Research Institute (PCORI).

12. Clinical Trials vs. Real World Evidence
• Controlled
• Randomized
• Double-blind
• Selection bias?
• Information bias?
• Confounding bias?

13. Nobody is Perfect
Powered to succeed?
• X vs. Y: OR 2.1 (0.9 – 3.8)
• Negative result?

14. Nobody is Perfect
Powered to succeed?
• X vs. Y: OR 2.1 (0.9 – 3.8)
• Negative result?
• Insufficiently powered to
answer the question.
Easy does it?
• Clinical trials take time.
• Answers may be out of date
by the time they arrive.

15. Solution: Avoid Most of the Bias
Perfect is the worst enemy of good enough.
I am 95%
confident that
you are the one!

16. Avoiding Selection Bias: Compare Like to Like
Compare like to like: patients in
intervention and control group must
be in similar clinical circumstances
• BESTMED: patients are being
started on one of five classes of
diabetes medications when their
blood glucose is elevated.
• Not like-to-like: patients who are
already taking diabetes medication
X vs. diabetes medication Y
≠

17. Outcomes: What Matters Most
Primary (Composite)
• Myocardial infarction
• Admission for heart failure
• Stroke
• Cardiovascular death
Secondary
• Individual components of the primary outcome
• Non-cardiovascular complications of diabetes
• Adverse reactions to study medications
• Balance: Composite of diabetes complications +
adverse reactions to diabetes medications that led
to a prolonged hospital stay
Include outcomes that a) matter to patients and b) can be reliably
ascertained from the available data.
BESTMED Outcomes

18. Outcomes: Creative Approach to Data
• Medullary thyroid cancer: rare cancer postulated to be a possible side
effects of diabetes medications (GLP1 agonists).
• There is no ICD code for medullary thyroid cancer – only for thyroid cancer
in general.
• Calcitonin level is used to a) diagnose and b) monitor for relapse of
medullary thyroid cancer.
• Solution: look for patients with
• An ICD code of (any) thyroid cancer +
• At least two calcitonin measurements (of any value) – a single measurement could be
used for (unconfirmed) diagnosis

19. Minimize Confounding Bias
Inverse Probability Weighting
A B
Higher probability of
treatment è lower weight
Lower probability of
treatment è greater weight

20. Garbage In = Garbage Out: Minimize
Information Bias
CHECK DATA QUALITY
• Check data ranges: blood glucose 10 mg/dL, BMI 11 kg/m2
• Compare data between study sites:
Patients Age MI Death Creatinine
Site-A 10,000 62 3.00% 0.25% 1.3
Site-B 100,000 71 4.50% 2.50% 2.0
Inappropriate
rounding of decimals
Missing
death data
Due to age difference:
OK

21. Intent-to-treat vs. Per-Protocol Analyses
• Patients who “break” the
protocol may be different from
those who don’t, introducing
bias in a per-protocol analysis.
• Adverse event of an intervention
leading to its discontinuation
should be counted as an
outcome.
• Answers the question What
happens if the patient is treated
with X? rather than the question
What happens if the patient
starts treatment with X?
• Takes cross-overs into account.
• Imbalance of patient
characteristics can be dealt with
by time-varying covariate
analysis.

22. Trust but Verify: Negative Controls
Pick an outcome that should not be affected by the study intervention
BESTMED
• Good negative control group: Lung cancer (not linked to any study drugs)
• Poor negative control group: Cholecystitis (side effect of a study drug)

23. Trust but Verify: Positive Controls
Pick an outcome that should be affected by the study intervention
BESTMED
• Good positive control group: HbA1c (GLP1 > DPP4)
• Poor positive control group: Blood pressure (small differences)

24. A PBM Case Study
Evaluating the Impact of Adherence to
Maintenance Medication on Outcomes Among
Patients with Chronic Obstructive Pulmonary
Disease: A Causal Inference Approach
Shivani Pandya, MS
Associate Research Director, HEOR
HealthCore, Inc.

25. Background
Inhaled maintenance
medications are the
standard of care for
many patients with
COPD1
Prior evidence indicative
of poor adherence which
is associated with
increased inpatient
admissions and total
cost2-4
Less is known of the
causal impact of
adherence on survival
and other outcomes for
patients
Objective: Assess effect
of adherence on
outcomes to support
pharmacy payers’
interventions
1Global Initiative for Chronic Obstructive Lung Disease 2018. https://goldcopd.org/. Accessed 24 February, 2021.
2Yu AP, Guérin A, Ponce de Leon D, et al. J Med Econ. 2011;14(4):486-496.
3Mannino D, Bogart M, Wu B, et al. Respir Med. 2022;197:106807.
4Davis JR, Wu B, Kern DM, et al. Am Health Drug Benefits. 2017;10(2):92-102

26. Two Different Analytic Approaches
Traditional Approach
• Examines association between exposure and
outcome
• Medication adherence regarded as a constant
exposure à Intent to Treat
• Accounts for only time invariant confounders
• Propensity score matching or weighting methods
• Relatively easy to implement
• Subject to following limitations
o Ambiguous temporal order
o Survival bias
o Assumption that medication adherence was constant
and not impacted by or in turn impacted other
covariates
Novel Causal Approach
• Examines average causal effect of exposure on
outcomes
• Medication adherence regarded as a time varying
exposure à As-Treated
• Accounts for time invariant & varying confounders
• Marginal structural model or other mixed models
• Complex data management and bias assessment
• Approach addresses several limitations:
o Ambiguous temporal order
o Survival bias
o Assuming constant nature of medication adherence
and confounders

27. Study Design Components #1
Data Source
• HealthCore Integrated Research Database
(HIRD®)
• Medical and pharmacy claims data linked with
mortality data
Cohort
• Retrospective, observational study design
• Patients with COPD aged ≥40 years with ≥1
maintenance regimen and ≥6 months follow-up
Study Period Start
01/01/2016
Study Period End
06/30/2021
Intake Period Start
01/01/2016
Intake Period End
12/31/2020
Initial Maintenance
Regimen Date
Index Date
≥6-months continuous
enrollment
6-months
Baseline Period
Notes:
Intake period was defined as the time period from 01/01/2016 – 12/31/2020 to examine the evidence of ≥1 inhaled maintenance medication regimen
The date of the first fill of inhaled maintenance medication was defined as the initial maintenance regimen date.
Index date was defined as the date 6 months after the initiation of first maintenance therapy
Figure 1. Study Design Overview

28. Study Design Components #2
Exposure
• Adherence estimated based on proportion of days
covered by the full COPD regimen on a daily rolling
basis; PDC ≥80% regarded as adherent – As Treated
• Discrete segments were created based on patients’
rolling adherence status until end of follow-up
Outcome
• All outcomes evaluated during each segment
• Clinical outcomes: All-cause mortality, all-cause and
COPD-related hospitalizations
• Economic outcomes: All-cause and COPD-related
medical and total costs
Study Period Start
01/01/2016
Study Period End
06/30/2021
Intake Period Start
01/01/2016
Intake Period End
12/31/2020
Initial Maintenance
Regimen Date
Index Date
7/1/2018
≥6-months continuous
enrollment
6-months
Baseline Period
Figure 2. Adherence assessment overview
6-month rolling period 1 (Adherent)
6-month rolling period 2 (Adherent)
6-month rolling period 3 (Non-Adherent)
6-month rolling period 4 (Non-Adherent)
7/2/2018 07/3/2018 7/4/2018
Segment 1 Segment 2

29. Study Design Components #3
Confounders
Measured confounding
• Time-invariant confounders: 6 months pre-
initial treatment
• Age, sex, region, plan and payor type,
initial regimen, Quan Charlson
comorbidity index, symptom burden,
all-cause and COPD-related resource
use and cost metrics
• Time-varying confounders: 6-months pre-
segment start
• Year of segment start, seasonality,
rescue medication fill rate, antibiotic
use, oxygen use, pulmonology visit,
other medication use, exacerbation
rate, all-cause and COPD-related
resource use and cost
Assumption of no unmeasured confounding
Notes: This figure demonstrates the assumed relationship between adherence and influential patient factors (time
varying and invariant) at every segment and its ultimate influence on outcomes. This figure is for illustration
purposes only (the study allowed for up to 20 time segments over an average follow-up period of 22 months)
Figure 3. Direct Acyclic Graph (DAG)

30. Study Design Components #4
Analytic Methodology
Marginal Structural Models:1-2 multi-step models to assess causal effect of adherence on
outcomes in the presence of measured time-varying and invariant confounders
• Step 1: Estimation of inverse probability of adherence vs non-adherence at each segment
using IPTW approach
o Confounder balance assessed based on standardized mean difference <10% pre- vs
post-IPTW
• Step 2: Generalized Estimating Equation (GEE) models fit in the weighted sample
o Estimates Average Treatment Effect (ATE)
o Results indicative of difference in average monthly costs and risk of survival or
hospitalization between adherent vs non-adherent segments
Sensitivity analyses planned to assess the effect of medication adherence on outcomes while
accounting for patients censored/loss to follow-up
1Robins JM, Hernán MA, Brumback B. Epidemiology. 2000 Sep;11(5):550-60.
2Yu AP, Yu YF, Nichol MB. Value Health. 2010 Dec;13(8):1038-45

31. Key Findings & Conclusions
Adherence to the full COPD regimen resulted in statistically significant and meaningful clinical and
economic benefits compared to non-adherence.
This robust real-world evidence can be leveraged by pharmacy payers to support the design and rollout of
their targeted adherence-based pharmacy initiatives that can influence quality performance metrics and
potentially result in total cost of care savings.
The study lays the analytic groundwork for robustly assessing the effect of medication adherence on
outcomes and is transferrable to different therapeutic areas in future.
This study is subject to limitations pertaining to use of claims data which are not generated for research
purposes and hence limits the ability to account for unobservable clinical and patient behavioral factors.
Publication planning is ongoing.

32. Thank You
Contact us at rwe@healthcore.com to find out more information.
healthcore.com @HealthCoreRWE

33. Finding evidence and truth
at the core of healthcare.

34. Thank you for participating!
CLICK HERE to learn more and
watch the webinar