Final Presentation given at the conclusion of the 2018 IMSM by the Savvysherpa Student Working Group. Group Members: Chixiang Chen, Ashley Gannon, Duwani Katmullage, Miaoqi Li, Mengfe Liu, Rebecca North and Jialu Wang

1. Identifying Precision Treatment for Rheumatoid
Arthritis with Reinforcement Learning
Chixiang Chen1
Ashley Gannon2
Duwani Katumullage3
Miaoqi Li4
Mengfei Liu5
Rebecca North6
Jialu Wang5
1Pennsylvania State University, Division of
Biostatistics and Bioinformatics
2Florida State University, Department of
Scientific Computing
3Sam Houston State University, Department of
Statistics
4University of Cincinnati, Department of
Mathematical Sciences
5The George Washington University,
Department of Statistics
6North Carolina State University, Department of
Statistics
July 25, 2018C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 1 / 24

2. Outline
1 Introduction
Background
The Problem
2 Methodology
3 Results
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 2 / 24

3. What is RA?
Rheumatoid Arthritis (RA)
A heterogeneous, chronic inflammatory disease that affects the lining
of the joints, causing pain and inflammatory flare ups
Other symptoms:
Issues of the heart, lungs, eyes and skin
60% increase in the risk of heart attack
Twice as likely as the average person to become depressed.
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 3 / 24

4. Current Treatments
Drug Therapies:
Disease-modifying
antirheumatic drugs
(DMARDs)
Synthetic or biologic
Glucocorticoids
NSAIDs or opioids
Other Therapies:
Physical and occupational
therapy
Surgical procedures
Assistive devices,
supportive products,
adjustments to home
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 4 / 24

5. The Problem
Motivation: The American College of Rheumatology (ACR) does not
provide an optimal treatment regime to follow after diagnosis.1
Research Goal: Use precision medicine techniques to develop a
dynamic treatment regime for the treatment of RA.
1
C.A. Wijbrandts and P.P. Tak. “Prediction of Response to Targeted Treatment in
Rheumatoid Arthritis”. In: Mayo Clinic Proceedings 92.7 (July 2017), pp. 1129–1143.
ISSN: 00256196. DOI: 10.1016/j.mayocp.2017.05.009. URL:
http://linkinghub.elsevier.com/retrieve/pii/S0025619617303658
(visited on 07/24/2018).
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 5 / 24

6. The Data
For each of 6,846 RA patients:
At least 6 months of weekly claims data prior to diagnosis
At least 12 months of weekly claims data after diagnosis
Comorbidity indicators prior to/after diagnosis & week of
development with respect to diagnosis
Age, gender, insurance subscriber indicator
For each week:
Counts for hospital (IP, OP, ER) and DR visits, and Rx scripts
Indicators of DMARD, NSAID, opioid, and glucocorticoid use
Treatment group
______ ______ ______ ______ ______1 10 0 0DMARD MTX nTNF TNF Tofa
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 6 / 24

7. The Data
For our analyses:
Only weeks after treatment initiation were included
We collapsed the data to be on a monthly scale
Summed hospital visit and Rx script counts
Indicator for NSAID or opioid use during the month
Indicator for glucocorticoid use during the month
Treatment group reassigned as maximum of each drug indicator
across the month
Comorbidity baseline adjusted to be at treatment initiation
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 7 / 24

8. Outline
1 Introduction
Background
The Problem
2 Methodology
3 Results
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 8 / 24

9. Q-Learning
Let’s narrow our focus to two stage decision-making problem.
Y: second stage outcome.
Ai: Treatment assignment at ith stage, i = 1, 2.
di(Xi): optimal decision rule at ith stage, i = 1, 2.
(d1(X1), d2(X2)) = argmin
a1,a2
E{Y|A1 = a1, X1, A2 = a2, X2}
Key questions:
How to model E{Y|A1 = a1, X1, A2 = a2, X2}?
How to predict optimize decision rule given a new observation?
How to evaluate the effect of optimized decision rule?
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 9 / 24

10. Q-Learning
Let’s deal with the first question under more rigorous formula:
Key formula
E{Y|A1, A2} = E E E{Y|A1, X1, A2, X2}|X1, A1 |A1, A2
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 10 / 24

11. Q-Learning
More specific, the pseudo algorithm we propose before is the idea of
Q-Learning2:
step 1: model E{Y|A1, X1, A2, X2} by some specified function
f2(A1, X1, A2, X2) and apply logistic regression to fit
step 2: predict the pseudo-outcome Y based on the fitted model
in step 1 under the optimized decision rules.
step 3 model E Y |X1, A1 by some function f1(A1, X1) based on
the pseudo-outcome Y and apply ordinary regression to fit
2
Rui Song et al. “Penalized Q-learning for dynamic treatment regimens”. In:
Statistica Sinica (2015). ISSN: 10170405. DOI: 10.5705/ss.2012.364. URL:
http:
//www3.stat.sinica.edu.tw/statistica/J25N3/J25N35/J25N35.html
(visited on 07/24/2018).
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 11 / 24

12. Q-Learning
Next, suppose we have a new patient stop by at the first month, then
optimized decision rule at first month will be given as:
ˆd1(X1) = argmin
a1
ˆf1(A1 = a1, X1)
If later we get the information about that patient at the second month,
then the corresponding optimal decision will be:
ˆd2(X1) = argmin
a2
ˆf2(A1 = ˆd1(X1), X1, A2 = a2, X2)
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 12 / 24

13. Estimation for expected outcome
Finally, a measure to evaluate how well our optimized decision rule
works:
E{Y|A1 = d1(X1), A2 = d1(X2)} ≤ E{Y|A1 = a1, A2 = a2}
Three Estimators for E{Y|A1, A2}:
naive estimation:
˜Y =
n
i Yi1{A1i}1{A2i}
n
i 1{A1i}1{A2i}
.
It might be biased under observational study
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 13 / 24

14. Estimation for expected outcome
IPTW estimator:
˜YIPTW =
n
i
Yi 1{A1i }1{A2i }
ˆP(A1i |x1i )ˆP(A2i |x2i )
n
i
1{A1i }1{A2i }
ˆP(A1i |x1i )ˆP(A2i |x2i )
P(A|X) is called propensity score modeled by multinomial logistic
regression.
We assume A1|X1 is independent of A2|X2
˜YIPTW is unbiased estimator only when propensity score P(A|X) is
correctedly specified
3
3
Erica E. M. Moodie, Bibhas Chakraborty, and Michael S. Kramer. “Q-learning for
estimating optimal dynamic treatment rules from observational data”. In: Canadian
Journal of Statistics 40.4 (Dec. 2012), pp. 629–645. ISSN: 03195724. DOI:
10.1002/cjs.11162. URL: http://doi.wiley.com/10.1002/cjs.11162
(visited on 07/24/2018).
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 14 / 24

15. Estimation for expected outcome
Double robust IPTW estimator:
˜YIPTW-rob =
1
n
n
i
Yi1(A1i)1(A2i)
ˆP(A1i|x1i)ˆP(A2i|x2i)
−
1(A1i)1(A2i) − ˆP(A1i|x1i)ˆP(A2i|x2i)
ˆP(A1i|x1i)ˆP(A2i|x2i)
× ˆf1(ˆd1(x1i), x1i)
same assumption and notation as before
˜YIPTW-rob is unbiased estimator if either one of ˆP(A|x) or
ˆf1(ˆd1(x1), x1), not both, is mis-specified (more recommended).
4
4
Baqun Zhang et al. “A Robust Method for Estimating Optimal Treatment
Regimes”. In: Biometrics 68.4 (Dec. 2012), pp. 1010–1018. ISSN: 0006341X. DOI:
10.1111/j.1541-0420.2012.01763.x. URL:
http://doi.wiley.com/10.1111/j.1541-0420.2012.01763.x (visited onC. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 15 / 24

16. Outline
1 Introduction
Background
The Problem
2 Methodology
3 Results
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 16 / 24

17. Results for Two-Stage Analysis
Table: Important Covariates for Two-Stage Analysis
Table: Stage 1
Age at diagnosis
Treatment group (month 1)
COPD
Hypertension
Angina
Skin ulcers
Gender
Table: Stage 2
Treatment group(month 2)
Glucocorticoid or painkiller(month 1)
Age at diagnosis
Prescriptions in month 1
Dr Visits in month 1
OP visits in month 1
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 17 / 24

18. Results for Two-Stage Analysis
Table: Important Covariates for Two-Stage Analysis
Table: Stage 1
Age at diagnosis
Treatment group (month 1)
COPD
Hypertension
Angina
Skin ulcers
Gender
Table: Stage 2
Treatment group(month 2)
Glucocorticoid or painkiller(month 1)
Age at diagnosis
Prescriptions in month 1
Dr Visits in month 1
OP visits in month 1
Under Optimal
Estimator Treatment Regime Under Observed Data
˜YIPTW 0.040 0.122
˜YIPTW-rob 0.059 0.112
Table: Results under Various Criteria
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 17 / 24

19. Visualization 1
factor(opt_trt)
1 TNF
1 DMARD
Visualization of the Categorical Response Variable and TreatmentID
When Angina = 1 and COPD = 1
Female
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 18 / 24

20. Visualization 2
factor(opt_trt)
1 TNF
1 MTX
Visualization of the Categorical Response Variable and TreatmentID
When Angina = 1 and COPD = 1
Male
1 DMARD
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 19 / 24

21. Visualization 3
factor(opt_trt)
1 TNF
1 MTX
Visualization of the Categorical Response Variable and TreatmentID
When Angina = 0 and COPD = 0
Female
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 20 / 24

22. Visualization 4
factor(opt_trt)
1 TNF
1 MTX
Visualization of the Categorical Response Variable and TreatmentID
When Angina = 0 and COPD = 0
Male
1 DMARD
1 DMARD &
1 MTX
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 21 / 24

23. Future Study
1 Making multiple decisions across half year can be extended.
2 Model selection procedure in multiple stages will be conducted.
3 We will obtain the confidence interval for predicted ˜Y.
4 Semi-parametric or non-parametric model will be implemented to
fit the data.
5 Methods dealing with not randomly censored data will be
developed.
6 Model global effect instead of the last stage effect will be
considered.
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 22 / 24

24. Acknowledgements
Grant Weller, Victoria Mansfield, Yinglong Guo
UnitedHealth Group, Research & Development
Daniel Luckett
University of North Carolina, Department of Biostatistics
SAMSI
Department of Mathematics, NC State University
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 23 / 24

25. Questions
Thank you for your attention.
Questions?
C. Chen et al. (SAMSI IMSM 2018) Precision Treatment for RA July 25, 2018 24 / 24