Developing appropriate methods for addressing missing data in health economic evaluation

1. Developing appropriate methods
for handling missing data in
health economic evaluation
Manuel Gomes
CHE seminar, University of York
November 20, 2013
Improving health worldwide
www.lshtm.ac.uk

2. Acknowledgments
 MRC Early Career Felloswship in Economics of Health
 PROMs
 Nils Gutacker
 Chris Bojke
 Andrew Street
 Others
 Rita Faria
 David Epstein
Improving health worldwide
www.lshtm.ac.uk

3. Overview
• Missing data problem in health economic evaluation
• Alternative methods for addressing missing data
• Why multiple imputation?
• Framework for sensitivity analysis
• Illustrate the methods in a re-analysis of PROMs for comparing
provider performance
Improving health worldwide
www.lshtm.ac.uk

4. Missing data in economic evaluation
• Cost-effectiveness analyses are prone to missing data:
– Patients lost to follow-up
– Incomplete resource use or quality-of-life questionnaires:
• Individual non-response
• Item non-response
• Key concern is that patients with missing information tend to be
systematically different from those with complete data
• Concerns face economic evaluations based on a single-study and
those that synthesise data from several sources in decision models
• Most published studies fail to address missing data (Noble et al 2012)

5. Reasons for missing data
• Missing completely at random (MCAR)
– Reasons for missing data are independent of both observed and unobserved
factors
E.g. Resource use questionnaires were lost
• Missing at random (MAR)
– Prob of missingness is unrelated to unobserved values, given the observed data:
any systematic differences between missing and observed values can be
explained by differences in observed factors.
E.g. Older people may be less likely to return their QoL questionnaire.
• Missing not at random (MNAR)
– Conditional on the observed data, the likelihood of non-response is still related
to unobserved values.
E.g. Patients in poor health may be less likely to return their EQ-5D
questionnaires because they are depressed.

6. Key considerations when dealing with
missing data in CEA
• Missing endpoints (e.g. costs and QALYs) are made up of multiple
individual components with potentially distinct missingness patterns
• The probability distribution of missing data may differ by treatment
group and endpoint
• Probability of observing one endpoint (e.g. costs) may depend on the
level of the other endpoint (e.g. QoL)
• Distribution of costs and QALYs are often non-Normal
• Model for the missing data must recognise the structure of the data
(e.g. hierarchical) and be compatible with model for the endpoints

7. Alternative methods for dealing with missing
data in CEA
Why multiple imputation?
Improving health worldwide

8. Unprincipled methods (MCAR)
• Complete-case analysis
• Available-case analysis
• Last value carried forward
• Mean Imputation
All assume MCAR
– very unlikely, particularly for patient-reported outcomes
– typically leads to biased estimates

9. Principled methods (MAR)
Regression (single) imputation
• Some sort of regression model is used to predict missing values
conditional on the observed data (MAR).
• Missing observations are replaced by the predicted values
• Analysis model is then applied to the complete dataset
• Do not recognise that the ‘imputed’ values are estimated rather
than known (uncertainty is underestimated)

10. Principled methods (MAR)
Likelihood-based approaches
• Maximum likelihood does not fill in the missing values (!)
• ML uses all observed data to search for the parameters that
maximise the likelihood
• ‘Borrows’ information from observed data to estimate parameters
for the incomplete variables
• Variables associated with missingness are often beyond those
included in the analysis model (e.g. post-randomisation variables)
• With missing covariates may require more complex algorithms
(E.g. Expectation-Maximisation).

11. Principled methods (MAR)
Inverse probability weighting (IPW)
• Complete cases are weighted by the inverse probability of being
observed
• Tend to be less efficient than, say ML methods, because uses only a
subset of all available information.
• Although efficiency can be improved (e.g. augmented IPW within a
SUR framework), implementation can be challenging in complex
analysis models
• Limited use when covariates are missing

12. Principled methods (MAR)
Full-Bayesian models
• Model for the missing data and analysis model are estimated
simultaneously (typically using MCMC methods)
• May be advantageous when prior evidence is available to inform
either model
• With flat priors, this method approximates multiple imputation
• Relatively complex to implement when covariates are missing
• May face convergence issues

13. Principled methods (MAR)
Multiple Imputation
• Each missing value is replaced by a set of plausible values from the
conditional distributional of the missing data given the observed
Key advantages when compared with previous methods:
– Imputation model is estimated separately from the analysis
model (e.g. allows for the inclusion of ‘auxiliary variables’)
– Recognises the uncertainty associated with the missing data and
the estimation of the imputed values
– Can handle missingness in both outcomes and covariates
– Provides a flexible framework for conducting sensitivity analyses

14. Principled methods (MAR)
Multiple Imputation
Key requirements for valid inference:
– Imputation model must be compatible with analysis model:
• E.g. Bivariate hierarchical model
– Imputation model is correctly specified
• E.g. Non-linear relationships between follow up and baseline
QALY
– Imputed values drawn from multivariate Normal distribution
• Normalising transformations (e.g. skewed costs)
• Latent Normal models (e.g. ordinal EQ-5D components)

15. Standard MI in CEA
• Joint imputation model for missing costs and outcomes
𝑐 𝑖 = 𝛽0𝑐 + 𝛽1𝑐 𝑋 𝑖 + 𝜀 𝑖𝑐
𝑒 𝑖 = 𝛽0𝑒 + 𝛽1𝑒 𝑋 𝑖 + 𝜀 𝑖𝑒
𝜀 𝑖𝑐
~𝐵𝑉𝑁
𝜀 𝑖𝑒
0
0
𝜎2
𝑐
𝜌𝜎 𝑐 𝜎 𝑒
𝜎2
𝑒
• Missingness predictors (X) allowed to differ by endpoint
• Joint model recognises missing costs may be associated with
outcomes, and vice-versa
• Can impute individual cost and outcome components (multivariate)
May be insufficient in more complex settings:
– Hierarchical studies (e.g. multicentre studies, meta-analysis)
– Non-randomised studies, where correctly specifying the
imputation model can be challenging

16. Multilevel MI
• Multilevel MI recognises hierarchical data structures
(probability of missing data may be more similar within than across centres)
• Imputes from multivariate hierarchical Normal (Gomes et al 2013):
𝑐
𝑐 𝑖𝑗 = 𝛽0𝑐 + 𝛽1𝑐 𝑋 𝑖𝑗 + 𝛽2𝑐 𝑍 𝑗 + 𝑢 𝑗𝑐 + 𝜀 𝑖𝑗
𝑒
𝑒 𝑖𝑗 = 𝛽0𝑒 + 𝛽1𝑒 𝑋 𝑖𝑗 + 𝛽2𝑒 𝑍 𝑗 + 𝑢 𝑗𝑒 + 𝜀 𝑖𝑗
𝑐
𝜀 𝑖𝑗
𝑒 ~𝐵𝑉𝑁
𝜀 𝑖𝑗
0
,
0
𝜎2
𝑐
𝜌𝜎 𝑐 𝜎 𝑒
𝜎2
𝑒
𝑢 𝑗𝑐
~𝐵𝑉𝑁
𝑢 𝑗𝑒
𝜏2
0
, 𝑐
0
𝜙𝜏 𝑐 𝜏 𝑒
𝜏2
𝑒
• Random effects MI found to perform better than fixed effects MI
(Diaz-Ordaz et al 2013)
•
R and Stata code available

17. MULTIPLE IMPUTATION FRAMEWORK
AND SENSITIVITY ANALYSIS
Improving health worldwide

18. Sensitivity Analysis 1: model
specification
‘Robust’ MI (Daniel and Kenward 2012)
• Robust MI is based on the concept of ‘double robustness’ which aims to
reduce reliance on correct specification of imputation model
• ‘Doubly robust’ estimators remain unbiased even if imputation model is
misspecified, given the probability of missingness is correctly specified
• Analysts can be more confident about the correct specification of the
latter (typically some sort of logistic model)

19. Sensitivity Analysis 1: model
specification
IMPLEMENTATION
1 – First, estimate the model for missing data, given the observed data.
2 – Obtain the predicted probabilities, say 𝜋 𝑖𝑗 , of observing the
endpoint
3 – Estimate your imputation model, by including the inverse of the
probabilities obtained in step 2 (𝜋 −1 ) as an additional predictor.
𝑖𝑗
4 – Combine the multiple estimates using Rubin’s rules as usual.
Remarks:
- estimates of the variance are not doubly robust
- extreme weights

20. Sensitivity analysis 2: Departures
from MAR
Framing the problem
• True data mechanism is unknown given the data at hand
• Assessing whether CEA inference is sensitive to alternative
assumptions about the reasons for the missing data is required
• Sensitivity parameters can be formulated:
1. In terms of missing data selection mechanism (departures from MAR), or
2. According to differences between distribution of observed and unobserved data
• Probability distribution for these parameters should ideally be
informed by external evidence (e.g. expert elicitation)

21. Sensitivity analysis 2: Departures
from MAR
1. Selection models
• Requires the specification of a model that explicitly recognises the
MNAR mechanism:
logit {𝑃 𝑅 𝑖𝑗 = 1 𝑊𝑖𝑗 , 𝑍 𝑗 } = 𝜂0 + 𝜂1 𝑊𝑖𝑗 + 𝜂2 𝑍 𝑗 + 𝜑 𝑗 + 𝜹𝒀 𝒊𝒋
• Selection model is jointly estimated with analysis model, typically using
MCMC methods in a Bayesian framework (Mason et al 2012)
• Prior distributions for 𝜹 are best informed by expert opinion

22. Sensitivity analysis 2: Departures
from MAR
Selection models approximated by importance weighting
• Avoids having to estimate 𝜹.
Implementation (Carpenter et al 2007)
• Impute missing data under MAR
• Results are combined by computing a weighted average instead of
standard Rubin’s rules
• For a plausible 𝛿, imputations judged to have a more plausible MNAR
mechanism are given a relatively higher weight.
𝑤𝑚=
𝑛1
𝑤𝑚
𝑚
𝑖=1
𝑤𝑚
−𝛿𝑌𝑖𝑗𝑚
𝑤 𝑚 = exp
𝑖=1

23. Sensitivity analysis 2: Departures
from MAR
2. Pattern-mixture models (Carpenter and Kenward 2013)
i) Approach starts by imputing the missing data under MAR
ii) Then, it is assumed the distribution of unobserved values differs from
that of observed values by, say 𝜃
𝑌𝑖𝑗 = 𝛽0 + 𝛽1 𝑋 𝑖𝑗 + 𝛽2 𝑍 𝑗 + 𝑢 𝑗 + 𝜀 𝑖𝑗
𝑖𝑓 𝑌𝑖𝑗 𝑖𝑠 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑
𝑌𝑖𝑗 = (𝛽0 +𝜽) + 𝛽1 𝑋 𝑖𝑗 + 𝛽2 𝑍 𝑗 + 𝑢 𝑗 + 𝜀 𝑖𝑗
𝑖𝑓 𝑌𝑖𝑗 𝑖𝑠 𝑢𝑛𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑
iii) Using the draws from 1 and probability distribution of 𝜃, impute missing
𝑌𝑖𝑗 using model above.
(Implementation code for R and Stata under way)

24. Sensitivity analysis 2: Departures
from MAR
Choice of method
• Choice of approach for conducting sensitivity analysis will be
dependent on the type of inference of interest.
• Example: Study comparing a new therapy for glycaemia control with
standard medical management.
Are we interested in?
• Making inference for the population (patients with diabetes), given
that they attended clinic visits as they should (de jure question)
• Making inference for the population, taking into account those whose
participation may be erratic (de facto question)

25. Case study
Patient-reported outcome measures (PROMs)
• Routinely collected in the English NHS and used to compare hospital
performance
• Prone to missing data:
– Providers may fail to administer a questionnaire at hospital admission
– Patients may refuse to participate or fail to return post-operative questionnaire
• Over 140,000 patients undergoing hip replacement in 2010-2012
• PROMs questionnaires linked to HES
• Post-operative EQ-5D was missing for 54% patients

26. Investigating the reasons for missing
data
Different forms of incomplete data in PROMs
80000
Number of episodes
70000
15%
14%
35%
26%
12%
14%
60000
50000
40000
30000
20000
Not linked: missing NHS
number or insufficient
match
Not linked: patient
refusal or questionnaires
not administered
Linked: incomplete
questionnaires
46%
38%
10000
Linked: complete
questionnaires
0
2010-2011
2011-2012
Year

27. Investigating the reasons for missing
data
Negative association
Positive association
Waiting time (over 3 months)
Length-of-stay (per extra day in hospital)
Charlson Index (>1)
Previous admitted to hospital
4th quintile
3rd quintile
2nd quintile
Most Deprived
Provider type (private)
Ethnicity (white)
Male
Age >75
Age 65-75
Age 55-65
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
Odds ratio of missingness
1.3
1.4
1.5

28. Investigating the reasons for missing
data
0
.2
.4
.6
.8
1
Association between provider-specific outcome and response rate
0
.5
1
Standardised response rate
1.5
2

29. Analysis model
Assessment of relative provider performance
• Provider-specific outcomes (𝑦2𝑗 ) were estimated using the NHS risk adjustment
method (Nuttall et al 2013):
𝑦2𝑖𝑗 = α + 𝑥 ′𝑖𝑗 𝛽 + 𝑦1𝑖𝑗 𝛾 + 𝑢 𝑗 + 𝜀 𝑖𝑗,
𝑦2𝑗 = 𝜌 𝑗 𝑦2,
𝜌𝑗 =
1
𝑛
𝑛
𝑖=1
𝜀 𝑖𝑗 ~𝑁(0, 𝜎 𝜀 )
𝑦2𝑖𝑗
𝑦2𝑖𝑗
• To help identify potential outliers we followed the methodology recommended
by Department of Health (funnel plots)

30. Imputation model
Multivariate multilevel ordered probit model (Goldstein et al 2009)
Let ℎ 𝑘,𝑖𝑗 be the kth component (k=1,…,5) of the EQ-5D, with M ordinal categories, m=1,…,3,
and 𝑦 ∗ its latent (unobserved) Normal response defined as:
𝑘,𝑖𝑗
ℎ 𝑘,𝑖𝑗
1 𝑖𝑓 𝑦 ∗ ≤ 𝜅1
𝑘,𝑖𝑗
∗
= 2 𝑖𝑓 𝜅1 < 𝑦 𝑘,𝑖𝑗 ≤ 𝜅2
3 𝑖𝑓 𝑦 ∗ > 𝜅2
𝑘,𝑖𝑗
Imputed values can then be drawn from the multivariate multilevel Normal distribution:
𝑘
𝑘
𝑘
𝑦 ∗ = 𝜹0 + 𝑤 ′ 𝜹1𝑘 + 𝑧 ′ 𝜹2 + 𝝊 𝑗𝑘 + 𝝂 𝑖𝑗
𝑘,𝑖𝑗
𝑖𝑗
𝑗
𝑘
Where 𝝂 𝑖𝑗 ~𝑀𝑉𝑁(0, Ω 𝜈 ) and 𝝊 𝑗𝑘 ~𝑀𝑉𝑁 0, Ω 𝜐 , and Ω is the 𝑘 × 𝑘 covariance matrix
We conducted 50 imputations and 5000 MCMC, with each set of imputed values
drawn from the posterior distribution every 100th iteration of the MCMC chain

31. Funnel plots & provider performance
Funnel plots and performance status
1.00
Provider-specific outcomes
Positive outlier: Alarm
0.90
Positive outlier: Alert
0.80
0.70
Negative outlier: Alert
0.60
Negative outlier:
Alarm
0.50
0
20
40
60
80
Volume
99.8%
95%
Target
100

32. MI, funnel plots & provider
performance
Funnel plots’ horizontal and vertical effects after MI

33. Complete-case analysis
.7
.8
.9
1
Funnel plot of provider-specific outcomes for complete cases
.6
22 negative
outliers: alarm
0
500
1000
1500
Volume
Upper 95% CI
Upper 99.8% CI
Adjusted post-operative EQ-5D
Lower 95% CI
Lower 99.8% CI

34. CCA vs Multiple Imputation
3
2
1
0
Density
4
5
6
Kernel densities for adjusted outcomes according to CCA and MI
0
.2
.4
.6
Adjusted post-operative EQ-5D
CCA
After MI
.8
1

35. Multiple Imputation
.7
.8
.9
1
Funnel plot of provider-specific outcomes after multiple imputation
.6
9 negative
outliers: alarm
0
1000
2000
3000
Volume
Upper 95% CI
Upper 99.8% CI
Adjusted post-operative EQ-5D
Lower 95% CI
Lower 99.8% CI

36. Multiple Imputation
.6
.7
.8
.9
1
Changes in the performance status
0
1000
2000
3000
Volume
Upper 95% CI
Lower 95% CI
Upper 99.8% CI
Lower 99.8% CI
Adjusted post-operative EQ-5D
Lost positive alarm status
Moved away from negative alarm status

37. Sensitivity analysis
Provider performance according to alternative methods for missing data
Post-operative EQ-5D
Control
limits
Upper @99.8% Upper @95% In control Lower @95% Lower @99.8%
(positive
(positive
(negative alert) (negative
alarm)
alert)
alarm)
CCA
(MCAR)
9 (3%)
26 (9%)
203
(71%)
25 (9%)
22 (8%)
Multilevel MI
(MAR)
8 (3%)
16 (5%)
254
(83%)
17 (6%)
9 (3%)
Robust MI
(MAR)
6 (2%)
23 (8%)
251
(82%)
14 (5%)
10 (3%)
MNAR
7 (2%)
21 (7%)
253
(83%)
11 (4%)
12 (4%)

38. Summary
• CEA context poses specific challenges to methods for addressing
missing data
• Unprincipled methods unlikely to be plausible in practice
• Under MAR, MI framework is flexible and naturally extends
to sensitivity analysis
• PROMs illustrates how MI can help address specific challenges
Future work
• Simulation work to compare relative merits of alternative
methods in complex circumstances
• Project website (R and Stata code, guindance, etc.)
Improving health worldwide
www.lshtm.ac.uk

39. References
•
•
•
•
•
•
•
•
•
Carpenter, J. & Kenward, M. 2013. Multiple Imputation and its Application, Chichester, UK., Wiley.
Carpenter, J. R., Kenward, M. G. & White, I. R. 2007. Sensitivity analysis after multiple imputation
under missing at random: a weighting approach. Stat Methods Med Res, 16, 259-75.
Daniel, R. M. & Kenward, M. G. 2012. A method for increasing the robustness of multiple
imputation. Computational Statistics & Data Analysis, 56, 1624-1643.
Diaz-Ordaz, K. Gomes, M. Grieve, R. and Kenward, M. 2013. Random effects versus fixed effects
multiple imputation for handling clustered missing data. For submission to Computational
Statistics & Data Analysis
Goldstein, H., Carpenter, J., Kenward, M. G. & Levin, K. A. 2009. Multilevel models with
multivariate mixed response types. Statistical Modelling, 9, 173-197.
Gomes, M. Diaz-Ordaz, K. Grieve, R. and Kenward, M. 2013. Multiple imputation methods for
handling missing data in CEA that use data from hierarchical studies: an application to cluster
randomized trials. Med Decis Making, 33: 1051-1063.
Mason A, Richardson S, Plewis I, Best N. Strategy for modelling nonrandom missing data
mechanisms in observational studies using Bayesian methods. J Off Stat. 2012;28(2):279–302.
Noble, S. M., Hollingworth, W. & Tilling, K. 2012. Missing data in trial-based cost-effectiveness
analysis: the current state of play. Health Econ, 21, 187-200.
Nuttall, D., Parkin, D. & Devlin, N. 2013. Inter-Provider Comparison of Patient-Reported
Outcomes: Developing an Adjustment to Account for Differences in Patient Case Mix. Health Econ.
Improving health worldwide
www.lshtm.ac.uk