1. UpdatingandAggregatingLungCancer
PredictionModels
Eoin Gray, Dr. Dawn Teare and Dr. John Stevens
University of Sheffield, School of Health and Related Research
1. Objectives
The research aimed to apply and evaluate
methods to update a single model or aggregate
multiple prediction models.
To achieve this the study had to;
1. Identify methods to update a single
model and aggregate multiple prediction
models that use an IPD.
2. Evaluate the methods practicality and
success when applied to lung cancer pre-
diction models.
3. Assess if an improved lung cancer pre-
diction model could be devised which
could be applied in selective screening
trials.
2. Methods
The methodology of the research was as follows;
1. Apply methods that can update a single prediction model using an external IPD to the
PLCOM2014 lung cancer model.
• The PLCOM2014 Model was selected for its leading performance in an external validation
of epidemiological lung cancer prediction models
2. Apply identified methods to aggregate multiple prediction models to the Bach, Hoggart, Pitts-
burgh and PLCOM2014 models.
• Two versions were conducted including and excluding the Bach Model as this model
limited the size of the IPD available to conduct the meta-modelling.
3. Externally validate the newly devised models in comparison to the original models to assess
if the methods could create a model with an improved performance.
4. Lung Cancer Models
The PLCOM2014 Model was considered for single updating and four models for model aggregation.
Model Never-Smokers Ever-Smokers Age Additional Duration Prediction
PLCOM2014 [4] X X 20+ 6 Years Incidence
Pittsburgh [3] X 6 Years Incidence
Hoggart [2] X 35+ 1(+) Years Incidence/Absolute
Bach [1] X 50-75 30+ PY 1(+) Years Incidence/Absolute
Table 1: Summary of Lung Cancer Prediction Models
The updated PLCOM2014 will predict 6 year risk of incidence for anyone aged 20+ years.
The aggregated model including the Bach Model will predict 6 year risk of incidence in ever-
smokers, aged 50-75, with a minimum 30+ pack year smoking history.
The aggregated model without the Bach Model will predict 6 year risk of incidence in ever-smokers
aged 35+ years.
3. Applied Methods
The literature review identified the following
methods to update the PLCOM2014 Model;
1. Update the intercept
2. Recalibration of the intercept and slope
3. Recalibration and selective re-estimation
4. Re-estimation
5. Re-estimation and extension
6. Selective re-estimation and selective extension
with recalibration
7. Re-estimation and selective extension without
recalibration
The following methods were identified for
model aggregation;
1. Model Averaging
2. Bayesian Model Averaging
• With an without an informative prior
5. Single Updating Results
The methods failed to improve the model dis-
crimination and prediction rules.
The model calibration improved in the internal
validation but struggled externally.
Method AUC Calibration
No Updating 0.7667 0 (823.89)
Recalibration 0.7667 0 (112.04)
Re-Estimation and Extension 0.7671 0 (319.17)
Table 2: External Validation of Leading Methods
Recalibration demonstrated the biggest calibra-
tion improvement.
Re-estimation and extension offered a simi-
lar discrimination and prediction rules to the
PLCOM2014 Model but not an improvement.
No method created an improved model.
6. Aggregating Results
Updating existing models using BMA with an
informative prior created a slightly improved
prediction model.
Model Model Weighting
PLCO 0.50942114
Hoggart 0.49057886
Pittsburgh 6.131e-16
Table 3: Weightings of Improved Model
The models reported a poor calibration.
The AUC marginally improved upon the
original models.
This model was optimal at 1.46% risk threshold
with a sensitivity of 84% and specificity of 36%.
Applying a 15.5% risk threshold would
maintain a specificity of 90% with a sensitivity
around 20%.
8. Conclusions
Single updating methods did not create an im-
proved model [5, 6].
The calibration, discrimination, and prediction
rules were all lower in the external validation
in comparison to the PLCOM2014 Model.
Model averaging and BMA were applied as
model aggregation techniques.
These did not improve the model calibration.
BMA with an informative prior offered a slight
improvement upon the original models.
The final model is a weighting of the Hoggart,
Pittsburgh, and PLCOM2014 models.
Weightings for the final model are in Table 3.
This model is optimal at 1.46% threshold and at
15.5% threshold would maintain a specificity of
90% for a sensitivity around 20%.
Contact Information
Web sheffield.ac.uk/scharr/sections/dts/staff
/eoingray
Email epgray1@sheffield.ac.uk
References
[1] Bach PB et al. Variations in lung cancer risk among
smokers. JNCI. 2003;95(6):470-8.
[2] Hoggart C et al. Cancer Prevention Research.
2012;5(6):834-46.
[3] Wilson DO et al. Lung Cancer. 2015;89(1):31-7
[4] Tammemagi MC et al. PLoS medicine.
2014;11(12):e1001764-e.
[5] Toll, D et al. Journal of Clinical Epidemiology, 61(11),
pp.1085-1094.
[6] Moons K et al. Heart, 98(9), pp.691-698.