1. ExternalValidationofEightLungCancer
PredictionModels
Eoin Gray, Dr. Dawn Teare and Dr. John Stevens
University of Sheffield, School of Health and Related Research
1. Objectives
An external validation was conducted for lung
cancer prediction models to;
1. Evaluate the predictive ability of pub-
lished lung cancer prediction models.
2. Compare multiple models and selective
screening trial criteria to assess their abil-
ity as a screening tool.
3. Identify the risk thresholds that allowed
the models to perform optimally.
The external validation shall identify the opti-
mal selective screening criteria.
2. Prediction Models
Eight predictions models were identified which
had distinct model designs:
Model Restrictions
Bach [1] Ever-smokers, 50-75 years, 30+ PY
LLP [2] 40-80 years
Spitz [3] 40+ years
Afr.-Amer. [4] 40+ years
Hoggart [5] Ever-smokers, 35+ years
Pittsburgh [6] Ever-smokers
PLCOM2012 [7] Ever-smokers, 20+ years
PLCOM2014 [8] 20+ years
Table 1: Summary of Prediction Models
3. Screening Trials
The NLST [9] and UKLS [10] screening trials
criteria were also validated.
Criteria NLST UKLS
Eligible Population 50-74 50-75
Entry Criteria 30+ PY ≥ 5% LLP risk
Additional Quit < 15 years
Table 2: Summary of Screening Trial Criteria
This criteria was also applied to the dataset to
assess their ability as a selective screening
criteria.
4. Validation Plan
Models were validated in 10 datasets provided
by ILCCO.
Calibration
Calibration was measured through the Hosmer-
Lemeshow test.
This assessed if the models could accurately
predict the observed lung cancer incidence rates
in the datasets.
Discrimination
Discrimination was measured by the Area
Under the Curve (AUC).
This evaluates how successful the model
assigns a higher risk to cases than controls.
Prediction Rules 1
The models were applied to their target popula-
tion to identify a risk threshold where the model
consistently performed strongly.
Prediction rules were evaluated at 0.1, 0.25, 0.5,
1, 1.5, and 2.5% thresholds.
The sensitivity, specificity, and positive likeli-
hood ratio (PLR) were reported.
The NLST criteria was also applied to provide a
baseline performance.
Prediction Rules 2
The models were evaluated in the UKLS target
population (Anyone aged 50 − 75).
Models were assessed while restricting unnec-
essary screening of controls.
Specificity was fixed at 90% which was deter-
mined by LLP Model at 5%.
The risk threshold and sensitivity, specificity,
and PLR were reported.
The UKLS guidelines were validated to offer a
baseline performance.
5. Calibration & AUC
The calibration and discrimination was validated;
Models struggled to exhibit a good calibration.
Pittsburgh Model marginally had the best
calibration (4 out of 10 datasets).
The PLCOM2014 Model had strongest AUC
(0.69-0.79 across datasets).
6. Validation One
The PLCOM2014 Model had best performance.
The NLST criteria would excluded many
participants but fail to capture a high
proportion of cases to screen.
PLCOM2012 and Pittsburgh models performed
well for ever-smokers.
The Spitz Model reported promising results
but needs to be validated in additional environ-
ments as it could only be applied in one dataset.
Model Optimal Sens. Spec. PLR
PLCO2014 0.5% 70% 75% 2.800
PLCO2012 0.5% 80% 55% 1.941
Pitts. 1% 65% 65% 1.857
Spitz 0.5% 72% 70% 2.400
NLST NA 35% 80% 1.750
Table 3: Key Validation One Results
7. Validation Two
The PLCOM2014 Model and PLCOM2012 Model
had the optimal performance.
For six year risk they should be applied at 3%
and 3.5% threshold for a sensitivity of 35%
This improved upon the UKLS guidelines.
The UKLS guidelines reported a sensitivity
around 20-22% for specificity of 90%.
The Pittsburgh Model also improved upon the
UKLS guidelines.
Model Optimal Sens. Spec. PLR
PLCO2014 3% 35% 90% 3.500
PLCO2012 3.5% 35% 90% 3.500
Pitts. 1% 30% 90% 3.000
UKLS 5% 20% 90% 2.000
Table 4: Key Validation Two Results
8. Conclusions
The PLCOM2014 Model should be used to iden-
tify a target population for screening.
The model should be applied at the 0.5% risk
threshold to identify a high proportion of cases.
The model would capture 70% of cases while
rejecting 75% of controls.
To PLCOM2014 Model should be applied at the
3.5% threshold to avoid screening 90% of con-
trols. The model would still manage to identify
35% of cases.
Contact Information
Web sheffield.ac.uk/scharr/sections/dts/staff
/eoingray
Email epgray1@sheffield.ac.uk
References
[1] Bach PB et al. Variations in lung cancer risk
among smokers. JNCI. 2003;95(6):470-8.
[2] Cassidy A et al. British Journal of Cancer.
2008;98(2):270-6.
[3] Spitz MR et al. Journal of the National Cancer In-
stitute. 2007;99(9):715-26
[4] Etzel CJ et al. Cancer Prevention Research.
2008;1(4):255-65.
[5] Hoggart C et al. Cancer Prevention Research.
2012;5(6):834-46.
[6] Wilson DO et al. Lung Cancer. 2015;89(1):31-7
[7] Tammemaegi MC et al. New England Journal of
Medicine. 2013;368(8):728-36.
[8] Tammemagi MC et al. PLoS medicine.
2014;11(12):e1001764-e.
[9] Gatsonis CA, Natl Lung Screening Trial Res T. Ra-
diology 2011;258(1):243-253
[10] McRonald, F et al. Cancer Prevention Research,
2014:7(3):362-371.