COVID-19 related prediction models for diagnosis and prognosis - a living systematic review

COVID-19 related prediction models for
diagnosis and prognosis: living SR
Maarten van Smeden
University Medical Center Utrecht
Julius Center for Health Sciences and Primary Care
The Netherlands
Twitter: @MvanSmeden
Email: M.vanSmeden@umcutrecht.nl
28 May 2020
Julius Seminar
I have no conflicts of interest to declare

Utrecht, 28 May 2020 Twitter: @MaartenvSmeden
Thanks to
Laure Wynants (Maastricht/Leuven)
Ewoud Schuit (Utrecht)
Gary Collins (Oxford) for the materials for these slides

https://coronavirus.jhu.edu/map.html (26/5/2020)

Utrecht, 28 May 2020 Twitter: @MaartenvSmedenDoi: 10.1136/bmj.m1328

The COVID-19 systematic review tsunami
https://www.crd.york.ac.uk/prospero

Utrecht, 28 May 2020 Twitter: @MaartenvSmedenData from: https://ispmbern.github.io/covid-19/ (publications in Pubmed/Embase; preprints in BioRxiv and MedRxiv

(Clinical) Prediction models
• Support clinical decision-making for individual patients
• Combining and giving appropriate weights to several inputs
(e.g., CT image characteristics, signs & symptoms, lab test
results, demographics, …)

What not?
• Virus, contact or transmission models
• “Risk factor” or prognostic factor finding studies
• Diagnostic accuracy studies
• Effects of treatments, policies or vaccines
https://www.crd.york.ac.uk/prospero

Utrecht, 28 May 2020 Twitter: @MaartenvSmeden10.1101/2020.03.05.20031906

Timeline
Doi: 10.1136/bmj.m1328

Timeline
Doi: 10.1136/bmj.m1328
1916 titles screened
15 studies included
describing 19 models
(first submitted version)

Timeline
Doi: 10.1136/bmj.m1328
1st submission

Timeline
Doi: 10.1136/bmj.m1328
2690 titles screened
27 studies included
describing 31 models
(published version)

Timeline
Doi: 10.1136/bmj.m1328
Received peer review
feedback

Timeline
Doi: 10.1136/bmj.m1328
Replied to peer
review + update

Timeline
Doi: 10.1136/bmj.m1328
Paper accepted

How?
• Used existing website to download COVID-19 related TIAB
(Pubmed/Embase/BioRxiv/MedRxiv, we manually added arXiv)
https://zika.ispm.unibe.ch/assets/data/pub/search_beta/

How?
• Used existing RoB and data extraction tool
Doi: 10.7326/M18-1377, 10.1371/journal.pmed.1001744

How?
• Used existing RoB and data extraction tool
• 14 experienced risk modelling reviewers from AT, BE, NL, UK.
Screening and data extraction independently by 2 reviewers
• Pre-submission inquiry (day 1), fast track peer review and
editorial process
• PRISMA and TRIPOD followed for our reporting

Flow diagram
Doi: 10.1136/bmj.m1328

Results
• 51 papers describing the development or validation of COVID-
19 related prediction models
• 45 preprints: medRxiv (n=30), arXiv (n=13), bioRxiv (n=2)
• 6 peer-reviewed & published: Clin Infect Dis, Crit Care, Eur
Radiol, Phys Eng Sci Med, PLoS One, Radiology
• 32 studies used data from China, 2 from Italy, 1 from
Singapore, 10 international data, 2 simulated data, 1 Medicare
data, 3 studies unclear
Doi: 10.1136/bmj.m1328

Results
• 3 models for predicting hospital admission from pneumonia
• 47 diagnosis models for COVID-19 or COVID-19 pneumonia
• 34 based on medical images (deep learning)
• 16 prognosis models for predicting mortality risk, progression
to severe disease, or length of stay
Doi: 10.1136/bmj.m1328

Results
• Hospital admission models: AUC range 0.73 to 0.81
• Diagnosis models: AUC range 0.85 to 0.99
• Diagnostic imaging models: AUC range 0.81 to 0.998
• Prognosis models: AUC range 0.85 to 0.99
Doi: 10.1136/bmj.m1328

PROBAST

PROBAST
• Risk of bias any flaw or shortcoming in the design, conduct or
analysis of a primary study that is likely to distort the predictive
performance of a model
• Assess risk of bias on four domains using “signaling questions”
• Participants (2 questions)
• Predictors (3 questions)
• Outcome (6 questions)
• Analysis (9 questions)
• If risk of bias was high in at least one domain, overall risk of bias
was judged to be high

PROBAST - results
• Participants domain: 24/51 at high risk of bias
• Non-representative of the target population (e.g., non-
consecutive patients)
• Predictors domain: 6/51 at high risk of bias
• Predictors not available at time of intended model use
• Outcome domain: 18/51 at high risk of bias
• Subjective or proxy outcomes
• Analysis domain: 50/51 at high risk of bias
• Small sample size (->overfitting & no adjustment), incomplete
reporting of model performance (e.g., no calibration)

All models at high
risk of bias

Poor reporting
• Unclear study designs
• Lack of clear description of the study population
• Basic characteristics not reported
• Age: not reported in 22 studies
• Sex: not reported in 22 studies
• Often unclear on the intend moment of use of the model, what
and how many predictors were examined, missing data
handling, sample size

Poor prediction models can make things worse
Doi: 10.1378/chest.1118087

Conclusion
• COVID-19 prediction models were poorly reported, at high risk of
bias, and their reported performance is probably optimistic
• Application of currently available prediction models not
recommended

From systematic review sprint to marathon
• 18 days from conception to paper accept
• BMJ first living review for coming months
• Collaboration with Cochrane Prognosis Methods Group
• April 7 -> 4909 TIAB screened -> 66 models total
(to be published shortly)
• May 4 -> 9306 TIAB screened (semi-automated) -> 148 models
total (analysis underway)

COVID-PRECISE
• Large IPD-MA initiative
• Partners in Sweden, UK, Zwitserland, USA and expanding
• Goals
• Develop and validate COVID-19 related prediction models
• Understand and model heterogeneity between healthcare
settings
• Lead also by Thomas Debray and Valentijn de Jong

Team science
Coordination team
Laure Wynants (Maastricht)
Maarten van Smeden (Utrecht)
Carl Moons
Ben Van Calster (Leuven)
Reviewers
Thomas Debray (Utrecht)
Valentijn de Jong
Ewoud Schuit
Hans Reitsma
Toshi Takada
Lotty Hooft
Anneke Damen
Constanza Navarro
Florien van Royen
Pauline Heus
Luc Smits (Maastricht)
Sander van Kuijk
Ewout Steyerberg (Leiden)
Anna Lohmann
Kim Luijken
Georg Heinze (Vienna)
Maria Haller
Christine Wallisch
Nina Kreuzberger (Cologne)
Nicole Skoetz
Darren Dahly (Cork)
Michael Harhay (Philadelphia)
Robert Wolff (York)
Ioana Tzoulaki (London)
Gary Collins (Oxford)
Jie Ma
Paula Dhiman
Richard Riley (Keele)
Kym Snell
Matthew Sperrin (Manchester)
Jamie Sergeant
Glen Martin
Jack Wilkinson
Chunhu Shi
Jan Verbakel (Leuven)
Information specialist
René Spijker (Utrecht)
Advisors
Marc Bonten (Utrecht)
Maarten De Vos (Leuven)
Liesbet Henckaerts

COVID-19 related prediction models for diagnosis and prognosis - a living systematic review

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to COVID-19 related prediction models for diagnosis and prognosis - a living systematic review

Similar to COVID-19 related prediction models for diagnosis and prognosis - a living systematic review (20)

More from Maarten van Smeden

More from Maarten van Smeden (11)

Recently uploaded

Recently uploaded (20)

COVID-19 related prediction models for diagnosis and prognosis - a living systematic review