Algorithm based medicine

Maarten van Smeden, PhD
AI & Big Data
Amsterdam UMC
June 15 2022
Algorithm based medicine

June 15, 2022 Twitter: @MaartenvSmeden
https://bit.ly/2CwW43A

Forsting, J Nuc Med, 2017, DOI: 10.2967/jnumed.117.190397

Proportion of studies indexed in Medline with the Medical Subject
Heading (MeSH) term “Artificial Intelligence” divided by the total number
of publications per year.
Faes et al. doi: 10.3389/fdgth.2022.833912

Reviewer #2

https://bit.ly/2TOdd0F

https://bit.ly/2v2aokk

Tech company business model

Tech company business model
https://bit.ly/2HSp8X5; https://bit.ly/2Z0Pfop; https://bit.ly/2KIcpHG; https://bit.ly/33IJhr9

Other success stories
https://go.nature.com/2VG2hS7; https://bbc.in/2Z1drXQ; https://bit.ly/2TAfRIP

IBM Watson winning Jeopardy! (2011)
https://bbc.in/2TMvV8I

IBM Watson for oncology
https://bit.ly/2LxiWGj

Machine learning everywhere
https://bit.ly/2ka0HLq; https://go.nature.com/33TQgO6; https://bit.ly/2kp6X23; https://bit.ly/2lZuKWt; https://bit.ly/2lI298g

Examples where
“ML” has done well

Example: retinal disease
Gulshan et al, JAMA, 2016, 10.1001/jama.2016.17216; Picture retinopathy: https://bit.ly/2kB3X2w
Diabetic retinopathy
Deep learning (= Neural network)
• 128,000 images
• Transfer learning (preinitialization)
• Sensitivity and specificity > .90
• Estimated from training data

Example: lymph node metastases
Bejnordi et al, JAMA, 2018, doi: 10.1001/jama.2017.14585. See our letter to the editor for a critical discussion: https://bit.ly/2kcYS0e
Deep learning competition
But:
• 390 teams signed up, 23 submitted
• “Only” 270 images for training
• Test AUC range: 0.56 to 0.99

Primary outcome: time to TB treatment.
Time to TB treatment lowered from a median of 11 days in
standard of care to 1 day with computer aided X-ray screening

10.1016/j.cell.2020.01.021

Misconceptions about
machine learning methods

https://bit.ly/38A1ng0

“Everything is an ML method”
https://bit.ly/2lEVn33

“ML methods are new computing methods”
• Artificial intelligence (since 1956)
• Machine learning (since 1959)
• Deep learning (since 1962)
• Reinforcement learning (since 1965)
Courtesy: Prof Daniel Oberski

“ML methods come from computer science”
https://bit.ly/2zhbwPv; https://stanford.io/2TVp1xK; https://stanford.io/2ZfED0k
Leo Breiman Jerome H Friedman Trevor Hastie
CART, random forest Gradient boosting Elements of statistical learning
Education Physics/Math Physics Statistics
Job title Professor of Statistics Professor of Statistics Professor of Statistics

“ML methods for prediction, statistics for explaining”
1See further: Kreiff and Diaz Ordaz; https://bit.ly/2m1eYdK
ML and causal inference, small selection1
• Superlearner (e.g. van der Laan)
• High dimensional propensity scores (e.g. Schneeweiss)
• Causal random forest (e.g. Wager)
• The book of why (Pearl)

Two cultures
Breiman, Stat Sci, 2001, DOI: 10.1214/ss/1009213726

Faes et al. doi: 10.3389/fdgth.2022.833912
Language

Robert Tibshirani: https://stanford.io/2zqEGfr
Machine learning: large grant = $1,000,000
Statistics: large grant = $50,000

ML refers to a culture, not to methods
Distinguishing between statistics and machine learning
• Substantial overlap methods used by both cultures
• Substantial overlap analysis goals
• Attempts to separate the two frequently result in disagreement
Pragmatic approach:
I’ll use “ML” to refer to models roughly outside of the traditional regression
types of analysis: decision trees (and descendants), SVMs, neural networks
(including Deep learning), boosting etc.

Beam & Kohane, JAMA, 2018, doi : 10.1001/jama.2017.18391

learning

Adversarial examples
https://bit.ly/2N4mQFo; https://bit.ly/2W7X9rF

”Class imbalance”
Doi: 10.1371/journal.pone.0158285

Potentially harmful idea: imbalance correction
• Imbalance between “events”
and “non-events” often
considered problematic
• Examples exist where >99%
of “non-events” were
discarded, to “correct” for
imbalance
• Can have very bad
consequences

Recidivism Algorithm
Pro-publica (2016) https://bit.ly/1XMKh5R

Skin cancer and rulers
Esteva et al., Nature, 2016, DOI: 10.1038/nature21056; https://bit.ly/2lE0vV0

Predicting mortality – the conclusion
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344

Predicting mortality – the results
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344

Predicting mortality – the media
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344; https://bit.ly/2Q6H41R; https://bit.ly/2m3RLrn

HYPE!

“As of today, we have deployed the system in 16 hospitals, and
it is performing over 1,300 screenings per day”
MedRxiv pre-print only, 23 March 2020,
doi.org/10.1101/2020.03.19.20039354

FDA APPROVED
FDA APPROVED

• Published on 7 April 2020
• 18 days between idea and article acceptance (sprint)
• Invited by BMJ as the first ever living review (marathon)

Living review (update 3)
doi: 10.1136/bmj.m1328

Living review (update 3)
Risk of bias assessment ursing PROBAST tool: https://www.probast.org/
doi: 10.1136/bmj.m1328

Doi: 10.1016/j.jclinepi.2021.01.009

Systematic review clinical prediction models
Christodoulou et al. Journal of Clinical Epidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004

Sources of prediction error
Y = 𝑓 𝑥 + 𝜀
For a model 𝑘 the expected test prediction error is:
σ!
+ bias! -
𝑓" 𝑥 + var -
𝑓" 𝑥
See equation 2.46 in Hastie et al., the elements of statistical learning, https://stanford.io/2voWjra
Irreducible error Mean squared prediction error
(with E 𝜀 = 0, var 𝜀 = 𝜎!
, values in 𝑥 are not random)
What we don’t model How we model
≈
≈

Y = 𝑓 𝑥 + 𝜀
σ!
+ bias! -
𝑓" 𝑥 + var -
𝑓" 𝑥
≈
≈
In words, two main components for error in predictions are:
• Mean squared predictor error
• Under control of the modeler

Y = 𝑓 𝑥 + 𝜀
σ!
+ bias! -
𝑓" 𝑥 + var -
𝑓" 𝑥
≈
≈
overfitting underfitting ”just right”

Y = 𝑓 𝑥 + 𝜀
σ!
+ bias! -
𝑓" 𝑥 + var -
𝑓" 𝑥
≈
≈
• Irreducible error
• Not under direct control of the modeler

Bias-variance trade-off
Irreducible error

Irreducible error is often large
• Health and lack thereof complex to measure (‘no gold standard’)
• Predictors of diseases are often imperfectly and partly
measured
• We often don’t know all the causal mechanisms at play
• much easier to predict if you know the causal mechanisms!
• “Prediction is very difficult, especially if it’s about the future!”
(Niels Bohr might have said this first)
Courtesy Cecile Janssens: https://bit.ly/2Jf5ft6

What can we do to reduce “irreducible” error?
• Changing the information
• Prognostication by text mining electronic health records
• e.g. predicting life expectancy
https://bit.ly/2k8Ao8e
• Analyzing social media posts
• e.g. pharmacovigilance, adverse events monitoring via Twitter posts
https://bit.ly/2m0KKrg
• Speech signal processing
• e.g. Parkinson‟s disease,
https://bit.ly/2v3ZdHR
• Medical imaging

Bias-variance trade-off revisited: double descent

But…

Flexible algorithms are data hungry
From slide deck Ben van Calster: https://bit.ly/38Aqmjs

Flexible algorithms are energy hungry
The costs of running (cloud computing) the Transformer
algorithm are estimated at 1 to 3 million Dollars
https://bit.ly/33Dj38X

Sample size for development and validation

Reporting

Pipeline of algorithmic medicine failure
Van Royen et al, ERJ, in press

Regulation
source: https://tinyurl.com/2yjtkz3h

https://www.leidraad-ai.nl/

What is the guideline?
• Guideline provides a description of what the work field
considers good professional conduct in the development,
testing and implementation of an Artificial Intelligence Prediction
Algorithm (AIPA) in the medical sector, including public
healthcare.
• Part of professional standard of professionals.
• Guideline is not legally binding, but it is relevant because it
describes what is considered as good professional conduct.

Comply or explain
• The guideline distinguishes
between requirements and
recommendations for good
professional conduct.
• The requirements are
indicated by mandatory.
Recommendations are
indicated by (strongly)
recommended.
• Use of the field standard
presupposes a comply or
explain approach.

Target group
Healthcare provider
Professional Scientific
/ medical associations
Education/training IT
suppliers
Citizen
Applying AI Assessing AI
Validator
Responsible
developer
Researcher
Data manager Data
supplier
Developing AI
(Internal)
supervisor
Notified body Peer
reviewer Privacy
officer Insurer
Patient(s)-
(associations)
Interest parties
Political parties
Interested citizen
Society

Phases guideline
Collection and
management of the
data
Phase 1
Development of the
AIP
Phase 2
Validation of the
AIPA
Phase 3
Development of the
required software
Phase 4
Impact assessment
of the AIPA in
combination with
the software
Phase 5
Implementation
and use of the AIPA
with software in
daily practice
Phase 6
Saskia Haitjema
Andre Dekker
Paul Algra
Amy Eikelenboom
Christian van
Ginkel
Martine de Vries
Daniel Oberski
Desy Kakiay
Kicky van
Leeuwen
Joran Lokkerbol
Evangelos
Kanoulas
Gabrielle
Davelaar
Wouter Veldhuis
Bart-Jan Verhoeff
Vincent Stirler
Daan van den
Donk
Huib Burger
Giovanni Cina
Martijn van der
Meulen
Maurits Kaptein
Floor van
Leeuwen
Egge van der Poel
Marcel Hilgersom
Sade Faneyte
Jonas Teuwen
Teus Kappen
Ewout Steyerberg
Leo Hovestadt
René Drost
Bart Geerts
Anne de Hond
René Verhaart
Nynke Breimer
Karen Wiegant
Laure Wynants
Lysette
Meuleman

Doi: 10.1093/eurheartj/ehac238

Algorithm based medicine
• Algorithms are high maintenance
• Developed models need repeated testing and updating to
remain useful over time and place
• Many new barriers: black box proprietary algorithms,
computing costs
• Regulation and quality control of algorithms
• Algorithms need testing, preferably in experimental fashion

Old statistics wine in new machine learning bottles?
Lots of…
• Hype
• Rebranding traditional analysis as ML and AI
• Methodological reinventions
• Traditional issues such as low sample size, lack of adequate
validation, poor reporting
Also, real developments in…
• Methods and architectures, allowing for modeling (unstructured)
data that could previously not easily be used
• Software
• Computing power
• Clinical trials showing benefit of AI assistance

Email: M.vanSmeden@umcutrecht.nl
Twitter: @MaartenvSmeden

Algorithm based medicine

More Related Content

What's hot

Similar to Algorithm based medicine

More from Maarten van Smeden

Recently uploaded

Algorithm based medicine