The early diagnosis of neurodegenerative diseases is crucial, as it could lead to early treatment and better chances of stopping the process. Machine learning algorithms provide an opportunity to diagnose these diseases earlier through automatic diagnosis, but their application to the medical domain is not straightforward. From data set size to interpretability, we will see why the beautiful, trendy and complex solution we can first think about might not be the best one.

1. Machine Learning for automatic
diagnosis: why your deep neural
network might not work
April 24th 2019

2. Projects 2
Amyloidosis prediction
• Predict the output of Amyloid PET
• Application : Recruit patients for clinical trials at a
lower cost
Predicting the evolution of a diagnosis
• Predict the development of Alzheimer’s disease
• Quantitative review

3. Data uncertainty 3
Reliability of the label
• Definition of AD not clear
• Subjective label established by specialist : sensitivity of 55 %
and specificity of 85 % (Beach et al 2012)
Noisy data
• Always look at the test – retest
(Koval et al 2019)

4. Data uncertainty 4
Biased missing values
• Example: proportion of hypertension in AD and non-AD subjects

5. Small data sets 5
ADNI Dataset
800 subjects with Mild Cognitive
Impairment (MCI), 550 followed for 3
years
MNIST
70k examples

6. Small data sets 6
Amyloidosis prediction
Work on 2 data sets:
• ADNI - 431 subjects, 6 features -> AUC = 69%
• INSIGHT – 318 subjects, 112 features -> AUC = 56%
• Lasso feature selection: AUC = 64 %
• Domain-knowledge summary variables (26): AUC = 68 %

7. Small data sets 7
Impact of smaller data sets:
• Lower performances
• Over-fitting
Over-fitting in deep learning
• High number of parameters
• Impact on performances

8. Lack of data 8
Data leakage
T –test feature
selection
Classification
(train – test)
T – test : separating
features
Paper 1
Paper 2
Paper 3
Common
knowledge
Feature selection
using domain
knowledge
Classification
(train – test)
Good… if different data sets are used

9. Lack of data 9
Solutions
• Using domain knowledge
• Validating on different data
sets
• Pooling cohorts to obtain
larger ones
Data set AUC
Trained and tested on INSIGHT 61.9
Trained on ADNI, tested on INSIGHT 62.0
Using both, same sample size 61.3
Using both, full sample size 67.5
Table: performances of amyloidosis prediction using MRI
features

10. Usability
What is the use case ?
• Information expected by the user
(doctor/patient)
• Inputs (unavailable / expensive features)
Interpretability
• A small increase in accuracy might not be
worth loss of interpretability

11. Take-home 11
Main issues
• Data uncertainty: unreliable labels, noisy features, biased missing values
➝ Take the time to explore, know your data
• Lack of data
➝ Use domain knowledge, pool different data bases, don’t over-fit
• Usability and need for interpretability
➝ It is not only about having the biggest accuracy !

12. References 12
Ansart, Manon, Stéphane Epelbaum, Geoffroy Gagliardi, Olivier Colliot, Didier Dormont, Bruno Dubois,
Harald Hampel, Stanley Durrleman, and for the Alzheimer’s Disease Neuroimaging Initiative* and the INSIGHT-
preAD study. 2019. “Reduction of Recruitment Costs in Preclinical AD Trials: Validation of Automatic Pre-
Screening Algorithm for Brain Amyloidosis.” Statistical Methods in Medical Research, January,
096228021882303. https://doi.org/10.1177/0962280218823036.
Ansart, Manon, Stéphane Epelbaum, Giulia Bassignana, Alexandre Bône, Simona Botani, Tiziana Cattai,
Raphaël Couronné, et al. 2019. “Predicting the Progression of Mild Cognitive Impairment Using Machine
Learning : A Systematic and Quantitative Review.” 49. (to be published)
Beach, Thomas G., Sarah E. Monsell, Leslie E. Phillips, and Walter Kukull. 2012. “Accuracy of the Clinical
Diagnosis of Alzheimer Disease at National Institute on Aging Alzheimer’s Disease Centers, 2005–2010.”
Journal of Neuropathology and Experimental Neurology 71 (4): 266–73.
https://doi.org/10.1097/NEN.0b013e31824b211b.
Koval, Igor, Stéphanie Allassonnière, and Stanley Durrleman. 2019. “Simulation of Virtual Cohorts Increases
Predictive Accuracy of Cognitive Decline in MCI Subjects.” ArXiv:1904.02921 [Cs, Stat], April.
http://arxiv.org/abs/1904.02921.
Junhao Wen, Elina Thibeau-Sutre, Jorge Samper- González, Alexandre Routier, Simona Bottani, Stanley
Durrleman, Ninon Burgos, Olivier Colliot. 2019. “Convolutional Neural Networks for Classification of
Alzheimer's Disease: Overview and Reproducible Evaluation.”