Maarten van Smeden discusses the role of machine learning (ML) and artificial intelligence (AI) in medicine, providing three illustrative examples: predicting mortality in patients with coronary artery disease, analyzing lymph node metastases in breast cancer, and clustering types of diabetes. He highlights the frequent confusion around what constitutes ML/AI, the misapplication of statistical principles in medical contexts, and the importance of involving statisticians in the evaluation of these technologies. The document concludes with a call for realistic simulations and collaboration between statisticians and computer scientists to improve the medical application of novel algorithms.

1. Thoughts on
Machine Learning and Artificial Intelligence
Maarten van Smeden, PhD
Leiden University Medical Center, Netherlands
STRATOS Lorenz Meeting
21/09/2018

2. Interested reader perspective
• Statistician by training
• Limited experience applying machine learning techniques
• Three examples that I think are illustrative for ML/AI in medicine
as it is applied nowadays
• Focus: prediction

3. Tech company business model

5. Apple Watch 4

6. FDA Approval
https://www.statnews.com/2018/09/13/heres-the-data-behind-the-new-apple-watch-ekg-app/?mc_cid=0fbfd65c13&mc_eid=75f1d5aea2

8. Impressive artificial intelligence
IBM Watson win against 2 Jeopardy’s champions in 2011

9. Reviewer #2

10. Less impressive artificial intelligence

11. Warning!
Statistical policing going on

12. Yesterday’s news

13. http://www.timvanderzee.com/the-wansink-dossier-an-overview/

14. Example 1: ML predicting mortality
• Caliber dataset (UK, EHR)
• N = 80,000 pre-existing coronary artery disease
• Predict all cause mortality (18,000 events, time horizon unclear)
• “used Cox models, random forests and elastic net regression”
• 586 candidate predictors vs 27 pre-selected variables
• Complete case / multiple imputation / missing indicator method
• Cox models: linear main effects only
• Split sample (1/3 test, 2/3 training)

15. Example 1: ML predicting mortality

16. Example 1: ML predicting mortality

17. Example 1: ML predicting mortality

18. One take
Linear regression is an example of
Machine Learning?
If so, what isn’t Machine Learning?

19. Perhaps more reasonable?
Beam & Kohane, JAMA, 2018

20. Example 2: lymph node metastases

21. Example 2: lymph node metastases

22. Example 2: lymph node metastases
• Researcher challenge competition
• Whole slide images of women diagnosed with breast cancer
• Training data: N = 270 (110 events); test data: N = 129 (49 events)
• 11 pathologists evaluating the test data
• 390 teams signed up for the competition
• 23 teams submitted 32 algorithms for evaluation

23. Example 2: lymph node metastases

24. Example 2: lymph node metastases
• Unfair comparison between pathologists and DL
• Pathologists no access to regularly available diagnostics
• AUC comparison DL (continuous) vs pathologists (5-item
scale)
• Promising algorithms overrepresented (390 teams -> 32
algorithms submitted)

25. Example 2: lymph node metastases
• No attention to risk prediction / calibration
• ML: attention classification only without probability
• Hugh (often implicit) difference between the traditional (risk)
prediction modeling in medicine and (traditional ML)
• Probably fine for Netflix recommendations; not so much for
real life medical decision making

26. Misuse of “risk"

27. Example 3: 5 types of diabetes

28. Example 3: 5 types of diabetes

29. Example 3: 5 types of diabetes
• Patients with newly diagnosed diabetes (N = 8980)
• 6 continuous variables
• K-means clustering (‘unsupervised learning’)

30. Example 3: 5 types of diabetes

31. Example 3: 5 types of diabetes

32. BS detection simulation
• Data generated from 2 independent MVN-distributions with .3 equal pairwise correlations
• “Sunday morning simulations”, code: https://github.com/MvanSmeden/DiabetesClusters

33. K-means clustering
“K-means finds a Voronoi partition, only if that partition coincides with a
"clustering" does it have a hope of actually doing clustering”
Max Little: https://twitter.com/MaxALittle/status/970277900871262213

34. Freak examples?
Probably?
Maybe?

35. What I observe is:
• Confusion and disagreement about what is and isn’t ML/AI
• Analyses labeled “ML/AI” have a tendency to concentrate on
classification (exceptions exist, e.g. high dimensional PS
approaches suggested that are called “ML”)
• Analyses labeled “ML/AI” in medicine are surprisingly often
done by people not thoroughly trained in statistics
• Basic statistical principles are often forgotten or ignored (e.g.
improper scoring rules)

36. Concluding remarks (1)
• Just because an algorithm is novel or flexible doesn’t mean it is
any good, obviously
• Dismissing the potential value of novel “ML/AI” algorithms out-
of-hand doesn’t make sense
• We need more realistic simulations and many applications to
compare the traditional vs more novel / flexible algorithms
• The primary issue in medical applications seems to be with the
modelers not so much with the models

37. Concluding remarks (2)
• Statisticians should be more involved in the application and
evaluation of novel / flexible algorithms, especially for risk
prediction
• Statisticians should be involved in studying performance of
novel / flexible algorithms (e.g. data hungriness) -> realistic
simulation studies
• Collaboration with computer scientists
• Computationally intensive -> may not be cheap
• Serious experimental design and reporting

38. Simulation is…
“…it is using simulation for multiplication that I find objectionable. Eight patients are
eight patients and so should remain.”

39. “All the impressive achievements of
deep learning amount to just curve
fitting”
Judea Pearl