The trustworthiness of risk estimates may suffer from substantial model uncertainty and heterogeneity, limiting the practical applicability of prediction models. This presentation examines to what extent AI and machine learning approaches can improve predictions for patients and care-givers. The presentations covers 3 main topics: 1. What do we need for individual patients? I discuss internal vs external validity; and performance measures such as calibration vs discrimination 2. Trustworthy processes to build a prediction model Can AI help, and what do humans achieve? 3. Types of uncertainty I discuss some statistical aspects, model uncertainty, and heterogeneity between contexts of application

1. March 2024, Aachen
Validation and Trustworthiness of AI based
Predictions
Ewout W. Steyerberg, PhD
Professor of Clinical Biostatistics and
Medical Decision Making
Dept of Biomedical Data Sciences
Leiden University Medical Center
Thanks to many for assistance and inspiration

2. Prediction
“10% risk”

3. Clinical prediction models
25-Mar-24
3 www.cancercalculators.org

4. 25-Mar-24
4 Insert > Header & footer

5. Validated  Trustworthy?
25-Mar-24
5 Insert > Header & footer

6. Data preparation
Veldnorm Medische AI
6
Predictive algorithms: Medical AI
Phase 1
Development AI
algorithm
Phase 2
Validation AI
algorithm
Phase 3
Software
environment
Phase 4
Impact
assessment
Phase 5
Implementation in
medical practice
Phase 6
1 2 3 4 5 6

7. Trustworthiness of predictions
Mathematical models needed:
• Complex processes
• No simple prediction via a deterministic theory
Modelling assumptions:
• Generally false
• Intelligent guesswork
Medical prediction: y ~ X

8. Topics: trustworthy predictions
1. What do we need for individual patients?
• Internal vs external validity
• Calibration vs discrimination
2. Trustworthy processes to build a prediction model?
• AI
• Humans
• Requirements
3. Types of uncertainty
• Statistical aspects
• Model uncertainty
• Heterogeneity between contexts of practical application

9. Validated = trustworthy?
• Classic:
• No validation, only internal  low ranking journal
• 1 convincing validation  top journal
• Modern
• Substantial heterogeneity in performance
 There is no such thing as a validated model

10. Example in neurotrauma, external validation
25-Mar-24
10 Insert > Header & footer
Modification of val.prob() in rms; val.prob.ci.2()
Steyerberg et al, PLoS Med 2008
well calibrated over prediction

11. Very heterogenous validations
25-Mar-24
11 Insert > Header & footer

12. Perspective at validation
25-Mar-24
12 Insert > Header & footer

13. Summary validation
• Internal: minimum, same underlying population
• External:
• Temporal
• Geographic
• Domain
• Efficient: internal-external cross-validation

14. Performance assessment
• What is the most commonly reported measure for performance of
prediction models?
• Area under the Receiver Operating Characteristic curve (AUC), or
concordance (c) statistic
• Discrimination = spread of predictions between individuals
• Higher if better predictors in model
• Higher --> more trustworthy??

15. 25-Mar-24
15 Insert > Header & footer

16. Trustworthiness for individuals
• Calibration = reliability of predictions per individual

17. 25-Mar-24
17 Insert > Header & footer
Modification of val.prob() in rms; val.prob.ci.2()
Steyerberg et al, PLoS Med 2008
well calibrated

18. Trustworthiness for individuals
• Calibration = reliability of predictions per individual
• True risk estimates UTOPIAN
• Calibration underreported

19. Topics: trustworthy predictions
1. What do we need for individual patients?
• Internal vs external validity
• Calibration vs discrimination
2. Trustworthy processes to build a prediction model?
• AI
• Humans
3. Types of uncertainty
• Statistical aspects
• Model uncertainty
• Heterogeneity between contexts of practical application

20. Trustworthiness of ChatGPT
ChatGPT: may be hallucinating
• Simple calculations:

21. Trustworthiness and AI
• Relation to evidence base
• Other popular terms related to AI
• Fairness
• Equity
 Let’s ask ChatGPT

25. Trustworthy models?
• Modeling flexibility: friend or foe?
Human oversight on:
• Classical modeling: selection of predictors; nonlinearity; interactions
• AI: hyperparameters; technique CART / RF / XGBoost / nnet / ..

27. Trustworthiness: poor for human modelers
Red cards and dark skin soccer players
https://psyarxiv.com/qkwst/
25-Mar-24
27 Insert > Header & footer

28. • 29 teams involving 61 analysts; same dataset; same research question:
whether soccer referees are more likely to give red cards to dark skin
toned players than light skin toned players
• Estimated odds ratios 0.89 –2.93 (median 1.3)
• 20 teams: statistically significant positive effect, 9: non-significant relation
25-Mar-24
28 Insert > Header & footer

29. Estimated odds ratios by 29 research teams
25-Mar-24
29 Insert > Header & footer

30. “Logistic regression”
25-Mar-24
30 Insert > Header & footer

31. Claimed trust in results
25-Mar-24
31 Insert > Header & footer

32. Trustworthiness: poor for human modelers
• 29 teams involving 61 analysts; same dataset; same research question:
whether soccer referees are more likely to give red cards to dark skin toned
players than light skin toned players
• Estimated odds ratios 0.89 –2.93 (median 1.3).
• 20 teams: statistically significant positive effect, 9: non-significant relation.
• 21 unique combinations of covariates
• “Variation in analysis of complex data may be difficult to
avoid, even by experts with honest intentions”
25-Mar-24
32 Insert > Header & footer

33. Explainable to humans = trustworthy?
• Explainable AI
• Algorithm trustworthy: if predictions are based on factors that are
acceptable to domain experts instead of on ‘spurious correlations’
• SHAP (SHapley Additive exPlanations) values
“By using SHAP values, researchers and practitioners can gain a
deeper understanding of how different features influence model
predictions, leading to improved model interpretability and trust.”
ChatGPT3.5

34. Topics: trustworthy predictions
1. What do we need for individual patients?
• Internal vs external validity
• Calibration vs discrimination
2. Trustworthy processes to build a prediction model?
• AI
• Humans
3. Types of uncertainty
• Statistical aspects
• Model uncertainty
• Heterogeneity between contexts of practical application

35. Approaches to uncertainty quantification
• Sample size
• specifically #events for binary outcome prediction
• ‘patients like you’ and exceptionality
• For risk communication: aleatory uncertainty
• For uncertainty communication: epistemic

36. Example on presentation by ‘the king of nomograms’
25-Mar-24
36 Insert > Header & footer

37. 25-Mar-24
37 Insert > Header & footer

38. N = 100?
25-Mar-24
38 Insert > Header & footer
GUSTO data, n=1200
(out of 40,830)

39. Uncertainty for rare, strong predictor: SHOCK
25-Mar-24
39 Insert > Header & footer

40. Predictions with and without SHOCK in the model
25-Mar-24
40 Insert > Header & footer

41. AI: arXiv paper
“Trustworthiness expresses whether a prediction is aligned with the
train data”
• Distance between the new patient and similar patients from the
training data estimates the trustworthiness of a prediction;
resembles the reference data
• Prediction for the new patient is close to the ground truth of the
neighbors in the training set.

42. Claim
”Effective N is an attractive concept to address epistemiologic uncertainty”
• Analytic solutions for regression models
• Minimum certainty, say, n>10, for model specification: selection / shrinkage?
• Approximate solutions for machine learning models
• Bootstrap
• Effective N: conditional on the model
25-Mar-24
42 Insert > Header & footer

43. Model uncertainty
• 246 biologists modeling
• 61 analysts in 29 teams on the Red Card problem
• …
• Comparisons between classic vs machine learning

44. Systematic review
25-Mar-24
44 Insert > Header & footer

46. Differences in discrimination

47. Topics: trustworthy predictions
1. What do we need for individual patients?
• Internal vs external validity
• Calibration vs discrimination
2. Trustworthy processes to build a prediction model?
• AI
• Humans
3. Types of uncertainty
• Statistical aspects
• Model uncertainty
• Heterogeneity between contexts of practical application

48. Heterogeneity
• Study design
• Selection of subjects
• Disease domain
• Measurement of covariates
• Measurement of outcomes
• Associations of covariates with outcome
• Overall outcome rates

49. Heterogeneity in performance
Steyerberg et al, PLoS Med 2008
Dijkland S et al; J Neurotrauma 2019

50. 25-Mar-24
50 Insert > Header & footer

51. 15 cohorts: 11 RCTs, 4 Observational studies
25-Mar-24
51 Insert > Header & footer

52. Heterogeneity in case-mix
25-Mar-24
52 Insert > Header & footer

53. Heterogeneity in predictor effects
25-Mar-24
53 Insert > Header & footer

54. Heterogeneity in predictions
25-Mar-24
54 Insert > Header & footer

55. Heterogeneity in individual predictions
25-Mar-24
55 Insert > Header & footer

56. Conclusions on trustworthy predictions
• Epistemic uncertainty: under the influence of the modeler
• Larger sample sizes
• Modest modeling, limit flexibility
• Heterogeneity: assess differences between settings
• Study design
• Distribution and effects of covariates
• Differences between predictions
• Model predictions suffer from multiple sources of uncertainty
• Transparency: for policy makers / physicians / patients
• Context dependency: Local versus global models
25-Mar-24
56 Insert > Header & footer