This is the deck from a presentation I gave to the Pittsburgh Industrial Statisticians Association (PISA) for their PISA23 event in a session on Artificial Intelligence and Machine Learning. The deck itself is not intended to be stand alone without the accompanying verbal presentation, however many of the slides contain key elements with references, and my contact information is available at the end if anyone has questions.

1. Validation of Clinical Artificial Intelligence
Where We Are and Where We Are Going
Delivered by:
Sean Manion, PhD
Chief Scientific Officer @ Equideum Health
14 May 2023
Prepared for PISA 2023
1

2. “We were using artificial intelligence
before [the 1956 Dartmouth Workshop],
we just called it Operations Research”
- Herb Simon, 1974 interview
2

3. TOP 10 IDEAS IN STATISTICS THAT HAVE POWERED THE AI
REVOLUTION
 1. Hirotugu Akaike (1973). Information Theory and an
Extension of the Maximum Likelihood Principle.
Proceedings of the Second International Symposium on
Information Theory.
 2. John Tukey (1977). Exploratory Data Analysis.
 3. Grace Wahba (1978). Improper Priors, Spline
Smoothing and the Problem of Guarding Against Model
Errors in Regression. Journal of the Royal Statistical
Society.
 4. Bradley Efron (1979). Bootstrap Methods: Another Look
at the Jackknife. Annals of Statistics.
 5. Alan Gelfand and Adrian Smith (1990). Sampling-based
Approaches to Calculating Marginal Densities. Journal of
the American Statistical Association.
 6. Guido Imbens and Joshua Angrist (1994). Identification
and Estimation of Local Average Treatment Effects.
Econometrica.
 7. Robert Tibshirani (1996). Regression Shrinkage and
Selection Via the Lasso. Journal of the Royal Statistical
Society.
 8. Leland Wilkinson (1999). The Grammar of Graphics.
 9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and
Yoshua Bengio (2014). Generative Adversarial Networks.
Proceedings of the International Conference on Neural
Information Processing Systems.
 10. Yoshua Bengio, Yann LeCun, and Geoffrey Hinton
(2015). Deep Learning. Nature.
KIM MARTINEAU, COLUMBIA NEWS, 06 JUL 2021; CURATED FROM ANDREW GELMAN & AKI VEHTARI (2021) WHAT ARE THE MOST IMPORTANT STATISTICAL IDEAS OF THE PAST 50 YEARS?,
JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION, 116:536, 2087-2097, DOI: 10.1080/01621459.2021.1938081
3

4. 4

5. HOW GOOD IS HEALTH AI IN A CLINICAL SETTING?
March 18, 2019
Reducing Sepsis Mortality by One-Fifth with
Epic
A predictive model helps clinicians provide faster
sepsis treatment (Epic)
“A predictive model in Epic helps clinicians at Lee
Health identify patterns that indicate adult patients in
the early stages of sepsis.”
June 21, 2021
The Epic Sepsis Model Falls Short—The
Importance of External Validation (JAMA)
“Despite generating alerts on 18% of all patients,
the ESM did not detect sepsis in 67% of patients
with sepsis.”
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH - SEAN.MANION@EQUIDEUM.COM 5

6. YEAH, BUT THAT EPIC/SEPSIS SITUATION IS JUST ONE CASE? RIGHT?
15 March 2021
Common pitfalls and recommendations for
using machine learning to detect and
prognosticate for COVID-19 using chest
radiographs and CT scans (Nature Machine
Intelligence)
“Our search identified 2,212 studies, of which 415
were included after initial screening and, after
quality screening, 62 studies were included in this
systematic review. Our review finds that none of
the models identified are of potential clinical
use due to methodological flaws and/or underlying
biases.”
Aug. 16, 2021
At a leading health tech conference, enthusiasm for
machine learning mixes with calls for greater
scrutiny (Stat)
“Throughout the four-day conference [HIMSS], the
largest annual event in health care technology,
industry leaders called for better ways to evaluate
the usefulness of machine learning algorithms,
audit them for bias, and put in place regulations
designed to ensure reliability, fairness, and
transparency.”
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH - SEAN.MANION@EQUIDEUM.COM 6

7. Generalizability types
de Hond, A.A.H., Shah, V.B., Kant, I.M.J. et al. Perspectives on validation of clinical predictive algorithms. npj Digit. Med. 6, 86
(2023). https://doi.org/10.1038/s41746-023-00832-9

8. “Machine learning in intensive care medicine: ready for take-of?” Fleuren et al 2020, Intensive Care Med (2020) 46:1486–1488 https://doi.org/10.1007/s00134-020-
06045-y
Clinical Validation of
AI = 1%

9. 9
“If errors of inaccuracy and errors of hypersensitivity always
seemed to be in opposite directions, on what basis could we
make a compromise between those errors? The answer was
that we could make such a compromise only on a statistical
basis.”
- Norbert Wiener (w/ Julian Bigelow), “Extrapolation,
Interpolation and Smoothing of Stationary Time Series”
(1942/1949) regarding their Anti-Aircraft Predictor

10. Illustration of varying degrees of relevance for protocol items across common
study types
Ciolino, J. D., Spino,
C., Ambrosius, W. T.,
Khalatbari, S.,
Cayetano, S. M.,
Lapidus, J. A., Nietert,
P. J., Oster, R. A.,
Perkins, S. M., Pollock,
B. H., Pomann, G. M.,
Price, L. L., Rice, T.
W., Tosteson, T. D.,
Lindsell, C. J., & Spratt,
H. (2021). Guidance for
biostatisticians on their
essential contributions
to clinical and
translational research
protocol
review. Journal of
clinical and
translational
science, 5(1), e161.

11. Is Machine
Learning
Generalizable
Research?
HHS has defined "research" as
a systematic investigation,
including research
development, testing and
evaluation, designed to develop
or contribute to generalizable
knowledge.
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH - SEAN.MANION@EQUIDEUM.COM 11

12. Belmont
Report
12
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH - SEAN.MANION@EQUIDEUM.COM

13. Institutional
Review Board
(IRB)
• Review Scientific, Medical, and
Ethical aspects of research
• Made of experts along with at least
one member whose interest is non-
scientific, and another not affiliated
with the university/trial site
• Some research is exempted, but
this generally requires an IRB rep to
determine, not the investigator
13
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH -
SEAN.MANION@EQUIDEUM.COM

14. AI-Institutional Review Board (AIRB)?
Should some medical AI be reviewed or exempted by
an IRB?
Should IRBs be utilized for expanded ethical review for
AI?
Should a parallel structure be developed?
◦ Future of Privacy Forum, “Designing an Artificial
Intelligence Research Review Committee” (2019)
◦ AI Global, “Responsible AI Guidelines: Independent
Review” (2020)
◦ Ethical Machines – Reid Blackman, Harvard
Business Review Press (2022)
14
SEAN MANION, CHIEF SCIENTIFIC OFFICER, EQUIDEUM HEALTH - SEAN.MANION@EQUIDEUM.COM

15. For Health, Blockchain is necessary
but not sufficient.
EmergingTechnologyConvergence:the EpicenterofHealthIndustryValueCreation
Trusted Privacy-in-Depth
A bundle of emerging privacy tech deployable via
blockchain networks: public, private, and hybrid.
Data Federation and Decentralized AI
• Federated analytics and learning, secure inference
• Incentivized training data sourcing with verifiable
provenance, data confidence fabrics
• Digital twins with self-sovereign AIs (avatars) in the
Metaverse, in silico research on digital twin simulations
Blockchain Networks
Public, Private, and Hybrid
Distributed ledgers
Web2 - Today’s Modern Distributed Web
Smart Contracts
Secure Automation Across
Organizational Boundaries
Optional Tokenization
Secured and Transferred,
Fungible and Non-Fungible
Decentralized Apps (“dApps”)
Web3 User Experience, Wallets,
Identities, Verifiable Credentials
Tightly-Coupled
Off-Chain
“Blockchain”
or “web3”
15
New
Software
Modalities
New
Hardware
Modalities

16. Source Data
Federated Data Architectures
Metadata
Technical
Descriptions of
Source Data
Enterprise Source Data has been exponentially
expanding for the last ~25 years.
We have already reached the cybersecurity, bioethical,
legal, and logistical extremes of data centralization
strategies, while “data hungry” AI’s continue to starve.
The growth rate is increasing from the edge with 5G and
IoT, along with our new understanding of social
determinant data as health-relevant data.
Continuous feeds of all this data will soon be under
the custody of the consumer, enabling
radical empowerment through self-sovereign AI.
Derived
Data
Analytical
Findings,
Insights
Value
Blockchain is the core enabler
of federated compute.
16
©2022 EQUIDEUM HEALTH | ALL RIGHTS RESERVED.
Blockchain makes it possible to bring the compute
to the data, vs. the data to the compute… in a
trusted, scalable, and secure manner.
It is no longer necessary to centralize
Source Data to create Derived Data.
Enterprise and Patient-Sovereign

17. 17
©2022 Equideum Health | All rights reserved.
Additional Information:
https://equideum.health/
sean.manion@equideum.com
https://www.linkedin.com/in/sean-manionphd/