Development and evaluation of prediction models: pitfalls and solutions (Part...BenVanCalster
Slides for the statistics in practice session for the Biometrisches Kolloquium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 16 March 2021.
Part I from Maarten van Smeden: https://www.slideshare.net/MaartenvanSmeden/development-and-evaluation-of-prediction-models-pitfalls-and-solutions
Dichotomania and other challenges for the collaborating biostatisticianLaure Wynants
Conference presentation at ISCB 41 in the session
"Biostatistical inference in practice: moving beyond false
dichotomies"
A comment in Nature, signed by over 800 researchers, called for the scientific community to “retire statistical significance”. The responses included a call to halt the use of the term „statistically significant”, and changes in journal’s author guidelines. The leading discourse among statisticians is that inadequate statistical training of clinical researchers and publishing practices are to blame for the misuse of statistical testing. In this presentation, we search our collective conscience by reviewing ethical guidelines for statisticians in light of the p-value crisis, examine what this implies for us when conducting analyses in collaborative work and teaching, and whether the ATOM (accept uncertainty; be thoughtful, open and modest) principles can guide us.
Development and evaluation of prediction models: pitfalls and solutions (Part...BenVanCalster
Slides for the statistics in practice session for the Biometrisches Kolloquium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 16 March 2021.
Part I from Maarten van Smeden: https://www.slideshare.net/MaartenvanSmeden/development-and-evaluation-of-prediction-models-pitfalls-and-solutions
Dichotomania and other challenges for the collaborating biostatisticianLaure Wynants
Conference presentation at ISCB 41 in the session
"Biostatistical inference in practice: moving beyond false
dichotomies"
A comment in Nature, signed by over 800 researchers, called for the scientific community to “retire statistical significance”. The responses included a call to halt the use of the term „statistically significant”, and changes in journal’s author guidelines. The leading discourse among statisticians is that inadequate statistical training of clinical researchers and publishing practices are to blame for the misuse of statistical testing. In this presentation, we search our collective conscience by reviewing ethical guidelines for statisticians in light of the p-value crisis, examine what this implies for us when conducting analyses in collaborative work and teaching, and whether the ATOM (accept uncertainty; be thoughtful, open and modest) principles can guide us.
A comment in Nature, signed by over 800 researchers, called for a rise up against statistical significance. This was followed by a special issue of The American Statistician aimed at halting the use of the term "statistically significant", and new guidelines for statistical reporting in the New England Journal of Medicine. These slides discuss the broader context of the "p-value crisis" and alternatives for communicating the conclusions after statistical analyses.
Target audience: Medical researchers; Scientists involved in conducting or interpreting analyses and communicating the results of scientific research, as well as readers of scientific publications.
Learning objectives:
To understand the context of the reproducibility crisis in medical research.
To learn about problems with p-values and alternatives to report findings.
To understand how (not) to interpret significant and insignificant findings.
To learn how to communicate research findings in a modest, thoughtful, and transparent way.
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Ewout Steyerberg
Title"Clinical prediction models in the age of artificial intelligence and big data", presented at the Basel Biometrics Society seminar Nov 1, 2019, Basel, by Ewout Steyerberg, with substantial inout from Maarten van Smeden and Ben van Calster
How to establish and evaluate clinical prediction models - StatsworkStats Statswork
A clinical prediction model can be used in various clinical contexts, including screening for asymptomatic illness, forecasting future events such as disease, and assisting doctors in their decision-making and health education. Despite the positive effects of clinical prediction models on practice, prediction modelling is a difficult process that necessitates meticulous statistical analysis and sound clinical judgments. Statswork offers statistical services as per the requirements of the customers. When you Order statistical Services at Statswork, we promise you the following always on Time, outstanding customer support, and High-quality Subject Matter Experts.
Read More With Us: https://bit.ly/3dxn32c
Why Statswork?
Plagiarism Free | Unlimited Support | Prompt Turnaround Times | Subject Matter Expertise | Experienced Bio-statisticians & Statisticians | Statistics across Methodologies | Wide Range of Tools & Technologies Supports | Tutoring Services | 24/7 Email Support | Recommended by Universities
Contact Us:
Website: www.statswork.com
Email: info@statswork.com
United Kingdom: 44-1143520021
India: 91-4448137070
WhatsApp: 91-8754446690
Large amounts of heterogeneous medical data have become available in various healthcare organizations (payers, providers, pharmaceuticals). Those data could be an enabling resource for deriving insights for improving care delivery and reducing waste. The enormity and complexity of these datasets present great challenges in analyses and subsequent applications to a practical clinical environment. More details are available here http://dmkd.cs.wayne.edu/TUTORIAL/Healthcare/
Development and evaluation of prediction models: pitfalls and solutionsMaarten van Smeden
Slides for the statistics in practice session for the Biometrisches Kolloqium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 18 March 2021
Workshop on "Building Successful Pipelines for Predictive Analytics in Healthcare" delivered by Danielle Belgrave, PhD, Researcher at Microsoft Research, Cambridge, UK.
How to establish and evaluate clinical prediction models - StatsworkStats Statswork
A clinical prediction model can be used in various clinical contexts, including screening for asymptomatic illness, forecasting future events such as disease, and assisting doctors in their decision-making and health education. Despite the positive effects of clinical prediction models on practice, prediction modeling is a difficult process that necessitates meticulous statistical analysis and sound clinical judgments. Statswork offers statistical services as per the requirements of the customers. When you Order statistical Services at Statswork, we promise you the following always on Time, outstanding customer support, and High-quality Subject Matter Experts.
Read More With Us: https://bit.ly/3dxn32c
Why Statswork?
Plagiarism Free | Unlimited Support | Prompt Turnaround Times | Subject Matter Expertise | Experienced Bio-statisticians & Statisticians | Statistics across Methodologies | Wide Range of Tools & Technologies Supports | Tutoring Services | 24/7 Email Support | Recommended by Universities
Contact Us:
Website: www.statswork.com
Email: info@statswork.com
United Kingdom: 44-1143520021
India: 91-4448137070
WhatsApp: 91-8754446690
Validation of Clinical Artificial Intelligence: Where We Are and Where We Are...Sean Manion PhD
This is the deck from a presentation I gave to the Pittsburgh Industrial Statisticians Association (PISA) for their PISA23 event in a session on Artificial Intelligence and Machine Learning.
The deck itself is not intended to be stand alone without the accompanying verbal presentation, however many of the slides contain key elements with references, and my contact information is available at the end if anyone has questions.
Lessons from COVID-19: How Are Data Science and AI Changing Future Biomedical...Jake Chen
: COVID-19 has profoundly impacted all our lives. Not all such impacts in science are negative. For example, how we adapt to online learning, remote mentorship, and online teamwork may become new “norms” of future scientific collaborations, breaking down institutional boundaries to communication. The COVID-19 pandemic has united the scientific community more than ever, through more than 3600 clinical trials, 60,000 peer-reviewed publications, 80,000 SARS-CoV-2 genome sequences, 100,000 COVID-19 open software tools, and a global community of scientists, with which all of us are working hard to find epidemiological patterns, diagnosis, therapeutics, and vaccines in a “War Against COVID-19”. In this talk, I will define and characterize data-driven medicine primarily through my personal journey in the past ten months, having witnessed the rapid “weaponizing of data science tools” in our community’s fight against COVID-19 (including ours, at http://covid19.ubrite.org/). I will review up-to-date COVID-19 literature, especially those related to how biomedical informatics, data science, and artificial intelligence have been applied in accelerating COVID-19 breakthrough discoveries, from basic research to clinical practice. I will end by sharing my thoughts on how the future of medicine in cancer and other translational areas can benefit from the proactive incorporation of new “data science engines.”
A comment in Nature, signed by over 800 researchers, called for a rise up against statistical significance. This was followed by a special issue of The American Statistician aimed at halting the use of the term "statistically significant", and new guidelines for statistical reporting in the New England Journal of Medicine. These slides discuss the broader context of the "p-value crisis" and alternatives for communicating the conclusions after statistical analyses.
Target audience: Medical researchers; Scientists involved in conducting or interpreting analyses and communicating the results of scientific research, as well as readers of scientific publications.
Learning objectives:
To understand the context of the reproducibility crisis in medical research.
To learn about problems with p-values and alternatives to report findings.
To understand how (not) to interpret significant and insignificant findings.
To learn how to communicate research findings in a modest, thoughtful, and transparent way.
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Ewout Steyerberg
Title"Clinical prediction models in the age of artificial intelligence and big data", presented at the Basel Biometrics Society seminar Nov 1, 2019, Basel, by Ewout Steyerberg, with substantial inout from Maarten van Smeden and Ben van Calster
How to establish and evaluate clinical prediction models - StatsworkStats Statswork
A clinical prediction model can be used in various clinical contexts, including screening for asymptomatic illness, forecasting future events such as disease, and assisting doctors in their decision-making and health education. Despite the positive effects of clinical prediction models on practice, prediction modelling is a difficult process that necessitates meticulous statistical analysis and sound clinical judgments. Statswork offers statistical services as per the requirements of the customers. When you Order statistical Services at Statswork, we promise you the following always on Time, outstanding customer support, and High-quality Subject Matter Experts.
Read More With Us: https://bit.ly/3dxn32c
Why Statswork?
Plagiarism Free | Unlimited Support | Prompt Turnaround Times | Subject Matter Expertise | Experienced Bio-statisticians & Statisticians | Statistics across Methodologies | Wide Range of Tools & Technologies Supports | Tutoring Services | 24/7 Email Support | Recommended by Universities
Contact Us:
Website: www.statswork.com
Email: info@statswork.com
United Kingdom: 44-1143520021
India: 91-4448137070
WhatsApp: 91-8754446690
Large amounts of heterogeneous medical data have become available in various healthcare organizations (payers, providers, pharmaceuticals). Those data could be an enabling resource for deriving insights for improving care delivery and reducing waste. The enormity and complexity of these datasets present great challenges in analyses and subsequent applications to a practical clinical environment. More details are available here http://dmkd.cs.wayne.edu/TUTORIAL/Healthcare/
Development and evaluation of prediction models: pitfalls and solutionsMaarten van Smeden
Slides for the statistics in practice session for the Biometrisches Kolloqium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 18 March 2021
Workshop on "Building Successful Pipelines for Predictive Analytics in Healthcare" delivered by Danielle Belgrave, PhD, Researcher at Microsoft Research, Cambridge, UK.
How to establish and evaluate clinical prediction models - StatsworkStats Statswork
A clinical prediction model can be used in various clinical contexts, including screening for asymptomatic illness, forecasting future events such as disease, and assisting doctors in their decision-making and health education. Despite the positive effects of clinical prediction models on practice, prediction modeling is a difficult process that necessitates meticulous statistical analysis and sound clinical judgments. Statswork offers statistical services as per the requirements of the customers. When you Order statistical Services at Statswork, we promise you the following always on Time, outstanding customer support, and High-quality Subject Matter Experts.
Read More With Us: https://bit.ly/3dxn32c
Why Statswork?
Plagiarism Free | Unlimited Support | Prompt Turnaround Times | Subject Matter Expertise | Experienced Bio-statisticians & Statisticians | Statistics across Methodologies | Wide Range of Tools & Technologies Supports | Tutoring Services | 24/7 Email Support | Recommended by Universities
Contact Us:
Website: www.statswork.com
Email: info@statswork.com
United Kingdom: 44-1143520021
India: 91-4448137070
WhatsApp: 91-8754446690
Validation of Clinical Artificial Intelligence: Where We Are and Where We Are...Sean Manion PhD
This is the deck from a presentation I gave to the Pittsburgh Industrial Statisticians Association (PISA) for their PISA23 event in a session on Artificial Intelligence and Machine Learning.
The deck itself is not intended to be stand alone without the accompanying verbal presentation, however many of the slides contain key elements with references, and my contact information is available at the end if anyone has questions.
Lessons from COVID-19: How Are Data Science and AI Changing Future Biomedical...Jake Chen
: COVID-19 has profoundly impacted all our lives. Not all such impacts in science are negative. For example, how we adapt to online learning, remote mentorship, and online teamwork may become new “norms” of future scientific collaborations, breaking down institutional boundaries to communication. The COVID-19 pandemic has united the scientific community more than ever, through more than 3600 clinical trials, 60,000 peer-reviewed publications, 80,000 SARS-CoV-2 genome sequences, 100,000 COVID-19 open software tools, and a global community of scientists, with which all of us are working hard to find epidemiological patterns, diagnosis, therapeutics, and vaccines in a “War Against COVID-19”. In this talk, I will define and characterize data-driven medicine primarily through my personal journey in the past ten months, having witnessed the rapid “weaponizing of data science tools” in our community’s fight against COVID-19 (including ours, at http://covid19.ubrite.org/). I will review up-to-date COVID-19 literature, especially those related to how biomedical informatics, data science, and artificial intelligence have been applied in accelerating COVID-19 breakthrough discoveries, from basic research to clinical practice. I will end by sharing my thoughts on how the future of medicine in cancer and other translational areas can benefit from the proactive incorporation of new “data science engines.”
Clinical Research Informatics Year-in-Review - 2023Peter Embi
Keynote presentation: Clinical Research Informatics & Implementation Science Year-in-Review, presented at the AMIA Informatics Summit, 2023. This covers major publications in CRI during latter part of 2021 through March 15, 2023.
The Randomized Controlled Trial: The Gold Standard of Clinical Science and a ...marcus evans Network
Tim Fayram, St. Jude Medical Inc. - Speaker at the marcus evans Medical Device R&D Summit Fall 2013, held in Palm Beach, FL delivered his presentation entitled The Randomized Controlled Trial: The Gold Standard of Clinical Science and a Barrier to Innovation?
Νικόλαος Κουρεντζής, Country Head Radiology-Ελλάδα, Κύπρος, Ισραήλ, Ρουμανία, Βουλγαρία, Μάλτα και Μολδαβία, Bayer
«Οι νέες προκλήσεις στην ιατρική απεικόνιση»
지난주말에 있었던 제 4회 대한신경집중치료학회 편집위원회 워크샵에서 발표했던 내용중에 발췌한 것입니다. 원래 제목은 "인공지능 관련 연구: 논문 작성과 심사에 관한 요령" 입니다. 최근에 deep learning in medical imaging으로 2편의 리뷰와 논문 1편, CADD 논문, 앙상블 논문 1편이 되면서 요청이 온것 같습니다.부족한 제가 하기 어려운 주제를 맡았는데, 혹시 도움이 되실 분이 있으면 도움을 되시라고 올려드립니다. 결론은 인공지능 연구라고 특별히 다르지는 않지만, 공학 연구와 의학연구가 다르고, 인공지능 특성을 잘 이해해야 한다 정도 될것 같습니다. (상당부분 저희병원 박성호 교수님의 radiology 논문 Methodology for Evaluation of Clinical Performance and Impact of Artificial Intelligence Technology for Medical Diagnosis and Prediction을 참고했습니다.)
Journal club and talk given to Health Data Analytics MSc, February 2023. Reflecting on how to do good machine learning over biomedical data, the pitfalls and good practices
Research in the time of Covid: Surveying impacts on Early Career ResearchersRebecca Grant
Based on a survey of over 4,500 researchers published in the white paper The State of Open Data 2020, this session will explore the impacts of the pandemic on early career reearchers (ECRs), their research practice, and how they interact with open data. We will discuss the specific challenges reported by ECRs, as well as the gaps in training and support that they have identified that would encourage their sharing and reuse of research data.
Presentation at the E-ARMA conference 2021.
Where AI will (and won't) revolutionize biomedicinePaul Agapow
Presented AI & Big Data Expo, London, December 2022.
Given the hype and success of machine learning and AI in other fields, its application in healthcare is only natural.
- However, the actual successes in medicine have been limited, with a number of high-profile failures.
- Here, I propose that biology is uniquely complex, with our lack of domain knowledge limiting the application of AI.
- However, there is reason for cautious optimism, with AI-lead approaches shifting the odds in our favour.
Deep learning in healthcare: Oppotunities and challenges with Electronic Medi...Thien Q. Tran
Interested in deep learning for healthcare has grown strongly recent years besides with the successes in other domains such as Computer Vision, Natural Language Processing, Speech Recognition and so forth. This talk will try to give a brief look into the recent effort of research in deep learning for healthcare. Especially, this talk focuses on the opportunities and challenges in using electronic health records (EHR) data, which is one of the most important data sources in healthcare domain.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
Str-AI-ght to heaven? Pitfalls for clinical decision support based on AI
1. Str-AI-ght to heaven?
Pitfalls for clinical decision support based on AI
Ben Van Calster
Department Development and Regeneration and EPI-centre, KU Leuven
Department Biomedical Data Sciences, LUMC Leiden
Research Ethics Committee, UZ Leuven
ben.vancalster@kuleuven.be; @BenVanCalster
ISUOG World Congress, 16 October 2021
2. Disclaimer
• Talk last year: “a plea for good methodology”
• This talk builds on that, in the context of AI and machine learning
• There is a lot of hype surrounding AI/ML. It may have potential, but we better
start to get real!
2
https://lawtomated.com/enough-with-the-a-i-hype-and-why/
Lawtomated
3. Do not celebrate too early…
3
Copyright Bas Czerwinski / Getty Images
Julian Alaphilippe, Liège-Bastogne-Liège (Oct. 4th, 2020)
Real winner: Primož Roglič
Real winner
4. Deep learning on medical images
4
Topol. Nat Med 2019;25:44-56. Zhu et al. Front Neurol 2019;10:869.
Titano et al. Nat Med 2018;24:1337-41; Nam et al. Radiology 2019;290:218-28; Ehteshami Bejnordi et al. JAMA 2017;318:2199-210;
Esteva et al. Nature 2017;542:115-8; De Fauw et al. Nat Med 2018;24:1342-50; Raman et al. Eye 2019;33:97-109.
5. Machine Learning for ‘EHR’ data
5
Rajkomar et al. Npj Digit Med 2018;1:18.
Rose. JAMA Netw Open 2018;1:e181404.
6. Reason for popularity?
6
“Very complex machine learning algorithms are highly flexible,
and hence find relationships we could not see before.
Therefore we make better predictions and better decisions.”
→ Guaranteed success!
Right?
7. Pitfalls for “predictive analytics”
7
1. Poor methodology
2. Lack of evidence
3. Considerable heterogeneity
4. (Financial) conflicts of interest
5. Actual implementation in clinical practice
8. 1. Methodology matters, not impact factors
8
Altman DG. BMJ 1994;308:283-284.
Van Calster et al, J Clin Epidemiol, in press.
Altman. BMJ 1994.
Our own frustration paper. JCE 2021.
9. ‘Predictive analytics’: covid-19
9
Wynants et al. BMJ 2020;369:m1328.
The review found more than 1 paper a day (!)
Results not trustworthy for 97% of the 231 models
Median sample size: 338
Non-representative sample: 42%
Representativity unclear: 25%
Data analysis problematic: 94%
No model validation at all: 22%
10. Predictive analytics for covid-19
10
Wynants et al. BMJ 2020;369:m1328
Deep learning models for covid-19 diagnosis using CT or RX
- No discussion of target population or setting
- Control group (without covid-19):
Images from pediatric population
Images from a different country
Images from different time periods
Barely defined, e.g. ‘healthy persons’
- Images from online repository, without further information
- Often not any demographic description (not even age or sex!)
12. Public covid-19 RX datasets
12
Santa Cruz et al. Med Image Analysis 2021;74:102225.
13. Complex algorithms are data hungry
So you dream of
having a Porsche?
If you cannot (or don’t want to) pay for it,
you may get this...
This also holds for predictive analytics. More fancy model? More expensive.
Currency: GOOD data.
13
14. Measurement and data quality
14
Missing values: the tricky importance of the invisible
Measurement: timing and procedure matters
Outcome: quality labels are key (see e.g. deep learning on medical images)
Beam & Kohane. JAMA 2018;319:1317-1318.
15. 2. Wanted: evidence
• Kleinrouweler (AJOG 2016): 263 models in obstetrics
• Only 23 of these (9%) had been externally validated…
Other examples of model overload:
• 1060 models predicting outcomes after CVD (1990-2015) (Wessler et al, 2017)
• 363 models predicting CVD (Damen et al, 2016)
• 231 models related to Covid-19 (Wynants et al, 2020), and counting!
• 116 models to diagnose ovarian malignancy (Kaijser et al, 2014)
15
Wessler et al. Diagn Progn Res 2017;1:20. Damen et al. BMJ 2016;353:i2416. Wynants et al. BMJ 2020;369:m1328.
Kleinrouweler et al. AJOG 2016;214:79-90. Kaijser et al. Hum Reprod Update 2014;20:229-62.
16. Smartphone apps for skin lesions
16
Freeman et al. BMJ 2020;368:m127
• 9 validation studies covering 6 apps
• 1132 lesions in total (average 126 per study)
• Methodological quality was poor
o Selective inclusions (non-representative)
o Images were taken and selected by clinicians
o Lots of unusable images
Scarce and poor evidence
17. Radiology AI
17
Van Leeuwen et al. Eur Radiol 2021;31:3797-3804
• 64/100: no evidence
• 18/100: evidence of diagnostic performance
• 18/100: evidence of potential impact
• Half of the studies were independent, the other half had conflicts of interest
18. 3. Expect (a lot of) heterogeneity
18
• Changes in care over time
• Differences in care between healthcare systems
• Differences in populations between practices/hospitals/regions
• Differences in hardware, software, and measurement procedures
• Differences in performance between patient subgroups (cf fairness)
Futoma et al. Lancet Digit Health 2020;2:e489-e492.
21. Hardware/software
21
Badgeley et al. npj Digit Med 2019;2:31.
Deep learning was better at predicting scanner model and brand
(AUC>=0.98) than at predicting hip fracture (AUC 0.78)
22. Where do DL datasets come from anyway?
22
Kaushal et al. JAMA 2020;324:1212-1213.
24. DL research (Sep 2021)
24
Perkonigg et al. Nat Comm 2021;12:5678.
25. 4. Proprietary datasets and models
25
Van Calster et al. JAMIA 2019;26:1651-1654.
https://hai.stanford.edu/news/flying-dark-hospital-ai-tools-arent-well-documented.
Not necessarily bad in principle: financial resources are needed
But it may hamper openness, availability, independent validation
COVID review: companies often did not react, but claimed that the model
was used on thousands of patients
27. Google’s Dermatology Assist (CE label)
27
https://www.statnews.com/2021/06/02/machine-learning-ai-methodology-research-flaws/.
Roxana Daneshjou (Stanford):
- No evaluation on external dataset.
- Insufficient variation in skin types.
- Outcome rarely based on biopsy.
- “I haven't seen data that makes me feel
comfortable with putting this in the hands of
patients or physicians.”
28. External validation of EPIC sepsis model
28
Wong et al. JAMA Intern Med 2021;181:1065-1070.
Model: penalized logistic regression with 80 variables
Data: 3 healthcare organizations, 2013-2015
AUC according to internal documentation: 0.78-0.83
Validation: 1 academic center, 2018-2019
AUC 0.63, calibration poor (risks way too high)
29. 5. Actual implementation
29
Logistical/practical issues to fit model in clinical workflow
Psychological issues regarding model use by healthcare staff
Medicolegal: Who is responsible when prediction is wrong?
https://www.statnews.com/2020/03/09/can-you-sue-artificial-intelligence-algorithm-for-malpractice/
Panch et al. npj Digit Med 2019;2:77.
30. Lack of evidence revisited: impact?
30
Clinical impact studies: scarce, difficult
Clinical decision support is a complex intervention (Kappen et al, 2018)
Endpoints of impact studies?
- Process-related: ‘easy’, but intermediate
- Long-term patient outcomes: difficult, lower effect sizes expected
Kappen et al. Diagn Progn Res 2018.
31. So, does medical AI ‘work’?
31
We still often don’t know!
Trust jeopardized by
- poor methodology
- lack of evidence
- lack of openness.
It may have potential if done well and evidence is gathered.
AI community / academia often shoots itself in the foot, this is a pity
Academia: wrong incentives (publish or perish)!
Companies: financial conflicts of interest!
32. That’s (not) all folks…
32
https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/.
https://spectrum.ieee.org/deep-learning-computational-cost
Thompson et al. IEEE Spectrum 2021.
Hao. MlT Technology review 2019.