The document discusses the application of medical AI, including examples of research and usage. It describes how medical AI can be used for classification, object detection, and segmentation of medical images like fundus images, endoscopic images, CT/MRI scans, and X-rays. It also discusses how AI can be applied to biological signals like ECG. Key metrics for medical AI like sensitivity and specificity are explained. Challenges in medical AI like data handling, domain knowledge requirements, and ensuring interpretability are also covered.
2. 2021 AMI Inc. ALL rights reserved.
2
About this documents
• This document has been revised based on the material presented at NeurIPS Meetup Japan
2021 .
3. 2021 AMI Inc. ALL rights reserved.
3
AMI Inc.
• AMI Inc.
• https://ami.inc/
• We’re developing Super stethoscope
• Stethoscope with remote medical diagnosis support function
stethoscope
Super stethoscope
Heart disease
AI
ECG
heartsound
4. 2021 AMI Inc. ALL rights reserved.
4
Agenda
• Usage of Medical AI
• Application of Medical AI
• Metric of Medical AI
• Features / Difficulties of Medical AI
• Examples of Medical AI Research and Application
5. 2021 AMI Inc. ALL rights reserved.
5
Usage of Medical AI
6. 2021 AMI Inc. ALL rights reserved.
6
Usage of Medical AI
• Usage of Medical AI
• Classification
• Object Detection
• Segmentation
• etc …
7. 2021 AMI Inc. ALL rights reserved.
7
Usage of Medical AI
• Classification
This Image from
https://www.kaggle.com/ratthachat/aptos-eye-preprocessing-in-diabetic-retinopathy
Eyes
(Detecting Diabetic Retinopathy)
This Image from
https://arxiv.org/abs/1612.00542
Breast
(Breast Mass Classification
from Mammograms)
This Image from
https://www.nature.com/articles/s41598-019-44839-3#Sec14
(Supplementary information)
Teeth
(Detection of Periodontal Bone Loss)
This Image from
A Deep Learning Model to Predict a Diagnosis of Alzheimer Disease by Using 18F-FDG PET of the Brain
Brain
(Alzheimer Disease )
This Image from
Artificial Intelligence-Assisted Polyp Detection for Colonoscopy: Initial Experience
Polyp
8. 2021 AMI Inc. ALL rights reserved.
8
Usage of Medical AI
• Object Detection
These Images from
A deep learning approach to automatic teeth detection and numbering based on object detection in dental periapical films
Automatic teeth detection and numbering
An Efficient Approach for Polyps Detection in Endoscopic Videos Based on Faster R-CNN
These Images from
Polyp detection
9. 2021 AMI Inc. ALL rights reserved.
9
Usage of Medical AI
• Segmentation
These Images from
Deep instance segmentation of teeth in panoramic X-ray images
Teeth segmentation
These Images from
Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery
Instrument segmentation
These Images from
Surface Muscle Segmentation Using 3D U-Net Based on
Selective Voxel Patch Generation in Whole-Body CT Images
Muscle / Bone Segmentation
10. 2021 AMI Inc. ALL rights reserved.
10
Application of Medical AI
11. 2021 AMI Inc. ALL rights reserved.
11
Application of Medical AI
• Application of Medical AI
• Image
• fundus image (eye)
• endoscopic image (cancer)
• CT/ MRI (aneurysm/ cancer)
• X-ray (bone)
• Biological signal
• ECG
12. 2021 AMI Inc. ALL rights reserved.
12
Image : fundus image (eye)
• Improved Automated Detection of Diabetic Retinopathy on a Publicly Available Dataset Through
Integration of Deep Learning
• Input : fundus images
• Output : diabetic retinopathy
• Model : CNN inspired by Alexnet23 (for more limited training sets) and the Oxford Visual
Geometry Group26 (for more extensive training sets) network architectures
• Result
• Sensitivity : 96.8%, Specificity :87.0%
• Related Company : IDx-DR
13. 2021 AMI Inc. ALL rights reserved.
13
Image : endoscopic image (cancer)
• Application of artificial intelligence using a convolutional neural network for detecting gastric
cancer in endoscopic images
• Input : endoscopic images
• Output : gastric cancer
• Model : Single Shot MultiBox Detector
• Result
• Sensitivity : 92.2%
• Related Company : AI Medical Service Inc.
14. 2021 AMI Inc. ALL rights reserved.
14
Image : CT/ MRI (aneurysm/ cancer)
• Deep Learning for MR Angiography: Automated Detection of Cerebral Aneurysms
• Input : MR images
• Output : aneurysm
• Model : ResNet18
• Result (top five candidate points with the highest probability of representing an aneurysm)
• Sensitivity : 93%
• Related Company : LPIXEL
15. 2021 AMI Inc. ALL rights reserved.
15
Image : CT/ MRI (aneurysm/ cancer)
• COVID-19 Pneumonia Image Analysis Program
• Input : CT images
• Output : the confidence level of COVID-19 pneumonia
• Result
• AUC : 0.780, Sensitivity : 89.6%, Specificity : 37.1%
• Related Company : MIC Medical, Alibaba Damo Technology, M3
https://corporate.m3.com/press_release/2020/20200629_001616.html
https://www.pmda.go.jp/files/000235943.pdf
16. 2021 AMI Inc. ALL rights reserved.
16
Image : CT/ MRI (aneurysm/ cancer)
• Lung nodule detection program
• Input : CT images
• Output : Pulmonary nodules
• Pulmonary nodules are whitish shadows in the lung area on X-ray or CT images, and if they are
seen, they may indicate lung cancer or other diseases.
• Result
• Sensitivity : 61.4% (doctor + program)
• Related Company : FUJIFILM
https://www.pmda.go.jp/PmdaSearch/kikiDetail/ResultDataSetPDF/671001_30200BZX00150000_A_01_03
https://www.fujifilm.com/jp/ja/news/list/4963
17. 2021 AMI Inc. ALL rights reserved.
17
Image : CT/ MRI (aneurysm/ cancer)
• Cancer detection in screening mammography
• Input : mammography images
• Output : cancer (positive / normal)
• Result
• AUC : 0.95, Sensitivity : 86.9%, Specificity :88.5%
• Related Company : CureMetrix
Image from
https://pubs.rsna.org/doi/pdf/10.1148/radiol.2019182908
https://www.accessdata.fda.gov/cdrh_docs/pdf18/K183285.pdf
18. 2021 AMI Inc. ALL rights reserved.
18
Image : CT/ MRI (aneurysm/ cancer)
• Denoised PET Images
• Input : PET Images
• Output : Denoised Images
• Related Company : SUBTLE MEDICAL
https://subtlemedical.com/usa/subtlepet/
https://www.accessdata.fda.gov/cdrh_docs/pdf18/K182336.pdf
19. 2021 AMI Inc. ALL rights reserved.
19
Image : X-ray (bone)
• Deep neural network improves fracture detection by clinicians
• Input : X-ray images
• Output : Fracture
• Model : 2D CNN
• Result
• Sensitivity : 91.5%, Specificity :93.9%
• Related Company : Imagen Technologies
20. 2021 AMI Inc. ALL rights reserved.
20
Biological signal : ECG
• Deep Neural Networks Can Predict New-Onset Atrial Fibrillation From the 12-Lead ECG and
Help Identify Those at Risk of Atrial Fibrillation–Related Stroke
• Input : 12-lead digital ECG wave data + Age, Sex
• 3 branch from 12-lead ECG
• I, II, V1, and V5, acquired from time (t) = 0 (start of data acquisition) to t=5 [s]
• V1, V2, V3, II, and V5 from t=5 to t=7.5 [s]
• V4, V5, V6, II, and V1 from t=7.5 to t=10 [s]
• Output : Risk of Atrial Fibrillation or not (1 year later)
• Model : 1D CNN
• Result
• AUC : 0.85
• Sensitivity : 69%, Specificity :81%
• Related Company : Tempus
3 branches from 12-lead ECG
21. 2021 AMI Inc. ALL rights reserved.
21
Metric of Medical AI
22. 2021 AMI Inc. ALL rights reserved.
22
Metric of Medical AI
• Main metric of Medical AI
• Sensitivity (Recall) : Ratio of people who have findings in people of disease
• High Sensitivity : useful for screening.
• If this test is negative, the probability of having the disease is very small.
• Specificity : Ratio of people who have findings in people of no disease
• High Specificity : useful for definitive diagnosis
• If this test is positive, the probability of having the disease are very high.
• AUROC
𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑁
𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 =
𝑇𝑁
𝐹𝑃 + 𝑇𝑁
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
23. 2021 AMI Inc. ALL rights reserved.
23
Metric of Medical AI
• High Sensitivity
• FN is few
• If test (prediction) is negative, no disease
• Whether a person with a disease be diagnosed as having a disease or not
• Effective when test results are negative, useful for screening.
• Test which High Sensitivity
• If this test is negative, the probability of having the disease is very small.
• e.g. High Sensitivity CRP
• as a marker for detecting micro-inflammation in atherosclerosis,
• It is used to predict the risk of myocardial infarction.
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
24. 2021 AMI Inc. ALL rights reserved.
24
Metric of Medical AI
• High Specificity
• FP is few
• If test (prediction) is positive, disease
• Whether properly diagnose a person as healthy if they have no diseases
• Effective when test results are positive, useful for definitive diagnosis
• High Specificity
• If that test is positive, the probability having that disease are very high.
• e.g.
• PCR test for COVID-19,
• Pregnancy test
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
25. 2021 AMI Inc. ALL rights reserved.
25
Metric of Medical AI
• Why Sensitivity and Specificity are used.
• Disease rate (prevalence) can be small. (Imbalanced data)
• PPV and NPV depend on disease rate
• If disease rate is low, PPV become small.
• FP is increased -> This is problem when screening for rare disease
• (If specificity is high, PPV become high)
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
positive negative
positive
negative
Test (Prediction)
Disease
(Ground Truth)
TP FN
FP TN
𝑃𝑃𝑉 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑃
Small Disease rate (prevalence)
26. 2021 AMI Inc. ALL rights reserved.
26
Metric of Medical AI
• AUROC
1-Specificity
Sensitivity
positive negative
positive
negative
AUROC
27. 2021 AMI Inc. ALL rights reserved.
27
Features / Difficulties of
Medical AI
28. 2021 AMI Inc. ALL rights reserved.
28
Features / Difficulties of Medical AI
• Data handling
• Data Collection / Normalization
• Data Feature
• Annotation
• How to split data
• For application
• Domain Knowledge
• Workflow to release
• Black box / Interpretability
• Domain shift
29. 2021 AMI Inc. ALL rights reserved.
29
Feature / Difficulty in Medical AI
• Data Collection / Normalization
• Unification of Terminology
• Notation variety, Data format depend on the facility
• item name, unit [m or cm], value type ([0, 1, ...] or [A, B, ...])
• Unification of Terminology and preprocessing for ML is important.
Item A
[Unit_A]
Item B_0
[Unit_B_0]
[0, 1, ..]
・
・
・
Item A
[Unit_A]
Item B_1
[Unit_B_1]
[A, B, …]
・
・
・
Facility A Facility B
30. 2021 AMI Inc. ALL rights reserved.
30
Features / Difficulties of Medical AI
• Data Feature
• Imbalanced data
• Ordered data (like severity)
Disease Normal Severity : 0 Severity : 1 Severity : 2
31. 2021 AMI Inc. ALL rights reserved.
31
Features / Difficulties of Medical AI
• Annotation
• The annotation will vary depending on the doctor’s ability and the facility’s policy.
Severity : 0 Severity : 1 Severity : 2
Severity : 0 Severity : 1 Severity : 2
Facility A Facility B
32. 2021 AMI Inc. ALL rights reserved.
32
Features / Difficulties of Medical AI
• How to split data
• We need to collect data from a variety of facilities.
Train / Validation Test
Patient A Patient B Patient C Patient D Patient E
Facility A Facility B Facility C Facility D Facility E
33. 2021 AMI Inc. ALL rights reserved.
33
Features / Difficulties of Medical AI
• Domain Knowledge
• ML engineers are not familiar with Medical terminology and Medical knowledge
• Medical knowledge for feature engineering
Blood test
TP, ALB, GOT, GPT, HbA ...
Medical knowledge
for feature engineering
Input data Output
ML
How to input
Terminology
in Medical domain
34. 2021 AMI Inc. ALL rights reserved.
34
Features / Difficulties of Medical AI
• Workflow to release product medical devices
• Product with medical AI is also applied this workflow. (case in Japan)
• In the case of relearning, application is needed again.
Development
Non-clinical
test
Clinical test
Application
for approval
Commercially
available
Biological safety
Electrical Safety
Safety of Machinery
Durability Tests
Stability Tests
etc.
35. 2021 AMI Inc. ALL rights reserved.
35
Feature / Difficulty in Medical AI
• Black box / Interpretability
• Need explanation of features for approval
• It is difficult to explain the detail because of black box
• Important points for approval
• Used Data for development
• Use case for application
Data Use case
Black box in AI
Input Output
Medical device
(w/o AI)
Known features
Important points for approval
36. 2021 AMI Inc. ALL rights reserved.
36
Feature / Difficulty in Medical AI
• Domain shift
• device, protocol of measurement
Facility A Facility B
37. 2021 AMI Inc. ALL rights reserved.
37
Examples of Medical AI
Research and Application
38. 2021 AMI Inc. ALL rights reserved.
38
Examples of Medical AI Research and Application
• Examples of Medical AI Research and Application
• Images
• fundus image (eye)
• endoscopic image (cancer)
• CT/ MRI (aneurysm/ cancer)
• X-ray (bone)
• Biological signal
• ECG
• Breath sound
39. 2021 AMI Inc. ALL rights reserved.
39
Image : fundus image (eye)
40. 2021 AMI Inc. ALL rights reserved.
40
Image : fundus image (eye)
• Improved Automated Detection of Diabetic Retinopathy on a Publicly Available Dataset Through
Integration of Deep Learning
• Input : fundus images
• Output : diabetic retinopathy
• Model : CNN inspired by Alexnet23 (for more limited training sets) and the Oxford Visual
Geometry Group26 (for more extensive training sets) network architectures
• Result
• Sensitivity : 96.8%, Specificity :87.0%
• Related Company : IDx-DR
41. 2021 AMI Inc. ALL rights reserved.
41
Image : endoscopic image
(cancer)
42. 2021 AMI Inc. ALL rights reserved.
42
Image : endoscopic image (cancer)
• Application of artificial intelligence using a convolutional neural network for detecting gastric
cancer in endoscopic images
• Input : endoscopic images
• Output : gastric cancer
• Model : Single Shot MultiBox Detector
• Result
• Sensitivity : 92.2%
• Related Company : AI Medical Service Inc.
43. 2021 AMI Inc. ALL rights reserved.
43
Image : CT/ MRI (aneurysm/
cancer)
44. 2021 AMI Inc. ALL rights reserved.
44
Image : CT/ MRI (aneurysm/ cancer)
• Deep Learning for MR Angiography: Automated Detection of Cerebral Aneurysms
• Input : MR images
• Output : aneurysm
• Model : ResNet18 (untrained)
• Result (top five candidate points with the highest probability of representing an aneurysm)
• Sensitivity : 93%
• Related Company : LPIXEL
45. 2021 AMI Inc. ALL rights reserved.
45
Image : CT/ MRI (aneurysm/ cancer)
• COVID-19 Pneumonia Image Analysis Program
• Input : CT images
• Output : the confidence level of COVID-19 pneumonia
• Result
• AUC : 0.780, Sensitivity : 89.6%, Specificity : 37.1%
• Related Company : MIC Medical, Alibaba Damo Technology, M3
https://corporate.m3.com/press_release/2020/20200629_001616.html
https://www.pmda.go.jp/files/000235943.pdf
46. 2021 AMI Inc. ALL rights reserved.
46
Image : CT/ MRI (aneurysm/ cancer)
• lung nodule detection program
• Input : CT images
• Output : Pulmonary nodules
• Pulmonary nodules are whitish shadows in the lung area on X-ray or CT images, and if they are
seen, they may indicate lung cancer or other diseases.
• Result
• Sensitivity : 61.4% (doctor + program)
• Related Company : FUJIFILM
https://www.pmda.go.jp/PmdaSearch/kikiDetail/ResultDataSetPDF/671001_30200BZX00150000_A_01_03
https://www.fujifilm.com/jp/ja/news/list/4963
47. 2021 AMI Inc. ALL rights reserved.
47
Image : CT/ MRI (aneurysm/ cancer)
• Cancer detection in screening mammography
• Input : mammography images
• Output : cancer (positive / normal)
• Result
• AUC : 0.95, Sensitivity : 86.9%, Specificity :88.5%
• Related Company : CureMetrix
Image from
https://pubs.rsna.org/doi/pdf/10.1148/radiol.2019182908 https://www.accessdata.fda.gov/cdrh_docs/pdf18/K183285.pdf
48. 2021 AMI Inc. ALL rights reserved.
48
Image : CT/ MRI (aneurysm/ cancer)
• Denoised PET Images
• Input : PET Images
• Output : Denoised Images
• Related Company : SUBTLE MEDICAL
https://subtlemedical.com/usa/subtlepet/
https://www.accessdata.fda.gov/cdrh_docs/pdf18/K182336.pdf
49. 2021 AMI Inc. ALL rights reserved.
49
Image : X-ray (bone)
50. 2021 AMI Inc. ALL rights reserved.
50
Image : X-ray (bone)
• Deep neural network improves fracture detection by clinicians
• Input : X-ray images
• Output : Fracture
• Model : 2D CNN
• Result
• Sensitivity : 91.5%, Specificity :93.9%
• Related Company : Imagen Technologies
51. 2021 AMI Inc. ALL rights reserved.
51
Biological signal : ECG
52. 2021 AMI Inc. ALL rights reserved.
52
ECG
• 12-lead ECG
• I, II, III, aVR, aVL, aVF
• V1, V2, V3, V4, V5, V6
http://www.chugaiigaku.jp/upfile/browse/browse2549.pdf
https://ja.wikipedia.org/wiki/%E5%BF%83%E9%9B%BB%E5%9B%B3
12-lead ECG (12 wave data)
ECG measurement
53. 2021 AMI Inc. ALL rights reserved.
53
ECG
• 12-lead ECG
• I, II, III, aVR, aVL, aVF
• V1, V2, V3, V4, V5, V6
http://www.chugaiigaku.jp/upfile/browse/browse2549.pdf
https://ja.wikipedia.org/wiki/%E5%BF%83%E9%9B%BB%E5%9B%B3
12-lead ECG (12 wave data)
54. 2021 AMI Inc. ALL rights reserved.
54
Biological signal : ECG
• Deep Neural Networks Can Predict New-Onset Atrial Fibrillation From the 12-Lead ECG and
Help Identify Those at Risk of Atrial Fibrillation–Related Stroke
• Input : 12-lead digital ECG wave data + Age, Sex
• Output : Risk of Atrial Fibrillation or not (1 year later)
• Feature Extraction : 3 branch from 12-lead ECG
• I, II, V1, and V5, acquired from time (t) = 0 (start of data acquisition) to t=5 [s]
• V1, V2, V3, II, and V5 from t=5 to t=7.5 [s]
• V4, V5, V6, II, and V1 from t=7.5 to t=10 [s]
• Model : 1D CNN
• Result
• AUC : 0.85
• Sensitivity : 69%, Specificity :81%
• Related Company : Tempus
3 branches from 12-lead ECG
55. 2021 AMI Inc. ALL rights reserved.
55
Biological signal : ECG
• Using Deep-Learning Algorithms to Simultaneously Identify Right and Left Ventricular
Dysfunction From the Electrocardiogram
• Input :
• 12-lead ECG wave data
• Table data
• Patient age
• Corrected QT interval, PR interval, atrial rate, and ventricular rate
• Output :
• LVEF (classification, regression)
• Left ventricular ejection fraction (LVEF) is the central measure of
left ventricular systolic function and using the diagnosis of heart failure.
• RVSD or RVD (classification)
• Feature Extraction : 12-lead ECG as image data
• Model : 2D EfficientNet
• Result : Classification
• AUROC, Sensitivity, Specificity
• LVEF<=40% : 0.94, 0.87, 0.85
• 40%<LVEF<=50% : 0.73, 0.78, 0.57
• LVEF>=50% : 0.87, 0.84, 0.81
• RVSD+RVD : 0.84, 0.77, 0.75
LVEF
RVSD, RVD
56. 2021 AMI Inc. ALL rights reserved.
56
Biological signal : ECG
• Deep Learning–Based Algorithm for Detecting Aortic Stenosis Using Electrocardiography
• Input : 12-lead ECG wave data, (or single-lead ECG) + Demographic Information
• Output : AS or not
• Feature Extraction :
• ECG raw data
• ECG feature
• Heart rate, AFIB/AFL, QT interval,
• QRS duration, QTc, R axis, T axis)
• Demographic Information
• Age, Sex, Weight, Height, BMI
• Model : 2D CNN + MLP
• Order of ECG
• V1, V2, V3, V4, V6, aVL, I, aVR, II, aVF, III
• Result
• AUC : 0.861, Sensitivity : 80.0%, Specificity :78.3%
57. 2021 AMI Inc. ALL rights reserved.
57
Biological signal : Breath
sound
58. 2021 AMI Inc. ALL rights reserved.
58
Biological signal : Breath sound
• COVID-19 Artificial Intelligence Diagnosis Using Only Cough Recordings
• Input : Cough (Audio Data)
• Output : COVID-19 or not
• Feature Extraction :
• MFCC (feature which is similar to the human hearing)
• Biomarker model
• Model : 2D CNN (ResNet50)
• Result
• AUC : 0.97, Sensitivity : 98.5%, Specificity : 94.2%
59. 2021 AMI Inc. ALL rights reserved.
59
Summary
• I have summarized following parts.
• Usage of Medical AI
• Application of Medical AI
• Metric of Medical AI
• Features / Difficulties of Medical AI
• Examples of Medical AI Research and Application
• Many doctors and medical staff need products to help medical task by using AI.
60. 2021 AMI Inc. ALL rights reserved.
60
References
• Metric of Medical AI
• 検査データの読み方と考え方
• https://www.jslm.org/books/guideline/2018/04.pdf
• 感度・特異度・ROC曲線
• https://www.jsph.jp/covid/files/5BAA6E3.pdf
• https://jeaweb.jp/files/about_epi_research/contest2016_1.pdf
• Others
• 計測・制御セレクションシリーズ 1 次世代医療AI - 生体信号を介した人とAIの融合 –
• 医療AIの知識と技術がわかる本 事例・法律から画像処理・データセットまで
• 医療AIとディープラーニングシリーズ 2020-2021年版 はじめての医用画像ディープラー
ニング -基礎・応用・事例-
• 医療AIとディープラーニングシリーズ 2021-2022年版 標準 医用画像のためのディープ
ラーニング-実践編-
• Useful websites for obtaining information on medical AI
• The Medical AI Times
• MedTech Online