Introduction of medical imaging AI, especially in digital pathology. The talk focused on how we come up with different projects, how to define the scope and challenges of these projects.
2. โข Basis concept of AI and common applications
โข Experiences of developing AI applications in aetherAI
โข Challenges & Impacts in medical AI
Outlines
3. What is Deep
Learning ?
A.I.
Ex. Expert System
Machine
Learning
Ex. Logistic regression
Representation
Learning
Ex. Shallow
autoencoder
Deep
Learning
Ex. MLPs
What is A.I.?
Source: Ian Goodfellow et al. Deep Learning
1956 Dartmouth AI Workshop
10. Some computer vision researches/applications in medical
image analysis
Mammographic
Mass Classification
Diabetic
Retinopathy
Detection
Breast Cancer
Metastasis
Detection
Airway
Segmentation of
Chest CT Image
Lung Nodule
Detection
Bone Suppression
in X-Ray Image
Skin Disease
Classification
https://arxiv.org/abs/1702.05747
11. Some computer vision researches/applications in medical
image analysis
Mammographic
Mass Classification
Diabetic
Retinopathy
Detection
Breast Cancer
Metastasis
Detection
Airway
Segmentation of
Chest CT Image
Lung Nodule
Detection
Bone Suppression
in X-Ray Image
Skin Disease
Classification
https://arxiv.org/abs/1702.05747
12. โข Ideal World
โข ๐ ๐๐๐ ๐๐๐ ๐ = ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
โข ๐ ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ = ๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐๐๐ฃ๐๐๐ ๐ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
How does deep learning work?
13. โข Ideal World
โข ๐ ๐๐๐ ๐๐๐ ๐ = ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
โข ๐ ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ = ๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐๐๐ฃ๐๐๐ ๐ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
โข Human Approximation
โข โ ๐๐๐๐๐๐๐๐๐โ = ๐๐๐ ๐๐๐ ๐
How does deep learning work?
14. โข Ideal World
โข ๐ ๐๐๐ ๐๐๐ ๐ = ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
โข ๐ ๐๐๐๐๐๐๐๐๐โ๐๐๐๐๐ ๐๐๐ ๐ = ๐๐๐ ๐๐๐ ๐ , ๐ โถ ๐๐๐ฃ๐๐๐ ๐ ๐ก๐๐ข๐กโ ๐๐ข๐๐๐ก๐๐๐
โข Human Approximation
โข โ ๐๐๐๐๐๐๐๐๐โ = ๐๐๐ ๐๐๐ ๐
โข Machine learning
โข ๐ ๐๐๐๐๐๐๐๐๐โ = ๐๐๐ ๐๐๐ ๐
โข m begins in random state
โข m is update by error backpropagation
How does deep learning work?
16. โข Classification
โข Accuracy may be biased when data
imbalanced
โข F1-score or AUC will be better.
โข Detection
โข Segmentation
Normal Event Guessing
Accuracy
5 1 83%
50 1 98%
5000 1 99.98%
General rules for evaluating different tasks under different
scenarios
17. โข Classification
โข Detection
โข mean Average Precision (mAP)
โข Evaluate precision under different
thresholds across classes
โข Segmentation
General rules for evaluating different tasks under different
scenarios
18. โข Classification
โข Detection
โข Segmentation
โข Intersection-of-Union
General rules for evaluating different tasks under different
scenarios
19. How Does Deep Learning Differ From Traditional Image Processing
Method
or ?
20. How Does Deep Learning Differ From Traditional Image Processing
Method
โข Traditional Image Processing
โข Cell contour segmentation
through thresholding
โข Nuclear segmentation through
thresholding
โข Calculating N/C ratio
โข Set a threshold for
classification
or ?
21. โข Deep Learning
โข Feed deep neural
network 5000 images of
lymphocyte and 5000
images of segmented
neutrophils
How Does Deep Learning Differ From Traditional Image Processing
Method
or ?
โข Traditional Image Processing
โข Cell contour segmentation
through thresholding
โข Nuclear segmentation through
thresholding
โข Calculating N/C ratio
โข Set a threshold for
classification
24. The Process of Making a Medical Image AI
Collecting Data &
Get Annotated Labels
Data Cleaning
Deploy
UI/UX
Identify Goals
Build and Train Model
Evaluate model
performance
Visualization & Error
analysis
25. โข How did we come up with the project
โข Challenges
Several use cases to discuss
26. โข How did we come up with the project
โข Challenges
Several use cases to discuss
28. How to Pickup Projects/Issues
What kinds of problems
can ML solve today
29. How to Pickup Projects/Issues
What kinds of problems
can ML solve today
Anything you are
writing rules for today!
30. โข Deep Learning
โข Feed deep neural
network 5000 images of
lymphocyte and 5000
images of segmented
neutrophils
How Does Deep Learning Differ From Traditional Image Processing
Method
or ?
โข Traditional Image Processing
โข Cell contour segmentation
through thresholding
โข Nuclear segmentation through
thresholding
โข Calculating N/C ratio
โข Set a threshold for
classification
These are
R U L E S !!
31. Before we dive into case studies โฆ
What is Digital Pathology?
32. Before we dive into case studies โฆ
What is Digital Pathology?
33. Before we dive into case studies โฆ
What is Digital Pathology?
35. โข Background
โข Adoption of digital pathology is slow because benefit is not clear
โข AI-powered diagnostic support may be the key
โข Goal
โข To train deep neural networks to recognize cancer cells in nasal
biopsy
Nasopharyngeal carcinoma screening
36. โข Hospital : Chang Gung Memorial Hospital
โข Physicians : ~6 physicians involved
โข 720 whole slide images
โข Digital slide scanner
โข aetherAI cloud platform for image viewing and annotation
โข Compute hardware for deep neural network training
โข 8 * Geforce 1080 Ti
Resource required
38. Two-Level AI Model for
Cancer Detection on
Whole Slide Image
Patch-level model (>10M Patches)
Background, Benign, Cancer
Performance
- Accuracy: 98%
- AUC: 0.99
Slide-level model
260 Training, 100 Testing
Performance
- Accuracy: 97%
- AUC: 0.98
Benign or NPC ?
Ground Truth : Cancer, Normal Tissue
Shadowed area : Cancer predicted by AI
39. โข Lymphoid tissue recognized as cancer
Error Analysis
background
benign
npc
40. Iteration of the model
Real Image Prediction
by Model V1
Prediction
by Model V2
43. How AI is Helping Pathologists with Challenging Cases
44. How AI is Helping Pathologists with Challenging Cases
45. Integration with our platform๏ผ
AI-Powered Target Navigation
(Quick Examination)
46. Digital Pathology AI Modules
97% Accuracy
Integration with our platform๏ผ
AI-Powered Calculation of Tumor Purity
(Quantification)
Tumor Purity : 80%
Region of Interest (1)
47. โข Annotation is very time-consuming
โข Error analysis requires deep understanding of histopathology
โข Integration with workflow requires careful design
โข Image quality differences among institutes
Main Challenges
51. โข Background:
โข Differential counting of bone marrow smear is cumbersome
โข Experts are in shortage, detailed counting often not performed
โข Materials and Methods:
โข Bone marrow smear slides from 1000 patients at NTUH
โข Images taken at 1000X
โข 500,000 cells to be annotated
โข Mask-RCNN based cell detection and classification model
Differential Counting of Bone Marrow Smear
58. Effect of Dataset Size on Precision
Class number of
data before
1/27
Precision
before 1/27
Plasma Cell 2,398 0.8
Segmented-neutrophil 2,635 0.47
Mature-limphocyte 4,087 0.37
Neutrophilic-band 1,226 0.25
Orthochromatic-
erythroblast
1,158 0.59
Eosinophils-and-
precursors
895 0.0
Polychromatophilic-
erythroblast
2,352 0.1
59. Effect of Dataset Size on Precision
More Data, More Benefit!
Class number of
data before
1/27
Precision
before 1/27
New labeled number of
data on 3/6
Precision on 3/6
Plasma Cell 2,398 0.8 3,566 (+48.7%) 0.93 (+16.3%)
Segmented-neutrophil 2,635 0.47 4,693 (+78.1%) 1.0 (+112.8%)
Mature-limphocyte 4,087 0.37 5,344 (+30.8%) 0.71 (+91.9%)
Neutrophilic-band 1,226 0.25 2,636 (+115%) 0.66 (+164%)
Orthochromatic-
erythroblast
1,158 0.59 1,600 (+38.2%) 0.66 (+11.9%)
Eosinophils-and-
precursors
895 0.0 1,229 (+37.3%) 0.5 (+50%)
Polychromatophilic-
erythroblast
2,352 0.1 4,502 (+91.4%) 0.3 (+200%)
60.
61. โข Annotation is very difficult ( and time-consuming )
โข Computation load (20M pixels, large model)
โข Unexpected imaging conditions when landing
โข Integration of workflow
Main Challenges
62. โข Model of microscopes
โข Lightness
โข Lens
โข Exposure
โข โฆ
โข Staining quality across
different institutes
โข Different habits between
algorithms and human
โข Counting in crowded
views
The landing challenge
63. โข Annotation is very difficult ( and time-consuming )
โข Computation load (20M pixels, large model)
โข Unexpected imaging conditions when landing
โข Integration of workflow
Main Challenges
71. Variation is too much and
annotation is too time-consuming
>1 hr/slide
72. โข Input size: 10000 x 10000 x 3 (RGB)
โข Model : Residual Networks
โข Training set : 780 images (357 NPC, 423 Benign)
โข Validation set size: 68 images (32 NPC, 36 Benign)
โข Hardware : QuantaGrid D52G nodes on Taiwania 2
Supercomputer, 8 Tesla V100(32gb) and 768 Gb system memory
per node
โข With batch size = 1, 360 Gb system memory is used for training
through Unified Memory
โข Each update takes 2.5 minutes.
Using images of entire specimen to train CNN
a.k.a. the no-fuss approach
73. Comparison of the two approaches
Patch-level model
No-fuss model
Classificatio
n probability
Grad-
CAM
output
74. Slide-Level Prediction Testset Performance
True vs False Positive
Precision-Recall
No-fuss
model
Two-stage
model
True
Positive
Precision
Recall False Positive
True
Positive
Precision
75. Result - Multi Nodes
โ Up to 274X speedup is achieved by running 32 nodes, compared to non-
optimized 1-GPU training.
โ Iteration of the model๏ผ from few weeks ๏ few days
77. Impact of AI On Medical Image Analysis
โข Laborious manual tasks may be taken over by AI
โข Results will be available faster, 24 hours / 7 days
โข Improved and constant quality of medical image analysis
โข Provide supportive medical services to rural areas -- universal
access to expert diagnosis
โข Medical care will be improved with new insights brought by AI
78. ๆธไฝ็ ็ AI โ ๅฏฆ็พ็ฒพๆบ้ซ็
Thank you
ๆๅ ถไปๅ้ก๏ผๆญก่ฟๆๅบ
seanyu@aetherai.com
Editor's Notes
Of course, AI is not a new idea. The field of AI research was born at a workshop at Dartmouth College in 1956. AI is a very broad definition. In the early days, AI mostly refered to expert system, where AI can react to an input according to a carefully constructed knowledge base. A subset of AI is called Machine learning, where algorithms can learn autonomously from features that human experts extract from data according to their prior knowledge. As a subset of machine learning, representation learning refers to methods where algorithms learn the relationships between input and output from data itself, but not features extracted by human experts. The smallest subset, the core technology behind this current AI revolution, is deep learning, where representation learning is achieved by deep neural networks.
Medical image analysis field was quick to catch up on the AI trend. For example, in mammography, AI algorithms can now detect and classify breast mass better than human experts. Google made big news when it trained a deep neural network to diagnose diabetic retinopathy, that is the disease of retina due to diabetes, better than ophthalmologists.
Medical image analysis field was quick to catch up on the AI trend. For example, in mammography, AI algorithms can now detect and classify breast mass better than human experts. Google made big news when it trained a deep neural network to diagnose diabetic retinopathy, that is the disease of retina due to diabetes, better than ophthalmologists.
Letโs assume, in the ideal world, there is a truth function that dictates how the state of a disease manifests in a radiograph. There is then, an inverse function that dictates how the radiographic appearance of a disease can be mapped to the disease. The job of a human radiologist is to learn this inverse truth function so that when he sees a radiograph, he can make a correct diagnosis. Machine learning is not very different. M here is a machine learning model that maps a radiographic appearance to the diagnosis of a disease. Nowadays, m is often a deep neural network. Deep neural network is very often composed by a series of matrix math operations, combined with nonlinear activation functions.
Letโs assume, in the ideal world, there is a truth function that dictates how the state of a disease manifests in a radiograph. There is then, an inverse function that dictates how the radiographic appearance of a disease can be mapped to the disease. The job of a human radiologist is to learn this inverse truth function so that when he sees a radiograph, he can make a correct diagnosis. Machine learning is not very different. M here is a machine learning model that maps a radiographic appearance to the diagnosis of a disease. Nowadays, m is often a deep neural network. Deep neural network is very often composed by a series of matrix math operations, combined with nonlinear activation functions.
Letโs assume, in the ideal world, there is a truth function that dictates how the state of a disease manifests in a radiograph. There is then, an inverse function that dictates how the radiographic appearance of a disease can be mapped to the disease. The job of a human radiologist is to learn this inverse truth function so that when he sees a radiograph, he can make a correct diagnosis. Machine learning is not very different. M here is a machine learning model that maps a radiographic appearance to the diagnosis of a disease. Nowadays, m is often a deep neural network. Deep neural network is very often composed by a series of matrix math operations, combined with nonlinear activation functions.
