A collection of medical image related deep learning projects i have experimented on

1. AGENDA
1. How to assess & select a sucessful computer vision
POC ?
2. What kind of data+label+annotation is required for
each type of models ?
3. The complexity vs time to develop the model
4. Inspirational ideas
– case study (inputs-vs-outputs of different type of
CV models)

2. SELECTING &
ASSESSING A GOOD
COMPUTER
VISION(=CV) POC

3. COMPUTER VISION POC ASSESSMENT
•Find ares of skin lesion <- define skin lesion
•Is it binary ? i.e is it normal vs skin lesion  Yes
•Define concrete logic flow to diagnose skin lesion vs
normal
•Scope for timeplan , model deliverables ...etc
Break down problems into
definitions
•Can skin lesion be identified by naked eyes(via logic above)
•Are the skin lesion areas annotated properly ?(show how)
•Data volume and pos vs neg per images(complexity of the
problem)
Naked-eye judgement +
annotate corretly
•Can I augment images like this (show how) ?
•Were the digitalized image always stored in jpeg format ?
•Are skin lesion alwaysbe rounded and browish-black ?
•Difference between normal tatoo vs skin lesion ?
•Do you need to understand how the model is built
(interpretability)
Subject matters
• What’s the value gain once you have the model ( time-to-diagnose ?)
• How is it making an impact (to others potentially)?
•Secure plans of model maintenance & lifecycle management
What’s the value add &
maintenance
•Is the existing datasets capable of answering the problem stated ?
•Is the data quality good ” enough ” to proceed ?
•Do we have enough time & budget to execute this ?
•Is GDPR mitigated on the dataset?
•Is the Azure environment set up properly for Data science activities ?
Is the problem solvable ( = goal
achivable )?
Assessment
(iterative evaluation)
skin lesion as an example POC steps

4. ASK YOURSELF (AND CUSTOMERS, AND
PARTNERS ) ...
You know what problem
you want to solve ?
Find areas of skin lesion
Are you able to show
data+labels+anotations
that answer to the
question ?
Yes, I do, and
”this (image +
logic)” is how the
diagnose is being
made !
Nope, I have no idea
Partially , I know the
class but I didn’t mark
the area of skin lesion
I have no idea, I just
want to find something
useful
Ideas brain storming &
prioritization workshop
Problem
statement
Assessing
data & labels
Type of algorithm
considered
Self-assessing
problem
statement
SupervisedUnsupervise
d
Acquire
annotations

5. WHAT KIND OF
DATA+LABEL
(+MASK/ANNOTATION
)
DO I NEED PER TYPE
OF COMPUTER
VISION MODEL ?
Skin lesion example continued ...

6. CLASSIFICATION MODEL
Melanoma
Class label =1LABE
L
DATA
No Melanoma
Class label = 0
images labels
1
1
1
images labels
0
0
0

7. SEGMENTATION MODEL
Melanoma
Class label =1LABE
L
DATA
No Melanoma
Class label = 0
images labels
=masks
images Labels=masks

8. OBJECT DETECTION MODEL
(BOUNDING BOX)
Melanoma
Class label =1LABE
L
DATA
No Melanoma
Class label = 0
images Labels +
annotation
Usually in xml format
images labels +
annotation
Usually in xml format
1
1
1
0
0
0

9. SKIPPING GENERAL ADVERSIAL KIND
OF MDOELS & OTHER TYPES OF
MODEL SUCH AS VISUAL QUESTIONS
AND ANSWERING

10. MODEL+DATA
COMPLEXITY VS
TIME-TO-DEVELOP
THE MODEL
Consideration for POC time planning and budgeting

11. MODEL COMPLEXITY VS TIME-
NEEDED
(GENERAL RULE-OF-THUMB , START WITH THE SIMPLEST
MODEL)
model + data
complexity
Time needed
To develop the
model
1. Data volume
2. good enough data
quality + annotation to
proceed
-------------------shortest -----
-----------------
(depends on num_of_classes )
--------------mid-
range-----------
(depends on num_of_classes + mask
-------- logest-------
(depends on num_of_classes + annotatiuon
quality + kind of model + interpretability

12. CASES OF
COMPUTER
MODELS AND ITS
INPUTS-VS-
In - What you feed to the model ( data requirements)
Out – what the prediction looks like ( model
deliverables)
During Model training and model prediction, you
feed different data to the model

13. CLASSIFICATION
CASES

14. INPUTs :
(image + label)
Feed to
NN
(trainin
g)
predict
labels
CLASSIFYING PNEUMONIA/NORMAL
(X-RAY IMAGES)
The Neural Networks
Pneumonia Norma
l
Image 1 Image 2
label:
Image ID:
Image 1 Image 2Image ID:
input
Train the model
use trained model
Prediction
( only the
labels)
Image1 :
Pneumonia
Image2: Normal

15. Feed to
NN
(trainin
g)
predict
labels
CLASSIFYING PARKINSON/NORMAL
HAND-DRAWING
(HAND-DRAWING SPIRAL IMAGE)
The Neural Networks
norma
l
parkinso
n
Image 1 Image 2
label:
Image ID:
Image 1 Image 2Image ID:
input
Train the model
use trained model
Prediction
( only the
labels)
Image1 : normal
Image2 :
parkinson

16. Feed to
CNN
(trainin
g)
predict
labels
CLASSIFYING
CONTRAST/NO_CONTRAST CT SCAN
(DICOM IMAGES)
contra
st
No
contrast
Image 1 Image 2
label:
Image ID:
Image 1 Image 2Image ID:
input
Train the model
use trained model
Prediction
( only the
labels)
Image1 : contrast
Image2 : no
contrast

17. SEGMENTATION
CASES

18. Feed to
train
the
model
Model
predict
s
masks
SEGMENTING BRAIN LESION
Model Prediction
train the model
label:
(=mask)
Image ID:
Image ID:
Use trained model
input
Prediction
(the area of interest)

19. Feed to
train
the
model
Model
predict
s
masks
SEGMENTING ULTRASOUND
Model Prediction
train the model
label:
(=mask)
Image ID:
Image ID:
Use trained model
input
Prediction
(the area of interest)

20. Feed to
train
the
model
Model
predict
s
masks
SEGMENTING LUNGS
Model Prediction
train the model
label:
(=mask)
Image ID:
Image ID:
Use trained model
input
Prediction
(the area of interest)

21. Feed to
train
the
model
Model
predict
s
masks
SEGMENTING EYE VESSELS
Model Prediction
train the model
label:
(=mask)
Image ID:
Test Image ID:
Use trained model
input
Prediction
(the eye vessels)

22. REAL TIME
SEGMENTATION IN
VIDEO

23. REAL TIME ABDOMINAL ORGAN
SEGMENTATION

24. OBJECT DETECTION
CASES

25. Feed to
train
the
model
Model
predict
s
masks
OBJECT DETECTION - SKIN LESION
train the model
Image ID:
Image ID:
Use trained model
input
Prediction
(bounding box + class
)
rectangle: class:
28 37 160 156 1annotati
on

26. Detect type of blood cells -1. training
(annotated blood cells class + location (point-per-point )
Labels:
(1) Classes
(2) Location (points)
Image ID: bloodcell_img_0001.jpg
Train the model
Feed to
train
the
model

27. Input
image
Use trained model
Get
predicti
on
Input an image
Get prediction of
(1) Classes
(2) Bounding box of
location for each
class
Detect type of blood cells -2 prediction
(annotated blood cells class + location (point-per-point )

28. IMAGE TRANSLATION
CASES

29. Feed to
train
the
model
CYCLEGANS TRANSLATION EYE +
EYE VESSELS
- 1. DURING TRAINING Model Prediction
train the model
Paris of image A(=the eye) and B (= the vessel )
A B

30. Model
predict
s
masks
CYCLEGANS TRANSLATION EYE 
EYE VESSELS
- 2. DURING PREDICTION
Sent image A
Use trained
model
input
Prediction
direction: A B or B
A
Sent image B
A
B
B
A

31. Generate realistic looking ultrasound image

32. OTHER NOVEL
COMPUTER VISION
CASES

33. IMAGE TRANSLATION CASES
Tracking throat cancer in a video – used to extract interesting
frames

34. 3D POINT-NET
Possible usage –
Instead of this furniture dataset, if we
could scan the entire body and obtain a
voxel grid kind of data , then we could
reconstruct a person’s 3D point cloud
similar to this demo in this slide.
The most common application is in VR/AR
domain, i.e used in Microsoft Hololens as
an example