The document summarizes a presentation on applying GANs in medical imaging. It discusses several papers on this topic: 1. A paper that used GANs to reduce noise in low-dose CT scans by training on paired routine-dose and low-dose CT images. This approach generated reconstructed low-dose CT images with improved quality. 2. A paper that used GANs for cross-modality synthesis, specifically generating skin lesion images from other modalities. 3. Additional papers discussed other medical imaging applications of GANs such as vessel-fundus image synthesis and organ segmentation.

1. Ph.D. Candidate: Cheng-Bin Jin (김성빈)
Computer Vision Lab., INHA University, Korea
TensorFlow Dev Summit 2019 Extended SONGDO
2019.04.06
1
의료영상 분야에서의 GAN응용
GAN in Medical Imaging
TensorFlow
DEV Summit 2019 EXTENDED

2. Bio.
2
Cheng-Bin Jin
(김성빈)
• Bachelor’s Degree (2009.06): YANBIAN University, China
• Master’s Degree (2014.06): YANBIAN University, China
• Ph.D. Candidate (2014.09 ~ Present): INHA University, Korea
• Part-time Researcher (2017.09 ~ Present): Team Elysium Inc., Korea
Research Topics:
 Cross Modality Medical Image Synthesis
 Biometrics (Fingerprint) / Machine vision
 Object detection-by-tracking in ADAS
 Action recognition in video surveillance
 Depth estimation from a single-view camera
ADAS: Advanced Driver Assistance System
• Email: sbkim0407@gmail.com
• Github: github.com/ChengBinJin
• Linkedin: linkedin.com/in/cheng-binjin/
• Facebook: facebook.com/chengbinjin0407
• Twitter: twitter.com/ChengBin_Jin
• Google Scholar: author page

3. Today’s Talk
3
I. Brief Introduction of GANs
II. GANs in Medical Imaging
III. Discussions
GANs: Generative Adversarial Networks

4. Brief Introduction of GANs
4

5. Taxonomy of Machine Learning
5
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
• Image classification
• Instance segmentation
• Object detection
• Image captioning
• Variational auto-encoders (VAE)
• Generative adversarial networks
• Autoregressive models
Classification
Classification +
Localization Object Detection
Instance
Segmentation
CAT CAT CAT, DOG, DUCK CAT, DOG, DUCK
Single object Multiple objects

6. Taxonomy of Machine Learning
6
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
Supervised learning
 Get probability of the label for given data instead of label itself
f
Cat: 0.98
Cake: 0.02
Dog: 0.00
y = f(x)
x
y
Image classification
Image segmentation
Object detection
Image captioning
Variational auto-encoders (VAE)
Generative adversarial networks
Autoregressive models

7. Supervised Learning
7
• Mathematical notation of classifying (greedy policy)
 y: label, x: data, θ*: fixed optimal parameter
 * *
argmax ;
y
y P Y X Optimal label
prediction
parameterized bygivenprobabilityget y when P is maximum
Linear model:
y = w1 * x + w2

8. Taxonomy of Machine Learning
8
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
Unsupervised Learning
• Find deterministic function f : z = f(x), x: data, z: latent vector
Image classification
Image segmentation
Object detection
Image captioning
Variational auto-encoders (VAE)
Generative adversarial networks
Autoregressive models
f [0.1, 0.3, -0.8, 0.4, …]

9. Taxonomy of Machine Learning
9
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
Unsupervised Learning
• Find deterministic function f : z = f(x), x: data, z: latent vector
Image classification
Image segmentation
Object detection
Image captioning
Variational auto-encoders (VAE)
Generative adversarial networks
Autoregressive models
f [0.1, 0.3, -0.8, 0.4, …]
Unsupervised Learning: unlimited data
g
Cat: 0.98
Cake: 0.02
Dog: 0.00
y = g(z)
Supervised Learning:
limited data
Weight initialization

10. Taxonomy of Machine Learning
10
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
Unsupervised Learning
• Find deterministic function f : z = f(x), x: data, z: latent vector
Image classification
Image segmentation
Object detection
Image captioning
Variational auto-encoders (VAE)
Generative adversarial networks
Autoregressive models
f [0.1, 0.3, -0.8, 0.4, …]
Unsupervised Learning: unlimited data
g
Cat: 0.98
Cake: 0.02
Dog: 0.00
y = g(z)
Supervised Learning:
limited data
Weight initialization

11. Self Learning
11
• Use data itself as label Convert unsupervised learning into reconstruction
self learning
• z = f(x), x = g(z) x = g(f(x))
f
(encoder)
g
(decoder)
z
x x'
Supervised Learning
with L2 loss ( = MSE)
MSE: mean squared error
Stacked Auto-Encoder
[0.1, 0.3, -0.8, 0.4, …]

12. Image-to-Image Translation
12
f
(encoder)
g
(decoder)
z
Latent vector
x y'
MSE: mean squared error
VAE: variational auto encoder
Stacked Auto-Encoder
Supervised Learning
with L2 loss ( = MSE)
y
• Use data itself as label Convert unsupervised learning into reconstruction
self learning
• z = f(x), x = g(z) x = g(f(x))

13. Random Image Generation
13
G: Generator
D: Discriminator
z: random vector
x: real data
z
x
G
D Real or Fake?
Gaussian noise
as an input for G
Test phase
• The generative model can be thought of as analogous to a team of
counterfeiters, trying to produce fake currency and use it without detection,
while the discriminative model is analogous to the police, trying to detect
the counterfeit currency. (From GAN paper NIPS2014)

14. Taxonomy of Machine Learning
14
I. Random image generation
II. Image-to-image translation
(Cross-modality synthesis)
Unsupervised
Learning
Supervised
Learning
Reinforcement
Learning
Semi-supervised
Learning
Random face generation
Image-to-image
translation
Paired data
Unpaired data

15. MMGAN (Minimax GAN)
15
• Ian J. Goodfellow, et al., “Generative adversarial Nets,” NIPS2014
         ~ ~
min max ( , ) log log 1datap pG D
V D G D D G          zx x z z
x z (1)
G: Generator
D: Discriminator
x~pz(x): Real data distribution
z~pz(z): Random vector (Gaussian or Uniform distribution)
P: probability for real data (0-1)
Real
P
D
x G(z)
Fake
z
G
Fully Connected
Neural Network
Fully Connected
Neural Network
(N, 100)
(N, 28, 28, 1)
Sigmoid()
(N, 28, 28, 1)

16. MMGAN (Minimax GAN)
16
         ~ ~
min max ( , ) log log 1datap pG D
V D G D D G          zx x z z
x z
1D wants to maximize the value function 0
0 0
(1)
• Ian J. Goodfellow, et al., “Generative adversarial Nets,” NIPS2014
G: Generator
D: Discriminator
x~pz(x): Real data distribution
z~pz(z): Random vector (Gaussian or Uniform distribution)
P: probability for real data (0-1)
Real
P
D
x G(z)
Fake
z
G
Fully Connected
Neural Network
Fully Connected
Neural Network
(N, 100)
(N, 28, 28, 1)
Sigmoid()
(N, 28, 28, 1)

17. MMGAN (Minimax GAN)
17
G: Generator
D: Discriminator
x~pz(x): Real data distribution
z~pz(z): Random vector (Gaussian or Uniform distribution)
P: probability for real data (0-1)
         ~ ~
min max ( , ) log log 1datap pG D
V D G D D G          zx x z z
x z
Real
P
D
x G(z)
Fake
z
G
Fully Connected
Neural Network
Fully Connected
Neural Network
(N, 100)
(N, 28, 28, 1)
Sigmoid()
(N, 28, 28, 1)
ConstantG wants to minimize the value function 1
Constant - Infinite
(1)
(N, 1)
• Ian J. Goodfellow, et al., “Generative adversarial Nets,” NIPS2014

18. NSGAN (Non-saturating GAN)
18
Objective Function for Generator in Real Implementation
         * * *
~ ~
min ( , ) log log 1datap pG
V D G D D G         zx x z z
x z
Ian J. Goodfellow, et al., “Generative adversarial Nets,” NIPS2014
small gradient
Slow at the beginning
     * *
~
min ( , ) logpG
V D G D G   
 zz z
z
x = G(z)
Minimax GAN (MMGAN)
Non-saturating GAN (NSGAN)
Note: Become to softmax cross entropy loss
log
yi
j
s
i s
j
e
L
e
 
  
 
 1
1 N
ii
L L
N 
  ,

19. GAN Codes in TensorFlow
19
Step 1: Forward G and D
Step 2: Calculate loss
Step 3: Define optimizers

20. Generative Models
20
• Fully visible belief nets
• Neural autoregressive
distribution estimator
(NADE)
• Masked autoencoder
for distribution
estimation (MADE)
• PxielRNN
• Change of variables
models (nonlinear ICA)
Tractable
density
…
Maximum Likelihood
Explicit density Implicit density
Approximate
density
Markov chain Direct
Variational Markov chain
• Variational
autoencoder (VAE)
• Boltzmann machine
• GSN • Generative
adversarial
networks
(GAN)

21. Generative Models
21
• Three image generation approaches are dominating the field:
VAE GAN Autoregressive Models
Pros.
- Efficient inference with
approximate latent variables.
- Generate sharp images.
- No need for any Markov chain or
approximate networks during
sampling.
- Very simple and stable training
process.
- Currently gives the best log likelihood.
- Tractable likelihood.
Cons. - Generated samples tend to be
blurry.
- Difficult to optimize due to unstable
training dynamics.
- Relatively inefficient during sampling
Variational Auto-Encoders (VAE)
Z
X
 z p z
 x p x z
 q z x Decoder
Encoder
Generative Adversarial Networks (GAN)
Z G
Fake
X
Real
D Real or
Fake?
generate
         ~ ~
min max ( , )
log log 1data
G D
p p
V D G
D D G

         zx x z z
x z
Autoregressive Models
   
2
1 1
1
,...,
n
i i
i
p p x x x 

 x

22. GANs in Medical Imaging
22

23. GANs in Medical Imaging
23Yi, Xin, Ekta Walia, and Paul Babyn. "Generative adversarial network in medical imaging: A review." arXiv preprint arXiv:1809.07294 (2018).
Fig. Number of GAN related papers published
from 2014.
Fig. Categorization of GAN related papers
according to canonical tasks.
• PubMed
• arXiv
• Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)
• SPIE Medical Imaging
• IEEE International Symposium on Biomedical Imaging (ISBI)
• International conference on Medical Imaging with Deep Learning (MIDL)

24. GANs in Medical Imaging
24
(5) Cross modality synthesis
(3) Vessel to fundus image
(2) Skin lesion synthesis(1) Low dose CT denoising
(4) Organ segmentation

25. Three Questions Need to be Answered
25
I. Why do we need this technology?
1) Patient (diagnostic accuracy)
2) Doctor (improve efficiency)
3) Hospital (more profit)
II. What method is used to solve this problem?
1) Classification / Segmentation / Detection / Synthesis
2) Traditional machine learning method / deep learning
III. How to verify the feasibility?
1) Experimental evaluation
2) Clinical evaluation
Note: the above is based on personal experience.
We will focus on these two
aspects for the following papers.
Targets:
• Patent
• Research
• Funding
• Business

26. (1) Low Dose CT Denoising [1/2]
26Wolterink, Jelmer M., et al. "Generative adversarial networks for noise reduction in low-dose CT." IEEE transactions on medical imaging 36.12 (2017): 2536-2545.
Routine-dose CT
(100%)
Low-dose CT
(20%)
Paired data
Noise Reconstructed CTLow-dose CT

27. (1) Low Dose CT Denoising [12/2]
27Wolterink, Jelmer M., et al. "Generative adversarial networks for noise reduction in low-dose CT." IEEE transactions on medical imaging 36.12 (2017): 2536-2545.
Fig. Overview of the proposed pipeline for noise reduction in
low-dose CT.
ILD: low-dose image
IRD: routine-dose image
     
2
1 22
,1G LD RD bce LDL G I I L D G I   
      ,1 , 0D bce RD bce LDL L D I L D G I 
(1)
(2)
Softmax cross entropy loss

28. (2) Skin Lesion Synthesis [1/2]
28
Yi, Xin, Ekta Walia, and Paul Babyn. "Unsupervised and semi-supervised learning with Categorical Generative Adversarial Networks assisted by Wasserstein
distance for dermoscopy image Classification." arXiv preprint arXiv:1804.03700 (2018).
Melanoma: 흑색종(피부암의 일종)
Benign: 양성
Hair Air bubble Ruler
• Melanoma is curable aggressive skin cancer if detected early.
• The diagnosis involves initial screening with subsequent biopsy and histopathological
examination if necessary.

29. (2) Skin Lesion Synthesis [2/2]
29
Yi, Xin, Ekta Walia, and Paul Babyn. "Unsupervised and semi-supervised learning with Categorical Generative Adversarial Networks assisted by Wasserstein
distance for dermoscopy image Classification." arXiv preprint arXiv:1804.03700 (2018).
Supervised Learning
Unsupervised
Learning
• Learning a feature representation uses 70 labeled samples

30. (3) Vessel to Fundus Image [1/2]
30Costa, Pedro, et al. "End-to-end adversarial retinal image synthesis." IEEE transactions on medical imaging 37.3 (2018): 781-791.
Dataset Training pairs Test pairs
DRIVE 40 20
STARE 10 10
Fig. Random samples of eye fundus images and corresponding vessel
networks generated by the model.
Table. Statistics information of the datasets
Paired data
Retinal image Vessel map• Annotated medical data is often scarce and
costly to obtain.

31. (3) Vessel to Fundus Image [2/2]
31
Test stage
Training stage
GAN (pix2pix)
Adversarial Auto-Encoder
Costa, Pedro, et al. "End-to-end adversarial retinal image synthesis." IEEE transactions on
medical imaging 37.3 (2018): 781-791.
ROC: Receiver Operating Characteristics
AUC: Area Under the Curve
TPR: True Positive Rate
FPR: False Positive Rate

32. (4) Organ Segmentation [1/2]
32
• Organ segmentation is a crucial step to obtain effective computer-aided detection on
Chest X-ray (CXR).
Dai, Wei, et al. "Scan: Structure correcting adversarial network for chest x-rays organ segmentation." arXiv preprint arXiv:1703.08770 (2017).
Original CXR image Lung field annotation
(N, 400, 400, 1)
(N, 400, 400, 4)
(N, 400, 400, 4)
(N, 400, 400, 1+4)

33. (4) Organ Segmentation [2/2]
33
JSRT (4 labels)
Dai, Wei, et al. "Scan: Structure correcting adversarial network for chest x-rays organ segmentation." arXiv preprint arXiv:1703.08770 (2017).
Montgomery (3 labels)

34. (5) Cross Modality Synthesis (MR CT) [1/4]
34J. M. Wolterink, et al., “Deep MR to CT Synthesis using Unpaired Data,” International Workshop on Simulation and Synthesis in Medical Imaging 2017
Supervised learning Unsupervised learning
• Examples showing local
misalignment between MR and CT
images after rigid registration using
mutual information.
• Although the skull is generally well-
aligned, misalignments may occur
in the throat, mouth, vertebrae, and
nasal cavities.

35. (5) Cross Modality Synthesis (MR CT) [2/4]
35J. M. Wolterink, et al., “Deep MR to CT Synthesis using Unpaired Data,” International Workshop on Simulation and Synthesis in Medical Imaging 2017
Figure. The CycleGAN model consists of a forward cycle and a backward cycle.
• DisCT: CT discriminator
• SynCT: CT synthetic network
• ICT: CT image
• DisMR: MR discriminator
• SynMR: MR synthetic network
• IMR: MR image
Forward cycle Backward cycle
     
2 2
1CT CT CT CT CT MRL Dis I Dis Syn I  
     
2 2
1MR MR MR MR MR CTL Dis I Dis Syn I  
     1 1Cycle MR CT MR MR CT MR CT CTL Syn Syn I I Syn Syn I I   
(1)
(2)
(3)
LSGAN loss

36. Discussions
36

37. Discussion
37
• Cross modality image synthesis (MR image domain): 50%
 Use down stream tasks such as segmentation or classification to validate the
quality of the generated sample.
• Segmentation & reconstruction (image-to-image translation): 35%
• Classification: 6%
• Others (detection & registration): 9%
• Finally, although there have many promising results reported in the literature, the adoption
of GANs in medical imaging is still in its infancy and there is currently no breakthrough
application as yet adopted clinically for GANs-based methods.
• No matter what research you do, be sure to work with doctors!

38. More Examples
38
Style Transform
Semantic Image Inpainting
Style Transform
github.com/ChengBinJin
Day to Night;
Night to Day

39. References
39
Review papers:
• Creswell, Antonia, et al. "Generative adversarial networks: An overview." IEEE Signal Processing
Magazine 35.1 (2018): 53-65.
• Kurach, Karol, et al. "The gan landscape: Losses, architectures, regularization, and
normalization." arXiv preprint arXiv:1807.04720 (2018).
• Huang, He, Phillip S. Yu, and Changhu Wang. "An introduction to image synthesis with generative
adversarial nets." arXiv preprint arXiv:1803.04469 (2018).
• Litjens, Geert, et al. "A survey on deep learning in medical image analysis." Medical image analysis 42
(2017): 60-88.
Githubs:
• https://github.com/hindupuravinash/the-gan-zoo
• https://github.com/xinario/awesome-gan-for-medical-imaging
• https://github.com/lzhbrian/image-to-image-papers

40. Thank you for your attention!
40
Q & A