Generative Adversarial Networks and Their Medical Imaging Applications

▪ Generative
–Can generate samples
▪ Adversarial
–Trained by competing each other
▪ Networks
–Use neural networks
Definition
Generative Adversarial Networks

▪ Discriminative Models
–Given X, predict Y
–We learn P(Y | X) directly
–We cannot can generate samples
–Examples
• Logistic regression, SVM, CRF, Decision trees
▪ Generative models
–Given X, predict P(X|Y)
–We learn P(X, Y)
–We can generate samples from P(X)
–Examples
• Markov chains, Naïve Bayes, GMM
Discriminative Models vs Generative Models
Generative Models?
Richard Feynmann

▪ Adversarial Training between Discriminator vs Generator
–Discriminator is a classifier that determines whether given images is real data from the
world or fake data generated by generator.
–Proposed by Ian Goodfellow(NIPS 2014)
Discriminator vs Generator
Adversarial Networks?
https://www.oreilly.com/learning/generative-adversarial-networks-for-beginners?imm_mid=0f6436&cmp=em-data-na-na-newsltr_ai_20170918
VS

▪ Alternate the training of discriminator and generator until convergence(may not happen)
Overview
Training GAN
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

▪ Fix the generator, then train discriminator to distinguish samples of real images from
samples generated by the generator
Discriminator Training
Training GAN

▪ Fix the discriminator, then train generator from the feedback of discriminator using
samples generated by the generator
Generator Training
Training GAN

▪ Training GAN is a minmax problem where
–The discriminator D tries to maximize its classification accuracy
–The generator G tries to minimize the discriminator’s classification accuracy
–The optimal solution for D
–The optimal solution for G
Mathematical Formulation
Training GAN
Iam Goodfellow et. al, NIPS 2014
Maximized by D Minimized by G
(in practice, we maximize )

Training Algorithm
Training GAN
Discriminator Training
Generator Training

▪ With the optimal discriminator, training GAN is equivalent to minimizing Jensen-Shannon
divergence as
What happens during the training of GAN?
Why GAN Works?
http://videolectures.net/site/normal_dl/tag=1129740/deeplearning2017_courville_generative_models_01.pdf
Christian Leidig et. al, CVPR 2017
the only solution is

Visualization of samples generated from trained Generator
Generating Samples from GAN

Deep Convolutional GAN
Improving Vanilla GAN
Alec Radford et. al, ICLR 2016

▪ LSUN bedroom dataset
Deep Convolutional GAN – Generated Samples

Deep Convolutional GAN – Vector Space Arithmetic

“GAN Zoo”
GAN Papers
https://github.com/hindupuravinash/the-gan-zoo

▪ Replace discriminator loss from binary cross entropy to least square loss
Least Square GAN(LS-GAN)
Improving Loss Function of GAN
Xudong Mao et. al, arXiv 2016
They don’t move.
0
1

▪ Is JS divergence good enough to train GAN?
Wasserstein GAN(WGAN)
Martin Arjovsky et. al, arXiv 2016
Continuous everywhere and differentiable almost everywhere
Vanishing gradient and bad convergence

▪ WGAN objective function
▪ How can we efficiently enforce the Lipschitz
constraint on the critic D ?
–Weight clipping
𝐷 𝑥1 − 𝐷 𝑥2 ≤ 𝐾 𝑥1 − 𝑥2 , K=1

▪ Learning curve and sample quality
Vanilla GAN
WGAN

▪ Issues with Weight Clipping
–Fail to capture higher moments of the data distribution
–Either exploding or vanishing gradient
Wasserstein GAN with Gradient Penalty(WGAN-GP)
Weight
Clipping
Gradient
Clipping
Ishaan Gulrajani et. al, NIPS 2017

▪ WGAN with Gradient Penalty
–Penalize the norm of the gradient instead of clipping the weights of critics

▪ Comparison of various loss functions and their sample quality

Generation of Conditioned Samples by GANs

▪ Conditioning the model on additional information for better multi-modal learning
Conditional GAN(CGAN)
Conditional Generation
Mirza et. al, arXiv 2014
https://github.com/hwalsuklee/tensorflow-generative-model-collections

▪ Disentangle individual dimensions in latent vector for capturing key attributes
InfoGAN
Xi Chen et al arXiv 2016

▪ Examples
InfoGAN
Xi Chen et al arXiv 2016

▪ Use both real/fake and label classifiers for discriminator training
Auxiliary Classifier GAN(AC-GAN)
Augustus Odena et al arXiv 2016

Recent GANs and Their Applications

▪ “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”
–Perceptual Loss
GAN for Image Super Resolution
SRGAN
Chrisitan Leidig et al, CVPR 2017

▪ Super Resolution Results
GAN for Image Super Resolution
SRGAN
Chrisitan Leidig et al, CVPR 2017

▪ “Image-to-Image Translation with Conditional Adversarial Networks”
Conditional GAN for Image Domain Transfer
Image-to-Image Translation
Phillip Isola et al, CVPR 2017

▪ Image Translation Results
Conditional GAN for Image Domain Transfer
Image-to-Image Translation
Phillip Isola et al, CVPR 2017

▪ “Generative Adversarial Text to Image Synthesis”
–Generator is conditioned on text embedding
–Discriminator uses both visual and textual features by concatenation
Conditional GAN for Text to Image Translation
Text2Image
Scott Reed et al, ICML 2016

▪ Text-to-Image Translation Results
Conditional GAN for Text to Image Translation
Text2Image
Scott Reed et al, ICML 2016

▪ “Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks”
–Generate high-resolution images from text by stacking conditional GANs
Stacked Conditional GAN for Text to Image Translation
StackGAN
Han Zhang et al, ICCV 2017

▪ Text-to-Image Translation Results
Stacked Conditional GAN for Text to Image Translation
StackGAN
Han Zhang et al, ICCV 2017

▪ “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”
Unpaired Image Translation
CycleGAN
Jun-Yan Zhu et al, ICCV 2017

▪ Paired/Unpaired and Generated Image/Reconstruction
CycleGAN

▪ Image Translation Example
CycleGAN

▪ “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation”
–Instead of training generators for each source-domain pair, we can train one generator handling
multiple domains to utilize all training dataset for all domain pairs and even for different dataset
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
for real
for fake
Yunjey Choi et. al, CVPR 2018

▪ Multi-domain Translation Result
StarGAN

▪ Use of ‘mask vector’ to jointly train multiple dataset
StarGAN

▪ Multi-domain Translation Results
StarGAN

▪ “Learning from Simulated and Unsupervised Images through Adversarial Training”
Refining Simulated Image for Data Augmentation
SimGAN
Ashish Shrivastava et. al, CVPR 2017
Self-regularization loss

▪ “Learning from Simulated and Unsupervised Images through Adversarial Training”
Refining Simulated Image for Data Augmentation
SimGAN
Ashish Shrivastava et. al, CVPR 2017
Visual Turing Test d=7 degree

–In many cases, data augmentation techniques used in natural images does not
semantically make sense in medical image
(flips, rotations, scale shifts, color shifts)
–Physically-plausible deformations or morphological transform can be used in limited
cases.
–More augmentation choices for texture classification problems.
Data Augmentation for Effective Training Set Expansion
Medical Data Generation
Source : H. R. Roth et. al., MICCAI, 2015

Generation of synthetic dataset for dataset expansion
M.J.M. Chuquicusma, ISBI 2018
▪ “How To Fool Radiologists with Generative Adversarial Networks? A Visual Turing Test For
Lung Cancer Diagnosis”

Generation of synthetic dataset for dataset expansion
M.J.M. Chuquicusma, ISBI 2018
▪ Visual Turing Test

▪ “Synthetic Medical Images from Dual Generative Adversarial Networks”
Generation of Fundus Image using Dual GANs
John T. Guibas et. al, NIPS Workshop 2017

▪ “Synthetic Data Augmentation using GAN for Improved Liver Lesion Classification”
Liver Lesion Generation for Data Augmentation
Maayan Frid-Ada et. al, ISBI 2018
Real Synthetic

▪ “Generalization of Deep Neural Networks for Chest Pathology Classification in X-ray using
Generative Adversarial Networks”
Chest X-ray Generation for Data Augmentation
Hojjat Salehinejad et. al, ICASSP 2018

▪ Generated Samples and Performance Improvement by Synthetic Augmentation
Chest X-ray Generation for Data Augmentation
Hojjat Salehinejad et. al, ICASSP 2018

▪ “Visual Feature Attribution using Wasserstein GANs “
VA-GAN : Understanding Visual Feature Attribution for Alzheimer Disease
Visual Feature Attribution
Christian F. Baumgatrner et. al, arXiv 2018
Overall Objective
GAN Loss Term
RegularizationTerm

▪ “Visual Feature Attribution using Wasserstein GANs “
VA-GAN : Understanding Visual Feature Attribution for Alzheimer Disease
Visual Feature Attribution
Christian F. Baumgatrner et. al, arXiv 2018

Enhance the quality of low-dose CT to normal-dose CT
Synthesis or Enhancement of Medical Image
Dong Nie et. al, MICCAI 2017
▪ “Medical Image Synthesis using Context-aware Generative Adversarial Networks”

Unpaired image translation from MR to CT using CycleGAN
Jelmer M. Wolterink et. al, MICCAI 2017 Workshop
Setting Mean Absolute Error
Paired Voxel-wise Loss 89.4 ± 6.8 HU
Unpaired Cycle Consistency Loss 73.7 ± 2.3 HU
▪ “Deep MR to CT Synthesis using Unpaired Data”

Enhance the quality of low-dose CT to normal-dose CT
J. M. Wolterink et. al, IEEE Trans. Medical Imaging
▪ “Generative Adversarial Networks for Noise Reduction in Low-Dose CT”

Deep learning for undersampled MRI reconstruction
Tran Min Quan et. al, arXiv 2018
▪ “Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss”
–Problem of compressed sensing MRI reconstruction
–CS-MRI using GAN objective

▪ “Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss”

▪ Experimental Results for Understampled MRI Reconstruction using GAN

Generation of segmentation mask undistinguishable from physician’s mask
Physician Friendly Loss for Segmentation
Source : P. Costa(2017), VUNO(2017)
▪ “Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks”

▪ “Adversarial Networks for the Detection of Aggressive Prostate Cancer”
Generation of segmentation mask undistinguishable from physician’s mask
Source : S. Kohl et. al(2017)

▪ “SegAN : Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation”
SegAN : Adversarial Network with Multi-scale Loss
Yuan Xue et. al, arXiv 2017

▪ “Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-
Consistency Generative Adversarial Network”
Segmentation of Multimodal Images using Image-to-Image Translation
Multimodal Image Segmentation
Zizhao Zhang et. al, CVPR 2018

▪ Image Generation
– Generation of rare cases
– Understanding latent structure of lesions
– Improving the performance of diagnostic models
▪ Image Synthesis and Translation
– Noise reduction, modality translation
– Accelerating image acquisition time
– Improving diagnostic performance
▪ Lesion Detection and Segmentation
– More physician friendly training
– Better performance for lesions or organs with complex structure
▪ Future of GANs in Medical Imaging
– More GANs to come in medical imaging with clinical and commercial values
More GANs in Medical Imaging
Future of GANs

khwan.jung@vuno.co
hello@vuno.co
Putting the world’s medical data to work

Generative Adversarial Networks and Their Medical Imaging Applications

More Related Content

What's hot

Similar to Generative Adversarial Networks and Their Medical Imaging Applications

Recently uploaded

In this document

Generative Adversarial Networks and Their Medical Imaging Applications