Medical Image Synthesis with Improved Cycle-GAN: CT from CECT

Medical Image Synthesis with Improved Cycle-GAN
CT from CECT
Workshop: Deep Learning for Biomedical Image Reconstruction
Boah Kim, Eung Yeop Kim, Jong Chul Ye

Contents
Medical Image Synthesis with Improved Cycle-GAN: CT from CECT
ISBI2020 Workshop Presentation 2
Boah Kim
1. Introduction
2. Method
3. Experiment
4. Conclusion

Introduction
Medical image synthesis
 An approach to modeling a mapping from given images to unknown images
• Potential risks of radiation exposure for multiple acquisition of medical images (ex. CT, PET)
• Not always accessible modalities for every patient
• Alignment issue on the analysis of multi-images
Multi-modality image synthesis Inter-modality image synthesis
MRI CT 3T MRI 7T MRI
 Necessity
Boah Kim
Nie, Dong, et al. "Medical image synthesis with deep convolutional adversarial networks." IEEE Transactions on Biomedical Engineering

Introduction
Deep learning approaches for image synthesis
 Based on generative adversarial networks (GAN) model
Conditional GAN model
Auto-context model with GAN
• Image generation by solving the min-max problem with generator and discriminator
• Useful for the dataset that does not have ground-truth images
Boah Kim
DCGAN model
Zhang, Qianqian, et al. ICHI, 2018 Dar, Salman UH, et al. IEEE TMI, 2019
Nie, Dong, et al. IEEE TBE, 2018

Introduction
CT image synthesis from CECT
 CECT and unenhanced CT
Boah Kim
CECT Unenhanced CT
• CECT : highlight specific tissues or parts of body by contrast agent
• Useful for diagnosis by extracting certain organ (ex. bone)
 Challenge
Design “CT image synthesis model using unsupervised deep learning”
• Decrease the risk of radiation exposure
• Do not require alignment for image analysis
• Not aligned CT & CECT → Do not have ground-truth images
• Only have to change the enhanced blood vessels
• Do not generate or remove certain organs/tissues in medical images

Contents
Boah Kim
1. Introduction
2. Method
3. Experiment
4. Conclusion

Method
Motivation: cycleGAN
 One-to-one mapping using cyclic constraint
Boah Kim
Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using
cycle-consistent adversarial networks,” CVPR, 2017
• Two Generators
Generate the unknown image from source image
• Two Discriminators
Distinguish between real and fake images
• Loss function
= 𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 𝐺𝐺𝐴𝐴𝐴𝐴, 𝐷𝐷𝐴𝐴 + 𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 𝐺𝐺𝐵𝐵𝐵𝐵, 𝐷𝐷𝐵𝐵
+ 𝜆𝜆1𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝐺𝐺𝐴𝐴𝐴𝐴, 𝐺𝐺𝐵𝐵𝐵𝐵 + 𝜆𝜆2𝐿𝐿𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖(𝐺𝐺𝐴𝐴𝐴𝐴, 𝐺𝐺𝐵𝐵𝐵𝐵)

Method
 Why using cycleGAN?
Boah Kim
B.Sim et al. “Optimal transport, cycleGAN, and penalized LS for unsupervised learning in inverse problems”
• Stochastic generalization of penalized least square (PLS)
Penalized least square loss with deep learning prior
Optimal transport (OT)
x: stochastic variable, y : given variable
x: stochastic variable, y: stochastic variable

Method
 Why using cycleGAN?
Boah Kim
• Stochastic generalization of penalized least square (PLS)
B.Sim et al. “Optimal transport, cycleGAN, and penalized LS for unsupervised learning in inverse problems”
Penalized least square loss with deep learning prior
Optimal transport (OT)
x: stochastic variable, y : given variable
x: stochastic variable, y: stochastic variable
Cyclic loss GAN loss
- cycleGAN can be explained by optimal transport.
- Cyclic loss is derived from the PLS.

Method
Overall framework
 Improved cycleGAN with residual learning
Boah Kim

Boah Kim
• Residual learning for generators
CNN
Prevent loss of
medical information
Method
Overall framework
• Three Discriminators
𝐷𝐷𝐴𝐴 : Distinguish real and fake CT
𝐷𝐷𝐵𝐵 : Distinguish real and fake CECT
𝑫𝑫𝑪𝑪 : Distinguish real and fake pair of CT & CECT
• Two Generators
𝐺𝐺𝐴𝐴𝐴𝐴 : CECT → CT (generate synthetic CT)
𝐺𝐺𝐵𝐵𝐵𝐵 : CT → CECT (generate synthetic CECT)

Boah Kim
Method
Overall framework
CNN
Prevent loss of
medical information
• Two Generators

 Training networks by solving the optimization problem
Boah Kim
𝐿𝐿 𝐺𝐺𝐴𝐴𝐴𝐴, 𝐺𝐺𝐵𝐵𝐵𝐵, 𝐷𝐷𝐴𝐴, 𝐷𝐷𝐵𝐵, 𝐷𝐷𝐶𝐶
= 𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 𝐺𝐺𝐴𝐴𝐴𝐴, 𝐷𝐷𝐴𝐴 + 𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 𝐺𝐺𝐵𝐵𝐵𝐵, 𝐷𝐷𝐵𝐵 + 𝐿𝐿𝐺𝐺𝐺𝐺𝑁𝑁′ 𝐺𝐺𝐴𝐴𝐴𝐴, 𝐺𝐺𝐵𝐵𝐵𝐵, 𝐷𝐷𝐶𝐶
min
𝐺𝐺𝐴𝐴𝐴𝐴,𝐺𝐺𝐵𝐵𝐵𝐵
max
𝐷𝐷𝐴𝐴,𝐷𝐷𝐵𝐵,𝐷𝐷𝐶𝐶
𝐿𝐿(𝐺𝐺𝐴𝐴𝐴𝐴, 𝐺𝐺𝐵𝐵𝐵𝐵, 𝐷𝐷𝐴𝐴, 𝐷𝐷𝐵𝐵,𝐷𝐷𝐶𝐶)
Method
Loss function

Boah Kim
𝑳𝑳𝑮𝑮𝑮𝑮𝑮𝑮(𝑮𝑮𝑨𝑨𝑨𝑨, 𝑫𝑫𝑨𝑨)
min
𝐺𝐺𝐴𝐴𝐴𝐴
𝔼𝔼𝑥𝑥𝐴𝐴~𝑃𝑃𝐴𝐴
[ 𝐷𝐷𝐴𝐴 𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐴𝐴 − 1
2
]
min
𝐷𝐷𝐴𝐴
1
2
𝔼𝔼𝑥𝑥𝐵𝐵~𝑃𝑃𝐵𝐵
[ 𝐷𝐷𝐴𝐴 𝑥𝑥𝐵𝐵 − 1 2
] +
1
2
[𝐷𝐷𝐴𝐴 𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐴𝐴
2
]
• Adversarial loss,
- Produce realistic images
- Apply to input 𝑥𝑥
min
max
Method
Loss function

Boah Kim
𝑳𝑳𝑮𝑮𝑮𝑮𝑵𝑵′(𝑮𝑮𝑨𝑨𝑨𝑨, 𝑮𝑮𝑩𝑩𝑩𝑩, 𝑫𝑫𝑪𝑪)
- Produce realistic images
- Apply to a paired input 𝑥𝑥𝐴𝐴 and 𝑥𝑥𝐵𝐵
min
𝔼𝔼𝑥𝑥𝐴𝐴~𝑃𝑃𝐴𝐴,𝑥𝑥𝐵𝐵~𝑃𝑃𝐵𝐵
[ 𝐷𝐷𝐶𝐶 𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐴𝐴 , 𝐺𝐺𝐵𝐵𝐵𝐵(𝑥𝑥𝐵𝐵 − 1 2
]
min
𝐷𝐷𝐶𝐶
1
2
𝐷𝐷𝐶𝐶 𝑥𝑥𝐵𝐵, 𝑥𝑥𝐴𝐴 − 1 2
+
1
2
[𝐷𝐷𝐶𝐶 𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐴𝐴 , 𝐺𝐺𝐵𝐵𝐵𝐵(𝑥𝑥𝐵𝐵) 2
]
min
max
Method
Loss function
• Adversarial loss,

Boah Kim
• Cyclic loss, 𝑳𝑳𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄(𝑮𝑮𝑨𝑨𝑨𝑨, 𝑮𝑮𝑩𝑩𝑩𝑩)
Guarantee one-to-one mapping of CT & CECT
𝐺𝐺𝐵𝐵𝐵𝐵 𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐴𝐴 − 𝑥𝑥𝐴𝐴 1
+𝔼𝔼𝑥𝑥𝐵𝐵~𝑃𝑃𝐵𝐵
𝐺𝐺𝐴𝐴𝐴𝐴 𝐺𝐺𝐵𝐵𝐵𝐵 𝑥𝑥𝐵𝐵 − 𝑥𝑥𝐵𝐵 1
min
max
Method
Loss function

Boah Kim
• Identity loss, 𝑳𝑳𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊(𝑮𝑮𝑨𝑨𝑨𝑨, 𝑮𝑮𝑩𝑩𝑩𝑩)
Preserve medical information between CT & CECT
𝐺𝐺𝐵𝐵𝐵𝐵 𝑥𝑥𝐴𝐴 − 𝑥𝑥𝐴𝐴 1
+𝔼𝔼𝑥𝑥𝐵𝐵~𝑃𝑃𝐵𝐵
𝐺𝐺𝐴𝐴𝐴𝐴 𝑥𝑥𝐵𝐵 − 𝑥𝑥𝐵𝐵 1
𝑮𝑮𝑩𝑩𝑩𝑩
𝑮𝑮𝑨𝑨𝑨𝑨
CT
CECT
CT
CECT
min
max
Method
Loss function

Boah Kim
• Minimize our designed loss function
(adversarial loss + cyclic loss + identity loss)
min
max
Method
Loss function

Boah Kim
3x3 Conv, BN, ReLU
2x2 Downconv
2x2 Upconv Skip + Concat
1x1 Conv, Tanh
1 64 64
.
256
2
64 128 128
.
128
2
128 256 256
.
64
2
256 512 512
.
32
2
64
2
128
2
256
2
256 256
128 128
64 64 1 1
1
256
2
128
2
64
2
32
2
31
2
30
2
64 128 256 512 1
4x4 Conv (stride 2), LeakyReLU
4x4 Conv (stride 2), BN, LeakyReLU
2x2 Conv (stride 1), BN, LeakyReLU
Fully connected Layer (4x4 Conv)
 Generator  Discriminator
Method
Network architecture

Contents
Boah Kim
1. Introduction
2. Method
3. Experiment
4. Conclusion

Experiment
 Head CT & CECT scans
Boah Kim
• Provided by Gacheon University College of Medicine
• 10 pair of CT & CECT scans (not aligned)
• 8 scans for training / 2 scans for test
CECT
CT
Dataset

Experiment
 Head CT & CECT scans
Boah Kim
• Provided by Gacheon University College of Medicine
• 10 pair of CT & CECT scans (not aligned)
• 8 scans for training / 2 scans for test
CECT
CT
Dataset
Implementation details
• Batch size = 4
• Learning rate = 0.0002 (decayed to zero until last epoch)
• Training for 200 epoch using a single GPU NVIDIA Geforce GTX 1080 Ti
• Data augmentation : 256x256 random crop, random horizontal/vertical flip, rotation

 Synthetic CT image results from CECT
Boah Kim
Input: CECT
[-300,600] HU
Target: CT cycleGAN Ours
Input - Target Input - cycleGAN Input - Ours
[-300,300] HU
Experiment
Results

Boah Kim
Input: CECT
[-300,600] HU
[-300,300] HU
Experiment
Results

Contents
Boah Kim
1. Introduction
2. Method
3. Experiment
4. Conclusion

Conclusion
 Present a medical image synthesis method with improved cycleGAN
Boah Kim
 Validate our model by synthesizing CT image from CECT image using real dataset
 Can be an important platform for medical image synthesis

Acknowledgement
• This work was supported by Institute of Information & Communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government(MSIT) [2016-0-00562(R0124-16-0002), Emotional
Intelligence Technology to Infer Human Emotion and Carry on Dialogue Accordingly]
• This work was also supported by the Industrial Strategic technology development program (10072064,
Development of Novel Artificial Intelligence Technologies To Assist Imaging Diagnosis of Pulmonary,
Hepatic, and Cardiac Diseases and Their Integration into Commercial Clinical PACS Platforms) funded by
the Ministry of Trade Industry and Energy (MI, Korea).
Boah Kim

Workshop: Deep Learning for Biomedical Image Reconstruction
Boah Kim
Thank you.

Medical Image Synthesis with Improved Cycle-GAN: CT from CECT

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Medical Image Synthesis with Improved Cycle-GAN: CT from CECT

Similar to Medical Image Synthesis with Improved Cycle-GAN: CT from CECT (20)

Recently uploaded

Recently uploaded (20)

Medical Image Synthesis with Improved Cycle-GAN: CT from CECT