Presentation for Switchable CycleGAN (IEEE TMI, 2021)

1. Continuous Conversion of CT Kernel
using Switchable CycleGAN with AdaIN
Serin Yang, Eung Yeop Kim, and Jong Chul Ye
Serin Yang
적응적 인스턴스 정규화가 적용된 전환 가능 CycleGAN을 이용한 연속적인 CT 커널 생성에 관한 연구

2. 2
Introduction

3. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 3
CT Reconstruction Kernels
Introduction
Soft kernel
Sharp kernel
Window
Level
400
Window
Width
1500
Window
Level
50
Window
Width
120
Sharp kernel (high pass filter)
• preserving higher spatial frequencies
• decreasing lower spatial frequencies
• more noise
• bone
Soft kernel (low pass filter)
• preserving lower spatial frequencies
• reducing higher spatial frequencies
• reduced noise and impaired spatial resolution
• brain or soft tissue
Kernel Properties

4. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 4
CT Reconstruction Kernels
Introduction
Soft kernel
Sharp kernel
Window
Level
400
Window
Width
1500
Window
Level
50
Window
Width
120
Sharp kernel (high pass filter)
• preserving higher spatial frequencies
• decreasing lower spatial frequencies
• more noise
• bone
Soft kernel (low pass filter)
• preserving lower spatial frequencies
• reducing higher spatial frequencies
• reduced noise and impaired spatial resolution
• brain or soft tissue
Kernel Properties
• Increased size of required storage
• Additional cost and inconvenience

5. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 5
• Supervised Method with Convolutional Neural Network
• Mapping function between two different kernels or among limited number of pairs.
Sang Min Lee, 2019 Da-in Eun, 2020
Deep Learning based Kernel Conversion
Introduction

6. 6
Proposed Method

7. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 7
Adaptive Instance Normalization (AdaIN)
Proposed Method
1 − 𝛽 𝑥 + 𝛽 𝜎 𝑦
𝑥 − 𝜇 𝑥
𝜎 𝑥
+ 𝜇 𝑦
Property
• 𝛽 ∈ [𝟎, 𝟏]
• Interpolation between styles of two inputs, 𝒙 and 𝒚
൞
𝑥, 𝛽 = 0
𝜎 𝑦
𝑥 − 𝜇 𝑥
𝜎 𝑥
+ 𝜇 𝑦 , 𝛽 = 1
• 𝒙: content input, 𝒚: style input
• AdaIN aligns the channel-wise mean and variance of 𝒙 to match those of 𝒚
• Not limited to a single style like Instance Normalization(IN)
• Adaptively computes affine parameters from the style input 𝒚
𝐴𝑑𝑎𝐼𝑁(𝑥, 𝑦) = 𝝈 𝒚
𝑥 − 𝜇 𝑥
𝜎 𝑥
+ 𝝁 𝒚
Definition

8. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 8
Switchable CycleGAN with AdaIN
Proposed Method
[1] Gu, Jawook, 2020
Vanilla CycleGAN
Switchable CycleGAN
in 2 domain[1]
Switchable CycleGAN
in 3 domain
𝐴𝑑𝑎𝐼𝑁(𝑥, 𝑦) = 𝝈 𝒚
𝑥 − 𝜇 𝑥
𝜎 𝑥
+ 𝝁 𝒚
Definition

9. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 9
Architecture of the Generator
Proposed Method
Input
Output
Encoder Decoder

10. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 10
Architecture of the Generator
Proposed Method
Input
Output
Encoder Decoder
Source Domain
AdaIN code generator
Target Domain
AdaIN code generator

11. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 11
Loss Function
Proposed Method
where ℓ𝒕𝒐𝒕𝒂𝒍(𝑮, 𝑭, 𝑫𝑺, 𝑫𝑯) = −ℓ𝒅𝒊𝒔𝒄(𝑮, 𝑭, 𝑫𝑺, 𝑫𝑯) + 𝜆𝑐𝑦𝑐ℓ𝒄𝒚𝒄𝒍𝒆(𝑮, 𝑭) + 𝜆𝑖𝑑ℓ𝒊𝒅𝒆𝒏𝒕𝒊𝒕𝒚(𝑮, 𝑭)
min
𝐺,𝐹
max
𝐷𝑆,𝐷𝐻
ℓ𝒕𝒐𝒕𝒂𝒍(𝑮, 𝑭, 𝑫𝑺, 𝑫𝑯),
2 Domain Switchable CycleGAN
G
D𝑆
D𝐻
G
G
G
ℓ𝒄𝒚𝒄𝒍𝒆 𝑮, 𝑭
G
D𝑆
D𝐻
G
G
G
D𝑆
D𝐻
ℓ𝒅𝒊𝒔𝒄 𝑮, 𝑭, 𝑫𝑺, 𝑫𝑯 ℓ𝒊𝒅𝒆𝒏𝒕𝒊𝒕𝒚 𝑮, 𝑭
G
D𝑆
D𝐻
G
G
G
G
G
Reconstructed
output
Input

12. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 12
Loss Function
Proposed Method
3 Domain Switchable CycleGAN
where ℓ𝒕𝒐𝒕𝒂𝒍 𝑮, 𝑭𝒆, 𝑭𝒅, 𝑫𝑺, 𝑫𝑯, 𝑫𝑴 = −ℓ𝒅𝒊𝒔𝒄 𝑮, 𝑭𝒆, 𝑭𝒅, 𝑫𝑺, 𝑫𝑯, 𝑫𝑴 + 𝜆𝑐𝑦𝑐ℓ𝒄𝒚𝒄𝒍𝒆 𝑮, 𝑭𝒆, 𝑭𝒅
+𝜆𝐴𝐸ℓ𝑨𝑬 𝑮, 𝑭𝒆, 𝑭𝒅 + 𝜆𝑠𝑐ℓ𝒔𝒄 𝑮, 𝑭𝒆, 𝑭𝒅
min
𝐺,𝐹𝑒,𝐹𝑑
max
𝐷𝑆,𝐷𝐻,𝐷𝑀
ℓ𝒕𝒐𝒕𝒂𝒍(𝑮, 𝑭𝒆, 𝑭𝒅, 𝑫𝑺, 𝑫𝑯, 𝑫𝑴),
( 0.0,1.0)
G
( 1.0,0.0)
G
H → M → S
H → S
S → M → H
S → H

13. 13
Experimental Results

14. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 14
Switchable CycleGAN in 2 domain - Interpolation
Results

15. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 15
Switchable CycleGAN in 2 domain - Interpolation
Results

16. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 16
Switchable CycleGAN in 2 domain - Interpolation
Results

17. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 17
Switchable CycleGAN in 2 domain – Comparative Studies
Results

18. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 18
Results Switchable CycleGAN in 3 domain

19. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 19
Results Switchable CycleGAN in 2 domain vs 3 domain

20. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 20
Results Effectiveness of self-consistency loss

21. 21
Conclusion

22. Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN 22
Conclusion
• Unsupervised deep-learning method for kernel conversion in CT images​
• Switchable CycleGAN combined with AdaIN allows training with single generator
• Preventing the model from overfitting problem in case of small dataset ↓​
• Computationally efficient​
• Various images were generated by synergistically combining the two given kernels at inference stage
• With intermediate kernel images, Switchable CycleGAN can be trained in 3 domain
• Improvement in the quality of the generated kernel images
• Range of interpolation can be controlled
• The improved performance of our proposed model was proved with extensive experimental results

23. Thank you