Presentation for CS-MRI using CycleGAN (IEEE 17th ISBI, 2020)

1. Unsupervised Learning for
Compressed Sensing MRI Using CycleGAN
Gyutaek Oh, Byeongsu Sim, and Jong Chul Ye

2. I. Introduction
Accelerated MRI
• Reduce the scan time of MRI
• Downsampled k-space  aliasing pattern
• Original image have to be reconstructed from downsampled image
Subsampling
ℱ/ℱ−1
ℱ/ℱ−1
Reconstruction

3. I. Introduction
• Main research thrust in the past for
accelerated MRI
• Long reconstruction time due to the
computational complexity
CS Algorithms
• High performance & significantly reduced run
time complexity
• Most of the existing deep learning approaches
require matched reference data (e.g. fully
sampled k-space)
Deep Learning
Approaches
Unpaired Deep Learning for Accelerated MRI

4. I. Introduction
Cycle-consistent Adversarial Network (CycleGAN)
• Unpaired image-to-image translation  Medical imaging field (e.g. CT, MRI)
• Two generators & discriminators
 domain of the generator output ≠ domain of the real data
[1] Kang, et al., Medical physics 46.2 (2019): 550-562.
[1]

5. I. Introduction
CycleGAN for Accelerated MRI
• Fully sampled images  downsampled images: deterministic measurement physics
• Replace one generator with the deterministic transform  one generator & discriminator
• Can be derived from optimal transport & penalized least squares cost
Generator
Fourier transform
&
Subsampling

6. II. Theory
𝒚 = ℋ𝒙 + 𝑤 ⋯ 1
• Noisy measurement 𝑦∈𝒴
• Unobserved image 𝑥∈𝒳
• Measurement noise 𝑤
• Measurement operator ℋ :𝒳↦𝒴
Inverse
Problem
Ill-posed
𝑐 𝒙; 𝒚 ≔ 𝒚 − ℋ𝒙 + 𝑅 𝒙 ⋯ 2
• Fixed (or given) variable 𝒚
• Random variable 𝒙
• Regularization function 𝑅 𝒙
Penalized Least
Squares (PLS)

7. II. Theory
• New PLS cost
𝑐 𝒙, 𝒚; Θ, ℋ ≔ 𝒚 − ℋ𝒙 + 𝐺Θ 𝒚 − 𝒙 ⋯ 3
• 𝒚: random variable, 𝒙: random variable
• 𝐺Θ: Neural network with the network parameter Θ
• Considering all combinations of 𝒙, 𝒚 = Finding transportation mapping between 𝒳 and 𝒴
• Optimal transport
𝕂 Θ, ℋ ≔ min
𝜋 𝒳×𝒴
𝑐 𝒙, 𝒚; Θ, ℋ 𝑑𝜋 𝒙, 𝒚 ⋯ 4
• Joint distribution 𝜋 𝒙, 𝒚
• Marginal distributions with respect to 𝒳 and 𝒴 are 𝝁 and 𝝂
Main Contribution

8. II. Theory
min
Θ,ℋ
𝕂 Θ, ℋ = min
Θ,ℋ
max
Φ,Ξ
𝛾ℓ𝑐𝑦𝑐𝑙𝑒 Θ, ℋ + ℓ𝑊𝐺𝐴𝑁 Θ, ℋ; Φ, Ξ ⋯ 5
• Cycle-consistency loss
ℓ𝑐𝑦𝑐𝑙𝑒 Θ, ℋ =
𝒳
𝒙 − 𝐺Θ ℋ𝒙 𝑑𝜇 𝒙 +
𝒴
𝒚 − ℋ𝐺Θ 𝒚 𝑑𝜈 𝒚 ⋯ 6
• Wasserstein GAN loss
ℓ𝑊𝐺𝐴𝑁 Θ, ℋ; Φ, Ξ =
𝒳
𝜑Φ 𝒙 𝑑𝜇 𝒙 −
𝒴
𝜑Φ 𝐺Θ 𝒚 𝑑𝜈 𝒚 +
𝒴
𝜓Ξ 𝒚 𝑑𝜈 𝒚 −
𝒳
𝜓Ξ ℋ𝒙 𝑑𝜇 𝒙 ⋯ 7
New PLS Cost + OT
Kantorovich Dual Formulation

9. II. Theory
• ℓ𝑐𝑦𝑐𝑙𝑒 Θ = 𝒳
𝒙 − 𝐺Θ ℋ𝒙 𝑑𝜇 𝒙 + 𝒴
𝒚 − ℋ𝐺Θ 𝒚 𝑑𝜈 𝒚 ⋯ 8
• ℓ𝑊𝐺𝐴𝑁 Θ; Φ = 𝒳
𝜑Φ 𝒙 𝑑𝜇 𝒙 − 𝒴
𝜑Φ 𝐺Θ 𝒚 𝑑𝜈 𝒚 ⋯ 9
Final Formulation
𝑦 = ℱ−1
𝒫Ωℱ𝑥 ⋯ 8
• 2-D Fourier transform 𝓕
• Sampling operation 𝓟𝛀
• 𝓗 = 𝓕−𝟏
𝓟𝛀𝓕 does not have to be estimated because the sampling mask 𝛀 is known
Accelerated MRI

10. III. Method
Conventional CycleGAN: two generators & discriminators
Can be replaced by 𝓕−𝟏
𝓟𝛀𝓕

11. III. Method
𝓕𝓗  Deterministic Fourier transform (𝓕−𝟏
𝓟𝓕)
Can be neglected

12. III. Method
Proposed CycleGAN: one generator & discriminator

13. IV. Experimental Results

14. V. Conclusion
• Our CycleGAN architecture for CS-MRI can be derived from OT & PLS
• The proposed method showed significantly improved results compared to conventional
CycleGAN
• Results of our CycleGAN were comparable to supervised learning
• It can be an important framework for accelerated MR images when matched reference data
are difficult to obtain