1. Optimal transport is a useful tool for designing unsupervised deep learning networks for inverse problems like denoising, deconvolution, and accelerated MRI. 2. CycleGAN can be interpreted as minimizing two Wasserstein distances between the input and target spaces in an unsupervised manner. 3. The document presents applications of CycleGAN and its variations to multiphase CT denoising, microscopy deconvolution, and accelerated MRI, showing it can produce good results without supervised labels.

1. O
Optimal Transport-driven CycleGAN for
Unsupervised Learning
in Inverse Problems

2. .LP-DHFKXO*UDGXDWH6FKRRORI$,
Dept. of Bio/Brain Engineering
Dept. of Mathematical Sciences

3. Classical Learning vs Deep Learning
D
Diagnosis
Classical machine learning
Deep learning
(no feature engineering)
Feature
Engineering
Esteva et al, Nature Medicine, (2019)

4. Deep Learning for Scientific Discovery
D
Diagnosis
Diagnosis analysis
New frontiers of deep learning: inverse problems

5. Penalized LS for Inverse Problems
Likelihood term
Data fidelity term
Prior term
Regularization term
Classical approaches for inverse problems
Tikhonov, TV, Compressed sensing
Top-down model
Transductive non-inductive
Computational expensive

6. Feed-Forward Neural Network Approaches
CNN as a direct inverse operation
Most simplest and fastest method
Supervised learning with lots of data

7. Model-based, PnP using CNN Prior
Likelihood term
Data fidelity term
CNN based regularization
CNN is used as a denoiser
Can use relative small CNNs fewer training data
Supervised learning
Still iterative
Aggarwal et al, IEEE TMI, 2018; Liu et al, IEEE JSTSP, 2020; Wu et al, IEEE JSTSP, 2020

8. Deep Image Prior (DIP)
CNN architecture as a regularization
Unsupervised learning
Extensive computation PLS
Ulyanov et al, CVPR, 2018

9. Unsupervised
Feed-forward CNN?

10. Yann LeCun’s Cake Analogy
Slide courtesy of Yann LeCun’s ICIP 2019 talk

11. Why Unsupervised Learning in Inverse Problems?
Low-dose CT
Remote sensing
Metal artifact removal
Blind deconvolution

12. Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020
Forward physics
(unknown, partially known, known)
inverse solution
Our Geometric View of Unsupervised Learning

13. Geometry of GAN
subject to
Lei, Na, arXiv:1710.05488 (2017)

14. Statistical Distances
f-Divergence
Wasserstein-1 metric
GAN, f-GAN
W-GAN
divergence
metric

15. Absolute Continuity

16. Optimal Transport
: A Gentle Review

17. Optimal Transport
Transportation map

18. Push-Forward of a Measure

19. Optimal Transport: Monge
Monge’s Original OT
Difficulty from the push-forward constraint
Transportation cost

20. Optimal Transport: Kantorovich
Kantorovich’s OT
Allows mass splitting
Probabilistic OT
Linear programming Nobel prize in economy
Transportation
cost
Joint
distribution

21. Kantorovich Dual Formulation
c-transform
marginal distribution
Kantorovich potential

22. Wasserstein-1 Metric and Its Dual
Kantorovich
Dual
1-Lipschitz. (Lip1)

23. Wasserstein GAN with Gradient Penalty
1-Lipschitz penalty
generator
discriminator

24. Generative Adversarial Nets (GANs)
Generator (transport map)
Discriminator
(Kantorovich Potential)
https://www.cfml.se/blog/generative_adversarial_networks/

25. Geometry of CycleGAN
Sim et al, arXiv:1909.12116, 2019
Khan et al, arXiv:2006.14773, 2020
Lee et al, arXiv:2007.03480, 2020

26. Geometry of CycleGAN
Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020
Forward physics
(unknown, partially known, known)
inverse solution

27. Two Wasserstein Metrics in Unsupervised Learning

28. Joint Minimization CycleGAN
Dual formulation Sim et al, arXiv:1909.12116, 2019

29. CycleGAN: Loss Function
1-Lipschitz
Cycle-consistency Discriminators
Forward operator
Unknown
Partial known
known

30. CycleGAN vs Penalized LS
Data fidelity term
Regularization term
Data fidelity term Regularization term
CycleGAN
PLS
y
D
D
D
D
D
D
D
D
D
Da
a
a
a
a
a
at
t
t
t
t
t
t
t
t
ta
a
a
a
a
a
a f
f
f
f
f
f
f
f
f
f
fi
i
i
i
i
i
i
i
i
id
d
d
d
d
d
d
d
d
d
de
e
e
e
e
e
el
l
l
l
l
l
l
l
l
l
li
i
i
i
i
i
i
i
i
it
t
t
t
t
t
t
t
t
ty
y
y
y
y
y
y
y t
t
t
t
t
t
t
t
t
te
e
e
e
e
e
er
r
r
r
r
r
rm
m
m
m
m
m
m
y R
R
R
R
R
R
R
R
R
R
R
R
Re
e
e
e
e
e
e
e
e
e
eg
g
g
g
g
g
g
g
g
g
g
g
gu
u
u
u
u
u
u
u
u
u
ul
l
l
l
l
l
l
l
l
l
l
l
la
a
a
a
a
a
a
a
a
a
ar
r
r
r
r
r
r
r
r
r
ri
i
i
i
i
i
i
i
i
i
i
i
iz
z
z
z
z
z
z
z
z
z
za
a
a
a
a
a
a
a
a
a
at
t
t
t
t
t
t
t
t
t
t
t
ti
i
i
i
i
i
i
i
i
i
i
i
io
o
o
o
o
o
o
o
o
o
on
n
n
n
n
n
n
n
n
n
n t
t
t
t
t
t
t
t
t
t
t
t
te
e
e
e
e
e
e
e
e
e
er
r
r
r
r
r
r
r
r
r
rm
m
m
m
m
m
m
m
m
m
m
CycleGAN can be considered as
stochastic generalization of PLS

31. Multiphase Cardiac CT denoising
Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose
Goal: dynamic changes of heart structure
No reference available
Kang et al, Medical Physics, 2018
Case 1: Unsupervised Denoising for Low-Dose CT

32. Unsupervised Denoising by CycleGAN
Kang et al, Medical Physics, 2019

33. Lose dose (5%) high dose
Input: phase 1 Proposed Target: phase 8 Input- output

34. (a) (b) (c) (d)
(e) (f) (g) (h)
Input: phase 1 Proposed Without
identity loss
GAN
Ablation Study

35. Input: phase 1 Proposed Without
identity loss
GAN
Ablation Study
(a) (b) (c) (d)
(e) (f) (g) (h)

36. Case 2: Unsupervised Deconvolution Microscopy
Lim et al, IEEE TCI, 2020

37. R
Results on Real Microscopy Data
Qualitative results

39. Lim et al, IEEE TCI, 2020

40. Case 3: Unsupervised Learning for Accelerated MRI
Sim et al, arXiv:1909.12116, 2019

41. R
Results on
Fast MR Data
Set
Sim et al, arXiv:1909.12116, 2019

42. Summary
Unsupervised learning becomes very important topics in
deep image reconstruction
Our theoretical findings
Optimal transport is an important mathematical tool for
designing unsupervised networks
CycleGAN can be derived by minimizing two
Wasserstein-1 distances in input and target spaces
Variation extensions of CycleGAN
Geometric view can be generalized for other problems