1. The document discusses Wasserstein GAN (WGAN), a type of generative adversarial network (GAN) that uses the Wasserstein distance rather than Jensen-Shannon divergence. WGAN has improved stability during training over traditional GANs. 2. WGAN trains the discriminator/critic to estimate the Wasserstein distance between real and generated distributions rather than classify samples. The gradient of the Wasserstein distance can be used to train the generator. 3. WGAN allows for using techniques like minibatch discrimination and batch normalization that previously caused issues with GAN training stability. WGAN has better theoretical properties and often produces higher quality samples than alternatives like DCGAN.

1. Wasserstein GAN
2017-07-12 NN #3 @ TFUG

2. Yuta Kashino ( )
BakFoo, Inc. CEO
Astro Physics /Observational Cosmology
Zope / Python
Realtime Data Platform for Enterprise / Prototyping

3. Yuta Kashino ( )
arXiv
stat.ML, stat.TH, cs.CV, cs.CL, cs.LG
math-ph, astro-ph
- PyCon2016
- PyCon2017 Edward
- 2017 8 TFUG
@yutakashino
https://www.slideshare.net/yutakashino/pyconjp2016

4. Wasserstein GAN

5. …
- WGAN: GAN
-
- DCGAN
-
-

6. Generative Adversarial
Networks

7. GAN 1
- Generative Adversarial Networks
- Ian Goodfelow
- Bengio , Theano/Pyleran2
- Google Brain
- 2016 NIPIS Tutorial
- : The GAN Zoo
https://goo.gl/uC8xn2
https://github.com/hindupuravinash/the-gan-zoo

8. GAN 2
- GAN …
- Meow Generator
- HDCGAN, WGAN, LSGAN…
https://ajolicoeur.wordpress.com/cats/
https://github.com/hindupuravinash/the-gan-zoo

9. Vanila GAN
- Generator Discriminator min/max
- G D
- MLP
https://goo.gl/vHUpqG https://goo.gl/7u4zS6

10. DCGAN
-
- CNN
- G /D
- Pool/ Full Batch Norm, Leaky ReLU
Unsupervised Representation Learning with
Deep Convolutional Generative Adversarial Networks
https://arxiv.org/abs/1511.06434
https://goo.gl/8EmZgT

11. GAN
- G/D JS
-
-
-
-
-

12. Wasserstein GAN

13. WGAN 1
- / /
- /
-
-
-
-
- (

14. WGAN 2
-
- Read-through: Wasserstein GAN
- Wasserstein GAN and the Kantorovich-Rubinstein
Duality
-
https://goo.gl/7ywVwc
https://goo.gl/40eCbR

15. WGAN
GAN Descriminator/Critic W
1. W
( 1, 2)
2. W
( 3)
3. W
4.

16. WGAN

17. 1. WGAN:

18. 4
- Total Variation(TV)
- Kullback-Leibler (KL) divergence
- Jenson-Shannon (JS) divergence
- Earth Mover (EM) / Wasserstein
(Pr, Pg) = sup
A
|Pr(A) Pg(A)|
KL(PrkPg) =
Z
x
log
✓
Pr(x)
Pg(x)
◆
Pr(x) dx
JS(Pr, Pg) =
1
2
KL(PrkPm) +
1
2
KL(PgkPm)
M = Pr/2 + Pg/2M
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤

19. 4
- W
- JS
- KL
- TV
KL(P0kP✓) = KL(P✓kP0) =
(
+1 if ✓ 6= 0 ,
0 if ✓ = 0 ,
(P0, P✓) =
(
1 if ✓ 6= 0 ,
0 if ✓ = 0 .
JS(P0, P✓) =
(
log 2 if ✓ 6= 0 ,
0 if ✓ = 0 ,
W(P0, P✓) = |✓|
U[0, 1]
https://goo.gl/40eCbR

20. 3 1
- 1:
W
- 2:
W
W
- 1, 2
W GAN Loss

21. 3 2
3: Kantorovich-Rubinstein
- W
- W
max
w2W
Ex⇠Pr
[fw(x)] Ex⇠P✓
[fw(x)]  sup
kfkLK
Ex⇠Pr
[f(x)] Ex⇠P✓
[f(x)]
= K · W(Pr, P✓)
r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

22. W/EM 1
-
-
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤
scypy.optimize.linprog
γ
https://goo.gl/7ywVwc
https://goo.gl/7ywVwc

23. W/EM 2
: Kantorovich-Rubinstein
-
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤
W(Pr, Pg) = sup
kfkL1
Ex⇠Pr
[f(x)] Ex⇠Pg
[f(x)]

24. 3 2( )
3: Kantorovich-Rubinstein
- W
- W
max
w2W
Ex⇠Pr
[fw(x)] Ex⇠P✓
[fw(x)]  sup
kfkLK
Ex⇠Pr
[f(x)] Ex⇠P✓
[f(x)]
= K · W(Pr, P✓)
r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

25. 2. WGAN:

26. r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

27. PyTorch
https://goo.gl/unktzn

28. 3. WGAN:

29. - WGAN

30. - W
DCGAN
JS
WGAN
W

31. DCGAN
WGAN
DCGAN

32. BatchNorm OK
BatchNorm
WGAN
BatchNorm
DCGAN

33. OK
MLP
WGAN
MLP
DCGAN

35. -
-
- WGAN
- GAN G D
Improved Training of Wasserstein GANs
https://arxiv.org/abs/1704.00028
Do GANs actually learn the distribution? An empirical study
https://arxiv.org/abs/1706.08224

36. WGAN
GAN Descriminator/Critic W
1. W
( 1, 2)
2. W
( 3)
3. W
4.

37. Questions
kashino@bakfoo.com
@yutakashino

38. BakFoo, Inc.
NHK NMAPS: +

39. BakFoo, Inc.
PyConJP 2015
Python

40. BakFoo, Inc.

41. BakFoo, Inc.
: SNS +