The document discusses different approaches to measuring the distance between distributions for unsupervised learning tasks. It describes maximum likelihood and Kullback-Leibler divergence, which can result in mode dropping. Generative adversarial networks (GANs) are introduced as an alternative that trains a generator and discriminator in opposition. However, GANs suffer from vanishing gradients. The document proposes that the Wasserstein distance may be a better measure as it is continuous and avoids mode dropping. It presents preliminary experiments showing the Wasserstein GAN loss correlates better with sample quality than other methods. However, further work is needed to optimize Wasserstein distances and enforce Lipschitz constraints without weight clipping.

1. On Different Distances Between Distributions
and Generative Adversarial Networks
Martin Arjovsky

2. Unsupervised learning
- We have samples from an unknown
distribution

3. Unsupervised learning
- We have samples from an unknown
distribution
- We want to approximate it by a parametric
distribution that’s close to in some sense.

4. Unsupervised learning
- We have samples from an unknown
distribution
- We want to approximate it by a parametric
distribution that’s close to in some sense.
- Close how?

5. Maximum Likelihood
- Maximum likelihood:

6. Maximum Likelihood
- Maximum likelihood:
- Assumptions: continuous with full support.

7. Maximum Likelihood
- Maximum likelihood:
- Assumptions: continuous with full support.
- Problems: restricted capacity distributes mass.
Modeling low dimensional distributions is impossible.

8. Kullback-Leibler Divergence
- Closeness measured by KL divergence (equivalent to
ML):

9. Kullback-Leibler Divergence
- Closeness measured by KL divergence (equivalent to
ML):
- When
integrand goes to infinity: high cost for mode
dropping.

10. Kullback-Leibler Divergence
- Closeness measured by KL divergence (equivalent to
ML):
- When
integrand goes to infinity: high cost for mode
dropping.

11. Generative Adversarial Networks
- Let be the law of for some simple (e.g.
Gaussian) r.v Z, passed through a complex function.

12. Generative Adversarial Networks
- Let be the law of for some simple (e.g.
Gaussian) r.v Z, passed through a complex function.
- Discriminator maximizes and generator minimizes

13. Generative Adversarial Networks
- Under optimal discriminator, minimizes
- Problems: vanishing gradients very quickly when D’s
accuracy is high.

14. Discriminator is pretty good...

15. Vanishing gradients, original cost

16. Alternate update
- Alternate update that has less vanishing gradients

17. Alternate update
- Alternate update that has less vanishing gradients
- Under optimality optimizes

18. Alternate update
- Alternate update that has less vanishing gradients
- Under optimality optimizes
- Problems: JSD with the wrong sign, reverse KL has

19. High variance updates

20. Problems of GANs (and divergences)
- When and lie on low dimensional manifolds,
there’s always a perfect discriminator, that provides
no usable gradients.

21. Problems of GANs (and divergences)
- When and lie on low dimensional manifolds,
there’s always a perfect discriminator, that provides
no usable gradients.
- Under the same assumptions

22. Problems of JSD, KLs et al.
- Doesn’t need to be a continuous function
of .
- Learning parallel lines.

23. Distances between distributions
- The topology JSD induces on probability measures is
too big, therefore few mappings to this space are
continuous.

24. Distances between distributions
- The topology JSD induces on probability measures is
too big, therefore few mappings to this space are
continuous.
- We can use the weak* topology, given by
Wasserstein

25. Wasserstein distance
- Wasserstein loss is continuous (lines ex):

26. Regularity of Wasserstein

27. Hierarchy of distances

28. Idea: optimize Wasserstein
- Wasserstein has a dual problem
- Idea: train one net f to maximize the dual, then do
gradient descent on theta.

29. Little Gaussian Experiment

30. WGAN loss correlates with sample quality!

31. Correlation for a normal GAN is terrible

32. Improved model stability

33. Improved model stability (cont.)

34. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!

35. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:

36. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?

37. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?

38. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?
- How do we optimize them? They all have duals, but they are much more
complicated. (E.g. for W2 replace lipschitz by convex and convex conj).

39. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?
- How do we optimize them? They all have duals, but they are much more
complicated. (E.g. for W2 replace lipschitz by convex and convex conj).
- Wasserstein requires a metric in X.

40. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?
- How do we optimize them? They all have duals, but they are much more
complicated. (E.g. for W2 replace lipschitz by convex and convex conj).
- Wasserstein requires a metric in X.
- Which one is wgan using? (Some combination of features / samples L2?)

41. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?
- How do we optimize them? They all have duals, but they are much more
complicated. (E.g. for W2 replace lipschitz by convex and convex conj).
- Wasserstein requires a metric in X.
- Which one is wgan using? (Some combination of features / samples L2?)
- Can we optimize W for a given metric? (And construct geodesics!)

42. Further work needed
- Weight clipping is a terrible way to enforce Lipschitz constraints!
- There are many Wasserstein distances aside from EM:
- What properties do they share? Make them different?
- How much mode dropping / sample quality focused are they?
- How do we optimize them? They all have duals, but they are much more
complicated. (E.g. for W2 replace lipschitz by convex and convex conj).
- Wasserstein requires a metric in X.
- Which one is wgan using? (Some combination of features / samples L2?)
- Can we optimize W for a given metric? (And construct geodesics!)
- Can we learn the metric simultaneously? (And learn geodesics!)