The document discusses Generative Adversarial Networks (GANs), highlighting their ability to learn and generate realistic data, including images and audio. It covers various GAN architectures such as CycleGAN, which enables unpaired image-to-image translation, and Self-Attention GANs (SAGAN), designed to capture both short and long-range dependencies. The document also references various applications and challenges associated with GANs, including their evaluation metrics and future work considerations.

1. Generative Adversarial
Networks
Palacode Narayana Iyer Anantharaman
29 Oct 2018

2. References
• https://github.com/lukedeo/keras-acgan/blob/master/acgan-analysis.ipynb
• https://github.com/keras-team/keras/blob/master/examples/mnist_acgan.py
• https://skymind.ai/wiki/generative-adversarial-network-gan
• https://junyanz.github.io/CycleGAN/
• Self Attention Generative Adversarial Networks: Zhang et al

3. Why GAN?
• GANs can learn to mimic any distribution and generate data
• The data may be images, speech or music
• The outputs from GANs are found to be quite realistic and impressive
• Thus, GANs have a number of applications: From being a feature in products like
Photoshop to generating synthetic datasets for image augmentation

4. GAN Architecture

5. GAN Architecture

6. GAN Workflow

7. Generator
• Generates synthetic images given the input noise z
• G is differentiable
• Typically a Gaussian distribution

8. Training
• Train on 2 mini batches simultaneously
• Training samples
• Generated samples
• Cost

9. Cost Function

10. Different Variants of GAN
Ref: https://github.com/lukedeo/keras-acgan/blob/master/acgan-analysis.ipynb

11. Cycle GAN (2017)
• Original Paper: “Unpaired Image-to-Image Translation using Cycle-Consistent
Adversarial Networks”, Zhu et al

12. Image to Image Translation
• Image to image translation is aimed at finding a mapping
between an input image (X) and its corresponding output
image (Y), where the pair X, Y are provided in the dataset
• This assumes that we are provided with such a labelled
dataset with pairings
• CycleGAN attempts to find a mapping between images from
source and target domains in the absence of paired
examples
Learn G: X → Y such that the distribution of images from G(X) is
indistinguishable from the distribution Y using an adversarial
loss.
Couple this with an inverse mapping F: Y → X and enforce a
cycle consistency loss to enforce F(G(X)) ≈ X

13. CycleGAN Approach

14. Cycle GAN: Objective Function
• Two discriminators: Dx and Dy where Dx aims to distinguish between images {x}
and translated images {F(y)}. In the same way Dy aims to discriminate between {y}
and {G(x)}
• The objective function has 2 parts representing the losses:
• adversarial losses for matching the distribution of generated images to the data distribution
in the target domain
• Cycle consistency losses that prevent the learned mappings G and F from contradicting each
other

15. Losses

16. Exercises
• Go through the original paper and answer the following:
• How is the model evaluated? What are the metrics?
• What are the main applications discussed in the paper?
• What are the limitations and future work?

17. SAGAN (2018) Zhang et al Abstract
• GANs often use a CNN as a generator
• CNNs capture short range dependencies very well (local receptive fields) but not
effective to capture long distance correlations
• Self Attention Generative Adversarial Networks (SAGAN) is aimed at generating
images that take in to account both short and long distance dependencies in the
source images

18. SAGAN

19. SAGAN Architecture