An Introduction to Pix2Pix GAN (Image-to-Image Translation with Conditional Adversarial Networks paper).

1. Image-to-ImageTranslation
with Pix2Pix GAN
Presenter: S.Shayan Daneshvar

2. Overview
1. What is Image-to-ImageTranslation?
2. The Problem of Image-to-ImageTranslation
3. Pix2Pix GAN for Image-to-ImageTranslation
4. Pix2Pix Architectural Details
5. Applications of the Pix2Pix GAN
6. Conclusion & References
1/20

3. What is Image-to-ImageTranslation?
1

4. ImageTranslation
Image-to-image translation is the controlled conversion of a given source
image to a target image.
3/20
1

5. The Problem of Image-to-ImageTranslation
2

6. Problem Definition
• It is a challenging problem that typically requires the development of a
specialized model and hand-crafted loss function for the type of translation
task being performed.
• Classical approaches , use per-pixel classification or regression models.
Ideally, a technique is required that is general, meaning that the same general
model and loss function can be used for multiple different image-to-image
translation tasks.
5/20
2

7. Pix2Pix GAN for Image-to-ImageTranslation
3

8. Pix2Pix
Pix2Pix is a Generative Adversarial Network, model designed for general purpose
image-to-image translation.The approach was presented by Phillip Isola, et al. in
their 2016 paper titled Image-to-ImageTranslation with Conditional Adversarial
Networks and presented at CVPR in 2017.
7/20
3

9. Pix2Pix
8/20
3
Demo available at https://affinelayer.com/pixsrv/

10. GAN and cGAN
9/20
3
The GAN architecture is an approach to training a generator model, typically
used for generating images. A discriminator model is trained to classify
images as real or fake and the generator is trained to fool the discriminator
model.
The Conditional GAN, is an extension of the GAN architecture that provides
control over the image that is generated, e.g. allowing an image of a given
class to be generated.

11. Pix2Pix GAN
10/20
3
The generator model is provided with a given image as input and generates a
translated version of the image.
The discriminator model is given an input image and a real or generated
paired image and must determine whether the paired image is real or fake.
Finally, the generator model is trained to both fool the discriminator model
and to minimize the loss between the generated image and the expected
target image.

12. Dataset Required for Pix2Pix
11/20
3
Pix2Pix GAN must be trained on image datasets that are comprised of input
images (before translation) and output or target images (after translation).

13. Pix2Pix Architectural Details
4

14. Pix2Pix Architecture
13/20
4
Pix2Pix GAN architecture involves the specification of the following:
• The generator model
• The discriminator model
• model optimization procedure
Both the generator and discriminator models use the standard Convolution-
BatchNormalization-ReLU blocks of layers.

15. U-Net Generator Model
14/20
4
A U-Net model architecture is used for the generator, instead of the common
encoder-decoder model. It is very similar to encoder-decoder model as it
involves down sampling to a bottleneck and up sampling again to an output
image, but links or skip-connections are made between layers of the same size
in the encoder and the decoder.

16. PatchGAN Discriminator Model
15/20
4
Unlike the standard GAN model that uses a deep convolutional neural network
to classify images, the Pix2Pix model uses a PatchGAN, which is a deep
convolutional neural network designed to classify patches of an input image as
real or fake, rather than the entire image.The output of the network is a single
feature map of real/fake predictions that can be averaged to give a single
score.

17. Composite Adversarial and L1 Loss
16/20
4
The discriminator model is trained in a standalone manner in the same way as
a standard GAN model.
The generator model is trained using both the adversarial loss for the
discriminator model and the mean absolute pixel difference between the
generated image and the expected image.
Generator Loss = Adversarial Loss + λ × L1 Loss

18. Applications of the Pix2Pix GAN
5

19. Applications
18/20
5
• Semantic labels ⇔ photo
• Architectural labels ⇒ photo
• Map ⇔ aerial photo
• Black andWhite ⇒ color photos.
• Edges ⇒ photo.
• Sketch ⇒ photo.
• Day ⇒ night photographs.
• Thermal ⇒ color photos.
• Photo with missing pixels ⇒ inpainted photo

20. Conclusion
19/20
6
• Image-to-image translation often requires specialized models and hand-
crafted loss functions.
• Pix2Pix GAN provides a general purpose model and loss function for image-
to-image translation.
• The Pix2Pix GAN was demonstrated on a wide variety of image generation
tasks, including translating photographs from day to night and product
sketches to photographs.

21. References
20/20
6
1. P. Isola, J. Zhu,T. Zhou and A. A. Efros, "Image-to-ImageTranslation with
Conditional Adversarial Networks," 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2017, pp. 5967-5976.
2. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.Warde-Farley, S. Ozair,
et al., "Generative adversarial nets", NIPS, 2014, pp. 2672–2680.
3. Generative Adversarial Networks with Python Deep Learning Generative
Models for Image Synthesis and ImageTranslation by Jason Brownlee.
4. Generative Adversarial Networks for Image-to-ImageTranslation by Arun
Solanki, Anand Nayyar, and Mohd Naved

22. ThankYou!
Any Questions?