Deep Learning JP: http://deeplearning.jp/hacks/

1. Image-to-Image Translation with
Conditional Adversarial Nets (Pix2Pix)
&
Perceptual Adversarial Networks for
Image-to-Image Transformation (PAN)
2017/10/2 DLHacks
Otsubo

2. Topic : image-to-image “translation”
1

3. Info
Pix2Pix [CVPR2017]
• Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
- iGAN [ECCV 2016]
- interactive-deep-colorization [SIGGRAPH 2017]
- Context-Encoder [CVPR 2016]
- Image Quilting [SIGGRAPH 2001]
- Texture Synthesis by Non-parametric Sampling [ICCV 1999]
• University of California
• 178 citations
PAN [arXiv2017]
• Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao
• University of Technology Sydney, The University of Sydney,
Universite Paris-Est
2

4. Background
• Many tasks are regarded as “translation”
from input image to output image
- Diverse methods exist for them
3
Is there single framework to achieve them?

5. Overview
Pix2Pix
• General-purpose solution to image-to-image
translation using single framework
- Single framework: conditional GAN (cGAN)
PAN
• Pix2Pix - (per-pixel loss)
+ (perceptual adversarial loss)
4

6. Naive Implementation : U-Net (①)
5
①per-pixel
loss (L1/L2)

7. Pix2Pix (①+②)
6
②adversarial
loss

8. Pix2Pix’s loss (①+②)
7
②
②
①

9. PAN (②+③)
8
③perceptual
adversarial loss

10. PAN’s loss (②+③)
9
L1 norm
②
②
③
③
m : constant

11. Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
10
Input Output
(Ground Truth)

12. Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
11
Input Output
(Ground Truth)
(cf. PAN uses discriminator’s
perceptual loss)

13. Example2 : Image Inpainting
• Globally and Locally Consistent Image
Completion [Iizuka, SIGGRAPH2017]
• Context Encoders (cGAN) [Pathak, CVPR2016]
- per-pixel loss
- adversarial loss
12
Input Output
(Ground Truth)

14. Example3 : Semantic Segmentation
Cityscape / Pascal VOC
• DeepLabv3 [Chen, arXiv2017]
• PSPNet [Zhao, CVPR2017]
http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?
cls=mean&challengeid=11&compid=6
Cell Tracking / CREMI
• Learned Watershed
[Wolf, ICCV2017]
• U-Net
[Ronneberger, MICCAI2015]
http://www.codesolorzano.com/Challenges/CTC/Welcome.html
13
Input Output
(Ground Truth)

15. Result1 : Image De-Raining
14
(≒pix2pix)→
(≒pix2pix)

16. Result2 : Image Inpainting
15

17. Result3 : Semantic Segmentation
16

18. Discussion
vs. No perceptual loss (Pix2Pix)
- Perceptual loss enables D to detect more
discrepancy between True/False images
vs. Pre-trained VGG perceptual loss (ID-GAN)
- VGG features tend to focus on content
- PAN features tend to focus on discrepancy
- PAN’s loss leads to avoid adversarial
examples [Goodfellow, ICLR2015] (?)
17
Why is perceptual adversarial loss so efficient?

19. Minor Difference
• Pix2Pix uses Patch-GAN
- Small size(70×70) patch-discriminator
- Final output of D is average of
patch-discriminator’s responses
(convolutionally applied)
18

20. To Do
• Implement
1. Pix2Pix (Patch Discriminator)
2. PAN (Patch Discriminator)
3. PAN (Normal Discriminator)
Wang et al. might compare 1 with 3.
19

21. 20

22. Implementation
2017/10/17 DLHacks
Otsubo

23. My Implementation
• https://github.com/DLHacks/pix2pix_PAN
• pix2pix
- https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
• PAN
- per-pixel loss à perceptual adversarial loss
- not same as paper’s original architecture
- num of parameters is same as pix2pix
22

24. My Experiments
• Facade (label à picture)
• Map (picture à Google map)
• Cityscape (picture à label)
23

25. Result (Facade pix2pix)
24

26. Result (Facade PAN)
25

27. Result (Map pix2pix)
26

28. Result (Map PAN)
27

29. Result (Cityscape pix2pix)
28

30. Result (Cityscape PAN)
29

31. Result (PSNR[dB])
30

32. Discussion – Why pix2pix > PAN?
• per-pixel loss is needed?
• patch discriminator is not suited for PAN?
• positive margin m
• (bad pix2pix implementation in PAN’s paper…?)
31