A short presentation on robust conditional GAN. This is based on papers accepted in ICLR 2019 [1] and IJCV 2020 [2]. The goal is to leverage structure in the target space of the model to make the model more robust to noise. [1] Grigorios G Chrysos, Jean Kossaifi, Stefanos Zafeiriou. "Robust Conditional Generative Adversarial Networks", International Conference on Learning Representations (ICLR) 2019. [2] Grigorios G Chrysos, Jean Kossaifi, Stefanos Zafeiriou. "RoCGAN: Robust Conditional GAN", International Journal of Computer Vision (IJCV) 2020.

1. Robust Conditional GAN
Grigorios Chrysos
Imperial College London
January 12, 2021
Grigorios Chrysos Robust Conditional GAN January 12, 2021 1 / 11

2. Conditional data generation: Visual examples
Figure: Image-to-image translation examples.
Phillip Isola, et al. ”A Image-to-image translation with conditional adversarial networks”, Conference on Computer Vision and
Pattern Recognition (CVPR) 2017.
Mehdi Mirza and Simon Osindero. ”Conditional generative adversarial nets”, CoRR 2014.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 2 / 11

3. Conditional data generation: Geometric intuition
(a) cGAN - manifold (b) cGAN - network schematic
Figure: Conditional GAN (cGAN) for conditional data generation. (a) Geometric
mapping learned, (b) Network schematic of the generator.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 3 / 11

4. cGAN: Robustness to noise
Figure: Failure cases of translation with cGANs. The left column depicts the
source input, the right column is the output.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 4 / 11

5. Robust conditional data generation: Geometric intuition
We propose Robust Conditional GAN (RoCGAN).
(a) cGAN - manifold (b) RoCGAN - manifold
Figure: Geometric mapping learned with (a) cGAN, (b) RoCGAN.
GC, Jean Kossaifi, Stefanos Zafeiriou. ”Robust Conditional Generative Adversarial Networks”, International Conference on
Learning Representations (ICLR) 2019.
GC, Jean Kossaifi, Stefanos Zafeiriou. ”RoCGAN: Robust Conditional GAN”, International Journal of Computer Vision (IJCV)
2020.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 5 / 11

6. Robust conditional data generation: Network design
(a) cGAN (b) RoCGAN
Figure: Network design in (a) cGAN, (b) RoCGAN.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 6 / 11

7. Evaluation: Noise models
The performance of the proposed vs cGAN is assessed in a number of
ways:
Using the a testing set similar to the training set.
Using additional noise model to perturb the input (during prediction
only).
The input is then ˜s = s + u(s), where s is the original input and u(s)
is a perturbation per pixel.
Specifically, a) Bernoulli, b) adversarial noise is used. Also, a
cross-noise experiment is conducted.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 7 / 11

8. Evaluation: Robustness to adversarial noise (IFGSM)
cGAN RoCGAN
0 2 4 6 8 10
# steps
0.5
0.6
0.7
0.8
0.9
1.0
SSIM
0 2 4 6 8 10
# steps
60
80
100
120
140
FIDFigure: Performance of cGAN/RoCGAN with respect to the number of steps in
the IFGSM noise (mean SSIM on the left, FID score on the right). A higher SSIM
(or a lower FID) indicates better performance.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 8 / 11

9. Evaluation: Robustness to adversarial noise (latent attack)
cGAN RoCGAN
0 1 2 3 4 5
# steps
0.5
0.6
0.7
0.8
0.9
1.0
SSIM
0 1 2 3 4 5
# steps
50
100
150
200
FIDFigure: Performance of cGAN/RoCGAN with respect to the number of steps in
the latent attack (mean SSIM on the left, FID score on the right). A higher SSIM
(or a lower FID) indicates better performance.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 9 / 11

10. Applications
Figure: Application of the RoCGAN model in deblurring.
GC, Paolo Favaro, Stefanos Zafeiriou. ”Motion deblurring of faces”, International Journal of Computer Vision (IJCV) 2018.
Ioannis Marras, GC, Ioannis Alexiou, Gregory Slabaugh, Stefanos Zafeiriou. ”Reconstructing the Noise Manifold for Image
Denoising”, European Conference on Computer Vision (ECCV) 2020.
Grigorios Chrysos Robust Conditional GAN January 12, 2021 10 / 11

11. Thank you for your attention!
Grigorios Chrysos Robust Conditional GAN January 12, 2021 11 / 11