Presentation of techniques, models and applications of unpaired image-to-image translation, for the intergroup meetings @ University of Bologna - Cesena.

1. Image-to-image translation with
Generative Adversarial Networks
(without math)
Gabriele Graffieti
Research Fellow
Alma Mater Studiorum Universit`a di Bologna
gabriele.graffieti@unibo.it
May 31, 2019
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 1 / 25

2. Image-to-image translation I
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 2 / 25

3. Image-to-image translation II
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 3 / 25

4. Image-to-image translation III
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 4 / 25

5. Image-to-image translation IV
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 5 / 25

6. Image-to-image translation V
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 6 / 25

7. Image-to-image translation VI
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 7 / 25

8. Image-to-image translation VII
How do we do it?
• We know features of the original domain that should not be present
in the results (snow, fog).
• We know features of the target domain that should be added in the
results (blue sky, hallucinated colors).
Nobody teach us what a summer scene looks like, we learn it from
data.
Key concept is realism!
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 8 / 25

9. Image-to-image translation VIII
And machines?
• The can learn concepts directly from data (machine learning).
• But how can we evaluate realism of the results?
• We are not able to define a metric for summerness or van goghness.
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 9 / 25

10. Image-to-image translation IX
My beloved opponent
• Translation can be seen as a particular case of imitation.
• The translated image should be similar to those on the target domain
(realism).
• We should exploit the forger-police officer game.
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 10 / 25

11. Proto-GAN framework
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 11 / 25

12. GAN framework for image-to-image translation
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 12 / 25

13. But the image content?
How we can ensure that the image content remain the same?
• The adversarial loss only ensure that the translated image looks
similar to the ones in the target domain.
• The content can be changed without any penalization.
• We don’t have a reference image in the target domain to compare
with the translated image.
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 13 / 25

14. Cycle consistency
Given a mapping G : X → Y and its inverse F : Y → X the two mappings
should be cycle consistent with each other.
F(G(x)) ≈ x
and
G(F(y)) ≈ y
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 14 / 25

15. CycleGAN
The CycleGAN model is composed of two different GANs, which learn
inverse translations between two image domains. These translations are
maintained cycle consistent through the difference between original and
recovered images.
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 15 / 25

16. Neural style transfer art I
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 16 / 25

17. Neural style transfer art II
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 17 / 25

18. Neural style transfer art III
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 18 / 25

19. Neural style transfer art IV
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 19 / 25

20. Photoshop 2.0?
• https:
//www.youtube.com/watch?time_continue=25&v=MXWm6w4E5q0
• http://gandissect.res.ibm.com/ganpaint.html?project=
churchoutdoor&layer=layer4
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 20 / 25

21. Deepfakes I
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 21 / 25

22. Deepfakes II
Good Utilizations
• Special effects in movies.
• ??
(Very) Bad Uses
• Fake porn.
• Video forging (discredit a politician, direct popular opinions, . . . ).
• False evidences in court cases.
• . . .
We will be able to distinguish fake data from reality?
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 22 / 25

23. Defogging I
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 23 / 25

24. Defogging II
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 24 / 25

25. Image-to-image translation with
Generative Adversarial Networks
(without math)
Gabriele Graffieti
Research Fellow
Alma Mater Studiorum Universit`a di Bologna
gabriele.graffieti@unibo.it
May 31, 2019
Gabriele Graffieti Image-to-image translation with GANs May 31, 2019 25 / 25