Oral defence for master thesis on the evaluation of images generated using conditional GANs

1. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
EVALUATION OF CONDITIONAL
IMAGES SYNTHESIS:
GENERATING A PHOTOREALISTIC
IMAGE FROM A FACE SKETCH
Laureanda:
Samantha Gallone
Relatore:
Prof. Andrea De Lorenzo

2. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
AGENDA
AGENDA
What are GANs?
Ø Structure and functioning
Ø Conditional GANs and applications
What’s next?
Ø Main limitations
Ø Three suggestions for future works
Project implementation
Ø How the dataset has been obtained?
Ø How the networks used are structured?
Evaluation’s results
Ø How the generated images have been tested?
Ø What can be said about their quality?
Agenda

3. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
INTRODUCTION
INTRODUCTION
Learn a generative model
Generative
Trained in an adversarial setting
Adversarial
Use deep Neural Networks
Networks
What are Generative Adversarial Networks?

4. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
INTRODUCTION
INTRODUCTION
Conditional GANs

5. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
APPLICATIONS
APPLICATIONS
Ø Generation of digital characters for:
• video games,
• movies, and
• animations
Ø Generation of photorealistic renderings of
suspect based on sketches obtained thanks to
eyewitness information
Ø Creation of fake identities
Possible applications

6. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
Ø ArtLine + sketch simplification
Ø XDoG edge detector + sketch simplification
Dataset preparation
Ø StyleGAN2
Ø ReStyle + pixel2Style2pixel (pSp)
Network architecture

7. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
Ø A very large dataset composed of pair of images (sketch + corresponding image) is needed
Ø Online there are available:
• CUHK Face Sketch FERET Database (CUFSF)
o 1’194 pair of images with both photo of a face and sketch of it
• FFHQ (Flickr-Face-HQ) dataset
o 70’000 face images
o no sketch
Dataset preparation - problem

8. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
Ø ArtLine
Ø Learning to Simplify (LtS)/Mastering sketching
Original image ArtLine LtS MSE+GAN pencil1 pencil2
Dataset preparation – 1st approach

9. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
Dataset preparation – 2nd approach
Ø Extended Difference of Gaussian (XDoG)
Ø Mastering sketching
Original image MSE+GAN
XDoG

10. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
(a) StyleGAN generator
Ø State-of-the-art deep learning generative model
Ø Developed by NVIDIA in 2018 to produce realistic-looking images
StyleGAN2

11. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
ReStyle
Ø Novel inversion scheme tasked with encoding real images into the extended 𝒲 + StyleGAN
latent space

12. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
pixel2Style2pixel
Ø Encoder network that directly generates a series of style vectors which are fed into a pretrained
StyleGAN generator

13. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
IMPLEMENTATION
IMPLEMENTATION
ReStyle – simplified encoder architecture
Ø Encoder architecture based on a variation of the pSp encoder
Ø All style features are derived from the final 16x16 feature map.

14. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Generated images

15. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Survey’s example
In your opinion, which images has inspired this drawing?

16. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Results – Correct vs incorrect responses

17. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Results – Lowest % of correct responses

18. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Results – Highest % of correct responses

19. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION RESULTS
RESULTS
Ø Spearman correlation coefficient used to determine if there is correlation between the time spent to
answer and the percentage of correct responses
Results – Further analysis

20. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION CONCLUSION
CONCLUSION
Limitations
Ø Not able to generate images of all races equally
Ø It is challenging to generate images of children and young people
Ø It is not able to capture some features like piercings, tattoos and freckles

21. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION CONCLUSION
CONCLUSION
Future works
Solve the current limitations
Apply Stable Diffusion to generate photorealistic images based on a face sketch
Adapt the proposed model to other domains

22. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
THANK YOU
FOR THE
ATTENTION

23. Bibliography (I)
1) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. GeneraJve adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N.
Lawrence, and K.Q. Weinberger, editors, Advances in Neural InformaJon Processing Systems, volume 27.
Curran Associates, Inc., 2014.
2) Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based style- gan encoder via iteraJve
refinement. In Proceedings of the IEEE/CVF InternaJonal Conference on Computer Vision (ICCV), October
2021.
3) Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.
Encoding in style: a stylegan encoder for image-to-image transla- Jon. In IEEE/CVF Conference on
Computer Vision and Pa`ern RecogniJon (CVPR), June 2021.
4) Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. Mastering Sketching: Adver- sarial AugmentaJon
for Structured PredicJon. ACM TransacJons on Graphics (TOG), 37(1), 2018.
5) Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. Learning to Simplify: Fully
ConvoluJonal Networks for Rough Sketch Cleanup. ACM TransacJons on Graphics (SIGGRAPH), 35(4),
2016.
6) Sven C. Olsen Holger Winnemöller, Jan Eric Kyprianidis. Xdog: An extended difference-of-gaussians
compendium including advanced image stylizaJon. Computers & Graphics, 36, 2012.

24. Bibliography (II)
7) NVIDIA. Ffhq dataset. https://github.com/NVlabs/ffhq-dataset.
8) Yu-Sheng Lin, Zhe-Yu Liu, Yu-An Chen, Yu-Siang Wang, Ya-Liang Chang, and Win- ston H. Hsu. Xcos: An
explainable cosine metric for face verification task. ACM Trans. Multimedia Comput. Commun. Appl.,
17(3s), nov 2021.
9) Timo Aila Tero Karras, Samuli Laine. A style-based generator architecture for genera- tive adversarial
networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
10) Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for
stylegan image manipulation. arXiv preprint arXiv:2102.02766, 2021.
11) H. J. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.
Cosface: Large margin cosine loss for deep face recognition. In IEEE/CVF Conference on Computer Vision
and Pattern Recognition, 2018.

25. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
APPENDIX

26. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
Explainable Cosine Metric - xCos
Ø It is based on the insight that humans tend to compare
different facial features to determine whether two face
images belong to the same person.
Ø It is built using a grid-based feature extraction
approach, in which each image is divided into
multiple local regions.
Ø It uses the cosine similarity to compute the similarity
score
Ø It includes an attention mechanism that identifies the
specific facial features that contribute the most to the
similarity score

27. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
Extended Difference of Gaussian
Ø Gaussian filter:
Ø Difference of two Gaussians with different 𝜎:
Ø XDoG :
Gσ(x) =
1
2πσ2
e− x2
2σ2
Dσ,k(x) = Gσ(x) − Gkσ(x) ≈ − (k − 1)σ2
∇2
G
Dσ,k,τ(x) = Gσ(x) − τ·Gkσ(x)
Tϵ,φ(u) =
{
1 u ≥ ϵ
1 + tanh(φ·(u − ϵ)) otherwise
Tϵ,φ(Dσ,k,τ * I)

28. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
Learning to Simplify (LtS)
Ø Technique to simplify rough sketches
Ø It consists of a Fully Convolutional Network to simplify the image
Ø It has been trained by the authors using pairs of rough and simplified sketches using a weighted
mean square error criterion as loss

29. INTRODUCTION APPLICATIONS
AGENDA CONCLUSION
RESULTS
IMPLEMENTATION
Mastering sketching
Ø Combines a fully convolutional network for sketch simplification with a discriminator network that
is able to distinguish real line drawings from those generated by the network
Ø It is trained a variation of a conditional GAN where instead of a random input z, it is used a
deterministic prediction
Ø For adversarial training, the prediction model S is trained together with the discriminator model
which is no conditioned on the input x.
S : x ↦ y = S(x)
D : y ↦ D(y) ∈ ℝ