Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 22nd Abstract. Generative adversarial networks (GANs) are one of the most popular models capable of producing high-quality images. However, most of the works generate images from the vector of random values, without explicit control of desired output properties. We study the ways of introducing such control for the user-selected region of interest (RoI). First, we overview and analyze the existing works in areas of image completion (inpainting) and controllable generation. Second, we propose our model based on GANs, which united approaches from the two mentioned areas, for the controllable local content generation. Third, we evaluate the controllability of our model on three accessible datasets – Celeba, Cats, and Cars – and give numerical and visual results of our method.

1. Author: Vadym Korshunov Supervisor: Dmytro Mishkin
REGION-SELECTED IMAGE GENERATION
WITH
GENERATIVE ADVERSARIAL NETWORKS
1

2. INTRODUCTION
CONTENTS
▸ Diving into image generation
▸ Defining goal, problem and prior definition about locally
controllable generation
▸ Top related works and methods overview
▸ Proposed method explanation
▸ Experiments summary and conclusions
2

3. INTRODUCTION
IMAGE GENERATION
Image generation — process of creating (sampling) artificial
images, based on the real samples. The trainable or non-
trainable model governs this process.
⟶ ̂x ∼ p( ⋅ |c, xr) ⟶
Left image source: unian.ua/politics/10837757-zelenskiy-vpevneniy-shcho-seredniy-biznes-pidtrimuye-reformi
Real data
Right image source: doublicat.com
Generated data
3

4. INTRODUCTION
IMAGE GENERATION
Image Inpainting ̂x ∼ p( ⋅ |xr, RoI)
Masked Generated Real xr
Images source: github.com/csxmli2016/SymmFCNet
4

5. HOW TO CONTROL GENERATED PROPERTIES?
CONTROLLABLE GENERATION
▸ Is this vector correlate with visual aspect of the artificial data?
▸ In the case of image inpainting, can we control concrete features
depends only on the selected region?
▸ Can we control one key feature per one element of the vector?
▸ Through existed and demonstrated models, there are no clear relation
between vector for control and outputs
, where — vector for control̂x ∼ p( ⋅ |c) c
5

6. HOW TO CONTROL GENERATED PROPERTIES?
CONTROLLABLE GENERATION
▸ Problem: poor controllability of image generation models,
i.e, unclear relation between outputs (visual features) and
vector for control (and prior random vector, if it exists)
▸ Goal: realize procedure of control in such way, that vector
of control correlates with the only limited number of
features, and check procedure on different datasets
6

7. HOW TO CONTROL GENERATED PROPERTIES?
CONTROLLABLE GENERATION
▸ Hypothesis 1. Local part of the image defined its limited number
features for control, and we can extract it in unsupervised way.
▸ Hypothesis 2. The vector for control may be associated with these
local properties, so we can create new content using as condition
selected region of interest — RoI (mask) — and vector for control c.
▸ Take methods from image inpainting domain, and consider
models in form
▸ Extend model with procedure of control, so model will be triple
conditional
p(z|x, RoI)
p(z|x, RoI, c)
Work consists of two parts:
7

8. GENERATIVE ADVERSARIAL NETWORKS
WHY GANS?
▸ Strong class of functions, whose effectiveness has been
proved in many tasks (image inpainting)
▸ Allows to create new content using conditional model
without labels, i.e, in unsupervised way
▸ Exist a lot of variations of GANs, so a lot of ideas can be
united to achieve concrete result
8

9. GENERATIVE ADVERSARIAL NETWORKS
GAN BASIC DEFINITIONS
▸ GAN consists of two networks: discriminator and generator, and
they compete each other
▸ A generator creates new content and should fool discriminator,
while the last one must distinguish real from fake ones
G Dz ∼ pz( ⋅ ) xr ∼ pr( ⋅ )
Gives feedback
Creates fake G(z) = xg
9

10. TOP RELATED WORKS
STYLE MODULATION
▸ Adaptive instance normalization in the internal layers
allows to significantly change the output properties by
only scaling and biasing
▸ It can be used to control output properties of the
generated images, using template
▸ Need few amount of memory and parameters
y
A(y) ⋅
xl − 𝔼(xl)
𝔻(xl) + ϵ
+ b(y)
10

11. TOP RELATED WORK
STYLE MODULATION WITH TEMPLATE
▸ For control used style image — in “gives” style to the
destination image (2017)
A(y) ⋅ ̂x + b(y)
Images source: arxiv.org/pdf/1703.06868.pdf
x
y
11

12. TOP RELATED WORK
STYLE MODULATION WITH MASK
▸ For control used mask on the image — it “gives” style to the
destination image with instance masks (2019)
A(y) ⋅ ̂x + b(y)
Images source: arxiv.org/pdf/1903.07291.pdf
x
y
12

13. TOP RELATED WORKS
INFO GAN
▸ Define additional for control
▸ Concatenate with prior random vector
▸ Statistically combine distribution for control with
generated distribution, using mutual information
maximization:
I(G(z, c), c) = 𝔻KL(pG(z,c), c − pG(z,c) ⋅ pc) max
c ∼ pc( ⋅ )
z ∼ pz( ⋅ )
simplified to ∼ ℍ(c|G(z, c)) min
13

14. PROPOSED APPROACH
KEY IDEA
▸ Define additional for control
▸ Concatenate with prior random vector
▸ Perform two-stage style modulation: with random vector and
with processed mask
▸ Train the generative conditional model , and
interpret mask also as random variable
▸ Increase visual properties, controllability and generative
properties using specified loss functions
c ∼ pc( ⋅ )
z ∼ pz( ⋅ )
G(z0, c|x, Mask)
14

15. PROPOSED APPROACH
DENORMALIZATION BLOCK (FIRST STAGE)
NORMALIZATION CONVOLUTION
ACTIVATION
x
x − 𝔼(x)
𝔻(x) + ϵ
̂RoI
RoI
Extracting non-linear
features from selected
region
15

16. PROPOSED APPROACH
DENORMALIZATION BLOCK (SECOND STAGE)
NORMALIZATION
x
x − 𝔼(x)
𝔻(x) + ϵ
̂RoI
z1 ⋅ ̂RoI + z2
CONVOLUTION
ACTIVATION
CONVOLUTION
ACTIVATION
Cz + d
z
z1, z2
A(z, RoI) b(z, RoI)
Obtaining scale and bias for
normalized input
Uniting random
vector with features
from mask
16

17. PROPOSED APPROACH
DENORMALIZATION BLOCK
NORMALIZATION
x
x − 𝔼(x)
𝔻(x) + ϵ
̂RoI
z1 ⋅ ̂RoI + z2
CONVOLUTION
ACTIVATION
CONVOLUTION
ACTIVATION
Cz + d
z
z1, z2
A(z, RoI) b(z, RoI)⋅ +
17

18. PROPOSED APPROACH
GENERATOR
▸ Gated convolution instead of standard
▸ U-Net based generator for optimal information sharing between layers
▸ Discriminator with few outputs
ENC1
ENC2 DEC2
DEC1
xr, RoI xg
Latent space
Linear z
18

19. PROPOSED METHOD
ADVERSARIAL LOSS
▸ Used Hinge loss as GAN objective with gradient penalty
▸ Used mutual information maximization and VAE loss functions to increase
generative properties and controllability
▸ Used style losses (content loss) to increase quality of generated image
and make it similar to real
▸ Used additional losses for training stabilization (feature matching)
19
LD = 𝔼x max(0, 1 − D(x)) + 𝔼z max(0, 1 + D(G(z)) + λ𝔼t(||∇t D(t)|| − 1)2
LG = − 𝔼zD(G(z))

20. EVALUATION
DATASETS
▸ Constructed GAN was evaluated on three datasets: Cats,
Cars and CelebA
▸ Generator and discriminator manipulates with 64x64
images
20

21. EVALUATION
FRÉCHET INCEPTION DISTANCE DEFINITION
▸ Used as a metric to compare quality of generated and real
images
▸ The metric uses features from pre-trained on ImageNet
neural network (Inception-v3)
▸ Motivation: pre-trained network saw a lot of real samples,
so it can estimate “realism” of the images
FID(Xg, Xr) = ||μr − μg ||2
+ Trace(Σr + Σg − 2(ΣrΣg)
1
2)
21

22. EVALUATION
NUMERICAL MODELS COMPARISON
Method FID (ell. RoIs) FID (rect. RoIs) FID (mix. RoIs)
Ours 0.323 0.074 0.106
FMM 0.539 0.111 0.197
GLCIC 0.485 0.638 0.618
▸ Compared three models on the CelebA
▸ Compared for different masks types (elliptic, rectangular and mixed)
22

23. EVALUATION
VISUAL MODELS COMPARISON
23

24. EVALUATION
CONTROLLABILITY INSPECTION EXAMPLE ONE
c = − 0.9 c = 0.9c = 0
— “makeup reduction"c1
— “aging”c2
— “surprise”c3
Real
Mask
Masked
Changing vector for control as sliders
24

25. EVALUATION
CONTROLLABILITY INSPECTION EXAMPLE TWO
c = − 0.9 c = 0.9c = 0
— “managing emotions”c1
— “smile”c2
— “nose height”c3
Real
Mask
Masked
Changing vector for control as sliders
25

26. EVALUATION
CONTROLLABILITY INSPECTION EXAMPLE (CATS AND CARS)
Changing vector for control as sliders
Real cats
Real cars
c = − 0.5 c = 0.5
— “texture”c1
— no
clear
meaning
c1
26

27. CONCLUSION
ANSWER TO THE REVIEWER
Q: “… So to shed more light on the impact of each
suggested modules it may be good to check how the
proposed method will do controllable generations for
fixed RoI. For example, for eyes or lips only…”
A: Yes, it’s good idea.
27

28. CONCLUSION
CONCLUSIONS
▸ Created method for local controllable generation — it
united image implanting with controllable generation
▸ Validated on different datasets and tested with different
(random) masks
▸ Experimentally proved the built generator allows
controllable generation in the selected region
28

29. CONCLUSION
Q&A
29

30. APPENDIX. UPSAMPLING BLOCK
UPSAMPLING BLOCK
1 − RoI
▸ Upsampling block has
specialization branches
▸ Left branch performed
generative and
controllable effect
▸ Right branch restored
information in the zone
opposite to RoI
30

31. APPENDIX. GENERATOR
GENERATOR
▸ Encoder is U-Net
based
▸ Model performs
global style
manipulation, like VAE
▸ Decoder part restores
latent representations
using mask
information and
generate content
inside mask
31

32. GENERATIVE ADVERSARIAL NETWORKS
GAN LOSS FUNCTIONS EVOLUTION
Loss type Discriminator Generator
Standard
LSGAN
WGAN
WGAN-GP
Hinge
−𝔼x log D(x) − 𝔼z log(1 − D(G(z)) 𝔼z log(1 − D(G(z)))
𝔼x(D(x) − 1)2
+ 𝔼zD(G(z))2 𝔼z(D(G(z)) − 1)2
−𝔼xD(x) + 𝔼zD(G(z)) −𝔼zD(G(z))
−𝔼xD(x) + 𝔼zD(G(z)) + λ𝔼t(||∇t D(t)|| − 1)2 −𝔼zD(G(z))
𝔼x max(0, 1 − D(x)) + 𝔼z max(0, 1 + D(G(z)) −𝔼zD(G(z))
32

33. APPENDIX. DISCRIMINATOR
DISCRIMINATOR
▸ Discriminator
consists of two
parts: global and
local
▸ Also discriminator
approximates
posterior
distribution
Q(c|x) ≈ ℙ(c|x)
33

34. APPENDIX. LOSSES
LOSSES 1
34

35. APPENDIX. LOSSES
LOSSES 2
35

36. APPENDIX. LOSSES
LOSSES 3
VAE-loss:
TV-loss:
36