2021 04-03-sean

2. Jaemin Jeong Seminar 2
Proposed Semantic rEgion-Adaptive Normalization
Control the style of each semantic region individually.
Good Performance
Reconstruction quality
Variabliity
Visual Quality
Abstract
SPADE is starting point for out research.

3. Jaemin Jeong Seminar 3
1. SPADE use only one style code to control the entire style of an
image, which is not sufficient for high quality synthesis or
detailed control.
2. we believe that inserting style information only in the beginning
of a network is not a good architecture choice.
We found two shorcomings ..

4. Jaemin Jeong Seminar 4
1. SEAN improves the quality of the synthesized images for
conditional GANs.
2. SEAN improves the per-region style encoding, so that
reconstructed images can be made more similar to the input style
images as measured by PSNR and visual inspection.
3. SEAN allows the user to select a different style input image for
each semantic region.
Advantages of SEAN

5. Jaemin Jeong Seminar 5
1. How to distill the per-region styles from the image according to
the mask ?
2. How to use the distilled per-region style codes to synthesize
photo-realistic images ?
Per Region Style Encoding and Control

6. Jaemin Jeong Seminar 6
Per-Region Style Encoder
 ST(Style Matrix) : 𝟓𝟓𝟓𝟓𝟓𝟓 × 𝒔𝒔
 𝒔𝒔 : number of semantic regions
set columns corresponding to regions that do not
exist in an input image to 0.
-> 항상 모든 semantic regions가 동일하게 있는게 아니니까
없는 건 0으로 합시다.
How to Encode Style?

7. Jaemin Jeong Seminar 7
 SEAN
 Semantic Region-Adaptive Normalization
𝛾𝛾𝑐𝑐,𝑦𝑦,𝑥𝑥 𝑆𝑆𝑆𝑆, 𝑀𝑀
ℎ𝑛𝑛,𝑐𝑐,𝑦𝑦,𝑥𝑥 − 𝜇𝜇𝑐𝑐
𝜎𝜎𝑐𝑐
+ 𝛽𝛽𝑐𝑐,𝑦𝑦,𝑥𝑥(𝑆𝑆𝑆𝑆, 𝑀𝑀)
ST : encoding per-region style code
M : segmentation mask
𝛾𝛾𝑐𝑐,𝑦𝑦,𝑥𝑥 𝑆𝑆𝑆𝑆, 𝑀𝑀 = 𝛼𝛼𝛾𝛾𝛾𝛾𝑐𝑐,𝑦𝑦,𝑥𝑥
𝑠𝑠
𝑆𝑆𝑆𝑆 + 1 − 𝛼𝛼𝛾𝛾 𝛾𝛾𝑐𝑐,𝑦𝑦,𝑥𝑥
𝑜𝑜
𝛽𝛽𝑐𝑐,𝑦𝑦,𝑥𝑥 𝑆𝑆𝑆𝑆, 𝑀𝑀 = 𝛼𝛼𝛾𝛾𝛽𝛽𝑐𝑐,𝑦𝑦,𝑥𝑥
𝑠𝑠
𝑆𝑆𝑆𝑆 + 1 − 𝛼𝛼𝛾𝛾 𝛽𝛽𝑐𝑐,𝑦𝑦,𝑥𝑥
𝑜𝑜
How to Control Style?

8. Jaemin Jeong Seminar 8
Network Architecture
1x1
conv

9. Jaemin Jeong Seminar 9
Network Architecture
Generator
Discriminator
Encoder

10. Jaemin Jeong Seminar 10
 We observed that the initial transformations are indispensable components of the
architecture because they transform the style codes according to the different roles of each
neural network layer.
 For example, early layers might control the hair styles (e.g. wavy, straight) of human face
images while later layers might refer to the lighting and color. In addition, we observed that
adding noise to the input of SEAN can improve the quality of synthesized images.
 The scale of such noise is adjusted by a per-channel scaling factor B that is learnt during
training, similar to StyleGAN.
Experiments

11. Jaemin Jeong Seminar 11
Quantitative comparisons.
Quantitative comparison of reconstruction quality.
Quantitative comparison using semantic segmentation performance (Deeplab v3+)

12. Jaemin Jeong Seminar 12
Qualitative results

13. Jaemin Jeong Seminar 13
Qualitative results

14. Jaemin Jeong Seminar 14
Ablation study on CelebAMask-HQ dataset.

15. Jaemin Jeong Seminar 15
Appendix Experiments
AMT Benchmark

16. Jaemin Jeong Seminar 16
 Proposed semantic region-adaptive normalization (SEAN).
 Our main idea is to extend SPADE, the currently best network, to control the style of each
semantic region individually.
 In summary, SEAN is better suited to encode, transfer, and synthesize style than the best
previous method in terms of reconstruction quality, variability, and visual quality.
 SEAN also pushes the frontier of interactive image editing.
 In future work, we plan to extend SEAN to processing meshes, point clouds, and textures on
surfaces.
Conclusion