1. The document discusses several deep learning methods for line art colorization including pix2pix, style2paints V3, and Tag2pix. 2. Style2paints V3 uses a two-stage UNet with content loss, adversarial loss, and positive regulation loss to first generate a draft colorization and then refine it. 3. Tag2pix uses color invariant and variant tags along with a UNet, squeeze-and-excitation network, and discriminators to colorize line art conditioned on text tags.

1. Sou Hasegawa
20191103 LT

2. Self-Introduce
https://medium.com/@crosssceneofwindff

3. My Deep Learning Work for Creativity

4. My Deep Learning Work for Creativity

5. Premises
1.
- ( )
- (CNN)
2.

6. What is Line Art Colorization ?
Brown_hair
Pink_dress
or or

7. Line Art Colorization Method
1.
a.
- PaintsChainer
- Two-stage Sketch Colorization (style2paints v3)[1]
- User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks[2]
2.
a.
- Tag2Pix[3]
3.
a.
- Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
(style2paints v1)[4]

8. Agenda
1. Basic Architecture
2. Hint
3. Tag
4. Reference

9. Agenda
1. Basic Architecture
2. Hint
3. Tag
4. Reference

10. pix2pix: Architecture
UNet
pix2pix[5] UNet
Discriminator Adversarial loss

11. pix2pix: Objective Function
1. Content loss
2. Adversarial loss

12. The Problems of Line Art Colorization
Kawaii Illustrations

13. Agenda
1. Basic Architecture
2. Hint
3. Tag
4. Reference

14. style2paints V3: Method
UNet
- Draft
-
- Content loss, Adversarial loss Positive regulation loss
- Refinement Draft
-
- Content loss, Adversarial loss
xc, i : c i
m: ×

15. style2paints V3: Architecture

16. style2paints V3: Refinement
Refinement Draft
3
1. ( etc)
a.
2.
a.
Fig. 7
3.
a. Spatial Transformer

17. Agenda
1. Basic Architecture
2. Hint
3. Tag
4. Reference

18. Tag2pix: Method
Tag
- Color Invariant Tag(CIT): ( ) 370
- Color Variant Tag(CVT): ( ) 115
- Semantic
- Content loss, Adversarial loss Content loss
Real/Fake
-
- Content loss, Adversarial loss Classification loss
CIT CVT

19. Tag2pix: Generator Architecture
UNet
SECat-ResNeXt CVT

20. Tag2pix: SECat Architecture
Squeeze-and-Excitation Network[6] GAP CVT Encode
AdaIN SECat
FC: Fully-Connected Layer
GAP: Global Average Pooling

21. Tag2pix: Discriminator Architecture
CVT
CIT

22. Agenda
1. Basic Architecture
2. Hint
3. Tag
4. Reference

23. style2paints V1: Method
Vgg19
Vgg19 (4096 ) UNet
Adversarial loss
- Discriminator 0
- Discriminator Vgg19
- UNet Guided Decoder
-
- Fig.7 Guide Decoder

24. style2paints V1: Architecture
Reference
Image
Input
Image
Output
Image
Guide
Decoder1
Output
Guide
Decoder2
Output

25. My Approach: Methods
style2paints V1 4096
( ….)
→Adaptive Instance Normalization(AdaIN)
-
- x, y
- FUNIT[7]
σ: µ:

26. My Approach: Architecture
Line Art
Reference
Content
Encoder
Style
Encoder
AdaIN ResBlocks Decoder
Colored
Art
Discriminator
Content Encoder Style Encoder
AdaIN ResBlocks
x: Output of content encoder
y: Output of style encoder

27. Summary
1. pix2pix
a. UNet + Discriminator
b.
2. : style2paints V3
a.
b.
3. : Tag2pix
a.
4. : style2paints V1
a. Vgg19 4096 Discriminator 4096
b. AdaIN

28. End

29. Appendix

30. Line Extraction Method
(Tag2pix )
1. sketchKeras[8]
- UNet
2. Sketch Simplification[9]
- Fully-Convolutional Network sketchKeras
3. XDoG[10]
- Gaussian

31. References

32. [1] Lvmin Zhang, et al., “Two-stage Sketch Colorization”. SIGGRAPH ASIA 2018
[2] Yuanzheng Ci, et al., “User-Guided Deep Anime Line Art Colorization with Conditional
Adversarial Networks”. ACM Multimedia Conference 2018
[3] Hyunsu Kim, et al., “Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing
Loss”. ICCV2019
[4] Lvmin Zhang, et al., “Style Transfer for Anime Sketches with Enhanced Residual U-net and
Auxiliary Classifier GAN”. ACPR2017
[5] Phillip Isola, et al., “Image-to-Image Translation with Conditional Adversarial Nets”. CVPR2017
[6] Jie Hu, et al., “Squeeze-and-Excitation Networks”. CVPR2018
[7] Ming-Yu Liu, et al., “Few-Shot Unsupervised Image-to-Image Translation”. ICCV2019
[8] Illyasviel, “sketchKeras”. https://github.com/lllyasviel/sketchKeras
[9] Edgar Simo-Serra, et al., “Learning to Simplify: Fully Convolutional Networks for Rough Sketch
Cleanup” SIGGRAPH2016
[10] Holger Winnemoeller, et al., “XDoG: An eXtended difference-of-Gaussians compendium
including advanced image stylization” Computer & Graphics 36(6):740-753 2012