5. 많은 이미지 변환 기법들이 있지만..
Generative Adversarial Networks(GANs)
Artistic/Photo Style Transfer
Deep Image Analogy
Variational Autoencoder (VAE)
Image Warping / Morphing
Texture Synthesis
…
26. Multi-modal Translation
MUNIT (ECCV2018)
DRIT (ECCV2018)
Lee, Hsin-Ying, et al. "Diverse image-to-image translation via disentangled representations." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
35. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
36. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
가장 잘됨!
37. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
가장 잘됨!
38. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
가장 잘됨!
배경색이 변하긴
하지만, 잘 되는 편..
39. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
가장 잘됨!
배경색이 변하긴
하지만, 잘 되는 편..
40. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
가장 잘됨!
배경색이 변하긴
하지만, 잘 되는 편..
Aligned Data에서는 잘 되지만,
Wild Image에서는 실패함.
41. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
혼합된 형태의 문제도 있음.
42. Style Transfer Object Transfiguration
with Geometry Change
Object Transfiguration
without Geometry Change
Aligned Data
Unaligned Data
(In the Wild)
혼합된 형태의 문제도 있음.
48. Attention GAN
Attention
Chen, Xinyuan, et al. "Attention-GAN for object transfiguration in wild images." Proceedings of the
European Conference on Computer Vision (ECCV). 2018.
50. TransGaGaWu, Wayne, et al. "Transgaga: Geometry-aware unsupervised image-to-image
translation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
63. U-GAT-IT:
Unsupervised Generative ATtentional Networks with Adaptive
Layer-Instance Normalization for Image-to-Image Translation
Junho Kim, MinJae Kim, Hyeonwoo Kang, Kwang Hee Lee
https://github.com/taki0112/UGATIT
https://arxiv.org/abs/1907.10830
65. Summary
- Propose a novel I2I model with the Attention module and the AdaLIN.
- Attention module: distinguish between source and target domains
- Guide the translation focusing on more important regions
- AdaLIN: flexible control the amount of change in shape and texture without
tuning (model architecture or hyperparameters)
- Learnable normalization function
66. Background: Class Activation Mapping(CAM)
Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of
the IEEE conference on computer vision and pattern recognition. 2016.
69. Background : BIN(Batch-Instance Normalization)
Nam, Hyeonseob, and Hyo-Eun Kim. "Batch-instance normalization for adaptively style-
invariant neural networks." Advances in Neural Information Processing Systems. 2018.
70. Background : BIN(Batch-Instance Normalization)
Nam, Hyeonseob, and Hyo-Eun Kim. "Batch-instance normalization for adaptively style-
invariant neural networks." Advances in Neural Information Processing Systems. 2018.
BN과 IN을 잘 결합하면 중요한 style은 유지하고
불필요한 style은 제거하는 효과를 가져옴.
71. Motivation : IN & LN
Instance Normalization Layer Normalization
▪ 주로 Image generation/style transfer에서 좋은
성능을 냄.
▪ Channel–wise statistics를 고려함.
▪ 상대적으로 source domain content의
structure를 잘 유지함.
▪ Global statistics를 고려함.
▪ 상대적으로 source domain content의 structure를
덜 유지하고, target domain의 style을 더 잘 반영함.
72. Motivation : IN & LN
Instance Normalization Layer Normalization
▪ 주로 Image generation/style transfer에서 좋은
성능을 냄.
▪ Channel–wise statistics를 고려함.
▪ 상대적으로 source domain content의
structure를 잘 유지함.
▪ Global statistics를 고려함.
▪ 상대적으로 source domain content의 structure를
덜 유지하고, target domain의 style을 더 잘 반영함.
BIN처럼 잘 합치면, task에 따라서 적절히
normalization을 수행 할 수 있지 않을까?
83. Conclusion and Future Work
- Attention module 과 새로운 learnable normalization 기법인 AdaLIN 제안.
- CAM기반의 attention module은 두 도메인간의 차이가 큰 영역을 더 잘
바꾸도록 가이드해 준다.
- AdaLIN을 통해서 shape과 style 변환의 양을 model이나 hyper-parameter의
변경 없이 조절할 수 있었다.
- Future work: 더 wild한 이미지 간의 변환, multi-mapping image translation