Yingxue Pang, Jianxin Lin, Tao Qin, Zhibo Chen, Image-to-Image Translation: Methods and Applications, IEEE Transactions on Multimedia, doi: 10.1109/TMM.2021.3109419, 2021. https://ieeexplore.ieee.org/document/9528943 https://arxiv.org/abs/2101.08629

1. Image-to-Image Translation:
Methods and Applications
Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen
IEEE Transactions on Multimedia, 2021
,
2022/6/17

2. ◼Image-to-Image translation (I2I)
I.
i.

3. ◼Image-to-Image translation (I2I)
•
•
•

4. ◼Variational Auto Encoder
[Kingma & Welling, arXiv2014]
• 𝓏 𝓍
𝑝𝜃(𝓍|𝓏)
• 𝓍 𝑝𝜃(𝓍)
• log 𝑝𝜃(𝓍)
◼Generative Adversarial Networks
[Goodfellow+, NeurIPS2014]
•
•
•

5. Generative Adversarial Network (GAN)
◼Unconditional GAN
• GAN [Goodfellow+, NeurIPS2014]
• DGANs [Radford+, ICLR2016]
•
◼Conditional GAN [Mirza & Osindero, arXiv2014]
•
•
or

6. ◼Image-to-Image translation (I2I)
I.
ii.

7. ◼ Peak signal-to-noise ratio (PSNR)
•
◼Structural similarity index (SSIM) [Wang+, IEEE2004]
• ground truth
◼Inception score (IS) [Salimans+, NeurIPS2016]
• Inception
◼Mask-SSIM and Mask-IS [Ma+, NeurIPS2017]
•
•

8. ◼Conditional inception score (CIS) [Huang+, ECCV2018]
• I2I
•
◼Perceptual distance (PD) [Johnson+, ECCV2016]
•
◼Fréchet inception distance (FID) [Heusel+, NeurIPS2017]
•

9. ◼Kernel inception distance (KID) [Binkowski+, ICLR2018]
• Inception network [Szegedy+, CVPR2016]
•
◼Single image Fréchet inception distance (SIFID) [Shaham+, ICCV2019]
•
• FID
◼Learned Perceptual Image Patch Similarity (LPIPS)
[Zhang+, CVPR2018]
•
•
•

10. ◼FCN scores [Isola+, CVPR2017]
• Semantic map
• IoU
◼Classification accuracy [Liu+, NeurIPS2017]
•
•
◼Density and Coverage (DC) [Naeem+, PMLR2020]
•
•

11. ◼Image-to-Image translation (I2I)
II. Two-domain I2I , Multi-domain I2I
i. Two-Domain Image-To-Image Translation

12. Two-Domain Image-To-Image Translation
◼Two-Domain I2I
• Image style transfer [Zhu+, ICCV2017]
• Semantic segmentation [Park+, CVPR2017]
• Image colorization [Suárez+, CVPR2017],
◼Two-Domain I2I
• Supervised
•
• Unsupervised
•
• Semi-supervised
•
• Few-shot
•
Image style transfer
Image colorization

13. Supervised I2I Single-modal Output
◼Single-modal Output
•
◼Pix2pix [Isola+, CVPR2017]
• Conditional GAN
• 𝐿1
◼Pix2PixHD [Wang+, CVPR2018]
• 2048 1024
◼SPADE [Park+, CVPR2019]
•
SPADE [Park+, CVPR2019]

14. Supervised I2I Multimodal Output
◼Multimodal Outputs
•
•
◼BicycleGAN [Zhu+, NeurIPS2017]
• cVAE-GAN [Hinton & Salakhutdinov, Science2006]
cLR-GAN [Chen+, NeurIPS2016]
•
• I2I
◼PixelNN [Bansal+, ICLR2018]
•
• ( ) PixelNN [Bansal+, ICLR2018]

15. Unsupervised I2I Single-modal Output
◼Translation using
a Cycle consistency Constraint
•
• Cycle-consistency Loss
•
loss
• CycleGAN [Zhu+, ICCV2017]
• 2
• U-GAT-IT [Kim+, ICLR2019]
•
•
CycleGAN [Zhu+, ICCV2017]

16. Unsupervised I2I Single-modal Output
◼Translation beyond Cycle-consistency Constraint
•
• TraVeLGAN [Amodio & Krishnaswamy, CVPR2019]
•
• CUT [Park+, ECCV2020]
•
•

17. Unsupervised I2I Single-modal Output
◼Translation of Fine-grained Objects
•
◼DAGAN [Ma+, CVPR2018]
◼Attention GAN
[Chen+, ECCV2018]
◼Attention guided GAN
[Mejjati+, NeurIPS2017]

18. Unsupervised I2I Single-modal Output
◼Translation by combining knowledge in other fields
•
◼RevGAN [Ouderaa & Worrall, CVPR2019]
◼Art2Real [Tomei+, CVPR2019]
◼GDWCT [Cho+, CVPR2019]
◼NICE-GAN [Chen+, CVPR2020]
Art2Real [Tomei+, CVPR2019]

19. Unsupervised I2I Multimodal Output
◼CycleGAN
• DSVIB [Kazemi+, NeurIPS2018]
• Augmented CycleGAN [Almahairi+, PMLR2018]
◼Disentangled representations
• cd-GAN [Lin+, CVPR2018]
• MUNIT [Huang+, ECCV2018]
• DRIT [Lee+, ECCV2018]
• EGSC-IT [Ma+, ICLR2018]
◼
• INIT [Shen+, CVPR2019]
• DSMAP [Chang+, ECCV2020]
DRIT [Lee+, ECCV2018]

20. Semi-supervised Image-to-Image Translation
◼TCR-SSIT [Mustafa & Mantiuk, ECCV2020]
• Transformation Consistency Regularization (TCR)
•

21. Few-Shot Image-to-Image Translation
◼ I2I
◼Transferring GAN
[Wang+, ECCV2018]
• GAN
•
◼MT-GAN [Lin+, arXiv2019]
•
•

22. One-shot Image-to-Image Translation
◼
• OST [Benaim & Wolf, NeurIPS2018]
•
•
• BiOST [Cohen & Wolf, ICCV2019]
•
•
◼
• TuiGAN [Lin+, ECCV2020]
•
• coarse-to-fine

23. ◼Image-to-Image translation (I2I)
II. Two-domain I2I , Multi-domain I2I
ii. Multi-Domain Image-To-Image Translation

24. Unsupervised I2I Single-modal Output
◼Training with multi-modules
• 2 I2I
•
• Domain-Bank [Hui+, ICPR2018]
• n
n
• ModularGAN [Zhao+, ECCV2018]
•

25. Training with one generator and discriminator pair
◼Training with one generator and discriminator pair
•
•
• StarGAN [Choi+, CVPR2018]
•
•
• AttGAN [He+, IEEE TIP2019]
•
•
• RelGAN [Wu+, ICCV2019] STGAN [Liu+, CVPR2019]
•
•
StarGAN, RelGAN

26. Training by combining knowledge in other fields
◼Training by combining knowledge in other fields
•
• Fixed-Point GAN [Siddiquee+, ICCV2019]
•
• SGN [Chang+, ICCV2019]
•
sym-parameter
• INIT [Cao+, ECCV2020]
•
• ADSPM [Wu+, ICCV2019]
•
SGN Structure

27. Unsupervised I2I Multimodal Output
◼ DosGAN [Lin+, arXiv2019]
• CNN
•
◼ GANimation [Pumarola+, ECCV2018]
•
•
◼ Disentanglement assumption
• UFDN [Liu+, NeurIPS2018]
• DMIT [Yu+, NeurIPS2019]
• StarGAN v2 [Choi+, CVPR2020]
• DRIT++ [Lee+, ECCV2018]
• GMM-UNIT [Liu+, arXiv2020]

28. Semi-Supervised Multi-domain I2I
◼SEMIT [Wang+, CVPR2020]
• Few-shot I2I
•
◼AGUIT [Li+, arXiv2019]
1.
2. AdaIN [Huang+, ICCV2017]
3. Cycle-consistency Loss Feature-consistency Loss

29. Few-shot Multi-domain I2I
◼FUNIT [Liu+, ICCV2019]
•
• I2I
• I2I
◼COCO-FUNIT [Saito+, ECCV2020]
•
•
◼ZstGAN [Lin+, arXiv2019]
• zero-shot I2I
•

30. ◼Image-to-Image translation (I2I)
II. Two-domain I2I , Multi-domain I2I
iii.

31. Experimental Evaluation
◼ Two-Domain I2I
• Pix2pix [Isola+, CVPR2017]
• BicycleGAN [Zhu+, NeurIPS2017]
• CycleGAN [Zhu+, ICCV2017]
• U-GAT-IT [Kim+, ICLR2019]
• GDWCT [Cho+, CVPR2019]
• CUT [Park+, ECCV2020]
• MUNIT [Huang+, ECCV2018]
◼ Multi-Domain I2I
• StarGAN [Choi+, CVPR2018]
• AttGAN [He+, IEEE TIP2019]
• STGAN [Liu+, CVPR2019]
• DosGAN [Lin+, arXiv2019]
• StarGANv2 [Choi+, CVPR2020]

32. ◼UT-Zap50K [Yu & Grauman, CVPR2019]
•
• 49826
• 200
• 256 256
◼CelebA [Liu, ICCV2015]
• 202599
• 40
• train, val, test 8:1:1
• 178 218 center crop
•

33. ◼Inception score (IS) [Salimans+, NeurIPS2016]
•
◼Fréchet inception distance (FID) [Heusel+, NeurIPS2017]
•
◼LPIPS [Zhang+, CVPR2018]
• LPIPS
•

34. Two-Domain I2I
◼Single-modal ◼Multimodal

35. Two-Domain I2I
◼Single-modal
•
• CUT FID IS
•
• StyleGAN
[Karras+, CVPR2019]
◼Multimodal
•
• LPIPS 0.047

36. MultiDomain I2I
◼Single-modal ◼Multimodal

37. MultiDomain I2I
◼StarGANv2
• FID IS LPIPS StarGANv2
• LPIPS 0.336

38. ◼Image-to-Image translation (I2I)
III. I2I
i. I2I
ii.

39. Application
◼
• [Park+, CVPR2019]
• [Yan+, ACM2017]
•
• [Isola+, CVPR2017]
• [Almahairi+, PMLR2018]
• [Mao+, CVPR2019]
• [Shocher+, CVPR2020]
• [Pathak+, CVPR2016]
• [Zheng+, AAAI2019]

40. ◼
• [Hicsinmez+, Image and Vision Computing, 2020]
• [Taigman+, ICLR2019]
◼
• [Zhu+, ICCV2017]
• [Kim+, ICML2017]
◼
• [Yi+, ICCV2017]
• [Park+, NeurIPS2020]

41. ◼
• [Yuan+, CVPR2018]
• [Manakov+, DART2019]
• [Zhang+, TCSVT2020]
• [Engin+, CVPRW2019]
• [Madam+, ECCV2018]

42. ◼
• I2I
◼
•

44. ◼AMT perceptual studies
• Amazon Mechanical Turk (AMT)
•
•

45. Two-domain, Multi-domain I2Iの分類

46. ◼Inception score (IS) Fréchet inception distance (FID)
• Two-Domain & Single-modal
•
• Two-Domain & Multimodal
• 19 FID
• Multi-Domain & Single-modal
• FID
• Multi-Domain & Multimodal
• 19
• FID