SlideShare a Scribd company logo
Unsupervised Image-to-Image Translation
via Pre-trained StyleGAN2 Network
Kwang Hee Lee
Abstract
• Traditional I2I translation
✓ Train data in two or more domains together
✓ Require lots of computation resources
✓ Lower quality and many artifacts
✓ Training process could be unstable when the data in different domains are not balanced
✓ Modal collapse is more likely to happen
• Proposed a new I2I translation method
✓ Generates a new model in the target domain
✓ via a series of model transformations
✓ on a pretrained StyleGAN2 model in the source domain
✓ Proposed an inversion method
Related Works
Image Translation
Paired Unpaired
Unimodal Multi-modal Unimodal Multi-modal Multi-domain Multi-mapping
Pix2Pix
Pix2PixHD
BicycleGAN CycleGAN
DiscoGAN
UNIT
MUNIT
DRIT
StarGAN DRIT++
DMIT
SDIT
StarGAN
v2
Related Works
Multi-modal
Multi-domain
Related Works-Multi-domain Translation
StarGAN (CVPR2018)
StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation Yunjey Choi1,2, Minje Choi1,2, Munyoung Kim2,3, Jung-Woo Ha2, Sung Kim2,4,
and Jaegul Choo1,2 IEEE CVPR, 2018 (Oral)
Related Works-Multi-modal Translation
MUNIT (ECCV2018)
DRIT (ECCV2018)
Lee, Hsin-Ying, et al. "Diverse image-to-image translation via disentangled representations." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
Related Works-Multi-mapping Translation
Lee, Hsin-Ying, et al. "DRIT++: Diverse Image-to-Image Translation via Disentangled Representations." arXiv preprint arXiv:1905.01270 (2019).
DRIT++(arXiv2019)
Related Works-Examplar Guided I2I
Wang, Miao, et al. "Example-guided style-consistent image synthesis from semantic
labeling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
Zhu, Zhen, et al. "Progressive pose attention transfer for person image
generation." Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. 2019.
Related Works-Current I2I
• Require online training on at least two domains
• Domain-distinguished generators and discriminators
• High demand on training resources in terms of both time and memory
• Modal collapse when training data are not balanced in any domains
Related Works-StyleGAN-based I2I
Related Works-GAN Inversion Mapping
Zhu, Jiapeng, et al. "In-domain gan inversion for real image editing." European
conference on computer vision. Springer, Cham, 2020.
Goal:
Image Editing
Image2Image Translation
Major Contributions
• Define the distance between two models to measure the semantic similarity between
two images generated by two models, based on the same input latent vector
• Propose an unsupervised I2I translation method via a pre-trained StyleGAN2 model
• Support multi-modal and multi-domain I2I translation
• Drastically improved the results and requires much less training resource
• Proposed an inversion method is based on an embedded GAN space providing a
boundary constraint for searching the latent code of the input image
Method
• 가정: source domain 과 target domain의 모델간 semantic similarity가 작을 경우에 좋은
결과를 얻을 수 있음.
• Worlflow
• Pretrained StyleGAN2 model on source domain dataset Ds
• Fine-tuning target domain model Gt using source domain model Gs and target dataset Dt
• Source domain의 image가 주어지면, source domain model로 부터 latent code를 찾음.
• 찾아진 latent code를 fine tuning된 target domain model에 넣어서 image를 생성함.
Method
• Fine-tuning with data in target domain Dt
• Freeze the FC layers in fine-tuning process
• To have the same embedded space as the base model
Fine tuning process
Fine-tuned model
Method
• During fine-tuning, the semantic similarity decreases due to domain difference.
• Layer-swapping: source domain 의 더 많은 feature 들을 유지시키기 위해서
• Source domain model과 target domain model 사이의 model distance를 줄여주는 역할
Method
• Model Transform
Fine-tuning
Layer-swapping
Model
Transformation
Method
• Image-to-code inversion
Method
From StyleGAN2 paper:
Problem: w가 embedded space에 속하는지 보장하지 못함.
Method
The difference between two latent codes:
(n: a specific semantic attribute)
(n에 대한 편미분)
Method
Method
• Multi-modal and Multi-domain I2I translation
• Style code injected at the higher layers(8x8, 16x16,…) can change a major structure of the
output (identity, hair style, face shape)
• Style code applied at lower layers can modify only minor features(color, light conditions and
other micro-structures)
• Multi-modal (다양한 스타일 생성)
• Multi-domain (다양한 도메인 생성)
를 통해서 을 얻을 수 있음.
Experiment
• Baseline models
• CycleGAN
• MUNIT
• DRIT++
• Different scenarios
• Face2portrait
• Face2cartoon
• Face2anime
• Cat2dog
• Cat2wild
• FFHQ for face cases, AFHQ for cats, dogs and wild animals
• Portrait (wikiart), anime(Danbooru2018) and cartoon(Toonify)
Expriment
• Implementation Details
• A2B
• Freeze FC part of the A model
• Fine-tuning: 12000(low resolution cases), 20000(high resolution cases) iterations
• 1024 x 1024 (face2portrait, face2cartoon), 256 x 256 (face2anime)
• 2080Ti GPU(11G), fine-tuning 2 days
• Kept the training strategy and loss functions the same as those in the original StyleGAN2
• Layer-swap: 0.3ms, 8x8, 16x16, 32x32 ,
• Inversion process: 1000 iterations, 0.8s~1s
Experiment
Experiment-Multi-modal, Multi-domain
Experiment-Reference guided image generation
Experiment-Ablation Study of Fine-tuning
Experiment
• FT(Fine-tuning)
• FC(Freeze FC layer)
• LS(Layer Swap)
Experiment-Layer Swap Analysis
Experiment-Model Distance
[55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of
deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 586–595, 2018.
Experiment
Experiment
• 2000 test images chosen randomly
• LPIPSd (Diversity): LPIPS between two randomly selected images in the generated results
• LPIPSs (Semantic Similarity): LPIPS between the generated image and the input
Experiment-Inversion Methods
Thank you.
StyleGAN

More Related Content

What's hot

Automatic Image Annotation (AIA)
Automatic Image Annotation (AIA)Automatic Image Annotation (AIA)
Automatic Image Annotation (AIA)
Farzaneh Rezaei
 
Step zhedong
Step zhedongStep zhedong
Step zhedong
哲东 郑
 
Modeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networksModeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networks
NAVER Engineering
 
Obscenity Detection in Images
Obscenity Detection in ImagesObscenity Detection in Images
Obscenity Detection in Images
Anil Kumar Gupta
 
Deep learning for person re-identification
Deep learning for person re-identificationDeep learning for person re-identification
Deep learning for person re-identification
哲东 郑
 
Face Detection System on Ada boost Algorithm Using Haar Classifiers
Face Detection System on Ada boost Algorithm Using Haar ClassifiersFace Detection System on Ada boost Algorithm Using Haar Classifiers
Face Detection System on Ada boost Algorithm Using Haar Classifiers
IJMER
 
[CVPR2020] Simple but effective image enhancement techniques
[CVPR2020] Simple but effective image enhancement techniques[CVPR2020] Simple but effective image enhancement techniques
[CVPR2020] Simple but effective image enhancement techniques
JaeJun Yoo
 
Color based image processing , tracking and automation using matlab
Color based image processing , tracking and automation using matlabColor based image processing , tracking and automation using matlab
Color based image processing , tracking and automation using matlab
Kamal Pradhan
 
Pratik ibm-open power-ppt
Pratik ibm-open power-pptPratik ibm-open power-ppt
Pratik ibm-open power-ppt
Vaibhav R
 
IRJET- Concepts, Methods and Applications of Neural Style Transfer: A Rev...
IRJET-  	  Concepts, Methods and Applications of Neural Style Transfer: A Rev...IRJET-  	  Concepts, Methods and Applications of Neural Style Transfer: A Rev...
IRJET- Concepts, Methods and Applications of Neural Style Transfer: A Rev...
IRJET Journal
 
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
Dongmin Choi
 
High PSNR Based Image Steganography
High PSNR Based Image SteganographyHigh PSNR Based Image Steganography
High PSNR Based Image Steganography
rahulmonikasharma
 
Image analytics - A Primer
Image analytics - A PrimerImage analytics - A Primer
Image analytics - A Primer
Gopi Krishna Nuti
 
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
IRJET Journal
 
Image Maximization Using Multi Spectral Image Fusion Technique
Image Maximization Using Multi Spectral Image Fusion TechniqueImage Maximization Using Multi Spectral Image Fusion Technique
Image Maximization Using Multi Spectral Image Fusion Technique
dbpublications
 
Hao_Guan_2011
Hao_Guan_2011Hao_Guan_2011
Hao_Guan_2011
Hao Guan
 
An Introduction to Computer Vision
An Introduction to Computer VisionAn Introduction to Computer Vision
An Introduction to Computer Vision
guestd1b1b5
 
Road signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencvRoad signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencv
MohdSalim34
 
Leveraging Deep Learning Representation for search-based Image Annotation
Leveraging Deep Learning Representation for search-based Image AnnotationLeveraging Deep Learning Representation for search-based Image Annotation
Leveraging Deep Learning Representation for search-based Image Annotation
mahyamk
 
Strategy for Foreground Movement Identification Adaptive to Background Variat...
Strategy for Foreground Movement Identification Adaptive to Background Variat...Strategy for Foreground Movement Identification Adaptive to Background Variat...
Strategy for Foreground Movement Identification Adaptive to Background Variat...
IJECEIAES
 

What's hot (20)

Automatic Image Annotation (AIA)
Automatic Image Annotation (AIA)Automatic Image Annotation (AIA)
Automatic Image Annotation (AIA)
 
Step zhedong
Step zhedongStep zhedong
Step zhedong
 
Modeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networksModeling perceptual similarity and shift invariance in deep networks
Modeling perceptual similarity and shift invariance in deep networks
 
Obscenity Detection in Images
Obscenity Detection in ImagesObscenity Detection in Images
Obscenity Detection in Images
 
Deep learning for person re-identification
Deep learning for person re-identificationDeep learning for person re-identification
Deep learning for person re-identification
 
Face Detection System on Ada boost Algorithm Using Haar Classifiers
Face Detection System on Ada boost Algorithm Using Haar ClassifiersFace Detection System on Ada boost Algorithm Using Haar Classifiers
Face Detection System on Ada boost Algorithm Using Haar Classifiers
 
[CVPR2020] Simple but effective image enhancement techniques
[CVPR2020] Simple but effective image enhancement techniques[CVPR2020] Simple but effective image enhancement techniques
[CVPR2020] Simple but effective image enhancement techniques
 
Color based image processing , tracking and automation using matlab
Color based image processing , tracking and automation using matlabColor based image processing , tracking and automation using matlab
Color based image processing , tracking and automation using matlab
 
Pratik ibm-open power-ppt
Pratik ibm-open power-pptPratik ibm-open power-ppt
Pratik ibm-open power-ppt
 
IRJET- Concepts, Methods and Applications of Neural Style Transfer: A Rev...
IRJET-  	  Concepts, Methods and Applications of Neural Style Transfer: A Rev...IRJET-  	  Concepts, Methods and Applications of Neural Style Transfer: A Rev...
IRJET- Concepts, Methods and Applications of Neural Style Transfer: A Rev...
 
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
Review : Structure Boundary Preserving Segmentation
for Medical Image with Am...
 
High PSNR Based Image Steganography
High PSNR Based Image SteganographyHigh PSNR Based Image Steganography
High PSNR Based Image Steganography
 
Image analytics - A Primer
Image analytics - A PrimerImage analytics - A Primer
Image analytics - A Primer
 
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
IRJET- Transformation of Realistic Images and Videos into Cartoon Images and ...
 
Image Maximization Using Multi Spectral Image Fusion Technique
Image Maximization Using Multi Spectral Image Fusion TechniqueImage Maximization Using Multi Spectral Image Fusion Technique
Image Maximization Using Multi Spectral Image Fusion Technique
 
Hao_Guan_2011
Hao_Guan_2011Hao_Guan_2011
Hao_Guan_2011
 
An Introduction to Computer Vision
An Introduction to Computer VisionAn Introduction to Computer Vision
An Introduction to Computer Vision
 
Road signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencvRoad signs detection using voila jone's algorithm with the help of opencv
Road signs detection using voila jone's algorithm with the help of opencv
 
Leveraging Deep Learning Representation for search-based Image Annotation
Leveraging Deep Learning Representation for search-based Image AnnotationLeveraging Deep Learning Representation for search-based Image Annotation
Leveraging Deep Learning Representation for search-based Image Annotation
 
Strategy for Foreground Movement Identification Adaptive to Background Variat...
Strategy for Foreground Movement Identification Adaptive to Background Variat...Strategy for Foreground Movement Identification Adaptive to Background Variat...
Strategy for Foreground Movement Identification Adaptive to Background Variat...
 

Similar to Unsupervised image to-image translation via pre-trained style gan2 network

"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr..."Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
Edge AI and Vision Alliance
 
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
Maurice Nsabimana
 
Smart environment for industry 4.0
Smart environment for industry 4.0Smart environment for industry 4.0
Smart environment for industry 4.0
JawadSajid2
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2
Vitaly Bondar
 
How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...
Wee Hyong Tok
 
Deep Generative Modelling
Deep Generative ModellingDeep Generative Modelling
Deep Generative Modelling
Petko Nikolov
 
HDF-EOS Java Application Programming Interfaces
HDF-EOS Java Application Programming InterfacesHDF-EOS Java Application Programming Interfaces
HDF-EOS Java Application Programming Interfaces
The HDF-EOS Tools and Information Center
 
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
Dongmin Choi
 
Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019
Aditya Bhattacharya
 
Image to image translation with Pix2Pix GAN
Image to image translation with Pix2Pix GANImage to image translation with Pix2Pix GAN
Image to image translation with Pix2Pix GAN
S.Shayan Daneshvar
 
Efficient Image Processing - Nicolas Roard
Efficient Image Processing - Nicolas RoardEfficient Image Processing - Nicolas Roard
Efficient Image Processing - Nicolas Roard
Paris Android User Group
 
OReilly AI Transfer Learning
OReilly AI Transfer LearningOReilly AI Transfer Learning
OReilly AI Transfer Learning
Danielle Dean
 
Strata London - Deep Learning 05-2015
Strata London - Deep Learning 05-2015Strata London - Deep Learning 05-2015
Strata London - Deep Learning 05-2015
Turi, Inc.
 
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr TeterwakLearn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
PyData
 
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
Catalina Arango
 
How Will AI Change the Role of the Data Scientist?
How Will AI Change the Role of the Data Scientist?How Will AI Change the Role of the Data Scientist?
How Will AI Change the Role of the Data Scientist?
Hugo Gävert
 
Project_Final_Review.pdf
Project_Final_Review.pdfProject_Final_Review.pdf
Project_Final_Review.pdf
DivyaGugulothu
 
MongoDB in FS
MongoDB in FSMongoDB in FS
MongoDB in FS
MongoDB
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMs
SylvainGugger
 
DiscoGAN
DiscoGANDiscoGAN
DiscoGAN
Il Gu Yi
 

Similar to Unsupervised image to-image translation via pre-trained style gan2 network (20)

"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr..."Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
"Enabling Ubiquitous Visual Intelligence Through Deep Learning," a Keynote Pr...
 
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
 
Smart environment for industry 4.0
Smart environment for industry 4.0Smart environment for industry 4.0
Smart environment for industry 4.0
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2
 
How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...
 
Deep Generative Modelling
Deep Generative ModellingDeep Generative Modelling
Deep Generative Modelling
 
HDF-EOS Java Application Programming Interfaces
HDF-EOS Java Application Programming InterfacesHDF-EOS Java Application Programming Interfaces
HDF-EOS Java Application Programming Interfaces
 
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
Review : Multi-Domain Image Completion for Random Missing Input Data [cdm]
 
Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019Computer vision-must-nit-silchar-ml-hackathon-2019
Computer vision-must-nit-silchar-ml-hackathon-2019
 
Image to image translation with Pix2Pix GAN
Image to image translation with Pix2Pix GANImage to image translation with Pix2Pix GAN
Image to image translation with Pix2Pix GAN
 
Efficient Image Processing - Nicolas Roard
Efficient Image Processing - Nicolas RoardEfficient Image Processing - Nicolas Roard
Efficient Image Processing - Nicolas Roard
 
OReilly AI Transfer Learning
OReilly AI Transfer LearningOReilly AI Transfer Learning
OReilly AI Transfer Learning
 
Strata London - Deep Learning 05-2015
Strata London - Deep Learning 05-2015Strata London - Deep Learning 05-2015
Strata London - Deep Learning 05-2015
 
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr TeterwakLearn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
 
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
Let's paint a Picasso - A Look at Generative Adversarial Networks (GAN) and i...
 
How Will AI Change the Role of the Data Scientist?
How Will AI Change the Role of the Data Scientist?How Will AI Change the Role of the Data Scientist?
How Will AI Change the Role of the Data Scientist?
 
Project_Final_Review.pdf
Project_Final_Review.pdfProject_Final_Review.pdf
Project_Final_Review.pdf
 
MongoDB in FS
MongoDB in FSMongoDB in FS
MongoDB in FS
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMs
 
DiscoGAN
DiscoGANDiscoGAN
DiscoGAN
 

Recently uploaded

Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
ElizabethGarrettChri
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
zsafxbf
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
eoxhsaa
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
NABLAS株式会社
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
9gr6pty
 
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative ClassifiersML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
MastanaihnaiduYasam
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
keesa2
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
dataschool1
 
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
slg6lamcq
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
nyvan3
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
eudsoh
 
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
asyed10
 
Digital Marketing Performance Marketing Sample .pdf
Digital Marketing Performance Marketing  Sample .pdfDigital Marketing Performance Marketing  Sample .pdf
Digital Marketing Performance Marketing Sample .pdf
Vineet
 
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
z6osjkqvd
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
jitskeb
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
Vietnam Cotton & Spinning Association
 
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdfOverview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
nhutnguyen355078
 
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理 原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
tzu5xla
 
Template xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptxTemplate xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptx
TeukuEriSyahputra
 
Drownings spike from May to August in children
Drownings spike from May to August in childrenDrownings spike from May to August in children
Drownings spike from May to August in children
Bisnar Chase Personal Injury Attorneys
 

Recently uploaded (20)

Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
 
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
 
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative ClassifiersML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
ML-PPT-UNIT-2 Generative Classifiers Discriminative Classifiers
 
一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理一比一原版悉尼大学毕业证如何办理
一比一原版悉尼大学毕业证如何办理
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
 
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
一比一原版南十字星大学毕业证(SCU毕业证书)学历如何办理
 
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
一比一原版英国赫特福德大学毕业证(hertfordshire毕业证书)如何办理
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
 
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
一比一原版美国帕森斯设计学院毕业证(parsons毕业证书)如何办理
 
Digital Marketing Performance Marketing Sample .pdf
Digital Marketing Performance Marketing  Sample .pdfDigital Marketing Performance Marketing  Sample .pdf
Digital Marketing Performance Marketing Sample .pdf
 
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
 
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
 
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdfOverview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
 
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理 原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
原版一比一爱尔兰都柏林大学毕业证(UCD毕业证书)如何办理
 
Template xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptxTemplate xxxxxxxx ssssssssssss Sertifikat.pptx
Template xxxxxxxx ssssssssssss Sertifikat.pptx
 
Drownings spike from May to August in children
Drownings spike from May to August in childrenDrownings spike from May to August in children
Drownings spike from May to August in children
 

Unsupervised image to-image translation via pre-trained style gan2 network

  • 1. Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network Kwang Hee Lee
  • 2. Abstract • Traditional I2I translation ✓ Train data in two or more domains together ✓ Require lots of computation resources ✓ Lower quality and many artifacts ✓ Training process could be unstable when the data in different domains are not balanced ✓ Modal collapse is more likely to happen • Proposed a new I2I translation method ✓ Generates a new model in the target domain ✓ via a series of model transformations ✓ on a pretrained StyleGAN2 model in the source domain ✓ Proposed an inversion method
  • 3. Related Works Image Translation Paired Unpaired Unimodal Multi-modal Unimodal Multi-modal Multi-domain Multi-mapping Pix2Pix Pix2PixHD BicycleGAN CycleGAN DiscoGAN UNIT MUNIT DRIT StarGAN DRIT++ DMIT SDIT StarGAN v2
  • 5. Related Works-Multi-domain Translation StarGAN (CVPR2018) StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation Yunjey Choi1,2, Minje Choi1,2, Munyoung Kim2,3, Jung-Woo Ha2, Sung Kim2,4, and Jaegul Choo1,2 IEEE CVPR, 2018 (Oral)
  • 6. Related Works-Multi-modal Translation MUNIT (ECCV2018) DRIT (ECCV2018) Lee, Hsin-Ying, et al. "Diverse image-to-image translation via disentangled representations." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  • 7. Related Works-Multi-mapping Translation Lee, Hsin-Ying, et al. "DRIT++: Diverse Image-to-Image Translation via Disentangled Representations." arXiv preprint arXiv:1905.01270 (2019). DRIT++(arXiv2019)
  • 8. Related Works-Examplar Guided I2I Wang, Miao, et al. "Example-guided style-consistent image synthesis from semantic labeling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. Zhu, Zhen, et al. "Progressive pose attention transfer for person image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  • 9. Related Works-Current I2I • Require online training on at least two domains • Domain-distinguished generators and discriminators • High demand on training resources in terms of both time and memory • Modal collapse when training data are not balanced in any domains
  • 11. Related Works-GAN Inversion Mapping Zhu, Jiapeng, et al. "In-domain gan inversion for real image editing." European conference on computer vision. Springer, Cham, 2020. Goal: Image Editing Image2Image Translation
  • 12. Major Contributions • Define the distance between two models to measure the semantic similarity between two images generated by two models, based on the same input latent vector • Propose an unsupervised I2I translation method via a pre-trained StyleGAN2 model • Support multi-modal and multi-domain I2I translation • Drastically improved the results and requires much less training resource • Proposed an inversion method is based on an embedded GAN space providing a boundary constraint for searching the latent code of the input image
  • 13. Method • 가정: source domain 과 target domain의 모델간 semantic similarity가 작을 경우에 좋은 결과를 얻을 수 있음. • Worlflow • Pretrained StyleGAN2 model on source domain dataset Ds • Fine-tuning target domain model Gt using source domain model Gs and target dataset Dt • Source domain의 image가 주어지면, source domain model로 부터 latent code를 찾음. • 찾아진 latent code를 fine tuning된 target domain model에 넣어서 image를 생성함.
  • 14. Method • Fine-tuning with data in target domain Dt • Freeze the FC layers in fine-tuning process • To have the same embedded space as the base model Fine tuning process Fine-tuned model
  • 15. Method • During fine-tuning, the semantic similarity decreases due to domain difference. • Layer-swapping: source domain 의 더 많은 feature 들을 유지시키기 위해서 • Source domain model과 target domain model 사이의 model distance를 줄여주는 역할
  • 18. Method From StyleGAN2 paper: Problem: w가 embedded space에 속하는지 보장하지 못함.
  • 19. Method The difference between two latent codes: (n: a specific semantic attribute) (n에 대한 편미분)
  • 21. Method • Multi-modal and Multi-domain I2I translation • Style code injected at the higher layers(8x8, 16x16,…) can change a major structure of the output (identity, hair style, face shape) • Style code applied at lower layers can modify only minor features(color, light conditions and other micro-structures) • Multi-modal (다양한 스타일 생성) • Multi-domain (다양한 도메인 생성) 를 통해서 을 얻을 수 있음.
  • 22. Experiment • Baseline models • CycleGAN • MUNIT • DRIT++ • Different scenarios • Face2portrait • Face2cartoon • Face2anime • Cat2dog • Cat2wild • FFHQ for face cases, AFHQ for cats, dogs and wild animals • Portrait (wikiart), anime(Danbooru2018) and cartoon(Toonify)
  • 23. Expriment • Implementation Details • A2B • Freeze FC part of the A model • Fine-tuning: 12000(low resolution cases), 20000(high resolution cases) iterations • 1024 x 1024 (face2portrait, face2cartoon), 256 x 256 (face2anime) • 2080Ti GPU(11G), fine-tuning 2 days • Kept the training strategy and loss functions the same as those in the original StyleGAN2 • Layer-swap: 0.3ms, 8x8, 16x16, 32x32 , • Inversion process: 1000 iterations, 0.8s~1s
  • 28. Experiment • FT(Fine-tuning) • FC(Freeze FC layer) • LS(Layer Swap)
  • 30. Experiment-Model Distance [55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  • 32. Experiment • 2000 test images chosen randomly • LPIPSd (Diversity): LPIPS between two randomly selected images in the generated results • LPIPSs (Semantic Similarity): LPIPS between the generated image and the input