SlideShare a Scribd company logo
1 of 42
Download to read offline
Analyzing and Improving the
Image Quality of StyleGAN
펀디멘탈 팀: 송헌, 고형권, 김동희, 김창연, 이민경, 이재윤
발표: 고형권 (hyungkwonko@gmail.com)
Progressive GAN (2018 ICLR) recap
Image from original paper
2
Image from original paper
StyleGAN (2019 CVPR) recap
3
Image from original paper
StyleGAN (2019 CVPR) recap
4
Image from original paper
StyleGAN (2019 CVPR) recap
5
https://github.com/justinpinkney/awesome-pretrained-stylegan2
Pretrained models for different datasets
6
Q & A
StyleGAN2 (2020 CVPR)
● In this paper, we focus all analysis solely on W, as it is the relevant latent
space from the synthesis network’s point of view.
8
StyleGAN2 (2020 CVPR)
Image from original paper
9
Image from original paper
The Problem of StyleGAN (Droplet artifacts)
10
The Problem of StyleGAN (Droplet artifacts)
https://www.youtube.com/watch?v=c-NJtV9Jvp0
11
B. Weight Demodulation
● AdaIN operation normalizes the mean and variance of each feature map
separately, thereby potentially destroying any information found in the
magnitudes of the features relative to each other.
● How to fix? Get rid of AdaIN structure
12
B. Weight Demodulation
● BEFORE (StyleGAN)
● W → affine transform → A
Image from original paper
13
B. Weight Demodulation
● AFTER (StyleGAN2)
Image from original paper
14
B. Weight Demodulation
Image from original paper
15
The Result of Weight Demodulation
https://www.youtube.com/watch?v=c-NJtV9Jvp0
16
The Result of Weight Demodulation (in style mixing)
https://www.youtube.com/watch?v=c-NJtV9Jvp0
17
Q & A
Perceptual Path Length score
● Perceptual Path Length score (PPL): Sampled from latent-space vectors using
interpolation → distance calculation using features of VGG
Image from original paper
19
Perceptual Path Length score
Image from original paper
20
Why LOW PPL == GOOD Quality?
● NOT immediately obvious…
● Hypothesis:
GOOD image region (STRETCHED, large latent space)
BAD image region (SQUEEZED, small latent space)
Image from original paper
21
Perceptual Path Length score
22
Why better than FID and Precision & Recall?
Geirhos et al., 2019 ICLR (oral)
23
D. Path Length Regularization
● We would like to encourage that a fixed-size step in W results in a non-zero,
fixed-magnitude change in the image.
● We can measure the deviation from this ideal empirically by stepping into
random directions in the image space and observing the corresponding w
gradients.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow.
Is generator conditioning causally related to GAN performance? CoRR, abs/1802.08768, 2018.
24
C. Lazy Regularization
● As the resulting regularization term is somewhat expensive to compute, we
first describe a general optimization that applies to any regularization
technique.
● … performed only once every 16 mini-batches
25
Q & A
The Problem of StyleGAN (Phase artifacts)
Image from original paper
27
The Problem of StyleGAN (Phase artifacts)
https://www.youtube.com/watch?v=c-NJtV9Jvp0
28
The Problem of StyleGAN (Phase artifacts)
● … We believe the problem is that in progressive growing each resolution
serves momentarily as the output resolution, forcing it to generate maximal
frequency details, which then leads to the trained network to have
excessively high frequencies in the intermediate layers, ...
29
MSG-GAN (2020 CVPR)
Image from original paper
30
E. No Growing, New G & D Architecture
Image from original paper
31
E. No Growing, New G & D Architecture
Image from original paper
32
The Result of New Architecture
https://www.youtube.com/watch?v=c-NJtV9Jvp0
33
F. Large Networks for Better Performance
… To verify this, we inspected the generated images
manually and noticed that they generally lack some of
the pixel-level detail...
Image from original paper
34
F. Large Networks for Better Performance
Image from original paper
35
Deep Fake Detection Algorithm
● How can we detect generated / synthesized image?
Image from original paper
36
Deep Fake Detection Algorithm
https://www.youtube.com/watch?v=c-NJtV9Jvp0
37
How to do this? LPIPS distance
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a
perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).
38
How to do this?
● (for StyleGAN) it appears that the
mapping from W to images is too
complex for this to succeed reliably in
practice.
● We find it encouraging that StyleGAN2
makes source attribution easier even
though the image quality has improved
significantly.
Image from original paper
39
https://github.com/justinpinkney/a
wesome-pretrained-stylegan2
Many pre-trained models exist
40
Outro
41
Q & A

More Related Content

What's hot

Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEDatabricks
 
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis taeseon ryu
 
Domain adaptation for Image Segmentation
Domain adaptation for Image SegmentationDomain adaptation for Image Segmentation
Domain adaptation for Image SegmentationDeepak Thukral
 
Stable Diffusion path
Stable Diffusion pathStable Diffusion path
Stable Diffusion pathVitaly Bondar
 
Generative Adversarial Networks and Their Medical Imaging Applications
Generative Adversarial Networks and Their Medical Imaging ApplicationsGenerative Adversarial Networks and Their Medical Imaging Applications
Generative Adversarial Networks and Their Medical Imaging ApplicationsKyuhwan Jung
 
Generative Adversarial Networks
Generative Adversarial NetworksGenerative Adversarial Networks
Generative Adversarial NetworksMustafa Yagmur
 
1 seaborn introduction
1 seaborn introduction 1 seaborn introduction
1 seaborn introduction YuleiLi3
 
Masked Autoencoders Are Scalable Vision Learners.pptx
Masked Autoencoders Are Scalable Vision Learners.pptxMasked Autoencoders Are Scalable Vision Learners.pptx
Masked Autoencoders Are Scalable Vision Learners.pptxSangmin Woo
 
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural imagesDeep Learning JP
 
Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks남주 김
 
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...홍배 김
 
Diffusion Deformable Model for 4D Temporal Medical Image Generation
Diffusion Deformable Model for 4D Temporal Medical Image GenerationDiffusion Deformable Model for 4D Temporal Medical Image Generation
Diffusion Deformable Model for 4D Temporal Medical Image GenerationBoahKim2
 
Evolution of the StyleGAN family
Evolution of the StyleGAN familyEvolution of the StyleGAN family
Evolution of the StyleGAN familyVitaly Bondar
 
GAN - Theory and Applications
GAN - Theory and ApplicationsGAN - Theory and Applications
GAN - Theory and ApplicationsEmanuele Ghelfi
 
[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation
[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation
[DL輪読会]Few-Shot Unsupervised Image-to-Image TranslationDeep Learning JP
 
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...Deep Learning JP
 
Image-to-Image Translation pix2pix
Image-to-Image Translation pix2pixImage-to-Image Translation pix2pix
Image-to-Image Translation pix2pixYasar Hayat
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Vitaly Bondar
 
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation..."Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...Edge AI and Vision Alliance
 
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Universitat Politècnica de Catalunya
 

What's hot (20)

Unified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIMEUnified Approach to Interpret Machine Learning Model: SHAP + LIME
Unified Approach to Interpret Machine Learning Model: SHAP + LIME
 
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
 
Domain adaptation for Image Segmentation
Domain adaptation for Image SegmentationDomain adaptation for Image Segmentation
Domain adaptation for Image Segmentation
 
Stable Diffusion path
Stable Diffusion pathStable Diffusion path
Stable Diffusion path
 
Generative Adversarial Networks and Their Medical Imaging Applications
Generative Adversarial Networks and Their Medical Imaging ApplicationsGenerative Adversarial Networks and Their Medical Imaging Applications
Generative Adversarial Networks and Their Medical Imaging Applications
 
Generative Adversarial Networks
Generative Adversarial NetworksGenerative Adversarial Networks
Generative Adversarial Networks
 
1 seaborn introduction
1 seaborn introduction 1 seaborn introduction
1 seaborn introduction
 
Masked Autoencoders Are Scalable Vision Learners.pptx
Masked Autoencoders Are Scalable Vision Learners.pptxMasked Autoencoders Are Scalable Vision Learners.pptx
Masked Autoencoders Are Scalable Vision Learners.pptx
 
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images
[DL輪読会]HoloGAN: Unsupervised learning of 3D representations from natural images
 
Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
 
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
 
Diffusion Deformable Model for 4D Temporal Medical Image Generation
Diffusion Deformable Model for 4D Temporal Medical Image GenerationDiffusion Deformable Model for 4D Temporal Medical Image Generation
Diffusion Deformable Model for 4D Temporal Medical Image Generation
 
Evolution of the StyleGAN family
Evolution of the StyleGAN familyEvolution of the StyleGAN family
Evolution of the StyleGAN family
 
GAN - Theory and Applications
GAN - Theory and ApplicationsGAN - Theory and Applications
GAN - Theory and Applications
 
[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation
[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation
[DL輪読会]Few-Shot Unsupervised Image-to-Image Translation
 
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...
[DL輪読会]Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval ...
 
Image-to-Image Translation pix2pix
Image-to-Image Translation pix2pixImage-to-Image Translation pix2pix
Image-to-Image Translation pix2pix
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2
 
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation..."Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
 
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
 

Similar to Style gan2 review

Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksFörderverein Technische Fakultät
 
Automatic Building detection for satellite Images using IGV and DSM
Automatic Building detection for satellite Images using IGV and DSMAutomatic Building detection for satellite Images using IGV and DSM
Automatic Building detection for satellite Images using IGV and DSMAmit Raikar
 
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...Universitat Politècnica de Catalunya
 
Finding connections among images using CycleGAN
Finding connections among images using CycleGANFinding connections among images using CycleGAN
Finding connections among images using CycleGANNAVER Engineering
 
Not Enough Measurements, Too Many Measurements
Not Enough Measurements, Too Many MeasurementsNot Enough Measurements, Too Many Measurements
Not Enough Measurements, Too Many MeasurementsMike McCann
 
Convolutional Neural Networks CNN
Convolutional Neural Networks CNNConvolutional Neural Networks CNN
Convolutional Neural Networks CNNAbdullah al Mamun
 
Contrast and resolution improvement of pocus using self consistent cycle gan
Contrast and resolution improvement of pocus using self consistent cycle ganContrast and resolution improvement of pocus using self consistent cycle gan
Contrast and resolution improvement of pocus using self consistent cycle ganShujaat Khan
 
Tutorial on Theory and Application of Generative Adversarial Networks
Tutorial on Theory and Application of Generative Adversarial NetworksTutorial on Theory and Application of Generative Adversarial Networks
Tutorial on Theory and Application of Generative Adversarial NetworksMLReview
 
Evaluation of conditional images synthesis: generating a photorealistic image...
Evaluation of conditional images synthesis: generating a photorealistic image...Evaluation of conditional images synthesis: generating a photorealistic image...
Evaluation of conditional images synthesis: generating a photorealistic image...SamanthaGallone
 
Unpaired Image Translations Using GANs: A Review
Unpaired Image Translations Using GANs: A ReviewUnpaired Image Translations Using GANs: A Review
Unpaired Image Translations Using GANs: A ReviewIRJET Journal
 
Decomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesisDecomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesisNaeem Shehzad
 
IRJET - mage Colorization using Self Attention GAN
IRJET - mage Colorization using Self Attention GANIRJET - mage Colorization using Self Attention GAN
IRJET - mage Colorization using Self Attention GANIRJET Journal
 
Image Translation with GAN
Image Translation with GANImage Translation with GAN
Image Translation with GANJunho Cho
 
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...MYEONGGYU LEE
 
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET Transform
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET TransformEnhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET Transform
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET TransformMonika Rout
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in VisionSangmin Woo
 
Project_Final_Review.pdf
Project_Final_Review.pdfProject_Final_Review.pdf
Project_Final_Review.pdfDivyaGugulothu
 
B-DCGAN Slides for ICONIP2019
B-DCGAN Slides for ICONIP2019B-DCGAN Slides for ICONIP2019
B-DCGAN Slides for ICONIP2019Hideo Terada
 

Similar to Style gan2 review (20)

Deep Generative Models - Kevin McGuinness - UPC Barcelona 2018
Deep Generative Models - Kevin McGuinness - UPC Barcelona 2018Deep Generative Models - Kevin McGuinness - UPC Barcelona 2018
Deep Generative Models - Kevin McGuinness - UPC Barcelona 2018
 
Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial Networks
 
Automatic Building detection for satellite Images using IGV and DSM
Automatic Building detection for satellite Images using IGV and DSMAutomatic Building detection for satellite Images using IGV and DSM
Automatic Building detection for satellite Images using IGV and DSM
 
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...
Generative Models and Adversarial Training (D2L3 Insight@DCU Machine Learning...
 
Finding connections among images using CycleGAN
Finding connections among images using CycleGANFinding connections among images using CycleGAN
Finding connections among images using CycleGAN
 
Not Enough Measurements, Too Many Measurements
Not Enough Measurements, Too Many MeasurementsNot Enough Measurements, Too Many Measurements
Not Enough Measurements, Too Many Measurements
 
Convolutional Neural Networks CNN
Convolutional Neural Networks CNNConvolutional Neural Networks CNN
Convolutional Neural Networks CNN
 
Contrast and resolution improvement of pocus using self consistent cycle gan
Contrast and resolution improvement of pocus using self consistent cycle ganContrast and resolution improvement of pocus using self consistent cycle gan
Contrast and resolution improvement of pocus using self consistent cycle gan
 
Tutorial on Theory and Application of Generative Adversarial Networks
Tutorial on Theory and Application of Generative Adversarial NetworksTutorial on Theory and Application of Generative Adversarial Networks
Tutorial on Theory and Application of Generative Adversarial Networks
 
Evaluation of conditional images synthesis: generating a photorealistic image...
Evaluation of conditional images synthesis: generating a photorealistic image...Evaluation of conditional images synthesis: generating a photorealistic image...
Evaluation of conditional images synthesis: generating a photorealistic image...
 
Unpaired Image Translations Using GANs: A Review
Unpaired Image Translations Using GANs: A ReviewUnpaired Image Translations Using GANs: A Review
Unpaired Image Translations Using GANs: A Review
 
Decomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesisDecomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesis
 
IRJET - mage Colorization using Self Attention GAN
IRJET - mage Colorization using Self Attention GANIRJET - mage Colorization using Self Attention GAN
IRJET - mage Colorization using Self Attention GAN
 
Image Translation with GAN
Image Translation with GANImage Translation with GAN
Image Translation with GAN
 
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...
(Paper Review)U-GAT-IT: unsupervised generative attentional networks with ada...
 
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET Transform
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET TransformEnhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET Transform
Enhancement Of Retinal Fundus Image Using 2-D GABOR WAVELET Transform
 
Transformer in Vision
Transformer in VisionTransformer in Vision
Transformer in Vision
 
MAHA DISS PROJECT 1-1.pptx
MAHA DISS PROJECT 1-1.pptxMAHA DISS PROJECT 1-1.pptx
MAHA DISS PROJECT 1-1.pptx
 
Project_Final_Review.pdf
Project_Final_Review.pdfProject_Final_Review.pdf
Project_Final_Review.pdf
 
B-DCGAN Slides for ICONIP2019
B-DCGAN Slides for ICONIP2019B-DCGAN Slides for ICONIP2019
B-DCGAN Slides for ICONIP2019
 

More from taeseon ryu

OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...taeseon ryu
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splattingtaeseon ryu
 
Hyperbolic Image Embedding.pptx
Hyperbolic  Image Embedding.pptxHyperbolic  Image Embedding.pptx
Hyperbolic Image Embedding.pptxtaeseon ryu
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정taeseon ryu
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdftaeseon ryu
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories taeseon ryu
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extractiontaeseon ryu
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learningtaeseon ryu
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Modelstaeseon ryu
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuningtaeseon ryu
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdftaeseon ryu
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdftaeseon ryu
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithmtaeseon ryu
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networkstaeseon ryu
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarizationtaeseon ryu
 

More from taeseon ryu (20)

VoxelNet
VoxelNetVoxelNet
VoxelNet
 
OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splatting
 
JetsonTX2 Python
 JetsonTX2 Python  JetsonTX2 Python
JetsonTX2 Python
 
Hyperbolic Image Embedding.pptx
Hyperbolic  Image Embedding.pptxHyperbolic  Image Embedding.pptx
Hyperbolic Image Embedding.pptx
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
 
YOLO V6
YOLO V6YOLO V6
YOLO V6
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories
 
RL_UpsideDown
RL_UpsideDownRL_UpsideDown
RL_UpsideDown
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extraction
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learning
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Models
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuning
 
mPLUG
mPLUGmPLUG
mPLUG
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithm
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networks
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarization
 

Recently uploaded

The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionJadeNovelo1
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...Chayanika Das
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024Jene van der Heide
 
cybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationcybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationSanghamitraMohapatra5
 
Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxzeus70441
 
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11GelineAvendao
 
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2AuEnriquezLontok
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxtuking87
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests GlycosidesNandakishor Bhaurao Deshmukh
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPirithiRaju
 
Unveiling the Cannabis Plant’s Potential
Unveiling the Cannabis Plant’s PotentialUnveiling the Cannabis Plant’s Potential
Unveiling the Cannabis Plant’s PotentialMarkus Roggen
 
Science (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsScience (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsDobusch Leonhard
 
whole genome sequencing new and its types including shortgun and clone by clone
whole genome sequencing new  and its types including shortgun and clone by clonewhole genome sequencing new  and its types including shortgun and clone by clone
whole genome sequencing new and its types including shortgun and clone by clonechaudhary charan shingh university
 
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Christina Parmionova
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Environmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxEnvironmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxpriyankatabhane
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsDanielBaumann11
 
Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxpriyankatabhane
 

Recently uploaded (20)

Introduction Classification Of Alkaloids
Introduction Classification Of AlkaloidsIntroduction Classification Of Alkaloids
Introduction Classification Of Alkaloids
 
The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and Function
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
 
cybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationcybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitation
 
Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptx
 
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
 
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
LESSON PLAN IN SCIENCE GRADE 4 WEEK 1 DAY 2
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPR
 
Unveiling the Cannabis Plant’s Potential
Unveiling the Cannabis Plant’s PotentialUnveiling the Cannabis Plant’s Potential
Unveiling the Cannabis Plant’s Potential
 
Science (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsScience (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and Pitfalls
 
whole genome sequencing new and its types including shortgun and clone by clone
whole genome sequencing new  and its types including shortgun and clone by clonewhole genome sequencing new  and its types including shortgun and clone by clone
whole genome sequencing new and its types including shortgun and clone by clone
 
PLASMODIUM. PPTX
PLASMODIUM. PPTXPLASMODIUM. PPTX
PLASMODIUM. PPTX
 
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Environmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptxEnvironmental acoustics- noise criteria.pptx
Environmental acoustics- noise criteria.pptx
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
 
Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptx
 

Style gan2 review

  • 1. Analyzing and Improving the Image Quality of StyleGAN 펀디멘탈 팀: 송헌, 고형권, 김동희, 김창연, 이민경, 이재윤 발표: 고형권 (hyungkwonko@gmail.com)
  • 2. Progressive GAN (2018 ICLR) recap Image from original paper 2
  • 3. Image from original paper StyleGAN (2019 CVPR) recap 3
  • 4. Image from original paper StyleGAN (2019 CVPR) recap 4
  • 5. Image from original paper StyleGAN (2019 CVPR) recap 5
  • 8. StyleGAN2 (2020 CVPR) ● In this paper, we focus all analysis solely on W, as it is the relevant latent space from the synthesis network’s point of view. 8
  • 9. StyleGAN2 (2020 CVPR) Image from original paper 9
  • 10. Image from original paper The Problem of StyleGAN (Droplet artifacts) 10
  • 11. The Problem of StyleGAN (Droplet artifacts) https://www.youtube.com/watch?v=c-NJtV9Jvp0 11
  • 12. B. Weight Demodulation ● AdaIN operation normalizes the mean and variance of each feature map separately, thereby potentially destroying any information found in the magnitudes of the features relative to each other. ● How to fix? Get rid of AdaIN structure 12
  • 13. B. Weight Demodulation ● BEFORE (StyleGAN) ● W → affine transform → A Image from original paper 13
  • 14. B. Weight Demodulation ● AFTER (StyleGAN2) Image from original paper 14
  • 15. B. Weight Demodulation Image from original paper 15
  • 16. The Result of Weight Demodulation https://www.youtube.com/watch?v=c-NJtV9Jvp0 16
  • 17. The Result of Weight Demodulation (in style mixing) https://www.youtube.com/watch?v=c-NJtV9Jvp0 17
  • 18. Q & A
  • 19. Perceptual Path Length score ● Perceptual Path Length score (PPL): Sampled from latent-space vectors using interpolation → distance calculation using features of VGG Image from original paper 19
  • 20. Perceptual Path Length score Image from original paper 20
  • 21. Why LOW PPL == GOOD Quality? ● NOT immediately obvious… ● Hypothesis: GOOD image region (STRETCHED, large latent space) BAD image region (SQUEEZED, small latent space) Image from original paper 21
  • 23. Why better than FID and Precision & Recall? Geirhos et al., 2019 ICLR (oral) 23
  • 24. D. Path Length Regularization ● We would like to encourage that a fixed-size step in W results in a non-zero, fixed-magnitude change in the image. ● We can measure the deviation from this ideal empirically by stepping into random directions in the image space and observing the corresponding w gradients. Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to GAN performance? CoRR, abs/1802.08768, 2018. 24
  • 25. C. Lazy Regularization ● As the resulting regularization term is somewhat expensive to compute, we first describe a general optimization that applies to any regularization technique. ● … performed only once every 16 mini-batches 25
  • 26. Q & A
  • 27. The Problem of StyleGAN (Phase artifacts) Image from original paper 27
  • 28. The Problem of StyleGAN (Phase artifacts) https://www.youtube.com/watch?v=c-NJtV9Jvp0 28
  • 29. The Problem of StyleGAN (Phase artifacts) ● … We believe the problem is that in progressive growing each resolution serves momentarily as the output resolution, forcing it to generate maximal frequency details, which then leads to the trained network to have excessively high frequencies in the intermediate layers, ... 29
  • 30. MSG-GAN (2020 CVPR) Image from original paper 30
  • 31. E. No Growing, New G & D Architecture Image from original paper 31
  • 32. E. No Growing, New G & D Architecture Image from original paper 32
  • 33. The Result of New Architecture https://www.youtube.com/watch?v=c-NJtV9Jvp0 33
  • 34. F. Large Networks for Better Performance … To verify this, we inspected the generated images manually and noticed that they generally lack some of the pixel-level detail... Image from original paper 34
  • 35. F. Large Networks for Better Performance Image from original paper 35
  • 36. Deep Fake Detection Algorithm ● How can we detect generated / synthesized image? Image from original paper 36
  • 37. Deep Fake Detection Algorithm https://www.youtube.com/watch?v=c-NJtV9Jvp0 37
  • 38. How to do this? LPIPS distance Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595). 38
  • 39. How to do this? ● (for StyleGAN) it appears that the mapping from W to images is too complex for this to succeed reliably in practice. ● We find it encouraging that StyleGAN2 makes source attribution easier even though the image quality has improved significantly. Image from original paper 39
  • 42. Q & A