SlideShare a Scribd company logo

Score based generative model

•
•
S
sangyun lee

This document discusses score-based generative models. It explains that score-based models estimate the score function or gradient of the data distribution rather than the probability density function. This allows flexible deep learning architectures to be used without normalization. Score functions can be estimated using methods like score matching, sliced score matching, and denoising score matching. Models can then generate samples via Langevin dynamics. Challenges include accurately modeling low density regions and recovering relative weights between modes. Noise Conditional Score Networks address these by modeling perturbed distributions. Score-based models can also solve inverse problems without retraining.

Science
1 of 28
Download to read offline
Score-based Generative Models 이상윤
Generative Models “Learning causal relation” https://web.eecs.umich.edu/~justincj/slides/eecs498/498_FA2019_lecture19.pdf
Generative Models GAN: More focusing on sampling VAE: Maximize the lower bound of likelihood using surrogate loss Normalizing Flow: Exact likelihood maximization via invertible transformations https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html
Generative Models - limitations GAN: High fidelity but hard to train VAE: Not exact likelihood maximization Normalizing Flow: Lack of flexibility (must be invertible) https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html
Motivations Sampling without computing probability density through langevin dynamics (similar to gradient ascend)
Motivations Why we need PDF? - MLE equal to minimizing KLD with real data distribution. - In PDF, all densities compete with each other. https://web.eecs.umich.edu/~justincj/slides/eecs498/498_FA2019_lecture19.pdf
Motivations : normalizing constant : unnormalized function or energy function - The integral part of normalizing constant is intractable. - AR, NF handle this problem via special structures which make the unit normalizing constant. It is difficult to define a flexible, high-capacity probability density function
Motivations Score-based models naturally bypass the problem.
Score matching
Score matching
Score matching
Score matching Gradient is zero in data points. Local maxima regularization But computing hessian trace needs O(D) backprop!
Score matching - sliced score matching Project the score onto random vectors Equals to Hessian-vector product, https://en.wikipedia.org/wiki/Hessian_automatic_differentiation -> single backprop
Score matching - denoising score matching Predict the score of the perturbed distribution instead of p(x). Minimizing the object function above give us the optimal score function But results could be noisy if is large Let be gaussian,
Pitfalls Now that we have score function, so let’s sample through langevin dynamics. Under some conditions, it is proved that is the exact sample from p(x)! But there are some problems...
Pitfalls - manifold hypothesis The manifold hypothesis: Data lie on low-dimensional manifold. Problems - Score function is inaccurate in the low-density regions. - It is difficult to recover the relative weights between modes.
Pitfalls - inaccurate score function is not well-defined in the low density regions. Since we train the objective function above via montecarlo estimation, a model observes less samples from the low density regions.
Score matching - recovering relative weights
Score matching - recovering relative weights Langevin dynamics failed to recover the relative weights between two modes.
Perturbed distribution The low density regions can be filled by injecting the noises. But how to determine the noise strength? -> gradually decrease the variance.
Noise Conditional Score Network (NCSN) A single network estimates the score of multiple perturbed data distribution.
Annealed Langevin Dynamics
Results Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS2019, Oral
Results SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS, ICLR2021 (Oral)
Inverse Problem Solving Inverse problem is Bayesian inference problem. We want to know p(x|y) when p(y|x) is given. E.g, super resolution, colorization, inpainting... score matching known forward process We could sample from p(x|y) via langevin dynamics!
Inverse Problem Solving Inverse problem solving without retraining!
Conclusion Very flexible architecture compared to NF or AR. Score function can be any function. Therefore, modern DL architectures can be used (Resnet, UNet, etc). Sampling from exact p(x) is possible compared to VAEs which use surrogate loss. GAN-level fidelity without minmax game. Naturally solves inverse problem.
References Song, Yang, and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution." arXiv preprint arXiv:1907.05600 (2019). Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020). Yang Song blog (https://yang-song.github.io/blog/2021/score/) Stefano Ermon seminar (https://www.youtube.com/watch?v=8TcNXi3A5DI&t=562s)

Recommended

PR-409: Denoising Diffusion Probabilistic Models
PR-409: Denoising Diffusion Probabilistic ModelsPR-409: Denoising Diffusion Probabilistic Models
PR-409: Denoising Diffusion Probabilistic Models
Hyeongmin Lee
 
Score based Generative Modeling through Stochastic Differential Equations
Score based Generative Modeling through Stochastic Differential EquationsScore based Generative Modeling through Stochastic Differential Equations
Score based Generative Modeling through Stochastic Differential Equations
Sungchul Kim
 
Score-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential EquationsScore-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential Equations
Sangwoo Mo
 
Learning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat MinimaLearning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat Minima
Sangwoo Mo
 
Disentangled Representation Learning of Deep Generative Models
Disentangled Representation Learning of Deep Generative ModelsDisentangled Representation Learning of Deep Generative Models
Disentangled Representation Learning of Deep Generative Models
Ryohei Suzuki
 
[PR12] Generative Models as Distributions of Functions
[PR12] Generative Models as Distributions of Functions[PR12] Generative Models as Distributions of Functions
[PR12] Generative Models as Distributions of Functions
JaeJun Yoo
 
Introduction to Diffusion Models
Introduction to Diffusion ModelsIntroduction to Diffusion Models
Introduction to Diffusion Models
Sangwoo Mo
 
Explicit Density Models
Explicit Density ModelsExplicit Density Models
Explicit Density Models
Sangwoo Mo
 

Recommended

PR-409: Denoising Diffusion Probabilistic Models
PR-409: Denoising Diffusion Probabilistic ModelsPR-409: Denoising Diffusion Probabilistic Models
PR-409: Denoising Diffusion Probabilistic Models
Hyeongmin Lee
 
Score based Generative Modeling through Stochastic Differential Equations
Score based Generative Modeling through Stochastic Differential EquationsScore based Generative Modeling through Stochastic Differential Equations
Score based Generative Modeling through Stochastic Differential Equations
Sungchul Kim
 
Score-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential EquationsScore-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential Equations
Sangwoo Mo
 
Learning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat MinimaLearning Theory 101 ...and Towards Learning the Flat Minima
Learning Theory 101 ...and Towards Learning the Flat Minima
Sangwoo Mo
 
Disentangled Representation Learning of Deep Generative Models
Disentangled Representation Learning of Deep Generative ModelsDisentangled Representation Learning of Deep Generative Models
Disentangled Representation Learning of Deep Generative Models
Ryohei Suzuki
 
[PR12] Generative Models as Distributions of Functions
[PR12] Generative Models as Distributions of Functions[PR12] Generative Models as Distributions of Functions
[PR12] Generative Models as Distributions of Functions
JaeJun Yoo
 
Introduction to Diffusion Models
Introduction to Diffusion ModelsIntroduction to Diffusion Models
Introduction to Diffusion Models
Sangwoo Mo
 
Explicit Density Models
Explicit Density ModelsExplicit Density Models
Explicit Density Models
Sangwoo Mo
 
Tutorial on Deep Generative Models
Tutorial on Deep Generative Models Tutorial on Deep Generative Models
Tutorial on Deep Generative Models
MLReview
 
Diffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesisDiffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesis
BeerenSahu
 
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII
 
Variational Autoencoder
Variational AutoencoderVariational Autoencoder
Variational Autoencoder
Mark Chang
 
Learning loss for active learning
Learning loss for active learningLearning loss for active learning
Learning loss for active learning
NAVER Engineering
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...
Universitat Politècnica de Catalunya
 
diffusion 모델부터 DALLE2까지.pdf
diffusion 모델부터 DALLE2까지.pdfdiffusion 모델부터 DALLE2까지.pdf
diffusion 모델부터 DALLE2까지.pdf
수철 박
 
Bayesian Model-Agnostic Meta-Learning
Bayesian Model-Agnostic Meta-LearningBayesian Model-Agnostic Meta-Learning
Bayesian Model-Agnostic Meta-Learning
Sangwoo Mo
 
RoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position Embedding
taeseon ryu
 
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
NAVER Engineering
 
Variational continual learning
Variational continual learningVariational continual learning
Variational continual learning
Nguyen Giang
 
【技術解説20】 ミニバッチ確率的勾配降下法
【技術解説20】 ミニバッチ確率的勾配降下法【技術解説20】 ミニバッチ確率的勾配降下法
【技術解説20】 ミニバッチ確率的勾配降下法
Ruo Ando
 
Deep Generative Models
Deep Generative Models Deep Generative Models
Deep Generative Models
Chia-Wen Cheng
 
Basic Generative Adversarial Networks
Basic Generative Adversarial NetworksBasic Generative Adversarial Networks
Basic Generative Adversarial Networks
Dong Heon Cho
 
SinGAN - Learning a Generative Model from a Single Natural Image
SinGAN - Learning a Generative Model from a Single Natural ImageSinGAN - Learning a Generative Model from a Single Natural Image
SinGAN - Learning a Generative Model from a Single Natural Image
Jishnu P
 
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
Deep Learning JP
 
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
Jinwon Lee
 
Efficient Neural Architecture Search via Parameter Sharing
Efficient Neural Architecture Search via Parameter SharingEfficient Neural Architecture Search via Parameter Sharing
Efficient Neural Architecture Search via Parameter Sharing
Jinwon Lee
 
Finding connections among images using CycleGAN
Finding connections among images using CycleGANFinding connections among images using CycleGAN
Finding connections among images using CycleGAN
NAVER Engineering
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2
Vitaly Bondar
 
Lecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning RecapLecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning Recap
butest
 
IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab Projects
Vijay Karan
 

More Related Content

What's hot

SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII
 
Learning loss for active learning
Learning loss for active learningLearning loss for active learning
Learning loss for active learning
NAVER Engineering
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...
Universitat Politècnica de Catalunya
 
RoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position Embedding
taeseon ryu
 
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
NAVER Engineering
 
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
Jinwon Lee
 
Finding connections among images using CycleGAN
Finding connections among images using CycleGANFinding connections among images using CycleGAN
Finding connections among images using CycleGAN
NAVER Engineering
 

What's hot (20)

Tutorial on Deep Generative Models
Tutorial on Deep Generative Models Tutorial on Deep Generative Models
Tutorial on Deep Generative Models
 
Diffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesisDiffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesis
 
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
SSII2019TS: Shall We GANs?​ ～GANの基礎から最近の研究まで～
 
Variational Autoencoder
Variational AutoencoderVariational Autoencoder
Variational Autoencoder
 
Learning loss for active learning
Learning loss for active learningLearning loss for active learning
Learning loss for active learning
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...
 
diffusion 모델부터 DALLE2까지.pdf
diffusion 모델부터 DALLE2까지.pdfdiffusion 모델부터 DALLE2까지.pdf
diffusion 모델부터 DALLE2까지.pdf
 
Bayesian Model-Agnostic Meta-Learning
Bayesian Model-Agnostic Meta-LearningBayesian Model-Agnostic Meta-Learning
Bayesian Model-Agnostic Meta-Learning
 
RoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer: Enhanced Transformer with Rotary Position Embedding
RoFormer: Enhanced Transformer with Rotary Position Embedding
 
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
1시간만에 GAN(Generative Adversarial Network) 완전 정복하기
 
Variational continual learning
Variational continual learningVariational continual learning
Variational continual learning
 
【技術解説20】 ミニバッチ確率的勾配降下法
【技術解説20】 ミニバッチ確率的勾配降下法【技術解説20】 ミニバッチ確率的勾配降下法
【技術解説20】 ミニバッチ確率的勾配降下法
 
Deep Generative Models
Deep Generative Models Deep Generative Models
Deep Generative Models
 
Basic Generative Adversarial Networks
Basic Generative Adversarial NetworksBasic Generative Adversarial Networks
Basic Generative Adversarial Networks
 
SinGAN - Learning a Generative Model from a Single Natural Image
SinGAN - Learning a Generative Model from a Single Natural ImageSinGAN - Learning a Generative Model from a Single Natural Image
SinGAN - Learning a Generative Model from a Single Natural Image
 
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
【DL輪読会】"Masked Siamese Networks for Label-Efficient Learning"
 
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
PR-330: How To Train Your ViT? Data, Augmentation, and Regularization in Visi...
 
Efficient Neural Architecture Search via Parameter Sharing
Efficient Neural Architecture Search via Parameter SharingEfficient Neural Architecture Search via Parameter Sharing
Efficient Neural Architecture Search via Parameter Sharing
 
Finding connections among images using CycleGAN
Finding connections among images using CycleGANFinding connections among images using CycleGAN
Finding connections among images using CycleGAN
 
Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2Latent diffusions vs DALL-E v2
Latent diffusions vs DALL-E v2
 

Similar to Score based generative model

Lecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning RecapLecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning Recap
butest
 
November, 2006 CCKM'06 1
November, 2006 CCKM'06 1 November, 2006 CCKM'06 1
November, 2006 CCKM'06 1
butest
 
GNA 13552928 deep learning for GAN a.ppt
GNA 13552928 deep learning for GAN a.pptGNA 13552928 deep learning for GAN a.ppt
GNA 13552928 deep learning for GAN a.ppt
ManiMaran230751
 
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORETEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
IJCI JOURNAL
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
butest
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
butest
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
butest
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
butest
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
butest
 
Representational Continuity for Unsupervised Continual Learning
Representational Continuity for Unsupervised Continual LearningRepresentational Continuity for Unsupervised Continual Learning
Representational Continuity for Unsupervised Continual Learning
MLAI2
 

Similar to Score based generative model (20)

Lecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning RecapLecture 17: Supervised Learning Recap
Lecture 17: Supervised Learning Recap
 
IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab Projects
 
Dynamic Routing Between Capsules
Dynamic Routing Between CapsulesDynamic Routing Between Capsules
Dynamic Routing Between Capsules
 
November, 2006 CCKM'06 1
November, 2006 CCKM'06 1 November, 2006 CCKM'06 1
November, 2006 CCKM'06 1
 
IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab Projects
 
Nips 2016 tutorial generative adversarial networks review
Nips 2016 tutorial generative adversarial networks reviewNips 2016 tutorial generative adversarial networks review
Nips 2016 tutorial generative adversarial networks review
 
GNA 13552928 deep learning for GAN a.ppt
GNA 13552928 deep learning for GAN a.pptGNA 13552928 deep learning for GAN a.ppt
GNA 13552928 deep learning for GAN a.ppt
 
Reading group gan - 20170417
Reading group gan - 20170417Reading group gan - 20170417
Reading group gan - 20170417
 
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORETEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
TEXT GENERATION WITH GAN NETWORKS USING FEEDBACK SCORE
 
02.cnn - CNN 파헤치기 3탄
02.cnn - CNN 파헤치기 3탄02.cnn - CNN 파헤치기 3탄
02.cnn - CNN 파헤치기 3탄
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Deep learning architectures
Deep learning architecturesDeep learning architectures
Deep learning architectures
 
Decomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesisDecomposing image generation into layout priction and conditional synthesis
Decomposing image generation into layout priction and conditional synthesis
 
[ICCV 21] Influence-Balanced Loss for Imbalanced Visual Classification
[ICCV 21] Influence-Balanced Loss for Imbalanced Visual Classification[ICCV 21] Influence-Balanced Loss for Imbalanced Visual Classification
[ICCV 21] Influence-Balanced Loss for Imbalanced Visual Classification
 
Overfitting & Underfitting
Overfitting & UnderfittingOverfitting & Underfitting
Overfitting & Underfitting
 
Representational Continuity for Unsupervised Continual Learning
Representational Continuity for Unsupervised Continual LearningRepresentational Continuity for Unsupervised Continual Learning
Representational Continuity for Unsupervised Continual Learning
 

Recently uploaded

Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
muralinath2
 
platelets- lifespan -Clot retraction-disorders.pptx
platelets- lifespan -Clot retraction-disorders.pptxplatelets- lifespan -Clot retraction-disorders.pptx
platelets- lifespan -Clot retraction-disorders.pptx
muralinath2
 
Mammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also FunctionsMammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also Functions
YOGESH DOGRA
 
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
Scintica Instrumentation
 
extra-chromosomal-inheritance[1].pptx.pdfpdf
extra-chromosomal-inheritance[1].pptx.pdfpdfextra-chromosomal-inheritance[1].pptx.pdfpdf
extra-chromosomal-inheritance[1].pptx.pdfpdf
DiyaBiswas10
 
Anemia_ different types_causes_ conditions
Anemia_ different types_causes_ conditionsAnemia_ different types_causes_ conditions
Anemia_ different types_causes_ conditions
muralinath2
 
Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...
Sérgio Sacani
 
ESR_factors_affect-clinic significance-Pathysiology.pptx
ESR_factors_affect-clinic significance-Pathysiology.pptxESR_factors_affect-clinic significance-Pathysiology.pptx
ESR_factors_affect-clinic significance-Pathysiology.pptx
muralinath2
 

Recently uploaded (20)

Hemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptxHemoglobin metabolism_pathophysiology.pptx
Hemoglobin metabolism_pathophysiology.pptx
 
Large scale production of streptomycin.pptx
Large scale production of streptomycin.pptxLarge scale production of streptomycin.pptx
Large scale production of streptomycin.pptx
 
Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
Circulatory system_ Laplace law. Ohms law.reynaults law,baro-chemo-receptors-...
 
EY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptxEY - Supply Chain Services 2018_template.pptx
EY - Supply Chain Services 2018_template.pptx
 
SCHIZOPHRENIA Disorder/ Brain Disorder.pdf
SCHIZOPHRENIA Disorder/ Brain Disorder.pdfSCHIZOPHRENIA Disorder/ Brain Disorder.pdf
SCHIZOPHRENIA Disorder/ Brain Disorder.pdf
 
In silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptxIn silico drugs analogue design: novobiocin analogues.pptx
In silico drugs analogue design: novobiocin analogues.pptx
 
platelets- lifespan -Clot retraction-disorders.pptx
platelets- lifespan -Clot retraction-disorders.pptxplatelets- lifespan -Clot retraction-disorders.pptx
platelets- lifespan -Clot retraction-disorders.pptx
 
FAIRSpectra - Towards a common data file format for SIMS images
FAIRSpectra - Towards a common data file format for SIMS imagesFAIRSpectra - Towards a common data file format for SIMS images
FAIRSpectra - Towards a common data file format for SIMS images
 
Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)Structural Classification Of Protein (SCOP)
Structural Classification Of Protein (SCOP)
 
Mammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also FunctionsMammalian Pineal Body Structure and Also Functions
Mammalian Pineal Body Structure and Also Functions
 
Predicting property prices with machine learning algorithms.pdf
Predicting property prices with machine learning algorithms.pdfPredicting property prices with machine learning algorithms.pdf
Predicting property prices with machine learning algorithms.pdf
 
GEOLOGICAL FIELD REPORT On Kaptai Rangamati Road-Cut Section.pdf
GEOLOGICAL FIELD REPORT On Kaptai Rangamati Road-Cut Section.pdfGEOLOGICAL FIELD REPORT On Kaptai Rangamati Road-Cut Section.pdf
GEOLOGICAL FIELD REPORT On Kaptai Rangamati Road-Cut Section.pdf
 
GBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram StainingGBSN- Microbiology (Lab 3) Gram Staining
GBSN- Microbiology (Lab 3) Gram Staining
 
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...
 
Seminar of U.V. Spectroscopy by SAMIR PANDA
Seminar of U.V. Spectroscopy by SAMIR PANDA Seminar of U.V. Spectroscopy by SAMIR PANDA
Seminar of U.V. Spectroscopy by SAMIR PANDA
 
extra-chromosomal-inheritance[1].pptx.pdfpdf
extra-chromosomal-inheritance[1].pptx.pdfpdfextra-chromosomal-inheritance[1].pptx.pdfpdf
extra-chromosomal-inheritance[1].pptx.pdfpdf
 
Anemia_ different types_causes_ conditions
Anemia_ different types_causes_ conditionsAnemia_ different types_causes_ conditions
Anemia_ different types_causes_ conditions
 
Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...
 
Astronomy Update- Curiosity’s exploration of Mars _ Local Briefs _ leadertele...
Astronomy Update- Curiosity’s exploration of Mars _ Local Briefs _ leadertele...Astronomy Update- Curiosity’s exploration of Mars _ Local Briefs _ leadertele...
Astronomy Update- Curiosity’s exploration of Mars _ Local Briefs _ leadertele...
 
ESR_factors_affect-clinic significance-Pathysiology.pptx
ESR_factors_affect-clinic significance-Pathysiology.pptxESR_factors_affect-clinic significance-Pathysiology.pptx
ESR_factors_affect-clinic significance-Pathysiology.pptx
 

Score based generative model

  • 2. Generative Models “Learning causal relation” https://web.eecs.umich.edu/~justincj/slides/eecs498/498_FA2019_lecture19.pdf
  • 3. Generative Models GAN: More focusing on sampling VAE: Maximize the lower bound of likelihood using surrogate loss Normalizing Flow: Exact likelihood maximization via invertible transformations https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html
  • 4. Generative Models - limitations GAN: High fidelity but hard to train VAE: Not exact likelihood maximization Normalizing Flow: Lack of flexibility (must be invertible) https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html
  • 5. Motivations Sampling without computing probability density through langevin dynamics (similar to gradient ascend)
  • 6. Motivations Why we need PDF? - MLE equal to minimizing KLD with real data distribution. - In PDF, all densities compete with each other. https://web.eecs.umich.edu/~justincj/slides/eecs498/498_FA2019_lecture19.pdf
  • 7. Motivations : normalizing constant : unnormalized function or energy function - The integral part of normalizing constant is intractable. - AR, NF handle this problem via special structures which make the unit normalizing constant. It is difficult to define a flexible, high-capacity probability density function
  • 12. Score matching Gradient is zero in data points. Local maxima regularization But computing hessian trace needs O(D) backprop!
  • 13. Score matching - sliced score matching Project the score onto random vectors Equals to Hessian-vector product, https://en.wikipedia.org/wiki/Hessian_automatic_differentiation -> single backprop
  • 14. Score matching - denoising score matching Predict the score of the perturbed distribution instead of p(x). Minimizing the object function above give us the optimal score function But results could be noisy if is large Let be gaussian,
  • 15. Pitfalls Now that we have score function, so let’s sample through langevin dynamics. Under some conditions, it is proved that is the exact sample from p(x)! But there are some problems...
  • 16. Pitfalls - manifold hypothesis The manifold hypothesis: Data lie on low-dimensional manifold. Problems - Score function is inaccurate in the low-density regions. - It is difficult to recover the relative weights between modes.
  • 17. Pitfalls - inaccurate score function is not well-defined in the low density regions. Since we train the objective function above via montecarlo estimation, a model observes less samples from the low density regions.
  • 18. Score matching - recovering relative weights
  • 19. Score matching - recovering relative weights Langevin dynamics failed to recover the relative weights between two modes.
  • 20. Perturbed distribution The low density regions can be filled by injecting the noises. But how to determine the noise strength? -> gradually decrease the variance.
  • 21. Noise Conditional Score Network (NCSN) A single network estimates the score of multiple perturbed data distribution.
  • 23. Results Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS2019, Oral
  • 24. Results SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS, ICLR2021 (Oral)
  • 25. Inverse Problem Solving Inverse problem is Bayesian inference problem. We want to know p(x|y) when p(y|x) is given. E.g, super resolution, colorization, inpainting... score matching known forward process We could sample from p(x|y) via langevin dynamics!
  • 26. Inverse Problem Solving Inverse problem solving without retraining!
  • 27. Conclusion Very flexible architecture compared to NF or AR. Score function can be any function. Therefore, modern DL architectures can be used (Resnet, UNet, etc). Sampling from exact p(x) is possible compared to VAEs which use surrogate loss. GAN-level fidelity without minmax game. Naturally solves inverse problem.
  • 28. References Song, Yang, and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution." arXiv preprint arXiv:1907.05600 (2019). Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020). Yang Song blog (https://yang-song.github.io/blog/2021/score/) Stefano Ermon seminar (https://www.youtube.com/watch?v=8TcNXi3A5DI&t=562s)