SlideShare a Scribd company logo

Representation Learning & Generative Modeling with Variational Autoencoder(VAE) / Auto-Encoding Variational Bayes Review

•Download as PPTX, PDF•
•
C
changedaeoh

This document summarizes the key ideas of auto-encoding variational Bayes. It discusses representation learning using latent variables to model high-dimensional sparse data on low-dimensional manifolds. It then explains generative modeling and the challenge of directly estimating complex data generating distributions. Finally, it introduces variational autoencoders as a way to approximate intractable posterior distributions over latent variables using variational inference and maximize a tractable evidence lower bound objective using the reparameterization trick, allowing end-to-end training of the encoder and decoder networks.

Science
1 of 28
Auto-Encoding Variational Bayes Diederik P. Kingma, Max Welling, 2013 TAVE Research Seminar 2021.05.18 Presenter : Changdae Oh bnormal16@naver.com
2 Topics  Representation Learning  Generative Modeling  Variational Auto-encoder Keywords 1. < Manifold hypothesis, Latent variable > 2. < Data generating distribution, Density estimation > 3. < Variational Inference, Evidence lower bound, Reparameterization >
3 1. Representation Learning Keywords < Manifold hypothesis, Latent variable >
4 Representation Learning X = (Latitude, Longitude, altitude) 3d X : Location of the car Representation & Manifold hypothesis X = (Distance from the datum) 1d satellit e navigati on
5 Representation Learning Image data And the other atypical data, too Very high dimensional, Very sparse !! (28*28) = 784 (256*256*3) = 196608
6 Representation Learning : Hidden variable that are not measured directly, but have a significant impact on the variation of data points. Latent Variable : Learning non-linear subspace with dense data points built by Hidden factors of variation. Manifold Learning Lower dimension, Dense space
7 Representation Learning Goal : Learn a function to map input ‘x’ -> output ‘y’ The behavior of intermediate layers - All features are projected down to two dim. (For visualization) - The classes become increasingly linearly separable https://deeplearning.cs.cmu.edu/F20/document/slides/lec17.representations.pdf Layers sequentially “straighten” the data manifold Manifold hypo. in Supervised Learning
8 Representation Learning ( can use latent variables ) Manifold hypo. in Unsupervised Learning http://cs231n.stanford.edu/slides/2021/lecture_12.pdf (Linear manifold) Goal : Learn some underlying hidden structure of the data
9 2. Generative Modeling Keywords < Data generating distribution, Density estimation >
10 Generative Modeling Density estimation Trainin g • 𝑃𝜃 𝑦 𝑥) • 𝑦 = 𝑓𝜃(𝑥) Discriminative model Generative model Use in 𝑓 ∶ 𝑥 → 𝑦 • 𝑔 ∶ 𝑠𝑒𝑒𝑑 → 𝑥 • 𝑔 ∶ 𝑠𝑒𝑒𝑑, 𝑦 → 𝑥 • 𝑃𝜃(𝑥) • 𝑃𝜃(𝑥, 𝑦) or 𝑃𝜃 𝑥 𝑦) Data generation Conditional prob. Estimation Predictio n Learn direct maps
11 Generative Modeling Process of the occurrence of natural images according to probability distribution 𝑃𝑑𝑎𝑡𝑎(𝐱) Generative model We want to learn 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 similar to 𝑃𝑑𝑎𝑡𝑎(𝐱) Machine Learning, Ilseok Oh. Lecture slide Data Generating Distribution * image from Fei-Dei Li, Justin Johnson, Serena Yeung, cs231n Stanford
12 Generative Modeling NIPS 2016 Tutorial: Generative Adversarial Networks
13 Generative Modeling We want to learn 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 similar to 𝑃𝑑𝑎𝑡𝑎(𝐱) Estimate directly via 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 ? * very challenging!! • Intractable • Require strong constraints Latent variable (generative) model * slide from Aaron Courville, IFT6266 Hiver 2017 : learn a mapping from some latent variable z to a complicated distribution on x
14 3. Variational Auto-encoder Keywords < Variational Inference, Evidence lower bound, Reparameterization >
15 Variational Auto-encoder • The data we observe in the real world are very high-dimensional and sparse. • A low-dimensional, high-density nonlinear manifold exists in the space where observational data are defined. • There is a latent variable describing the manifold, which is very closely related to the variation of observed data x. Story so far • Want to get a model that generate data similar to observational data x. • To do that, we need to estimate the distribution of the data P(x). • However, direct estimation of P(x) is challenging. • Instead, let's model a conditional distribution P(x|z) using the latent variable z.
16 Variational Auto-encoder • where does ‘z’ come from? • How can ‘z’ be defined and obtained? Proble m * image from the cs236, Stanford 2019f - Deep Generative Models, lectue5 Since z is literally a latent variable, it is very difficult to define it manually and impossible to measure directly.
17 Variational Auto-encoder • 𝑥𝑖 ~ 𝑃 𝑥 𝑧) • 𝑧𝑖 ~ 𝑃(𝑧) • 𝑧𝑖 ~ 𝑃 𝑧 𝑥) Distributional assumptions Can use this sample directly but the performance is not good. Still, there is a problem… Overview for data generating process Assume a familiar distribution. 𝑃 𝑧 𝑥) = 𝑃 𝑥 𝑧)𝑃 𝑧 /𝑃(𝑥) Learn the distribution of the latent variable z, which is well explained from x. And sampling z from that dist. Assume a familiar distribution. Intractable
18 Variational Auto-encoder Variational Inference 𝑝𝜃 𝑧 𝑥) ≈ 𝑞𝜙(𝑧 | 𝑥) General family of methods for approximating Complicated densities by a simpler class of densities * slide from shakir Mohamed(Google DeepMind), Imperial College, London, 2015 intractab le tractabl e, familiar
19 Variational Auto-encoder Find objective http://cs231n.stanford.edu/slides/2021/lecture_12.pdf assumptions Want to maximize
20 Variational Auto-encoder Find objective ≥ 0 Expression 1 - generic Expression 2 - practical Tractable Variational Lower Bound!! (also called Evidence Lower Bound) Let’s maximize this ELBO ! Totally MC approx. est. KLD : Analytical solution Expectation term : MC approx. est.
21 Variational Auto-encoder END-TO-END Learning ! ( reparameterization )
22 Variational Auto-encoder * slide from CPSC 532L lecture 11, the University of British Columbia Problem!
23 Variational Auto-encoder Reparameterization trick 𝑧 ~ 𝑞𝜙 𝑧 𝑥) = 𝑁(𝜇 𝑥 , Σ(𝑥)) https://arxiv.org/abs/1606.05908
24 Variational Auto-encoder End2End learning pros cons • Interpretable latent space • Allows inference of q(z|x), can be useful feature representation for other tasks • Approx’ optimal • Samples are blurrier * slide from Aaron Courville, IFT6266 Hiver 2017
25 Variational Auto-encoder Experiments on loss term https://www.jeremyjordan.me/variational-autoencoders/
26 Variational Auto-encoder Learned latent space
27 Main Reference Paper • Auto-Encoding Variational Bayes, Diederik P Kingma, Max Welling, 2013. [link] • Tutorial on Variational Autoencoders, Carl Doersch, 2016. [link] • NIPS 2016 Tutorial: Generative Adversarial Networks, Ian Goodfellow, 2016. [link] Slide • cs231n lecture slide, stanfold, 2021s. [link] • cs236n lecture slide, stanfold, 2019f. [link] • IFT6266-H2017, University of Montreal [link] Book • Deep Learning, Ian Goodfellow et al, 2016. [e-book] • Machine Learning, Ilseok Oh, 2018. etc. • Tutorial - what is a variational autoencoder? [link] • Everything about the autoencoder [video]
28 Changdae Oh bnormal16@naver.com https://velog.io/@changdaeoh https://github.com/changdaeoh

Recommended

Reviews on Deep Generative Models in the early days / GANs & VAEs paper review
Reviews on Deep Generative Models in the early days / GANs & VAEs paper reviewReviews on Deep Generative Models in the early days / GANs & VAEs paper review
Reviews on Deep Generative Models in the early days / GANs & VAEs paper reviewchangedaeoh
 
Deep Advances in Generative Modeling
Deep Advances in Generative ModelingDeep Advances in Generative Modeling
Deep Advances in Generative Modelingindico data
 
InfoGAN and Generative Adversarial Networks
InfoGAN and Generative Adversarial NetworksInfoGAN and Generative Adversarial Networks
InfoGAN and Generative Adversarial NetworksZak Jost
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Universitat Politècnica de Catalunya
 
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...Universitat Politècnica de Catalunya
 
Convolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelsConvolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelstuxette
 
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...Universitat Politècnica de Catalunya
 
Generative Adversarial Network (+Laplacian Pyramid GAN)
Generative Adversarial Network (+Laplacian Pyramid GAN)Generative Adversarial Network (+Laplacian Pyramid GAN)
Generative Adversarial Network (+Laplacian Pyramid GAN)NamHyuk Ahn
 

Recommended

Reviews on Deep Generative Models in the early days / GANs & VAEs paper review
Reviews on Deep Generative Models in the early days / GANs & VAEs paper reviewReviews on Deep Generative Models in the early days / GANs & VAEs paper review
Reviews on Deep Generative Models in the early days / GANs & VAEs paper reviewchangedaeoh
 
Deep Advances in Generative Modeling
Deep Advances in Generative ModelingDeep Advances in Generative Modeling
Deep Advances in Generative Modelingindico data
 
InfoGAN and Generative Adversarial Networks
InfoGAN and Generative Adversarial NetworksInfoGAN and Generative Adversarial Networks
InfoGAN and Generative Adversarial NetworksZak Jost
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Universitat Politècnica de Catalunya
 
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...
Generative Adversarial Networks (D2L5 Deep Learning for Speech and Language U...Universitat Politècnica de Catalunya
 
Convolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelsConvolutional networks and graph networks through kernels
Convolutional networks and graph networks through kernelstuxette
 
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...
Generative Models and Adversarial Training (D3L4 2017 UPC Deep Learning for ...Universitat Politècnica de Catalunya
 
Generative Adversarial Network (+Laplacian Pyramid GAN)
Generative Adversarial Network (+Laplacian Pyramid GAN)Generative Adversarial Network (+Laplacian Pyramid GAN)
Generative Adversarial Network (+Laplacian Pyramid GAN)NamHyuk Ahn
 
Generative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their ApplicationsGenerative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their ApplicationsArtifacia
 
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
Graph Neural Network in practice
Graph Neural Network in practiceGraph Neural Network in practice
Graph Neural Network in practicetuxette
 
DTLC-GAN
DTLC-GANDTLC-GAN
DTLC-GANShinagawa Seitaro
 
Icml2017 overview
Icml2017 overviewIcml2017 overview
Icml2017 overviewTatsuya Shirakawa
 
Graph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype PredictionGraph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype Predictiontuxette
 
Creating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural netsCreating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural netsAkin Osman Kazakci
 
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...The Statistical and Applied Mathematical Sciences Institute
 
Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"tuxette
 
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...AMIDST Toolbox
 
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic SegmentationSemantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation岳華 杜
 
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
About functional SIR
About functional SIRAbout functional SIR
About functional SIRtuxette
 
Anomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class ClassifierAnomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class Classifier홍배 김
 
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...Rizwan Habib
 
Learning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for GraphsLearning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for GraphsMathias Niepert
 
2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper reading2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper readingShinagawa Seitaro
 
SEGAN: Speech Enhancement Generative Adversarial Network
SEGAN: Speech Enhancement Generative Adversarial NetworkSEGAN: Speech Enhancement Generative Adversarial Network
SEGAN: Speech Enhancement Generative Adversarial NetworkUniversitat Politècnica de Catalunya
 
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 

More Related Content

What's hot

Generative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their ApplicationsGenerative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their ApplicationsArtifacia
 
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
Graph Neural Network in practice
Graph Neural Network in practiceGraph Neural Network in practice
Graph Neural Network in practicetuxette
 
Graph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype PredictionGraph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype Predictiontuxette
 
Creating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural netsCreating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural netsAkin Osman Kazakci
 
Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"tuxette
 
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...AMIDST Toolbox
 
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic SegmentationSemantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation岳華 杜
 
About functional SIR
About functional SIRAbout functional SIR
About functional SIRtuxette
 
Anomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class ClassifierAnomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class Classifier홍배 김
 
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...Rizwan Habib
 
Learning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for GraphsLearning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for GraphsMathias Niepert
 
2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper reading2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper readingShinagawa Seitaro
 

What's hot (20)

Generative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their ApplicationsGenerative Adversarial Networks and Their Applications
Generative Adversarial Networks and Their Applications
 
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
 
Graph Neural Network in practice
Graph Neural Network in practiceGraph Neural Network in practice
Graph Neural Network in practice
 
DTLC-GAN
DTLC-GANDTLC-GAN
DTLC-GAN
 
Icml2017 overview
Icml2017 overviewIcml2017 overview
Icml2017 overview
 
Graph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype PredictionGraph Neural Network for Phenotype Prediction
Graph Neural Network for Phenotype Prediction
 
Creating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural netsCreating new classes of objects with deep generative neural nets
Creating new classes of objects with deep generative neural nets
 
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...
Deep Learning Opening Workshop - Horseshoe Regularization for Machine Learnin...
 
Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"Reading revue of "Inferring Multiple Graphical Structures"
Reading revue of "Inferring Multiple Graphical Structures"
 
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
Parallel Filter-Based Feature Selection Based on Balanced Incomplete Block De...
 
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
Convolutional Neural Networks (DLAI D5L1 2017 UPC Deep Learning for Artificia...
 
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
Generative Adversarial Networks GAN - Santiago Pascual - UPC Barcelona 2018
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic SegmentationSemantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
 
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
 
About functional SIR
About functional SIRAbout functional SIR
About functional SIR
 
Anomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class ClassifierAnomaly Detection and Localization Using GAN and One-Class Classifier
Anomaly Detection and Localization Using GAN and One-Class Classifier
 
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
NYAI - A Path To Unsupervised Learning Through Adversarial Networks by Soumit...
 
Learning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for GraphsLearning Convolutional Neural Networks for Graphs
Learning Convolutional Neural Networks for Graphs
 
2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper reading2018.01.12 AHClab SD-study paper reading
2018.01.12 AHClab SD-study paper reading
 
SEGAN: Speech Enhancement Generative Adversarial Network
SEGAN: Speech Enhancement Generative Adversarial NetworkSEGAN: Speech Enhancement Generative Adversarial Network
SEGAN: Speech Enhancement Generative Adversarial Network
 

Similar to Representation Learning & Generative Modeling with Variational Autoencoder(VAE) / Auto-Encoding Variational Bayes Review

Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Chris Rackauckas
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
010_20160216_Variational Gaussian Process
010_20160216_Variational Gaussian Process010_20160216_Variational Gaussian Process
010_20160216_Variational Gaussian ProcessHa Phuong
 
causality_discussion_slides_final.pdf
causality_discussion_slides_final.pdfcausality_discussion_slides_final.pdf
causality_discussion_slides_final.pdfssuser8cde591
 
20230213_ComputerVision_연구.pptx
20230213_ComputerVision_연구.pptx20230213_ComputerVision_연구.pptx
20230213_ComputerVision_연구.pptxssuser7807522
 
Machine Learning Foundations for Professional Managers
Machine Learning Foundations for Professional ManagersMachine Learning Foundations for Professional Managers
Machine Learning Foundations for Professional ManagersAlbert Y. C. Chen
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Ian Morgan
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Bayes Nets meetup London
 
Tutorial on Deep Generative Models
Tutorial on Deep Generative Models Tutorial on Deep Generative Models
Tutorial on Deep Generative ModelsMLReview
 
The Success of Deep Generative Models
The Success of Deep Generative ModelsThe Success of Deep Generative Models
The Success of Deep Generative Modelsinside-BigData.com
 
Nonlinear dimension reduction
Nonlinear dimension reductionNonlinear dimension reduction
Nonlinear dimension reductionYan Xu
 
NIPS2007: deep belief nets
NIPS2007: deep belief netsNIPS2007: deep belief nets
NIPS2007: deep belief netszukun
 
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に - 最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に - Hiroshi Fukui
 

Similar to Representation Learning & Generative Modeling with Variational Autoencoder(VAE) / Auto-Encoding Variational Bayes Review (20)

Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
 
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
Unsupervised Deep Learning (D2L1 Insight@DCU Machine Learning Workshop 2017)
 
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
 
010_20160216_Variational Gaussian Process
010_20160216_Variational Gaussian Process010_20160216_Variational Gaussian Process
010_20160216_Variational Gaussian Process
 
causality_discussion_slides_final.pdf
causality_discussion_slides_final.pdfcausality_discussion_slides_final.pdf
causality_discussion_slides_final.pdf
 
Deep Learning for Computer Vision: Unsupervised Learning (UPC 2016)
Deep Learning for Computer Vision: Unsupervised Learning (UPC 2016)Deep Learning for Computer Vision: Unsupervised Learning (UPC 2016)
Deep Learning for Computer Vision: Unsupervised Learning (UPC 2016)
 
XGBoostLSS - An extension of XGBoost to probabilistic forecasting, Alexander ...
XGBoostLSS - An extension of XGBoost to probabilistic forecasting, Alexander ...XGBoostLSS - An extension of XGBoost to probabilistic forecasting, Alexander ...
XGBoostLSS - An extension of XGBoost to probabilistic forecasting, Alexander ...
 
20230213_ComputerVision_연구.pptx
20230213_ComputerVision_연구.pptx20230213_ComputerVision_연구.pptx
20230213_ComputerVision_연구.pptx
 
alexVAE_New.pdf
alexVAE_New.pdfalexVAE_New.pdf
alexVAE_New.pdf
 
Machine Learning Foundations for Professional Managers
Machine Learning Foundations for Professional ManagersMachine Learning Foundations for Professional Managers
Machine Learning Foundations for Professional Managers
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
 
Tutorial on Deep Generative Models
Tutorial on Deep Generative Models Tutorial on Deep Generative Models
Tutorial on Deep Generative Models
 
2018 Modern Math Workshop - Nonparametric Regression and Classification for M...
2018 Modern Math Workshop - Nonparametric Regression and Classification for M...2018 Modern Math Workshop - Nonparametric Regression and Classification for M...
2018 Modern Math Workshop - Nonparametric Regression and Classification for M...
 
The Success of Deep Generative Models
The Success of Deep Generative ModelsThe Success of Deep Generative Models
The Success of Deep Generative Models
 
Nonlinear dimension reduction
Nonlinear dimension reductionNonlinear dimension reduction
Nonlinear dimension reduction
 
NIPS2007: deep belief nets
NIPS2007: deep belief netsNIPS2007: deep belief nets
NIPS2007: deep belief nets
 
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018
Deep Learning without Annotations - Xavier Giro - UPC Barcelona 2018
 
最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に - 最近の研究情勢についていくために - Deep Learningを中心に -
最近の研究情勢についていくために - Deep Learningを中心に -
 

Recently uploaded

Spermiogenesis or Spermateleosis or metamorphosis of spermatid
Spermiogenesis or Spermateleosis or metamorphosis of spermatidSpermiogenesis or Spermateleosis or metamorphosis of spermatid
Spermiogenesis or Spermateleosis or metamorphosis of spermatidSarthak Sekhar Mondal
 
Luciferase in rDNA technology (biotechnology).pptx
Luciferase in rDNA technology (biotechnology).pptxLuciferase in rDNA technology (biotechnology).pptx
Luciferase in rDNA technology (biotechnology).pptxAleenaTreesaSaji
 
Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)PraveenaKalaiselvan1
 
Isotopic evidence of long-lived volcanism on Io
Isotopic evidence of long-lived volcanism on IoIsotopic evidence of long-lived volcanism on Io
Isotopic evidence of long-lived volcanism on IoSérgio Sacani
 
VIRUSES structure and classification ppt by Dr.Prince C P
VIRUSES structure and classification ppt by Dr.Prince C PVIRUSES structure and classification ppt by Dr.Prince C P
VIRUSES structure and classification ppt by Dr.Prince C PPRINCE C P
 
Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​ ​Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​ ​kaibalyasahoo82800
 
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |aasikanpl
 
Natural Polymer Based Nanomaterials
Natural Polymer Based NanomaterialsNatural Polymer Based Nanomaterials
Natural Polymer Based NanomaterialsAArockiyaNisha
 
Recombination DNA Technology (Nucleic Acid Hybridization )
Recombination DNA Technology (Nucleic Acid Hybridization )Recombination DNA Technology (Nucleic Acid Hybridization )
Recombination DNA Technology (Nucleic Acid Hybridization )aarthirajkumar25
 
Work, Energy and Power for class 10 ICSE Physics
Work, Energy and Power for class 10 ICSE PhysicsWork, Energy and Power for class 10 ICSE Physics
Work, Energy and Power for class 10 ICSE Physicsvishikhakeshava1
 
Boyles law module in the grade 10 science
Boyles law module in the grade 10 scienceBoyles law module in the grade 10 science
Boyles law module in the grade 10 sciencefloriejanemacaya1
 
Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Nistarini College, Purulia (W.B) India
 
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCESTERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCEPRINCE C P
 
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxUnlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxanandsmhk
 
Orientation, design and principles of polyhouse
Orientation, design and principles of polyhouseOrientation, design and principles of polyhouse
Orientation, design and principles of polyhousejana861314
 
G9 Science Q4- Week 1-2 Projectile Motion.ppt
G9 Science Q4- Week 1-2 Projectile Motion.pptG9 Science Q4- Week 1-2 Projectile Motion.ppt
G9 Science Q4- Week 1-2 Projectile Motion.pptMAESTRELLAMesa2
 
Disentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTDisentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTSérgio Sacani
 
Scheme-of-Work-Science-Stage-4 cambridge science.docx
Scheme-of-Work-Science-Stage-4 cambridge science.docxScheme-of-Work-Science-Stage-4 cambridge science.docx
Scheme-of-Work-Science-Stage-4 cambridge science.docxyaramohamed343013
 
Analytical Profile of Coleus Forskohlii | Forskolin .pdf
Analytical Profile of Coleus Forskohlii | Forskolin .pdfAnalytical Profile of Coleus Forskohlii | Forskolin .pdf
Analytical Profile of Coleus Forskohlii | Forskolin .pdfSwapnil Therkar
 

Recently uploaded (20)

Spermiogenesis or Spermateleosis or metamorphosis of spermatid
Spermiogenesis or Spermateleosis or metamorphosis of spermatidSpermiogenesis or Spermateleosis or metamorphosis of spermatid
Spermiogenesis or Spermateleosis or metamorphosis of spermatid
 
Luciferase in rDNA technology (biotechnology).pptx
Luciferase in rDNA technology (biotechnology).pptxLuciferase in rDNA technology (biotechnology).pptx
Luciferase in rDNA technology (biotechnology).pptx
 
Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)Recombinant DNA technology (Immunological screening)
Recombinant DNA technology (Immunological screening)
 
Engler and Prantl system of classification in plant taxonomy
Engler and Prantl system of classification in plant taxonomyEngler and Prantl system of classification in plant taxonomy
Engler and Prantl system of classification in plant taxonomy
 
Isotopic evidence of long-lived volcanism on Io
Isotopic evidence of long-lived volcanism on IoIsotopic evidence of long-lived volcanism on Io
Isotopic evidence of long-lived volcanism on Io
 
VIRUSES structure and classification ppt by Dr.Prince C P
VIRUSES structure and classification ppt by Dr.Prince C PVIRUSES structure and classification ppt by Dr.Prince C P
VIRUSES structure and classification ppt by Dr.Prince C P
 
Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​ ​Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​ ​
 
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
Call Us ≽ 9953322196 ≼ Call Girls In Mukherjee Nagar(Delhi) |
 
Natural Polymer Based Nanomaterials
Natural Polymer Based NanomaterialsNatural Polymer Based Nanomaterials
Natural Polymer Based Nanomaterials
 
Recombination DNA Technology (Nucleic Acid Hybridization )
Recombination DNA Technology (Nucleic Acid Hybridization )Recombination DNA Technology (Nucleic Acid Hybridization )
Recombination DNA Technology (Nucleic Acid Hybridization )
 
Work, Energy and Power for class 10 ICSE Physics
Work, Energy and Power for class 10 ICSE PhysicsWork, Energy and Power for class 10 ICSE Physics
Work, Energy and Power for class 10 ICSE Physics
 
Boyles law module in the grade 10 science
Boyles law module in the grade 10 scienceBoyles law module in the grade 10 science
Boyles law module in the grade 10 science
 
Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...Bentham & Hooker's Classification. along with the merits and demerits of the ...
Bentham & Hooker's Classification. along with the merits and demerits of the ...
 
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCESTERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
STERILITY TESTING OF PHARMACEUTICALS ppt by DR.C.P.PRINCE
 
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptxUnlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
Unlocking the Potential: Deep dive into ocean of Ceramic Magnets.pptx
 
Orientation, design and principles of polyhouse
Orientation, design and principles of polyhouseOrientation, design and principles of polyhouse
Orientation, design and principles of polyhouse
 
G9 Science Q4- Week 1-2 Projectile Motion.ppt
G9 Science Q4- Week 1-2 Projectile Motion.pptG9 Science Q4- Week 1-2 Projectile Motion.ppt
G9 Science Q4- Week 1-2 Projectile Motion.ppt
 
Disentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTDisentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOST
 
Scheme-of-Work-Science-Stage-4 cambridge science.docx
Scheme-of-Work-Science-Stage-4 cambridge science.docxScheme-of-Work-Science-Stage-4 cambridge science.docx
Scheme-of-Work-Science-Stage-4 cambridge science.docx
 
Analytical Profile of Coleus Forskohlii | Forskolin .pdf
Analytical Profile of Coleus Forskohlii | Forskolin .pdfAnalytical Profile of Coleus Forskohlii | Forskolin .pdf
Analytical Profile of Coleus Forskohlii | Forskolin .pdf
 

Representation Learning & Generative Modeling with Variational Autoencoder(VAE) / Auto-Encoding Variational Bayes Review

  • 1. Auto-Encoding Variational Bayes Diederik P. Kingma, Max Welling, 2013 TAVE Research Seminar 2021.05.18 Presenter : Changdae Oh bnormal16@naver.com
  • 2. 2 Topics  Representation Learning  Generative Modeling  Variational Auto-encoder Keywords 1. < Manifold hypothesis, Latent variable > 2. < Data generating distribution, Density estimation > 3. < Variational Inference, Evidence lower bound, Reparameterization >
  • 3. 3 1. Representation Learning Keywords < Manifold hypothesis, Latent variable >
  • 4. 4 Representation Learning X = (Latitude, Longitude, altitude) 3d X : Location of the car Representation & Manifold hypothesis X = (Distance from the datum) 1d satellit e navigati on
  • 5. 5 Representation Learning Image data And the other atypical data, too Very high dimensional, Very sparse !! (28*28) = 784 (256*256*3) = 196608
  • 6. 6 Representation Learning : Hidden variable that are not measured directly, but have a significant impact on the variation of data points. Latent Variable : Learning non-linear subspace with dense data points built by Hidden factors of variation. Manifold Learning Lower dimension, Dense space
  • 7. 7 Representation Learning Goal : Learn a function to map input ‘x’ -> output ‘y’ The behavior of intermediate layers - All features are projected down to two dim. (For visualization) - The classes become increasingly linearly separable https://deeplearning.cs.cmu.edu/F20/document/slides/lec17.representations.pdf Layers sequentially “straighten” the data manifold Manifold hypo. in Supervised Learning
  • 8. 8 Representation Learning ( can use latent variables ) Manifold hypo. in Unsupervised Learning http://cs231n.stanford.edu/slides/2021/lecture_12.pdf (Linear manifold) Goal : Learn some underlying hidden structure of the data
  • 9. 9 2. Generative Modeling Keywords < Data generating distribution, Density estimation >
  • 10. 10 Generative Modeling Density estimation Trainin g • 𝑃𝜃 𝑦 𝑥) • 𝑦 = 𝑓𝜃(𝑥) Discriminative model Generative model Use in 𝑓 ∶ 𝑥 → 𝑦 • 𝑔 ∶ 𝑠𝑒𝑒𝑑 → 𝑥 • 𝑔 ∶ 𝑠𝑒𝑒𝑑, 𝑦 → 𝑥 • 𝑃𝜃(𝑥) • 𝑃𝜃(𝑥, 𝑦) or 𝑃𝜃 𝑥 𝑦) Data generation Conditional prob. Estimation Predictio n Learn direct maps
  • 11. 11 Generative Modeling Process of the occurrence of natural images according to probability distribution 𝑃𝑑𝑎𝑡𝑎(𝐱) Generative model We want to learn 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 similar to 𝑃𝑑𝑎𝑡𝑎(𝐱) Machine Learning, Ilseok Oh. Lecture slide Data Generating Distribution * image from Fei-Dei Li, Justin Johnson, Serena Yeung, cs231n Stanford
  • 12. 12 Generative Modeling NIPS 2016 Tutorial: Generative Adversarial Networks
  • 13. 13 Generative Modeling We want to learn 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 similar to 𝑃𝑑𝑎𝑡𝑎(𝐱) Estimate directly via 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝑃𝑚𝑜𝑑𝑒𝑙 𝐱 ; 𝜃 ? * very challenging!! • Intractable • Require strong constraints Latent variable (generative) model * slide from Aaron Courville, IFT6266 Hiver 2017 : learn a mapping from some latent variable z to a complicated distribution on x
  • 14. 14 3. Variational Auto-encoder Keywords < Variational Inference, Evidence lower bound, Reparameterization >
  • 15. 15 Variational Auto-encoder • The data we observe in the real world are very high-dimensional and sparse. • A low-dimensional, high-density nonlinear manifold exists in the space where observational data are defined. • There is a latent variable describing the manifold, which is very closely related to the variation of observed data x. Story so far • Want to get a model that generate data similar to observational data x. • To do that, we need to estimate the distribution of the data P(x). • However, direct estimation of P(x) is challenging. • Instead, let's model a conditional distribution P(x|z) using the latent variable z.
  • 16. 16 Variational Auto-encoder • where does ‘z’ come from? • How can ‘z’ be defined and obtained? Proble m * image from the cs236, Stanford 2019f - Deep Generative Models, lectue5 Since z is literally a latent variable, it is very difficult to define it manually and impossible to measure directly.
  • 17. 17 Variational Auto-encoder • 𝑥𝑖 ~ 𝑃 𝑥 𝑧) • 𝑧𝑖 ~ 𝑃(𝑧) • 𝑧𝑖 ~ 𝑃 𝑧 𝑥) Distributional assumptions Can use this sample directly but the performance is not good. Still, there is a problem… Overview for data generating process Assume a familiar distribution. 𝑃 𝑧 𝑥) = 𝑃 𝑥 𝑧)𝑃 𝑧 /𝑃(𝑥) Learn the distribution of the latent variable z, which is well explained from x. And sampling z from that dist. Assume a familiar distribution. Intractable
  • 18. 18 Variational Auto-encoder Variational Inference 𝑝𝜃 𝑧 𝑥) ≈ 𝑞𝜙(𝑧 | 𝑥) General family of methods for approximating Complicated densities by a simpler class of densities * slide from shakir Mohamed(Google DeepMind), Imperial College, London, 2015 intractab le tractabl e, familiar
  • 20. 20 Variational Auto-encoder Find objective ≥ 0 Expression 1 - generic Expression 2 - practical Tractable Variational Lower Bound!! (also called Evidence Lower Bound) Let’s maximize this ELBO ! Totally MC approx. est. KLD : Analytical solution Expectation term : MC approx. est.
  • 22. 22 Variational Auto-encoder * slide from CPSC 532L lecture 11, the University of British Columbia Problem!
  • 23. 23 Variational Auto-encoder Reparameterization trick 𝑧 ~ 𝑞𝜙 𝑧 𝑥) = 𝑁(𝜇 𝑥 , Σ(𝑥)) https://arxiv.org/abs/1606.05908
  • 24. 24 Variational Auto-encoder End2End learning pros cons • Interpretable latent space • Allows inference of q(z|x), can be useful feature representation for other tasks • Approx’ optimal • Samples are blurrier * slide from Aaron Courville, IFT6266 Hiver 2017
  • 25. 25 Variational Auto-encoder Experiments on loss term https://www.jeremyjordan.me/variational-autoencoders/
  • 27. 27 Main Reference Paper • Auto-Encoding Variational Bayes, Diederik P Kingma, Max Welling, 2013. [link] • Tutorial on Variational Autoencoders, Carl Doersch, 2016. [link] • NIPS 2016 Tutorial: Generative Adversarial Networks, Ian Goodfellow, 2016. [link] Slide • cs231n lecture slide, stanfold, 2021s. [link] • cs236n lecture slide, stanfold, 2019f. [link] • IFT6266-H2017, University of Montreal [link] Book • Deep Learning, Ian Goodfellow et al, 2016. [e-book] • Machine Learning, Ilseok Oh, 2018. etc. • Tutorial - what is a variational autoencoder? [link] • Everything about the autoencoder [video]

Editor's Notes

  1. 자동치의 위치는 삼차원 공간에 무작위로 분포하지 않고, 도로라는 일차원 비선형 공간에 분포한다. 대부분의 자동차가 도로 위에 있는데 간혹 갓길로 벗어난 자동차도 있을 것이다. 공중에 둥둥떠다니는 자동차는 없음. 아주 낮은확률로 태풍에 날라다닐때 포착되겠지 매니폴드 가정. : 샘플은 원본 데이터가 표현되는 d차원 공간에 무작위적으로 분포하는 것이 아니라, 그보다 훨씬 낮은 차원의 공간에 분포 - 다음으로 이미지 데이터를 살펴보자.
  2. 구체적으로 살펴보자 차원수가 저렇다는거고 실제로 가질수잇는 값이 0 ~ 255까지니까 나타낼 수 있는 256^(저 숫자) 인거
  3. 잠재변수 : 인간이 인위적으로 정의한 변수 / 특성이 아닌 데이터 내부에 잠재된 변수 매니폴드학습이 대표적인 representation learning이라고 볼 수 있겠음.
  4. 사실 움짤임
  5. 차원축소 / clustering / 밀도추정 등등 (밀도 추정의 경우 manifold랑 직접 연관이 없다고 생각하실 수도 있지만 어떤 유용한 Latent variable을 이용해서 추정하는 경우가 많고, 그 유용한 latent variable은 매니폴드의 기저가 되기때문에 )
  6. 일반적인 분별모델에서의 x -> y의 관계가 여기서 잠재변수 생성모델의 z -> x
  7. 분포 x|z는 z가 given이므로 예측대상이 실세계의 데이터여서 다루기 쉬운 분포로 가정할 수 있음. Z의 마지널 분포인 P(z)를 아무렇게나 단순하게 가정할 수 있다. 그리고 p(z)로부터 z를 샘플링하여 decoder에 전달할 수 있다. 그러나 좀더 x와 밀접한 관련이 있는 z를 샘플링 하여 생성모델에 넘기고 싶다. 분포 z|x는 x가 given이고 분포의 예측대상이 실세계에서 관측불가능한 잠재변수이므로 이를 p(x|z)*p(z) / p(x) 로 구해야되는데 분포 p(x)를 계산할 수 없음
  8. 일반적인 분별모델에서의 x -> y의 관계가 여기서 잠재변수 생성모델의 z -> x
  9. 일반적인 분별모델에서의 x -> y의 관계가 여기서 잠재변수 생성모델의 z -> x
  10. 일반적인 분별모델에서의 x -> y의 관계가 여기서 잠재변수 생성모델의 z -> x