Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Autoencoders
NN club, March 21 2018
Jonathan Ronen
Agenda
● PCA and linear autoencoders
● Deep and nonlinear autoencoders
● Variational autoencoders
PCA for dimensionality reduction
PCA for dimensionality reduction
● The U that maximizes the variance of PC1
● also minimizes the reconstruction error
○ No...
PCA through backpropagation
●
● This is an autoencoder
● If the neurons are linear, it is similar to PCA
○ Caveat: PCs are...
PCA vs linear autoencoders for MNIST
PCA vs linear autoencoders for MNIST
Autoencoders can be nonlinear
Nonlinear autoencoder with 32 hidden neurons
Autoencoders can be deep
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
Sigmoid
Sigmoid
Sigmoid
Sigmoid
Sigmoid
Deep autoencoder (bottleneck of 2)
Guess which one is deep (has intermediate layer)?
Many variations of autoencoders
● Sparse autoencoders
● Denoising autoencoders
● Convolutional autoencoders
○ UNet is a so...
Variational autoencoders
Variational Bayesian Inference
Variational Inference
+
autoencoders
z
x observation
latent variab...
Variational Inference (quick overview)
z
x observation
latent variable
Variational Inference (quick overview)
z
x observation
latent variable
problematic...
Variational Inference (quick overview)
z
x observation
latent variable
problematic...
Variational Inference Solution:
Chos...
Side note on
● Information
○ “How many bits do we need to represent event x if we optimized for p(x)?”
● Entropy
○ “What i...
Variational Inference (quick overview)
skipping the math...
Maximizing the Evidence LOwer Bound (ELBO)
Variational inference is methods to maximize ELBO
How does it fit in with autoencoders?
What if autoencoders were probabilistic?
What if autoencoders were probabilistic?
Regular autoencoder
Variational autoencoder
Variational Autoencoder loss - negative ELBO
reconstruction error divergence from prior
Backpropagation through VAEs
sampling
Backpropagation through VAEs - reparameterizing
VAE 2d embedding
VAEs are a generative model
Regular autoencoder as a generative model?
Jupyter Notebook with all analysis in this talk
https://nbviewer.jupyter.org/gist/jonathanronen/69902c1a97149ab4aae42e099d...
Further reading
● https://arxiv.org/abs/1312.6114
● https://www.youtube.com/watch?v=uaaqyVS9-rM
● https://www.jeremyjordan...
Upcoming SlideShare
Loading in …5
×

Jonathan Ronen - Variational Autoencoders tutorial

175 views

Published on

Presentation at Deep Learning club, BIMSB, MDC, Berlin, march 2018.
Jonathan Ronen, Akalin lab.

Published in: Data & Analytics
  • Be the first to comment

Jonathan Ronen - Variational Autoencoders tutorial

  1. 1. Autoencoders NN club, March 21 2018 Jonathan Ronen
  2. 2. Agenda ● PCA and linear autoencoders ● Deep and nonlinear autoencoders ● Variational autoencoders
  3. 3. PCA for dimensionality reduction
  4. 4. PCA for dimensionality reduction ● The U that maximizes the variance of PC1 ● also minimizes the reconstruction error ○ Note: this is not the same as OLS, which minimizes There are efficient solvers for this, but we could also use backpropagation
  5. 5. PCA through backpropagation ● ● This is an autoencoder ● If the neurons are linear, it is similar to PCA ○ Caveat: PCs are orthogonal, autoencoded components are not - but they will span the same space
  6. 6. PCA vs linear autoencoders for MNIST
  7. 7. PCA vs linear autoencoders for MNIST
  8. 8. Autoencoders can be nonlinear
  9. 9. Nonlinear autoencoder with 32 hidden neurons
  10. 10. Autoencoders can be deep ReLu ReLu ReLu ReLu ReLu ReLu ReLu ReLu ReLu ReLu Sigmoid Sigmoid Sigmoid Sigmoid Sigmoid
  11. 11. Deep autoencoder (bottleneck of 2) Guess which one is deep (has intermediate layer)?
  12. 12. Many variations of autoencoders ● Sparse autoencoders ● Denoising autoencoders ● Convolutional autoencoders ○ UNet is a sort of autoencoder ● And more… ● I’d like to introduce Variational Autoencoders
  13. 13. Variational autoencoders Variational Bayesian Inference Variational Inference + autoencoders z x observation latent variable
  14. 14. Variational Inference (quick overview) z x observation latent variable
  15. 15. Variational Inference (quick overview) z x observation latent variable problematic...
  16. 16. Variational Inference (quick overview) z x observation latent variable problematic... Variational Inference Solution: Chosen to be a distribution we can work with
  17. 17. Side note on ● Information ○ “How many bits do we need to represent event x if we optimized for p(x)?” ● Entropy ○ “What is the expected amount of information in each event drawn from p(x)?” (how many bits?) ● Cross-entropy ○ “What is the expected amount of information in p(x) if we optimized for q(x)?” (how many bits?) ● Kullback-Leibler divergence: “cross-entropy - entropy” ○ “How many more bits will we need to represent events from p(x) if we optimized for q(x)?
  18. 18. Variational Inference (quick overview) skipping the math... Maximizing the Evidence LOwer Bound (ELBO)
  19. 19. Variational inference is methods to maximize ELBO How does it fit in with autoencoders?
  20. 20. What if autoencoders were probabilistic?
  21. 21. What if autoencoders were probabilistic? Regular autoencoder Variational autoencoder
  22. 22. Variational Autoencoder loss - negative ELBO reconstruction error divergence from prior
  23. 23. Backpropagation through VAEs sampling
  24. 24. Backpropagation through VAEs - reparameterizing
  25. 25. VAE 2d embedding
  26. 26. VAEs are a generative model
  27. 27. Regular autoencoder as a generative model?
  28. 28. Jupyter Notebook with all analysis in this talk https://nbviewer.jupyter.org/gist/jonathanronen/69902c1a97149ab4aae42e099d1d1367
  29. 29. Further reading ● https://arxiv.org/abs/1312.6114 ● https://www.youtube.com/watch?v=uaaqyVS9-rM ● https://www.jeremyjordan.me/variational-autoencoders/ ● https://blog.keras.io/building-autoencoders-in-keras.html

×