Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Belief Networks (D2L1 Deep Learning for Speech and Language UPC 2017)

1,181 views

Published on

https://telecombcn-dl.github.io/2017-dlsl/

Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN.

The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.

Published in: Data & Analytics
  • Be the first to comment

Deep Belief Networks (D2L1 Deep Learning for Speech and Language UPC 2017)

  1. 1. Day 2 Lecture 1 Deep Belief Networks (DBN) Elisa Sayrol
  2. 2. Restrictive Boltzmann Machine (RBM) Training. Contrastive Divergence (CD) Deep Belief Networks (DBN) Overview
  3. 3. 3 Restricted Boltzmann Machine (RBM) Figure: Geoffrey Hinton (2013) Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007. ● Shallow two-layer net. ● Restricted=No two nodes in a layer share a connection ● Bipartite graph. ● Bidirectional graph ○ Shared weights. ○ Different biases.
  4. 4. 4 Restricted Boltzmann Machine (RBM) Figure: Geoffrey Hinton (2013) Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.
  5. 5. 5 Restricted Boltzmann Machine (RBM) DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”. Forward pass
  6. 6. 6 Restricted Boltzmann Machine (RBM) DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”. Backward pass c c c c
  7. 7. 7 Restricted Boltzmann Machine (RBM) DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”. Backward pass The reconstructed values at the visible layer are compared with the actual ones with the KL Divergence.
  8. 8. 8 What are the Maths behind RBMs? (Estimation of the parameters) Geoffrey Hinton, "Introduction to Deep Learning & Deep Belief Nets” (2012) Geoorey Hinton, “Tutorial on Deep Belief Networks”. NIPS 2007.
  9. 9. 9 What are the Maths behind RBMs? Other references: Deeplearning.net: Restricted Boltzmann Machines (with Theano functions and concepts) Hugo Larochelle: Course on NN Let’s take a look at some of his slides on RBM….
  10. 10. 10 What are the Maths behind RBMs? Hugo Larochelle Slides
  11. 11. Hugo Larochelle Slides
  12. 12. Hugo Larochelle Slides
  13. 13. Hugo Larochelle Slides
  14. 14. Hugo Larochelle Slides
  15. 15. Hugo Larochelle Slides
  16. 16. Hugo Larochelle Slides
  17. 17. Hugo Larochelle Slides
  18. 18. Hugo Larochelle Slides
  19. 19. 19 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. ● Architecture like an MLP. ● Training as a stack of RBMs.
  20. 20. 20 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. ● Architecture like an MLP. ● Training as a stack of RBMs.
  21. 21. 21 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. ● Architecture like an MLP. ● Training as a stack of RBMs.
  22. 22. 22 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. ● Architecture like an MLP. ● Training as a stack of RBMs.
  23. 23. 23 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. ● Architecture like an MLP. ● Training as a stack of RBMs… ● ...so they do not need labels: Unsupervised learning
  24. 24. 24 Deep Belief Networks (DBN) Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554. After the DBN is trained, it can be fine-tuned with a reduced amount of labels to solve a supervised task with superior performance. Supervised learning Softmax
  25. 25. Thank You!

×