Baptiste Wicht, profile picture
Uploaded byBaptiste Wicht
9,659 views

Deep learning presentation

The document discusses deep learning, focusing on various architectures like Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN), including their definitions, history, algorithms, and applications. It highlights the complexities involved in implementing these models and the challenges of training them effectively. Additionally, it covers future directions for research and potential refinements in deep learning techniques.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Downloaded 319 times
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Deep Learning Baptiste Wicht baptiste.wicht@gmail.com September 12, 2014 Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Table of Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Contents 1 Deep Learning Definition History Usages Difficulties 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition Deep Learning (Wikipedia) Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using model architectures composed of multiple non-linear transformations Deep Learning (deeplearning.net) Deep Learning is a new area of Machine Learning research, which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence. Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition (cont.d) Goal: Imitate the nature Set of algorithms Generally structures with multiple layers Often unsupervised feature learning Time-consuming training Sometimes large amount of data Generally complex data New name for an old thing hot topic Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties History 1960: Neural networks 1985: Multilayer Perceptrons 1986: Restricted Boltzmann Machine 1995: Support Vector Machine 2006: Hinton presents the Deep Belief Network (DBN) New interests in deep learning and RBM State of the art MNIST 2009: Deep Recurrent Neural Network 2010: Convolutional DBN 2011: Max-Pooling CDBN Many competitions won and state of the art results Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Names Geoffrey Hinton Andrew Y. Ng Yoshua Bengio Honglak Lee Yann LeCun ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Algorithms Deep Neural Networks Deep Belief Networks Convolutional Deep Belief Networks Deep SVM Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Usages Text recognition Facial Expression Recognition Object Recognition Audio classification Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Difficulties Large number of free variables Few insights on how to set them Complex to implement Large variations between papers Lot of refinements were proposed Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contents 1 Deep Learning 2 Restricted Boltzmann Machine Definition Training Units Variants 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition Restricted Boltzmann Machine Function: Learn a probability distribution over the input Generative stochastic neural network Visible and hidden neurons Neurons form a bipartite graph V visible units and visible biases H hidden units and hidden biases VxH weights Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition (Cont.d) Binary units (Bernoulli RBM) p(hj = 1|v) = (cj + mX i viwi,j ) p(vi = 1|h) = (bi + nX j hjwi,j ) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Usages Unsupervised feature learning Classification with other techniques (linear classifier, SVM, ...) Limited to one layer of abstraction Stacking for higher-level models and classification Deep Belief Network Deep Boltzmann Machines Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Training Objective: Maximizing the log-likelihood Intractable Other methods have been developed: Markov Chain Monte Carlo (MCMC) (Too slow) Contrastive Divergence (CD) (Hinton) Persistent CD Mean-Field CD (mf-CD) Parallel Tempering Annealed Importance Sampling Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence For each data point 1 Compute gradients g between t = k and t = k − 1 2 Add g to the weights and the biases Repeat for several epochs Experiments have shown that CD1 (k = 1) works well Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence When to stop training ? 1 Proxies to log-likelihood: Reconstruction error Pseudo-likelihood (PCD) 2 Visual inspection of the filters Training is relatively fast Can be trained on GPU Hard to compare two RBMs Hard to test an implementation correctly Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence Options Mini-batch training Momentum Weight decay Sparsity Target ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Units RBM Was initially developed with binary units Different types of units can be used: Gaussian visible units for real-value inputs Softmax hidden unit for classification (last layer) Rectified Linear Unit (ReLU) units for hidden/visible Can be capped Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Variants Convolutional RBM (see later) mean-covariance RBM (mcRBM) Sparse RBM (SRBM) Third-Order RBM Spike And Slab RBM Nonnegative RBM ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network Definition Training 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition Deep Belief Network Generative graphical model Type of Deep Neural Network Multiple layer of hidden units Stack of RBMs Can be implemented with other autoencoders Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition (Cont.d) Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Training 1 Train each layer, from bottom to top, with Contrastive Divergence (Unsupervised) 2 Then treat the DBN as a MLP 3 If necessary, fine-tune the last layer for classification (Supervised) Back propagation nonlinear Conjugate Gradient method Limited Memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) Hessian-Free CG (Martens) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM Definition Training Probabilistic Max Pooling 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition Convolutional RBM Motivation: Translation-invariance Scaling to full-size images Variant of RBM, concepts remain the same NV xNV binary visible units K groups of hidden units NKxNK binary hidden units per group Each group has a NWxNW filter (NW , NV − NH + 1) A bias bk for each hidden group A single bias c for all visible units Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(hk j = 1|v) = (bk + (W~ k v v)j ) p(vi = 1|h) = (c + KX k (Wk f hk )i) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Training Contrastive Divergence Gradients computations are done with convolutions Same refinements can be used (weight decay, momentum, ...) CRBM is highly overcomplete Sparse learning is very important Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Probabilistic Max Pooling Shrink the representation by a constant factor C Allows higher-level to be invariant to small translations Reduces computational effort Generative version of standard Max Pooling Pooling layer with K groups of pooling units Each group has NPxNP units NP , NH/C Each hidden block (CxC) is connected to exactly one pooling unit Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(vi = 1|h) = (c + KX k (Wk f hk )i) I(hk j ) , bk + (W~ k v v)j p(hk j = 1|v) = exp(I(hk i )) 1 + P j02
exp(I(hk i0)) p(pk = 0|v) = 1 1 + P j02
exp(I(hk i0)) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Stack of Convolutional RBM With or without Probabilistic Max Pooling Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Feature Learning Source: Honglak Lee Each layer learns a different abstraction of features 1 Stroke 2 Parts of faces 3 Faces Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Implementation Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation Deep Learning Library (DLL) https://github.com/wichtounet/dll RBM Binary, Gaussian, Softmax, ReLU units CD and PCD Momentum, Weight Decay, Sparsity Target Convolutional RBM Standard version Probabilistic Max Pooling Various units CD and PCD Momentum, Weight Decay, Sparsity Target Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
Deep Learning Restricted Boltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation DBN Pretraining with RBM Fine-tuning with Conjugate Gradient Fine-tuning with Stochastic Gradient Descent Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning

More Related Content

PPTX
Deep Belief nets
bybutest
 
PDF
Restricted Boltzmann Machine - A comprehensive study with a focus on Deep Bel...
byIndraneel Pole
 
PPT
NIPS2007: deep belief nets
byzukun
 
PDF
Deep Belief Networks
byHasan H Topcu
 
PDF
P05 deep boltzmann machines cvpr2012 deep learning methods for vision
byzukun
 
PDF
Deep Belief Networks (D2L1 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
PDF
Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hier...
byMad Scientists
 
PPTX
Deep Belief Networks for Spam Filtering
bybutest
 
Deep Belief nets
bybutest
 
Restricted Boltzmann Machine - A comprehensive study with a focus on Deep Bel...
byIndraneel Pole
 
NIPS2007: deep belief nets
byzukun
 
Deep Belief Networks
byHasan H Topcu
 
P05 deep boltzmann machines cvpr2012 deep learning methods for vision
byzukun
 
Deep Belief Networks (D2L1 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hier...
byMad Scientists
 
Deep Belief Networks for Spam Filtering
bybutest
 

What's hot

PPTX
Deep belief networks for spam filtering
bySOYEON KIM
 
PPTX
Multilayer & Back propagation algorithm
byswapnac12
 
PPT
Deep Learning
byRoshan Chettri
 
PPTX
Review-image-segmentation-by-deep-learning
byTrong-An Bui
 
PPTX
Deep learning lecture - part 1 (basics, CNN)
bySungminYou
 
PDF
Introduction to Applied Machine Learning
bySheilaJimenezMorejon
 
PPT
Chapter 09 class advanced
byHouw Liong The
 
PDF
Lecture 7: Recurrent Neural Networks
bySang Jun Lee
 
PPTX
Acoustic modeling using deep belief networks
byYueshen Xu
 
PDF
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
bycsandit
 
PPTX
Convolutional neural network from VGG to DenseNet
bySungminYou
 
PDF
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
byIOSR Journals
 
PPT
Ann
byvini89
 
PDF
deep CNN vs conventional ML
byChao Han chaohan@vt.edu
 
PDF
09 classadvanced
byJoonyoungJayGwak
 
PPT
lecture07.ppt
bybutest
 
PDF
RNN and its applications
bySungjoon Choi
 
PPTX
A Framework for Scene Recognition Using Convolutional Neural Network as Featu...
byTahmid Abtahi
 
PPTX
Neural networks
byIndira Priyadarsini
 
PDF
P03 neural networks cvpr2012 deep learning methods for vision
byzukun
 
Deep belief networks for spam filtering
bySOYEON KIM
 
Multilayer & Back propagation algorithm
byswapnac12
 
Deep Learning
byRoshan Chettri
 
Review-image-segmentation-by-deep-learning
byTrong-An Bui
 
Deep learning lecture - part 1 (basics, CNN)
bySungminYou
 
Introduction to Applied Machine Learning
bySheilaJimenezMorejon
 
Chapter 09 class advanced
byHouw Liong The
 
Lecture 7: Recurrent Neural Networks
bySang Jun Lee
 
Acoustic modeling using deep belief networks
byYueshen Xu
 
Objective Evaluation of a Deep Neural Network Approach for Single-Channel Spe...
bycsandit
 
Convolutional neural network from VGG to DenseNet
bySungminYou
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
byIOSR Journals
 
Ann
byvini89
 
deep CNN vs conventional ML
byChao Han chaohan@vt.edu
 
09 classadvanced
byJoonyoungJayGwak
 
lecture07.ppt
bybutest
 
RNN and its applications
bySungjoon Choi
 
A Framework for Scene Recognition Using Convolutional Neural Network as Featu...
byTahmid Abtahi
 
Neural networks
byIndira Priyadarsini
 
P03 neural networks cvpr2012 deep learning methods for vision
byzukun
 

Viewers also liked

PDF
Deep Learning - The Past, Present and Future of Artificial Intelligence
byLukas Masuch
 
PDF
Deep Learning through Examples
bySri Ambati
 
PDF
Deep Learning - Convolutional Neural Networks
byChristian Perone
 
PDF
Deep Learning for NLP: An Introduction to Neural Word Embeddings
byRoelof Pieters
 
PDF
Convolutional Neural Networks (CNN)
byGaurav Mittal
 
PDF
Introduction to Machine Learning and Deep Learning
byTerry Taewoong Um
 
PPTX
Deep Learning in Computer Vision
bySungjoon Choi
 
PDF
Deep learning - Conceptual understanding and applications
byBuhwan Jeong
 
PDF
Deep Learning Computer Build
byPetteriTeikariPhD
 
PDF
Passive stereo vision with deep learning
byYu Huang
 
PDF
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
byRoelof Pieters
 
PDF
Large Scale Deep Learning with TensorFlow
byJen Aman
 
PDF
Data Science - Part XVII - Deep Learning & Image Processing
byDerek Kane
 
PPTX
Deep neural networks
bySi Haem
 
PDF
DSRLab seminar Introduction to deep learning
byPoo Kuan Hoong
 
PPTX
Ravi ubana presentation on cloud
byRavi Ubana
 
KEY
Intalio|Cloud Presentation
byintalio_bpm
 
PDF
Howest presentation "Cloud computing"
byPhilippe Creytens
 
PDF
CENDOO & ai one SemTech presentation
byCENDOO AG
 
PDF
Deep learning review
byManas Gaur
 
Deep Learning - The Past, Present and Future of Artificial Intelligence
byLukas Masuch
 
Deep Learning through Examples
bySri Ambati
 
Deep Learning - Convolutional Neural Networks
byChristian Perone
 
Deep Learning for NLP: An Introduction to Neural Word Embeddings
byRoelof Pieters
 
Convolutional Neural Networks (CNN)
byGaurav Mittal
 
Introduction to Machine Learning and Deep Learning
byTerry Taewoong Um
 
Deep Learning in Computer Vision
bySungjoon Choi
 
Deep learning - Conceptual understanding and applications
byBuhwan Jeong
 
Deep Learning Computer Build
byPetteriTeikariPhD
 
Passive stereo vision with deep learning
byYu Huang
 
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
byRoelof Pieters
 
Large Scale Deep Learning with TensorFlow
byJen Aman
 
Data Science - Part XVII - Deep Learning & Image Processing
byDerek Kane
 
Deep neural networks
bySi Haem
 
DSRLab seminar Introduction to deep learning
byPoo Kuan Hoong
 
Ravi ubana presentation on cloud
byRavi Ubana
 
Intalio|Cloud Presentation
byintalio_bpm
 
Howest presentation "Cloud computing"
byPhilippe Creytens
 
CENDOO & ai one SemTech presentation
byCENDOO AG
 
Deep learning review
byManas Gaur
 

Similar to Deep learning presentation

PPTX
MODULE V_REINFORCEMENT LEARNING _Machine Learning.pptx
byarchanaonlineexam
 
PPTX
Introduction to un supervised learning
byRishikesh .
 
PPTX
Deep belief network.pptx
bySushilAcharya18
 
PDF
Deep Generative Models
byMijung Kim
 
PPT
Deep Beleif Networks
byDeepak Singh
 
PPTX
Boltzmann Machines in Deep learning and machine learning also used for traini...
byvenkatasaisumanth74
 
PDF
A Survey of Deep Learning Algorithms for Malware Detection
byIJCSIS Research Publications
 
PPTX
Deep learning from a novice perspective
byAnirban Santara
 
PPTX
Georgia Tech cse6242 - Intro to Deep Learning and DL4J
byJosh Patterson
 
PDF
Unsupervised Learning (DLAI D9L1 2017 UPC Deep Learning for Artificial Intell...
byUniversitat Politècnica de Catalunya
 
PDF
Fundamental of deep learning
byStanley Wang
 
PPTX
Presentation on Deep Learning lecture_30.pptx
bymasudcse1
 
PDF
Dl1 deep learning_algorithms
byArmando Vieira
 
PDF
ProjectReport
byPritish Yuvraj
 
PPTX
Tutorial on Deep learning and Applications
byNhatHai Phan
 
PPTX
Introsuction to restricted boltzman machine
byRajanRock3
 
PDF
2010 deep learning and unsupervised feature learning
byVan Thanh
 
PPTX
Thesis Presentation
byReuben Feinman
 
PDF
Deep learning
bySimon Belak
 
PDF
Restricted Boltzmann Machine | Neural Network Tutorial | Deep Learning Tutori...
byEdureka!
 
MODULE V_REINFORCEMENT LEARNING _Machine Learning.pptx
byarchanaonlineexam
 
Introduction to un supervised learning
byRishikesh .
 
Deep belief network.pptx
bySushilAcharya18
 
Deep Generative Models
byMijung Kim
 
Deep Beleif Networks
byDeepak Singh
 
Boltzmann Machines in Deep learning and machine learning also used for traini...
byvenkatasaisumanth74
 
A Survey of Deep Learning Algorithms for Malware Detection
byIJCSIS Research Publications
 
Deep learning from a novice perspective
byAnirban Santara
 
Georgia Tech cse6242 - Intro to Deep Learning and DL4J
byJosh Patterson
 
Unsupervised Learning (DLAI D9L1 2017 UPC Deep Learning for Artificial Intell...
byUniversitat Politècnica de Catalunya
 
Fundamental of deep learning
byStanley Wang
 
Presentation on Deep Learning lecture_30.pptx
bymasudcse1
 
Dl1 deep learning_algorithms
byArmando Vieira
 
ProjectReport
byPritish Yuvraj
 
Tutorial on Deep learning and Applications
byNhatHai Phan
 
Introsuction to restricted boltzman machine
byRajanRock3
 
2010 deep learning and unsupervised feature learning
byVan Thanh
 
Thesis Presentation
byReuben Feinman
 
Deep learning
bySimon Belak
 
Restricted Boltzmann Machine | Neural Network Tutorial | Deep Learning Tutori...
byEdureka!
 

Recently uploaded

PDF
Transcript: What Thema can do: Leveraging metadata to support the discoverabi...
byBookNet Canada
 
PDF
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
PDF
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
PPTX
INSY_7213_Theme Sessions 47 & 48 Advanced databases.pptx
bystormzy56
 
PDF
Quick Learn Laravel for Beginner from Zero to Survive
byiDev Semarang
 
PDF
Parental Control App for Phones_ The Complete 2026 Guide for Safer, Smarter P...
byRyan Cooper
 
PDF
Computer-Based Training (CBT) The Backbone of Modern Technical & Defence Trai...
bycomputerbasedtrainin1
 
PDF
Afsar Ahmed - Digital Marketing Portfolio
byAfsar Ahmed
 
PDF
React Mastery: Visual Mental Models to Understand React Deeply
byThomas Gaye
 
PDF
"When every team does things "right", but together it turns into chaos", Yozh...
byFwdays
 
PDF
EU regulations for the North American book supply chain - Tech Forum 2026
byBookNet Canada
 
PDF
2026_01_28 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
PPTX
2026 SCORM Troubleshooting Rustici + dominKnow.pptx
byRustici Software
 
PDF
Fortinet Certified Fundamentals in Cybersecurity
byVICTOR MAESTRE RAMIREZ
 
PDF
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
PDF
Taxonomy Alignment for SDG Tagging - ASHA Case Study
byEnterprise Knowledge
 
PPTX
Jisc Equipment Data Service Update Workshop 3 December 2025
byJisc
 
PPTX
Code Like Bro -A guide for better Coding
byMadan Panthi
 
PDF
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
PDF
JCB_3CX_SPECS (24 págs) companywrench.com .pdf
byAquilesOyarzn1
 
Transcript: What Thema can do: Leveraging metadata to support the discoverabi...
byBookNet Canada
 
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
INSY_7213_Theme Sessions 47 & 48 Advanced databases.pptx
bystormzy56
 
Quick Learn Laravel for Beginner from Zero to Survive
byiDev Semarang
 
Parental Control App for Phones_ The Complete 2026 Guide for Safer, Smarter P...
byRyan Cooper
 
Computer-Based Training (CBT) The Backbone of Modern Technical & Defence Trai...
bycomputerbasedtrainin1
 
Afsar Ahmed - Digital Marketing Portfolio
byAfsar Ahmed
 
React Mastery: Visual Mental Models to Understand React Deeply
byThomas Gaye
 
"When every team does things "right", but together it turns into chaos", Yozh...
byFwdays
 
EU regulations for the North American book supply chain - Tech Forum 2026
byBookNet Canada
 
2026_01_28 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
2026 SCORM Troubleshooting Rustici + dominKnow.pptx
byRustici Software
 
Fortinet Certified Fundamentals in Cybersecurity
byVICTOR MAESTRE RAMIREZ
 
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
Taxonomy Alignment for SDG Tagging - ASHA Case Study
byEnterprise Knowledge
 
Jisc Equipment Data Service Update Workshop 3 December 2025
byJisc
 
Code Like Bro -A guide for better Coding
byMadan Panthi
 
The Enterprise Web3 Landscape - a view from Kaleido based on the past 10 year...
byPeter Broadhurst
 
JCB_3CX_SPECS (24 págs) companywrench.com .pdf
byAquilesOyarzn1
 

Deep learning presentation

  • 1.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Deep Learning Baptiste Wicht baptiste.wicht@gmail.com September 12, 2014 Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 2.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Table of Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 3.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Contents 1 Deep Learning Definition History Usages Difficulties 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 4.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition Deep Learning (Wikipedia) Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using model architectures composed of multiple non-linear transformations Deep Learning (deeplearning.net) Deep Learning is a new area of Machine Learning research, which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence. Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 5.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Definition (cont.d) Goal: Imitate the nature Set of algorithms Generally structures with multiple layers Often unsupervised feature learning Time-consuming training Sometimes large amount of data Generally complex data New name for an old thing hot topic Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 6.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties History 1960: Neural networks 1985: Multilayer Perceptrons 1986: Restricted Boltzmann Machine 1995: Support Vector Machine 2006: Hinton presents the Deep Belief Network (DBN) New interests in deep learning and RBM State of the art MNIST 2009: Deep Recurrent Neural Network 2010: Convolutional DBN 2011: Max-Pooling CDBN Many competitions won and state of the art results Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 7.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Names Geoffrey Hinton Andrew Y. Ng Yoshua Bengio Honglak Lee Yann LeCun ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 8.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Algorithms Deep Neural Networks Deep Belief Networks Convolutional Deep Belief Networks Deep SVM Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 9.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Usages Text recognition Facial Expression Recognition Object Recognition Audio classification Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 10.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition History Usages Difficulties Difficulties Large number of free variables Few insights on how to set them Complex to implement Large variations between papers Lot of refinements were proposed Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 11.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contents 1 Deep Learning 2 Restricted Boltzmann Machine Definition Training Units Variants 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 12.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition Restricted Boltzmann Machine Function: Learn a probability distribution over the input Generative stochastic neural network Visible and hidden neurons Neurons form a bipartite graph V visible units and visible biases H hidden units and hidden biases VxH weights Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 13.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Definition (Cont.d) Binary units (Bernoulli RBM) p(hj = 1|v) = (cj + mX i viwi,j ) p(vi = 1|h) = (bi + nX j hjwi,j ) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 14.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 15.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 16.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 17.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 18.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Example Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 19.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Usages Unsupervised feature learning Classification with other techniques (linear classifier, SVM, ...) Limited to one layer of abstraction Stacking for higher-level models and classification Deep Belief Network Deep Boltzmann Machines Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 20.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Training Objective: Maximizing the log-likelihood Intractable Other methods have been developed: Markov Chain Monte Carlo (MCMC) (Too slow) Contrastive Divergence (CD) (Hinton) Persistent CD Mean-Field CD (mf-CD) Parallel Tempering Annealed Importance Sampling Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 21.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence For each data point 1 Compute gradients g between t = k and t = k − 1 2 Add g to the weights and the biases Repeat for several epochs Experiments have shown that CD1 (k = 1) works well Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 22.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence When to stop training ? 1 Proxies to log-likelihood: Reconstruction error Pseudo-likelihood (PCD) 2 Visual inspection of the filters Training is relatively fast Can be trained on GPU Hard to compare two RBMs Hard to test an implementation correctly Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 23.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Contrastive Divergence Options Mini-batch training Momentum Weight decay Sparsity Target ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 24.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Units RBM Was initially developed with binary units Different types of units can be used: Gaussian visible units for real-value inputs Softmax hidden unit for classification (last layer) Rectified Linear Unit (ReLU) units for hidden/visible Can be capped Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 25.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Units Variants Variants Convolutional RBM (see later) mean-covariance RBM (mcRBM) Sparse RBM (SRBM) Third-Order RBM Spike And Slab RBM Nonnegative RBM ... Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 26.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network Definition Training 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 27.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition Deep Belief Network Generative graphical model Type of Deep Neural Network Multiple layer of hidden units Stack of RBMs Can be implemented with other autoencoders Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 28.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Definition (Cont.d) Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 29.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Training 1 Train each layer, from bottom to top, with Contrastive Divergence (Unsupervised) 2 Then treat the DBN as a MLP 3 If necessary, fine-tune the last layer for classification (Supervised) Back propagation nonlinear Conjugate Gradient method Limited Memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) Hessian-Free CG (Martens) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 30.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM Definition Training Probabilistic Max Pooling 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 31.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition Convolutional RBM Motivation: Translation-invariance Scaling to full-size images Variant of RBM, concepts remain the same NV xNV binary visible units K groups of hidden units NKxNK binary hidden units per group Each group has a NWxNW filter (NW , NV − NH + 1) A bias bk for each hidden group A single bias c for all visible units Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 32.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(hk j = 1|v) = (bk + (W~ k v v)j ) p(vi = 1|h) = (c + KX k (Wk f hk )i) Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 33.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Training Contrastive Divergence Gradients computations are done with convolutions Same refinements can be used (weight decay, momentum, ...) CRBM is highly overcomplete Sparse learning is very important Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 34.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Probabilistic Max Pooling Shrink the representation by a constant factor C Allows higher-level to be invariant to small translations Reduces computational effort Generative version of standard Max Pooling Pooling layer with K groups of pooling units Each group has NPxNP units NP , NH/C Each hidden block (CxC) is connected to exactly one pooling unit Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 35.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Training Probabilistic Max Pooling Definition (Cont.d) Binary units: p(vi = 1|h) = (c + KX k (Wk f hk )i) I(hk j ) , bk + (W~ k v v)j p(hk j = 1|v) = exp(I(hk i )) 1 + P j02
  • 36.
    exp(I(hk i0)) p(pk = 0|v) = 1 1 + P j02
  • 37.
    exp(I(hk i0)) BaptisteWichtbaptiste.wicht@gmail.com Deep Learning
  • 38.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 39.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Definition Stack of Convolutional RBM With or without Probabilistic Max Pooling Each RBM takes input from previous layer output Each layer forms a higher-level representation of the data Number of hidden units in each layer can be tuned Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 40.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Feature Learning Source: Honglak Lee Each layer learns a different abstraction of features 1 Stroke 2 Parts of faces 3 Faces Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 41.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Contents 1 Deep Learning 2 Restricted Boltzmann Machine 3 Deep Belief Network 4 Convolutional RBM 5 Convolutional DBN 6 Conclusion Implementation Conclusion Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 42.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation Deep Learning Library (DLL) https://github.com/wichtounet/dll RBM Binary, Gaussian, Softmax, ReLU units CD and PCD Momentum, Weight Decay, Sparsity Target Convolutional RBM Standard version Probabilistic Max Pooling Various units CD and PCD Momentum, Weight Decay, Sparsity Target Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning
  • 43.
    Deep Learning RestrictedBoltzmann Machine Deep Belief Network Convolutional RBM Convolutional DBN Conclusion Implementation Conclusion Implementation DBN Pretraining with RBM Fine-tuning with Conjugate Gradient Fine-tuning with Stochastic Gradient Descent Baptiste Wichtbaptiste.wicht@gmail.com Deep Learning