SlideShare a Scribd company logo

Deep neural networks

S
Si Haem

Deep learning and neural networks are inspired by biological neurons. Artificial neural networks (ANN) can have multiple layers and learn through backpropagation. Deep neural networks with multiple hidden layers did not work well until recent developments in unsupervised pre-training of layers. Experiments on MNIST digit recognition and NORB object recognition datasets showed deep belief networks and deep Boltzmann machines outperform other models. Deep learning is now widely used for applications like computer vision, natural language processing, and information retrieval.

1 of 36
Download to read offline
Deep Learning and Application in
       Neural Networks


   Hugo Larochelle    Geoffrey Hinton
    Yoshua Bengio       Andrew Ng.
  Jerome Louradour   Andrew L. Nelson
    Pascal Lamblin   R. Salskhutdinov
Biological Neurons




                                        Terminal Branches
                     Dendrites
                                             of Axon




                                 Axon
Artificial Neural Networks (ANN)

                                Terminal Branches
             Dendrites
                                     of Axon
  x1
       w1
  x2
       w2
  x3   w3




                         Axon

       wn
  xn




                                Slide credit : Andrew L. Nelson
Layered Networks




Input nodes                                        Output nodes
                       Hidden nodes
        Connections

              Output : yi   f ( wi1x1 wi2x 2 wi3x 3  wim x m )
                            f(       wij x j )
                                 j
Neural network application
• ALVINN drives 70 mph on highways




                                     Slide credit : T.Michell
Neural network application




                      Slide credit : Andraw Ng.
The simplest model- the Perceptron
  •The Perceptron was introduced in 1957 by
  Frank Rosenblatt.



Perceptron:                              -                             D0


                                        d
                                                                       D1



Activation                                                             D2
functions:



Learning:                                     Input     Output
                                              Layer               Destinations
                                                        Layer
              Update




                                                      Slide credit : Geoffrey Hinton
Second generation neural networks (~1985)
                      Compare outputs with
Back-propagate        correct answer to get
error signal to get   error signal
derivatives for
learning                                             outputs




                                                            hidden
                                                            layers


                                                    input vector

                                              Slide credit : Geoffrey Hinton
BP-algorithm
                                       1
                                       .5

                                       0
                                            -5   0   5
 Activations




                       errors




The error:
                       Update




Update
Weights:
                                Slide credit : Geoffrey Hinton
Back Propagation
 Advantages
 • Multi layer Perceptron network can be
 trained by the back propagation algorithm
 to perform any mapping between the
 input and the output.

What is wrong with back-propagation?
 •It requires labeled training data.
       Almost all data is unlabeled.
 •The learning time does not scale well
       It is very slow in networks with
       multiple hidden layers.
 •It can get stuck in poor local optima.




                                             Slide credit : Geoffrey Hinton
Why Deep multi-layered neural network

              object models




              object parts
              (combination
              of edges)



             edges




         pixels
Deep neural networks
Deep Neural Networks
• Standard learning strategy
  – Randomly initializing the weights of the network
  – Applying gradient descent using backpropagation
• But, backpropagation does not work well
  (if randomly initialized)
  – Deep networks trained with back-propagation
    (without unsupervised pre-train) perform worse
    than shallow networks
  – ANN have limited to one or two layers
Deep neural networks
Deep neural networks
Deep neural networks
Unsupervised greedy layer-wise
     training procedure.
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Deep neural networks
Layer-Local Unsupervised Learning
• Restricted Boltzmann Machine (SRBM) (Hinton et al,
    NC’2006)
•   Auto-encoders (Bengio et al, NIPS’2006)
•   Sparse auto-encoders (Ranzato et al, NIPS’2006)
•   Kernel PCA (Erhan 2008)
•   Denoising auto-encoders (Vincent et al, ICML’2008)
•   Unsupervised embedding (Weston et al, ICML’2008)
•   Slow features (Mohabi et al, ICML’2009, Bergstra & Bengio
    NIPS’2009)
Experiments
• MNIST data set
  – A benchmark for handwritten digit recognition
  – The number of classes is 10 (corresponding to the
    digits from 0 to 9)
  – The inputs were scaled between 0 and 1




                                Exploring Strategies for Training Deep Neural Networks.
 Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, Pascal Lamblin; 10(Jan):1--40, 2009.
Classification error on MNIST training
Experiments
• Histograms presenting the test errors obtained on MNIST
  using models trained with or without pre-training.




         The difficulty of training deep architectures and the effect of unsupervised pre-training.
       Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent;
                                                                              pages 153-160, 2009.
Experiments
• NORB data set
  – 5 object categories, 5 difference objects within
    each category.
• Classification error rate
  – DBM : 10.8 %
  – SVM : 11.6 %
  – Logistic Regression : 22.5 %
  – KNN : 18.4 %


                               Efficient Learning of Deep Boltzmann Machines.
                                        Ruslan Salakhutdinov, Hugo Larochelle ;
                                                   JMLR W&CP 9:693-700, 2010.
Conclusion
• Deep Learning : powerful arguments &
  generalization priciples
• Unsupervised Feature Learning is crucial
  many new algorithms and applications in
  recent years
• Deep Learning suited for multi-task learning,
  domain adaptation and semi-learning with
  few labels
Application
• Classification (Bengio et al., 2007; Ranzato et al., 2007b;
    Larochelle et al., 2007; Ranzato et al.,2008)
• Regression (Salakhutdinov and Hinton, 2008)
• Dimensionality Reduction (Hinton Salakhutdinov, 2006;
    Salakhutdinov and Hinton, 2007)
•   Modeling Textures (Osindero and Hinton, 2008)
•   Information Retrieval (Salakhutdinov and Hinton, 2007)
•   Robotics (Hadsell et al., 2008)
•   Natural Language Processing (Collobert and Weston, 2008;
    Weston et al., 2008)
• Collaborative Filtering (Salakhutdinov et al., 2007)
Recent Deep Learning Highlights
Recent Deep Learning Highlights
• Google Goggles uses Stacked Sparse Auto Encoders
  (Hartmut Neven @ ICML 2011)
• The monograph or review paper Learning Deep Architectures for AI
  (Foundations & Trends in Machine Learning, 2009).
• Exploring Strategies for Training Deep Neural Networks, Hugo
  Larochelle, Yoshua Bengio, Jerome Louradour and Pascal Lamblin
  in: The Journal of Machine Learning Research, pages 1-40, 2009.
• The LISA publications database contains a deep architectures
  category. http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/
  ReadingOnDeepNetworks
• Deep Machine Learning – A New Frontier in Artificial Intelligence
  Research – a survey paper by Itamar Arel, Derek C. Rose, and
  Thomas P. Karnowski.
Thank You

Recommended

Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep LearningOswald Campesato
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...Simplilearn
 
Deep Learning With Neural Networks
Deep Learning With Neural NetworksDeep Learning With Neural Networks
Deep Learning With Neural NetworksAniket Maurya
 
Machine Learning: Introduction to Neural Networks
Machine Learning: Introduction to Neural NetworksMachine Learning: Introduction to Neural Networks
Machine Learning: Introduction to Neural NetworksFrancesco Collova'
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learningleopauly
 
Deep Learning - Convolutional Neural Networks
Deep Learning - Convolutional Neural NetworksDeep Learning - Convolutional Neural Networks
Deep Learning - Convolutional Neural NetworksChristian Perone
 

More Related Content

What's hot

An introduction to Deep Learning
An introduction to Deep LearningAn introduction to Deep Learning
An introduction to Deep LearningJulien SIMON
 
Deep Learning Tutorial
Deep Learning TutorialDeep Learning Tutorial
Deep Learning TutorialAmr Rashed
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkPrakash K
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Suraj Aavula
 
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...Simplilearn
 
Artificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesArtificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesMohammed Bennamoun
 
Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNNNoura Hussein
 
Intro to deep learning
Intro to deep learning Intro to deep learning
Intro to deep learning David Voyles
 
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...Edureka!
 
Deep Belief Networks
Deep Belief NetworksDeep Belief Networks
Deep Belief NetworksHasan H Topcu
 
Introduction Of Artificial neural network
Introduction Of Artificial neural networkIntroduction Of Artificial neural network
Introduction Of Artificial neural networkNagarajan
 
Adaptive Resonance Theory
Adaptive Resonance TheoryAdaptive Resonance Theory
Adaptive Resonance TheoryNaveen Kumar
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronMostafa G. M. Mostafa
 
Image classification with Deep Neural Networks
Image classification with Deep Neural NetworksImage classification with Deep Neural Networks
Image classification with Deep Neural NetworksYogendra Tamang
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentMuhammad Rasel
 

What's hot (20)

An introduction to Deep Learning
An introduction to Deep LearningAn introduction to Deep Learning
An introduction to Deep Learning
 
Deep Learning Tutorial
Deep Learning TutorialDeep Learning Tutorial
Deep Learning Tutorial
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)
 
Multi Layer Network
Multi Layer NetworkMulti Layer Network
Multi Layer Network
 
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...
Deep Learning Tutorial | Deep Learning Tutorial For Beginners | What Is Deep ...
 
Artificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rulesArtificial Neural Networks Lect3: Neural Network Learning rules
Artificial Neural Networks Lect3: Neural Network Learning rules
 
Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNN
 
Intro to deep learning
Intro to deep learning Intro to deep learning
Intro to deep learning
 
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...
What is Deep Learning | Deep Learning Simplified | Deep Learning Tutorial | E...
 
Deep learning
Deep learningDeep learning
Deep learning
 
Deep Belief Networks
Deep Belief NetworksDeep Belief Networks
Deep Belief Networks
 
Lecture11 - neural networks
Lecture11 - neural networksLecture11 - neural networks
Lecture11 - neural networks
 
Introduction Of Artificial neural network
Introduction Of Artificial neural networkIntroduction Of Artificial neural network
Introduction Of Artificial neural network
 
Adaptive Resonance Theory
Adaptive Resonance TheoryAdaptive Resonance Theory
Adaptive Resonance Theory
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
 
Image classification with Deep Neural Networks
Image classification with Deep Neural NetworksImage classification with Deep Neural Networks
Image classification with Deep Neural Networks
 
Deep learning
Deep learningDeep learning
Deep learning
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descent
 

Viewers also liked

Deep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsDeep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsBuhwan Jeong
 
Deep Learning through Examples
Deep Learning through ExamplesDeep Learning through Examples
Deep Learning through ExamplesSri Ambati
 
Bắt đầu học data science
Bắt đầu học data scienceBắt đầu học data science
Bắt đầu học data scienceHong Ong
 
Bắt đầu nghiên cứu Big Data
Bắt đầu nghiên cứu Big DataBắt đầu nghiên cứu Big Data
Bắt đầu nghiên cứu Big DataHong Ong
 
Hands-on Deep Learning in Python
Hands-on Deep Learning in PythonHands-on Deep Learning in Python
Hands-on Deep Learning in PythonImry Kissos
 
Hadoop and Machine Learning
Hadoop and Machine LearningHadoop and Machine Learning
Hadoop and Machine Learningjoshwills
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)Prof. Dr. Diego Kuonen
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data ScientistDaniel Tunkelang
 
Data By The People, For The People
Data By The People, For The PeopleData By The People, For The People
Data By The People, For The PeopleDaniel Tunkelang
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning SystemsXavier Amatriain
 
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Data Science London
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientistryanorban
 
A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013Philip Zheng
 
Deep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDeep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDevashish Shanker
 
Introduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningIntroduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningVarad Meru
 
An Introduction to Supervised Machine Learning and Pattern Classification: Th...
An Introduction to Supervised Machine Learning and Pattern Classification: Th...An Introduction to Supervised Machine Learning and Pattern Classification: Th...
An Introduction to Supervised Machine Learning and Pattern Classification: Th...Sebastian Raschka
 
Machine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesMachine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesPier Luca Lanzi
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsDavid Pittman
 
Tutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsTutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsNhatHai Phan
 
Tips for data science competitions
Tips for data science competitionsTips for data science competitions
Tips for data science competitionsOwen Zhang
 

Viewers also liked (20)

Deep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsDeep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applications
 
Deep Learning through Examples
Deep Learning through ExamplesDeep Learning through Examples
Deep Learning through Examples
 
Bắt đầu học data science
Bắt đầu học data scienceBắt đầu học data science
Bắt đầu học data science
 
Bắt đầu nghiên cứu Big Data
Bắt đầu nghiên cứu Big DataBắt đầu nghiên cứu Big Data
Bắt đầu nghiên cứu Big Data
 
Hands-on Deep Learning in Python
Hands-on Deep Learning in PythonHands-on Deep Learning in Python
Hands-on Deep Learning in Python
 
Hadoop and Machine Learning
Hadoop and Machine LearningHadoop and Machine Learning
Hadoop and Machine Learning
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
 
Data By The People, For The People
Data By The People, For The PeopleData By The People, For The People
Data By The People, For The People
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
 
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientist
 
A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013
 
Deep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDeep Learning for Natural Language Processing
Deep Learning for Natural Language Processing
 
Introduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningIntroduction to Mahout and Machine Learning
Introduction to Mahout and Machine Learning
 
An Introduction to Supervised Machine Learning and Pattern Classification: Th...
An Introduction to Supervised Machine Learning and Pattern Classification: Th...An Introduction to Supervised Machine Learning and Pattern Classification: Th...
An Introduction to Supervised Machine Learning and Pattern Classification: Th...
 
Machine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesMachine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification Rules
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data Scientists
 
Tutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsTutorial on Deep learning and Applications
Tutorial on Deep learning and Applications
 
Tips for data science competitions
Tips for data science competitionsTips for data science competitions
Tips for data science competitions
 

Similar to Deep neural networks

introduction to deeplearning
introduction to deeplearningintroduction to deeplearning
introduction to deeplearningEyad Alshami
 
AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...
AI&BigData Lab. Артем Чернодуб  "Распознавание изображений методом Lazy Deep ...AI&BigData Lab. Артем Чернодуб  "Распознавание изображений методом Lazy Deep ...
AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...GeeksLab Odessa
 
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...GeeksLab Odessa
 
Fundamental of deep learning
Fundamental of deep learningFundamental of deep learning
Fundamental of deep learningStanley Wang
 
What's Wrong With Deep Learning?
What's Wrong With Deep Learning?What's Wrong With Deep Learning?
What's Wrong With Deep Learning?Philip Zheng
 
Evolution of Deep Learning and new advancements
Evolution of Deep Learning and new advancementsEvolution of Deep Learning and new advancements
Evolution of Deep Learning and new advancementsChitta Ranjan
 
Deep learning from a novice perspective
Deep learning from a novice perspectiveDeep learning from a novice perspective
Deep learning from a novice perspectiveAnirban Santara
 
An Introduction to Deep Learning
An Introduction to Deep LearningAn Introduction to Deep Learning
An Introduction to Deep LearningPoo Kuan Hoong
 
Big Data Malaysia - A Primer on Deep Learning
Big Data Malaysia - A Primer on Deep LearningBig Data Malaysia - A Primer on Deep Learning
Big Data Malaysia - A Primer on Deep LearningPoo Kuan Hoong
 
MDEC Data Matters Series: machine learning and Deep Learning, A Primer
MDEC Data Matters Series: machine learning and Deep Learning, A PrimerMDEC Data Matters Series: machine learning and Deep Learning, A Primer
MDEC Data Matters Series: machine learning and Deep Learning, A PrimerPoo Kuan Hoong
 
Details of Lazy Deep Learning for Images Recognition in ZZ Photo app
Details of Lazy Deep Learning for Images Recognition in ZZ Photo appDetails of Lazy Deep Learning for Images Recognition in ZZ Photo app
Details of Lazy Deep Learning for Images Recognition in ZZ Photo appPAY2 YOU
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classificationCenk Bircanoğlu
 
DEF CON 24 - Clarence Chio - machine duping 101
DEF CON 24 - Clarence Chio - machine duping 101DEF CON 24 - Clarence Chio - machine duping 101
DEF CON 24 - Clarence Chio - machine duping 101Felipe Prado
 
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중datasciencekorea
 
2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learning2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learningVan Thanh
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 
Convolutional neural networks 이론과 응용
Convolutional neural networks 이론과 응용Convolutional neural networks 이론과 응용
Convolutional neural networks 이론과 응용홍배 김
 

Similar to Deep neural networks (20)

introduction to deeplearning
introduction to deeplearningintroduction to deeplearning
introduction to deeplearning
 
AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...
AI&BigData Lab. Артем Чернодуб  "Распознавание изображений методом Lazy Deep ...AI&BigData Lab. Артем Чернодуб  "Распознавание изображений методом Lazy Deep ...
AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...
 
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...
AI&BigData Lab 2016. Артем Чернодуб: Обучение глубоких, очень глубоких и реку...
 
Fundamental of deep learning
Fundamental of deep learningFundamental of deep learning
Fundamental of deep learning
 
What's Wrong With Deep Learning?
What's Wrong With Deep Learning?What's Wrong With Deep Learning?
What's Wrong With Deep Learning?
 
Evolution of Deep Learning and new advancements
Evolution of Deep Learning and new advancementsEvolution of Deep Learning and new advancements
Evolution of Deep Learning and new advancements
 
AINL 2016: Filchenkov
AINL 2016: FilchenkovAINL 2016: Filchenkov
AINL 2016: Filchenkov
 
Deep learning from a novice perspective
Deep learning from a novice perspectiveDeep learning from a novice perspective
Deep learning from a novice perspective
 
Deep learning 1
Deep learning 1Deep learning 1
Deep learning 1
 
An Introduction to Deep Learning
An Introduction to Deep LearningAn Introduction to Deep Learning
An Introduction to Deep Learning
 
Big Data Malaysia - A Primer on Deep Learning
Big Data Malaysia - A Primer on Deep LearningBig Data Malaysia - A Primer on Deep Learning
Big Data Malaysia - A Primer on Deep Learning
 
MDEC Data Matters Series: machine learning and Deep Learning, A Primer
MDEC Data Matters Series: machine learning and Deep Learning, A PrimerMDEC Data Matters Series: machine learning and Deep Learning, A Primer
MDEC Data Matters Series: machine learning and Deep Learning, A Primer
 
Details of Lazy Deep Learning for Images Recognition in ZZ Photo app
Details of Lazy Deep Learning for Images Recognition in ZZ Photo appDetails of Lazy Deep Learning for Images Recognition in ZZ Photo app
Details of Lazy Deep Learning for Images Recognition in ZZ Photo app
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classification
 
DEF CON 24 - Clarence Chio - machine duping 101
DEF CON 24 - Clarence Chio - machine duping 101DEF CON 24 - Clarence Chio - machine duping 101
DEF CON 24 - Clarence Chio - machine duping 101
 
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중
Deep Learning - 인공지능 기계학습의 새로운 트랜드 :김인중
 
2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learning2010 deep learning and unsupervised feature learning
2010 deep learning and unsupervised feature learning
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
 
Convolutional neural networks 이론과 응용
Convolutional neural networks 이론과 응용Convolutional neural networks 이론과 응용
Convolutional neural networks 이론과 응용
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learning
 

Recently uploaded

Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdf
Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdfDr.M.Florence Dayana-Cloud Computing-unit - 4.pdf
Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdfDr.Florence Dayana
 
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...EduSkills OECD
 
Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsOrganic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsDr.M.Geethavani
 
Google Ad Grants Services at TechSoup.pdf
Google Ad Grants Services at TechSoup.pdfGoogle Ad Grants Services at TechSoup.pdf
Google Ad Grants Services at TechSoup.pdfTechSoup
 
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTS
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTSA TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTS
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTSDr.M.Geethavani
 
Unit 5th topic Drugs used in congestive Heart failure and shock.pdf
Unit 5th topic Drugs used in congestive Heart failure and shock.pdfUnit 5th topic Drugs used in congestive Heart failure and shock.pdf
Unit 5th topic Drugs used in congestive Heart failure and shock.pdfSUMIT TIWARI
 
Data Modeling - Enhanced ER diagrams & Mapping.pdf
Data Modeling - Enhanced ER diagrams & Mapping.pdfData Modeling - Enhanced ER diagrams & Mapping.pdf
Data Modeling - Enhanced ER diagrams & Mapping.pdfChristalin Nelson
 
Discussing the new Competence Framework for project managers in the built env...
Discussing the new Competence Framework for project managers in the built env...Discussing the new Competence Framework for project managers in the built env...
Discussing the new Competence Framework for project managers in the built env...Association for Project Management
 
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Aggregage
 
Learner Digital Skills Toolkit DRAFT.docx
Learner Digital Skills Toolkit DRAFT.docxLearner Digital Skills Toolkit DRAFT.docx
Learner Digital Skills Toolkit DRAFT.docxGeorgeMilliken2
 
2024-02-24_Session 1 - PMLE_UPDATED.pptx
2024-02-24_Session 1 - PMLE_UPDATED.pptx2024-02-24_Session 1 - PMLE_UPDATED.pptx
2024-02-24_Session 1 - PMLE_UPDATED.pptxgdgsurrey
 
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...SUMIT TIWARI
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Sue Beckingham
 
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdf
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdfSSC Hindu Religion and Moral Education MCQ Solutions 2024.pdf
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdfMohonDas
 
Relational_Algebra_Calculus Operations.pdf
Relational_Algebra_Calculus Operations.pdfRelational_Algebra_Calculus Operations.pdf
Relational_Algebra_Calculus Operations.pdfChristalin Nelson
 
Mycobacteriology update 2024 Margie Morgan.ppt
Mycobacteriology update 2024 Margie Morgan.pptMycobacteriology update 2024 Margie Morgan.ppt
Mycobacteriology update 2024 Margie Morgan.pptMargie Morgan
 
How To Create Record Rules in the Odoo 17
How To Create Record Rules in the Odoo 17How To Create Record Rules in the Odoo 17
How To Create Record Rules in the Odoo 17Celine George
 
The basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptxThe basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptxdeputymitchell2
 
Kartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentKartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentInMediaRes1
 

Recently uploaded (20)

Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdf
Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdfDr.M.Florence Dayana-Cloud Computing-unit - 4.pdf
Dr.M.Florence Dayana-Cloud Computing-unit - 4.pdf
 
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...
Managing Choice, Coherence and Specialisation in Upper Secondary Education - ...
 
Organic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional GroupsOrganic Synthesis and Estimation of Functional Groups
Organic Synthesis and Estimation of Functional Groups
 
Google Ad Grants Services at TechSoup.pdf
Google Ad Grants Services at TechSoup.pdfGoogle Ad Grants Services at TechSoup.pdf
Google Ad Grants Services at TechSoup.pdf
 
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTS
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTSA TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTS
A TEXTBOOK OF INTELLECTUAL ROPERTY RIGHTS
 
Unit 5th topic Drugs used in congestive Heart failure and shock.pdf
Unit 5th topic Drugs used in congestive Heart failure and shock.pdfUnit 5th topic Drugs used in congestive Heart failure and shock.pdf
Unit 5th topic Drugs used in congestive Heart failure and shock.pdf
 
Data Modeling - Enhanced ER diagrams & Mapping.pdf
Data Modeling - Enhanced ER diagrams & Mapping.pdfData Modeling - Enhanced ER diagrams & Mapping.pdf
Data Modeling - Enhanced ER diagrams & Mapping.pdf
 
Discussing the new Competence Framework for project managers in the built env...
Discussing the new Competence Framework for project managers in the built env...Discussing the new Competence Framework for project managers in the built env...
Discussing the new Competence Framework for project managers in the built env...
 
Advance Mobile Application Development class 04
Advance Mobile Application Development class 04Advance Mobile Application Development class 04
Advance Mobile Application Development class 04
 
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
 
Learner Digital Skills Toolkit DRAFT.docx
Learner Digital Skills Toolkit DRAFT.docxLearner Digital Skills Toolkit DRAFT.docx
Learner Digital Skills Toolkit DRAFT.docx
 
2024-02-24_Session 1 - PMLE_UPDATED.pptx
2024-02-24_Session 1 - PMLE_UPDATED.pptx2024-02-24_Session 1 - PMLE_UPDATED.pptx
2024-02-24_Session 1 - PMLE_UPDATED.pptx
 
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...
Bilingual notes of Pharmacognosy chapter 4Glycosides, Volatile oils,Tannins,R...
 
Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...Digital Footprints to Career Pathways - Building a Strong Professional Online...
Digital Footprints to Career Pathways - Building a Strong Professional Online...
 
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdf
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdfSSC Hindu Religion and Moral Education MCQ Solutions 2024.pdf
SSC Hindu Religion and Moral Education MCQ Solutions 2024.pdf
 
Relational_Algebra_Calculus Operations.pdf
Relational_Algebra_Calculus Operations.pdfRelational_Algebra_Calculus Operations.pdf
Relational_Algebra_Calculus Operations.pdf
 
Mycobacteriology update 2024 Margie Morgan.ppt
Mycobacteriology update 2024 Margie Morgan.pptMycobacteriology update 2024 Margie Morgan.ppt
Mycobacteriology update 2024 Margie Morgan.ppt
 
How To Create Record Rules in the Odoo 17
How To Create Record Rules in the Odoo 17How To Create Record Rules in the Odoo 17
How To Create Record Rules in the Odoo 17
 
The basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptxThe basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptx
 
Kartik Nair In Media Res Media Component
Kartik Nair In Media Res Media ComponentKartik Nair In Media Res Media Component
Kartik Nair In Media Res Media Component
 

Deep neural networks

  • 1. Deep Learning and Application in Neural Networks Hugo Larochelle Geoffrey Hinton Yoshua Bengio Andrew Ng. Jerome Louradour Andrew L. Nelson Pascal Lamblin R. Salskhutdinov
  • 2. Biological Neurons Terminal Branches Dendrites of Axon Axon
  • 3. Artificial Neural Networks (ANN) Terminal Branches Dendrites of Axon x1 w1 x2 w2 x3 w3 Axon wn xn Slide credit : Andrew L. Nelson
  • 4. Layered Networks Input nodes Output nodes Hidden nodes Connections Output : yi f ( wi1x1 wi2x 2 wi3x 3  wim x m ) f( wij x j ) j
  • 5. Neural network application • ALVINN drives 70 mph on highways Slide credit : T.Michell
  • 6. Neural network application Slide credit : Andraw Ng.
  • 7. The simplest model- the Perceptron •The Perceptron was introduced in 1957 by Frank Rosenblatt. Perceptron: - D0 d D1 Activation D2 functions: Learning: Input Output Layer Destinations Layer Update Slide credit : Geoffrey Hinton
  • 8. Second generation neural networks (~1985) Compare outputs with Back-propagate correct answer to get error signal to get error signal derivatives for learning outputs hidden layers input vector Slide credit : Geoffrey Hinton
  • 9. BP-algorithm 1 .5 0 -5 0 5 Activations errors The error: Update Update Weights: Slide credit : Geoffrey Hinton
  • 10. Back Propagation Advantages • Multi layer Perceptron network can be trained by the back propagation algorithm to perform any mapping between the input and the output. What is wrong with back-propagation? •It requires labeled training data. Almost all data is unlabeled. •The learning time does not scale well It is very slow in networks with multiple hidden layers. •It can get stuck in poor local optima. Slide credit : Geoffrey Hinton
  • 11. Why Deep multi-layered neural network object models object parts (combination of edges) edges pixels
  • 13. Deep Neural Networks • Standard learning strategy – Randomly initializing the weights of the network – Applying gradient descent using backpropagation • But, backpropagation does not work well (if randomly initialized) – Deep networks trained with back-propagation (without unsupervised pre-train) perform worse than shallow networks – ANN have limited to one or two layers
  • 17. Unsupervised greedy layer-wise training procedure.
  • 27. Layer-Local Unsupervised Learning • Restricted Boltzmann Machine (SRBM) (Hinton et al, NC’2006) • Auto-encoders (Bengio et al, NIPS’2006) • Sparse auto-encoders (Ranzato et al, NIPS’2006) • Kernel PCA (Erhan 2008) • Denoising auto-encoders (Vincent et al, ICML’2008) • Unsupervised embedding (Weston et al, ICML’2008) • Slow features (Mohabi et al, ICML’2009, Bergstra & Bengio NIPS’2009)
  • 28. Experiments • MNIST data set – A benchmark for handwritten digit recognition – The number of classes is 10 (corresponding to the digits from 0 to 9) – The inputs were scaled between 0 and 1 Exploring Strategies for Training Deep Neural Networks. Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, Pascal Lamblin; 10(Jan):1--40, 2009.
  • 29. Classification error on MNIST training
  • 30. Experiments • Histograms presenting the test errors obtained on MNIST using models trained with or without pre-training. The difficulty of training deep architectures and the effect of unsupervised pre-training. Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent; pages 153-160, 2009.
  • 31. Experiments • NORB data set – 5 object categories, 5 difference objects within each category. • Classification error rate – DBM : 10.8 % – SVM : 11.6 % – Logistic Regression : 22.5 % – KNN : 18.4 % Efficient Learning of Deep Boltzmann Machines. Ruslan Salakhutdinov, Hugo Larochelle ; JMLR W&CP 9:693-700, 2010.
  • 32. Conclusion • Deep Learning : powerful arguments & generalization priciples • Unsupervised Feature Learning is crucial many new algorithms and applications in recent years • Deep Learning suited for multi-task learning, domain adaptation and semi-learning with few labels
  • 33. Application • Classification (Bengio et al., 2007; Ranzato et al., 2007b; Larochelle et al., 2007; Ranzato et al.,2008) • Regression (Salakhutdinov and Hinton, 2008) • Dimensionality Reduction (Hinton Salakhutdinov, 2006; Salakhutdinov and Hinton, 2007) • Modeling Textures (Osindero and Hinton, 2008) • Information Retrieval (Salakhutdinov and Hinton, 2007) • Robotics (Hadsell et al., 2008) • Natural Language Processing (Collobert and Weston, 2008; Weston et al., 2008) • Collaborative Filtering (Salakhutdinov et al., 2007)
  • 34. Recent Deep Learning Highlights
  • 35. Recent Deep Learning Highlights • Google Goggles uses Stacked Sparse Auto Encoders (Hartmut Neven @ ICML 2011) • The monograph or review paper Learning Deep Architectures for AI (Foundations & Trends in Machine Learning, 2009). • Exploring Strategies for Training Deep Neural Networks, Hugo Larochelle, Yoshua Bengio, Jerome Louradour and Pascal Lamblin in: The Journal of Machine Learning Research, pages 1-40, 2009. • The LISA publications database contains a deep architectures category. http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/ ReadingOnDeepNetworks • Deep Machine Learning – A New Frontier in Artificial Intelligence Research – a survey paper by Itamar Arel, Derek C. Rose, and Thomas P. Karnowski.