Uploaded byBineeshJose99
PDF, PPTX578 views

Classification By Back Propagation

This document discusses classification using backpropagation in deep neural networks. It provides an overview of key concepts like perceptrons, multi-layer perceptrons, training MLPs, neural networks as classifiers, backpropagation, backpropagation with 3 hidden layers, gradient descent backpropagation, and challenges in deep neural network training. The document is authored by Bineesh Jose, a research scholar at the School of Computer Science, M G University in Kottayam.

Embed presentation

Download as PDF, PPTX
Classification by Backpropagation DEEP NEURAL NETWORK(DNN) March 29, 2019 Bineesh Jose Research Scholar School of Computer Science M G University Kottayam
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Contents Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers
17 Classification by Backpropagation Bineesh Jose 2 Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Perceptron
17 Classification by Backpropagation Bineesh Jose Perceptron 3 Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Multi-Layer Perceptron When MLP contains two or more hidden layers then such an MLP is called Deep Neural Network DNN
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron 4 MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam MLP Cntd.. Multi-Layer Perceptron • The inputs to the network correspond to the attributes measured for each training tuple • Inputs are fed simultaneously into the units making up the input layer • They are then weighted and fed simultaneously to a hidden layer • The number of hidden layers is arbitrary, although usually only one • The weighted outputs of the last hidden layer are input to units making up the output layer, which emits the network's prediction • The network is feed-forward in that none of the weights cycles back to an input unit or to an output unit of a previous layer
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. 5 Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Training MLP • Training MLP was a big challenge, because it was needed to handle derivatives at multiple layers. • In MLP, the activation function was a step function. z = w0x0 + w1x1 + . . . + wnxn = wT x (1) • Gradients cannot move on a flat surface. • Backpropagation algorithm was developed to solve this problem. • The step function is replaced with a logistic function:
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP 6 Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Neural Network as a Classifier Strength • High tolerance to noisy data and Ability to classify untrained patterns • Well-suited for continuous-valued inputs and outputs • Successful on a wide array of real-world data • Techniques have recently been developed for the extraction of rules from trained neural networks Weakness • Long training time. • Require a number of parameters typically best determined empirically, e.g., the network topology or structure.. • Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier 7 Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation • Backpropagation: A neural network learning algorithm. • Started by psychologists and neurobiologists to develop and test computational analogues of neurons. • During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class labelof the input tuples • Also referred to as connectionist learning due to the connections between units
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier 8 Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Cntd..
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation 9 Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation with 3 hidden layers
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers 10 Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Gradient Descent Backpropagation This is described as Gradient Descent using reverse-mode autodiff. • Error gradient along all connection weights were measured by propagating the error from output layer.. • First, a forward pass is performed - output of every neuron in every layer is computed.. • Output error is estimated • Then compute how much each neuron in last hidden layer contributed to output error. • This is repeated backwards until input layer.. • Last step is Gradient Descent on all connection weights using error gradients estimated in previous steps.
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation 11 Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Gradient Descent Mean Squared Error (MSE) cost function
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent 12 Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Example
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example 13 Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Algorithm
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm 14 Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Efficiency and Interpretability
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability 15 Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Challenges in DNN training Vanishing Gradients • The gradients often gets smaller and smaller when progressing to lower layers. • This happens in Directed Acyclic Graph models. • So, no weights get updated in lower layers (no convergence to good solution) Exploding Gradients • The gradients often gets bigger and bigger during backprop. • This happens in Directed Cyclic Graph models. • Weights get large updates.
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training 16 HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Handwritten Digit Recognition Using MNIST dataset
17 Classification by Backpropagation Bineesh Jose Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation 17 Backpropagation with 4 hidden layers SOCS M.G University Kottayam HDR Using Backpropagation with 4 hidden layers
Thank you!

More Related Content

PPT
Reinforcement Learning Q-Learning
byMelaku Eneayehu
 
PPTX
Ensemble Method (Bagging Boosting)
byAbdullah al Mamun
 
PPTX
Decision tree induction \ Decision Tree Algorithm with Example| Data science
byMaryamRehman6
 
PPT
2.5 backpropagation
byKrish_ver2
 
PPT
Classification using back propagation algorithm
byKIRAN R
 
PPTX
Bagging.pptx
byComsatsSahiwal1
 
PPTX
Support vector machine
byMusa Hawamdah
 
PDF
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 
Reinforcement Learning Q-Learning
byMelaku Eneayehu
 
Ensemble Method (Bagging Boosting)
byAbdullah al Mamun
 
Decision tree induction \ Decision Tree Algorithm with Example| Data science
byMaryamRehman6
 
2.5 backpropagation
byKrish_ver2
 
Classification using back propagation algorithm
byKIRAN R
 
Bagging.pptx
byComsatsSahiwal1
 
Support vector machine
byMusa Hawamdah
 
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 

What's hot

PPTX
Ensemble methods in machine learning
bySANTHOSH RAJA M G
 
PPTX
Association Rule Learning Part 1: Frequent Itemset Generation
byKnoldus Inc.
 
PDF
Introduction to Recurrent Neural Network
byKnoldus Inc.
 
PPTX
15 puzzle problem using branch and bound
byAbhishek Singh
 
PPTX
Machine Learning - Ensemble Methods
byAndrew Ferlitsch
 
PPTX
Deep learning
byRajgupta258
 
PPTX
Image classification using CNN
byNoura Hussein
 
PPTX
Naive bayes
byAshraf Uddin
 
PPT
Association rule mining
byAcad
 
DOCX
S/MIME
bymaria azam
 
PDF
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
byMohammed Bennamoun
 
PDF
Design principle of pattern recognition system and STATISTICAL PATTERN RECOGN...
byTEJVEER SINGH
 
PPTX
Regularization in deep learning
byKien Le
 
PPT
Np cooks theorem
byNarayana Galla
 
PPT
Svm
bywltongxing
 
PPTX
Image classification using cnn
bySumeraHangi
 
PDF
GoogLeNet Insights
byAuro Tripathy
 
PPTX
Convolution Neural Network (CNN)
bySuraj Aavula
 
PPT
2.3 bayesian classification
byKrish_ver2
 
PPTX
Introdution and designing a learning system
byswapnac12
 
Ensemble methods in machine learning
bySANTHOSH RAJA M G
 
Association Rule Learning Part 1: Frequent Itemset Generation
byKnoldus Inc.
 
Introduction to Recurrent Neural Network
byKnoldus Inc.
 
15 puzzle problem using branch and bound
byAbhishek Singh
 
Machine Learning - Ensemble Methods
byAndrew Ferlitsch
 
Deep learning
byRajgupta258
 
Image classification using CNN
byNoura Hussein
 
Naive bayes
byAshraf Uddin
 
Association rule mining
byAcad
 
S/MIME
bymaria azam
 
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
byMohammed Bennamoun
 
Design principle of pattern recognition system and STATISTICAL PATTERN RECOGN...
byTEJVEER SINGH
 
Regularization in deep learning
byKien Le
 
Np cooks theorem
byNarayana Galla
 
Svm
bywltongxing
 
Image classification using cnn
bySumeraHangi
 
GoogLeNet Insights
byAuro Tripathy
 
Convolution Neural Network (CNN)
bySuraj Aavula
 
2.3 bayesian classification
byKrish_ver2
 
Introdution and designing a learning system
byswapnac12
 

Similar to Classification By Back Propagation

PPTX
Machine Learning DR PRKRao-PPT UNIT-II.pptx
bycsithodsdes
 
PDF
Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation
byMohammed Bennamoun
 
PPT
Chapter 9. Classification Advanced Methods.ppt
bySubrata Kumer Paul
 
PPTX
back propagation1_presenation_lab 6.pptx
bysomeyamohsen2
 
PPTX
Artificial Neural Networks_Unit_3_3.pptx
byanil kumar
 
PDF
Classification by back propagation, multi layered feed forward neural network...
bybihira aggrey
 
PPT
Intro to Deep learning - Autoencoders
byAkash Goel
 
PDF
09 classadvanced
byJoonyoungJayGwak
 
PPTX
Classification_by_back_&propagation.pptx
bySadiaSaleem301
 
PDF
22PCOAM16 ML UNIT 2 NOTES & QB QUESTION WITH ANSWERS
byGuru Nanak Technical Institutions
 
PDF
22PCOAM16 _ML_ Unit 2 Full unit notes.pdf
byGuru Nanak Technical Institutions
 
PPTX
Introduction to Perceptron and Neural Network.pptx
byPoonam60376
 
PPTX
Deeplearning for Computer Vision PPT with
bynaveenraghavendran10
 
PDF
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
byManish Saraswat
 
PPTX
Neural net and back propagation
byMohit Shrivastava
 
PPTX
Unit one ppt of deeep learning which includes Ann cnn
bykartikaursang53
 
PPT
Data Mining: Concepts and techniques classification _chapter 9 :advanced methods
bySalah Amean
 
PPT
Chapter 09 classification advanced
byHouw Liong The
 
PPT
Chapter 09 class advanced
byHouw Liong The
 
PPT
Addvanced Classification Algorithmns in DWDM
byMrSKanthiKiran
 
Machine Learning DR PRKRao-PPT UNIT-II.pptx
bycsithodsdes
 
Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation
byMohammed Bennamoun
 
Chapter 9. Classification Advanced Methods.ppt
bySubrata Kumer Paul
 
back propagation1_presenation_lab 6.pptx
bysomeyamohsen2
 
Artificial Neural Networks_Unit_3_3.pptx
byanil kumar
 
Classification by back propagation, multi layered feed forward neural network...
bybihira aggrey
 
Intro to Deep learning - Autoencoders
byAkash Goel
 
09 classadvanced
byJoonyoungJayGwak
 
Classification_by_back_&propagation.pptx
bySadiaSaleem301
 
22PCOAM16 ML UNIT 2 NOTES & QB QUESTION WITH ANSWERS
byGuru Nanak Technical Institutions
 
22PCOAM16 _ML_ Unit 2 Full unit notes.pdf
byGuru Nanak Technical Institutions
 
Introduction to Perceptron and Neural Network.pptx
byPoonam60376
 
Deeplearning for Computer Vision PPT with
bynaveenraghavendran10
 
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
byManish Saraswat
 
Neural net and back propagation
byMohit Shrivastava
 
Unit one ppt of deeep learning which includes Ann cnn
bykartikaursang53
 
Data Mining: Concepts and techniques classification _chapter 9 :advanced methods
bySalah Amean
 
Chapter 09 classification advanced
byHouw Liong The
 
Chapter 09 class advanced
byHouw Liong The
 
Addvanced Classification Algorithmns in DWDM
byMrSKanthiKiran
 

Recently uploaded

PPTX
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Track Phone Location via Number (2026)_ What Actually Works, What’s a Scam, a...
byEmily Woods
 
PDF
Title Installing Windows, Linux, MacOS.pdf
byGeraldAbadajos
 
PDF
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
PPTX
TechSprint (SJBIT) 2025-26 Hackathon Winners & Awards Ceremony
bysuhasspgdg
 
PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
OpenCharacter AI Reviews: Features, Plans, and Best Alternative
byOpenCharacter AI
 
PPTX
How Does an ICO Launchpad Work Step-by-Step Breakdown.pptx
byTom Hardy S
 
PDF
Startup Formation Collapses While Acquisition Activity Hits New High!
byMemoori
 
PDF
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
PDF
AI and Zero Trust: What it takes to do it right
byMark Simos
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
PDF
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
PDF
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
PDF
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
PPTX
WEEK 1-introduction of industrial arts grade 7.pptx
byrosierufinomaliongan
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Track Phone Location via Number (2026)_ What Actually Works, What’s a Scam, a...
byEmily Woods
 
Title Installing Windows, Linux, MacOS.pdf
byGeraldAbadajos
 
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
TechSprint (SJBIT) 2025-26 Hackathon Winners & Awards Ceremony
bysuhasspgdg
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
OpenCharacter AI Reviews: Features, Plans, and Best Alternative
byOpenCharacter AI
 
How Does an ICO Launchpad Work Step-by-Step Breakdown.pptx
byTom Hardy S
 
Startup Formation Collapses While Acquisition Activity Hits New High!
byMemoori
 
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
AI and Zero Trust: What it takes to do it right
byMark Simos
 
UiPath Automation Developer Associate Training Series 2026 - Session 1
byDianaGray10
 
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
WEEK 1-introduction of industrial arts grade 7.pptx
byrosierufinomaliongan
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
UiPath Automation Developer Associate Training Series 2026 - Session 2
byDianaGray10
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 

Classification By Back Propagation

  • 1.
    Classification by Backpropagation DEEPNEURAL NETWORK(DNN) March 29, 2019 Bineesh Jose Research Scholar School of Computer Science M G University Kottayam
  • 2.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Contents Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers
  • 3.
    17 Classification by Backpropagation Bineesh Jose 2Perceptron Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Perceptron
  • 4.
    17 Classification by Backpropagation Bineesh Jose Perceptron 3Multi-Layer Perceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Multi-Layer Perceptron When MLP contains two or more hidden layers then such an MLP is called Deep Neural Network DNN
  • 5.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron 4 MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam MLP Cntd.. Multi-Layer Perceptron • The inputs to the network correspond to the attributes measured for each training tuple • Inputs are fed simultaneously into the units making up the input layer • They are then weighted and fed simultaneously to a hidden layer • The number of hidden layers is arbitrary, although usually only one • The weighted outputs of the last hidden layer are input to units making up the output layer, which emits the network's prediction • The network is feed-forward in that none of the weights cycles back to an input unit or to an output unit of a previous layer
  • 6.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. 5 Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam DEEPNET Training MLP • Training MLP was a big challenge, because it was needed to handle derivatives at multiple layers. • In MLP, the activation function was a step function. z = w0x0 + w1x1 + . . . + wnxn = wT x (1) • Gradients cannot move on a flat surface. • Backpropagation algorithm was developed to solve this problem. • The step function is replaced with a logistic function:
  • 7.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP 6 Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Neural Network as a Classifier Strength • High tolerance to noisy data and Ability to classify untrained patterns • Well-suited for continuous-valued inputs and outputs • Successful on a wide array of real-world data • Techniques have recently been developed for the extraction of rules from trained neural networks Weakness • Long training time. • Require a number of parameters typically best determined empirically, e.g., the network topology or structure.. • Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights
  • 8.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier 7 Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation • Backpropagation: A neural network learning algorithm. • Started by psychologists and neurobiologists to develop and test computational analogues of neurons. • During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class labelof the input tuples • Also referred to as connectionist learning due to the connections between units
  • 9.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier 8 Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Cntd..
  • 10.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation 9 Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation with 3 hidden layers
  • 11.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers 10 Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Gradient Descent Backpropagation This is described as Gradient Descent using reverse-mode autodiff. • Error gradient along all connection weights were measured by propagating the error from output layer.. • First, a forward pass is performed - output of every neuron in every layer is computed.. • Output error is estimated • Then compute how much each neuron in last hidden layer contributed to output error. • This is repeated backwards until input layer.. • Last step is Gradient Descent on all connection weights using error gradients estimated in previous steps.
  • 12.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation 11 Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Gradient Descent Mean Squared Error (MSE) cost function
  • 13.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent 12 Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Example
  • 14.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example 13 Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Backpropagation Algorithm
  • 15.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm 14 Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Efficiency and Interpretability
  • 16.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability 15 Challenges in DNN training HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Challenges in DNN training Vanishing Gradients • The gradients often gets smaller and smaller when progressing to lower layers. • This happens in Directed Acyclic Graph models. • So, no weights get updated in lower layers (no convergence to good solution) Exploding Gradients • The gradients often gets bigger and bigger during backprop. • This happens in Directed Cyclic Graph models. • Weights get large updates.
  • 17.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training 16 HDR Using Backpropagation Backpropagation with 4 hidden layers SOCS M.G University Kottayam Handwritten Digit Recognition Using MNIST dataset
  • 18.
    17 Classification by Backpropagation Bineesh Jose Perceptron Multi-LayerPerceptron MLP Cntd.. Training MLP Neural Network as a Classifier Backpropagation Backpropagation with 3 hidden layers Gradient Descent Backpropagation Gradient Descent Backpropagation Example Backpropagation Algorithm Efficiency and Interpretability Challenges in DNN training HDR Using Backpropagation 17 Backpropagation with 4 hidden layers SOCS M.G University Kottayam HDR Using Backpropagation with 4 hidden layers
  • 19.