Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Backpropagation: Understanding How to
Update ANNs Weights Step-by-Step
Ahmed Fawzy Gad
ahmed.fawzy@ci.menofia.edu.eg
MENOU...
Train then Update
โ€ข The backpropagation algorithm is used to update the NN weights
when they are not able to make the corr...
Train then Update
โ€ข The backpropagation algorithm is used to update the NN weights
when they are not able to make the corr...
Neural Network Training Example
๐— ๐Ÿ ๐— ๐Ÿ ๐Ž๐ฎ๐ญ๐ฉ๐ฎ๐ญ
๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ‘ ๐ŸŽ. ๐ŸŽ๐Ÿ‘
๐–๐Ÿ ๐–๐Ÿ ๐›
๐ŸŽ. ๐Ÿ“ ๐ŸŽ. ๐Ÿ“ 1. ๐Ÿ–๐Ÿ‘
Training Data Initial Weights
๐ŸŽ. ๐Ÿ
...
Network Training
โ€ข Steps to train our network:
1. Prepare activation function input
(sum of products between inputs
and we...
Network Training: Sum of Products
โ€ข After calculating the sop between inputs
and weights, next is to use this sop as the
i...
Network Training: Activation Function
โ€ข In this example, the sigmoid activation
function is used.
โ€ข Based on the sop calcu...
Network Training: Prediction Error
โ€ข After getting the predicted outputs,
next is to measure the prediction error
of the n...
How to Minimize Prediction Error?
โ€ข There is a prediction error and it should be minimized until reaching
an acceptable er...
Weights Update Equation
โ€ข We can use the weights update equation:
๏‚ง ๐‘พ ๐’๐’†๐’˜: new updated weights.
๏‚ง ๐‘พ ๐’๐’๐’…: current weights. ...
Weights Update Equation
๐‘พ ๐’๐’†๐’˜ = ๐‘พ ๐’๐’๐’… + ฮท ๐’… โˆ’ ๐’€ ๐‘ฟ
= [๐Ÿ. ๐Ÿ–๐Ÿ‘, ๐ŸŽ. ๐Ÿ“, ๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐ŸŽ๐Ÿ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’ [+๐Ÿ, ๐ŸŽ. ๐Ÿ, ๐ŸŽ. ๐Ÿ‘
= [๐Ÿ. ๐Ÿ–๐Ÿ‘, ๐ŸŽ....
Weights Update Equation
โ€ข The new weights are:
โ€ข Based on the new weights, the network will be re-trained.
๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐’ƒ...
Weights Update Equation
โ€ข The new weights are:
โ€ข Based on the new weights, the network will be re-trained.
โ€ข Continue thes...
Why Backpropagation Algorithm is Important?
โ€ข The backpropagation algorithm is used to answer these questions
and understa...
Forward Vs. Backward Passes
โ€ข When training a neural network, there are two
passes: forward and backward.
โ€ข The goal of th...
Backward Pass
โ€ข Let us work with a simpler example:
โ€ข How to answer this question: What is the effect on the output Y
give...
Calculating Derivatives
โ€ข The derivative
๐๐’€
๐๐‘ฟ
can be calculated as follows:
โ€ข Based on these two derivative rules:
โ€ข The ...
Prediction Error โ€“ Weight Derivative
E W?
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
Change in Y wrt X
๐๐’€
๐››๐‘ฟ
Change in E wrt W
๐๐‘ฌ
๐››๐‘พ
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•)
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” =
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐Ÿ
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•...
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐Ÿ
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•...
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐Ÿ
๐’” = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ...
Prediction Error โ€“ Weight Derivative
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•...
Multivariate Chain Rule
Predicted
Output
Prediction
Error
sop Weights
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’‡ ๐’™ =
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐’” = ๐‘ฟ ...
Multivariate Chain Rule
Predicted
Output
Prediction
Error
sop Weights
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’‡ ๐’™ =
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
๐’” = ๐‘ฟ ...
Error-Predicted (
๐๐‘ฌ
๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
) Partial Derivative
Substitution
๐๐‘ฌ
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
=
๐
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
(
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ...
Predicted-sop (
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
๐๐’”
) Partial Derivative
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
๐๐’”
=
๐
๐๐’”
(
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
)
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
๐๐’”
=
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’”
(๐Ÿ โˆ’
๐Ÿ
๐Ÿ +...
Sop-๐‘Š1 (
๐๐’”
๐››๐‘พ ๐Ÿ
) Partial Derivative
๐๐’”
๐››๐‘พ ๐Ÿ
=
๐››
๐››๐‘พ ๐Ÿ
(๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ)
= ๐Ÿ โˆ— ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ
๐Ÿโˆ’๐Ÿ + ๐ŸŽ + ๐ŸŽ
= ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ...
๐๐’”
๐››๐‘พ ๐Ÿ
=
๐››
๐››๐‘พ ๐Ÿ
(๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ)
= ๐ŸŽ + ๐Ÿ โˆ— ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ
๐Ÿโˆ’๐Ÿ + ๐ŸŽ
= ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ
๐ŸŽ
)= ๐‘ฟ ๐Ÿ(๐Ÿ
๐๐’”
๐››๐‘พ ๐Ÿ
= ๐‘ฟ ๐Ÿ
๐๐’”
๐››๐‘พ ๐Ÿ
= ๐‘ฟ ...
Error-๐‘Š1 (
๐››๐‘ฌ
๐››๐‘พ ๐Ÿ
) Partial Derivative
โ€ข After calculating each individual derivative, we can multiply all of
them to get...
Error-๐‘Š2 (
๐››๐‘ฌ
๐››๐‘พ ๐Ÿ
) Partial Derivative
๐๐‘ฌ
๐๐‘พ ๐Ÿ
=
๐๐‘ฌ
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
โˆ—
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
๐๐’”
โˆ—
๐๐’”
๐๐‘พ ๐Ÿ
๐๐‘ฌ
๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…
= ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’
๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†...
Interpreting Derivatives
โ€ข There are two useful pieces of information from the derivatives
calculated previously.
Increasi...
Updating Weights
โ€ข Each weight will be updated based on its derivative according to this
equation:
๐‘พ๐’Š๐’๐’†๐’˜ = ๐‘พ๐’Š๐’๐’๐’… โˆ’ ฮท โˆ—
๐››๐‘ฌ
...
Second Example
Backpropagation for NN with Hidden Layer
ANN with Hidden Layer
๐‘พ ๐Ÿ ๐‘พ ๐Ÿ ๐‘พ ๐Ÿ‘ ๐‘พ ๐Ÿ’ ๐‘พ ๐Ÿ“ ๐‘พ ๐Ÿ” ๐’ƒ ๐Ÿ ๐’ƒ ๐Ÿ ๐’ƒ ๐Ÿ‘
๐ŸŽ. ๐Ÿ“ ๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ”๐Ÿ ๐ŸŽ. ๐Ÿ โˆ’๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ‘ ๐ŸŽ. ๐Ÿ’ โˆ’๐ŸŽ. ๐Ÿ ๐Ÿ. ๐Ÿ–๐Ÿ‘
๐— ๐Ÿ ๐— ๐Ÿ ๐Ž๐ฎ๐ญ๐ฉ๐ฎ๐ญ
...
ANN with Hidden Layer
Initial
Weights PredictionTraining
ANN with Hidden Layer
Initial
Weights PredictionTraining
BackpropagationUpdate
Forward Pass โ€“ Hidden Layer Neurons
๐’‰ ๐Ÿ๐’Š๐’ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐Ÿ
= ๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ“ + ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐Ÿ’
๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ’๐Ÿ–
๐’‰...
Forward Pass โ€“ Hidden Layer Neurons
๐’‰ ๐Ÿ๐’Š๐’ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ
= ๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ”๐Ÿ + ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐Ÿ
๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐ŸŽ๐Ÿ๐Ÿ...
Forward Pass โ€“ Output Layer Neuron
๐’๐’–๐’•๐’Š๐’ = ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘
= ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– โˆ— โˆ’๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” โˆ— ๐ŸŽ. ๐Ÿ‘ + ๐Ÿ. ๐Ÿ–๐Ÿ‘
๐’๐’–๐’•๐’Š...
Forward Pass โ€“ Prediction Error
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘
๐‘ฌ =
๐Ÿ
๐Ÿ
๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’๐’–๐’• ๐’๐’–๐’•
๐Ÿ
=
๐Ÿ
๐Ÿ
๐ŸŽ. ๐ŸŽ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ”๐Ÿ“ ๐Ÿ
๐‘ฌ = ๐ŸŽ. ๐Ÿ‘๐Ÿ’๐Ÿ—
๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ...
Partial Derivatives Calculation
Eโˆ’๐‘Š5 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ“
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ“
Eโˆ’๐‘Š5 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ“
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ“
๐๐‘ฌ
๐๐’๐’–๐’• ๐’๐’–๐’•
=
๐
๐๐’๐’–๐’• ๐’๐’–๐’•
(
๐Ÿ
๐Ÿ
๐’…๐’†...
Eโˆ’๐‘Š5 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ“
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ“
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
=
๐
๐๐’๐’–๐’•๐’Š๐’
(
๐Ÿ
๐Ÿ ...
Eโˆ’๐‘Š5 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ“
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ“
๐๐’๐’–๐’•๐’Š๐’
๐๐‘พ ๐Ÿ“
=
๐
๐๐‘พ ๐Ÿ“
(๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ...
Eโˆ’๐‘Š5 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ“
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ“
๐๐’๐’–๐’•๐’Š๐’
๐๐‘พ ๐Ÿ“
= ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ–
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–...
Eโˆ’๐‘Š6 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ”
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ”
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ”
Eโˆ’๐‘Š6 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ”
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ”
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ”
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’...
Eโˆ’๐‘Š6 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ”
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ“
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ”
๐๐’๐’–๐’•๐’Š๐’
๐๐‘พ ๐Ÿ”
=
๐
๐๐‘พ ๐Ÿ”
(๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ...
Eโˆ’๐‘Š6 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ”
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ”
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐‘พ ๐Ÿ”
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’...
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
๐...
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
P...
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
P...
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
P...
Eโˆ’๐‘Š1 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
๐...
Eโˆ’๐‘Š2 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative:
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
Eโˆ’๐‘Š2 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative:
๐๐‘ฌ
๐››๐‘พ ๐Ÿ
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ
...
Eโˆ’๐‘Š2 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative:
Partial Derivative
Substitution
๐๐’‰๐Ÿ๐’Š๐’
๐๐‘พ ๐Ÿ
=
๐
๐๐‘พ ๐Ÿ
(๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐Ÿ)
= ๐ŸŽ ...
Eโˆ’๐‘Š2 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ
) Parial Derivative:
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“
๐๐’‰ ๐Ÿ๐’๐’–๐’•
๐๐’‰ ๐Ÿ๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ”
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
= ...
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐‘ฌ
๐››๐‘พ ๐Ÿ‘
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ‘
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐‘ฌ
๐››๐‘พ ๐Ÿ‘
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ‘
...
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐’๐’–๐’•๐’Š๐’
๐๐’‰๐Ÿ ๐’๐’–๐’•
=
๐
๐๐’‰๐Ÿ ๐’๐’–๐’•
(๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘)
= ๐ŸŽ + (๐’‰ ๐Ÿ๐’๐’–๐’•) ๐Ÿโˆ’๐Ÿโˆ— ๐‘พ ๐Ÿ”...
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐’‰๐Ÿ ๐’๐’–๐’•
๐๐’‰๐Ÿ๐’Š๐’
=
๐
๐๐’‰ ๐Ÿ๐’Š๐’
(
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’
)
๐๐’‰๐Ÿ ๐’๐’–๐’•
๐๐’‰๐Ÿ๐’Š๐’
= (
๐Ÿ
๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’
)(๐Ÿ โˆ’
๐Ÿ
๐Ÿ +...
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐’‰๐Ÿ๐’Š๐’
๐๐‘พ ๐Ÿ‘
=
๐
๐๐‘พ ๐Ÿ‘
(๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ)
= ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ
= (๐‘ฟ ๐Ÿ...
Eโˆ’๐‘Š3 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
) Parial Derivative:
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’
๐๐’‰๐Ÿ ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ‘
๐๐’‰ ๐Ÿ๐’๐’–๐’•
๐๐’‰ ๐Ÿ๐’Š๐’
= ๐ŸŽ....
Eโˆ’๐‘Š4 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ’
) Parial Derivative:
๐๐‘ฌ
๐››๐‘พ ๐Ÿ’
=
๐››๐‘ฌ
๐››๐’๐’–๐’• ๐’๐’–๐’•
โˆ—
๐››๐’๐’–๐’• ๐’๐’–๐’•
๐››๐’๐’–๐’•๐’Š๐’
โˆ—
๐››๐’๐’–๐’•๐’Š๐’
๐››๐’‰๐Ÿ ๐’๐’–๐’•
โˆ—
๐››๐’‰๐Ÿ ๐’๐’–๐’•
๐››๐’‰๐Ÿ๐’Š๐’
โˆ—
๐››๐’‰๐Ÿ๐’Š๐’
๐››๐‘พ ๐Ÿ’
Eโˆ’๐‘Š4 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ’
) Parial Derivative:
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’
๐๐’‰๐Ÿ ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ‘
๐๐’‰ ๐Ÿ๐’๐’–๐’•
๐๐’‰ ๐Ÿ๐’Š๐’
= ๐ŸŽ....
Eโˆ’๐‘Š4 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ’
) Parial Derivative:
๐๐’‰๐Ÿ๐’Š๐’
๐๐‘พ ๐Ÿ’
=
๐
๐๐‘พ ๐Ÿ’
(๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ)
= ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ
= ๐ŸŽ + ...
Eโˆ’๐‘Š4 (
๐๐‘ฌ
๐๐‘พ ๐Ÿ’
) Parial Derivative:
๐๐’๐’–๐’• ๐’๐’–๐’•
๐๐’๐’–๐’•๐’Š๐’
= ๐ŸŽ. ๐Ÿ๐Ÿ‘
๐๐‘ฌ
๐๐’๐’–๐’• ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’
๐๐’‰๐Ÿ ๐’๐’–๐’•
= ๐ŸŽ. ๐Ÿ‘
๐๐’‰ ๐Ÿ๐’๐’–๐’•
๐๐’‰ ๐Ÿ๐’Š๐’
= ๐ŸŽ....
All Error-Weights Partial Derivatives
๐๐‘ฌ
๐๐‘พ ๐Ÿ’
= ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ‘
๐๐‘ฌ
๐๐‘พ ๐Ÿ‘
= ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ—
๐๐‘ฌ
๐๐‘พ ๐Ÿ
= โˆ’. ๐ŸŽ๐ŸŽ๐Ÿ‘
๐๐‘ฌ
๐๐‘พ ๐Ÿ
= โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ
๐››๐‘ฌ
๐››๐‘พ ๐Ÿ”
= ๐ŸŽ. ๐ŸŽ๐Ÿ—...
Updated Weights
๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ—
๐๐‘ฌ
๐๐‘พ ๐Ÿ
= ๐ŸŽ. ๐Ÿ“ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ = ๐ŸŽ. ๐Ÿ“๐ŸŽ๐ŸŽ๐ŸŽ๐Ÿ
๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ—
๐๐‘ฌ
๐๐‘พ ๐Ÿ
= ๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ—...
Upcoming SlideShare
Loading in โ€ฆ5
×

Backpropagation: Understanding How to Update ANNs Weights Step-by-Step

This presentation explains how the backpropagation algorithm is useful in updating the artificial neural networks (ANNs) weights using two examples step by step. Readers should have a basic understanding of how ANNs work, partial derivatives, and multivariate chain rule.

This presentation won`t dive directly into the details of the algorithm but will start by training a very simple network. This is because the backpropagation algorithm is meant to be applied over a network after training. So, we should train the network before applying it to catch the benefits of backpropagation algorithm and how to use it.

  • Be the first to comment

Backpropagation: Understanding How to Update ANNs Weights Step-by-Step

  1. 1. Backpropagation: Understanding How to Update ANNs Weights Step-by-Step Ahmed Fawzy Gad ahmed.fawzy@ci.menofia.edu.eg MENOUFIA UNIVERSITY FACULTY OF COMPUTERS AND INFORMATION INFORMATION TECHNOLOGY โ€ซุงู„ู…ู†ูˆููŠุฉโ€ฌ โ€ซุฌุงู…ุนุฉโ€ฌ โ€ซูˆุงู„ู…ุนู„ูˆู…ุงุชโ€ฌ โ€ซุงู„ุญุงุณุจุงุชโ€ฌ โ€ซูƒู„ูŠุฉโ€ฌ โ€ซุงู„ู…ุนู„ูˆู…ุงุชโ€ฌ โ€ซุชูƒู†ูˆู„ูˆุฌูŠุงโ€ฌ โ€ซุงู„ู…ู†ูˆููŠุฉโ€ฌ โ€ซุฌุงู…ุนุฉโ€ฌ
  2. 2. Train then Update โ€ข The backpropagation algorithm is used to update the NN weights when they are not able to make the correct predictions. Hence, we should train the NN before applying backpropagation. Initial Weights PredictionTraining
  3. 3. Train then Update โ€ข The backpropagation algorithm is used to update the NN weights when they are not able to make the correct predictions. Hence, we should train the NN before applying backpropagation. Initial Weights PredictionTraining BackpropagationUpdate
  4. 4. Neural Network Training Example ๐— ๐Ÿ ๐— ๐Ÿ ๐Ž๐ฎ๐ญ๐ฉ๐ฎ๐ญ ๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ‘ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐–๐Ÿ ๐–๐Ÿ ๐› ๐ŸŽ. ๐Ÿ“ ๐ŸŽ. ๐Ÿ“ 1. ๐Ÿ–๐Ÿ‘ Training Data Initial Weights ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘ ๐‘ฟ ๐Ÿ In Out ๐‘พ ๐Ÿ ๐‘พ ๐Ÿ +๐Ÿ ๐’ƒ ๐‘ฟ ๐Ÿ
  5. 5. Network Training โ€ข Steps to train our network: 1. Prepare activation function input (sum of products between inputs and weights). 2. Activation function output. ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘
  6. 6. Network Training: Sum of Products โ€ข After calculating the sop between inputs and weights, next is to use this sop as the input to the activation function. ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘ ๐’” = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ ๐’” = ๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ“ + ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ + ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐’” = ๐Ÿ. ๐Ÿ—๐Ÿ’
  7. 7. Network Training: Activation Function โ€ข In this example, the sigmoid activation function is used. โ€ข Based on the sop calculated previously, the output is as follows: ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘ ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ—๐Ÿ’ = ๐Ÿ ๐Ÿ + ๐ŸŽ. ๐Ÿ๐Ÿ’๐Ÿ’ = ๐Ÿ ๐Ÿ. ๐Ÿ๐Ÿ’๐Ÿ’ ๐’‡ ๐’” = ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’
  8. 8. Network Training: Prediction Error โ€ข After getting the predicted outputs, next is to measure the prediction error of the network. โ€ข We can use the squared error function defined as follows: โ€ข Based on the predicted output, the prediction error is: ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’ ๐Ÿ = ๐Ÿ ๐Ÿ โˆ’๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ ๐Ÿ = ๐Ÿ ๐Ÿ ๐ŸŽ. ๐Ÿ•๐Ÿ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ‘๐Ÿ“๐Ÿ•
  9. 9. How to Minimize Prediction Error? โ€ข There is a prediction error and it should be minimized until reaching an acceptable error. What should we do in order to minimize the error? โ€ข There must be something to change in order to minimize the error. In our example, the only parameter to change is the weight. How to update the weights? โ€ข We can use the weights update equation: ๐‘พ ๐’๐’†๐’˜ = ๐‘พ ๐’๐’๐’… + ฮท ๐’… โˆ’ ๐’€ ๐‘ฟ
  10. 10. Weights Update Equation โ€ข We can use the weights update equation: ๏‚ง ๐‘พ ๐’๐’†๐’˜: new updated weights. ๏‚ง ๐‘พ ๐’๐’๐’…: current weights. [1.83, 0.5, 0.2] ๏‚ง ฮท: network learning rate. 0.01 ๏‚ง ๐’…: desired output. 0.03 ๏‚ง ๐’€: predicted output. 0.874 ๏‚ง ๐‘ฟ: current input at which the network made false prediction. [+1, 0.1, 0.3] ๐‘พ ๐’๐’†๐’˜ = ๐‘พ ๐’๐’๐’… + ฮท ๐’… โˆ’ ๐’€ ๐‘ฟ
  11. 11. Weights Update Equation ๐‘พ ๐’๐’†๐’˜ = ๐‘พ ๐’๐’๐’… + ฮท ๐’… โˆ’ ๐’€ ๐‘ฟ = [๐Ÿ. ๐Ÿ–๐Ÿ‘, ๐ŸŽ. ๐Ÿ“, ๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐ŸŽ๐Ÿ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’ [+๐Ÿ, ๐ŸŽ. ๐Ÿ, ๐ŸŽ. ๐Ÿ‘ = [๐Ÿ. ๐Ÿ–๐Ÿ‘, ๐ŸŽ. ๐Ÿ“, ๐ŸŽ. ๐Ÿ + โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ–๐Ÿ’[+๐Ÿ, ๐ŸŽ. ๐Ÿ, ๐ŸŽ. ๐Ÿ‘ = [๐Ÿ. ๐Ÿ–๐Ÿ‘, ๐ŸŽ. ๐Ÿ“, ๐ŸŽ. ๐Ÿ + [โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ–๐Ÿ’, โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐ŸŽ๐Ÿ–๐Ÿ’, โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ๐Ÿ“ = [๐Ÿ. ๐Ÿ–๐Ÿ๐Ÿ, ๐ŸŽ. ๐Ÿ’๐Ÿ—๐Ÿ—, ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ–
  12. 12. Weights Update Equation โ€ข The new weights are: โ€ข Based on the new weights, the network will be re-trained. ๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐’ƒ ๐’๐’†๐’˜ ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ– ๐ŸŽ. ๐Ÿ’๐Ÿ—๐Ÿ— ๐Ÿ. ๐Ÿ–๐Ÿ๐Ÿ ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐ŸŽ. ๐Ÿ‘
  13. 13. Weights Update Equation โ€ข The new weights are: โ€ข Based on the new weights, the network will be re-trained. โ€ข Continue these operations until prediction error reaches an acceptable value. 1. Updating weights. 2. Retraining network. 3. Calculating prediction error. ๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐‘พ ๐Ÿ๐’๐’†๐’˜ ๐’ƒ ๐’๐’†๐’˜ ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ– ๐ŸŽ. ๐Ÿ’๐Ÿ—๐Ÿ— ๐Ÿ. ๐Ÿ–๐Ÿ๐Ÿ ๐ŸŽ. ๐Ÿ In Out ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ’๐Ÿ—๐Ÿ— ๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ– +๐Ÿ ๐’ƒ = ๐Ÿ. ๐Ÿ–22 ๐ŸŽ. ๐Ÿ‘
  14. 14. Why Backpropagation Algorithm is Important? โ€ข The backpropagation algorithm is used to answer these questions and understand effect of each weight over the prediction error. New Weights !Old Weights
  15. 15. Forward Vs. Backward Passes โ€ข When training a neural network, there are two passes: forward and backward. โ€ข The goal of the backward pass is to know how each weight affects the total error. In other words, how changing the weights changes the prediction error? Forward Backward
  16. 16. Backward Pass โ€ข Let us work with a simpler example: โ€ข How to answer this question: What is the effect on the output Y given a change in variable X? โ€ข This question is answered using derivatives. Derivative of Y wrt X ( ๐๐’€ ๐๐‘ฟ ) will tell us the effect of changing the variable X over the output Y. ๐’€ = ๐‘ฟ ๐Ÿ ๐’ + ๐‘ฏ
  17. 17. Calculating Derivatives โ€ข The derivative ๐๐’€ ๐๐‘ฟ can be calculated as follows: โ€ข Based on these two derivative rules: โ€ข The result will be: ๐๐’€ ๐››๐‘ฟ = ๐›› ๐››๐‘ฟ (๐‘ฟ ๐Ÿ ๐’ + ๐‘ฏ) ๐’€ = ๐‘ฟ ๐Ÿ ๐’ + ๐‘ฏ ๐›› ๐››๐‘ฟ ๐‘ฟ ๐Ÿ = ๐Ÿ๐‘ฟSquare ๐›› ๐››๐‘ฟ ๐‘ช = ๐ŸŽConstant ๐๐’€ ๐››๐‘ฟ = ๐Ÿ๐‘ฟ๐’ + ๐ŸŽ = ๐Ÿ๐‘ฟ๐’
  18. 18. Prediction Error โ€“ Weight Derivative E W? ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ Change in Y wrt X ๐๐’€ ๐››๐‘ฟ Change in E wrt W ๐๐‘ฌ ๐››๐‘พ
  19. 19. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
  20. 20. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
  21. 21. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•)
  22. 22. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’”
  23. 23. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐Ÿ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’”
  24. 24. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐Ÿ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’”
  25. 25. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐Ÿ ๐’” = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’”
  26. 26. Prediction Error โ€“ Weight Derivative ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ (๐‘ช๐’๐’๐’”๐’•๐’‚๐’๐’•) ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‡ ๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐’” = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’(๐‘ฟ1โˆ— ๐‘พ1+ ๐‘ฟ2โˆ—๐‘พ2+๐’ƒ) ๐Ÿ
  27. 27. Multivariate Chain Rule Predicted Output Prediction Error sop Weights ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’‡ ๐’™ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐’” = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐‘พ ๐Ÿ, ๐‘พ ๐Ÿ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’(๐‘ฟ1โˆ— ๐‘พ1+ ๐‘ฟ2โˆ—๐‘พ2+๐’ƒ) ๐Ÿ ๐๐‘ฌ ๐๐‘พ = ๐ ๐๐‘พ ( ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’(๐‘ฟ ๐Ÿโˆ— ๐‘พ ๐Ÿ+ ๐‘ฟ ๐Ÿโˆ—๐‘พ ๐Ÿ+๐’ƒ) ๐Ÿ ) Chain Rule
  28. 28. Multivariate Chain Rule Predicted Output Prediction Error sop Weights ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ ๐’‡ ๐’™ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ๐’” = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐‘พ ๐Ÿ, ๐‘พ ๐Ÿ ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐››๐’” ๐๐’” ๐››๐‘พ ๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ Letโ€™s calculate these individual partial derivatives. ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ— ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” โˆ— ๐๐’” ๐๐‘พ ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ— ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” โˆ— ๐๐’” ๐๐‘พ ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ— ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” โˆ— ๐๐’” ๐๐‘พ ๐Ÿ
  29. 29. Error-Predicted ( ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ) Partial Derivative Substitution ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ( ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ) = ๐Ÿ โˆ— ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿโˆ’๐Ÿ โˆ— (๐ŸŽ โˆ’ ๐Ÿ) )= (๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’…) โˆ— (โˆ’๐Ÿ = ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ’ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ’ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’‘๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐Ÿ
  30. 30. Predicted-sop ( ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” ) Partial Derivative ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” = ๐ ๐๐’” ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ) ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” (๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ) ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” (๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’” ) = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ—๐Ÿ’ (๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ—๐Ÿ’ ) = ๐Ÿ ๐Ÿ + ๐ŸŽ. ๐Ÿ๐Ÿ’๐Ÿ’ (๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐ŸŽ. ๐Ÿ๐Ÿ’๐Ÿ’ ) = ๐Ÿ ๐Ÿ. ๐Ÿ๐Ÿ’๐Ÿ’ (๐Ÿ โˆ’ ๐Ÿ ๐Ÿ. ๐Ÿ๐Ÿ’๐Ÿ’ ) = ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’(๐Ÿ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’) = ๐ŸŽ. ๐Ÿ–๐Ÿ•๐Ÿ’(๐ŸŽ. ๐Ÿ๐Ÿ๐Ÿ”) ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐››๐’” = ๐ŸŽ. ๐Ÿ๐Ÿ Substitution ๐๐ซ๐ž๐๐ข๐œ๐ญ๐ž๐ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’”
  31. 31. Sop-๐‘Š1 ( ๐๐’” ๐››๐‘พ ๐Ÿ ) Partial Derivative ๐๐’” ๐››๐‘พ ๐Ÿ = ๐›› ๐››๐‘พ ๐Ÿ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ) = ๐Ÿ โˆ— ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ ๐Ÿโˆ’๐Ÿ + ๐ŸŽ + ๐ŸŽ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ ๐ŸŽ )= ๐‘ฟ ๐Ÿ(๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ Substitution ๐๐’” ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ ๐ฌ = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ
  32. 32. ๐๐’” ๐››๐‘พ ๐Ÿ = ๐›› ๐››๐‘พ ๐Ÿ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ) = ๐ŸŽ + ๐Ÿ โˆ— ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ ๐Ÿโˆ’๐Ÿ + ๐ŸŽ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ ๐ŸŽ )= ๐‘ฟ ๐Ÿ(๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ = ๐ŸŽ. ๐Ÿ‘ Substitution ๐๐’” ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ‘ ๐ฌ = ๐‘ฟ1 โˆ— ๐‘พ1 + ๐‘ฟ2 โˆ— ๐‘พ2 + ๐’ƒ Sop-๐‘Š1 ( ๐๐’” ๐››๐‘พ ๐Ÿ ) Partial Derivative
  33. 33. Error-๐‘Š1 ( ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ ) Partial Derivative โ€ข After calculating each individual derivative, we can multiply all of them to get the desired relationship between the prediction error and each weight. ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ— ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” โˆ— ๐๐’” ๐๐‘พ ๐Ÿ ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐››๐’” = ๐ŸŽ. ๐Ÿ๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐ŸŽ๐Ÿ Calculated Derivatives
  34. 34. Error-๐‘Š2 ( ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ ) Partial Derivative ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐๐‘ฌ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… โˆ— ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐๐’” โˆ— ๐๐’” ๐๐‘พ ๐Ÿ ๐๐‘ฌ ๐››๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ ๐๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… ๐››๐’” = ๐ŸŽ. ๐Ÿ๐Ÿ ๐๐’” ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ‘ ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ–๐Ÿ’๐Ÿ’ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ‘ Calculated Derivatives
  35. 35. Interpreting Derivatives โ€ข There are two useful pieces of information from the derivatives calculated previously. Increasing/decreasing weight increases/decreases error. Derivative MagnitudeDerivative Sign Positive Increasing/decreasing weight decreases/increases error. Negative Increasing/decreasing weight by P increases/decreases error by MAG*P. Increasing/decreasing weight by P decreases/increases error by MAG*P. Positive Sign Negative Sign In our example, because both ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ and ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ are positive, then we would like to decrease the weights in order to decrease the prediction error. ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐ŸŽ๐Ÿ
  36. 36. Updating Weights โ€ข Each weight will be updated based on its derivative according to this equation: ๐‘พ๐’Š๐’๐’†๐’˜ = ๐‘พ๐’Š๐’๐’๐’… โˆ’ ฮท โˆ— ๐››๐‘ฌ ๐››๐‘พ๐’Š ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ— ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ โˆ’ 0.01 โˆ— ๐ŸŽ. ๐ŸŽ๐Ÿ ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐ŸŽ. ๐Ÿ’๐Ÿ—๐Ÿ—๐Ÿ—๐Ÿ ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ— ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ โˆ’ 0.01 โˆ— ๐ŸŽ. ๐ŸŽ๐Ÿ๐Ÿ– ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ—๐Ÿ• Updating ๐‘พ ๐Ÿ Updating ๐‘พ ๐Ÿ Continue updating weights according to derivatives and re-train the network until reaching an acceptable error.
  37. 37. Second Example Backpropagation for NN with Hidden Layer
  38. 38. ANN with Hidden Layer ๐‘พ ๐Ÿ ๐‘พ ๐Ÿ ๐‘พ ๐Ÿ‘ ๐‘พ ๐Ÿ’ ๐‘พ ๐Ÿ“ ๐‘พ ๐Ÿ” ๐’ƒ ๐Ÿ ๐’ƒ ๐Ÿ ๐’ƒ ๐Ÿ‘ ๐ŸŽ. ๐Ÿ“ ๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ”๐Ÿ ๐ŸŽ. ๐Ÿ โˆ’๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ‘ ๐ŸŽ. ๐Ÿ’ โˆ’๐ŸŽ. ๐Ÿ ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐— ๐Ÿ ๐— ๐Ÿ ๐Ž๐ฎ๐ญ๐ฉ๐ฎ๐ญ ๐ŸŽ. ๐Ÿ ๐ŸŽ. ๐Ÿ‘ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ Training Data Initial Weights
  39. 39. ANN with Hidden Layer Initial Weights PredictionTraining
  40. 40. ANN with Hidden Layer Initial Weights PredictionTraining BackpropagationUpdate
  41. 41. Forward Pass โ€“ Hidden Layer Neurons ๐’‰ ๐Ÿ๐’Š๐’ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐Ÿ = ๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ“ + ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐Ÿ’ ๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ’๐Ÿ– ๐’‰ ๐Ÿ๐’๐’–๐’• = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐Ÿ’๐Ÿ– ๐’‰ ๐Ÿ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– ๐’‰ ๐Ÿ In Out
  42. 42. Forward Pass โ€“ Hidden Layer Neurons ๐’‰ ๐Ÿ๐’Š๐’ = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ = ๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ”๐Ÿ + ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐Ÿ ๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐ŸŽ๐Ÿ๐Ÿ ๐’‰ ๐Ÿ๐’๐’–๐’• = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐ŸŽ๐Ÿ๐Ÿ ๐’‰ ๐Ÿ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” ๐’‰ ๐Ÿ In Out
  43. 43. Forward Pass โ€“ Output Layer Neuron ๐’๐’–๐’•๐’Š๐’ = ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– โˆ— โˆ’๐ŸŽ. ๐Ÿ + ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” โˆ— ๐ŸŽ. ๐Ÿ‘ + ๐Ÿ. ๐Ÿ–๐Ÿ‘ ๐’๐’–๐’•๐’Š๐’ = ๐Ÿ. ๐Ÿ–๐Ÿ“๐Ÿ– ๐’๐’–๐’• ๐’๐’–๐’• = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’๐’–๐’• ๐’Š๐’ = ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ–๐Ÿ“๐Ÿ– ๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ”๐Ÿ“ ๐’๐’–๐’• In Out
  44. 44. Forward Pass โ€“ Prediction Error ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐‘ฌ = ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’๐’–๐’• ๐’๐’–๐’• ๐Ÿ = ๐Ÿ ๐Ÿ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐Ÿ–๐Ÿ”๐Ÿ“ ๐Ÿ ๐‘ฌ = ๐ŸŽ. ๐Ÿ‘๐Ÿ’๐Ÿ— ๐‘ท๐’“๐’†๐’…๐’Š๐’„๐’•๐’†๐’… = ๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ”๐Ÿ“ ๐๐‘ฌ ๐๐‘พ ๐Ÿ , ๐๐‘ฌ ๐๐‘พ ๐Ÿ , ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ , ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ , ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ , ๐๐‘ฌ ๐๐‘พ ๐Ÿ”
  45. 45. Partial Derivatives Calculation
  46. 46. Eโˆ’๐‘Š5 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ“
  47. 47. Eโˆ’๐‘Š5 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ“ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ ๐๐’๐’–๐’• ๐’๐’–๐’• ( ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’๐’–๐’• ๐’๐’–๐’• ๐Ÿ ) = ๐Ÿ โˆ— ๐Ÿ ๐Ÿ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’๐’–๐’• ๐’๐’–๐’• ๐Ÿโˆ’๐Ÿ โˆ— (๐ŸŽ โˆ’ ๐Ÿ) = ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… โˆ’ ๐’๐’–๐’• ๐’๐’–๐’• โˆ— (โˆ’๐Ÿ) ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐’๐’–๐’• ๐’๐’–๐’• โˆ’ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐’๐’–๐’• ๐’๐’–๐’• โˆ’ ๐’…๐’†๐’”๐’Š๐’“๐’†๐’… = ๐ŸŽ. ๐Ÿ–๐Ÿ”๐Ÿ“ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ Partial Derivative Substitution
  48. 48. Eโˆ’๐‘Š5 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ“ ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ ๐๐’๐’–๐’•๐’Š๐’ ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’๐’–๐’• ๐’Š๐’ ) ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’๐’–๐’• ๐’Š๐’ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’๐’–๐’• ๐’Š๐’ ) ๐œ•๐’๐’–๐’• ๐’๐’–๐’• ๐œ•๐’๐’–๐’•๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ–๐Ÿ“๐Ÿ– )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐Ÿ.๐Ÿ–๐Ÿ“๐Ÿ– ) = ( ๐Ÿ ๐Ÿ. ๐Ÿ“๐Ÿ” )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ. ๐Ÿ“๐Ÿ” ) = ๐ŸŽ. ๐Ÿ”๐Ÿ’๐Ÿ ๐Ÿ โˆ’ ๐ŸŽ. ๐Ÿ”๐Ÿ’๐Ÿ = ๐ŸŽ. ๐Ÿ”๐Ÿ’๐Ÿ ๐ŸŽ. ๐Ÿ‘๐Ÿ“๐Ÿ— ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ Partial Derivative Substitution
  49. 49. Eโˆ’๐‘Š5 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ“ = ๐ ๐๐‘พ ๐Ÿ“ (๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘) = ๐Ÿ โˆ— ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— (๐‘พ ๐Ÿ“) ๐Ÿโˆ’๐Ÿ + ๐ŸŽ + ๐ŸŽ ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ“ = ๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ“ = ๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ“ = ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– Partial Derivative Substitution
  50. 50. Eโˆ’๐‘Š5 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ“ = ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ = ๐ŸŽ. ๐Ÿ๐Ÿ๐Ÿ—
  51. 51. Eโˆ’๐‘Š6 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ” ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ”
  52. 52. Eโˆ’๐‘Š6 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ” ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ” ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“
  53. 53. Eโˆ’๐‘Š6 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ” ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ“ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ” ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ” = ๐ ๐๐‘พ ๐Ÿ” (๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘) = ๐ŸŽ + ๐Ÿ โˆ— ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— (๐‘พ ๐Ÿ”) ๐Ÿโˆ’๐Ÿ +๐ŸŽ ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ” = ๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ” = ๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ” = ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” Partial Derivative Substitution
  54. 54. Eโˆ’๐‘Š6 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ” ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐‘พ ๐Ÿ” ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐‘พ ๐Ÿ” = ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” ๐๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ“๐ŸŽ๐Ÿ” ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐ŸŽ. ๐ŸŽ๐Ÿ—๐Ÿ•
  55. 55. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ
  56. 56. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“
  57. 57. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ Partial Derivative Substitution ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ ๐๐’‰๐Ÿ ๐’๐’–๐’• (๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘) = (๐’‰ ๐Ÿ๐’๐’–๐’•) ๐Ÿโˆ’๐Ÿ โˆ— ๐‘พ ๐Ÿ“ + ๐ŸŽ + ๐ŸŽ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐‘พ ๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐‘พ ๐Ÿ“ ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• = โˆ’๐ŸŽ. ๐Ÿ
  58. 58. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ Partial Derivative Substitution ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ๐ ๐๐’‰ ๐Ÿ๐’Š๐’ ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐Ÿ’๐Ÿ– )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐Ÿ’๐Ÿ– ) ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ”
  59. 59. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ Partial Derivative Substitution ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ ๐๐‘พ ๐Ÿ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐Ÿ) = ๐‘ฟ ๐Ÿ โˆ— (๐‘พ ๐Ÿ) ๐Ÿโˆ’๐Ÿ+ ๐ŸŽ + ๐ŸŽ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ
  60. 60. Eโˆ’๐‘Š1 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ” ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• = โˆ’๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— โˆ’๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ” โˆ— ๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ
  61. 61. Eโˆ’๐‘Š2 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative: ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ
  62. 62. Eโˆ’๐‘Š2 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative: ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ” ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• = โˆ’๐ŸŽ. ๐Ÿ
  63. 63. Eโˆ’๐‘Š2 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative: Partial Derivative Substitution ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ ๐๐‘พ ๐Ÿ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ + ๐’ƒ ๐Ÿ) = ๐ŸŽ + ๐‘ฟ ๐Ÿ โˆ— (๐‘พ ๐Ÿ) ๐Ÿโˆ’๐Ÿ+๐ŸŽ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐‘ฟ ๐Ÿ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ
  64. 64. Eโˆ’๐‘Š2 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ ) Parial Derivative: ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ” ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• = โˆ’๐ŸŽ. ๐Ÿ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ‘ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— โˆ’๐ŸŽ. ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘๐Ÿ” โˆ— ๐ŸŽ. ๐Ÿ‘ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = โˆ’. ๐ŸŽ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ
  65. 65. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘
  66. 66. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘ ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“
  67. 67. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ ๐๐’‰๐Ÿ ๐’๐’–๐’• (๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ“ + ๐’‰ ๐Ÿ๐’๐’–๐’• โˆ— ๐‘พ ๐Ÿ” + ๐’ƒ ๐Ÿ‘) = ๐ŸŽ + (๐’‰ ๐Ÿ๐’๐’–๐’•) ๐Ÿโˆ’๐Ÿโˆ— ๐‘พ ๐Ÿ” + ๐ŸŽ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐‘พ ๐Ÿ” Partial Derivative Substitution ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐‘พ ๐Ÿ” ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘
  68. 68. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ๐ ๐๐’‰ ๐Ÿ๐’Š๐’ ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) Partial Derivative Substitution ๐๐’‰๐Ÿ ๐’๐’–๐’• ๐๐’‰๐Ÿ๐’Š๐’ = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐’‰ ๐Ÿ๐’Š๐’ ) = ( ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐ŸŽ๐Ÿ๐Ÿ )(๐Ÿ โˆ’ ๐Ÿ ๐Ÿ + ๐’†โˆ’๐ŸŽ.๐ŸŽ๐Ÿ๐Ÿ ) ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ“ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘
  69. 69. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ‘ = ๐ ๐๐‘พ ๐Ÿ‘ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ) = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ = (๐‘ฟ ๐Ÿ) ๐Ÿโˆ’๐Ÿโˆ— ๐‘พ ๐Ÿ‘ + ๐ŸŽ + ๐ŸŽ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ‘ = ๐‘พ ๐Ÿ‘ Partial Derivative Substitution ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ‘ = ๐‘พ ๐Ÿ‘ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ”๐Ÿ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘
  70. 70. Eโˆ’๐‘Š3 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ ) Parial Derivative: ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ‘ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ“ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ”๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ”๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ— ๐๐‘ฌ ๐››๐‘พ ๐Ÿ‘ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ‘
  71. 71. Eโˆ’๐‘Š4 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ ) Parial Derivative: ๐๐‘ฌ ๐››๐‘พ ๐Ÿ’ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ’
  72. 72. Eโˆ’๐‘Š4 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ ) Parial Derivative: ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ‘ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ“ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ’ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ’
  73. 73. Eโˆ’๐‘Š4 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ ) Parial Derivative: ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ’ = ๐ ๐๐‘พ ๐Ÿ’ (๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ) = ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ‘ + ๐‘ฟ ๐Ÿ โˆ— ๐‘พ ๐Ÿ’ + ๐’ƒ ๐Ÿ = ๐ŸŽ + (๐‘ฟ ๐Ÿ) ๐Ÿโˆ’๐Ÿโˆ— ๐‘พ ๐Ÿ’ + ๐ŸŽ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ’ = ๐‘พ ๐Ÿ’ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ’ = ๐‘พ ๐Ÿ’ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐Ÿ Partial Derivative Substitution ๐๐‘ฌ ๐››๐‘พ ๐Ÿ’ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ’
  74. 74. Eโˆ’๐‘Š4 ( ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ ) Parial Derivative: ๐๐’๐’–๐’• ๐’๐’–๐’• ๐๐’๐’–๐’•๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ‘ ๐๐‘ฌ ๐๐’๐’–๐’• ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ ๐๐’๐’–๐’•๐’Š๐’ ๐๐’‰๐Ÿ ๐’๐’–๐’• = ๐ŸŽ. ๐Ÿ‘ ๐๐’‰ ๐Ÿ๐’๐’–๐’• ๐๐’‰ ๐Ÿ๐’Š๐’ = ๐ŸŽ. ๐Ÿ๐Ÿ“ ๐๐’‰๐Ÿ๐’Š๐’ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐Ÿ–๐Ÿ‘๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ‘ โˆ— ๐ŸŽ. ๐Ÿ๐Ÿ“ โˆ— ๐ŸŽ. ๐Ÿ ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐››๐‘พ ๐Ÿ’ = ๐››๐‘ฌ ๐››๐’๐’–๐’• ๐’๐’–๐’• โˆ— ๐››๐’๐’–๐’• ๐’๐’–๐’• ๐››๐’๐’–๐’•๐’Š๐’ โˆ— ๐››๐’๐’–๐’•๐’Š๐’ ๐››๐’‰๐Ÿ ๐’๐’–๐’• โˆ— ๐››๐’‰๐Ÿ ๐’๐’–๐’• ๐››๐’‰๐Ÿ๐’Š๐’ โˆ— ๐››๐’‰๐Ÿ๐’Š๐’ ๐››๐‘พ ๐Ÿ’
  75. 75. All Error-Weights Partial Derivatives ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ = โˆ’. ๐ŸŽ๐ŸŽ๐Ÿ‘ ๐๐‘ฌ ๐๐‘พ ๐Ÿ = โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ ๐››๐‘ฌ ๐››๐‘พ ๐Ÿ” = ๐ŸŽ. ๐ŸŽ๐Ÿ—๐Ÿ• ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ = ๐ŸŽ. ๐Ÿ๐Ÿ๐Ÿ—
  76. 76. Updated Weights ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ“ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ = ๐ŸŽ. ๐Ÿ“๐ŸŽ๐ŸŽ๐ŸŽ๐Ÿ ๐‘พ ๐Ÿ๐’๐’†๐’˜ = ๐‘พ ๐Ÿ โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ = ๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— โˆ’๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ๐ŸŽ๐ŸŽ๐ŸŽ๐Ÿ‘ ๐‘พ ๐Ÿ‘๐’๐’†๐’˜ = ๐‘พ ๐Ÿ‘ โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ”๐Ÿ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ— = ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ—๐Ÿ—๐Ÿ ๐‘พ ๐Ÿ’๐’๐’†๐’˜ = ๐‘พ ๐Ÿ’ โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ’ = ๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— ๐ŸŽ. ๐ŸŽ๐ŸŽ๐Ÿ‘ = ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ—๐Ÿ• ๐‘พ ๐Ÿ“๐’๐’†๐’˜ = ๐‘พ ๐Ÿ“ โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ“ = โˆ’๐ŸŽ. ๐Ÿ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— ๐ŸŽ. ๐Ÿ”๐Ÿ๐Ÿ– = โˆ’๐ŸŽ. ๐Ÿ๐ŸŽ๐Ÿ”๐Ÿ๐Ÿ– ๐‘พ ๐Ÿ”๐’๐’†๐’˜ = ๐‘พ ๐Ÿ” โˆ’ ฮท โˆ— ๐๐‘ฌ ๐๐‘พ ๐Ÿ” = ๐ŸŽ. ๐Ÿ‘ โˆ’ ๐ŸŽ. ๐ŸŽ๐Ÿ โˆ— ๐ŸŽ. ๐ŸŽ๐Ÿ—๐Ÿ• = ๐ŸŽ. ๐Ÿ๐Ÿ—๐Ÿ—๐ŸŽ๐Ÿ‘ Continue updating weights according to derivatives and re-train the network until reaching an acceptable error.

ร—