ML

1. Neural network and learning machines
Backpropagation for updating weights

2. Neural Network training steps
Weight Initialization
Inputs Application
Sum of inputs - Weights product
Activation functions
Weights Adaptations
Back to step 2
1
2
3
4
5
6

3. 0 ≤ α ≤ 1
0 ≤  ≤ 1
Learning Rate 
First method:
Regarding 5th step: Weights Adaptation

4. Second method: Back propagation
Regarding 5th step: Weights Adaptation
Feedforward
Inputs
Outputs
Backward
 Fowrward VS Backword passes
The Backpropagation algorithm is a sensible
approach for dividing the contribution of each
weight.
Fowrward Input
weights
backward
SOP
Prediction
Output
Prediction
Error
Prediction
Error
Prediction
Output
SOP
Input
weights

5. Backward pass
Let us work with a simpler example
𝒚 = 𝒙𝟐 z+ c
How to answer this question: What is the effect on the output Y
given a change in variable X?
This question is answered using derivatives. Derivative of Y wrt X ( 𝜕𝑦ൗ 𝜕𝑥
) will tell us the effect of changing the variable X over the output Y.

6. Backward pass
Calculating the Derivatives
𝒚 = 𝒙𝟐 z+ c
The Derivative ( 𝜕𝑦ൗ 𝜕𝑥) can be calculated as
follow 𝝏
𝝏𝒙
𝒙𝟐 z+ c
Based on these two derivative rules:
Square Constant
𝝏
𝝏𝒙
𝒙𝟐 =2x
𝝏𝒙
𝝏
c=0
The Result will be :
𝝏
𝝏𝒙
𝒙𝟐 z+ c=2zx+0=2zx

7. Backward pass
Calculating the Derivative of prediction Error wrt Weights
𝟐
𝑬 =
𝟏
(𝒅𝒆𝒔𝒊𝒓𝒆𝒅 − 𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅)𝟐
𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 = 𝒇(𝒔) =
𝟏
𝟏 + 𝒆−𝒔
𝒑𝒓𝒆𝒅𝒆𝒔𝒊𝒓𝒆𝒅 = 𝑪𝒐𝒏𝒔𝒕𝒂𝒏𝒕
𝟐
𝑬 =
𝟏
(𝒅𝒆𝒔𝒊𝒓𝒆𝒅 −
𝟏
𝟏 + 𝒆−𝒔
)𝟐
m
w j i  b i
s   x i
j
) 2
n
j
1
x i w i j
 b i
e  
E 
1
( d 
2

8. second method: Back propagation
Regarding 5th step: Weights Adaptation
 Backword pass
What is the change in prediction Error (E) given the change in weight (W) ?
Get partial derivative of E W.R.T W E
W
1
E  (d  y)2
2
d (desired output) Const
y ( predicted output)
s
1e
1
f (s) 
s (Sum Of Product SOP )
m
s   xi w ji  bi
j
) 2
n
j
1
x i w i j
 b i
e 
E 
1
( d 
2
w1, w2

9. second method: Back propagation
Regarding 5th step: Weights Adaptation
 Weight derivative
E 
1
(d  y)2
2
1
1es
y  f (s)  s  x1 w1  x2 w2  b w1, w2
W
E
y
E
s
y
w1 w2
s
,
s
Chain Rule
 w2 y s  w2
E

E
x
y
x
s
 w1 y s  w1
E

E
x
y
x
s

10. second method: Back propagation
Regarding 5th step: Weights Adaptation
 Weight derivative
 E


y y 2
1
(d  y ) 2
 y  d
)
(1
1 1
1 es
1 es

s 1 es
s
y

 1
1 1 2 2 1
1
1
w w
s


x wx w b  x 2
1 1 2 2
2
2
w w
s


x w  x w b  x
i
i
) x
(1
1 1
1 es
1 es
E
 (y  d)
 w

11. second method: Back propagation
Regarding 5th step: Weights Adaptation
 Update the Weights
In order to update the weights , use the Gradient Descent
f(w)
+ slop
w
Wnew= Wold - (+ve)
f(w)
- slop
w
Wnew= Wold - (-ve)

12. Example
Learning rate : 0.01