covers ANN basic concept . Linear and Non linear Model, Gradient decent algorithm , Non Linear activation function

1. Artificial Neural Network
Esther Pushpam V S , M.E

2. ▪ ANN
▪ Linear model
▪ Non-linear model
▪ Local Minima, Global Minima
▪ Gradient Decent Algorithm
▪ Non – Linear Activation Function
Outline
9 October 2021 Artificial Neural Network 2

3. Artificial neural network (ANN)
9 October 2021 Artificial Neural Network 3
This Photo by Unknown Author is licensed under CC BY-SA
This Photo by Unknown Author is licensed under CC BY-SA
Y = ∑ WiXi + b
Where , b is bias

4. Linear Model
y = f(x) = x1w1 + b
Note : line equation y = mx + c which similar
to neuron model , i.e w1 as m slope and b as
constant , if we adjust the weight and bias the
curve will be adjusted
b
x1
w1
y
This Photo by Unknown Author is licensed under CC BY-SA

5. Ultimate goal to fit the line on the points , the should cover maximum points , the
points which deviated from line considered error
The points above the line gives positive error. Below the line gives negative error
value.
Hence square of error or
mean square of error(E) will be considered as
E = target output – predicted output
Line fitting
9 October 2021 Artificial Neural Network 5
This Photo by Unknown Author is licensed under CC BY-SA-NC

6. We should find the convergence point ,
That convergence point called as minima
The surface may not have unique minimum
The learning starts from any point and
move towards minima
In order to identify common minima ,
we need to go through iteration
What will happen if the line is not linear
9 October 2021 Artificial Neural Network 6
This Photo by Unknown Author is licensed under CC BY-SA-NC

7. Global Minima, Local Minima
9 October 2021 Artificial Neural Network 7
This Photo by Unknown Author is licensed under CC BY-SA

8. Gradient Decent algorithm
9 October 2021 Artificial Neural Network 8
This Photo by Unknown Author is licensed under CC BY
Its is difficult to fit the curve in
multidimensional model, it has so many slope
So that gradient decent is used to optimize the
error
gradient decent (GD) =
𝜕𝐸
𝜕𝑊𝑖𝑗
the error E does not depends on only one
weight (slope) , its depends on other weight too
hence partial derivative is used.

9. Example
In multi- dimensional model, we may have several output to control the system ,
Example: input satellite pixel image to classify output 1. land , 2. water, 3. forest
x1
x2
x3
xn
Y1 = f1(x1,x2,x3…xn) = land
Y2= f2(x1,x2,x3…xn) = water
Y3= f2(x1,x2,x3…xn) = forest

10. E =
1
2
Σ 𝑇𝑜 − 𝑌𝑜 2 …………………… 2
To – Target value , Yo – predicted output
To fit the model in the Multidimensional space ,
local minima in the dimension need to converge.
Gradient decent(GD)=−
𝜕𝐸
𝜕𝑊𝑖𝑗
= To − Yo Xi = η To − Yo Xi = ∆Wij
new wij = wij + ∆wij , η – learning rate.
Gradient Decent algorithm derivation
9 October 2021 Artificial Neural Network 10
This Photo by Unknown Author is licensed under CC BY-SA-NC

11. Derivation
By chain rule
𝜕𝐸
𝜕𝑊𝑖𝑗
=
𝜕𝐸
𝜕𝑦𝑜
.
𝜕𝑦𝑜
𝜕𝑊𝑖𝑗
𝜕𝐸
𝜕𝑦𝑜
=
𝜕
𝜕𝑦𝑜
1
2
Σ 𝑇𝑜 − 𝑌𝑜 2 = - 𝑇𝑜 − 𝑌𝑜
𝜕𝑦𝑜
𝜕𝑊𝑖𝑗
= 𝜕
𝜕𝑊𝑖𝑗
σ𝑗=0
𝑁
𝑊𝑖𝑗𝑋𝑗 = Xj
Hence ,
𝜕𝐸
𝜕𝑊𝑖𝑗
= - 𝑇𝑜 − 𝑌𝑜 Xj

12. Activation function
Non-linear activation function
1.Sigmoid =
1
1+exp −𝜆𝑛𝑒𝑡
{ 0 to 1}
2.Tanh=
2
1+exp −𝜆𝑛𝑒𝑡
− 1 {-1 to 1}
This Photo by Unknown Author is licensed under CC BY-SA

13. RAJASEKARAN, S., and PAI, G. A. VIJAYALAKSHMI. NEURAL NETWORKS, FUZZY LOGIC AND
GENETIC ALGORITHM: SYNTHESIS AND APPLICATIONS (WITH CD). India, PHI Learning, 2004.
Lec-3 Gradient Descent Algorithm – YouTube-
References
9 October 2021 Artificial Neural Network 13