bnm

1. CSEN3011 ARTIFICIAL NEURAL
NETWORKS
O. V. Ramana Murthy

2. Course Outcome 1
2
Understand the origin, ideological basics, Learning process and
various Neural Architectures ofANN.

3. Contents
3
 Artificial Neuron
 Neural Network
 Training with Back propagation
 Practical issues
 Common Neural Architectures

4. Reference
4
 Chapter 1, 2
Charu C.Aggarwal,“Neural Networks and Deep Learning”,
Springer International PublishingAG
 Deep Learning- Charu Aggarwal
https://www.youtube.com/playlist?list=PLLo1RD8Vbbb_6g
CyqxG_qzCLOj9EKubw7

5. Neuron
5

6. Perceptron
6
The simplest neural network is referred to as the perceptron. This neural
network contains a single input layer and an output node.

7. Artificial Neuron
7
w2
xn
x1
wn
w1

1
fact
^
𝑦
b
y
x2

8. Numerical Example 1
8
Calculate the output assuming Binary step activation function
0.6
0.2
0.7
0.3
1
fact
^
𝑦
0.45
y=0.93
^
𝑦 = 𝑓 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 ( 𝑦 )=
{1 𝑖𝑓𝑦 ≥ 0
0 𝑖𝑓𝑦 <0

9. Artificial Neuron model
9
Sigmoidal function
=1 is Binary sigmoidal function
x2
x1
w2
w1
y

10. ACTIVATION FUNCTIONS
10
(A)Identity/linear
(B) Bipolar Binary step
(C) Bipolar sigmoidal step
Source [2]

11. Numerical Example 2
11
Calculate the output assuming binary sigmoidal activation function
0.6
0.2
0.7
0.3
1
fact
^
𝑦=0.72
0.45
y=0.93
=1 is Binary sigmoidal function

12. Pre- and Post-Activation Values
12

13. Linear Separability
13

14. Linearly Separable – AND gate
14
x1
(input)
x2
(input)
Y
(output)
0 0 0
0 1 0
1 0 0
1 1 1

15. Linearly Separable – AND gate
15
 Two input sources =>Two input neurons
 One output => One output Neuron.
 Activation function is binary sigmoidal
 Derivative

16. Linearly Separable – AND gate
16
Y
𝑦
x2
x1
1
w2
w1
w0
 f(.)

17. Back-propagation training/algorithm
17
Given: Input vector i th
instant ,Target .
Initialize weights w0, w1, w2 and learning rate with some random
values in the range [0 1]
1. Output
2. Activation function sigmoidal activation function
3. Compute error:
4. Backpropagate the error to crossing activation function
where is the derivative of activation function selected.
for sigmoidal activation function

18. Back-propagation training/algorithm
18
5. Compute change in weights and bias
, ,
6. Update the changes in weights and bias
7. Keep repeating the steps 1 – 6, for all input combinations ( 4
nos).This is one epoch.
8. Run multiple Epochs till the error decreases and stabilizes.

19. Matrix Notation – AND gate
19
Forward pass
After activation function
Loss function

20. Matrix Notation – AND gate
20
 Backpropagation
 UpdateWeights:
 Update using
This iterative process continues until convergence

21. (4 Rules)Backpropagating Error
21
Y
𝑦
 f(.)
xi
𝑦
 f(.)
wi
1. Output Neuron
2.Across Link
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒𝑥𝑖
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒 𝑦
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒 𝑦
3.Weights Update

22. w2
(4 Rules)Backpropagating Error
22
xi
 f(.)
w1
4.Across Link (>1 hidden layer)
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒𝑥𝑖
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒 𝑦1
𝑦 2
 f(.)
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒 𝑦2
𝑦 1
wn 𝑦 𝑛
 f(.)
𝑒𝑎𝑡 𝑠𝑡𝑎𝑔𝑒 𝑦𝑛

23. The power of nonlinear activation functions in
transforming a data set to linear separability
23

24. Linearly not Separable – XOR gate
24
x1
(input)
x2
(input)
Y
(output)
0 0 0
0 1 1
1 0 1
1 1 0

25. Linearly not Separable – XOR gate
25
 Two input sources =>Two input neurons
 One output => One output Neuron.
 One hidden layer => 2 neurons
 Activation function is binary sigmoidal
 Derivative

26. Linearly not Separable – XOR gate
26

27. w2
w1
v21
v12
27
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
Input layer Hidden layer Output layer

28. Back-propagation Training
28
Given: Inputs , target .
Initialize weights and learning rate with some random
values
1. Hidden unit , j = 1 to p hidden neurons
2. output , sigmoidal activation function
3. Output unit
4. Output sigmoidal activation function
Feed-forward
Phase

29. Back-propagation Training
29
5. Compute error correction term
where is derivative
6. Compute change in weights and bias ,
send to previous layer
7. Hidden unit
8. Calculate error term
9. Compute change in weights and bias
,
Back-propagation
of
error
Phase

30. Back-propagation Training
30
10. Each output unit, k = 1 to m update weights and bias
11. Each hidden unit, j = 1 to p update weights and bias
12. Check for stopping criterion e.g. certain number of
epochs or when targets are equal/close to network
outputs
Weights
and
Bias
update
phase

31. w2
w1
v21
v12
31
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑧1=v11 𝑥1+v 21 𝑥2+𝑣 01
𝑧2=v12 𝑥1+v 22 𝑥2+𝑣02
Hidden neuron input computation

32. w2
w1
v21
v12
32
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑍 1=𝑓 𝑠𝑖𝑔 (𝑧 1)
𝑍 2=𝑓 𝑠𝑖𝑔 (𝑧 2)
𝑧 1
𝑧 2
Hidden neuron output computation

33. w2
w1
v21
v12
33
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑦=w1 𝑍 1+w 2𝑍 2+𝑤0
𝑧 1
𝑧 2
Output neuron input computation

34. w2
w1
v21
v12
34
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑌 =𝑓 𝑠𝑖𝑔 (𝑦)
𝑧 1
𝑧 2
Output neuron Output computation
( 𝑦 )

35. w2
w1
v21
v12
35
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝛿=(𝑡 −𝑌 ) 𝑓 ′
( 𝑦)
𝑧 1
𝑧 2
Output Error correction computation
( 𝑦 )

36. w2
w1
v21
v12
36
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
∆𝑤1=𝛿. 𝑍1
𝑧 1
𝑧 2
Output neuron changes updates computation
( 𝑦 )
𝛿
∆𝑤2=𝛿. 𝑍2
∆𝑤 0=1. 𝛿

37. w2
w1
v21
v12
37
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑧 1
𝑧 2
Hidden neuron error propagation computation
( 𝑦 )
𝛿
𝛿1=𝛿.𝑤1
𝛿2=𝛿.𝑤2

38. w2
w1
v21
v12
38
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑧 1
𝑧 2
Hidden neuron error correction computation
( 𝑦 )
𝛿
𝛿11=𝛿1. 𝑓 ′
(𝑧 1)
𝛿22=𝛿2. 𝑓 ′
(𝑧 2)
𝛿1
𝛿2

39. w2
w1
v21
v12
39
x2
x1
v11
Y
1 1
v22
v01
v02
Z2
Z1
w0
𝑧 1
𝑧 2
Hidden neuron changes updates computation
( 𝑦 )
𝛿
𝛿1
𝛿2
∆ 𝑣11=𝛿11𝑥1
1
∆ 𝑣01=𝛿11
𝛿11
𝛿22
2
2
∆ 𝑣02=𝛿22

40. Matrix Notation – XOR gate
40
Given

41. Matrix Notation – XOR gate
41
Forward pass Hidden layer
After activation function

42. Matrix Notation – XOR gate
42
Forward pass Output layer
After activation function
Loss function

43. Matrix Notation – XOR gate
43
 Backpropagation
 UpdateWeights:
Hidden layer gradient

44. Matrix Notation – XOR gate
44
Update using
This iterative process continues until convergence

45. NN with Two Hidden Layers (HW)
45
𝑤𝑖 , 𝑗
(1)

46. Practical Issues – Softmax Layer
46

47. Example
47
z =[2.0 1.0 0.1]
Then
LetTargets beT = [ 1 0 0]
Define Loss function

48. Example
48
Homework.
Loss function is cross entropy

49. Regularization (to avoid Overfitting)
49
One of the primary causes of corruption of the generalization
process is overfitting.
The objective is to determine a curve that defines the border of
the two groups using the training data.

50. Overfitting
50
One of the primary causes of corruption of the generalization
process is overfitting.
The objective is to determine a curve that defines the border of
the two groups using the training data.

51. Overfitting
51
Some outliers penetrate the area of the other group and disturb
the boundary. As Machine Learning considers all the data, even
the noise, it ends up producing an improper model (a curve in
this case).This would be penny-wise and pound-foolish.

52. Remedy : Regularization
52
 Regularization is a numerical method that attempts to
construct a model structure as simple as possible. The
simplified model can avoid the effects of overfitting at the
small cost of performance.
 Cost function Sum of squared errors

53. Remedy : Regularization
53
 For this reason, overfitting of the neural network can be
improved by adding the sum of weights to the cost function,
(new) Cost function
 In order to drop the value of the cost function, both the error
and weight should be controlled to be as small as possible.
 However, if a weight becomes small enough, the associated
nodes will be practically disconnected. As a result,
unnecessary connections are eliminated, and the neural
network becomes simpler.

54. Add L1 Regularization to XOR Network
54
 New Loss function
 The gradient of the regularized loss w.r.t a weight w is:
Update rule for weights w is

55. Add L2 Regularization to XOR Network
55
 New Loss function
 The gradient of the regularized loss w.r.t a weight w is:
Update rule for weights w is

56. XOR implementation with L1
56
# Apply L2 regularization to weights
hidden_layer_weights += learning_rate *
(np.dot(hidden_layer_output.T, output_layer_delta) –
sign(hidden_layer_weights))
input_layer_weights += learning_rate *
(np.dot(inputs.T, hidden_layer_delta) -
sign(input_layer_weights))
# Update biases (no regularization applied to biases)
hidden_layer_bias += np.sum(output_layer_delta,
axis=0, keepdims=True) * learning_rate
input_layer_bias += np.sum(hidden_layer_delta, axis=0,
keepdims=True) * learning_rate

57. XOR implementation with L2
57
# Apply L2 regularization to weights
hidden_layer_weights += learning_rate *
(np.dot(hidden_layer_output.T, output_layer_delta) -
hidden_layer_weights)
input_layer_weights += learning_rate *
(np.dot(inputs.T, hidden_layer_delta) -
input_layer_weights)
# Update biases (no regularization applied to biases)
hidden_layer_bias += np.sum(output_layer_delta,
axis=0, keepdims=True) * learning_rate
input_layer_bias += np.sum(hidden_layer_delta, axis=0,
keepdims=True) * learning_rate

58. Matrix Notation – XOR gate with L2
Regularization
58
Expanding from Slide 44.
Only weights will be updated as follows. Bias values won’t change.
This iterative process continues until convergence.
L2 regularization penalizes large weights, resulting in slightly
smaller weight updates compared to the non-regularized case.

59. Common Neural Architectures
59
 Autoencoder (Module 2)
 Deep Neural Network (Module 3)
 Attractor Neural Networks (Module 4)
 Self-organizing Maps (module 5)

60. Autoencoder
60

61. Autoencoder
61

62. Deep Neural Network
62

63. Attractor Neural Networks
63

64. Self-Organizing Maps
64

65. 65

66. Appendix: Example Implementation
66
Using Back-
propagation
network, find the
new weights for the
network shown
aside. Input = [0 1]
and target output is
1. use learning rate
0.25 and binary
sigmoidal activation
function

67. 1. Consolidate the information
67
 Given: Inputs [0 1], target 1.
 [
 []=[0.4 0.1 0.2]
 Learning rate
 Activation function is binary sigmoidal
 Derivative

68. 2. Feed-forward Phase
68
1. Hidden unit , j = 1,2
2. Output , sigmoidal activation function
3. Output unit
4. Output sigmoidal activation function

69. 2. Feed-forward Phase
69
1. Hidden unit , j = 1,2
2. Output , sigmoidal activation function ,
3. Output unit
4. Output sigmoidal activation function

70. 3. Back-propagation of error Phase
70
5. Compute error correction term
6. Compute change in weights and bias
,
,
,

71. 3. Back-propagation of error Phase
71
5. Compute error correction term
6. Compute change in weights and bias
,
,
,
7. Hidden unit

72. 3. Back-propagation of error Phase
72
5. Compute error correction term
6. Compute change in weights and bias
,
,
,
7. Hidden unit

73. 3. Back-propagation of error Phase
73
8. Calculate error term

74. 3. Back-propagation of error Phase
74
8. Calculate error term

75. 3. Back-propagation of error Phase
75
8. Calculate error term
9. Compute change in weights and bias
,

76. 3. Back-propagation of error Phase
76
8. Calculate error term
9. Compute change in weights and bias
,
0.0118
0.0118

77. 3. Back-propagation of error Phase
77
8. Calculate error term
9. Compute change in weights and bias
,
0.0118
0.0118
0.0
0.00245

78. 3. Back-propagation of error Phase
78
8. Calculate error term
9. Compute change in weights and bias
,
0.0118
0.0118
0.0
0.00245

79. 4. Weights and Bias update phase
79
10. Each output unit, k = 1 to m update weights and bias
,

80. 4. Weights and Bias update phase
80
10. Each output unit, k = 1 to m update weights and bias
,

81. 4. Weights and Bias update phase
81
11. Each hidden unit, j = 1 to p update weights and bias

82. 4. Weights and Bias update phase
82
11. Each hidden unit, j = 1 to p update weights and bias

83. 4. Weights and Bias update phase
83
11. Each hidden unit, j = 1 to p update weights and bias

84. 84
Epoch v 11 v21 v01 v12 v22 v02
0 0.6 -0.1 0.3 -0.3 0.4 0.5
1 0.6 -0.097 0.303 -0.3 0.401 0.501
Write a program for this case and cross-verify your answers.
After how many epochs will the output converge?
Epoch z1 z2 w1 w2 w0 y
0 0.549 0.711 0.4 0.1 -0.2 0.523
1 0.5513 0.7113 0.416 0.121 -0.17 0.5363