Multilayer feedforward networks

1. UNIT II
Neural Networks
Course :Machine Learning
By: Dr P Indira priyadarsini B.Tech,M.Tech,Ph.D
3/4/2022
DEPARTMENT OF Computer science and Engineering 1

2. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 2
MULTI LAYERED FEED FORWARD
NEURAL NETWORK ARCHITECTURES
Multilayer networks solve the classification problem for non linear sets by
employing hidden layers, whose neurons are not directly connected to the output.
The additional hidden layers can be interpreted geometrically as additional hyper-
planes, which enhance the separation capacity of the network. Figure 2.2 shows
typical multilayer network architectures.
This new architecture introduces a new question: how to train the hidden units for
which the desired output is not known. The Back propagation algorithm offers a
solution to this problem.

3. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 3
Input Nodes – The Input nodes provide information from the outside world to the
network and are together referred to as the “Input Layer”. No computation is
performed in any of the Input nodes – they just pass on the information to the hidden
nodes.
Hidden Nodes – The Hidden nodes have no direct connection with the outside world
(hence the name “hidden”). They perform computations and transfer information
from the input nodes to the output nodes. A collection of hidden nodes forms
a “Hidden Layer”.
While a feedforward network will only have a single input layer and a single output
layer, it can have zero or multiple Hidden Layers.
Output Nodes – The Output nodes are collectively referred to as the “Output Layer”
and are responsible for computations and transferring information from the network
to the outside world.

4. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 4
The training occurs in a supervised style. The basic idea is to present the input vector
to the network;
calculate in the forward direction the output of each layer and the final output of the
network.
For the output layer the desired values are known and therefore the weights can be
adjusted as for a single layer network; in the case of the BP algorithm according to the
gradient decent rule.
To calculate the weight changes in the hidden layer the error in the output layer is
back-propagated to these layers according to the connecting weights.
This process is repeated for each sample in the training set. One cycle through the
training set is called an epoch.
The number of epochs needed to train the network depends on various parameters,
especially on the error calculated in the output layer.
The following description of the Back propagation algorithm is based on the
descriptions in [rume86],[faus94], and [patt96].
The assumed architecture is depicted in Figure 2.3. The input vector has n dimensions,
the output vector has m dimensions, the bias (the used constant input) is -1, there is
one hidden layer with g neurons. The matrix V holds the weights of the neurons in the
hidden layer. The matrix W defines the weights of the neurons in the output layer. The
learning parameter is η, and the momentum is α.

5. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 5
• In machine learning, gradient descent and back propagation often appear at the
same time, and sometimes they can replace each other.
• Back propagation can be considered as a subset of gradient descent, which is the
implementation of gradient descent in multi-layer neural networks.
• Back propagation, also named the Generalized Delta Rule, is an algorithm used in
the training of ANNs for supervised learning. (generalizations exist for other
artificial neural networks) It efficiently computes the gradient of the error
function with respect to the weights of the network for a single input-output
example.
• This makes it feasible to use gradient methods for training multi-layer networks,
updating the weights to minimize loss.
• Since the same training rule is applied recursively for each layer of the neural
network, we can calculate the contribution of each weight to the total error
inversely from the output layer to the input layer.
BACKPROPAGATION

6. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 6
Back Propagation Neural Networks Continued….
• Back Propagation Neural Network is a multilayer neural network consisting of
the input layer, at least one hidden layer and output layer.
• As its name suggests, back propagating will take place in this network.
• The error which is calculated at the output layer, by comparing the target output
and the actual output, will be propagated back towards the input layer.
Architecture
• As shown in the diagram, the architecture of BPN has three interconnected
layers having weights on them.
• The hidden layer as well as the output layer also has bias, whose weight is
always 1, on them.
• As is clear from the diagram, the working of BPN is in two phases.
• One phase sends the signal from the input layer to the output layer, and the
other phase back propagates the error from the output layer to the input layer.

7. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 7
• Simply put, we’re propagating the total error backward through the
connections in the network layer by layer, calculate the contribution
(gradient) of each weight and bias to the total error in every layer, then use
gradient descent algorithm to optimize the weights and biases, and
eventually minimize the total error of the neural network.
• The back propagation algorithm has two phases:
• Forward pass phase: feed-forward propagation of input pattern signals
through the network, from inputs towards the network outputs.
• Backward pass phase: computes ‘error signal’ – propagation of error
(difference between actual and desired output values) backwards through
network, starting form output units towards the input units.
• Visualizing this can help understand how the backpropagation algorithm
works step by step
BACKPROPAGATION (CONTINUED…. )

8. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 8

9. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 9
3/4/2022 9
The back-propagation training algorithm
Step 1: Initialization
Set all the weights and threshold levels of the
network to random numbers uniformly
distributed inside a small range:
where Fi is the total number of inputs of neuron i
in the network. The weight initialization is done
on a neuron-by-neuron basis.










i
i F
F
4
.
2
,
4
.
2

10. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY
10
Step 2: Activation
Activate the back-propagation neural network
by applying inputs x1(p), x2(p),…, xn(p) and
desired outputs yd,1(p), yd,2(p),…, yd,n(p).
(a) Calculate the actual outputs of the neurons
in the hidden layer:
where n is the number of inputs of neuron j in
the hidden layer, and sigmoid is the sigmoid
activation function.











 

j
n
i
ij
i
j p
w
p
x
sigm
oid
p
y
1
)
(
)
(
)
(

11. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY
11
(b) Calculate the actual outputs of the neurons
in the output layer:
Step 2 : Activation (continued)
where m is the number of inputs of neuron k in
the output layer.











 

k
m
j
jk
jk
k p
w
p
x
sigm
oid
p
y
1
)
(
)
(
)
(

12. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 12
12
Step 3: Weight training
Update the weights in the back-propagation
network propagating backward the errors
associated with output neurons.
(a) Calculate the error gradient for the neurons in
the output layer:
where
Calculate the weight corrections:
Update the weights at the output neurons:
)
(
)
(
)
1
( p
w
p
w
p
w jk
jk
jk 



)
(
)
(
1
)
(
)
( p
e
p
y
p
y
p k
k
k
k 



)
(
)
(
)
( , p
y
p
y
p
e k
k
d
k 

)
(
)
(
)
( p
p
y
p
w k
j
jk 




13. DEPARTMENT OF INFORMATION TECHNOLOGY
13
(b) Calculate the error gradient for the neurons
in the hidden layer:
Step 3: Weight training (continued)
Calculate the weight corrections:
Update the weights at the hidden neurons:
)
(
)
(
)
(
1
)
(
)
(
1
]
[ p
w
p
p
y
p
y
p jk
l
k
k
j
j
j 





)
(
)
(
)
( p
p
x
p
w j
i
ij 



)
(
)
(
)
1
( p
w
p
w
p
w ij
ij
ij 




14. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 14
3/4/2022 14
Step 4: Iteration
Increase iteration p by one, go
back to Step 2 and repeat the process until the
selected error criterion is satisfied.
As an example, we may consider the three-layer
back-propagation network. Suppose that the
network is required to perform logical operation
Exclusive-OR. Recall that a single-layer
perceptron could not do this operation. Now we
will apply the three-layer net.

15. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 15
Working Example
Back propagation (mynotes)

16. 3/4/2022
DEPARTMENT OF INFORMATION TECHNOLOGY 16
APPLICATIONS OF FEEDFORWARD NEURAL
NETWORKS
There are a wide variety of applications of neural networks to real world problems.
1. Gene Expression Profiling for predicting Clinical Outcomes in cancer patients.
2. Steering an Autonomous Vehicle.
3. Call admission control in ATM Networks.
4. Robot Arm Control and Navigation.