The document summarizes five basic learning algorithms of artificial neural networks: Hebbian learning, memory-based learning, backpropagation, competitive learning, Adaline network, and Madaline network. It provides details on each algorithm, including mathematical formulas, steps involved, advantages and disadvantages, and applications.

1. Learning Algorithm of ANN
By:
Waseem Khan
F/o Engg. & Technology
Jamia Millia Islamia
New Delhi-110025

2. INTRODUCTION
 Learning rules are algorithms which direct changes
in the weights of the connections in a network.
They are incorporating an error reduction
procedure by employing the difference between
the desired output and an actual output to change
its weights during training.The learning rule is
typically applied repeatedly to the same set of
training inputs across a large number of epochs
with error gradually reduced across epochs as the
weights are fine-tuned.

3. FIVE BASIC LEARNING
ALGORITHM OF ANN
1. Hebbian Learning
2. Memory Based Learning
3. Back propagation
4. Competitive learning
5. Adaline network
6. Madaline network

4. Hebbian Learning
 Hebbian Learning is a learning rule which is the oldest and most famous
of all learning rules.
 Hebb’s principle can be described as a method of determining how to
alter the weights between model neurons.
 The Hebbian rule mathematically:
wkj (n)=F(yk(n), xj(n))
where F() is a function of both signals. The above formula can take
many specific forms.

5. HEBBIAN RULE
Typical examples are:
 Hebb’s hypothesis: In the simplest case we have
just the product of the two signals (it is also called
the activity product rule):
wkj (n)=yk(n) xj(n)
where  is a learning rate. This form emphasizes
the correlational nature of a Hebbian synapse.

6. HEBBIAN RULE
 Covariance hypothesis: In this case we replace
the product of pre- and post-synaptic signals with
the departure of of the same signals from their
respective average values over a certain time
interval. If x* and y* is their time-averaged value
then the covariance form is defined by:
wkj (n)=(yk(n)-y*) (xj(n)-x*)

7. Memory based Learning
 In memory-based learning, most of the past
experiences are explicitly stored in a large memory of
correctly classified input-output
 All memory-based learning algorithms involve two
factors
1. Criterion
2. Learning rule applied in local neighbourhood

8. Advantages of Memory-Based Methods
 Lazy learning:
o never need to learn a global model
o many simple local models taken together can represent
a more complex global model
o better focussed learning
o handles missing values, time varying distributions
 Very efficient cross-validation
 Intelligible learning method to many users
 Nearest neighbours support explanation and training

9. Weaknesses of Memory-Based Methods
 Curse of Dimensionality
 Run-time cost scales with training set size
 Large training sets will not fit in memory
 Many MBL methods are strict averages
 Sometimes doesn’t seem to perform as well as
other methods such as neural nets
 Predicted values for regression not continuous

10. COMPETITIVE LEARNING
 In competitive learning, neurons compete among
themselves to be activated.
 In this nodes compete for the right to respond to a
subset of the input data.
 Competitive learning works by increasing the
specialization of each node in the network.
 It is well suited to finding clusters within data.

11.  N inputs units
 P output neurons
 P x N weights
x1
x2
xN
W11
W12
W22
WP1
WPN
Y1
Y2
YP
Pi
N
j
jiji XWh
...2,1
1




01oriY

12. Three basic steps
 a set of neurons that are all the same
 a limit imposed on the strength of each neuron
 a mechanism that permits the neurons to compete- a
winner-takes-all
 The standard competitive learning rule
wkj = (xj-wkj) if neuron k wins the competition
= 0 if neuron k loses the competition

13. ADALINE LEARNING
 Network with a single linear unit.
 It receives input from several units and also
from one unit called bias.
 Uses bipolar activation for its input signals
and its target output.

14.  The total input received by the output neuron is given by
 Apply activation function over net input :
 Square of error

16. DEMERITS OF ADALINE
 Only for linearly separable problems
 Solves problems where have only one
global minimum.

17. APLICATIONS OF
ADALINE LEARNING
 Pattern Classifications
 Better convergence property than Perceptron
 Noise cancellation
 Echo cancellation
 Face recognition
 Signature recognition

18. MADALINE
 Combination of many Adalines.
 Architectures:
 Hidden layers of adaline nodes
 Output nodes differ
 Learning
 Error driven, but not by gradient descent
 Minimum disturbance: smaller change of
weights is preferred, provided it can reduce the
error

20. APLICATIONS OF
MADALINE LEARNING
 Logical Calculation
 Signal Processing
 Vehicle inductive signature recognition
 Forecasting and risk assessments.
 Used in several adaptive filtering process.
 Used to solve three Monks problems, two Led
display problems, and the And-Xor problem

21. ADVANTAGES OF
MADALINE LEARNING
 Solves non-separable problems
 It is easy for description of discrete tasks without
extra requirement of discretization
 Simple in computation and interpretation with
hard-limit activation function and limited input
and output states
 Facilitative for hardware implementation with the
available VLSI technology

22. DISADVANTAGES OF
MADALINE LEARNING
 Larger architecture
 Effect of variation of network parameters on its
output
 Madaline sensitivity

23. BACK PROPAGATION
 Back propagation is a multilayer feed forward
network with one layer of z hidden units.
 The y output units and z hidden units has b bias.
 The input layer is connected to hidden layer and
output layer is connected to the output layer by
means of interconnection weights.

24. Input
layer
xi
x1
x2
xn
1
2
i
n
Output
layer
1
2
k
l
yk
y1
y2
yl
Input signals
Error signals
wjk
Hidden
layer
wij
1
2
j
m

25. MERITS OF
BACK PROPAGATION
 Relatively simple implementation.
 It does not require any special mention of the
features of the function to be learnt.
 Computing time is reduced if the weights chosen
are small at the beginning.
 Batch updates of weights exist, which provides a
smoothing effect on the weight correction terms.

26. DEMERITS OF
BACKPROPAGATION
 Slow and inefficient.
 Large amount of input/output data is available.
 Outputs can be fuzzy or non numeric.
 The solutions of the problem may change over time within
the bounds of given input and output parameters.
 Back propagation learning does not require normalization of
input vectors; however, normalization could improve
performance.
 Gradient descent with back propagation is not guaranteed to
find the global minimum of the error function

27. APLICATIONS OF
BACKPROPAGATION
 Load forecasting problems in power systems.
 Image processing.
 Fault diagnosis and fault detection.
 Gesture recognition, speech recognition.
 Signature verification.
 Bioinformatics.
 Structural engineering design (civil).