Artificial neural network

1. Artificial Neural Networks

2. WHAT IS NEURAL NETWORK
• Brain inspired mathematical models
• Biological and artificial NN

3. BIOLOGICAL NEURON

4. Biological Neural Network

5. BRAIN AND ANN
• Brain:
–1010 neurons
–1013 connections
–speed at milisec
–Fully parallel
• ANN:
–20 000 neurons
–Speed at nanosec
–Simulating parallel
computation

6. Evolution of ANN
Year Neural Network Designer Description
1943 McCulloch and Pitts
Neuron
Mcculloch Pitts Logic gates
1949 Hebb Hebb Strength increases if
neurons are active
1958-1988 Perceptron Rosenblatt Weights of path can be
adjusted
1960 Adaline Widrow and Hoff Mean squared error
1972 SOM Kohenen Clustering
1982 Hopfield John Hopfield Associative memory
nets
1986 Back Propagation Rumelhard Multilayer
1987-90 ART Carpenter Used for both binary
and analog

7. Biological and Artificial Neuron
• Basic parts: Body(soma), Dendrites(inputs), Axons (outputs),
Synapses (connections)

8. Basic parts of artificial neuron
• Input summing function
• Activation function
• Weighted inputs
• Output
output = f (w1in1+ …+wninn)

11. Activation Functions

12. Activation Functions

13. The McCulloch-Pitts Neuron
This vastly simplified model of real neurons is also known as a
Threshold Logic Unit
A set of connections brings in activations from other neurons.
A processing unit sums the inputs, and then applies a non-linear activation
function (i.e. squashing/transfer/threshold function).
An output line transmits the result to other neurons
y = F(w1x1+ w2x2 - b)

14. Common Models of Neurons
Binary
perceptrons
Continuous perceptrons

15. 1st Neural Network: AND function
Threshold(Y) = 2
X1
Y
X2
1
1
1
2
Y
1st Neural Network: OR function
X1
Y
X2
2
2
Threshold(Y) = 2
1
2
Y

16. 1st Neural Network: AND not function
Threshold(Y) = 2
1
2
Y
X1
Y
X2
2
-1
1st Neural Network: XOR function
X1
Y
X2
2
-1
Z2
Z1
2
-1
2
2
Threshold(Y) = 2
1
2
Y

17. McCulloch-Pitts Neuron Model

20. Classification Based on Interconnections
Interconnections
Feed
forward
Single
layer
Multilayer
Feed Back Recurrent
Single
layer
Multilayer
Network architecture is the arrangement of neurons to form layers and
connection patterns formed within and between layers

21. Neural Network Architectures

22. Neural network training/learning
• Learning procedure: adjusting connection weights, until network
gets desired behaviour
 Supervised Learning
 Unsupervised Learning
 Reinforcement Learning

23. Supervised Learing

24. Perceptron Learning Rule
• Learning signal is the difference between the desired and actual
neuron’s response
• Learning is supervised

27. Widrow-Hoff learning Rule
• Also called as least mean square learning rule
• Introduced by Widrow(1962), used in supervised learning
• Independent of the activation function
• Special case of delta learning rule wherein activation function is an identity function ie
f(net)=net
• Minimizes the squared error between the desired output value di and neti

28. ADALINE
Linear transfer function
Linear combination of inputs
y = w1x1+w2x2+…+wnxn,
x1
x2
x3
w1
w2
w3
y

29. Delta Learning Rule
• Only valid for continuous activation function
• Used in supervised training mode
• Learning signal for this rule is called delta
• The aim of the delta rule is to minimize the error over all training patterns

30. Learning rule is derived from the condition of least squared error.
Calculating the gradient vector with respect to wi
Minimization of error requires the weight changes to be in the negative
gradient direction

34. Unsupervised Learning

35. Hebbian Learning Rule
• The learning signal is equal to the neuron’s output
Feed Forward Unsupervised Learning

36. Features of Hebbian Learning
•Feedforward unsupervised learning
•“When an axon of a cell A is near enough to excite a
cell B and repeatedly and persistently takes place in
firing it, some growth process or change takes place in
one or both cells increasing the efficiency”
•If oixj is positive the results is increase in weight else
vice versa

39. Reinforcement learning

40. Multi Layer Perceptron
• Basic perceptron extended with
one or more layers of hidden
neurons between input and
output layer
• Differentiable neuron transfer
functions (tanh, sigmoid)
• Using Backpropagation
algorithm for learning, which is
based on LMS algorithm
• Able to solve complex
problems

41. Backpropagation algorithm
• For Multi Layer Perceptron training – able to adjust
weights in hidden layers
• Supervised learning algorithm based on LMS algorithm
• Multi Layer Perceptron with Backpropagation is
universal aproximator

46. Backpropagation settings
• Max error
• Max iterations
• Learning rate
• Momentum
• Batch mode