Neural Network Learning Algorithm

1. Name Ammar Muhammad
Subject Intelligent Control
Application
Topic Neural Network &
Learning Algorithm
Professor Chan Zhou

2. What is the Artificial Neural
Network?
• An Artificial Neural Network (ANN) is a computational model inspired
by the structure and function of the human brain's neural networks.
• It resembles the brain in two respects.
 It comprises interconnected nodes (neurons) that process information, learn
patterns from data.
 Make predictions or classifications by adjusting connections (synaptic weights )
between nodes.
• ANN are considered nonlinear statistical data modeling tools
where complex relationships between inputs and outputs are
modeled or patterns are found.

3. Biological Inspiration
• The Brain is a massively parallel information processing system.
• Our brains are a huge network of processing elements.
• A typical brain contains a network of 10 billion neurons.

4. CNS- Brain and Neuron
• The brain is highly complex nonlinear and massively parallel
computing machine.

5. Motivation behind Artificial
Neural Network
A neuron is a basic unit of the brain processes and
transmitting the information.
• Dendrite: Receive signal from
other neurons.
• Soma(Cell body): sums all the
inputs.
• Axon: it is used to transmit the
electric signal to other
neurons.
• Synaptic Terminal: Release
the neurotransmitter to
transmit information to
dendrites.

6. Historical Perception

7. Association with the Biological
neurons and Artificial neurons.
Inputs
Output
Liner / nonlinear model
BNN ANN
Soma Node
Dendrites Inputs
Synapse Weight or
interconnection
/Link
Axon Output

8. Perceptron
• A perceptron is a most fundamental unit of neural network (ANN)
that does certain computations to detect feature or business
intelligence in the input data.
• The perceptron is linear model for the supervised learning used
for the binary classification,
• A perceptron is a most fundamental unit of neural network (ANN)
that does certain computations to detect feature or business
intelligence in the input data.
• The perceptron is linear model for the supervised learning used
for the binary classification,
• Perceptron consist of 4 parts
• Inputs
• weights & bias
• summation function Activation function
• Output
• Perceptron learning rule.
• Perceptron learns the weight for the input single sin order to draw a
linear decision boundary.
Types of perceptron
• Single layer perceptron
• Multi layer perceptron

9. Activation Function
The function AN receives the net input signal and bias and determines the output of
the neuron. This function is referred as the activation function.

10. Network Architecture
Network Architecture
Single Layer ANN
Architecture
Feedforward ANN
Architectures
Unsupervised
(Kohonen)
Supervised
(MLP, RBF)
Recurrent ANN
Architectures
Unsupervised
(ART)
Supervised
(Elman, Jordan,
Hopfield)
• The way the neurons of a neural networks are structured is intimately
linked with the learning algorithm used to train the network.
• Generally, three fundamentally different classes of the network
architecture.

11. Single Layer ANN Architecture
• It is one of the oldest and first introduced neural networks.
• It was proposed by Frank Rosenblatt in 1958. Perceptron is also known as an artificial neural
network.
• Perceptron is mainly used to compute the logical gate like AND, OR, and XOR which has binary
input and binary output.

12. Feedforward ANN Architectures
• It is also known as the Multi- Layered
Neural Network.
• A multi-layer perceptron has one input
layer and for each input, there is one
neuron(or node), it has one output
layer with a single node for each
output and it can have any number of
hidden layers and each hidden layer
can have any number of nodes.
• Capable of handling the non-linearly
separable data.

13. Recurrent ANN Architectures
• Recurrent Neural Network(RNN) is a type of Neural Network where the output from
the previous step is fed as input to the current step.
• In traditional neural networks, all the inputs and outputs are independent of each
other.
• Two types of recurrent network
• Without hidden layers
• With Hidden layers

14. ANN Learning Techniques
 Depending on types and characteristics of the encoding and recall
process, ANN learning techniques are categorized as
 Supervised learning
 Un supervised learning
 Reinforcement learning
 Competitive learning

15. Supervised Learning
• In the supervise learning system there is a trainer(teacher) who provides the input
and the corresponding target (output ) patterns.
• A learning algorithm is employed to determine a unique set of the network
parameters that jointly satisfy the input–output interrelationship of patterns(encoding)
• Then excitation with an unknown input pattern can generate its corresponding output
pattern.
• ANN trained by supervised learning algorithm behaves like a multi-input-multi-output
function approximator.
• There are many supervised learning using ANN.
• Most popular is back propagation algorithm.

16. Unsupervised Learning
• An unsupervised learning system employs no teacher.
• Interrelation among patterns is not known.
• One or more input patterns is automatically mapped to one pattern cluster.
• Most systems employ recursive leaning rule that automatically adjusts the
network parameters for attaining some criteria like minimization f the
network energy states.
• Imagine you have a folder filled with various pictures from a trip, but these
images aren't categorized or labeled. You want to organize them based on
similarities without manually labeling each photo.
• Hopfield nets and associative memory are most popular unsupervised
learning system.

17. Reinforcement Learning
• Reinforcement Learning bridges the gap between the supervised and unsupervised
learning.
• Learning scheme employs internal critic that examines the response of the
environment in turn of the action of the learning system on the environment.
• If the response is in favor of the goal, then action is rewarded otherwise it is
penalized.
• Determination of the status of the action: reward or penalty may require quite a long
time.
• Imagine you're teaching a dog a new trick using reinforcement learning concepts: like
fetching a ball.
• Q-learning is the most common reinforcement learning technique.

18. Competitive learning
• In Competitive learning scheme neurons compete with one another to
satisfy the given goal.
• The output neurons compete among themselves to become active.
• Only as single output neuron is active at any one time.
• The neuron that wins the competition is called winner takes all neurons.
• Imagine you have a basket of fruits (apples, oranges, and bananas) that
vary in color, and you want to sort them into different bins based on their
colors using a competitive learning approach:
• You have three bins labeled "Red," "Orange," and "Yellow," representing the
colors of fruits (like apples, oranges, and bananas).

19. Hebb’s Postulate of Learning
• Donald Hebb the origination of the behavior (1949)
• Hebb’s Postulate “ When an axon of cell A is near enough to excite a cell B and
repeatedly or presently takes part in firing it, some growth process or metabolic
change takes place in one or both cell such that A’s efficiency as one pf the cell firing
B is increased.
• Mathematically,
• Where xi is the input and y is the output.
• Bipolar input or output (-1 or +1)
• Limitation –can classify linearly separable patterns only.

21. Advantages / Disadvantages
 Advantages
 Adapt to unknown situations
 Powerful, it can model complex functions.
 Ease of use, learns by example, and very little user
domain‐specific expertise needed
 Disadvantages
 Forgets
 Not exact
 Large complexity of the network structure

22. Conclusion
 Artificial Neural Networks are an imitation of the
biological neural networks, but much simpler ones.
 The computing would have a lot to gain from neural
networks.
 Their ability to learn by example makes them very
flexible and powerful furthermore there is need to device
an algorithm to perform a specific task.