595

1. Biological Neuron Artificial Neuron
• Bio-ANN
[https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligen
ce_neural_networks.htm] [https://ujjwalkarn.me/2016/08/09/quick-intro-
neural-networks/]
• Activation functions [https://en.wikipedia.org/wiki/Activation_function]
• Layer of Neurons []
• Role of Bias- [https://stackoverflow.com/questions/2480650/role-of-bias-
in-neural-networks]
• McCulloch Pitts Model (unsup)
[https://machinelearningknowledge.ai/mcculloch-pitts-neuron-model/]-
• Perceptron (sup) []
• Learning: Supervised/Unsupervised/Reinforcement
[https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligen
ce_neural_networks.htm]
• Applications of Neural Network
• ANN learning methods
• Desirable properties of ANN- stability, plasticity
• Introduction to Back Propagation Networks,

2. Biological NN Vs ANN
• Characteristic abilities of Biological
neural systems
– pattern recognition
– perception
– motor control
– Memorize
– Learn
– Generalize
• Components
– Neurons - basic building blocks of
biological neural systems are nerve
cells, referred to as
– Synapses – interconnection between
the axon of one neuron and a
dendrite of another neuron
• Algorithmic models of the features of
biological neural systems are called
“artificial neural networks (ANN)”
Order of 10-500 billion neurons in the human cortex, with 60 trillion synapses.
Arranged in approximately 1000 main modules, each with 500 neural networks.

3. Types of Learning in ANN
• FeedForward ANN
– the information flow is
unidirectional.
– A unit sends information
to other unit from which
it does not receive any
information.
– There are no feedback
loops.
– Application: pattern
generation/recognition/cl
assification.
• Feedback ANN
– feedback loops are
allowed.
– Application: content
addressable memories

4. More Types of ANNs
• Single-layer NNs, such as the Hopfield network;
• Multilayer feedforward NNs, including, for example, standard
backpropagation, functional link and product unit networks;
• Temporal NNs, such as the Elman and Jordan simple recurrent
networks as well as time-delay neural networks;
• Self-organizing NNs, such as the Kohonen self-organizing
feature maps and the learning vector quantizer;
• Combined feedforward and self-organizing NNs, such as the
radial basis function networks.

5. Single Neuron
• Components
– X1, X2: Numerical Input
– f is non-linear and is called the Activation Function - takes a single
number and performs a certain fixed mathematical operation on it
– 1: Bias with weight b .

6. Activation Fuction
[Ref- Engelbrecht Andries P., Computational Intelligence: An Introduction, Wiley]
Linear : produces a linearly
modulated output, where
ß is a constant.
f (net -)= ß(net - )
Step: takes a real-valued
input and squashes it to
the range [ ß1, ß2], binary
or bipolar.
f (net -)= ß1 if (net ≥ )
= ß2 if (net <)
Ramp: is a combination of the
linear and step functions
f (net -) = ß if (net- ≥ ß)
= net - if |net -|<ß
= -ß if (net- < ß)

7. Sigmoid: takes a real-
valued input and squashes
it to range (0, 1).
ß controls the steepness.
tanh: takes a real-valued
input and squashes it to
the range [-1, 1].
ß controls the steepness.
ReLU (Rectified Linear Unit):
takes a real-valued input
and thresholds it at zero
(replaces negative values
with zero)
f(x) = max(0, x)
Activation Fuction
[Ref- Engelbrecht Andries P., Computational Intelligence: An Introduction, Wiley]

8. Layers of Neurons: (e.g. in Feedforward NN)
• Input nodes
– No computation is performed in any of the Input nodes
– They just pass on the information to the hidden nodes
• Hidden nodes
– They perform computations and transfer information from the input
nodes to the output nodes.
– There can be zero or multiple hidden layers
• Output nodes
– Responsible for computations and transferring information from the
network to the outside world
– One output node for one decision parameter

9. McCulloch-Pitts-neuron
• First ever primitive model of biological neuron was conceptualized by Warren
Sturgis McCulloch and Walter Harry Pitts in 1943
• Elements-
– Neuron:computational in which the input signals are computed and an output is fired
• Summation Function- This simply calculates the sum of incoming inputs(excitatory).
• Activation Function - Essentially this is the step function which sees if the summation
is more than equal to a preset Threshold value , if yes then neuron should fire (i.e.
output =1 ) if not the neuron should not fire (i.e. output =0).
– Neuron fires: Output =1 , if Summation >= Threshold
– Neuron does not fires: Output =0 , if Summation < Threshold
– Excitatory Input : This is an incoming binary signals to neuron, which can have only two
values 0 (=OFF) or 1 (=ON)
– Inhibitory Input : If this input is on, this will now allow neuron to fire , even if there are
other excitatory inputs which are on.
– Output : The value of 0 indicates that the neuron does not fire, the value of 1 indicates
the neuron does fire.

10. Function of McCulloch-Pitts Model
• Design-
– McCulloch-Pitts neuron model can be used to compute some
simple functions which involves binary input and output.
• Steps -
– The input signals are switched on and the neuron is activated.
– If Neuron detects that Inhibitory input is switched on, the
output is straightaway zero, which means the neuron does not
fire.
– If there is no Inhibitory input, then neuron proceeds to calculate
the sum of number of excitatory inputs that are switched on.
– If this sum is greater than equal to the preset threshold value,
the neuron fires (output=1) , otherwise the neuron does not fire
(output=0)

11. Illustration of McCulloch-Pitts Model
https://machinelearningknowledge.ai/mcculloch-pitts-neuron-model/

12. Illustration of McCulloch-Pitts Model

13. Design of McCulloch-Pitts Neuron
for AND Function
• For the neuron to fire, both excitatory input signals have to be
enabled.
• So it is very intuitive that the threshold value should be 2 .
• Additionally if the inhibitory input is on, then irrespective of
any other input, the neuron will not fire.
https://machinelearningknowledge.ai/mcculloch-pitts-neuron-model/

14. Design of McCulloch-Pitts Neuron
for OR Function
• For the neuron to fire, at least 1 excitatory input signals has to
be enabled.
• So it is very intuitive that the threshold value should be 1 .
• Additionally if the inhibitory input is on, then irrespective of
any other input, the neuron will not fire.
https://machinelearningknowledge.ai/mcculloch-pitts-neuron-model/

15. Design of McCulloch-Pitts Neuron
for Real Life Decision Making
• Problem - You like going to for a particular movie if it is a new
release. But you watch a movie if the ticket price is cheap. Further
you cant plan movie on your weekdays as they are busy
• Design
– Excitatory Inputs
• X1- IsMovieNew
• X2- IsTicketCheap
– Output Function : AND (since Ouput is 1 only if both X1 and X2 are 1)
– Inhibitory
• IsWeekday : If it is on for the neuron, then there is no question of planning for the movie.
https://machinelearningknowledge.ai/mcculloch-pitts-neuron-model/

16. Limitation of McCulloch-Pitts Model
• There is No (machine) learning in this model
• This model was not built to work as machine
learning model in the first place.
• Rather McCulloch and Pitts just wanted to
build a mathematical model to represent the
workings of biological neuron.
• But this humble looking model actually
inspired other researchers to come up with
true machine learning based neural models in
the later years

17. Learning Methods
• Supervised Learning
– The model is trained using examples of expected output values for
each input combination.
– For example, pattern recognizing. The ANN comes up with a guess for
given input vector, then compares the guess with the corresponding
“correct” output value and makes adjustments in weights according to
errors.
• Unsupervised Learning
– It is required when there is no example data set with known answers.
– For example, searching for a hidden pattern. In this case, clustering
i.e. dividing a set of elements into groups according to some unknown
pattern is carried out based on the existing data sets present.
• Reinforcement Learning
– This strategy is built on observation.
– In this method, the ANN makes a decision by observing its
environment. If the observation is negative, the network adjusts its
weights to be able to make a different required decision the next time.

18. Applications of ANN
• Classification - where the aim is to predict the class of an input
vector;
• Pattern matching - where the aim is to produce a pattern best
associated with a given input vector;
• Pattern completion - where the aim is to complete the missing parts
of a given input vector;
• Optimization-where the aim is to find the optimal values of
parameters in an optimization problem;
• Control - where, given an input vector, an appropriate action is
suggested;
• Function approximation/times series modeling - where the aim is
to learn the functional relationships between input and desired
output vectors;
• Data mining - with the aim of discovering hidden patterns from
data – also referred to as knowledge discovery.