This presentation covers the basics of neural network along with the back propagation training algorithm and a code for image classification at the end.

1. Neural Networks
• Contents :-
• Why to study Neural Network
• What are Neural Network
• Brain Inspired Neural Network
• Simple Neuron Model
• Neural Network Model
• Neural Network for Boolean function
• Neural Network for Non Linear function
• Neural Network for Multiclass classification
MOHIT SHRIVASTAVA
DEPARTMENT OF COMPUTER SCIENCE AND
ENGINEERING
NIT WARANGAL

2. Why to study
Neural
Network
• Neural Network are used in problems where we need to learn complex non-linear
hypotheses.
• These non-linear hypotheses requires both higher degree and feature multiplicative
terms.

3. Why to study Neural Network
• The decision boundry for
the nonlinear classification
problem is complex and
can't be represented by
simple linear hypothesis.
• For such problem we fit a
logistic regression and it
will have both higher
degree and feature
multiplicative terms.

4. Why to study Neural Network
• If the input for such
classification is a image then
consider a small image of
100*100 pixel so 10000
features.
• If we fit a nonlinear
regression the number of
multiplicative feature terms
will increase to (n*(n+1))/2
• In this case it rises to 50
millions that is huge and not
efficient.
• Neural network helps us to
solve such problems

5. What are Neural Network
• Origins: Algorithms that try to mimic the brain.
• Very widely used in 80s and early 90s; popularity diminished in late 90s.
• Recent resurgence: State-of-the-art technique for many applica1ons
• Artificial neural networks are not nearly as complex or intricate as the actual brain
structure

6. What are
Neural
Network

7. Brain Inspired Neural
Network
• Brain function occurs as the result of the firing of
neurons
• Neurons connect to each other through synapses,
which propagate action potential (electrical impulses)
by releasing neurotransmitters
• Synapses can be excitatory (potential-increasing) or
inhibitory (potential-decreasing), and have varying
thresholds.
• Dinitrides receives the input.
• Axon gives the output.

8. Brain Inspired Neural
Network
• Synapses in neural network are weights between
layers.
• Thresholds of synapses in brain are activation
function in neural network.
• Axon and Dendrites are layer's input and output in
neural network.

9. Simple
Neuron
Model
Theta: Weights
X : Input with bias
Yellow circle: Activation of neuron
Each input is multiplied by weight
and summed up and then activation
function is applied here that is
Sigmoid function

10. Neural
Network
Model
A neural network with 3 layers

12. Neural
Network for
Boolean
function
A neuron represents the AND
Boolean function using sigmoid
activation function.

13. Neural
Network for
Boolean
function
Similar model like AND for OR
NOT using sigmoid activation
function.

14. Neural Network for
Non Linear function
• Combining Representations to
Create Non-Linear Functions like
XOR by adjusting weights.
• Here we are solving the problem of
more features that we had in
logistic regression by having more
layers into our network.

15. Neural Network for multiclass classification
• Till now we had hypothesis value as either
0 or 1 or it’s a binary classification
• Now we try to use neural network with
multiclass classification where there are
more than one classes

16. Neural Network for multiclass classification
• Number of input unit = number of
features
• Number of output unit = number of
classes
• Here for image classification number of
classes are 4 so 4 neuron in output layer

17. Neural Networks with backpropagation
algorithm
• Contents :-
• What is backporpagation algorithim
• Flow in backprop

18. What is backpropagation algorithm
• Backpropagation was derived by multiple researchers in the early 60's
• Paul Werbos was first in the US to propose that it could be used for
neural nets after analyzing it in depth in his 1974 PhD Thesis.
• In 1986, through the work of David E. Rumelhart, Geoffrey E.
Hinton, Ronald J. Williams, and James McClelland, backpropagation
gained recognition.
• Backpropagation is a training method used for a multi-layer neural
network.
• It is also called the generalized delta rule.

19. Flow in backpropagation algorithm
The training of a neural network by back propagation takes place in
three stages
1. Feedforward of the input pattern
2. Calculation and Back propagation of the associated error
3. Adjustments of the weights

20. Forward propagation
• Initially activation of
input layer is the
features only
• Z is the weighted sum of
activation and weights
• Note- bias are also
present.

21. Cost function
• It is generalization of
the logistic regression
cost function
• K is the no. Of neu in
output layer
• We just add up the cost
of each neuron in
output layer.
• At each iteration the
cost depresses.
• Note- regularization
term is also present.

22. Backpropagation:
Error Computation
• Each hidden node j is “responsible”
for some fraction of the error δ (l) in
each of the output nodes j to which
it connects
• δ (l) is divided according to the
strength of the j connection
between hidden node and the
output node
• Then, the “blame” is propagated
back to provide the error values for
the hidden layer

23. Training a Neural
Network via Gradient
Descent with Backprop
• Green box is the backprop
component.
• Computation of average regularized
gradient is optional.
• Note- We never initialize weights in
neural net as 0

24. Backpropagation for image classification
• We will now demonstrate how to use backpropagation for image classification
• We will use the Olivetti faces dataset.
• This dataset contains a set of face images taken between April 1992 and April 1994 at AT&T
Laboratories Cambridge.
• There are ten different images of each of 40 distinct subjects.
• For some subjects, the images were taken at different times, varying the lighting, facial expressions
(open / closed eyes, smiling / not smiling) and facial details (glasses / no glasses).
• All the images were taken against a dark homogeneous background with the subjects in an upright,
frontal position (with tolerance for some side movement).

25. Downloading the dataset and printing the
data.

26. Description
• Each image is a row in data along with its label.
Upper left corner shows the label and lower left shows the data item
number.

27. Multi-layer Perceptron classifier
• We will use classifier from sklearn scientific library
• Available under module sklearn.neural_network.MLPClassifier
• This model optimizes the cost function using LBFGS or stochastic gradient descent

28. Code:-