Deep learning is the subset of Machine learning. In this neural network is designed with more than one hidden layer between input and output layer.

1. DEEP LEARNING
By
A. Yamini Saraswathi
Y18MTSP401
Under the guidance of
Mr. T. Krishna chaithanya (M.Tech)

2. Contents
• Artificial intelligence, machine learning
• What is deep learning
• computer vision features
• Fundamentals of deep learning
• Back propogation algorithm
• Convolutional neural network
• Recurrent neural network
• Deep learning framework
• Applications

3. Artificial intelligence
Artficial intelligence is the simulation of human intelligence
processes by machines especially computer systems. In this process
include learning, reasoning to reach to an approximate or accurate
output.

5. Continution....
Fig: Artificial neural network
Fig: Biological neuron

6. Continution....
Applications of artficial intelligence
• Autonomous vehicles(as drones and self driven cars)
• Medical diagnosis
• Playing games (such as chess)
• Online assistance (such as siri)
• Image recognition
• Speech recognition
• Natural language processing
• Etc......

7. Continution...
Learning of a neural netwwork can be of two types
• Supervised learning:-
It is used to perform classification when we want to map input
data to output labels. In this data is labeled with classes.
• Unsupervised learning:-
This learning is used to perform clustering and in this data is not
labeled. It is mostly used in exploratory analysis.

8. Machine learning
• Which is a subclass of artificial intelligence that automatially learn
and improve from the experience without being epxlicitly
programmed.
• Intelligent systems also built on machine learning algorithms which
have the capability to learn from the past experience.

9. Continution....
MACHINE
Trained data:
Rose
Tulips
Jasmine
Chrysanthymum
Input data
Feature
extraction
Chrysanthy
mum
Output

10. Limitations of machine learning
• High dimensionality of the data
• Unable to solve crucial AI problems
• One of the biggest challenge is feature extraciton
• Requires only significant amount of data

11. What is deep learning
Which is a subclass of
machine learning. In this neural
network is designed with more
than one hidden layer between
input and output layer. It reduces
the complexity of high dimensions
of the data.
Fig: Deep neural network

12. Computer vision features
• Camera obscura
• www.youtube.com/watch?v=10Hay06LJ4
• Black world
• SIFT feature
• Object recognition
• Etc......

13. Fundamentals of deep learning
Activation function:-
These are really important for artificial neural network to learn and
make sense of something really complicated and make non-linear complex
functional mapping between inputs and response variable. It should be
always non-linear and there are so many types
Sigmoid
Tanh
Softmax
Relu
Leaky Relu

14. Continution....
Sigmoid
It looks like an S- shaped curve. It's mathematical form is f(x) = 1 / 1 +
exp(-x) and its range is from 0 to 1 easy to understand and apply but it
has some major issues
Vanishing gradient
It's output is not zero centered
Sigmoids saturate and kill gradients
Sigmoids have slow convergence

15. Continution....
Tanh
It also mostly looked like S-shaped
curve and its range is from -1 to 1
and it's mathematical form
f(x) = 1-exp(-2x) / 1 + exp(-2x)
in this optimizations is easier. it's
major issue is
Vanishing gradient problem

16. Continution....
Relu
It has become more popular in last
couple of years. It was recently proved
that it had 6 times improvement in
convergence compared to Tanh
function it also solves vanishing
gradient problem there are two major
issues
• It can be used in hidden layers only
• Some gradients may die

17. Continution....
Leaky Relu
To fix the problem of dying neurons in the
Relu we will use Leaky Relu which is the
modification of Relu which introdues
a small slope to keep the updates alive

18. Loss function
• It is having an important part in ANN which is used to measure the
inconsistency between predicted value and actual value. These are
very helpful for training the neural network.
Y=Predicted output-actual output
• Many types of loss functions are present to deal with different types
of tasks but most widely used loss function is Mean Square
Error(MSE)

19. Optimization
Weights and bias are modifiedusing a function calledoptimization
function in such a way error is minimized. For updating weights we use an
equation
Optimazation algorithms are pf 2 types
• First order optmization algorithm
• Second order optimization algorithm

20. Optimzation algorithms
Gradient descent is the most popularly used optmization algorithm.
Which is of two types
• Stochastich gradient descent
θ=θ−η∇J(θ) where ∇J(θ) is the Gradient of Loss function J(θ) w.r.t
parameters 'θ’
• Mini batch gradient descent

21. Weight initialization
The method that we follow for weight initilalization is 'Xavier
intialization'. This method tries to intialize the weights with a smarter
value

22. Normalization
It is most important part in ANN because because most of the
data obtained from the real world are very distant each other. So to
keep the data at a comparable range we need to normalize the input
data of input layer. Common normalization approach includes.
• Statistical normalization
• Sigmoid normalization

23. Batch normalizaton
• Training deep neural network with more number of layers is more
challenging one to avoid this problem 'Batch normalization' is
introduced.
• It trains deep neural network that standardizes the inputs to a
layernfor each mini-batch and
Benifits of this batch normalization are
• Network train
• Makes weights easier to initialize
• Simplifies the creation of deeper network
• Makes more activation functions viable

24. Continution...Normalization
Batch normalization
Fig: Deep neural network

25. Regularization
When we build a neural network it performs well for trained data but
not on the test data. This is the sign of over fitting to reduce this
problem there are two ways
• Use more data
• Regularization
Getting more data is sometimes impossible and expoensive so for this
purpose we use 'Regularization'
So to solve this we should make our model generalize over the trained
data which is done using various regularization techniques.

26. Continuation...
Regularization techniques
• L1 & L2 regularization
Which updates general cost function by adding another term known as
regularization term
There is a small difference between L1 and L2 in the equation. Mostly L1 is
preferred

27. Continuation....
• Dropout regularization
This is mostly used one in the deep learning. At every iteration it randomly
selects some nodes and removes remaining with all of their incoming
signals and outgoing connections
So at each iteration has a different set of nodes
which result a different set of output
At last we ensemble models which usually perform
better than a normal neural network
Fig: Drop out Neural network

28. Back propogation algorithm
Neural network is trained by 'Back propogation algorithm' after
forwarding inputs to neural network then inputs and weights are summed
and multiplied then apply to activation function at last we predict the error
so to reduce this error we need to update weights using gradient descent
and bias which propogate backwwards to network

29. Convolutional neural network
In todays, convnets are used in every where it has been very effective
in such as image classification and image recognition it has been very
successful in identifying faces, objects and traffic signs
LeNet was one of the first convolutional neural network which helped
the path to deep learning
Convent is mainly build based on 3 layers
• Convolution layer & Non linearity
• Pooling layer
• Fully connected layer

30. Continution...
Fig: Convolutional neural network

31. Continution....
Convolution operation
Fig:Input image Fig: Kernal filter
Fig: Feature map
K=1

32. Continution....
Padding & pooling operation
Fig: Pooling operation
Fig: Padding operation

33. Continution....
Visulalisation of Convnet

34. Continuation...
Fig: Practical example of Convnet

35. Recurrent Neural Network
• RNN are also a feed forward neural network, however with ecurrent
meomory loops which takes input from the previous and /or same
layers of states
• This gives them a unique capability to model along the time
dimension and arbitary sequence of even inputs.
• RNN's are used for sequenced data analysis such as time series,
sentimental nalysis, NLP, language translation, speech recognition,
image captioning, script recognition and other othings.
• These are aslo calles networks with memory as the previous inputs or
states may persist in the model to do a sequential analysis. These
memories become input as well

36. Google Net
• It is almost different compared to other architectures like Alex Net
• It is the winner of Image Net 2014 and can train network faster which
is developed by Google
• In this global average pooling is used at the end of architecture it
leads to reduce number of weights are reduced and accuracy
increases to 0.6%

37. Continuation…

38. Deep Learning Framework
• Deep learning frameworks were built in academia in first generation Caffe
from Berkley university
Torch from NYU and also collaboration with Facebook
• In second generation all deep learning frameworks were developed from
industry's
Caffe2 from Facebook
Pytarch from Facebook
Tensor flow is from Facebook
MXNet from Amazon
CNTK form Microsoft

39. Importance of Deep learning frame work
• Easily can build big computational graphs
• Easily compute gradients in computational graphs
• Run it all efficiently n GPU

40. Continuation...
• This framework is an open source which aims to simplify the
implementation of compels and large scale deep learning models
using these amazing frameworks we can build complex models like
CNN. Some types of frameworks listed below.
Tensor flow
Keras
Pytorch
Caffe
Deeplearning4j
• One of the most popular frame work is 'Tensorflow'

41. Applications
• Automatic speech recognition
• Image recognition
• Visual art processing
• Natural language processing
• Drug discovery and toxicology
• Bioinformatics
• Medical Image Analysis
• Mobile advertising
• Image restoration
• Financial fraud detection
• Military

42. Google Arts & Culture app
This app uses automatic image analysis to compare lot of pictures in
the museums and in institutions throughout the world with your
picture. For this process mainly deep learning is mainly involved and in
the next slide I have taken the pictures practically.