The document outlines the syllabus for a deep learning course, covering historical trends, basics of machine learning, and various deep learning algorithms. It details early developments in neural networks, significant milestones, and various types of machine learning, including supervised, unsupervised, and reinforcement learning. Additionally, it emphasizes the importance of linear algebra in machine learning and provides insights into specific algorithms such as CNNs, LSTMs, and GANS.

1. 20CSE471A
DEEP LEARNING
N VIJAYA KUMAR M.E(CSE), M.B.A

2. SYLLABUS
• UNIT1:INTRODUCTION
• Historical Trends in Deep Learning–
• Machine Learning basics –
• earning algorithms –
• Linear Algebra for Machine Learning –
• Testing –
• Cross Validation –
• Dimensionality Reduction –
• fitting –
• Hyper parameters and validation sets –
• Estimators – Bias – Variance - Loss Function-
Regularization.

3. HISTORICAL TRENDS IN DEEP
LEARNING
• A Short History Of Deep Learning:-
• Deep learning is a topic that is making big
waves at the moment.
• It is basically a branch of machine learning
(another hot topic) that uses algorithms to
e.g. recognize objects and understand human
speech.

4. • Scientists have used deep learning algorithms
with multiple processing layers (hence
“deep”) to make better models from large
quantities of unlabelled data (such as photos
with no description, voice recordings or videos
on YouTube).
• It’s one kind of supervised machine learning,
in which a computer is provided a training set
of examples to learn a function, where each
example is a pair of an input and an output
from the function.

5. • Very simply: if we give the computer a picture
of a cat and a picture of a ball, and show it
which one is the cat, we can then ask it to
decide if subsequent pictures are cats. The
computer compares the image to its training
set and makes an answer. Today’s algorithms
can also do this unsupervised; that is, they
don’t need every decision to be pre-
programmed.

6. • Of course, the more complex the task, the
bigger the training set has to be. Google’s
voice recognition algorithms operate with a
massive training set — yet it’s not nearly big
enough to predict every possible word or
phrase or question you could put to it.

7. • But it’s getting there. Deep learning is
responsible for recent advances in computer
vision, speech recognition, natural language
processing, and audio recognition.
• Deep learning is based on the concept of
artificial neural networks, or computational
systems that mimic the way the human brain
functions. And so, our brief history of deep
learning must start with those neural
networks.

8. • 1943: Warren McCulloch and Walter Pitts create
a computational model for neural networks
based on mathematics and algorithms called
threshold logic.
• 1958: Frank Rosenblatt creates the perceptron,
an algorithm for pattern recognition based on a
two-layer computer neural network using simple
addition and subtraction. He also proposed
additional layers with mathematical notations,
but these wouldn’t be realized until 1975.
• .

9. • 1980: Kunihiko Fukushima proposes the
Neoconitron, a hierarchical, multilayered
artificial neural network that has been used
for handwriting recognition and other pattern
recognition problems.
• 1989: Scientists were able to create
algorithms that used deep neural networks,
but training times for the systems were
measured in days, making them impractical
for real-world use.

10. • 1992: Juyang Weng publishes Cresceptron, a
method for performing 3-D object recognition
automatically from cluttered scenes.
• Mid-2000s: The term “deep learning” begins to
gain popularity after a paper by Geoffrey Hinton
and Ruslan Salakhutdinov showed how a many-
layered neural network could be pre-trained one
layer at a time.
• 2009: NIPS Workshop on Deep Learning for
Speech Recognition discovers that with a large
enough data set, the neural networks don’t need
pre-training, and the error rates drop significantly.

11. • 2012: Artificial pattern-recognition algorithms
achieve human-level performance on certain
tasks. And Google’s deep learning
algorithm discovers cats.
• 2014: Google buys UK artificial intelligence startup
Deepmind for £400m
• 2015: Facebook puts deep learning technology –
called DeepFace – into operations to automatically
tag and identify Facebook users in photographs.
Algorithms perform superior face recognition tasks
using deep networks that take into account 120
million parameters.

12. • 2016: Google DeepMind’s algorithm AlphaGo
masters the art of the complex board game
Go and beats the professional go player Lee
Sedol at a highly publicized tournament in
Seoul

13. MACHINE LEARNING BASICS
What is Machine Learning?
• In 1959, Arthur Samuel, a computer scientist
who pioneered the study of artificial
intelligence, described machine learning as
“the study that gives computers the ability to
learn without being explicitly programmed.”

14. • Alan Turing’s seminal paper (Turing, 1950)
introduced a benchmark standard for
demonstrating machine intelligence, such that
a machine has to be intelligent and responsive
in a manner that cannot be differentiated from
that of a human being.
• Machine Learning is an application of artificial
intelligence where a computer/machine learns
from the past experiences (input data) and
makes future predictions. The performance of
such a system should be at least human level.

15. • A more technical definition given by Tom M.
Mitchell’s (1997) : “A computer program is
said to learn from experience E with respect
to some class of tasks T and performance
measure P, if its performance at tasks in T, as
measured by P, improves with experience E.”

16. • Example:
• A handwriting recognition learning problem:
Task T: recognizing and classifying
handwritten words within images
• Performance measure P: percent of words
correctly classified, accuracy
• Training experience E: a data-set of
handwritten words with given classifications
• In order to perform the task T, the system
learns from the data-set provided. A data-set
is a collection of many examples. An example
is a collection of features.

17. Machine Learning Categories
• Machine Learning is generally categorized into
three types:
• Supervised Learning,
• Unsupervised Learning,
• Reinforcement learning

18. • Supervised Learning: In supervised learning the
machine experiences the examples along with
the labels or targets for each example. The
labels in the data help the algorithm to
correlate the features.
• Two of the most common supervised machine
learning tasks is classification and regression.
• In classification problems the machine must
learn to predict discrete values. That is, the
machine must predict the most probable
category, class, or label for new examples.

19. • Applications of classification include predicting
whether a stock's price will rise or fall, or deciding if a
news article belongs to the politics or leisure section.
In regression problems the machine must predict the
value of a continuous response variable. Examples of
regression problems include predicting the sales for a
new product, or the salary for a job based on its
description. Unsupervised Learning: When we have
unclassified and unlabeled data, the system attempts
to uncover patterns from the data. There is no label
or target given for the examples. One common task is
to group similar examples together called clustering.

20. • Reinforcement Learning: Reinforcement
learning refers to goal-oriented algorithms,
which learn how to attain a complex objective
(goal) or maximize along a particular
dimension over many steps. This method
allows machines and software agents to
automatically determine the ideal behavior
within a specific context in order to maximize
its performance. Simple reward feedback is
required for the agent to learn which action is
best; this is known as the reinforcement signal.
For example, maximize the points won in a
game over many moves.

21. DEEP LEARNING ALGORITHMS
• 1. Convolutional Neural Networks (CNNs)
• 2. Long Short Term Memory Networks (LSTMs)
• 3. Recurrent Neural Networks (RNNs)
• 4. Generative Adversarial Networks (GANs)
• 5. Radial Basis Function Networks (RBFNs)
• 6. Multilayer Perceptrons (MLPs)
• 7. Self Organizing Maps (SOMs)
• 8. Deep Belief Networks (DBNs)
• 9. Restricted Boltzmann Machines (RBMs)

22. 1. Convolutional Neural Networks
(CNNs)
• CNN's popularly known as ConvNets majorly
consists of several layers and are specifically
used for image processing and detection of
objects.
• It was developed in 1998 by Yann LeCun and
was first called LeNet.

23. • Back then, it was developed to recognize
digits and zip code characters.
• CNNs have wide usage in identifying the
image of the satellites, medical image
processing, series forecasting, and anomaly
detection.

24. 2. Long Short Term Memory
Networks (LSTMs)
• LSTMs can be defined as Recurrent Neural
Networks (RNN) that are programmed to learn and
adapt for dependencies for the long term.
• It can memorize and recall past data for a greater
period and by default, it is its sole behavior.
• LSTMs are designed to retain over time and
henceforth they are majorly used in time series
predictions because they can restrain memory or
previous inputs.

25. 3. Recurrent Neural Networks
(RNNs)
• Recurrent Neural Networks or RNNs consist of
some directed connections that form a cycle
that allow the input provided from the LSTMs
to be used as input in the current phase of
RNNs.
• These inputs are deeply embedded as inputs
and enforce the memorization ability of LSTMs
lets these inputs get absorbed for a period in
the internal memory.

26. • RNNs are therefore dependent on the inputs
that are preserved by LSTMs and work under
the synchronization phenomenon of LSTMs.
• RNNs are mostly used in captioning the image,
time series analysis, recognizing handwritten
data, and translating data to machines.

27. 4. Generative Adversarial
Networks (GANs)
• GANs are defined as deep learning algorithms
that are used to generate new instances of
data that match the training data.
• GAN usually consists of two components
namely a generator that learns to generate
false data and a discriminator that adapts
itself by learning from this false data

28. • Over some time, GANs have gained immense
usage since they are frequently being used to
clarify astronomical images and
simulate lensing the gravitational dark matter.

29. 5. Radial Basis Function Networks
(RBFNs)
• RBFNs are specific types of neural networks
that follow a feed-forward approach and
make use of radial functions as activation
functions.
• They consist of three layers namely the input
layer, hidden layer, and output layer which
are mostly used for time-series prediction,
regression testing, and classification.

30. 6. Multilayer Perceptrons (MLPs)
• MLPs are the base of deep learning
technology. It belongs to a class of feed-
forward neural networks having various layers
of perceptrons.
• These perceptrons have various activation
functions in them.
• MLPs also have connected input and output
layers and their number is the same.

31. • Also, there's a layer that remains hidden
amidst these two layers. MLPs are mostly
used to build image and speech
recognition systems or some other types of
the translation software.

32. 7. Self Organizing Maps (SOMs)
• SOMs were invented by Teuvo Kohenen for
achieving data visualization to understand the
dimensions of data through artificial and self-
organizing neural networks.
• The attempts to achieve data visualization to
solve problems are mainly done by what
humans cannot visualize.
• These data are generally high-dimensional so
there are lesser chances of human
involvement and of course less error.

33. 8. Deep Belief Networks (DBNs)
• DBNs are called generative models because they
have various layers of latent as well as stochastic
variables.
• The latent variable is called a hidden
unit because they have binary values.
• DBNs are also called Boltzmann
Machines because the RGM layers are stacked
over each other to establish communication with
previous and consecutive layers.

34. • DBNs are used in applications like video and
image recognition as well as capturing
motional objects.

35. 9. Restricted Boltzmann Machines
(RBMs)
• RBMs were developed by Geoffrey Hinton and
resemble stochastic neural networks that learn
from the probability distribution in the given input
set.
• This algorithm is mainly used in the field of
dimension reduction, regression and classification,
topic modeling and are considered the building
blocks of DBNs. RBIs consist of two layers namely
the visible layer and the hidden layer.

36. Linear algebra for Machine
Learning
• Machine learning has a strong connection
with mathematics.
• Each machine learning algorithm is based on
the concepts of mathematics & also with the
help of mathematics.
• one can choose the correct algorithm by
considering training time, complexity, number
of features, etc.

37. • Linear Algebra is an essential field of
mathematics, which defines the study of
vectors, matrices, planes, mapping, and lines
required for linear transformation.
• The term Linear Algebra was initially
introduced in the early 18th
century to find out
the unknowns in Linear equations and solve
the equation easily.

38. • hence it is an important branch of
mathematics that helps study data.
• Also, no one can deny that Linear Algebra is
undoubtedly the important and primary thing
to process the applications of Machine
Learning. It is also a prerequisite to start
learning Machine Learning and data science.

39. • Linear algebra plays a vital role and key
foundation in machine learning, and it
enables ML algorithms to run on a huge
number of datasets.
• The concepts of linear algebra are widely used
in developing algorithms in machine learning.
Although it is used almost in each concept of
Machine learning, specifically, it can perform
the following task:

40. • Optimization of data.
• Applicable in loss functions, regularization,
covariance matrices, Singular Value
Decomposition (SVD), Matrix Operations,
and support vector machine classification.
• Implementation of Linear Regression in
Machine Learning.

41. Why learn Linear Algebra before
learning Machine Learning?
• Linear Algebra is just similar to the flour of
bakery in Machine Learning.
• As the cake is based on flour similarly, every
Machine Learning Model is also based on Linear
Algebra.
• Further, the cake also needs more ingredients
like egg, sugar, cream, soda. Similarly, Machine
Learning also requires more concepts as vector
calculus, probability, and optimization theory.

42. some popular examples of linear
algebra in Machine learning:
• Datasets and Data Files
• Linear Regression
• Recommender Systems
• One-hot encoding
• Regularization
• Principal Component Analysis
• Images and Photographs
• Singular-Value Decomposition
• Deep Learning
• Latent Semantic Analysis

43. benefits of learning Linear Algebra
before Machine learning
• Better Graphic experience
• Improved Statistics
• Creating better Machine Learning algorithms
• Estimating the forecast of Machine Learning
• Easy to Learn

44. • Notation: Notation in linear algebra enables
you to read algorithm descriptions in papers,
books, and websites to understand the
algorithm's working. Even if you use for-loops
rather than matrix operations, you will be able
to piece things together.

45. • Operations: Working with an advanced level
of abstractions in vectors and matrices can
make concepts clearer, and it can also help in
the description, coding, and even thinking
capability. In linear algebra, it is required to
learn the basic operations such as addition,
multiplication, inversion, transposing of
matrices, vectors, etc.

46. • Matrix Factorization: One of the most
recommended areas of linear algebra is matrix
factorization, specifically matrix deposition
methods such as SVD and QR.