This document provides an overview of various deep learning techniques including recurrent neural networks, convolutional neural networks, the universal approximation theorem, and generative adversarial networks. It describes what each technique is used for as well as key aspects of how they work, such as RNNs using sequential data and CNNs being applied to visual imagery. The document also discusses regularization techniques used in CNNs and implications of the universal approximation theorem.

1. Unit 3
BY:SURBHI SAROHA

2.  Deep Learning
 Recurrent Neural Networks
 Convolutional Neural Networks
 The UniversalApproximationTheorem
 GenerativeAdversarial Networks

3.  Deep learning is a computational technique that
uses the often hidden information contained in
vast datasets to solve questions of interest.
 It’s been used widely in fields such as games,
speech and voice recognition, autonomous cars,
science and medicine.
 Deep learning (also known as deep structured
learning) is part of a broader family of machine
learning methods based on artificial neural
networks with representation learning.
 Learning can be supervised, semi-
supervised or unsupervised.

4.  Deep-learning architectures such as deep neural
networks, deep belief networks, recurrent neural
networks and convolutional neural networks have
been applied to fields including computer
vision, machine vision, speech recognition, natural
language processing, audio recognition, social
network filtering, machine
translation, bioinformatics, drug design, medical
image analysis, material inspection and board
game programs, where they have produced results
comparable to and in some cases surpassing human
expert performance.

5.  A recurrent neural network (RNN) is a type of artificial
neural network which uses sequential data or time series
data.
 These deep learning algorithms are commonly used for
ordinal or temporal problems, such as language
translation, natural language processing (nlp), speech
recognition, and image captioning; they are incorporated
into popular applications such as Siri, voice search, and
GoogleTranslate.
 Like feedforward and convolutional neural networks
(CNNs), recurrent neural networks utilize training data to
learn.They are distinguished by their “memory” as they
take information from prior inputs to influence the current
input and output.

6.  While traditional deep neural networks
assume that inputs and outputs are
independent of each other, the output of
recurrent neural networks depend on the
prior elements within the sequence.
 While future events would also be helpful in
determining the output of a given sequence,
unidirectional recurrent neural networks
cannot account for these events in their
predictions.

7.  In deep learning, a convolutional neural
network (CNN, or ConvNet) is a class of deep neural
networks, most commonly applied to analyzing visual
imagery.
 They are also known as shift invariant or space
invariant artificial neural networks (SIANN), based
on their shared-weights architecture and translation
invariance characteristics.
 They have applications in image and video
recognition, recommender systems, image
classification, medical image analysis, natural
language processing, brain-computer interfaces, and
financial time series.

8.  CNNs are regularized versions of multilayer perceptrons.
Multilayer perceptrons usually mean fully connected
networks, that is, each neuron in one layer is connected to
all neurons in the next layer.
 The "fully-connectedness" of these networks makes them
prone to overfitting data.
 Typical ways of regularization include adding some form
of magnitude measurement of weights to the loss
function.
 CNNs take a different approach towards regularization:
they take advantage of the hierarchical pattern in data and
assemble more complex patterns using smaller and
simpler patterns.Therefore, on the scale of connectedness
and complexity, CNNs are on the lower extreme.

9.  In the mathematical theory of artificial neural networks, universal
approximation theorems are results that establish the density of
an algorithmically generated class of functions within a given
function space of interest.
 Typically, these results concern the approximation capabilities of
the feedforward architecture on the space of continuous functions
between two Euclidean spaces, and the approximation is with
respect to the compact convergence topology.
 However, there are also a variety of results between non-
Euclidean spaces and other commonly used architectures and,
more generally, algorithmically generated sets of functions, such
as the convolutional neural network (CNN) architecture, radial
basis-functions, or neural networks with specific properties

10.  Most universal approximation theorems can be parsed
into two classes.
 The first quantifies the approximation capabilities of
neural networks with an arbitrary number of artificial
neurons ("arbitrary width" case) and the second focuses on
the case with an arbitrary number of hidden layers, each
containing a limited number of artificial neurons
("arbitrary depth" case).
 Universal approximation theorems imply that neural
networks can represent a wide variety of interesting
functions when given appropriate weights.
 On the other hand, they typically do not provide a
construction for the weights, but merely state that such a
construction is possible.

11.  A generative adversarial network ( GAN) is a
class of machine learning frameworks designed
by Ian Goodfellow and his colleagues in 2014.
 Two neural networks contest with each other in
a game (in the form of a zero-sum game, where
one agent's gain is another agent's loss).
 Given a training set, this technique learns to
generate new data with the same statistics as
the training set.

12.  The core idea of a GAN is based on the
"indirect" training through the discriminator,
which itself is also being updated
dynamically.
 This basically means that the generator is not
trained to minimize the distance to a specific
image, but rather to fool the discriminator.
 This enables the model to learn in an
unsupervised manner.