The document outlines the history and key concepts of deep learning, including milestones in neural networks from their inception in the 1940s to advancements like convolutional and generative adversarial networks. It discusses various learning techniques such as backpropagation, regularization, and semi-supervised learning, emphasizing their significance in model training and performance. Additionally, it highlights the applications of deep learning in fields like natural language processing, computer vision, and medical imaging.

1. DEEP LEARNING
(UNIT 2)
BY: SURBHI SAROHA

2. SYLLABUS
DEEP NETWORKS :
History of Deep Learning- A Probabilistic Theory of Deep Learning
Backpropagation and regularization,
batch normalization- VC Dimension and Neural Nets-Deep Vs Shallow Networks-Convolutional
Networks- Generative Adversarial Networks (GAN),
Semi supervised Learning

3. History of Deep Learning- A Probabilistic
Theory of Deep Learning
Deep learning, a subset of machine learning, has a rich history that dates back several decades.
Here are some key milestones:
1940s-1950s: Early Neural Networks
◦ 1943: Warren McCulloch and Walter Pitts introduced the concept of artificial neurons, the basic units of
neural networks.
◦ 1950s: Frank Rosenblatt developed the Perceptron, an early neural network model capable of learning
from data.
1980s: Backpropagation and Multi-Layer Perceptrons
◦ 1986: Geoffrey Hinton, David Rumelhart, and Ronald Williams popularized the backpropagation
algorithm, which allowed multi-layer perceptrons to learn from data more effectively.

4. Cont…..
The backpropagation algorithm enabled the training of neural networks with multiple hidden
layers, marking a significant breakthrough.
1990s-2000s: Convolutional and Recurrent Neural Networks
1998: Yann LeCun developed LeNet, a convolutional neural network (CNN) for digit recognition,
demonstrating the potential of CNNs in image processing tasks.
1997: Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory (LSTM)
networks, a type of recurrent neural network (RNN) capable of learning long-term
dependencies, crucial for tasks like speech recognition and language modeling.
2010s: Deep Learning Renaissance

5. Cont…..
2012: AlexNet, a deep CNN developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton,
won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), significantly
outperforming traditional methods.
This success sparked widespread interest in deep learning and led to the development of many
more advanced architectures like VGG, GoogLeNet, ResNet, and more.
Late 2010s-Present: Advances and Applications
Transformers and attention mechanisms, introduced in 2017 by Vaswani et al., revolutionized
natural language processing (NLP) and led to state-of-the-art models like BERT, GPT, and T5.

6. A Probabilistic Theory of Deep Learning
A probabilistic theory of deep learning aims to provide a theoretical framework to understand
how deep learning models generalize from data. Key concepts include:
Bayesian Inference
◦ Bayesian methods provide a probabilistic framework for learning from data by combining prior
knowledge with observed data to update beliefs.
◦ In deep learning, Bayesian inference can be used to estimate model parameters and uncertainty,
leading to more robust predictions.
Variational Inference
Variational inference is a technique for approximating complex probability distributions in a
computationally efficient manner.

7. Cont….
In deep learning, variational autoencoders (VAEs) use variational inference to learn latent
representations of data, facilitating tasks like generation and anomaly detection.
Stochastic Gradient Descent (SGD)
SGD is a probabilistic optimization method widely used in deep learning to minimize the loss
function by updating model parameters based on random subsets of data (mini-batches).
The stochastic nature of SGD introduces randomness into the training process, which can help
escape local minima and improve generalization.
Regularization Techniques
Regularization methods, such as dropout and weight decay, introduce probabilistic elements
into the training process to prevent overfitting.

8. Cont….
Deep Generative Models
Deep generative models, such as GANs (Generative Adversarial Networks) and VAEs, provide
probabilistic frameworks for generating new data samples from learned distributions.
These models have applications in image synthesis, text generation, and more.

9. Backpropagation and regularization
Backpropagation
Backpropagation (backward propagation of errors) is the key algorithm used to train artificial
neural networks. It involves a two-step process: forward pass and backward pass.
Forward Pass
Input Data: The input data is fed into the network.
Propagation: The input data propagates through the network, layer by layer, to produce an
output.
Loss Calculation: The output is compared with the actual target output to calculate the loss
(error).

10. Backward Pass
Compute Gradients: The loss is propagated backward through the network to compute
gradients of the loss with respect to each weight using the chain rule of calculus.
Update Weights: The weights are updated using gradient descent (or one of its variants) to
minimize the loss. This is done by subtracting the gradient multiplied by the learning rate from
the current weights.
The process is repeated for many iterations (epochs) until the network converges to a state
where the loss is minimized.

11. Regularization
Regularization techniques are used to prevent overfitting, where the model performs well on
the training data but poorly on new, unseen data.
Here are some common regularization methods:
Dropout:
Randomly sets a fraction of the input units to zero at each update during training time. This
prevents units from co-adapting too much.
Data Augmentation:
Artificially enlarges the training dataset by creating modified versions of the existing data,
helping the model generalize better.
Early Stopping: Monitors the performance of the model on a validation set and stops training
when the performance stops improving, thus preventing overfitting.

12. Batch normalization- VC Dimension
Batch Normalization improves the training process of neural networks by normalizing layer
inputs, leading to more stable and faster training.
VC Dimension is a measure of a model's capacity to classify data, helping to understand the
balance between model complexity and generalization ability.
Benefits of Batch Normalization
Stabilizes Training: Reduces internal covariate shift, making the training process more stable.
Higher Learning Rates: Allows the use of higher learning rates, speeding up convergence.
Regularization Effect: Acts as a regularizer, potentially reducing the need for dropout.

13. Cont…..
VC Dimension
VC (Vapnik-Chervonenkis) dimension is a measure of the capacity (complexity) of a statistical
classification algorithm. It is used to understand the learning capability and overfitting tendency of a
model.
Definition
The VC dimension of a model is defined as the largest number of points that can be shattered by the
model. A set of points is said to be shattered by a model if the model can correctly classify all possible
labelings of those points.
Importance of VC Dimension
Model Complexity: A higher VC dimension indicates a more complex model that can fit more intricate
patterns.
Generalization: Models with a very high VC dimension relative to the number of training samples are
prone to overfitting, as they can fit the training data very well but may not generalize to unseen data.

14. Neural Nets-Deep Vs Shallow Networks
Neural networks, a key technology in machine learning, come in various forms, with deep and shallow
networks being two primary types.
The distinction between them lies mainly in their architecture and complexity:
Shallow Neural Networks
Architecture: Consist of an input layer, one or two hidden layers, and an output layer.
Complexity: Less complex due to fewer hidden layers.
Training Time: Generally faster to train due to fewer parameters.
Computational Resources: Require fewer computational resources.
Performance: Suitable for simpler tasks where the relationship between input and output is not
highly complex.
Examples of Use Cases: Basic classification tasks, simple pattern recognition, linear or mildly
nonlinear relationships.

15. Deep Neural Networks
Architecture: Consist of an input layer, multiple hidden layers (often more than three), and an
output layer.
Complexity: More complex due to the increased number of layers.
Training Time: Longer training times due to the greater number of parameters.
Computational Resources: Require significantly more computational resources, including
advanced hardware like GPUs.
Performance: Capable of capturing highly complex relationships and patterns in data, making
them suitable for more complex tasks.
Examples of Use Cases: Image and speech recognition, natural language processing,
autonomous driving, and other tasks involving large datasets and complex feature relationships.

16. Shallow Networks
Advantages:
◦ Faster training times.
◦ Simpler to implement and understand.
◦ Require less computational power.
Disadvantages:
◦ Limited ability to model complex data.
◦ May under fit complex datasets.

17. Deep Networks
Advantages:
◦ Can model complex patterns and relationships.
◦ More powerful in handling large and complex datasets.
Disadvantages:
◦ Longer training times and higher computational cost.
◦ Risk of overfitting if not properly regularized.

18. Networks-Convolutional Networks-
Generative Adversarial Networks (GAN)
Convolutional Neural Networks (CNNs) are specialized deep neural networks primarily used for processing
structured grid data such as images.
They are particularly effective for tasks involving visual data due to their unique architecture.
Key Features of CNNs
Convolutional Layers: These layers apply a set of filters to the input data, producing feature maps. Filters help in
detecting patterns such as edges, textures, and shapes.
Pooling Layers: These layers reduce the spatial dimensions of the feature maps, lowering computational cost and
helping in abstraction.
Fully Connected Layers: Typically placed at the end, these layers combine the extracted features to make the final
prediction.
Activation Functions: Commonly used activation functions like ReLU (Rectified Linear Unit) add non-linearity,
enabling the network to learn complex patterns.
Dropout and Batch Normalization: Techniques to prevent overfitting and ensure faster convergence during
training.

19. Applications of CNNs
Image Classification: Assigning a label to an image (e.g., recognizing objects in photos).
Object Detection: Identifying and localizing objects within an image.
Image Segmentation: Classifying each pixel in an image into predefined categories.
Face Recognition: Identifying or verifying a person from an image.
Medical Image Analysis: Detecting anomalies or diseases from medical scans.

20. Generative Adversarial Networks (GANs)
•Generative Adversarial Networks (GANs) are a class of machine learning frameworks designed
for generative modeling.
•They consist of two neural networks, the generator and the discriminator, that are trained
simultaneously in a competitive manner.
Key Components of GANs
Generator: This network generates new data instances. It tries to produce data that is
indistinguishable from real data.
Discriminator: This network evaluates the authenticity of the data, distinguishing between real
data from the training set and fake data produced by the generator.
Adversarial Process: The generator and discriminator are trained together in a zero-sum game.
The generator aims to fool the discriminator, while the discriminator aims to correctly identify
real versus fake data.

21. Training GANs
Initialization: Both networks are initialized.
Adversarial Training:
◦ The generator creates fake data.
◦ The discriminator evaluates the data.
◦ The discriminator is trained on real and fake data.
◦ The generator is trained to produce data that the discriminator classifies as real.
Iterative Improvement: Through this iterative process, both networks improve, resulting in the
generator producing highly realistic data over time.

22. Applications of GANs
Image Generation: Creating realistic images from noise (e.g., generating faces, artwork).
Data Augmentation: Generating additional training data to improve machine learning models.
Super-Resolution: Enhancing the resolution of images.
Style Transfer: Applying the artistic style of one image to another.
Text-to-Image Synthesis: Generating images from textual descriptions.

23. Semi supervised Learning
Semi-supervised learning is a type of machine learning that combines a small amount of labeled
data with a large amount of unlabeled data during training.
This approach leverages the labeled data to improve learning accuracy and can significantly
reduce the cost and effort of data labeling.
It lies between supervised learning (using only labeled data) and unsupervised learning (using
only unlabeled data).

24. Key Concepts in Semi-Supervised
Learning
Labeled Data: A small dataset where each example is paired with an associated label.
Unlabeled Data: A larger dataset where examples do not have associated labels.
Label Propagation: The process of spreading label information from labeled to unlabeled data
based on similarity.
Self-Training: An iterative method where the model is initially trained on labeled data, predicts
labels for the unlabeled data, and then retrained on the newly labeled data.
Co-Training: Uses multiple classifiers trained on different views of the data to label the
unlabeled data.
Generative Models: Models that assume a distribution for the data and try to estimate it using
both labeled and unlabeled data.

25. Techniques in Semi-Supervised Learning
•Self-Training:
•The model is trained on labeled data.
•It predicts labels for the unlabeled data.
•High-confidence predictions are added to the training set as pseudo-labeled data.
•The process is repeated iteratively.
•Co-Training:
•Uses two or more different classifiers.
•Each classifier is trained on different subsets of features.
•Classifiers label the unlabeled data, and their predictions are added to the training set for the
other classifiers.

26. Cont…..
•Semi-Supervised SVM:
•Extends Support Vector Machines (SVM) to use unlabeled data.
•Finds the decision boundary that maximizes the margin on both labeled and unlabeled data.
•Graph-Based Methods:
•Construct a graph where nodes represent data points, and edges represent similarities.
•Use label propagation to spread label information through the graph.
•Generative Models:
•Assume a probabilistic model for the data distribution.
•Use labeled data to learn parameters for the distribution and then use these parameters to assign
labels to the unlabeled data.

27. Applications of Semi-Supervised Learning
Natural Language Processing (NLP): Tasks like text classification, named entity recognition, and
sentiment analysis often use semi-supervised learning to leverage large amounts of unlabeled
text data.
Computer Vision: Object detection, image classification, and segmentation can benefit from
semi-supervised learning due to the high cost of labeling images.
Speech Recognition: Large volumes of audio data can be used to improve speech recognition
models with minimal labeled data.
Bioinformatics: Protein function prediction, gene expression analysis, and other tasks where
labeled data is scarce but unlabeled data is abundant.
Medical Imaging: Diagnosis and treatment planning can use semi-supervised learning to analyze
medical images with limited labeled examples.

28. Advantages and Challenges
Advantages
Reduced Labeling Cost: Significantly reduces the need for labeled data, which can be expensive
and time-consuming to obtain.
Improved Performance: Can lead to better performance than using only labeled data, especially
when the amount of labeled data is small.
Utilization of Unlabeled Data: Makes effective use of the large amounts of unlabeled data that
are often available.

29. Challenges
Complexity: Implementing semi-supervised learning techniques can be more complex than
supervised or unsupervised learning.
Quality of Pseudo-Labels: Incorrect pseudo-labels can degrade model performance.
Model Assumptions: Many semi-supervised learning methods rely on specific assumptions (e.g.,
the cluster assumption), which may not hold true for all datasets.
In summary, semi-supervised learning is a powerful technique that bridges the gap between
supervised and unsupervised learning, leveraging the strengths of both to create more accurate
and cost-effective models.
It is particularly useful in scenarios where labeled data is scarce but unlabeled data is abundant.

30. THANK YOU 