Deep learning is a class of machine learning algorithms that uses multiple layers of nonlinear processing units for feature extraction and transformation. It can be used for supervised learning tasks like classification and regression or unsupervised learning tasks like clustering. Deep learning models include deep neural networks, deep belief networks, and convolutional neural networks. Deep learning has been applied successfully in domains like computer vision, speech recognition, and natural language processing by companies like Google, Facebook, Microsoft, and others.

1. Introduc)on
to
Deep
Learning
Massimiliano
Ruocco

2. Outline
• Introduction and Motivation for DL
• From NN to Deep Learning
• Deep Learning Models
• Deep Learning in the Real World
• Conclusion

3. Introduction and
Motivation for DL

4. Introduction
Deep Learning - WHAT
Class of ML training algorithm

5. Introduction
Deep Learning - Motivations
• ML Algorithms:
– Supervised
– Unsupervised
– Semi-supervised
– Reinforcement Learning

6. • ML Algorithms: unsupervised learning
Data
Representa)on
Input
Clustering
Output
Example (Marketing/Customer segmentation):
• Input : Customers of a specific product
• Output: Customer subgroups
Introduction
Deep Learning - Motivations

7. • ML Algorithms: supervised learning
Data
Representa)on
Input
Classifica)on/
Regression
Output
Training
Labeled
DataSet
Data
Representa)on
Example (spam detection):
• Input : Email
• Output: Spam/NotSpam
• Training Set: Data set of mail labeled as Spam/Not Spam
Introduction
Deep Learning in ML and AI

8. • ML Algorithms: supervised learning
Data
Representa)on
Input
Classifica)on/
Regression
Output
Training
Labeled
DataSet
Data
Representa)on
Example (spam detection):
• Input : Email
• Output: Spam/NotSpam
• Training Set: Data set of mail labeled as Spam/Not Spam
Introduction
Deep Learning – Representation Problem

9. • Data Representation:
– feature set selection
– #features
• Main Issues:
– Course of dimensionality
– Overfitting
– Handcrafted features
• How to tackle: Representation Learning
Introduction
Deep Learning – Representation Problem

10. • Deep learning methods:
– Representations are expressed in terms of
other, simpler representations
Introduction
Deep Learning - WHAT

11. • Deep Learning algorithm as application of Machine
Learning to Artificial intelligence
Ar#ficial
Intelligence
(i.e.
knowledge
bases)
Machine
Learning
(i.e.
Support
Vector
Machine)
Representa#on
Learning
(i.e.
Autoencoders)
Deep
Learning
(i.e.
Mul=layer
Perceptron)
Introduction
Deep Learning in ML and AI

12. From Neural Network to
Deep Learning

13. • Neural Network: Basic
– Different layers of neurons/perceptrons
– Human brain analysis
– Input, Hidden Layer, Output
• Neural Network: Applications
– Classification (Spam Detection)
– Pattern Recognition (Character recognition)
Introduction
From Neural Network to Deep Learning

14. • The core: Neuron
Introduction
From Neural Network to Deep Learning
W1
W2
W3
x1
x2
xn
Sigmoid
func)on
1/(1+e-­‐z)
Output
hw(x)
x
=
[x0…xn]T
w
=
[w0…wn]T
z
=
wTx

15. • Neural Network – Single layer
Introduction
From Neural Network to Deep Learning

16. • Forward Propagation:
– process of computing the output
Introduction
From Neural Network to Deep Learning
x1
x2
x3
a1
2
a2
2
W(1)
W(2)
a(2)
z(2)
z(3)
X
z(2)
=
XW(1)
a(2)
=
f(z(2))
z(3)
=
a(2)W(2)
y
=
f(z(3))

17. • Training a Neural Network:
– Learning the parameters (weights)
• Supervised
• Unsupervised
• Reinforcement Learning
• Employing a Neural Network:
– Selecting the Architecture
– # Layers
– # Units per layer
– Kind of learning algorithm
Introduction
From Neural Network to Deep Learning

18. • Training a Neural Network:
– Backward Propagation
• Gradient descent
• Objective: Minimize the cost function J
Introduction
From Neural Network to Deep Learning
x1
x2
x3
a1
2
a2
2
W(1)
W(2)
a(2)
z(2)
z(3)
X

19. • DNN à Typically artificial neural netwok
with 3 or more levels of non-linear
operations
Introduction
From Neural Network to Deep Learning

20. • Using Back propagation for Deep NN
– Does not scale
– Bad performance for random initialization
– Local Optima
– Vanishing gradient problem
Introduction
Issues in Training DNN

21. Introduction
The Breakthrough
2006*+
Backward
Propaga#on
Greedy-­‐layer
wise
training
+
Supervised
fine
tuning
* Hinton et al. A fast learning algorithm for deep belief nets.
Neural Computation, 18:1527–1554, 2006
+ Ranzato et al. Efficient learning of sparse representations with an energy-based model.
Advances in Neural Information Processing Systems 19 (NIPS’06),

22. • Deep learning methods:
– Class of ML algorithm
– Use cascade of many levels of non linear
processing units for feature extraction
– Hierarchy of concepts
– Multiple-layered model
– NN with high number of hidden layers
– NEW LEARNING ALGORITHM Overcoming previous
training problems
Introduction
Deep Learning - Summary

23. Deep
Learning
Models

24. Deep Learning Models
Introduction
• Two main classes:
– Generative
• Deep Network for supervised Learning
– Discriminative
• Deep Network for unsupervised learning
– Hybrid

25. Deep Learning Models
Generative – Deep Belief Network
• Generative graphic model
• Mix directed and undirected between vars
• Learn to reconstruct the input

26. Deep Learning Models
Generative – Deep Belief Network
• Training algorithm
– Iteratively apply RBM training to each pair of
layers

27. Deep Learning Models
Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition
– Feed-forward multilayer network
– Kind of back propagation for learning
– Receptive fields
– Learn suitable representation of the image

28. Deep Learning Models
Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition
– Key concepts:
• Max pooling
• Sparse Connectivity
• Convolution

29. Deep
Learning
in
the
Real
World

30. • NLP
• Image Classification/Computer Vision
• Speech Recognition
Introduction
Deep Learning – Application Field

31. • [Google] 2013
acquired DNNresearch of professor Geoff
Hinton to improve the state of the art in
image recognition in photos
• [Facebook] 2013
hired deep learning expert Yann to head up
the company’s new artificial intelligence lab
specialized in deep learning for computer
vision and image recognition
• [Pinterest] 2014
announced it has acquired Visual Graph
• [Google + Baidu]:
20G13 - Deep Learning Visual Search Engine
Deep Learning in the Real World
Facts

32. • [Baidu] 2013:
Deep Learning Visual Search Engine
• [Google] 2013
Photo Search Engine
• [Microsoft] 2013
Search by voice on Xbox console
• [Google] 2014
word2vec for word tagging or text messaging
suggestion
Deep Learning in the Real World
Products

33. Thanks
for
the
aUen)on