A brief introduction to deep learning, with focus on industry favorite networks, viz. Convolutional Neural Network and Recurrent Neural Network.

1. Deep Learning with Convolutional Neural
Network and Recurrent Neural Network
Ashray Bhandare

2. Page  2
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning

3. Page  3
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning

4. Page  4
Introduction
 For a computer to do a certain task, We have to give them a set of
instruction that they will follow.
 To give these instruction we should know what the answer is before hand.
Therefore the problem cannot be generalized.
 What if we have a problem where we don’t know anything about it?
 To overcome this, we use Neural networks as it is good with pattern
recognition.
EECS 4750/5750 Machine Learning

5. Page  5
Neural Network Refresher
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

6. Page  6
Deep network
EECS 4750/5750 Machine Learning

7. Page  7
Reasons to go Deep
 Historically, computers have only been useful for tasks that we can
explain with a detailed list of instructions.
 Computers fail in applications where the task at hand is fuzzy, such as
recognizing patterns.
EECS 4750/5750 Machine Learning
Pattern Complexity
Simple Methods like SVM,
regression
Moderate Neural Networks
outperform
Complex Deep Nets – Practical
Choice
EECS 4750/5750 Machine Learning

8. Page  8
Reasons to go Deep
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

9. Page  9
Reasons to go Deep
 A downside of training a deep network is the computational cost.
 The resources required to effectively train a deep net were prohibitive in
the early years of neural networks. However, thanks to advances in high-
performance GPUs of the last decade, this is no longer an issue
 Complex nets that once would have taken months to train, now only take
days.
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

10. Page  10
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

11. Page  11
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

12. Page  12
How to choose a Deep Net
 Convolutional Neural Network (CNN)
 Recurrent Neural Network (RNN)
 Deep Belief Network (DBN)
 Recursive Neural Tensor Network (RNTN)
 Restricted Boltzmann Machine (RBM)
EECS 4750/5750 Machine Learning
Applications
Text Processing RNTN, RNN
Image Recognition CNN, DBM
Object Recognition CNN, RNTN
Speech Recognition RNN
Time series Analysis RNN
Unlabeled data – pattern
recognition
RBM
EECS 4750/5750 Machine Learning

13. Page  13
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

14. Page  14
Introduction
 A convolutional neural network (or ConvNet) is a type of feed-forward
artificial neural network
 The architecture of a ConvNet is designed to take advantage of the 2D
structure of an input image.
 
 A ConvNet is comprised of one or more convolutional layers (often with a
pooling step) and then followed by one or more fully connected layers as
in a standard multilayer neural network.
EECS 4750/5750 Machine Learning
VS
EECS 4750/5750 Machine Learning

15. Page  15
Motivation behind ConvNets
 Consider an image of size 200x200x3 (200 wide, 200 high, 3 color
channels)
– a single fully-connected neuron in a first hidden layer of a regular Neural
Network would have 200*200*3 = 120,000 weights.
– Due to the presence of several such neurons, this full connectivity is wasteful
and the huge number of parameters would quickly lead to overfitting
 However, in a ConvNet, the neurons in a layer will only be connected to a
small region of the layer before it, instead of all of the neurons in a fully-
connected manner.
– the final output layer would have dimensions 1x1xN, because by the end of the
ConvNet architecture we will reduce the full image into a single vector of class
scores (for N classes), arranged along the depth dimension
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

16. Page  16
MLP VS ConvNet
EECS 4750/5750 Machine Learning
Input
Hidden
Output
Input
Hidden
Output
Multilayered
Perceptron:
All Fully
Connected
Layers
Convolutional
Neural Network
With Partially
Connected
Convolution Layer
EECS 4750/5750 Machine Learning

17. Page  17
MLP vs ConvNet
A regular 3-layer Neural
Network.
A ConvNet arranges its
neurons in three dimensions
(width, height, depth), as
visualized in one of the
layers.
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

18. Page  18
How ConvNet Works
 For example, a ConvNet takes the input as an image which can be
classified as ‘X’ or ‘O’
 In a simple case, ‘X’ would look like:
X or OCNN
A two-dimensional
array of pixels
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

19. Page  19
How ConvNet Works
 What about trickier cases?
CNN
X
CNN
O
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

20. Page  20
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 1 -1 -1 -1
-1 -1 1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 1 -1 -1
-1 -1 -1 1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=
?
How ConvNet Works – What Computer Sees
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

21. Page  21
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 1 -1 -1 -1
-1 -1 1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 1 -1 -1
-1 -1 -1 1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=x
How ConvNet Works
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

22. Page  22
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 X -1 -1 -1 -1 X X -1
-1 X X -1 -1 X X -1 -1
-1 -1 X 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 X -1 -1
-1 -1 X X -1 -1 X X -1
-1 X X -1 -1 -1 -1 X -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
How ConvNet Works – What Computer Sees
 Since the pattern doesnot match exactly, the computer will not be able to
classify this as ‘X’
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

23. Page  23
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

24. Page  24
ConvNet Layers (At a Glance)
 CONV layer will compute the output of neurons that are connected to local
regions in the input, each computing a dot product between their weights
and a small region they are connected to in the input volume.
 RELU layer will apply an elementwise activation function, such as the
max(0,x) thresholding at zero. This leaves the size of the volume
unchanged.
 POOL layer will perform a downsampling operation along the spatial
dimensions (width, height).
 FC (i.e. fully-connected) layer will compute the class scores, resulting in
volume of size [1x1xN], where each of the N numbers correspond to a
class score, such as among the N categories.
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

25. Page  25
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 X -1 -1 -1 -1 X X -1
-1 X X -1 -1 X X -1 -1
-1 -1 X 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 X -1 -1
-1 -1 X X -1 -1 X X -1
-1 X X -1 -1 -1 -1 X -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Recall – What Computer Sees
 Since the pattern doesnot match exactly, the computer will not be able to
classify this as ‘X’
 What got changed?
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

26. Page  26
=
=
=
 Convolution layer will work to identify patterns (features) instead of
individual pixels
Convolutional Layer
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

27. Page  27
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 1
-1 1 -1
1 -1 -1
1 -1 1
-1 1 -1
1 -1 1
Convolutional Layer - Filters
 The CONV layer’s parameters consist of a set of learnable filters.
 Every filter is small spatially (along width and height), but extends through
the full depth of the input volume.
 During the forward pass, we slide (more precisely, convolve) each filter
across the width and height of the input volume and compute dot products
between the entries of the filter and the input at any position.
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

28. Page  28
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 1
-1 1 -1
1 -1 -1
1 -1 1
-1 1 -1
1 -1 1
Convolutional Layer - Filters
 Sliding the filter over the width and height of the input gives 2-dimensional
activation map that responds to that filter at every spatial position.
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

29. Page  29
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
 Strides = 1, Filter Size = 3 X 3 X 1, Padding = 0
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

30. Page  30
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

31. Page  31
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

32. Page  32
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

33. Page  33
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

34. Page  34
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

35. Page  35
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

36. Page  36
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

37. Page  37
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Navigation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

38. Page  38
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

39. Page  39
1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

40. Page  40
1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

41. Page  41
1 1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

42. Page  42
1 1 1
1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

43. Page  43
1 1 1
1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

44. Page  44
1 1 1
1 1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

45. Page  45
1 1 1
1 1 1
1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

46. Page  46
1 1 1
1 1 1
1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

47. Page  47
1 1 1
1 1 1
1 1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

48. Page  48
1
1 1 1
1 1 1
1 1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

49. Page  49
1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

50. Page  50
1 1 -1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

51. Page  51
1 1 -1
1 1 1
-1 1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

52. Page  52
1
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
1 1 -1
1 1 1
-1 1 1
55
1 1 -1
1 1 1
-1 1 1
Convolutional Layer – Filters – Computation Example
EECS 4750/5750 Machine Learning

53. Page  53
1 -1 -1
-1 1 -1
-1 -1 1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
Convolutional Layer – Filters – Computation Example
Input Size (W): 9
Filter Size (F): 3 X 3
Stride (S): 1
Filters: 1
Padding (P): 0
9 X 9 7 X 7
Feature Map Size = 1+ (W – F + 2P)/S
= 1+ (9 – 3 + 2 X 0)/1 = 7
EECS 4750/5750 Machine Learning

54. Page  54
1 -1 -1
-1 1 -1
-1 -1 1
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
=
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-1 -1 1
-1 1 -1
1 -1 -1
1 -1 1
-1 1 -1
1 -1 1
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
=
=
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Filters – Output Feature Map
 Output Feature
Map of One
complete
convolution:
– Filters: 3
– Filter Size: 3 X 3
– Stride: 1
 Conclusion:
– Input Image:
9 X 9
– Output of
Convolution:
7 X 7 X 3
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

55. Page  55
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolutional Layer – Output
EECS 4750/5750 Machine Learning

56. Page  56
Rectified Linear Units (ReLUs)
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
0.77
Max(0,x)
EECS 4750/5750 Machine Learning

57. Page  57
0.77 0
Rectified Linear Units (ReLUs)
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
Max(0,x)
EECS 4750/5750 Machine Learning

58. Page  58
0.77 0 0.11 0.33 0.55 0 0.33
Rectified Linear Units (ReLUs)
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
Max(0,x)
EECS 4750/5750 Machine Learning

59. Page  59
0.77 0 0.11 0.33 0.55 0 0.33
0 1.00 0 0.33 0 0.11 0
0.11 0 1.00 0 0.11 0 0.55
0.33 0.33 0 0.55 0 0.33 0.33
0.55 0 0.11 0 1.00 0 0.11
0 0.11 0 0.33 0 1.00 0
0.33 0 0.55 0.33 0.11 0 0.77
Rectified Linear Units (ReLUs)
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
Max(0,x)
EECS 4750/5750 Machine Learning

60. Page  60
ReLU layer
0.77 0 0.11 0.33 0.55 0 0.33
0 1.00 0 0.33 0 0.11 0
0.11 0 1.00 0 0.11 0 0.55
0.33 0.33 0 0.55 0 0.33 0.33
0.55 0 0.11 0 1.00 0 0.11
0 0.11 0 0.33 0 1.00 0
0.33 0 0.55 0.33 0.11 0 0.77
0.33 0 0.11 0 0.11 0 0.33
0 0.55 0 0.33 0 0.55 0
0.11 0 0.55 0 0.55 0 0.11
0 0.33 0 1.00 0 0.33 0
0.11 0 0.55 0 0.55 0 0.11
0 0.55 0 0.33 0 0.55 0
0.33 0 0.11 0 0.11 0 0.33
0.33 0 0.55 0.33 0.11 0 0.77
0 0.11 0 0.33 0 1.00 0
0.55 0 0.11 0 1.00 0 0.11
0.33 0.33 0 0.55 0 0.33 0.33
0.11 0 1.00 0 0.11 0 0.55
0 1.00 0 0.33 0 0.11 0
0.77 0 0.11 0.33 0.55 0 0.33
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.11 0.33 -0.77 1.00 -0.77 0.33 -0.11
0.11 -0.55 0.55 -0.77 0.55 -0.55 0.11
-0.55 0.55 -0.55 0.33 -0.55 0.55 -0.55
0.33 -0.55 0.11 -0.11 0.11 -0.55 0.33
EECS 4750/5750 Machine Learning

61. Page  61
Pooling Layer
 The pooling layers down-sample the previous layers feature map.
 Its function is to progressively reduce the spatial size of the representation
to reduce the amount of parameters and computation in the network
 The pooling layer often uses the Max operation to perform the
downsampling process
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

62. Page  62
1.00
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

63. Page  63
1.00 0.33
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning

64. Page  64
1.00 0.33 0.55
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning

65. Page  65
1.00 0.33 0.55 0.33
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning

66. Page  66
1.00 0.33 0.55 0.33
0.33
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning

67. Page  67
1.00 0.33 0.55 0.33
0.33 1.00 0.33 0.55
0.55 0.33 1.00 0.11
0.33 0.55 0.11 0.77
Pooling
 Pooling Filter Size = 2 X 2, Stride = 2
EECS 4750/5750 Machine Learning

68. Page  68
1.00 0.33 0.55 0.33
0.33 1.00 0.33 0.55
0.55 0.33 1.00 0.11
0.33 0.55 0.11 0.77
0.33 0.55 1.00 0.77
0.55 0.55 1.00 0.33
1.00 1.00 0.11 0.55
0.77 0.33 0.55 0.33
0.55 0.33 0.55 0.33
0.33 1.00 0.55 0.11
0.55 0.55 0.55 0.11
0.33 0.11 0.11 0.33
Pooling
EECS 4750/5750 Machine Learning

69. Page  69
Layers get stacked
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
1.00 0.33 0.55 0.33
0.33 1.00 0.33 0.55
0.55 0.33 1.00 0.11
0.33 0.55 0.11 0.77
0.33 0.55 1.00 0.77
0.55 0.55 1.00 0.33
1.00 1.00 0.11 0.55
0.77 0.33 0.55 0.33
0.55 0.33 0.55 0.33
0.33 1.00 0.55 0.11
0.55 0.55 0.55 0.11
0.33 0.11 0.11 0.33
EECS 4750/5750 Machine Learning

70. Page  70
Layers Get Stacked - Example
224 X 224 X 3 224 X 224 X 64
CONVOLUTION
WITH 64 FILTERS
112 X 112 X 64
POOLING
(DOWNSAMPLING)
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

71. Page  71
Deep stacking
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
EECS 4750/5750 Machine Learning

72. Page  72
Fully connected layer
 Fully connected layers are the normal flat
feed-forward neural network layers.
 These layers may have a non-linear
activation function or a softmax activation
in order to predict classes.
 To compute our output, we simply re-
arrange the output matrices as a 1-D
array.
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

73. Page  73
Fully connected layer
 A summation of product of inputs and weights at each output node
determines the final prediction
X
O
0.55
1.00
1.00
0.55
0.55
0.55
0.55
0.55
1.00
0.55
0.55
1.00
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

74. Page  74
Putting it all together
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
X
O
EECS 4750/5750 Machine Learning

75. Page  75
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

76. Page  76
Hyperparameters (knobs)
 Convolution
– Filter Size
– Number of Filters
– Padding
– Stride
 Pooling
– Window Size
– Stride
 Fully Connected
– Number of neurons
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

77. Page  77
Case Studies
 LeNet – 1998
 AlexNet – 2012
 ZFNet – 2013
 VGG – 2014
 GoogLeNet – 2014
 ResNet – 2015
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

78. Page  78
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

79. Page  79
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

80. Page  80
 Humans don’t start their thinking from scratch every second. You don’t
throw everything away and start thinking from scratch again. Your
thoughts have persistence.
 You make use of context and previous knowledge to understand what is
coming next
 Recurrent neural networks address this issue. They are networks with
loops in them, allowing information to persist.
EECS 4750/5750 Machine Learning
Recurrent Neural Network -Intro
EECS 4750/5750 Machine Learning

81. Page  81
What’s for dinner?
pizza
sushi
waffles
day
of the
week
month
of the
year
late
meeting
Recurrent Neural Network -Intro
EECS 4750/5750 Machine Learning

82. Page  82
pizza
sushi
waffles
pizza
yesterday
sushi
yesterday
waffles
yesterday
What’s for dinner?
Recurrent Neural Network -Intro
EECS 4750/5750 Machine Learning

83. Page  83
pizza
sushi
waffles
pizza yesterday
sushi yesterday
waffles yesterday
predicted pizza for
yesterday
predicted sushi for yesterday
predicted waffles for
yesterday
Recurrent Neural Network -Intro
EECS 4750/5750 Machine Learning

84. Page  84
A vector is a list of values
Weather vectorHigh
temperature
67
43
13
.25
.83
Low
temperature
Wind speed
Precipitation
Humidity
==
“High is 67 F.
Low is 43 F.
Wind is 13 mph.
.25 inches of rain.
Relative humidity
is 83%.”
67
43
13
.25
.83
Vectors as Inputs
EECS 4750/5750 Machine Learning

85. Page  85
A vector is a list of values
0
0
1
0
0
0
0
Day of week vector
Sunday 0
0
1
0
0
0
0
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
==“It’s
Tuesday”
Vectors as Inputs
EECS 4750/5750 Machine Learning

86. Page  86
A vector is a list of values
Dinner prediction vector
Pizza 0
1
0
Sushi
Waffles
==
“Tonight I
think
we’re
going to
have
sushi.”
0
1
0
Vectors as Inputs
EECS 4750/5750 Machine Learning

87. Page  87
pizza
sushi
waffles
pizza yesterday
sushi yesterday
waffles
yesterday
predicted pizza for yesterday
predicted sushi for yesterday
predicted waffles for yesterday
How RNN’s Work
EECS 4750/5750 Machine Learning

88. Page  88
prediction for
today
dinner
yesterday
predictions for yesterday
How RNN’s Work
EECS 4750/5750 Machine Learning

89. Page  89
predicti
on
new
information
How RNN’s Work
EECS 4750/5750 Machine Learning

90. Page  90
predicti
on
How RNN’s Work
EECS 4750/5750 Machine Learning

91. Page  91
Unrolled predictions
pizza
sushi
waffles
yesterdaytwo days ago
...
...
...
How RNN’s Work
EECS 4750/5750 Machine Learning

92. Page  92
RNN’s – Overall Structure
These loops make recurrent neural networks seem kind of
mysterious.
A recurrent neural network can be thought of as multiple
copies of the same network, each passing a message to a
successor. Consider what happens if we unroll the loop:
EECS 4750/5750 Machine Learning

93. Page  93
Consider Simple statements
Harry met Sally.
Sally met James.
James met Harry.
...
Dictionary : {Harry, Sally, James, met, .}
RNN’s - Example
For the sake of illustration, lets build an RNN that looks at
only one previous word.
EECS 4750/5750 Machine Learning

94. Page  94
predicti
on
new
information
Sally
Harry
James
saw
.
Sally
Harry
James
met
.
Sally
Harry
James
met
.
RNN’s - Example
EECS 4750/5750 Machine Learning

95. Page  95
predicti
on
new
information
.
RNN’s - Example
Sally
Harry
James
saw
.
Sally
Harry
James
met
.
Sally
Harry
James
met
.
EECS 4750/5750 Machine Learning

96. Page  96
predicti
on
new
information
RNN’s - Example
Sally
Harry
James
saw
.
Sally
Harry
James
met
.
Sally
Harry
James
met
.
EECS 4750/5750 Machine Learning

97. Page  97
prediction
new
information
recurrent
neural
network
RNN’s - Example
EECS 4750/5750 Machine Learning

98. Page  98
Hyperbolic tangent (tanh) squashing function
Your number goes
in here
The squashed
version comes out
here
No matter what you start with, the answer stays between -1 and 1.
EECS 4750/5750 Machine Learning

99. Page  99
prediction
new
information
recurrent
neural
network
RNN’s – Overall Structure
EECS 4750/5750 Machine Learning

100. Page  100
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

101. Page  101
Mistakes an RNN can makeHarry met Harry.
Sally met James met Harry met …
James. Harry. Sally.
Why RNN’s Fail
This can be easily resolved by going back multiple time steps
EECS 4750/5750 Machine Learning

102. Page  102
“the clouds are in the sky”
Why RNN’s Fail –Vanishing or Exploding Gradient
In such cases, where the gap between the relevant
information and the place that it’s needed is small, RNNs
can learn to use the past information.
EECS 4750/5750 Machine Learning

103. Page  103
Why RNN’s Fail –Vanishing or Exploding Gradient
EECS 4750/5750 Machine Learning

104. Page  104
“I grew up in France. I lived there for about 20 years. The people there are
very nice. I can speak fluent French”
Why RNN’s Fail –Vanishing or Exploding Gradient
Unfortunately, as that gap grows,
RNNs become unable to learn to
connect the information.
EECS 4750/5750 Machine Learning

105. Page  105
Why RNN’s Fail –Vanishing or Exploding Gradient
EECS 4750/5750 Machine Learning

106. Page  106
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

107. Page  107
prediction
new
information
recurrent
neural
network
Solution – Long Short Term Memory
EECS 4750/5750 Machine Learning

108. Page  108
prediction
new
information
Solution – Long Short Term Memory
EECS 4750/5750 Machine Learning

109. Page  109
memory
memory
forgetting
prediction
Prediction +
Memories
new
information
Forget Gate Layer
EECS 4750/5750 Machine Learning

110. Page  110
Plus junction: element-by-element addition
6
7
8
3
4
5
=
3 + 6
4 + 7
5 + 8
9
11
13
=
EECS 4750/5750 Machine Learning

111. Page  111
Times junction: element-by-element multiplication
6
7
8
3
4
5
=
3 x 6
4 x 7
5 x 8
18
28
40
=
EECS 4750/5750 Machine Learning

112. Page  112
Gating
1.0
0.5
0.0
0.8
0.8
0.8
=
0.8 x 1.0
0.8 x 0.5
0.8 x 0.0
0.8
0.4
0.0
=
Signal
On / Off
gating
EECS 4750/5750 Machine Learning

113. Page  113
Logistic (sigmoid) squashing function
1.0
0.5
1.0 2.0-1.0-2.0
No matter what you start with, the answer stays between 0 and 1.
EECS 4750/5750 Machine Learning

114. Page  114
memory
memory
forgetting
prediction
Prediction +
Memories
new
information
Forget Gate Layer
EECS 4750/5750 Machine Learning

115. Page  115
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
new
information
Long Short Term Memory
EECS 4750/5750 Machine Learning

116. Page  116
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
new
information
Long Short Term Memory
EECS 4750/5750 Machine Learning

117. Page  117
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

118. Page  118
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
Harry met Sally.
James …
LSTM in Action
EECS 4750/5750 Machine Learning

119. Page  119
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
Harry met Sally.
James …
James,
Harry,
Sally
LSTM in Action
EECS 4750/5750 Machine Learning

120. Page  120
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
Prediction
Harry met Sally.
James …
James,
Harry,
Sally
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

121. Page  121
ignoring
collected
possibilities
selection
memory
forgetting
Prediction
Harry met Sally.
James …
James,
Harry,
Sally
met,
James
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

122. Page  122
ignoring
selection
memory
forgetting
Prediction
Harry met Sally.
James …
James,
Harry,
Sally
met,
James
met,
James
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

123. Page  123
ignoring
selection
memory
forgetting
Harry met Sally.
James …
James,
Harry,
Sally
met,
James
met,
James
met,
James
met
LSTM in Action
EECS 4750/5750 Machine Learning

124. Page  124
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
new
information
LSTM in Action
EECS 4750/5750 Machine Learning

125. Page  125
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
new
information
met
LSTM in Action
EECS 4750/5750 Machine Learning

126. Page  126
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
met
Harry met Sally.
James met…
LSTM in Action
EECS 4750/5750 Machine Learning

127. Page  127
filtered
possibilitiesignoring
collected
possibilities
selection
memory
forgetting
possibilities
Prediction
met
Harry met Sally.
James met…
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

128. Page  128
filtered
possibilitiesignoring
collected
possibilities
selection
James
forgetting
possibilities
Prediction
met
Harry met Sally.
James met…
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

129. Page  129
filtered
possibilitiesignoring
collected
possibilities
selection
James
forgetting
James,
Harry,
Sally
Prediction
met
Harry met Sally.
James met…
met,
James
LSTM in Action
EECS 4750/5750 Machine Learning

130. Page  130
ignoring
collected
possibilities
selection
James
forgetting
James,
Harry,
Sally
Prediction
met
Harry met Sally.
James met…
met,
James
James,
Harry,
Sally
LSTM in Action
EECS 4750/5750 Machine Learning

131. Page  131
ignoring
Harry,
Sally
selection
James
forgetting
James,
Harry,
Sally
Prediction
met
Harry met Sally.
James met…
met,
James
James,
Harry,
Sally
LSTM in Action
EECS 4750/5750 Machine Learning

132. Page  132
ignoring
Harry,
Sally
selection
James
forgetting
James,
Harry,
Sally
Harry,
Sally
met
Harry met Sally.
James met…
met,
James
James,
Harry,
Sally
LSTM in Action
EECS 4750/5750 Machine Learning

133. Page  133
Agenda
 Introduction to Deep Learning
– Neural Nets Refresher
– Reasons to go Deep
 Demo 1 – Keras
 How to Choose a Deep Net
 Introcuction to CNN
– Architecture Overview
– How ConvNet Works
 ConvNet Layers
– Convolutional Layer
– Pooling Layer
– Normalization Layer (ReLU)
– Fully-Connected Layer
 Hyper Parameters
EECS 4750/5750 Machine Learning
 Demo 2 – MNIST Classification
 Introduction to RNN
– Architecture Overview
– How RNN’s Works
– RNN Example
 Why RNN’s Fail
 LSTM
– Memory
– Selection
– Ignoring
 LSTM Example
 Demo 3 – Imdb Review
Classification
 Image Captioning
EECS 4750/5750 Machine Learning

134. Page  134
References
 Karpathy, A. (n.d.). CS231n Convolutional Neural Networks for Visual Recognition.
Retrieved from http://cs231n.github.io/convolutional-networks/#overview
 Rohrer, B. (n.d.). How do Convolutional Neural Networks work?. Retrieved from
http://brohrer.github.io/how_convolutional_neural_networks_work.html
 Brownlee, J. (n.d.). Crash Course in Convolutional Neural Networks for Machine Learning.
Retrieved from http://machinelearningmastery.com/crash-course-convolutional-neural-
networks/
 Lidinwise (n.d.). The revolution of depth. Retrieved from https://medium.com/@Lidinwise/the-
revolution-of-depth-facf174924f5#.8or5c77ss
 Nervana. (n.d.). Tutorial: Convolutional neural networks. Retrieved from
https://www.nervanasys.com/convolutional-neural-networks/
 Olah, C. (n.d.). Tutorial: Understanding LSTM Networks. Retrieved from
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
 Rohrer, B. (n.d.). Tutorial: How Recurrent Neural Networks and Long Short-Term Memory
Work. Retrieved from https://brohrer.github.io/how_rnns_lstm_work.html
 Serrano, L. (n.d.). A friendly introduction to Recurrent Neural Networks. Retrieved from
https://www.youtube.com/watch?v=UNmqTiOnRfg&t=87s
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning

135. Page  135
Thank you
EECS 4750/5750 Machine Learning
EECS 4750/5750 Machine Learning