Deep Learning in Computer Vision

Introduction to
Deep Learning
Presenter: Sungjoon Choi
(sungjoon.choi@cpslab.snu.ac.kr)

Optimization methods
CNN basics
Semantic segmentation
Weakly supervised localization
Image detection
RNN
Visual QnA
Word2Vec
Image Captioning
Contents

What is deep learning?
3
“Deep learning is a branch of machine learning based on a set of
algorithms that attempt to model high-level abstractions in data by
using multiple processing layers, with complex structures or otherwise,
composed of multiple non-linear transformations.”
Wikipedia says:
Machine
Learning
High-level
abstraction Network

Is it brand new?
4
Neural Nets McCulloch & Pitt 1943
Perception Rosenblatt 1958
RNN Grossberg 1973
CNN Fukushima 1979
RBM Hinton 1999
DBN Hinton 2006
D-AE Vincent 2008
AlexNet Alex 2012
GoogLeNet Szegedy 2015

Deep architectures
5
Feed-Forward: multilayer neural nets, convolutional nets
Feed-Back: Stacked Sparse Coding, Deconvolutional Nets
Bi-Directional: Deep Boltzmann Machines, Stacked Auto-Encoders
Recurrent: Recurrent Nets, Long-Short Term Memory

CNN
7
CNNs are basically layers of convolutions followed by
subsampling and fully connected layers.
Intuitively speaking, convolutions and subsampling
layers works as feature extraction layers while a fully
connected layer classifies which category current input
belongs to using extracted features.

Gradient descent?
There are three variants of gradient descent
Differ in how much data we use to compute
gradient
We make a trade-off between the accuracy
and computing time

Batch gradient descent
In batch gradient decent, we use the entire
training dataset to compute the gradient.

Stochastic gradient descent
In stochastic gradient descent (SGD), the
gradient is computed from each training
sample, one by one.

Mini-batch gradient decent
In mini-batch gradient decent, we take the
best of both worlds.
Common mini-batch sizes range between 50
and 256 (but can vary).

Challenges
Choosing a proper learning rate is cumbersome.
 Learning rate schedule
Avoiding getting trapped in suboptimal local
minima

Adagrad
It adapts the learning rate to the parameters,
performing larger updates for infrequent and
smaller updates for frequent parameters.
𝜃𝑡+1,𝑖 = 𝜃𝑡,𝑖 −
𝜂
𝐺𝑡,𝑖𝑖 + 𝜖
𝑔𝑡,𝑖
Performing larger updates for infrequent and
smaller updates for frequent parameters.

Adadelta
Adadelta is an extension of Adagrad that seeks
to reduce its monotonically decreasing learning
rate.
It restricts the window of accumulated past
gradients to some fixed size 𝑤.
𝐸 𝑔2
𝑡 = 𝛾𝐸 𝑔2
𝑡−1 + 1 − 𝛾 𝑔𝑡
2
𝐸 ∆𝜃2
𝑡 = 𝛾𝐸 ∆𝜃2
𝑡−1 + 1 − 𝛾 ∆𝜃𝑡
2
𝜃𝑡+1 = 𝜃𝑡 −
𝐸 ∆𝜃2
𝑡 + 𝜖
𝐸 𝑔2
𝑡 + 𝜖
𝑔𝑡
No learning rate!

RMSprop
RMSprop is an unpublished, adaptive learning
rate method proposed by Geoff Hinton in his
lecture..
𝐸 𝑔2
𝑡 = 𝛾𝐸 𝑔2
𝑡−1 + 1 − 𝛾 𝑔𝑡
2
𝜃𝑡+1 = 𝜃𝑡 −
𝜂
𝐸 𝑔2
𝑡 + 𝜖
𝑔𝑡

Adam
Adaptive Moment Estimation (Adam) stores both
exponentially decaying average of past gradients
and and squared gradients.
𝑚 𝑡 = 𝛽1 𝑚 𝑡−1 + 1 − 𝛽1 𝑔𝑡
𝑣 𝑡 = 𝛽2 𝑣 𝑡−1 + 1 − 𝛽2 𝑔𝑡
2
𝜃𝑡+1 = 𝜃𝑡 −
𝜂
𝑣 𝑡 + 𝜖
1 − 𝛽2
𝑡
1 − 𝛽1
𝑡 𝑚 𝑡
Momentum
Running average of
gradient squares

Adam
Adaptive Moment Estimation (Adam) stores both
exponentially decaying average of past gradients
and and squared gradients.
𝑚 𝑡 = 𝛽1 𝑚 𝑡−1 + 1 − 𝛽1 𝑔𝑡
𝑣 𝑡 = 𝛽2 𝑣 𝑡−1 + 1 − 𝛽2 𝑔𝑡
2
𝜃𝑡+1 = 𝜃𝑡 −
𝜂
𝑣 𝑡 + 𝜖
1 − 𝛽2
𝑡
1 − 𝛽1
𝑡 𝑚 𝑡

Semantic Segmentation?
lion
dog
giraffe
Image Classification
bicycle
person
ball
dog
Object Detection
person
person
person
person person
bicyclebicycle
Semantic Segmentation

Weakly
supervised
localization

76

77

Weakly Supervised Object Localization
78
Usually supervised learning of localization is annotated with bounding box
What if localization is possible with image label without bounding box
annotations?
Today’s seminar: Learning Deep Features for Discriminative
Localization
1512.04150v1 Zhou et al. 2015 CVPR2016

Architecture
79
AlexNet+GAP+places205
Living room
11x11 Avg Pooling: Global Average Pooling (GAP)
11x11x512
512 205
227x227x3

Class activation map (CAM)
80
• Identify important image regions by projecting back
the weights of output layer to convolutional feature
maps.
• CAMs can be generated for each class in single image.
• Regions for each categories are different in given image.
• palace, dome, church …

Results
81
• CAM on top 5 predictions on an image
• CAM for one object class in images

GAP vs. GMP
82
• Oquab et al. CVPR2015
Is object localization for free? weakly-supervised learning with convolutional neural
networks.
• Use global max pooling(GMP)
• Intuitive difference between GMP and GAP?
• GAP loss encourages identification on the extent of an object.
• GMP loss encourages it to identify just one discriminative part.
• GAP, average of a map maximized by finding all discriminative
parts of object
• if activations is all low, output of particular map reduces.
• GMP, low scores for all image regions except the most
discriminative part
• do not impact the score when perform MAX
pooling

GAP & GMP
83
• GAP (upper) vs GMP (lower)
• GAP outperforms GMP
• GAP highlights more complete
object regions and less
background noise.
• Loss for average pooling
benefits when the network
identifies all discriminative
regions of an object

Concept localization
85
Concept localization in weakly
labeled images
• Positive set: short phrase in text caption
• Negative set: randomly selected images
• Model catch the concept, phrases are
much more abstract than object name.
Weakly supervised text detector
• Positive set: 350 Google StreeView
images that contain text.
• Negative set: outdoor scene images in
SUN dataset
• Text highlighted without bounding box
annotations.

R-CNN
141
Image Regions Resize Convolution
Features
Classify

SPP net
142
Image Convolution Features SPPRegions Classify

R-CNN vs. SPP net
143
R-CNN SPP net

Fast R-CNN
144
Image
Convolution Features
Regions
RoI Pooling
Layer
Class Label
Confidence
RoI Pooling
Layer
Class Label
Confidence

R-CNN vs. SPP net vs. Fast R-CNN
145
R-CNN SPP net
Fast R-CNN

Faster R-CNN
146
Image Fully Convolutional
Features
Bounding Box
Regression
BB Classification
FastR-CNN

R-CNN vs. SPP net vs. Fast R-CNN
147
R-CNN SPP net
Fast R-CNN Faster R-CNN

Recurrent Neural Network
155
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Recurrent Neural Network
156

LSTM comes in!
157
Long Short Term Memory
This is just a standard RNN.

LSTM comes in!
158
Long Short Term Memory
This is just a standard RNN.This is the LSTM!

Overall Architecture
159
(Cell) state
Hidden State
Forget Gate
Input Gate
Output Gate
Next (Cell) State
Next Hidden State
Input
Output
Output = Hidden state

The Core Idea
160

VQA: Dataset and Problem definition
162
VQA dataset - Example
Q: How many dogs are seen?
Q: What animal is this?
Q: What color is the car?
Q: What is the mustache made of?Q: Is this vegetarian pizza?

Solving VQA
163
Approach
[Malinowski et al., 2015] [Ren et al., 2015] [Andres et al., 2015]
[Ma et al., 2015] [Jiang et al., 2015]
Various methods have been proposed

DPPnet
164
Motivation
Common pipeline of using deep learning for vision
CNN trained on ImageNet
Switch the final layer and fine-tune for the New Task
In VQA, Task is determined by a question
Observation:

DPPnet
165
Main Idea
Switching parameters of a layer based on a question
Dynamic Parameter Layer
Question Parameter Prediction Network

DPPnet
166
Parameter Explosion
Number of parameter for fc-layer (R):
DynamicParameterLayer
Question Feature
Predicted Parameter
M
N
Q
P
: Dimension of hidden state
fc-layer
N=Q×P R=Q×P×M Q=1000, P=1000, M=500
For example:
R=500,000,000
1.86GB for single layer
Number of parameters for
VGG19: 144,000,000

DPPnet
167
Parameter Explosion
Number of parameter for fc-layer (R):
Question Feature
Predicted Parameter
M
N
Q
P
: Dimension of hidden state
fc-layer
Solution:
R=Q×P×M R= N×M
N=Q×P N<Q×P
We can control N

DPPnet
168
Weight Sharing with Hashing Trick
Weights of Dynamic Parameter Layer are picked from Candidate weights by Hashing
Question Feature
Candidate Weights
fc-layer
0.11.2-0.70.3-0.2
0.1 0.1 -0.2 -0.7
1.2 -0.2 0.1 -0.7
-0.7 1.2 0.3 -0.2
0.3 0.3 0.1 1.2
Hasing
[Chen et al., 2015]

DPPnet
169
Final Architecture
End-to-End Fine-tuning is possible (Fully-differentiable)

DPPnet
170
Qualitative Results
Q: What is the boy holding?
DPPnet: surfboard DPPnet: bat

DPPnet
171
Qualitative Results
Q: What animal is shown?
DPPnet: giraffe DPPnet: elephant

DPPnet
172
Qualitative Results
Q: How does the woman feel?
DPPnet: happy
Q: What type of hat is she wearing?
DPPnet: cowboy

DPPnet
173
Qualitative Results
Q: How many cranes are in the image?
DPPnet: 2 (3)
Q: How many people are on the bench?
DPPnet: 2 (1)

How to combine image and question?
174

175

176

177

178

179

180

181

Multimodal Compact Bilinear Pooling
182

183

184

185

Preliminaries
245
Understanding Deep Image
Representations by Inverting Them
CVPR2015
Texture Synthesis Using
Convolutional Neural Networks
NIPS2015

A Neural Algorithm of Artistic Style
246

247

248
-NIPS2015
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Visual texture synthesis
250
Which one do you think is real?
Right one is real.
Goal of texture synthesis is to produce (arbitrarily many)
new samples from an example texture.

Results of this work
251
Right ones are given sources!

Texture Model
253
𝑋 𝑎
Input a
𝐹𝑎
1
𝐹𝑎
2
𝐹𝑎
3
𝑋 𝑏
Input b
𝐹𝑏
1
𝐹𝑏
2
𝐹𝑏
3
number of filters

Feature Correlations
254
𝑋 𝑎
Input a
𝐹𝑎
1
𝐹𝑎
2
𝐹𝑎
3
𝑋 𝑏
Input b
𝐹𝑏
1
𝐹𝑏
2
𝐹𝑏
3
number of filters
𝐺 𝑎
2
= 𝐹𝑎
2 𝑇
𝐹𝑎
2
(Gram matrix)

Feature Correlations
255
𝐺 𝑎
2
𝐹𝑎
2
𝐹𝑎
2
=
number of filters W*H
𝐹𝑎
2
𝐺 𝑎
2 = 𝐹𝑎
2 𝑇 𝐹𝑎
2
(Gram matrix)
number of filters

Texture Generation
256
𝑋 𝑎
Input a
𝐹𝑎
1 𝐹𝑎
2 𝐹𝑎
3
𝑋 𝑏
Input b
𝐹𝑏
1
𝐹𝑏
2
𝐹𝑏
3
𝐺 𝑎
1
𝐺 𝑏
1
𝐺 𝑎
1
𝐺 𝑏
1
𝐺 𝑎
1
𝐺 𝑏
1

Texture Generation
257
𝑋 𝑎
Input a
𝐹𝑎
1 𝐹𝑎
2 𝐹𝑎
3
𝑋 𝑏
Input b
𝐹𝑏
1
𝐹𝑏
2
𝐹𝑏
3
𝐺 𝑎
1
𝐺 𝑏
1
𝐺 𝑎
1
𝐺 𝑏
1
𝐺 𝑎
1
𝐺 𝑏
1
Element-wise squared loss
Total layer-wise loss function

260
-CVPR2015
Aravindh Mahendran, Andrea Vedaldi (VGGgroup)

Reconstruction from feature map
261

Reconstruction from feature map
262
𝑋 𝑎
Input a
𝐹𝑎
1 𝐹𝑎
2 𝐹𝑎
3
𝑋 𝑏
Input b
𝐹𝑏
1
𝐹𝑏
2
𝐹𝑏
3
number of filters
Let’s make this features similar!
By changing the input image!

264
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

How?
265
Style Image
Content Image
Mixed ImageNeural Art

How?
266
Style Image
Content Image
Mixed ImageNeural Art

Neural Art
268
𝑝: original photo, 𝑎: original artwork
𝑥: image to be generated
Content Style
Total loss = content loss + style loss

Deep Learning in Computer Vision

More Related Content

What's hot

Similar to Deep Learning in Computer Vision

More from Sungjoon Choi

Recently uploaded

Deep Learning in Computer Vision