Convolutional Neural Network (CNN) presentation from theory to code in Theano

M&S
Convolutional Neural
Network from Theory
to Code
Seongwon Hwang

M&S
Tensor :
(Index Notation)
ky
kl
ijy
ijklA
kj
a
ijA iA
ki
jjB

M&S
Range
𝑎 𝑏
𝑐 𝑑
𝑎 𝑏 𝑐
𝑑 𝑒 𝑓
𝑔 ℎ 𝑖
𝑎 𝑏
𝑒 𝑓
𝑐 𝑑
𝑔 ℎ
𝑖 𝑗
𝑚 𝑛
𝑘 𝑙
𝑜 𝑝
𝐴𝑖𝑗
i = 2, j = 2
i = 3, j = 3 i = 4, j = 4
𝑎 𝑏
𝑐 𝑑
𝑒 𝑓
i = 3, j = 2

M&S
Free index : unrepeated index
𝑦1 = 𝑎11 𝑥1+𝑎12 𝑥2
𝑦2 = 𝑎21 𝑥1+𝑎22 𝑥2
𝑦 𝑘 = 𝑎 𝑘1 𝑥1+𝑎 𝑘2 𝑥2, k = 1,2


2
1i
ikik xay
),( 21 yyy 

M&S
Dummy index : repeated index
1,2kixay ikik  ,
Free index Dummy index


2
1i
ikik xay
𝑦1 = 𝑎11 𝑥1+𝑎12 𝑥2
𝑦2 = 𝑎21 𝑥1+𝑎22 𝑥2

M&S
Example
iji BA jj BABA 2211 =
212111,1 BABAj 
222121,2 BABAj 
),( 222121212111 BABABABA iji BA =
i, j = 1,2

M&S
Example
ijjA 332211 iii AAA 
),,( 333322311233222211133122111 AAAAAAAAA 
133122111,1 AAAi 
233222211,2 AAAi 
333322311,3 AAAi 
=
i, j = 1,2,3

M&S
Example – Determinant
bcad
dc
ba
aa
aa
A  detdetdet
2221
1211
bcad
aaaa
aaeaaeaaeaae
aae jiij




00 21122211
221222211221221112211111
21

M&S
Example – Determinant
)(...det 21... matrixNNaaaeA Nkjikij 

M&S
Derivatives
kjjkjj
xx
y
y
x
y
y






2
,, ,
1,2,3jφ
x
φ
z
φ
y
φ
x
φ
φgrad jj










 ),,( ,
Example – Gradient

M&S
CNN Tensor notation in Theano
- Input Images -
4D tensor
1D tensor
[number of feature maps at layer m, number of feature maps at layer m-1,
filter height, filter width]
ij
klx
op
qrW
mb
[ i, j, k, l ] =
[ o, p, q, r ] =
[ m ] =
- Weight -
- Bias -
[n’th feature map number]
[mini-batch size, number of input feature
maps, image height, image width]

M&S
Network
Theory I

M&S
Convolution?





daatwax
twxty
)()(
))(()(




a
anwaxny ][][][
- Continuous Variables -
- Discrete Variables -

M&S
Convolution?




a
anwaxny ][][][
- Discrete Variables -
][][ awax ][][ awax 
)]([][ nawax 
Y-axis transformation

M&S
Y축
X축
Convolution?
X축
X축

M&S
Cross-Correlation?




a
nawaxnwxny ][][])[(][
- Discrete Variables (In real number) -
][][ awax ][][ nawax n step move
★

M&S
Convolution VS. Cross-Correlation
Cross-CorrelationConvolution

M&S
Cross-Correlation in 2D
Output (y) Kernel (w) Input (x)
 






n m
nmwjnimx
jiwxjiy
],[],[
],)[(],[

M&S
Intuition for Cross-Correlation
wx
★

M&S
wx
★

M&S
wx
★
Input
)( mapFreature
neuronHidden
fieldrecptiveLocal
filterorkernel

M&S
Network
Theory II

M&S
CNN overall procedures
Input Image Convolutional Layer Pooling MLP Softmax Output

M&S
CNN overall procedures
Input Image
Convolutional
Layer
Pooling
MLP
Softmax Output

M&S
Input Image
Input Image
1D - Input neurons
2D - Input neurons

M&S
Convolutional Layer
Input Image
Convolutional
Layer
Input neurons
Hidden neuron

M&S
Traditional Neural Network
Input layer Hidden layer Output layer

M&S
CNN - Sparse Connectivity
Input layer Hidden layer
(Feature map)
Output layer

M&S
CNN – Dimension shrinkage ( + Pooling)
Input layer Output layerHidden layer
(Feature map)

M&S
Cross-Correlation Input (or Hidden) layer with weight
Input layer Output layer
1W
2W
3W
'3W
'2W
'1W
Hidden layer
(Feature map)(Receptive field)

M&S
Shared Weight representation
Input layer Output layer
1W
2W
3W
'3W
'2W
'1W
Hidden layer
(Feature map)

M&S
Multiple Feature maps
Input Image
Convolutional
Layer
Input neurons
First hidden layer

M&S
Max Pooling
Input Image
Convolutional
Layer
Pooling
Max pooling with
2x2 filters and stride 2

M&S
Why pooling?
Input Image
Convolutional
Layer
Pooling
1. Low computation 2. Translation invariance
3. Transformation invariance
4. Scaling invariance

M&S
Several types of Pooling
Input Image
Convolutional
Layer
Pooling

M&S
Transform data dimension before MLP
Input Image
Convolutional
Layer
Pooling
MLP
1D - Output neurons
2D - Output neurons

M&S
Multilayer Perceptron (MLP)
Input Image
Convolutional
Layer
Pooling
MLP
Input layer Hidden layer 1 Hidden layer 2

M&S
Softmax Output
Input Image
Convolutional
Layer
Pooling
MLP
Softmax Output
Hidden layer 2 Output layer

M&S
Intuition for CNN
Input Image
Convolutional
Layer
Pooling
MLP
Softmax Output

M&S
Network
Code in Theono

M&S
Input Image
Input Image
- Input Images -
4D tensor
[mini-batch size, number of input feature
maps, image height, image width]ij
klx
5
...
28
28
500
7
[ i, j, k, l ] =
Mini batch 1
5
...
28
28
500
8
Mini batch 100
. . .
50,000 images
in the training data

M&S
Weight tensor
Input Image
Convolutional
Layer
4D tensor
[number of feature maps at layer m, number of feature maps at layer m-1,
filter height, filter width]
op
qrW [ o, p, q, r ] =
- Weight -

M&S
Exercise for Input and Weight tensor
11
11x
11
11W
Input layer
Convolutional layer 1 Convolutional layer 2
[ 1, 1, 1, 1 ]
[ 1, 1, 1, 1 ]

M&S
Code for Convolutional Layer
28
28
8
def evaluate_lenet5(learning_rate=0.1,
n_epochs=2, dataset=‘minist.pkl.gz’,
nkerns=[20, 50], batch_size=500):
LeNetConvPoolLayer
image_shape=(batch_size, 1, 28, 28)
filter_shape=(nkerns[0], 1, 5, 5)
poolsize=(2, 2)
image_shape=(batch_size, nkerns[0], 12, 12)
filter_shape=(nkerns[1], nkerns[0], 5, 5)
poolsize=(2, 2)
Layer0 – Convolutional layer 1
Layer1 – Convolutional layer 2
5
5
20
24
24
20
5
5
12
12
8
8
4
4
20 50 50 50
28 – 5 + 1 = 24
Convolution
24 / 2 = 12
Pooling
12 – 5 + 1 = 8
Convolution
8 / 2 = 4
Pooling
Class

M&S
Zero-Padding Input layer Output layerHidden layer
(Feature map)
Input Image
Convolutional
Layer
Pooling

M&S
(Feature map)
0
0

M&S
(Feature map)
0
0
0
0

M&S
Zero-Padding
No zero-padding Zero-padding 1 Zero-padding 2
Dimension Reduction Dimension Equality Dimension Increase
Zero-
padding in
Theano
Default in
Theano

M&S
Code for Zero-Padding setting in Theano Library
conv.py (In Library File)
def conv2d(input, filters, image_shape=None, filter_shape=None,
border_mode=‘valid’, subsample=(1, 1), **kargs):
‘valid’ ‘full’
We should modify dimension changes in code by ourselves!
Image_shape + filter_shape – 1Image_shape - filter_shape + 1
28 + 5 – 1 = 3228 - 5 + 1 = 24

M&S
1. Border in Pooling
No problem in Border Ingore_border = False Ingore_border = True

M&S
1. Border in Pooling (Code)
No problem in Border
Ingore_border = False Ingore_border = True
pooled_out =
downsample.max_pool_2d(input=conv_out,
ds=poolsize, ignore_border=True)
LeNetConvPoolLayer
Class
Default in Theano library is False!

M&S
2. Stride in Pooling
Default in Theano
stride size = poolsize
Unduplicated!

M&S
2
Default in Theano
stride size = poolsize
Unduplicated!

M&S
2
Default in Theano
stride size = poolsize = (2,2)
Unduplicated!

M&S
stride size = (1, 1)
6

M&S
2. Stride in Pooling (Code)
76 8
6 7 8
3 3 4
pooled_out =
ds=poolsize, ignore_border=True)
pooled_out =
ds=poolsize, ignore_border=True, st = (1,1))

M&S
Activation Function in Convolutional Layer
self.output=T.tanh(pooled_out +
self.b.dimshuffle(‘x’,0,‘x’,‘x’)
Convolution Pooling Activation
LeNetConvPoolLayer
Class

M&S
Dimension Reduction 2D  1D
Input Image
Convolutional
Layer
Pooling
MLP
layer2_input = layer1_input.output.flatten(2)
1D - Output neurons
2D - Output neurons

M&S
Code for MLP in Theano
Input Image
Convolutional
Layer
Pooling
MLP layer2 = HiddenLayer( rng,
input=layer2_input,
n_in-nkerns[1] * 4 * 4,
n_out=500,
activation = T.tanh)
HiddenLayer
Class
Last output size for C+P
Number of node at Hidden layer
Activation function at Hidden layer
***In order to extend the number of Hidden Layer in MLP,
We need to make layer3 by copying this code***

M&S
Code for Softmax Output in Theano
Input Image
Convolutional
Layer
Pooling
MLP
Softmax Output
layer3 =
LogisticRegression( input=layer2.output,
n_in=500,
n_out=10)
LogisticRegression
Class
cost = layer3.negative_log_likelihood(y)
Number of node at previous
Hidden layer
Final Output size
Ex) 0, 1, 2, ,,, 9
Softmax activation function

M&S
CNN application in bioinformatics problem

Convolutional Neural Network (CNN) presentation from theory to code in Theano

More Related Content

What's hot

Similar to Convolutional Neural Network (CNN) presentation from theory to code in Theano

Recently uploaded

Convolutional Neural Network (CNN) presentation from theory to code in Theano

Editor's Notes