Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CNN Structure: from LeNet to ShuffleNet
Dalin Zhang
School of CSE, UNSW
17/Jul/2017
Preliminary
Local receptive fields
Shared weights and biases
Multiple feature maps
Subsampling maps
LeNet: Hello World!
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner,
Gradient-based learning applied to document recognitio...
ILSVRC
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
More than 1.2 Million Images 1000 classes
Impressive new...
AlexNet: ILSVRC 2012 winner
C(11x11)P-C(5x5)P-C(3x3)-C(3x3)-C(3x3)P
Max pooling
Relu activation function
8 layers
A. Krizh...
VGGNet: ILSVRC 2014 2nd
All convolutional layer kernels are of size 3x3
MaxPooling of size 2x2 is done after 2 or 3 layers...
GoogleNet: ILSVRC 2014 Winner
Let the network choose the kernel size itself
Pointwise convolution (1x1 convolution) reduce...
ResNet: ILSVRC 2015 Winner
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
Deep Residual Learning for Image Recogni...
ResNet: ILSVRC 2015 Winner
Introduce skip connections
Pointwise convolution reduce and restore feature maps
152 layers, to...
ResNet: ILSVRC 2015 Winner
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
Deep Residual Learning for Image Recogni...
Xception: Depthwise Separable Convolutions
François Chollet
Xception: Deep Learning with Depthwise Separable Convolution (...
Xception: Depthwise Separable Convolutions
François Chollet
Xception: Deep Learning with Depthwise Separable Convolution (...
ResNeXt: Group Convolutions ILSVRC 2016 2nd
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, Kaiming He
Aggregated Re...
ResNeXt: Group Convolutions ILSVRC 2016 2nd
More clear case
Group convolution reduce the complexity compared to the simila...
ShuffleNeXt: pointwise group conv+channel shuffle
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin and Jian Sun
ShuffleNet: An Extr...
ShuffleNeXt: pointwise group conv+channel shuffle
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin and Jian Sun
ShuffleNet: An Extr...
Summary
 Stack simple structures
 Skip connection
 Pointwise convolution
 Depthwise convolution
 Group convolution
 ...
Upcoming SlideShare
Loading in …5
×

CNN Structure: From LeNet to ShuffleNet

301 views

Published on

slides for group learning

Published in: Technology
  • Be the first to comment

CNN Structure: From LeNet to ShuffleNet

  1. 1. CNN Structure: from LeNet to ShuffleNet Dalin Zhang School of CSE, UNSW 17/Jul/2017
  2. 2. Preliminary Local receptive fields Shared weights and biases Multiple feature maps Subsampling maps
  3. 3. LeNet: Hello World! Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE 86(11): 2278–2324, 1998. C(5x5)-P(2x2) pair repeat Average pooling Sigmoid or tanh activation function
  4. 4. ILSVRC ImageNet Large Scale Visual Recognition Challenge (ILSVRC) More than 1.2 Million Images 1000 classes Impressive new CNN structures from ILSVRC www.image- net.org/challenges/LSVRC/
  5. 5. AlexNet: ILSVRC 2012 winner C(11x11)P-C(5x5)P-C(3x3)-C(3x3)-C(3x3)P Max pooling Relu activation function 8 layers A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012
  6. 6. VGGNet: ILSVRC 2014 2nd All convolutional layer kernels are of size 3x3 MaxPooling of size 2x2 is done after 2 or 3 layers of convolutions Pooling stride is 2 Stacking building blocks of the same shape K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015
  7. 7. GoogleNet: ILSVRC 2014 Winner Let the network choose the kernel size itself Pointwise convolution (1x1 convolution) reduce parameters 22 layers C. Szegedy et al., Going deeper with convolutions, CVPR 2015
  8. 8. ResNet: ILSVRC 2015 Winner Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep Residual Learning for Image Recognition, CVPR 2016 (Best Paper) ResNet: 152 layers
  9. 9. ResNet: ILSVRC 2015 Winner Introduce skip connections Pointwise convolution reduce and restore feature maps 152 layers, top-5 error rate 3.57% vs. 5.1% of human expert Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep Residual Learning for Image Recognition, CVPR 2016 (Best Paper) Directly performing 3x3 convolutions: Parameters: 256x256x3x3 ~ 600K Residual module structure: Parameters: 64x256x1x1 ~ 16K 64x64x3x3 ~ 36K 256x64x1x1 ~ 16K Total ~70K
  10. 10. ResNet: ILSVRC 2015 Winner Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep Residual Learning for Image Recognition, CVPR 2016 (Best Paper) Problem: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Deeper network is not easy to optimize. Cause: In some cases some neuron can “die”(output zero) in the training and become ineffective/useless. This can cause information loss, sometimes very important information. Solution: Skip connections carry important information in the previous layer to the next layers.
  11. 11. Xception: Depthwise Separable Convolutions François Chollet Xception: Deep Learning with Depthwise Separable Convolution (2017 Apr) Important Hypothesis: The mapping of cross-channels correlations and spatial correlations in the feature maps of convolutional neural networks can be entirely decoupled. output input
  12. 12. Xception: Depthwise Separable Convolutions François Chollet Xception: Deep Learning with Depthwise Separable Convolution (2017 Apr)
  13. 13. ResNeXt: Group Convolutions ILSVRC 2016 2nd Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, Kaiming He Aggregated Residual Transformations for Deep Neural Networks (2017 Apr) Introduce group convolution to the ResNet unit, thus introduce a new dimension “cardinality” (the number of groups) to ResNet.
  14. 14. ResNeXt: Group Convolutions ILSVRC 2016 2nd More clear case Group convolution reduce the complexity compared to the similar ResNet structure. Gain better performance at the same complexity Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, Kaiming He Aggregated Residual Transformations for Deep Neural Networks (2017 Apr)
  15. 15. ShuffleNeXt: pointwise group conv+channel shuffle Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin and Jian Sun ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Channel shuffle: help information flow across feature maps (B, g x n, H, W) – reshape(B, g, n, H, W) – transpose(B, n, g, H, W) – reshape(B, g, n, H, W)
  16. 16. ShuffleNeXt: pointwise group conv+channel shuffle Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin and Jian Sun ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Pointwise group convolution: Reduce complexity allowing more feature maps, especially important to small networks
  17. 17. Summary  Stack simple structures  Skip connection  Pointwise convolution  Depthwise convolution  Group convolution  Channel shuffle

×