The document discusses MobileNets, a framework for efficient convolutional neural networks aimed at mobile vision applications, emphasizing the need for low power, real-time performance, and small model sizes in safety-critical gadgets. Key techniques for optimizing small deep neural networks include removing fully-connected layers, kernel and channel reduction, depthwise separable convolutions, and varying hyperparameters like width and resolution multipliers. Depthwise separable convolutions significantly reduce computational costs while maintaining performance, making them suitable for resource-constrained environments.

1. MobileNets:
Efficient Convolutional Neural Networks for
MobileVision Applications
29th October, 2017
PR12 Paper Review
Jinwon Lee
Samsung Electronics

2. MobileNets

3. Key Requirements for Commercial Computer
Vision Usage
• Data-centers(Clouds)
 Rarely safety-critical
 Low power is nice to have
 Real-time is preferable
• Gadgets – Smartphones, Self-driving cars, Drones, etc.
 Usually safety-critical(except smartphones)
 Low power is must-have
 Real-time is required
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

4. What’s the “Right” Neural Network for Use in a
Gadget?
• Desirable Properties
 Sufficiently high accuracy
 Low computational complexity
 Low energy usage
 Small model size
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

5. Why Small Deep Neural Networks?
• Small DNNs train faster on distributed hardware
• Small DNNs are more deployable on embedded processors
• Small DNNs are easily updatable Over-The-Air(OTA)
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

6. Techniques for Small Deep Neural Networks
• Remove Fully-Connected Layers
• Kernel Reduction ( 3x3  1x1 )
• Channel Reduction
• Evenly Spaced Downsampling
• Depthwise Separable Convolutions
• Shuffle Operations
• Distillation & Compression
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

7. Key Idea : Depthwise Separable Convolution!

8. Recap – Convolution Operation
1 1 1 1 0
1 1 1 1 0
0 0 0 1 1
0 1 1 1 0
0 1 1 0 0
0 1 1 0 1
0 1 1 0 0
0 0 1 1 0
0 0 1 1 1
1 1 1 0 0
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0
-1 0 0
0 1 0
0 0 -1
0 -1 0
-1 1 -1
0 -1 0
1 0 1
0 1 0
1 0 1
-1 0 -1
0 1 0
0 0 -1
0 -1 0
-1 1 0
0 -1 0
1 0 1
0 -1 0
1 0 1
1 -1 1
0 -1 -1
3 1 0
3 0 1
-2 0 2
0 2 3
=
convolution
Input channel : 3 Output channel : 2# of filters : 2

9. Recap –VGG, Inception-v3
• VGG – use only 3x3 convolution
 Stack of 3x3 conv layers has same effective receptive
field as 5x5 or 7x7 conv layer
 Deeper means more non-linearities
 Fewer parameters: 2 x (3 x 3 x C) vs (5 x 5 x C)
 regularization effect
• Inception-v3
 Factorization of filters

10. Why should we always consider all channels?

11. Standard Convolution
Depthwise convolution
Figures from http://machinethink.net/blog/googles-mobile-net-architecture-on-iphone/

12. Depthwise Separable Convolution
• Depthwise Convolution + Pointwise Convolution(1x1 convolution)
Depthwise convolution Pointwise convolution
Figures from http://machinethink.net/blog/googles-mobile-net-architecture-on-iphone/

13. Standard Convolution vs Depthwise Separable
Convolution

14. Standard Convolution vs Depthwise Separable
Convolution
• Standard convolutions have the computational cost of
 DK x DK x M x N x DF x DF
• Depthwise separable convolutions cost
 DK x DK x M x DF x DF + M x N x DF x DF
• Reduction in computations
 1 / N + 1 / DK
2
 If we use 3x3 depthwise separable convolutions, we get between 8 to 9 times
less computations
DK : width/height of filters
DF : width/height of feature maps
M : number of input channels
N : number of output channels(number of filters)

15. Depthwise Separable Convolutions

16. Model Structure

17. Width Multiplier & Resolution Multiplier
• Width Multiplier –Thinner Models
 For a given layer and width multiplier α, the number of input channels M
becomes αM and the number of output channels N becomes αN – where α
with typical settings of 1, 0.75, 0.6 and 0.25
• Resolution Multiplier – Reduced Representation
 The second hyper-parameter to reduce the computational cost of a neural
network is a resolution multiplier ρ
 0<ρ≤1, which is typically set of implicitly so that input resolution of network
is 224, 192, 160 or 128(ρ = 1, 0.857, 0.714, 0.571)
• Computational cost:
DK x DK x αM x ρDF x ρDF + αM x αN x ρDF x ρDF

18. Width Multiplier & Resolution Multiplier

19. Experiments – Model Choices

20. Model Shrinking Hyperparameters

21. Model Shrinking Hyperparameters

22. Results

23. Results
PlaNet : 52M parameters, 5.74B mult-adds
MobilNet : 13M parameters, 0.58M mult-adds

24. Results

25. Results

26. Tensorflow Implementation
https://github.com/Zehaos/MobileNet/blob/master/nets/mobilenet.py