Squeeze methods for deep learning model. - use pre-training - small architecture

1. Squeeeze Models
On Deep Learning

2. Going Deeper & Larger

3. Operation Energy [pJ] Relative Cost
32 bit int ADD 0.1 1
32 bit float ADD 0.9 9
32 bit Register File 1 10
32 bit int MULT 3.1 31
32 bit float MULT 3.7 37
32 bit SRAM Cache 5 50
32 bit DRAM Memory 640 6400
Energy Consume
Network Compression and Speedup, Shuochao Yao, Yiwen Xu, Daniel Calzada

4. No Problem!

5. Mobile is Always problem
1GB
...
1GB
3830 mAh
4GB
4000 mAh

7. Singular Value Decomposition
Denton, Emily L., et al. "Exploiting linear structure within convolutional networks for efficient
evaluation." Advances in Neural Information Processing Systems. 2014.

8. Singular Value Decomposition
Storage for 𝑊: 𝑂(𝑚𝑘)
Storage for ෩𝑊: 𝑂(𝑚𝑡 + 𝑡 + 𝑡𝑘)
Truncated SVD : minimize(A – A’)
Use for Compress FC Layer

9. Singular Value Decomposition
http://darkpgmr.tistory.com/106
600x367
100
50
* 원래는… 데이터 압축, 노이즈 제거 사용

10. Singular Value Decomposition
• Trained on ImageNet 2012 database, then compressed
• 5 convolutional layers, 3 fully connected layers, softmax output layer

11. Pruning
Han, Song, et al. "Learning both weights and connections for efficient
neural network." NIPS. 2015.
https://www.slideshare.net/embeddedvision/techniques-for-efficient-implementation-of-deep-neural-networks-a-presentation-from-stanford

12. Deep Compression
Pruning Method
Prune the weights of which magnitudes are less than a threshold 𝝉

13. Deep Compression

14. Pruning and splicing
Guo, Yiwen, et al. "Dynamic Network Surgery for Efficient DNNs." NIPS. 2016.

15. Pruning Problem
Incorrect pruning is Irreversible => need splicing
previous this work
Less iter, better result

16. Pruning and Splicing
⨀ is the element-wise product
Train
Network
Update
T_k
Update
W_k
* At start, T_k is all 1

17. Pruning and Splicing

18. SQUEEZE NET
Iandola, Forrest N., et al. "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size."

19. SqueezeNet
Use pre-trained model
VS
Design Small CNN Architecture

20. SqueezeNet

21. SqueezeNet

22. Also Densenet…