Download as PDF, PPTX
![Main Ideas:
✓ Convolution, local receptive fields, shared weights
✓ Spatial subsampling
LENET [LECUN ET AL. 1998]](https://image.slidesharecdn.com/gaohuang-190826085652/75/Architecture-Design-for-Deep-Neural-Networks-I-5-2048.jpg)
![Main Ideas:
✓ ReLU (Rectified Linear Unit)
✓ Dropout
✓ Local Response Normalization, Overlapping Pooling
✓ Data Augmentation, Multiple GPUs
ALEXNET [KRIZHEVSKY ET AL. 2012]](https://image.slidesharecdn.com/gaohuang-190826085652/75/Architecture-Design-for-Deep-Neural-Networks-I-6-2048.jpg)
![Main Idea:
✓ Skip connection: promotes gradient propagation
RESNET [HE ET AL. 2016]
Identity mappings promote gradient propagation.
: Element-wise addition](https://image.slidesharecdn.com/gaohuang-190826085652/75/Architecture-Design-for-Deep-Neural-Networks-I-7-2048.jpg)
![Main Idea:
✓ Dense connectivity: creates short path in the network and encourages feature reuse.
ResNet
GoogleNet
FractalNet
DENSENET [HUANG ET AL. 2017]](https://image.slidesharecdn.com/gaohuang-190826085652/75/Architecture-Design-for-Deep-Neural-Networks-I-8-2048.jpg)
![Main Idea:
✓ Automatic architecture search using reinforcement learning, genetic/evolutional
algorithms or differentiable approaches.
AutoML is a very active research field, see www.automl.org
Neural Architecture Search [Zoph and Le, 2017 and many]](https://image.slidesharecdn.com/gaohuang-190826085652/75/Architecture-Design-for-Deep-Neural-Networks-I-18-2048.jpg)
Uploaded byWanjin Yu
Architecture Design for Deep Neural Networks I
This document summarizes Gao Huang's presentation on neural architectures for efficient inference. The presentation covered three parts: 1) macro-architecture innovations in convolutional neural networks (CNNs) such as ResNet, DenseNet, and multi-scale networks; 2) micro-architecture innovations including group convolution, depthwise separable convolution, and attention mechanisms; and 3) moving from static networks to dynamic networks that can adaptively select simpler or more complex models based on input complexity. The key idea is to enable faster yet accurate inference by matching computational cost to input difficulty.
More Related Content
![[AIoTLab]attention mechanism.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/aiotlabattentionmechanism-230406114603-e5ba0365-thumbnail.jpg?width=640&height=640&fit=bounds)
![ViT (Vision Transformer) Review [CDM]](https://cdn.slidesharecdn.com/ss_thumbnails/vitreviewcdm-201012184226-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Architecture Design for Deep Neural Networks I
![Number_Guessing_Game_Dsbsbssbzboc[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/numberguessinggamedoc1-251206215042-a076fc05-thumbnail.jpg?width=640&height=640&fit=bounds)
Architecture Design for Deep Neural Networks I
- 1. Gao Huang Assistant Professor Departmentof Automation, Tsinghua University Neural Architectures for Efficient Inference
- 2. OUTLINE 1. Macro-architecture innovationsin ConvNets 2. Micro-architecture innovations in ConvNets 3. From static network to dynamic network
- 3.
- 4. CONVOLUTIONAL NETWORKS LeNet AlexNet VGGInception ResNet DenseNet
- 5. Main Ideas: ✓ Convolution,local receptive fields, shared weights ✓ Spatial subsampling LENET [LECUN ET AL. 1998]
- 6. Main Ideas: ✓ ReLU(Rectified Linear Unit) ✓ Dropout ✓ Local Response Normalization, Overlapping Pooling ✓ Data Augmentation, Multiple GPUs ALEXNET [KRIZHEVSKY ET AL. 2012]
- 7. Main Idea: ✓ Skipconnection: promotes gradient propagation RESNET [HE ET AL. 2016] Identity mappings promote gradient propagation. : Element-wise addition
- 8. Main Idea: ✓ Denseconnectivity: creates short path in the network and encourages feature reuse. ResNet GoogleNet FractalNet DENSENET [HUANG ET AL. 2017]
- 9. REDUNDANCY IN DEEPMODELS Classifier Input Prediction Low-level features Mid-level features High-level features Classifier
- 10.
- 11. DENSE CONNECTIVITY C CC C : Channel-wise concatenationC
- 12. DENSE AND SLIM kchannels k channels k channels k channels k : Growth Rate C C C C
- 13. Model #Layers #ParametersValidation Error ResNet 18 11.7M 30.43% DenseNet 121 ?? ?? DENSE AND SLIM ARCHITECTURE
- 14. Model #Layers #ParametersValidation Error ResNet 18 11.7M 30.43% DenseNet 121 8.0M 25.03% DENSE AND SLIM ARCHITECTURE More connections, less computation!
- 15. 21.0 22.5 24.0 25.5 27.0 Top-1error(%) GFLOPs ResNet DenseNet 21.0 22.5 24.0 25.5 27.0 Top-1error(%) # Parameters(M) ResNet DenseNet ResNet-152 ResNet-101 ResNet-50 ResNet-34 ResNet-152 ResNet-101 ResNet-50 ResNet-34 DenseNet-232(k=48) DenseNet-264 DenseNet-201 DenseNet-169 DenseNet-121 DenseNet-121 DenseNet-169 DenseNet-201 DenseNet-264 DenseNet-232(k=48) RESULTS ON IMAGENET
- 16. THELOSSSURFACE https://www.cs.umd.edu/~tomg/projects/landscapes/ VGG-56 VGG-110 DenseNet-121ResNet-56 Visualizing the losslandscape of neural nets. Li, Hao, et al. NIPS 2018.a
- 17. Main Idea: ✓ Multi-scalefeature fusion: merge signals with different frequencies. Interlinked CNN (Zhou et al, ISNN’15) Neural Fabric (Saxena & Verbeek, NIPS’16) MSDNet (Huang et al, ICLR’18) HRNet (Sun et al, CVPR’19) MULTI-SCALE NETWORKS
- 18. Main Idea: ✓ Automaticarchitecture search using reinforcement learning, genetic/evolutional algorithms or differentiable approaches. AutoML is a very active research field, see www.automl.org Neural Architecture Search [Zoph and Le, 2017 and many]
- 19.
- 20. Main Idea: ✓ Splitconvolution into multiple groups Standard Convolution Group Convolution 𝑂 𝐶 × 𝐶 𝐺 𝑂 𝐶 × 𝐶 Networks using Group Convolution: ✓ AlexNet (Krizhevsky et al, NIPS’12) ✓ ResNeXt (Xie et al, CVPR’17) ✓ CondenseNet (Huang et al, CVPR’18) ✓ ShuffleNet (Zhang et al, CVPR’18) ✓ … Group Convolution
- 21. Main Idea: ✓ Splitconvolution into multiple groups, each group has one channel Networks using DSC: ✓ Xception (Chollet, CVPR’17) ✓ MobileNet (Howard et al, CVPR’18) ✓ MobileNet V2 (Sandler et al, 2018) ✓ ShuffleNet V2 (Ma et al, CVPR’19) ✓ NasNet (Zoph, CVPR’18) ✓ … Depth-wise Separable Convolution (DSC)
- 22. Main Idea: ✓ Channel-wiseattention: second order operations Squeeze and excitation network (Hu et al, CVPR’18)
- 23. Main Idea: ✓ Increasereceptive field via filter dilation Dilated convolution (Yu & Koltun, ICLR’18)
- 24. Main Idea: ✓ Learnthe offset filed for convolutional filters Deformable convolution (Dai et al, CVPR’18)
- 25.
- 26. 57. 68.7 70.5 77.8 79.6 0. 20. 40. 60. 80. 100. AlexNetGoogleNet VGG ResNet-152 DenseNet-264 Top-1Accuracy DEVELOPMENT OF DEEP LEARNING 2012 2014 2014 2015 2017
- 27.
- 28. *Photo Courtesy ofPixel Addict (CC BY-ND 2.0) BIGGER IS BETTER Bigger models are needed for those noncanonical images.
- 29. *Photo Courtesy ofWillian Doyle(CC BY-ND 2.0) BIGGER IS BETTER
- 30. Why do weuse the same expensive model for all images?
- 31. Can we usesmall&cheap models for easy images; big&expensive models for hard ones?
- 32. A NAIVE IDEAOF ADAPTIVE EVALUATION AlexNet Inception ResNet "easy" horse "hard" hoarse
- 33. CHALLENGE: LACK OFCOARSE-LEVEL FEATURES Down-sampling Linear Output Classifiers only work well on coarse-scale feature maps Nearly all computation has been done before getting a coarse feature Fine-level features Mid-level features Coarse-level features Down-sampling Down-sampling Input Classifier Classifier Classifier
- 34. SOLUTION: MULTI-SCALE ARCHITECTURE Fine-levelfeatures Mid-level features Coarse-level features
- 35. Fine-level features Mid-level features Coarse-levelfeatures SOLUTION: MULTI-SCALE ARCHITECTURE
- 36. Classifier 4Classifier 2Classifier 3Classifier 1 … … … Test Input … MULTI-SCALE FEATURES Classifiers only operate on high level features! Fine-level features Mid-level features Coarse-level features
- 37. Classifier 4Classifier 2Classifier 3Classifier 1 … … … Test Input … MULTI-SCALE DENSENET
- 38. Classifier 4Classifier 2Classifier 3Classifier 1 … … … …Classifier 2 Classifier 3Classifier 1 cat: 0.2 0.2 ≱ threshold cat: 0.4 0.4 ≱ threshold cat: 0.6 0.6 > threshold MULTI-SCALE DENSENET
- 39. MULTI-SCALE DENSENET Results 2x-5x speedupover DenseNet
- 40. Classifier 4Classifier 2Classifier 3Classifier 1 … … … Test Input … VISUALIZATION
- 41. Class: red wine Class: volcano "easy" "hard" VISUALIZATION (exitat first classifier) (exit at last classifier)
- 42. MORE RESULTS ANDDISCUSSIONS Please refer to Multi-scale dense network for efficient image classification, ICLR Oral, 2018 (Acceptance rate 2.2%, Rank 4/935)
- 43. ADAPTIVE INFERENCE ISA CHALLENGING PROBLEM How to design proper network architectures? How to effectively training dynamic networks? How to efficiently perform dynamic evaluation? Adaptive inference for object detection and segmentation? Spatial adaptive and temporal adaptive?
- 44.