Nas net where model learn to generate models
•
1 like•317 views
Walk through NAS net and a few papers applied NAS search space as well as the approach for architecture search to achieve SOTA in accuracy for ImageNet
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Nas net where model learn to generate models
Similar to Nas net where model learn to generate models (20)
Recently uploaded
Recently uploaded (20)
Nas net where model learn to generate models
- 1. NAS Net Where Model can be generated by a model PHAM QUANG KHANG
- 2. History of remarkable CNNs models 2019/8/17 PHAM QUANG KHANG 2 Google Net - Inception ResNET
- 3. Direction of developing models 2019/8/17 PHAM QUANG KHANG 3 Model ImageNet Accuracy Million Mult-Adds Million Parameters GoogleNet 69.8% 1550 6.8 MobileNetV1 70.6 575 4.2 ShuffleNet(1.5) 71.5% 292 3.4 ShuffleNet (x2) 73.7% 524 5.4 NasNet-A 74% 564 5.3 MobileNetV2 72.0% 300 3.4 Go deep for greater accuracy Smaller, less FLOP for real life application
- 4. For a model to be used in real life app? 1. Accuracy: of course no one wants a low accurate app 2. Latency: of course it has to be fast enough so that you don’t take a picture and wait for a few minutes for some result to come out, at least in inference mode 3. Easy solution: design a model that can achieve close to SOTA in accuracy but also fast enough like less parameter, less computation etc 4. Example of these “easy” solution: ① Inception to reduce parameter from VGG ② MobileNet to reduce params and computation from RESNet 5. Do we reach limit of human in designing new model? 2019/8/17 PHAM QUANG KHANG 4
- 5. Search in space of models? 1. Possible parameters to search in space of model: ① Params within each Conv layer: stride, kernel size, padding etc ② Number of Conv, Pooling layers ③ Order of layers and connection (skip connection like in RESNet) ④ Any else? 2. Search method: ① Random search ② Grid search ③ Reinforcement Learning 2019/8/17 PHAM QUANG KHANG 5 Use RNNs as controller to generate models with REINFORCE rule to optimize the reward of generating “good” architecture Barret Zoph, Quoc V.Le: Neural Architecture Search with Reinforcement Learning (ICLR 2017)
- 6. NASNet: transferable architectures 1. Contribution: • A search space enables transferability: NASNet search space • Regularization technique: ScheduledDropPath • NASNet architecture with performance of • 82.7% Top 1 and 96.2% top-5 on ImageNet • 28% less computation to current SOTA • Used as feature extraction for Faster-RCNN and surpass 4.0% to SOTA for COCO 2. Approach: • Define a narrow search space for feasibility • Use Neural Architecture Search (RNNs with RL) • Search a generic architecture in small data set then transfer to larger 3. Limitation (Khang’s opinion): • Still a lot of pre-defined constraint • Huge effort for search process • Marginal result from random search from RL 2019/8/17 PHAM QUANG KHANG 6 Learning Transferable Architectures for Scalable Image Recognition Barret et. al. CVPR 2018
- 7. Model concept Define an architecture: • Predetermined as composed of conv cells repeated many times with each cell has same architecture • Contains: conv cells return feature map of same dimension, conv cells reduce height and width by factor of 2 2019/8/17 PHAM QUANG KHANG 7
- 8. NASNet search space 2019/8/17 PHAM QUANG KHANG 8 • Element-wise add • Concat Step 1 Step 2 Step 3 Step 4 Step 5
- 9. Experiment and results (1) Search setup: 500 GPUs for 4 days to generate good architectures 2019/8/17 PHAM QUANG KHANG 9 Personal opinion: Marginal result from random search from RL ScheduledDropPath: each path in the cell is dropped out with a probability that is linearly increased over the course of training 0.015
- 10. Experiment and results (2) Search setup: 500 GPUs for 4 days to generate good architectures 2019/8/17 PHAM QUANG KHANG 10
- 11. MnasNet/ProxyLessNas: NAS for mobile MnasNet: platform aware ProxyLessNas: search on task and hardware 2019/8/17 PHAM QUANG KHANG 11 • Learn architecture for task and hardware • No restriction on repeat blocks
- 12. Latency on mobile for ImageNet MNasNet ProxyLessNas 2019/8/17 PHAM QUANG KHANG 12 Agreed SOTA for ImageNet on Mobile?
- 13. Proposed architectures MNasNet ProxyLessNas 2019/8/17 PHAM QUANG KHANG 13
- 14. EfficientNet: Model Scaling for CNNs Uniformly scales all dimensions of depth/width/resolution to get smaller architecture yet still achieved SOTA on ImageNet 2019/8/17 PHAM QUANG KHANG 14
- 15. Effect of scaling and constraint for search Use 1 base-line and scale up with 3 dimensions: 2019/8/17 PHAM QUANG KHANG 15 Constraints and objective for search space (using MNasNet method to search) Search Space Optimization goal
- 16. Use base of NasNet to win on all level 2019/8/17 PHAM QUANG KHANG 16
- 17. Thank you 2019/8/17 PHAM QUANG KHANG 17