Download as PDF, PPTX
![Agenda
• AutoML @ NeurIPS 2018
• 1
“Massively Parallel Hyperparameter Tuning” [Li, et al.]
• 2
“Neural Architecture Optimization” [Luo, et al.]
2](https://image.slidesharecdn.com/neuripsautoml-190126062134/85/AutoML-in-NeurIPS-2018-2-320.jpg)
![Today’s Papers
• Hyperparameter Optimization
“Massively Parallel Hyperparameter Tuning” [Li, et al.]
• Neural Architecture Search
“Neural Architecture Optimization” [Luo, et al.]
9](https://image.slidesharecdn.com/neuripsautoml-190126062134/85/AutoML-in-NeurIPS-2018-9-320.jpg)
![Related Work: Successive Halving (SHA)
13
• ( )
•
• Hyperband [16, Li, et al.]](https://image.slidesharecdn.com/neuripsautoml-190126062134/85/AutoML-in-NeurIPS-2018-13-320.jpg)
![Related Work: NASNet Search Space
30
• [16, Zoph et al.]
• NASNet Space ImageNet SOTA [17, Zoph et al.]
–
– ResNet ResBlock](https://image.slidesharecdn.com/neuripsautoml-190126062134/85/AutoML-in-NeurIPS-2018-30-320.jpg)
Uploaded byShotaro Sano
AutoML in NeurIPS 2018
The document discusses two papers presented at NeurIPS 2018 on AutoML: 1) "Massively Parallel Hyperparameter Tuning" which proposes an asynchronous successive halving algorithm to parallelize hyperparameter tuning. This approach improves on synchronous successive halving by avoiding mis-promoting configurations. 2) "Neural Architecture Optimization" which uses a neural network to learn embeddings of neural network architectures for optimization. The approach achieves state-of-the-art results on CIFAR-10 and shows good transferability to other tasks.
More Related Content
![[DL輪読会]Randomized Prior Functions for Deep Reinforcement Learning](https://cdn.slidesharecdn.com/ss_thumbnails/20190614-190614005503-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DL輪読会]Deep Neural Networks as Gaussian Processes](https://cdn.slidesharecdn.com/ss_thumbnails/dl0216okamoto2162-180323031830-thumbnail.jpg?width=640&height=640&fit=bounds)
AutoML in NeurIPS 2018
- 1. AutoML NeurIPS 2018 Yomikai@ PFN (2019/01/26) Shotaro Sano
- 2. Agenda • AutoML @NeurIPS 2018 • 1 “Massively Parallel Hyperparameter Tuning” [Li, et al.] • 2 “Neural Architecture Optimization” [Luo, et al.] 2
- 3. What is AutoML? •Hyperparameter Optimization (HPO): • Neural Architecture Search (NAS): NN • Meta Learning: 3 “The user simply provides data, and the AutoML system automatically determines approach that performs best for particular applications.” Futter et al., 2018 AutoML: Methods, Systems, Challenges
- 4. AutoML @ NeurIPS2018 • AutoML • HPO, NAS, Meta Learning • Meta Learning • AutoML Meetup @ Google AI • – System for ML – Meta Learning – NeurIPS 2018 Competition Track 4
- 5. Hyperparameter Optimization @NeurIPS 2018 • Bayesian Optimization Meta-learning • 10 @ – “Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior” – “Automating Bayesian Optimization with Bayesian Optimization” – etc. • Systems for ML – “Massively Parallel Hyperparameter Tuning” – “Population Based Training as a Service” – etc. 5
- 6. Neural Architecture Search@ NeurIPS 2018 • , Semantic Segmentation • 4+ @ – “Neural Architecture Optimization” – “Neural Architecture Search with Bayesian Optimization and Optimal Transport” – etc. • 2019 AutoDL 6
- 7. Meta Learning @NeurIPS 2018 • Keywords: Model-agnostic Meta-Learning, Few-shot Learning, Transfer Learning, etc. • 20 @ – “Bayesian Model-Agnostic Meta Learning” – “Meta-Reinforcement Learning of Structured Exploration Strategies” – etc. • Meta Learning – HPO NAS 7
- 8. Competition Track: AutoML3@ NeurIPS 2018 • AutoML3: – – • Tree-parzen Estimator + LightGBM/XGBoost 8 Train&Test Task A Task B Task C Task D Task E
- 9. Today’s Papers • HyperparameterOptimization “Massively Parallel Hyperparameter Tuning” [Li, et al.] • Neural Architecture Search “Neural Architecture Optimization” [Luo, et al.] 9
- 10. Systems for MLWorkshop (NeurIPS 2018) Massively Parallel Hyperparameter Tuning
- 11. 11 Blackbox Optimization such as GridSearch Bayesian Optimization … Hyperparameter Tuner LR: 0.00001 ~ 0.1 Dropout: 0.0 ~ 0.5
- 12. Massively Parallel HyperparameterTuning 12 • • – – Optuna – Successive Halving
- 13. Related Work: SuccessiveHalving (SHA) 13 • ( ) • • Hyperband [16, Li, et al.]
- 14. Related Work: SuccessiveHalving (SHA) 14 N resource config config config config config config config config config
- 15. resource config config config config config config config config config Related Work: SuccessiveHalving (SHA) 15 N / η η ( η=3 )
- 16. Related Work: SuccessiveHalving (SHA) 16 N / η2 η2 resource config config config config config config config config config ( η=3 )
- 17. Related Work: SuccessiveHalving (SHA) 17 resource config config config config config config config config config
- 18. Simple and Powerful! 18 SuccessiveHalving Random Search Faster & better!
- 19. rung 2 rung 111 1 1 2 2 3 resource 19 Related Work: Synchronous SHA parallelize rung 3 config config config config 1 2 3 worker 1 1 1 2 worker 2 1 rung 1 rung 2 rung 3
- 20. • : rung •/ 20 Problem with Synchronous SHA Synchronous Asynchronous ( ) Config worker 1 1 1 2 worker 2 1 2 31 rung 1 rung 2 rung 3
- 21. • “ ”1/η config • config config 21 Proposed Method: Asynchronous SHA 1 1 ? 1 2 1 2 ? ( η=2 ) 1 2 1
- 22. • PROS: rung •CONS: mis-promote – Config – Mis-promote N -1/2 22 Proposed Method: Asynchronous SHA PROS (Massively Parallel Hyperparameter Tuning) 1 1 2 1 2 31 1 2 1 1 2 31 2 3 1
- 23. 23 Experiments: Single WorkerSetting Mis-promote Synchronous ( ) Synchronous SHA Asynchronous SHA
- 24. 24 Experiments: Multi WorkerSetting config ( ) Synchronous SHA Asynchronous SHA
- 25.
- 26.
- 27. Neural Architecture Search • •ImageNet SOTA 27
- 28. 28 Chain-structured Space Tree-structured Space Multi-branchNetwork Cell Search Space … Full Evaluation Lower Fidelities Learning Curve Extrapolation One-shot Architecture Search … Reinforcement Learning Evolutionary Search Bayesian Optimization Monte Carlo Tree Search …
- 29. Neural Architecture Optimization 29 • • –CIFAR (+cutout) SOTA – 2018 PFN Neural Architecture Search
- 30. Related Work: NASNetSearch Space 30 • [16, Zoph et al.] • NASNet Space ImageNet SOTA [17, Zoph et al.] – – ResNet ResBlock
- 31. Proposed Method: NAONet • –? • NASNet ( ) 31
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39. 39 Experiments: SOTA onCIFAR-10 SOTA
- 40. 40 Experiments: Transferring CIFAR-10to CIFAR-100 SOTA
- 41. 41 Experiments: Transferring PTBto WikiText-2
- 42.
- 43. Define-by-run style hyperparametersearch framework. ! Fat config & poor control syntax! " High modularity! " High representation power!