Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quoc Le at AI Frontiers : Automated Machine Learning

291 views

Published on

Traditional machine learning systems are hand-designed and tuned by machine learning experts. To scale up the impact of machine learning to many real-world applications, we must figure out a way to automate the designing process of these pipelines. In this talk, I will discuss the use of machine learning to automate the process of designing neural architectures and data augmentation strategies (Neural Architecture Search and AutoAugment).

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Quoc Le at AI Frontiers : Automated Machine Learning

  1. 1. AutoML - Automated Machine Learning Quoc Le @quocleix Joint work with many colleagues in the Google Brain team
  2. 2. Current: Solution = ML Expertise + Data + Computation
  3. 3. Current: Solution = ML Expertise + Data + Computation But can we turn this into: Solution = Data + 100X Computation
  4. 4. Data processing Machine Learning Model Data
  5. 5. Data processing Machine Learning Model Data
  6. 6. Importance of architectures for Vision ● Designing neural network architectures is hard ● Lots of human efforts go into tuning them ● Can we try and learn good architectures automatically? Canziani et al, 2017
  7. 7. Convolutional Architectures Krizhevsky et al, 2012
  8. 8. Neural Architecture Search ● We can specify the structure and connectivity of a neural network by a configuration string ○ [“Filter Width: 5”, “Filter Height: 3”, “Num Filters: 24”] ● Use a RNN (“Controller”) to generate this string (the “Child Network”) ● Train the Child Network to see how well it performs on a validation set ● Use reinforcement learning to update the Controller based on the accuracy of the Child Network
  9. 9. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network
  10. 10. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search
  11. 11. Neural Architecture Search for Convolutional Networks Controller RNN Softmax classifier Embedding
  12. 12. Neural Architecture Search for CIFAR-10
  13. 13. Performance of cell on ImageNet
  14. 14. Platform aware Architecture Search
  15. 15. Platform aware Architecture Search
  16. 16. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity
  17. 17. Confidential + Proprietary
  18. 18. Confidential + Proprietary
  19. 19. Confidential + Proprietary Strange hump Basically linear
  20. 20. Confidential + Proprietary Mobile NASNet-A on ImageNet
  21. 21. Data processing Machine Learning Model Data Focus of machine learning research
  22. 22. Data processing Machine Learning Model Data Focus of machine learning research Very important but manually tuned
  23. 23. Data Augmentation
  24. 24. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity / Augmentation Strategy
  25. 25. AutoAugment: Example Policy Probability of applying Magnitude
  26. 26. CIFAR-10 State-of-art: 2.1% error AutoAugment: 1.5% error ImageNet State-of-art: 3.9% error AutoAugment: 3.5% error
  27. 27. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity / Augmentation Strategy Summary of AutoML and its progress
  28. 28. CIFAR-10 AutoML Accuracy ML Experts
  29. 29. ImageNet AutoML Top-1Accuracy ML Experts
  30. 30. References ● Neural Architecture Search with Reinforcement Learning. Barret Zoph and Quoc V. Le. ICLR, 2017 ● Learning Transferable Architectures for Large Scale Image Recognition. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. CVPR, 2018 ● AutoAugment: Learning Augmentation Policies from Data. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, Quoc V. Le. Arxiv, 2018 ● Searching for Activation Functions. Prajit Ramachandran, Barret Zoph, Quoc Le. ICLR Workshop, 2018

×