Successfully reported this slideshow.
Your SlideShare is downloading. ×

Quoc Le at AI Frontiers : Automated Machine Learning

Quoc Le at AI Frontiers : Automated Machine Learning

Download to read offline

Traditional machine learning systems are hand-designed and tuned by machine learning experts. To scale up the impact of machine learning to many real-world applications, we must figure out a way to automate the designing process of these pipelines. In this talk, I will discuss the use of machine learning to automate the process of designing neural architectures and data augmentation strategies (Neural Architecture Search and AutoAugment).

Traditional machine learning systems are hand-designed and tuned by machine learning experts. To scale up the impact of machine learning to many real-world applications, we must figure out a way to automate the designing process of these pipelines. In this talk, I will discuss the use of machine learning to automate the process of designing neural architectures and data augmentation strategies (Neural Architecture Search and AutoAugment).

More Related Content

More from AI Frontiers

Related Books

Free with a 30 day trial from Scribd

See all

Quoc Le at AI Frontiers : Automated Machine Learning

  1. 1. AutoML - Automated Machine Learning Quoc Le @quocleix Joint work with many colleagues in the Google Brain team
  2. 2. Current: Solution = ML Expertise + Data + Computation
  3. 3. Current: Solution = ML Expertise + Data + Computation But can we turn this into: Solution = Data + 100X Computation
  4. 4. Data processing Machine Learning Model Data
  5. 5. Data processing Machine Learning Model Data
  6. 6. Importance of architectures for Vision ● Designing neural network architectures is hard ● Lots of human efforts go into tuning them ● Can we try and learn good architectures automatically? Canziani et al, 2017
  7. 7. Convolutional Architectures Krizhevsky et al, 2012
  8. 8. Neural Architecture Search ● We can specify the structure and connectivity of a neural network by a configuration string ○ [“Filter Width: 5”, “Filter Height: 3”, “Num Filters: 24”] ● Use a RNN (“Controller”) to generate this string (the “Child Network”) ● Train the Child Network to see how well it performs on a validation set ● Use reinforcement learning to update the Controller based on the accuracy of the Child Network
  9. 9. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network
  10. 10. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search
  11. 11. Neural Architecture Search for Convolutional Networks Controller RNN Softmax classifier Embedding
  12. 12. Neural Architecture Search for CIFAR-10
  13. 13. Performance of cell on ImageNet
  14. 14. Platform aware Architecture Search
  15. 15. Platform aware Architecture Search
  16. 16. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity
  17. 17. Confidential + Proprietary
  18. 18. Confidential + Proprietary
  19. 19. Confidential + Proprietary Strange hump Basically linear
  20. 20. Confidential + Proprietary Mobile NASNet-A on ImageNet
  21. 21. Data processing Machine Learning Model Data Focus of machine learning research
  22. 22. Data processing Machine Learning Model Data Focus of machine learning research Very important but manually tuned
  23. 23. Data Augmentation
  24. 24. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity / Augmentation Strategy
  25. 25. AutoAugment: Example Policy Probability of applying Magnitude
  26. 26. CIFAR-10 State-of-art: 2.1% error AutoAugment: 1.5% error ImageNet State-of-art: 3.9% error AutoAugment: 3.5% error
  27. 27. Controller: proposes Child Networks Train & evaluate Child Networks 20K times Iterate to find the most accurate Child Network Reinforcement Learning or Evolution Search Architecture / Optimization Algorithm / Nonlinearity / Augmentation Strategy Summary of AutoML and its progress
  28. 28. CIFAR-10 AutoML Accuracy ML Experts
  29. 29. ImageNet AutoML Top-1Accuracy ML Experts
  30. 30. References ● Neural Architecture Search with Reinforcement Learning. Barret Zoph and Quoc V. Le. ICLR, 2017 ● Learning Transferable Architectures for Large Scale Image Recognition. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. CVPR, 2018 ● AutoAugment: Learning Augmentation Policies from Data. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, Quoc V. Le. Arxiv, 2018 ● Searching for Activation Functions. Prajit Ramachandran, Barret Zoph, Quoc Le. ICLR Workshop, 2018

Editor's Notes

  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Bayesian optimization scales cubically, and can only work over fixed length spaces
    Evolutionary algorithms
  • Bayesian optimization scales cubically, and can only work over fixed length spaces
    Evolutionary algorithms
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Each prediction is carried out by a softmax classifier
    Example of a convolutional network
  • Mention the networks are much smaller than any of the other networks on the list
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time
  • Many different algorithms could be used, in this we used the basic REINFORCE
    Goal is to make the Controller sampler better and better architectures over time

×