Traditional machine learning systems are hand-designed and tuned by machine learning experts. To scale up the impact of machine learning to many real-world applications, we must figure out a way to automate the designing process of these pipelines. In this talk, I will discuss the use of machine learning to automate the process of designing neural architectures and data augmentation strategies (Neural Architecture Search and AutoAugment).

1. AutoML - Automated Machine Learning
Quoc Le
@quocleix
Joint work with many colleagues in the Google Brain team

2. Current:
Solution =
ML Expertise + Data + Computation

3. Current:
Solution =
ML Expertise + Data + Computation
But can we turn this into:
Solution =
Data + 100X Computation

4. Data processing
Machine
Learning Model
Data

5. Data processing
Machine
Learning Model
Data

6. Importance of architectures for Vision
● Designing neural network architectures is hard
● Lots of human efforts go into tuning them
● Can we try and learn good architectures automatically?
Canziani et al, 2017

7. Convolutional Architectures
Krizhevsky et al, 2012

8. Neural Architecture Search
● We can specify the structure and connectivity of a neural network by a
configuration string
○ [“Filter Width: 5”, “Filter Height: 3”, “Num Filters: 24”]
● Use a RNN (“Controller”) to generate this string (the “Child Network”)
● Train the Child Network to see how well it performs on a validation set
● Use reinforcement learning to update the Controller based on the accuracy of
the Child Network

9. Controller: proposes Child Networks Train & evaluate Child Networks
20K
times
Iterate to
find the
most
accurate
Child
Network

10. Controller: proposes Child Networks Train & evaluate Child Networks
20K
times
Iterate to
find the
most
accurate
Child
Network
Reinforcement Learning
or Evolution Search

11. Neural Architecture Search for Convolutional Networks
Controller RNN
Softmax classifier
Embedding

12. Neural Architecture Search for CIFAR-10

13. Performance of cell on ImageNet

14. Platform aware Architecture Search

15. Platform aware Architecture Search

16. Controller: proposes Child Networks Train & evaluate Child Networks
20K
times
Iterate to
find the
most
accurate
Child
Network
Reinforcement Learning
or Evolution Search
Architecture / Optimization Algorithm /
Nonlinearity

17. Confidential + Proprietary

18. Confidential + Proprietary

19. Confidential + Proprietary
Strange hump
Basically
linear

20. Confidential + Proprietary
Mobile NASNet-A on ImageNet

21. Data processing
Machine
Learning Model
Data
Focus of machine
learning research

22. Data processing
Machine
Learning Model
Data
Focus of machine
learning research
Very important but
manually tuned

23. Data Augmentation

24. Controller: proposes Child Networks Train & evaluate Child Networks
20K
times
Iterate to
find the
most
accurate
Child
Network
Reinforcement Learning
or Evolution Search
Architecture / Optimization Algorithm /
Nonlinearity / Augmentation Strategy

25. AutoAugment: Example Policy
Probability of applying
Magnitude

26. CIFAR-10
State-of-art: 2.1% error
AutoAugment: 1.5% error
ImageNet
State-of-art: 3.9% error
AutoAugment: 3.5% error

27. Controller: proposes Child Networks Train & evaluate Child Networks
20K
times
Iterate to
find the
most
accurate
Child
Network
Reinforcement Learning
or Evolution Search
Architecture / Optimization Algorithm /
Nonlinearity / Augmentation Strategy
Summary of AutoML and its progress

28. CIFAR-10
AutoML
Accuracy
ML Experts

29. ImageNet
AutoML
Top-1Accuracy
ML Experts

30. References
● Neural Architecture Search with Reinforcement Learning. Barret Zoph and Quoc
V. Le. ICLR, 2017
● Learning Transferable Architectures for Large Scale Image Recognition. Barret
Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. CVPR, 2018
● AutoAugment: Learning Augmentation Policies from Data. Ekin D. Cubuk, Barret
Zoph, Dandelion Mane, Vijay Vasudevan, Quoc V. Le. Arxiv, 2018
● Searching for Activation Functions. Prajit Ramachandran, Barret Zoph, Quoc Le.
ICLR Workshop, 2018