Deep learning & Humanity's Grand Challenges

1. Deep Learning and
Humanity’s Grand Challenges

2. Deep learning is causing
a machine learning revolution

3. “cat”
Deep Learning
Modern Reincarnation of Artificial Neural Networks
Collection of simple trainable mathematical units, organized in layers, that work together to solve
complicated tasks
Key Benefit
Learns features from raw, heterogeneous, noisy data
No explicit feature engineering required
What’s New
new network architectures,
new training math, *scale*

4. ConvNets

5. input
Pixels:
output
“lion”
Functions a Deep Neural Network Can Learn

6. input
Pixels:
Audio:
output
“lion”
“How cold is it outside?”
Functions a Deep Neural Network Can Learn

7. input
Pixels:
Audio:
“Hello, how are you?”
output
“lion”
“How cold is it outside?”
“Bonjour, comment allez-vous?”
Functions a Deep Neural Network Can Learn

8. input
Pixels:
Audio:
“Hello, how are you?”
Pixels:
output
“lion”
“How cold is it outside?”
“Bonjour, comment allez-vous?”
“A blue and yellow train
travelling down the tracks”
Functions a Deep Neural Network Can Learn

9. But why now?

10. Accuracy
Scale (data size, model size)
1980s and 1990s
neural networks
other approaches

11. more
computeAccuracy
Scale (data size, model size)
neural networks
other approaches
1980s and 1990s

12. more
computeAccuracy
Scale (data size, model size)
neural networks
other approaches
Now

13. 5% errors
humans
2011
26% errors

14. 2016
3% errors
2011
5% errors
humans
26% errors

15. 2008: Grand Engineering Challenges for 21st Century
● Make solar energy affordable
● Provide energy from fusion
● Develop carbon sequestration methods
● Manage the nitrogen cycle
● Provide access to clean water
● Restore & improve urban infrastructure
● Advance health informatics
● Engineer better medicines
● Reverse-engineer the brain
● Prevent nuclear terror
● Secure cyberspace
● Enhance virtual reality
● Advance personalized learning
● Engineer the tools for scientific
discovery
www.engineeringchallenges.org/challenges.aspx

16. Engineer the tools for scientific discovery

17. AutoML: Automated machine learning
(“learning to learn”)

18. Current:
Solution = ML expertise + data +
computation

19. Current:
Solution = ML expertise + data +
computation
Can we turn this into:
Solution = data + 100X computation
???

20. Idea: model-generating model trained via reinforcement learning
(1) Generate ten models
(2) Train them for a few hours
(3) Use loss of the generated models as reinforcement learning signal
Neural Architecture Search with Reinforcement Learning, Zoph & Le, ICLR 2016
arxiv.org/abs/1611.01578
Neural Architecture Search

21. CIFAR-10 Image Recognition Task

22. Early encouraging signs that we can build flexible
systems that can solve new problems
automatically…

23. Early encouraging signs that we can build flexible
systems that can solve new problems
automatically…
But, we’ll need more computation

24. Special computation properties
reduced
precision
ok
about 1.2
× about 0.6
about 0.7
1.21042
× 0.61127
0.73989343
NOT

25. handful of
specific
operations
× =
reduced
precision
ok
about 1.2
× about 0.6
about 0.7
1.21042
× 0.61127
0.73989343
NOT
Special computation properties

26. Deep neural networks and machine learning are producing
significant breakthroughs that are solving some of the
world’s grand challenges

27. Advance Personalized Learning

28. Lindsey, Shroyer, Pashler & Mozer. Improving students’ long-term knowledge retention through
personalized review. Psychological Science, vol. 25, pp. 639-647, 2014.
Three approaches:
1. Massed review (current educational practice)
2. Generic spaced review (“one size fits all”)
3. Personalized spaced review (adaptive selection based on student’s past performance)
Distribution of study time

29. Lindsey, Shroyer, Pashler & Mozer. Improving students’ long-term knowledge retention through
personalized review. Psychological Science, vol. 25, pp. 639-647, 2014.
Student Performance Improves Significantly
Overall performance Performance by chapter
+12.4%

30. Robotics

31. Combining Vision with Robotics
“Deep Learning for Robots: Learning
from Large-Scale Interaction”, Google
Research Blog, March, 2016
“Learning Hand-Eye Coordination for
Robotic Grasping with Deep Learning
and Large-Scale Data Collection”,
Sergey Levine, Peter Pastor, Alex
Krizhevsky, & Deirdre Quillen,
Arxiv, arxiv.org/abs/1603.02199

32. Self-Supervised and End-to-end Pose Estimation

33. Automated machine
learning (“learning to learn”)

34. Current:
Solution = ML expertise + data +
computation

35. Current:
Solution = ML expertise + data +
computation
Can we turn this into:
Solution = data + 100X computation
???

36. Early encouraging signs
Trying multiple different approaches:
(1)Reinforcement learning-based architecture search
(2)Model architecture evolution

37. More computational power is needed
Deep learning is transforming how we
design computers