Deep learning has made significant progress in solving problems like visual recognition, speech recognition, and natural language processing. The field began with the perceptron in 1957 but had many limitations. Multilayer perceptrons and backpropagation in the 1980s-90s helped address these issues. Breakthroughs like convolutional neural networks in the 1990s and large datasets like ImageNet in 2012 helped deep learning scale. Factors like GPUs, very deep models, and big data fueled further advances in areas such as image captioning. Future progress may come from faster processing, recurrent and hierarchical models, attentional models, and simulated worlds to generate more training data.

1. The History and
Near Future of
Deep Learning
David Kammeyer
Kammeyer Development
kammeyer@kammeyer.org
Big Data Beers 15.9.2015

2. What’s the big Deal?

3. Solving Problems that are
Easy for Humans, Hard
for Computers
• Visual Recognition, including OCR
• Speech Recognition
• Natural Language Processing (Translation,
Sentiment Analysis

4. Where did this all
come from?

5. 1957: The Perceptron
Frank Rosenblatt @ Cornell, MIT, ONR

6. How the Perceptron Works

7. Limitations and
Winter #1
Perceptrons cannot learn
the XOR function, or any
nonmonotonic function.

8. Multilayer Perceptrons
1989: Cybenko’s Universal Approximation theorem for
Single Hidden Layer Perceptrons

9. Backpropagation

10. Training Methods and
Winter #2
• Just because you can represent a
function as a single hidden layer net
doesn’t mean you can learn it (Might
need more layers to be able to learn)
• SVMs provided better learning
guarantees

11. The Renaissance

12. Convolutional Neural Networks
LeCun, 1993

13. ImageNet 2012
A. Krizhevsky’s AlexNet wins ImageNet Competition

14. Image Captioning
Karpathy 2015

15. What Changed?

16. GPUs
• 40x Speedup relative to
CPUs, allows the training
of much larger models
than before

17. Very Deep Models
• Allows for Hierarchical Representation of Knowledge

18. Big Data

19. Newer Techniques
RNN, LSTM, Deep Q-Learning, New Activation
Functions, Max Pooling

20. What’s Next?

21. Faster Processing
• Faster GPUs
• FPGAs
• ASICS

22. More Recurrence,
Bidirectional Hierarchies
• LSTM and RNN models have taken over at the
state of the art.
• Next step is Deep Recurrent models to capture
conceptual hierarchies
• Will Require new learning algorithms

23. Hierarchical Representations in the Brain

24. Attentional Models
Allow the network to sequentially focus attention on a
particular part of the input

25. Simulated (or Real) Worlds
• Lots of Data Needed to Train Large Models
• We’re going to have to Generate it, or Capture it from the Real World

26. More Researchers

27. Questions?
Dave Kammeyer
Kammeyer Development
kammeyer@kammeyer.org

28. Thanks!