This document discusses the history and development of deep learning and neural networks. It explains that early neural networks from the 1940s-1980s had limitations due to limited computational power. Starting in 2006, deep neural networks enabled more layers of representation to be learned from data in an end-to-end fashion. The document contrasts shallow learning approaches that used hand-crafted features with deep learning, which allows automatic feature extraction directly from data using trainable neural networks.

4. Observable
Data
Model Information

5. Observable
Data
Model Information
Speech
Recognition
I am a boy

6. Observable
Data
Model Information
Image
Classify
Cat

7. Observable
Data
Model Information
HOW?

12. • Core visual object recognition
Feedback

13. • Weakness in kernel machine(SVM …):
• It does not scale well with sample size.
• Based on matching local templates.
• the training data is referenced for test data
• Local representation VS distributed representation
• N N(Neural Network) -> Kernel machine -> Deep NN

14. • Deep learning is all about deep neural networks
• 1949 : Hebbian learning
• Donald Hebb : the father of neural networks
• 1958 : (single layer) Perceptron
• Frank Rosenblatt
- Marvin Minsky, 1969
• 1986 : Multilayer Perceptron(Back propagation)
• David Rumelhart, Geoffrey Hinton, and Ronald Williams
• 2006 : Deep Neural Networks
• Geoffrey Hinton and Ruslan Salakhutdinov

15. Hand-
Crafted
Features
Trainable
Generic
Classifier

16. F(X;𝜃)
𝜃
Simple
Classifier

17. Layer
1
Simple
Classifier
Layer
2
Layer
N

18. Layer
1
Simple
Classifier
Layer
2
Layer
N

19. Layer
1
Simple
Classifier
Layer
2
Layer
N

20. Layer
1
Simple
Classifier
Layer
2
Layer
N

21. Layer
1
Simple
Classifier
Layer
2
Layer
N
Trainable
Generic
Classifier
Hand-
crafted
Features

22. Layer
1
Simple
Classifier
Layer
2
Layer
N
Trainable
Generic
Classifier
Hand-
crafted
Features

23. Layer
1
Simple
Classifier
Layer
2
Layer
N

25. Shallow learning Deep learning
feature extraction by domain experts
(SIFT, SURF, orb...)
automatic feature extraction from data
separate modules
(feature extractor + trainable classifier)
unified model : end-to-end learning
(trainable feature + trainable classifier)

43. i j

94. • http://www.cs.toronto.edu/~hinton/MatlabForSciencePaper.html
• convolutional neural networks (popular): LeCun
• Alex Krizhevsky: Hinton (python, C++)
• https://code.google.com/p/cuda-convnet/
• Caffe: UC Berkeley (C++)
• http://caffe.berkeleyvision.org/