The document is an introduction to deep learning that discusses what it is capable of, how it works, and how to get started with it. It can be used for tasks like image recognition, text understanding, and driving cars by finding patterns in data. Deep learning uses neural networks with many layers to learn representations of data with multiple levels of abstraction. Frameworks like TensorFlow and Caffe can be used along with tutorials to help get started with deep learning.

1. Medical Data Retrieval
rene.donner@contextflow.com
René Donner
Deep Learning

3. René Donner Deep Learning
Overview
3
The (amazing) things Deep Learning can do
How does it work?
How can you start with DL?

4. René Donner Deep Learning
Roughly …
4
Deep learning finds patterns
in data corresponding to
high-level, abstract concepts

5. What can it do?

6. René Donner Deep Learning
What it can be used for
6
Image recognition
Text understanding, translation
Voice recognition
Playing video games
Driving cars
…

7. René Donner Deep Learning
Image recognition
7

8. René Donner Deep Learning
Scene labeling
8
http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html

9. René Donner Deep Learning
Text recognition
9
http://www.pyimagesearch.com/2014/09/22/
getting-started-deep-learning-python/
Large-Scale Deep Learning for Intelligent Computer Systems,
Jeff Dean, Google, BayLearn 2015

10. René Donner Deep Learning
Text understanding
10
2013 Glove: Global Vectors for Word Representation, Jeffrey
Pennington, Richard Socher and Christopher D. Manning

11. René Donner Deep Learning
Word embeddings
11

12. René Donner Deep Learning
Information extraction / Reasoning
12
MetaMind

13. René Donner Deep Learning
Some well know research groups
13
Stanford / Baidu
Andrew Ng
NYU / Facebook
Yann LeCun
UToronto / Google
Geoffrey Hinton

14. René Donner Deep Learning
NVIDIA
14
brand new: M40
(same as Geforce GTX Titan X)
Images: NVIDIA website

15. How does it work?

16. René Donner Deep Learning
Difference to classic ML
16
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

17. René Donner Deep Learning
Deep learning
17
http://theanalyticsstore.ie/deep-learning/

18. René Donner Deep Learning
Visualization
18
1. Layer
higher Layers
Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012

19. René Donner Deep Learning
Deep learning
19
How does it work?
http://theanalyticsstore.ie/deep-learning/
http://stats.stackexchange.com/questions/114385/
what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma

20. René Donner Deep Learning
Optimization
20
Stochastic gradient descent
Automatic differentiation
blog.datumbox.com

21. René Donner Deep Learning
Local minima
21
Less problematic than thought - saddle points
https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg

22. René Donner Deep Learning
Deep learning
22
Low level features of color images
https://www.coursera.org/course/neuralnets

23. René Donner Deep Learning
Deep learning
23
http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

24. René Donner Deep Learning
ImageNet topologies
24
ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015

25. René Donner Deep Learning
MNIST - Demo
25

26. René Donner Deep Learning
MNIST
26
http://deeplearning4j.org/rbm-mnist-tutorial.html

27. René Donner Deep Learning
Deep learning - why does it work?
27
Can cope with huge amounts of data
Learns small invariances
Overcomplete, sparse, representations
Learn Embedding
Lots of data
Recent advance: it is actually computable!

28. René Donner Deep Learning
Deep learning - pros
28
Not-domain specific
Supervised / Semi-supervised / Unsupervised
Classification / regression in last layer
Simple math
Hip

29. René Donner Deep Learning
Deep learning - cons
29
Lots of meta-parameters
Needs a lot of data
Very compute intensive
Hip

30. Getting started with DL

31. René Donner Deep Learning
Frameworks
31
Many different DL toolboxes
Efficiency important (GPU)
Attention to numerical issues

32. René Donner Deep Learning
Frameworks
32
Caffe
http://caffe.berkeleyvision.org/
Plain text files
Fastest CNN, GPU
Keras
https://github.com/fchollet/keras
Python, on top of Theano
TensorFlow
http://tensorflow.org/
Python, by Google
MXNet
https://github.com/dmlc/mxnet
Python, R, Julia
Slidefromcaffetutorial

33. René Donner Deep Learning
Tensorflow
33
General gradient descent library

34. René Donner Deep Learning
Tutorials
34
Stanford tutorial
https://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial
Matlab code snippets
videolectures.net
http://videolectures.net/deeplearning2015_montreal/
coursera
https://www.coursera.org/course/neuralnets

35. René Donner Deep Learning
Practical hints
35
Bengio Arxiv
Practical Recommendations for Gradient-Based Training of Deep Architectures
http://arxiv.org/abs/1206.5533
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
Kaggle
http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge
http://benanne.github.io/2014/04/05/galaxy-zoo.html
Relevant conferences
NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)
CVPR, ICML
Many interesting papers on arxiv.org

36. René Donner Deep Learning
Current research topics
36
Parallelization
What is deep learning, actually?
Alternative, faster, simpler methods
Multi-domain, transfer learning

37. Medical Data Retrieval
rene.donner@contextflow.com
René Donner
Deep Learning