Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Deep Learning:

and Deep Data-Science
12	
  May	
  2015
@graphific
Roelof Pieters
www.csc.kth.se/~roelof/
roelof@kth.se Gra...
BUT FIRST…
CAT PERSON? DOG PERSON?
are you a…
in the next few minutes
DETECTOR
we ll be making a
main Libraries
•sckikit-learn (machine learning) 

http://scikit-learn.org
•caffe (deep learning) - for training deep neur...
Code is ahead, soon… 

I promise :)
“Data science is clearly a blend of the
hackers' art, statistics and machine
learning…”
—Hilary Mason & Chris Wiggins, 201...
(Drew Connoway 2010)
1 feature
> Features = Awesomeness
1 feature
2 features
> Features = Awesomeness
1 feature
2 features
too few features/dimensions = overfitting
1 feature
2 features
too few features/dimensions = overfitting
3 features
> Features = Awesomeness
More Features = Awesomeness!
1 feature
2 features 3 features
++ Data Needs also grow!
(picture by Dato)
(picture by Dato)
Deep Learning?
•A host of statistical machine learning
techniques
•Enables the automatic learning of feature
hierarchies
•...
(picture by Dato)
Deep Learning
Deep Learning?
•Manually designed features are often over-specified,
incomplete and take a long time to design and validate...
23
2006+ : The Deep Learning Conspirators
(chart by Clarifai)
Audio Recognition
Image Recognition
(chart by Clarifai)
Natural Langauge Processing
…
Natural Langauge Processing
… …
DL? How ?
almost at the code…
(picture by Dato)
Coding time!
our ingredients…
(picture by Dato)
Kaggle’s Cat vs Dog
dataset (25k dog/cat
pictures)
https://www.kaggle.com/c/dogs-vs-cats/data
Pretrained
Convolutional 

Neural Net (CNN)
(picture by Dato)
97% accuracy in < 1h
MultiLayer
Perceptron
Cooking Instructions1 Load Pretrained Net
features
image
2 Extract features for all training images
3. train MLP on those ...
Cooking Instructions1 Load Pretrained Net
https://github.com/BVLC/caffe/wiki/Model-Zoo
No Free Lunch… But Free Models!
# imports demo
# load pretrained deep neural net
(convnet from Krizhevsky et al.'s NIPS 2012 ImageNet classification paper)
demo
Cooking Instructions
features
image
2 Extract features for all training images
1 Load Pretrained Net
# feed image into the network and return internal feature
representation of layer fc6 demo
#extract features from images demo
(…)
#dump features as pickle file demo
Cooking Instructions1 Load Pretrained Net
features
image
2 Extract features for all training images
3. train MLP on those ...
Pylearn2:
Multilayer Perceptron (MLP) on top of
extracted features
#imports demo
#load earlier extracted features and labels
#convert to input that pylearn understands demo
# create

layers of

MLP
# with 

softmax

as final layer
# trainer
initialized

with SGD, 

momentum,

dropout
demo
# train/test splits demo
#start our MLP (pylearn experiment method)
(…)
demo
already after 5 min:
90%
1 hour
50%
accuracy
97%
1 hour
94%
start at iteration #2
accuracy
(…)
So are YOU more like a Dog or Cat?
DETECTOR
(I might put it up as a Flask site online, if people are
interested?)
What 

about

me?
EASY PIEZY…
THATS ALL!
as PhD candidate KTH/CSC:
“Always interested in discussing
Machine Learning, Deep
Architectures, Graphs, and
Language Tech...
Wanna Know More?
bit.ly/SthlmDL
• Theano - CPU/GPU symbolic expression compiler in
python (from LISA lab at University of Montreal).
http://deeplearning.n...
• RNNLM (Mikolov)

http://rnnlm.org
• NB-SVM

https://github.com/mesnilgr/nbsvm
• Word2Vec (skipgrams/cbow)

https://code....
• cuda-convnet2 (Alex Krizhevsky, Toronto) (c++/
CUDA, optimized for GTX 580) 

https://code.google.com/p/cuda-convnet2/
•...
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Deep Learning as a Cat/Dog Detector
Upcoming SlideShare
Loading in …5
×

Deep Learning as a Cat/Dog Detector

34,827 views

Published on

Talk given at PYCON Stockholm 2015
Intro to Deep Learning + taking pretrained imagenet network, extracting features, and RBM on top = 97 Accuracy after 1 hour (!) of training (in top 10% of kaggle cat vs dog competition)

Deep Learning as a Cat/Dog Detector

  1. 1. Deep Learning:
 and Deep Data-Science 12  May  2015 @graphific Roelof Pieters www.csc.kth.se/~roelof/ roelof@kth.se Graph Technologies R&D roelof@graph-technologies.com slides online at:
 https://www.slideshare.net/roelofp/deep-learning-as-a-catdog-detector
  2. 2. BUT FIRST… CAT PERSON? DOG PERSON? are you a…
  3. 3. in the next few minutes DETECTOR we ll be making a
  4. 4. main Libraries •sckikit-learn (machine learning) 
 http://scikit-learn.org •caffe (deep learning) - for training deep neural nets (for today: loading a pre-trained one) 
 http://caffe.berkeleyvision.org •theano (efficient gpu-powered math)
 http://www.deeplearning.net/software/theano/ •ipython notebook
 http://ipython.org/notebook.html
  5. 5. Code is ahead, soon… 
 I promise :)
  6. 6. “Data science is clearly a blend of the hackers' art, statistics and machine learning…” —Hilary Mason & Chris Wiggins, 2010 Data Science ?
  7. 7. (Drew Connoway 2010)
  8. 8. 1 feature > Features = Awesomeness
  9. 9. 1 feature 2 features > Features = Awesomeness
  10. 10. 1 feature 2 features too few features/dimensions = overfitting
  11. 11. 1 feature 2 features too few features/dimensions = overfitting 3 features > Features = Awesomeness
  12. 12. More Features = Awesomeness! 1 feature 2 features 3 features
  13. 13. ++ Data Needs also grow!
  14. 14. (picture by Dato)
  15. 15. (picture by Dato)
  16. 16. Deep Learning? •A host of statistical machine learning techniques •Enables the automatic learning of feature hierarchies •Generally based on artificial neural networks
  17. 17. (picture by Dato) Deep Learning
  18. 18. Deep Learning? •Manually designed features are often over-specified, incomplete and take a long time to design and validate •Learned Features are easy to adapt, fast to learn
 •Deep learning provides a very flexible, (almost?) universal, learnable framework for representing world, visual and linguistic information. •Deep learning can learn unsupervised (from raw text/audio/images/whatever content) and supervised (with specific labels like positive/ negative) (as summarised by Richard Socher 2014)
  19. 19. 23 2006+ : The Deep Learning Conspirators
  20. 20. (chart by Clarifai) Audio Recognition
  21. 21. Image Recognition (chart by Clarifai)
  22. 22. Natural Langauge Processing …
  23. 23. Natural Langauge Processing … …
  24. 24. DL? How ? almost at the code…
  25. 25. (picture by Dato) Coding time! our ingredients…
  26. 26. (picture by Dato) Kaggle’s Cat vs Dog dataset (25k dog/cat pictures)
  27. 27. https://www.kaggle.com/c/dogs-vs-cats/data
  28. 28. Pretrained Convolutional 
 Neural Net (CNN)
  29. 29. (picture by Dato) 97% accuracy in < 1h MultiLayer Perceptron
  30. 30. Cooking Instructions1 Load Pretrained Net features image 2 Extract features for all training images 3. train MLP on those features
  31. 31. Cooking Instructions1 Load Pretrained Net
  32. 32. https://github.com/BVLC/caffe/wiki/Model-Zoo No Free Lunch… But Free Models!
  33. 33. # imports demo
  34. 34. # load pretrained deep neural net (convnet from Krizhevsky et al.'s NIPS 2012 ImageNet classification paper) demo
  35. 35. Cooking Instructions features image 2 Extract features for all training images 1 Load Pretrained Net
  36. 36. # feed image into the network and return internal feature representation of layer fc6 demo
  37. 37. #extract features from images demo
  38. 38. (…) #dump features as pickle file demo
  39. 39. Cooking Instructions1 Load Pretrained Net features image 2 Extract features for all training images 3. train MLP on those features
  40. 40. Pylearn2: Multilayer Perceptron (MLP) on top of extracted features #imports demo
  41. 41. #load earlier extracted features and labels #convert to input that pylearn understands demo
  42. 42. # create
 layers of
 MLP # with 
 softmax
 as final layer # trainer initialized
 with SGD, 
 momentum,
 dropout demo
  43. 43. # train/test splits demo
  44. 44. #start our MLP (pylearn experiment method) (…) demo already after 5 min:
  45. 45. 90% 1 hour 50% accuracy
  46. 46. 97% 1 hour 94% start at iteration #2 accuracy (…)
  47. 47. So are YOU more like a Dog or Cat? DETECTOR
  48. 48. (I might put it up as a Flask site online, if people are interested?) What 
 about
 me?
  49. 49. EASY PIEZY… THATS ALL!
  50. 50. as PhD candidate KTH/CSC: “Always interested in discussing Machine Learning, Deep Architectures, Graphs, and Language Technology” roelof@kth.se www.csc.kth.se/~roelof/ Data Science ConsultancyAcademic/Research roelof@graph-systems.com www.graph-technologies.com Gve Systems Graph Technologies 64 In Touch!
  51. 51. Wanna Know More? bit.ly/SthlmDL
  52. 52. • Theano - CPU/GPU symbolic expression compiler in python (from LISA lab at University of Montreal). http://deeplearning.net/software/theano/ • Pylearn2 - library designed to make machine learning research easy. http://deeplearning.net/software/ pylearn2/ • Torch - Matlab-like environment for state-of-the-art machine learning algorithms in lua (from Ronan Collobert, Clement Farabet and Koray Kavukcuoglu) http://torch.ch/ • more info: http://deeplearning.net/software links/ Wanna Play ? 66 Wanna Play ? General Deep Learning
  53. 53. • RNNLM (Mikolov)
 http://rnnlm.org • NB-SVM
 https://github.com/mesnilgr/nbsvm • Word2Vec (skipgrams/cbow)
 https://code.google.com/p/word2vec/ (original)
 http://radimrehurek.com/gensim/models/word2vec.html (python) • GloVe
 http://nlp.stanford.edu/projects/glove/ (original)
 https://github.com/maciejkula/glove-python (python) • Socher et al / Stanford RNN Sentiment code:
 http://nlp.stanford.edu/sentiment/code.html • Deep Learning without Magic Tutorial:
 http://nlp.stanford.edu/courses/NAACL2013/ 67 Wanna Play ? NLP
  54. 54. • cuda-convnet2 (Alex Krizhevsky, Toronto) (c++/ CUDA, optimized for GTX 580) 
 https://code.google.com/p/cuda-convnet2/ • Caffe (Berkeley) (Cuda/OpenCL, Theano, Python)
 http://caffe.berkeleyvision.org/ • OverFeat (NYU) 
 http://cilvr.nyu.edu/doku.php?id=code:start 68 Wanna Play ? Computer Vision

×