Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python for Image Understanding: Deep Learning with Convolutional Neural Nets

84,102 views

Published on

Talk given at PyData 2015 London. June 21, 2015. Practical approach to how to train your deep neural net for images.

  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @ https://www.ThesisScientist.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I, Mr. Mohamed Bah of EU Capital Investment Company, as marketing manager with full corporate authority and responsibility under penalty of perjury declares that i have the right to sell the Gold dust/bars and rough uncut Diamonds which is clean, clear and from Non-criminal origin and guarantees to export to Buyers chosen refinery. The Buyer, with full corporate authority and responsibility under penalty of perjury declares that he is ready, willing and able to buy with clean, clear United States currency from Non-Criminal Original. Please we are dealing with direct buyers or buyers mandate not brokers. Thanks in advance Regards Mr. Bah +232-25-314067 +232-76-955142 Email: mohamedb20@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • i am so interested in your works, i would love to learn more
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Python for Image Understanding: Deep Learning with Convolutional Neural Nets

  1. 1. for Image Understanding: Deep Learning with Convolutional Neural Nets Roelof Pieters PhD candidate at KTH & Data Science consultant at Graph Technologies  @graphific London 2015roelof@graph-technologies.com
  2. 2. What is Deep Learning?
  3. 3. “Deep learning is a set of algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction.” (much debated definition) A Definition
  4. 4. • A host of statistical machine learning techniques • Enables the automatic learning of feature hierarchies • Generally based on artificial neural networks A typology Deep learning is:
  5. 5. Manually designed features are often over-specified, incomplete and take a long time to design and validate Learned features are easy to adapt, fast to learn Deep learning provides a very flexible, (possibly?) universal, learnable framework for representing world, visual and linguistic information. Deep learning can learn unsupervised (from raw text/ audio/images/whatever content) and supervised (with specific labels like positive/negative)
 Summary by Richard Socher. Old vs new school?
  6. 6. No More Handcrafted Features ! 6
  7. 7. “Brain”-like: Feature Hierarchies
  8. 8. input layer output layer hidden layers Feature Hierarchies: Vision
  9. 9. input layer output layer hidden layers Feature Hierarchies: Audio
  10. 10. cars elephants chairs Feature Hierarchies: And so on…
  11. 11. History: audio recognition chart by Clarifai
  12. 12. History: image recognition chart by Clarifai
  13. 13. History: image recognition Krizhevsky et al. ImageNet Classification with Deep Convolutional Neural Networks, ILSVRC2010
  14. 14. 16 Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions Image-Text: Joint Visual Semantic embeddings
  15. 15. Video Scene Detection
  16. 16. http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html
  17. 17. http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html
  18. 18. How does Deep Learning work?
  19. 19. Activation Functions Sigmoid
  20. 20. Activation Functions Tanh
  21. 21. Activation Functions ReLU often approximated by just
  22. 22. optimization strategies
  23. 23. So what’s there for Python?
  24. 24. python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level (efficient gpu-powered math) (computer-vision oriented DL framework,
 model-zoo, prototxt model definitions) 
 pythonification ongoing! (wrapper for theano, yaml, experiment-oriented) (theano-wrapper, models in python code, 
 abstracts theano away) (theano-extension, models in python code, 
 theano not hidden) and of course:
  25. 25. python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org deeplearning.net/software/pylearn2 keras.io lasagne.readthedocs.org/en/latest and of course:
  26. 26. python has a wide range of deep learning-related libraries available Deep Learning with Python Low level High level deeplearning.net/software/theano caffe.berkeleyvision.org deeplearning.net/software/pylearn2 keras.io lasagne.readthedocs.org/en/latest and of course: we will use lasagne in our examples
  27. 27. Doing Deep Learning?
  28. 28. 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
  29. 29. 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
  30. 30. • Mean subtraction • Normalization • PCA and Whitening 1. Preprocess the data
  31. 31. • Mean subtraction • Normalization • PCA and Whitening 1. Preprocess the data: Normalization (mean image visualised of cifar-10)
  32. 32. • Mean subtraction • Normalization • PCA and Whitening 1. Preprocess the data: PCA & Whitening
  33. 33. 1. Preprocess the data: PCA & Whitening
  34. 34. 1. Preprocess the data, the right way Warning: • compute preprocessing statistics on training data • apply on all (training / validation / test) data
  35. 35. 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
  36. 36. • Deep Belief Network (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
  37. 37. • Deep Belief Network (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
  38. 38. • Deep Belief Network (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
  39. 39. • Deep Belief Network (DBN) • Convolutional Net (CNN) • Recurrent Net (RNN) 2. Choosing the right architecture
  40. 40. Convolutional Neural Net Pbird Psunset Pdog Pcat
  41. 41. Convolutional Neural Net
  42. 42. DrawCNN: visualizing the units' connections Agrawal, et al. Analyzing the performance of multilayer neural networks for object recognition. ECCV, 2014 Szegedy, et al. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013 Zeiler, M. et al. Visualizing and Understanding Convolutional Networks, ECCV 2014
  43. 43. 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
  44. 44. 1. Preprocess the data 2. Choose architecture 3. Train (Code Finally!) 4. Optimize/Regularize 5. Tips/Tricks Training a (deep) Neural Network
  45. 45. Training a (deep) Neural Network (…) layer definitions layer
 parameters
  46. 46. Training a (deep) Neural Network hyper
 parameters
  47. 47. Training a (deep) Neural Network Lift off!
  48. 48. Training a (deep) Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Debug
  49. 49. Debug Training: Visualize Loss Curve
  50. 50. Debug Training: Visualize Loss Curve
  51. 51. Debug Training: Visualize Loss Curve This looks (too) linear: lower your learning rate! This looks (too) wide: increase your batch size!
  52. 52. Debug Training: Visualize Accuracy big gap: overfitting: regularize! no gap: underfitting (increase model size)
  53. 53. Debug Training: Visualize Weights (usually: first layer) complete mess, doesn't get past the random initialisation
  54. 54. Debug Training: Visualize Weights (usually: first layer) better, but still “noisy” weights mostly solvable by stronger regularisation
  55. 55. Debug Training: Visualize Weights (usually: first layer) good: now operates as “edge detector”
  56. 56. Training a (deep) Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Tips/Tricks Debug
  57. 57. • Tweak Hyperparameters / Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
  58. 58. • Tweak Hyperparameters / Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
  59. 59. • Grid search won't work on your millions + parameters • Random Search? Mwah… • Bayesian Optimization: Yeh baby! • Spearmint: https://github.com/HIPS/Spearmint • Hypergrad: https://github.com/HIPS/hypergrad Choosing Hyperparameters
  60. 60. • Tweak Hyperparameters / Architecture • Data Augmentation • Dropout • Batch Normalization Overfitting
  61. 61. Data Augmentation
  62. 62. Data Augmentation http://benanne.github.io/2015/03/17/plankton.html (realtime data augmentation at Kaggle’s #1 National Data Science Bowl
 ≋ Deep Sea ≋ team) rotation: random with angle between 0° and 360° (uniform) translation: random with shift between -10 and 10 pixels (uniform) rescaling: random with scale factor between 1/1.6 and 1.6 (log- uniform) flipping: yes or no (bernoulli) shearing: random with angle between -20° and 20° (uniform) stretching: random with stretch factor between 1/1.3 and 1.3 (log- uniform)
  63. 63. • Tweak Hyperparameters / Architecture • Data Augmentation • Dropout • Batch Normalization Optimize / Regularize
  64. 64. Dropout as Regularization (naively trained net) Overfits !
  65. 65. Dropout as Regularization (naively trained net) Overfits ! Dropout!
  66. 66. Dropout as Regularization (naively trained net) Overfits ! Dropout!
  67. 67. Dropout as Regularization (naively trained net) (net with dropout) less strongly overfitted & 
 can run for more epochs higher accuracy
  68. 68. Overfitting • Tweak Hyperparameters / Architecture • Data Augmentation • Dropout • Batch Normalization
  69. 69. •Normalize the activations in each layer within a minibatch •Learn the mean and variance of each layer as parameters Batch Normalization as regularization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift 
 S Ioffe and C Szegedy (2015) 85% 50% 15%
  70. 70. Training a (deep) Neural Network 1. Preprocess the data 2. Choose architecture 3. Train 4. Optimize/Regularize 5. Further Tips & Tricks to improve Model Accuracy Debug
  71. 71. • Ensembles • Finetuning pre-trained/earlier-trained net • Sticking extracted layer features in another classifier (ie SVM) Other “Tricks”
  72. 72. • Ensembles • Finetuning pre-trained/earlier-trained net • Sticking extracted layer features in another classifier (ie SVM) Other “Tricks”
  73. 73. Ensembles • majority vote when hard predictions (ie classes) • average vote for soft predictions (continious scale) • make sure classifiers are uncorrelated • cross validate ensemble weights (by grid search, or rank average) • stacked • blending
  74. 74. Ensembles (10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
  75. 75. avg: 0.3647 Ensembles (10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
  76. 76. predict by mean of all: 0.4244 avg: 0.3647 Ensembles (10 similar nets with varying hyperparameters on same tiny-imagenet dataset)
  77. 77. (10 similar nets with varying hyperparameters on same tiny-imagenet dataset) predict by mean of all: 0.4244 leave out model9: 0.4259 avg: 0.3647 Ensembles
  78. 78. (10 similar nets with varying hyperparameters on same tiny-imagenet dataset) 0.4259 0.4244 0.3647 Ensembles
  79. 79. danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/ (ensembling specialist nets by Daniel Nouri, Kaggle facial keypoint tutorial) Ensembles “Specialists” Ensemble
  80. 80. https://www.kaggle.com/c/facial-keypoints-detection/ try it yourself :) 3 similar nets trained on the same data but with different hyper parameters.
 RMSE’s: •2,08449 •2,04575 •2.01565 together: 
 1.93397 disclaimer: Kaggle is not real life, people…
  81. 81. Machine learning systems can easily be fooled but beware… / no free lunch:
  82. 82. Saliency Maps K. Simonyan, A. Vedaldi, A. Zisserman , "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps", ICLR Workshop 2014 first we predict on a pixel level
  83. 83. Fooling ConvNets Szegedy, Christian, et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images." arXiv preprint then we do our “magic”
  84. 84. Fooling ConvNets Szegedy, Christian, et al. "Intriguing properties of neural networks." arXiv preprint, 2013. Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images." arXiv preprint then we do our “magic”
  85. 85. Failing ConvNets
  86. 86. Failing ConvNets “Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
  87. 87. Failing ConvNets “Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
  88. 88. Failing ConvNets “Suddenly, a leopard print sofa appears”, rocknrollnerd.github.io
  89. 89. Failing ConvNets
  90. 90. thanks for listening ;) questions? or find me @graphific
  91. 91. • Computer Vision: 
 Fei-Fei Li & Andrej Karpathy, Stanford course “Convolutional Neural Networks for Visual Recognition”
 http://vision.stanford.edu/teaching/cs231n • Natural Language Processing:
 Richard Socher, Stanford course “Deep Learning for Natural Language Processing”,
 http://cs224d.stanford.edu/ • Neural Nets:
 Geoffrey Hinton, Coursera/Toronto, “Neural Networks for Machine Learning"
 https://www.coursera.org/course/neuralnets Wanna Play? • Bunch of tutorials: 
 http://deeplearning.net/tutorial/ • Book:
 Yoshua Bengio, et al, “Deep Learning”
 http://www.iro.umontreal.ca/~bengioy/dlbook/ • UFLDL Tutorial
 http://deeplearning.stanford.edu/tutorial/ • Reading Lists:
 http://deeplearning.net/reading-list/ 
 http://memkite.com/deep-learning-bibliography/ • Podcast
 Talking Machines, http://www.thetalkingmachines.com/

×