Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hands-on Deep Learning in Python

25,947 views

Published on

TLV meetup Deep Learning

Published in: Science
  • DOWNLOAD TH1S BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK Here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB EBOOK Here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK Here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB EBOOK Here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc EBOOK Here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Hands-on Deep Learning in Python

  1. 1. Hands-on Deep Learning in Python Imry Kissos Deep Learning Meetup TLV August 2015
  2. 2. Outline ● Problem Definition ● Training a DNN ● Improving the DNN ● Open Source Packages ● Summary 2
  3. 3. Problem Definition 3 Deep Convolution Network 1 http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/
  4. 4. Tutorial ● Goal: Detect facial landmarks on (normal) face images ● Data set provided by Dr. Yoshua Bengio ● Tutorial code available: https://github.com/dnouri/kfkd-tutorial/blob/master/kfkd.py 4
  5. 5. Flow 5 Predict Points on Test Set Train Model General Train Model “Nose Tip” Train Model “Mouth Corners”
  6. 6. Flow 6 Train Images Train Points Fit Trained Net
  7. 7. Flow 7 Test Images Predict Predicted Points
  8. 8. Python Deep Learning Framework nolearn - Wrapper to Lasagne Lasagne - Theano extension for Deep Learning Theano - Define, optimize, and mathematical expressions Efficient Cuda GPU for DNN 8 Low Level High Level HW Supports: GPU & CPU OS: Linux, OS X, Windows
  9. 9. Training a Deep Neural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN 9
  10. 10. Training a Deep Neural Network 1. Data Analysis a. Exploration + Validation b. Pre-Processing c. Batch and Split 2. Architecture Engineering 3. Optimization 4. Training the DNN 10
  11. 11. Data Exploration + Validation Data: ● 7K gray-scale images of detected faces ● 96x96 pixels per image ● 15 landmarks per image (?) Data validation: ● Some Landmarks are missing 11 1
  12. 12. Pre-Processing 12 Data Normalization Shuffle train data
  13. 13. Batch - - t - train batch - validation batch - - test batch ⇐One Epoch’s data 13train/valid/test splits are constant
  14. 14. Train / Validation Split 14 Classification - Train/Validation preserve classes proportion
  15. 15. Training a Deep Neural Network 1. Data Analysis 2. Architecture Engineering a. Layers Definition b. Layers Implementation 3. Optimization 4. Training 15
  16. 16. Architecture 16 X Y Conv Pool Dense Output
  17. 17. Layers Definition 17
  18. 18. Activation Function 18 1 ReLU
  19. 19. Dense Layer 19
  20. 20. Dropout 20
  21. 21. Dropout 21
  22. 22. Training a Deep Neural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization a. Back Propagation b. Objective c. SGD d. Updates e. Convergence Tuning 4. Training the DNN 22
  23. 23. Back Propagation Forward Path 23 Conv Dense X Y Output Points
  24. 24. Back Propagation Forward Path 24 X Y Conv Output PointsDense X Y Training Points
  25. 25. Back Propagation Backward Path 25 X Y Conv Dense
  26. 26. Back Propagation Update 26 Conv Dense For All Layers:
  27. 27. Objective 27
  28. 28. S.G.D 28 Updates the network after each batch Karpathy - “Babysitting”: weights/updates ~1e3
  29. 29. Optimization - Updates 29 Alec Radford
  30. 30. Adjusting Learning Rate & Momentum 30 Linear in epoch
  31. 31. Convergence Tuning 31 stops according to validation loss returns best weights
  32. 32. Training a Deep Neural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN a. Fit b. Fine Tune Pre-Trained c. Learning Curves 32
  33. 33. Fit 33 Loop over validation batchs Forward Loop over train batchs Forward+BackProp
  34. 34. Fine Tune Pre-Trained fgd 34 change output layer load pre-trained weight fine tune specialist
  35. 35. Learning Curves Loop over 6 Nets: 35 Epochs
  36. 36. Learning Curves Analysis 36 Net 1 Net 2 OverfittingConvergence Jittering EpochsEpochs RMSE RMSE
  37. 37. Part 1 Summary Training a DNN: 37
  38. 38. Part 1 End Break
  39. 39. Part 2 Beyond Training
  40. 40. Outline ● Problem Definition ● Motivation ● Training a DNN ● Improving the DNN ● Open Source Packages ● Summary 40
  41. 41. Beyond Training 1. Improving the DNN a. Analysis Capabilities b. Augmentation c. Forward - Backward Path d. Monitor Layers’ Training 2. Open Source Packages 3. Summary 41
  42. 42. Improving the DNN Very tempting: ● >1M images ● >1M parameters ● Large gap: Theory ↔ Practice ⇒Brute force experiments?! 42
  43. 43. Analysis Capabilities 1. Theoretical explanation a. Eg. dropout and augmentation decrease overfit 2. Empirical claims about a phenomena a. Eg. normalization improves convergence 3. Numerical understanding a. Eg. exploding / vanishing updates 43
  44. 44. Reduce Overfitting Solution: Data Augmentation 44 Net 1 Net 2 Overfitting Epochs
  45. 45. Data Augmentation Horizontal Flip Perturbation 45 1
  46. 46. Advanced Augmentation http://benanne.github.io/2015/03/17/plankton.html 46
  47. 47. Convergence Challenges 47 Need to monitor forward + backward path EpochsEpochs RMSE Data ErrorNormalization
  48. 48. Forward - Backward Path Forward Backward: Gradient w.r.t parameters 48
  49. 49. Monitor Layers’ Training nolearn - visualize.py 49
  50. 50. Monitor Layers’ Training 50 X. Glorot ,Y. Bengio, Understanding the difficulty of training deep feedforward neural networks: “Monitoring activation and gradients across layers and training iterations is a powerful investigation tool” Easy to monitor in Theano Framework
  51. 51. Weight Initialization matters (1) 51 Layer 1- Gradient are close to zero - vanishing gradients
  52. 52. Weight Initialization matters (2) 52 Network returns close to zero values for all inputs
  53. 53. Monitoring Activation plateaus sometimes seen when training neural networks 53 For most epochs the network returns close to zero output for all inputs Objective plateaus sometimes can be explained by saturation
  54. 54. Max of Weights of Conv1: Max of Updates of Conv1: 54http://cs231n.github.io/neural-networks-3/#baby Monitoring weights/update ratio 3e-1 2e-1 1e-1 0 3e-3 2e-3 1e-3 0 Epoch Epoch
  55. 55. Beyond Training 1. Improving the DNN 2. Open Source Packages a. Hardware and OS b. Python Framework c. Deep Learning Open Source Packages d. Effort Estimation 3. Summary 55
  56. 56. Hardware and OS ● Amazon Cloud GPU: AWS Lasagne GPU Setup Spot ~ $0.0031 per GPU Instance Hour ● IBM Cloud GPU: http://www-03.ibm.com/systems/platformcomputing/products/symphony/gpuharvesting.html ● Your Linux machine GPU: pip install -r https://raw.githubusercontent.com/dnouri/kfkd- tutorial/master/requirements.txt ● Window install http://deeplearning.net/software/theano/install_windows.html#install-windows 56
  57. 57. Starting Tips ● Sanity Checks: ○ DNN Architecture : “Overfit a tiny subset of data” Karpathy ○ Check Regularization ↗ Loss ↗ ● Use pre-trained VGG as a base line ● Start with ~3 conv layer with ~16 filter each - quickly iterate 57
  58. 58. Python ● Rich eco-system ● State-of-the-art ● Easy to port from prototype to production 58 Podcast : http://www.reversim.com/2015/10/277-scientific-python.html
  59. 59. Python Deep Learning Framework 59Keras ,pylearn2, OpenDeep, Lasagne - common base
  60. 60. Tips from Deep Learning Packages Torch code organization Caffe’s separation configuration ↔code NeuralNet → YAML text format defining experiment’s configuration 60
  61. 61. Deep Learning Open Source Packages 61 Caffe for applications Torch and Theano for research on Deep Learning itself http://fastml.com/torch-vs-theano/ Black BoxWhite Box Open source progress rapidly→ impossible to predict industry’s standard
  62. 62. Disruptive Effort Estimation Feature Eng Deep Learning 62Still requires algorithmic expertise
  63. 63. Summary ● Dove into Training a DNN ● Presented Analysis Capabilities ● Reviewed Open Source Packages 63
  64. 64. References Hinton Coursera Neuronal Network https://www.coursera.org/course/neuralnets Technion Deep Learning course http://moodle.technion.ac.il/course/view.php?id=4128 Oxford Deep Learning course https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu CS231n CNN for Visual Recognition http://cs231n.github.io/ Deep Learning Book http://www.iro.umontreal.ca/~bengioy/dlbook/ Montreal DL summer school http://videolectures.net/deeplearning2015_montreal/ 64
  65. 65. Questions? 65 Deep Convolution Regression Network

×