Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introduction to Deep Learning with TensorFlow

24,821 views

Published on

A practical guide for using TensorFlow with examples

Published in: Engineering

Introduction to Deep Learning with TensorFlow

  1. 1. Terry Taewoong Um (terry.t.um@gmail.com) University of Waterloo Department of Electrical & Computer Engineering Terry Taewoong Um INTRODUCTION TO DEEP NEURAL NETWORK WITH TENSORFLOW 1
  2. 2. Terry Taewoong Um (terry.t.um@gmail.com) CONTENTS 2 1. Why Deep Neural Network
  3. 3. Terry Taewoong Um (terry.t.um@gmail.com) 3 EXAMPLE CASE - Imagine you have extracted features from sensors - The dimension of each sample (which represents one of gestures) is around 800 - You have 70,000 samples (trial) - What method would you apply?
  4. 4. Terry Taewoong Um (terry.t.um@gmail.com) 4 EXAMPLE CASE - Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique ☞ What you did here is “Finding a good representation” - Then, you may apply a classification methods to classify 10 classes • You may have several ways to do it • But, what if - You have no idea for feature selection? - The dimension is much higher than 800 and you have more classes. ?
  5. 5. Terry Taewoong Um (terry.t.um@gmail.com) 5 EXAMPLE CASE - Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique ☞ What you did here is “Finding a good representation” - Then, you may apply a classification methods to classify 10 classes • You may have several ways to do it • But, what if - You have no idea for feature selection? - The dimension is much higher than 800 and you have more classes. MNIST dataset (65000spls * 784dim) MNIST dataset (60000spls * 1024dim)
  6. 6. Terry Taewoong Um (terry.t.um@gmail.com) 6 CLASSIFICATION RESULTS error rate : 28% → 15% → 8% (2010) (2014)(2012) http://rodrigob.github.io/are_we_there_yet/bu ild/classification_datasets_results.html
  7. 7. Terry Taewoong Um (terry.t.um@gmail.com) 7 PARADIGM CHANGE Knowledge PRESENT Representation (Features) How can we find a good representation? IMAGE SPEECH Hand-Crafted Features
  8. 8. Terry Taewoong Um (terry.t.um@gmail.com) 8 PARADIGM CHANGE IMAGE SPEECH Hand-Crafted Features Knowledge PRESENT Representation (Features) Can we learn a good representation (feature) for the target task as well?
  9. 9. Terry Taewoong Um (terry.t.um@gmail.com) 9 UNSUPERVISED LEARNING “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representation”, Lee et al., 2012
  10. 10. Terry Taewoong Um (terry.t.um@gmail.com) 10 THREE TYPES OF DEEP LEARNING • Unsupervised learning method Autoencoder http://goo.gl/s6kmqY - Restricted Boltzmann Machine(RBM), Autoencoder, etc. - It helps to avoid local minima problem (It regularizes the training data) - But it is not necessary when we have large amount of data. (Drop-out is enough for regularization) • Convolutional Neural Network (ConvNet) • Recurrent Neural Network (RNN) + Long-Short Term Memory (LSTM) - ConvNet has shown outstanding performance in recognition tasks (image, speech) - ConvNet contains hierarchical abstraction process called pooling. - RNN+LSTM makes use of long-term memory → Good for time-series data - RNN is a generative model: It can generate new data
  11. 11. Terry Taewoong Um (terry.t.um@gmail.com) CONTENTS 11 2. DNN with TensorFlow Thanks to Sungjoon Choi https://github.com/sjchoi86/
  12. 12. Terry Taewoong Um (terry.t.um@gmail.com) 12 DEEP LEARNING LIBRARIES
  13. 13. Terry Taewoong Um (terry.t.um@gmail.com) 13 DEEP LEARNING LIBRARY
  14. 14. Terry Taewoong Um (terry.t.um@gmail.com) 14 DEEP LEARNING LIBRARY • Karpathy’s Recommendation
  15. 15. Terry Taewoong Um (terry.t.um@gmail.com) 15 BASIC WORKFLOW OF TF 1. Load data 2. Define the NN structure 3. Set optimization parameters 4. Run! https://github.com/terryum/TensorFlow_Exercises
  16. 16. Terry Taewoong Um (terry.t.um@gmail.com) 16 EXAMPLE 1 https://github.com/terryum/TensorFlow_Exercises
  17. 17. Terry Taewoong Um (terry.t.um@gmail.com) 17 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/2_LogisticRegression_MNIST_160516.ipynb
  18. 18. Terry Taewoong Um (terry.t.um@gmail.com) 18 1. LOAD DATA
  19. 19. Terry Taewoong Um (terry.t.um@gmail.com) 19 2. DEFINE THE NN STRUCTURE 3. SET OPTIMIZATION PARAMETERS
  20. 20. Terry Taewoong Um (terry.t.um@gmail.com) 20 4. RUN
  21. 21. Terry Taewoong Um (terry.t.um@gmail.com) 21 4. RUN (C.F.)
  22. 22. Terry Taewoong Um (terry.t.um@gmail.com) 22 EXAMPLE 2 https://github.com/terryum/TensorFlow_Exercises
  23. 23. Terry Taewoong Um (terry.t.um@gmail.com) 23 NEURAL NETWORK Hugo Larochelle, http://www.dmi.usherb.ca/~larocheh/index_en.html • Activation functions http://goo.gl/qMQk5H • Basic NN structure
  24. 24. Terry Taewoong Um (terry.t.um@gmail.com) 24 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/3a_MLP_MNIST_160516.ipynb
  25. 25. Terry Taewoong Um (terry.t.um@gmail.com) 25 2. DEFINE THE NN STRUCTURE
  26. 26. Terry Taewoong Um (terry.t.um@gmail.com) 26 3. SET OPTIMIZATION PARAMETERS
  27. 27. Terry Taewoong Um (terry.t.um@gmail.com) 27 4. RUN
  28. 28. Terry Taewoong Um (terry.t.um@gmail.com) 28 EXAMPLE 3 https://github.com/terryum/TensorFlow_Exercises
  29. 29. Terry Taewoong Um (terry.t.um@gmail.com) 29 CONVOLUTION http://colah.github.io/posts/2014-07- Understanding-Convolutions/ http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
  30. 30. Terry Taewoong Um (terry.t.um@gmail.com) 30 CONVOLUTIONAL NN • How can we deal with real images which is much bigger than MNIST digit images? - Use not fully-connected, but locally-connected NN - Use convolutions to get various feature maps - Abstract the results into higher layer by using pooling - Fine tune with fully-connected NN https://goo.gl/G7kBjI https://goo.gl/Xswsbd http://goo.gl/5OR5oH
  31. 31. Terry Taewoong Um (terry.t.um@gmail.com) 31 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/4a_CNN_MNIST_160517.ipynb
  32. 32. Terry Taewoong Um (terry.t.um@gmail.com) 32 2. DEFINE THE NN STRUCTURE
  33. 33. Terry Taewoong Um (terry.t.um@gmail.com) 33 2. DEFINE THE NN STRUCTURE
  34. 34. Terry Taewoong Um (terry.t.um@gmail.com) 34 3. SET OPTIMIZATION PARAMETERS
  35. 35. Terry Taewoong Um (terry.t.um@gmail.com) 35 4. RUN
  36. 36. Terry Taewoong Um (terry.t.um@gmail.com) 36 4. RUN (C.F.)
  37. 37. Terry Taewoong Um (terry.t.um@gmail.com) 37 Thank you https://www.facebook.com/terryum http://terryum.io/ http://t-robotics.blogspot.kr/

×