Successfully reported this slideshow.
Upcoming SlideShare
×

# Introduction to Deep Learning with TensorFlow

A practical guide for using TensorFlow with examples

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

### Introduction to Deep Learning with TensorFlow

1. 1. Terry Taewoong Um (terry.t.um@gmail.com) University of Waterloo Department of Electrical & Computer Engineering Terry Taewoong Um INTRODUCTION TO DEEP NEURAL NETWORK WITH TENSORFLOW 1
2. 2. Terry Taewoong Um (terry.t.um@gmail.com) CONTENTS 2 1. Why Deep Neural Network
3. 3. Terry Taewoong Um (terry.t.um@gmail.com) 3 EXAMPLE CASE - Imagine you have extracted features from sensors - The dimension of each sample (which represents one of gestures) is around 800 - You have 70,000 samples (trial) - What method would you apply?
4. 4. Terry Taewoong Um (terry.t.um@gmail.com) 4 EXAMPLE CASE - Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique ☞ What you did here is “Finding a good representation” - Then, you may apply a classification methods to classify 10 classes • You may have several ways to do it • But, what if - You have no idea for feature selection? - The dimension is much higher than 800 and you have more classes. ?
5. 5. Terry Taewoong Um (terry.t.um@gmail.com) 5 EXAMPLE CASE - Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique ☞ What you did here is “Finding a good representation” - Then, you may apply a classification methods to classify 10 classes • You may have several ways to do it • But, what if - You have no idea for feature selection? - The dimension is much higher than 800 and you have more classes. MNIST dataset (65000spls * 784dim) MNIST dataset (60000spls * 1024dim)
6. 6. Terry Taewoong Um (terry.t.um@gmail.com) 6 CLASSIFICATION RESULTS error rate : 28% → 15% → 8% (2010) (2014)(2012) http://rodrigob.github.io/are_we_there_yet/bu ild/classification_datasets_results.html
7. 7. Terry Taewoong Um (terry.t.um@gmail.com) 7 PARADIGM CHANGE Knowledge PRESENT Representation (Features) How can we find a good representation? IMAGE SPEECH Hand-Crafted Features
8. 8. Terry Taewoong Um (terry.t.um@gmail.com) 8 PARADIGM CHANGE IMAGE SPEECH Hand-Crafted Features Knowledge PRESENT Representation (Features) Can we learn a good representation (feature) for the target task as well?
9. 9. Terry Taewoong Um (terry.t.um@gmail.com) 9 UNSUPERVISED LEARNING “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representation”, Lee et al., 2012
10. 10. Terry Taewoong Um (terry.t.um@gmail.com) 10 THREE TYPES OF DEEP LEARNING • Unsupervised learning method Autoencoder http://goo.gl/s6kmqY - Restricted Boltzmann Machine(RBM), Autoencoder, etc. - It helps to avoid local minima problem (It regularizes the training data) - But it is not necessary when we have large amount of data. (Drop-out is enough for regularization) • Convolutional Neural Network (ConvNet) • Recurrent Neural Network (RNN) + Long-Short Term Memory (LSTM) - ConvNet has shown outstanding performance in recognition tasks (image, speech) - ConvNet contains hierarchical abstraction process called pooling. - RNN+LSTM makes use of long-term memory → Good for time-series data - RNN is a generative model: It can generate new data
11. 11. Terry Taewoong Um (terry.t.um@gmail.com) CONTENTS 11 2. DNN with TensorFlow Thanks to Sungjoon Choi https://github.com/sjchoi86/
12. 12. Terry Taewoong Um (terry.t.um@gmail.com) 12 DEEP LEARNING LIBRARIES
13. 13. Terry Taewoong Um (terry.t.um@gmail.com) 13 DEEP LEARNING LIBRARY
14. 14. Terry Taewoong Um (terry.t.um@gmail.com) 14 DEEP LEARNING LIBRARY • Karpathy’s Recommendation
15. 15. Terry Taewoong Um (terry.t.um@gmail.com) 15 BASIC WORKFLOW OF TF 1. Load data 2. Define the NN structure 3. Set optimization parameters 4. Run! https://github.com/terryum/TensorFlow_Exercises
16. 16. Terry Taewoong Um (terry.t.um@gmail.com) 16 EXAMPLE 1 https://github.com/terryum/TensorFlow_Exercises
17. 17. Terry Taewoong Um (terry.t.um@gmail.com) 17 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/2_LogisticRegression_MNIST_160516.ipynb
18. 18. Terry Taewoong Um (terry.t.um@gmail.com) 18 1. LOAD DATA
19. 19. Terry Taewoong Um (terry.t.um@gmail.com) 19 2. DEFINE THE NN STRUCTURE 3. SET OPTIMIZATION PARAMETERS
20. 20. Terry Taewoong Um (terry.t.um@gmail.com) 20 4. RUN
21. 21. Terry Taewoong Um (terry.t.um@gmail.com) 21 4. RUN (C.F.)
22. 22. Terry Taewoong Um (terry.t.um@gmail.com) 22 EXAMPLE 2 https://github.com/terryum/TensorFlow_Exercises
23. 23. Terry Taewoong Um (terry.t.um@gmail.com) 23 NEURAL NETWORK Hugo Larochelle, http://www.dmi.usherb.ca/~larocheh/index_en.html • Activation functions http://goo.gl/qMQk5H • Basic NN structure
24. 24. Terry Taewoong Um (terry.t.um@gmail.com) 24 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/3a_MLP_MNIST_160516.ipynb
25. 25. Terry Taewoong Um (terry.t.um@gmail.com) 25 2. DEFINE THE NN STRUCTURE
26. 26. Terry Taewoong Um (terry.t.um@gmail.com) 26 3. SET OPTIMIZATION PARAMETERS
27. 27. Terry Taewoong Um (terry.t.um@gmail.com) 27 4. RUN
28. 28. Terry Taewoong Um (terry.t.um@gmail.com) 28 EXAMPLE 3 https://github.com/terryum/TensorFlow_Exercises
29. 29. Terry Taewoong Um (terry.t.um@gmail.com) 29 CONVOLUTION http://colah.github.io/posts/2014-07- Understanding-Convolutions/ http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
30. 30. Terry Taewoong Um (terry.t.um@gmail.com) 30 CONVOLUTIONAL NN • How can we deal with real images which is much bigger than MNIST digit images? - Use not fully-connected, but locally-connected NN - Use convolutions to get various feature maps - Abstract the results into higher layer by using pooling - Fine tune with fully-connected NN https://goo.gl/G7kBjI https://goo.gl/Xswsbd http://goo.gl/5OR5oH
31. 31. Terry Taewoong Um (terry.t.um@gmail.com) 31 1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/ master/4a_CNN_MNIST_160517.ipynb
32. 32. Terry Taewoong Um (terry.t.um@gmail.com) 32 2. DEFINE THE NN STRUCTURE
33. 33. Terry Taewoong Um (terry.t.um@gmail.com) 33 2. DEFINE THE NN STRUCTURE
34. 34. Terry Taewoong Um (terry.t.um@gmail.com) 34 3. SET OPTIMIZATION PARAMETERS
35. 35. Terry Taewoong Um (terry.t.um@gmail.com) 35 4. RUN
36. 36. Terry Taewoong Um (terry.t.um@gmail.com) 36 4. RUN (C.F.)
37. 37. Terry Taewoong Um (terry.t.um@gmail.com) 37 Thank you https://www.facebook.com/terryum http://terryum.io/ http://t-robotics.blogspot.kr/