Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to 딥러닝 개요 (2015-05-09 KISTEP)
Similar to 딥러닝 개요 (2015-05-09 KISTEP) (20)
More from Keunwoo Choi
More from Keunwoo Choi (10)
Recently uploaded
Recently uploaded (20)
딥러닝 개요 (2015-05-09 KISTEP)
- 1. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Deep Learning A brief explanation Keunwoo.Choi @qmul.ac.uk Centre for Digital Music, Queen Mary University of London, UK 1/24
- 2. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary 1 Introduction 2 Machine-Learning 3 Deep learning Overview Nonlinearity Weights SGD Training 4 Issues Overfitting Batch processing Back-propagation Other architectures ImageNet 5 Summary 2/24
- 3. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Keunwoo Choi PhD, QMUL, EECS, c4dm, 2014-present Supervised by Mark Sandler and George Fazekas Music Recommendation, (Deep) Machine Learning Internship, Naver Labs, July-Oct 2015 Visiting PhD, New York University, July-Dec 2016 ETRI, 2011-2014 3D Audio (WFS) Master’s, SNU EECS, 2009-2011 Applied Acoustics Laboratory, 3D Audio, Music Signal Processing Bachelor. SNU EECS, 2005-2009 3/24
- 4. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Research Topics Music Feature Extractions Analysis of deep CNNs (ISMIR LDB 2015, MLSP 2016) Auto-Tagging using deep CNN (ISMIR 2016) Playlist Generation RNN-based playlist generation (ICML workshop 2016) Music Captioning Automatic Composition Text-based chords and drums (CSMC 2016) 4/24
- 5. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Machine Learning More correctly, supervised learning Given a goal Given data x, y Train an algorithm that best matches x ! y and validated using unseen x (good generalisation) ”Do not memorise the examples!” Conventional approaches: Feature extraction + Classifier Researchers and experts hand-craft the features Classifier (e.g. SVM) is trained to achieve the goal 5/24
- 6. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Machine Learning Problems of the conventional approaches Hand-crafting takes resources E.g. MFCCs (speech recognition), Histogram of Gradient, SIFT (computer vision) Hand-crafting is not automatically optimisable but a Jang-in-jeong-sin thingy. Is a Jang-in better than machines? 6/24
- 7. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary NN vs. DNN 1 Logistic regression: No hidden layer Neural Networks: 1 hidden layer Deep NN: N hidden layers (N>1) 1 extremetech.com 7/24
- 8. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary DEMO TensorFlow Playground Logistic regression: No hidden layer Neural Networks: 1 hidden layer Deep NN: N hidden layers (N>1) Logistic regression Logistic regression fails NN works well! NN fails Shallow NN is okay The bigger, the better 8/24
- 9. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary DL Overview A motivation to deep leanring Brain and human sensory system Neurons are identical Many (100B) identical neurons with suitable structures Human learns by examples Human sensory systems are deep Parallel and serial neuron structure 9/24
- 10. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary DL Overview A motivation to deep leanring Do not need to hand-craft features Black box includes [feature extraction ! classifier] The whole procedure is computationally optimised to achieve the goal by iterative, heavy-computational methods have outperformed many Jang-in’s 10/24
- 11. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary Comparison Example task: Speech recognition Method Conventional ML Deep Learning Feature MFCCs (FFT ! mel-scale aggregation! DCT!time- derivative!ignore first coe↵!..) FFT!NN Classifier SVM, GMM NN Every computation, parameters, weights is automatically decided by during training 11/24
- 12. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary DL Nonlinearity Single layer performs a nonlinear mapping using Let x=input vector, y=output vector, NN: y = 2(W2 1(W1x)) DNN: Stacked (=deep) layers perform a more nonlinear and complex mapping y = 6(W6 5(W5 4(W4 3(W3 2(W2 1(W1x)))))) Stacked layers = stacked Nonlinearity! 2 Multiple linear layers, otherwise, can be compressed into one layer 2 best explained in Colah’s blog 12/24
- 13. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary DL Weights (= parameters) NN = nonlinear () and weights W For (), we use ReLU and its variants DNN = Combination of ReLU and many Wi ’s We want... the network to be trained to do the all dirty works - feature extraction and classification (=Wi ’s that do what we order to do) the network to learn by examples (=find the optimal W using training data) How do we train? ! SGD 13/24
- 14. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary Deep Learning How it learns - by SGD! SGD: Stochastic Gradient Descent SGD computationally finds w so that J(w) is minimised SGD iteratively finds w so that J(w) is minimised SGD gradually finds w so that J(w) is minimised w is updated to minimise J(w) (J(w) J(w) @J(w) @w ) ...if J(w) is di↵erentiable 14/24
- 15. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary Deep Learning How it learns - by SGD! Loss function J(w) A function that we want to minimise to achieve the goal yestimation = 4(W4 3(W3 2(W2 1(W1x)))) ytrue is given in the dataset E.g. l2: J(w) = (yestimation ytrue)2 Loss function measures how well the current algorithm is performing 15/24
- 16. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary Deep Learning How it learns We have (a set of) x and ytrue (aka dataset) We decide a loss function yestimation = 4(W4 3(W3 2(W2 1(W1x)))) J(w) = a function of (yestimation, ytrue) w is updated and becomes better weights = training is performed by SGD = the DNN is optimised 16/24
- 17. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Overview Nonlinearity Weights SGD Training Issues Summary Deep Learning The whole learning procedure Prepare a training dataset (x, y) Get a DNN configured (number of layers, nodes, loss function) for many times: for every x, y: (do SGD) compute yestimation = f (x, w) (go through DNN) update W according to the current loss, loss(ytrue , yestimation) Done! 17/24
- 19. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Overfitting Batch processing Back- propagation Other architectures ImageNet Summary Overfitting Overfitting When the network memorises the training data and fails to generalise 3 A general problem in ML Example: , 3 hrefhttp://cs231n.github.io/neural-networks-3/cs231n 19/24
- 20. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Overfitting Batch processing Back- propagation Other architectures ImageNet Summary Batch Gradient Descent Batch Gradient Descent Compute GD with seeing more than 1 examples simultaneously Every computation of yestimation = 4(W4 3(W3 2(W2 1(W1x)))) is done by matrix computations Quicker in GPU (because GPU is specialised at computing large matrix computations) Less zig-zag 4 4 www.holehouse.org 20/24
- 21. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Overfitting Batch processing Back- propagation Other architectures ImageNet Summary Back-propatagion aka backprop 5 The essence inside Gradient Descent of NN The way to compute the derivatives of all weights, @J(w) @w so that J(w) can be updated as J(w) @J(w) @w Discovered by Rumelhart, Hinton, and Williams (1986) 5 extremetech.com 21/24
- 22. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Overfitting Batch processing Back- propagation Other architectures ImageNet Summary Other architectures Convolutional Networks by LeCun (in Facebook AI Research and NYU) Biological visual systems Very widely used in almost every DL problem Recurrent networks Sequences (text) and time-series data (speech, weather, stock price,...) 22/24
- 23. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Overfitting Batch processing Back- propagation Other architectures ImageNet Summary ImageNet competition 6 14M images in 1K categories Have enabled to test new algorithms in DL 6 Slide from NVIDIA 23/24
- 24. Deep Learning Keunwoo.Choi @qmul.ac.uk Introduction Machine- Learning Deep learning Issues Summary Resources Deeplearning4j tutorials (Korean) ML lecture in Coursera, Stanford cs231n from Stanford 24/24