Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)

789 views

Published on

https://github.com/telecombcn-dl/dlmm-2017-dcu

Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)

  1. 1. Eva Mohedano eva.mohedano@insight-centre.org PhD Student Insight Centre for Data Analytics Dublin City University Deep Neural Networks
  2. 2. Overview ● ● ● ●
  3. 3. Perceptron (Neuron)
  4. 4. Linear decision decision boundary Class 0 Class 1 x
  5. 5. Linear decision decision boundary Class 0 Class 1 Parameters of the line. They are find based on training data - Learning Stage. x
  6. 6. Limitations: XOR problem Input 1 Input 2 Desired Output 0 0 0 0 1 1 1 0 1 1 1 0 XOR logic table 1 0 0 1 Input 1 Input 2 ?
  7. 7. Non-linear decision boundaries ● ● ● 7
  8. 8. Non-linear decision boundaries ○ ○ ○ ○ ○ ○ ○ ○ 8
  9. 9. Principle of deep learning ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ Data Pixels Deep learning Convolutional Neural Network Data Pixels Low-level features SIFT Representation BoW/VLAD/Fisher Classifier SVM EngineeredUnsupervisedSupervised Supervised 9
  10. 10. Example: feature engineering in computer vision 10 ϕ(x)
  11. 11. Neural networks: single neuron We already seen the single neuron. This is just a linear classifier (or regressor) Inputs: ● x1 , x2 Parameters ● w1 , w2 , b
  12. 12. Neural networks A composition of these simple neurons into several layers Each neuron simply computes a linear combination of its inputs, adds a bias, and passes the result through an activation function g(x) The network can contain one or more hidden layers. The outputs of these hidden layers can be thought of as a new representation of the data (new features). The final output is the target variable (y = f(x))
  13. 13. Activation functions ● ● ● ● ● ● Sigmoid Tanh ReLU 13 If we only use linear layers we are only able to learn linear transformations of our input.
  14. 14. Multilayer perceptrons 14 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 When each node in each layer is a linear combination of all inputs from the previous layer then the network is called a multilayer perceptron (MLP) Weights can be organized into matrices. Fully connected network Forward pass computes h0 h1 h2 h3
  15. 15. Multilayer perceptrons 15 Forward pass computes w11 w12 w13 w14 w21 w22 w23 w24 w31 w32 w33 w34 w41 w42 w43 w44 W1 x1 x2 x3 x4 b1 b2 b3 b4 b1 h0 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 h0 h1 h2 h3
  16. 16. Multilayer perceptrons 16 Forward pass computes w11 w12 w13 w14 w21 w22 w23 w24 w31 w32 w33 w34 w41 w42 w43 w44 W1 x1 x2 x3 x4 b1 b2 b3 b4 b1 h0 x1 x2 x3 x4 y1 Layer 1 Layer 2 Layer 3 Layer 0 y2 h0 h1 h2 h3
  17. 17. Universal approximation theorem Universal approximation theorem states that “the standard multilayer feed-forward network with a single hidden layer, which contains finite number of hidden neurons, is a universal approximator among continuous functions on compact subsets of Rn , under mild assumptions on the activation function.” If a 2 layer NN is a universal approximator, then why do we need deep nets?? The universal approximation theorem: ● Says nothing about the how easy/difficult it is to fit such approximators ● Needs a “finite number of hidden neurons”: finite may be extremely large In practice, deep nets can usually represent more complex functions with less total neurons (and therefore, less parameters)
  18. 18. Example: MNIST digit classification ● ● ● ● ● ● ● ● 18
  19. 19. Example: MNIST digit classification ● ● ● ● ● 28 28 19 Layer #Weights #Biases Total 1 784 x 512 512 401,920 2 512 x 512 512 262,656 3 512 x 10 10 5,130 669,706
  20. 20. Example: MNIST digit classification ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20
  21. 21. Permutation invariant MNIST ● ● ● ● ● ● Permute 21
  22. 22. Convolutional neural networks (CNNs, convnets) ● ● 22
  23. 23. 1D convolution 23 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter dot product -3 = .
  24. 24. 1D convolution 24 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 . .
  25. 25. 1D convolution 25 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 -7 . . .
  26. 26. 1D convolution 26 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 9 -7 -4 13 -6 . . . . . .
  27. 27. 1D convolution 27 0 1 5 0 2 8 1 0 -1 2 -1 Signal Kernel/filter -3 -7 13 . . . Hyperparameters
  28. 28. Convolution on a grid 28
  29. 29. Convolution on a volume 32 32 5 5 3 A 5x5 convolution on a volume of depth 3 (e.g. an image) needs a filter (kernel) with 5x5x3 elements (weights) + a bias Andrej Karpathy's demo: http://cs231n.github.io/convolutional-networks/# conv
  30. 30. Convolution with multiple filters Stack results into a 3D volume (depth 4) Apply filter 1 Apply filter 2 Apply filter 3 Apply filter 4
  31. 31. Pooling layers 1 5 0 2 8 1 10 2 4 9 0 3 8 9 3 7 8 2 3 8 9 6 0 5 16 7 2 2 7 3 6 3 0 5 2 2 10 9 16 7
  32. 32. Convnets ● ● LeNet-5 [LeCun et al. 1998] 32
  33. 33. Alexnet ● 8 parameter layers (5 convolution, 3 fully connected) ● Softmax output ● 650,000 units ● 60 million free parameters ● Trained on two GPUs (two streams) for a week ● Ensemble of 7 nets used in ILSVRC challenge Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS, 2012.
  34. 34. Features of Alexnet: Convolutions
  35. 35. Features of Alexnet: max pooling
  36. 36. Features of Alexnet: ReLu Sigmoid ReLU
  37. 37. Filters learnt by Alexnet Visualization of the 96 11 x 11 filters learned by bottom layer Krizhevsky et al. ImageNet classification with deep convolutional neural networks. NIPS, 2012.
  38. 38. Example: a convnet on MNIST ● ● ● ● ● ● ● ● ● ● https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py 38
  39. 39. Advantages of convnets ● ○ ○ ● ○ ● ○ ● ○ ● ○ ● 39
  40. 40. Summary ● ● ● ● ● 40
  41. 41. Questions? 41
  42. 42. ● ● ● ● ● Layer #Weights #Biases Total 1 784 x 512 512 401,920 2 512 x 512 512 262,656 3 512 x 10 10 5,130 669,706 ● ● ● ● ● ● ● Layer #Weights #Biases Total 1 3 x 3x 32 1 10 2 3x3x32 32 320 3 14X14x128 128 25,216 4 128X10 10 1,290 35,526

×