Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to PyConZA'17 Deep Learning for Computer Vision
Similar to PyConZA'17 Deep Learning for Computer Vision (20)
Recently uploaded
Recently uploaded (20)
PyConZA'17 Deep Learning for Computer Vision
- 1. Deep Learning for Computer Vision Executive-ML 2017/09/21 Neither Proprietary nor Confidential – Please Distribute ;) Alex Conway alex @ numberboost.com @alxcnwy PyConZA’17
- 2. Hands up!
- 3. Check out the Deep Learning Indaba videos & practicals! http://www.deeplearningindaba.com/videos.html http://www.deeplearningindaba.com/practicals.html
- 4. Deep Learning is Sexy (for a reason!) 4
- 5. Image Classification 5 http://yann.lecun.com/exdb/mnist/ https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py (99.25% test accuracy in 192 seconds and 70 lines of code)
- 7. Image Classification 7 ImageNet Classification with Deep Convolutional Neural Networks, Krizhevsky et. Al. Advances in Neural Information Processing Systems 25 (NIPS2012)
- 10. Object detection
- 11. Image Captioning & Visual Attention XXX 11 https://einstein.ai/research/knowing-when-to-look-adaptive- attention-via-a-visual-sentinel-for-image-captioning
- 16. 16 Original input Rear Window (1954) Pix2pix output Fully Automated Remastered Painstakingly by Hand https://hackernoon.co m/remastering-classic- films-in-tensorflow- with-pix2pix- f4d551fa0503
- 20. Deep learning is MagicDeep learning is MagicDeep learning is EASY!
- 21. 1. What is a neural network? 2. What is a convolutional neural network? 3. How to use a convolutional neural network 4. More advanced Methods 5. Case studies & applications 21
- 22. 1.What is a neural network?
- 23. What is a neuron? 24 • 3 inputs [x1,x2,x3] • 3 weights [w1,w2,w3] • Element-wise multiply and sum • Apply activation function f • Often add a bias too (weight of 1) – not
- 24. What is an Activation Function? 25 Sigmoid Tanh ReLU Nonlinearities … “squashing functions” … transform neuron’s output NB: sigmoid output in [0,1]
- 25. What is a (Deep) Neural Network? 26 Inputs outputs hidden layer 1 hidden layer 2 hidden layer 3 Outputs of one layer are inputs into the next layer
- 26. How does a neural network learn? 27 • We need labelled examples “training data” • We initialize network weights randomly and initially get random predictions • For each labelled training data point, we calculate the error between the network’s predictions and the ground-truth labels • Use ‘backpropagation’ (chain rule), to update the network parameters (weights + convolutional filters ) in the opposite direction to the error
- 27. How does a neural network learn? 28 New weight = Old weight Learning rate- Gradient of weight with respect to Error( )x “How much error increases when we increase this weight”
- 30. What is a Neural Network? For much more detail, see: 1. Michael Nielson’s Neural Networks & Deep Learning free online book http://neuralnetworksanddeeplearning.com/chap1.html 2. Anrej Karpathy’s CS231n Notes http://neuralnetworksanddeeplearning.com/chap1.html 31
- 31. 2. What is a convolutional neural network?
- 32. What is a Convolutional Neural Network? 33 “like a ordinary neural network but with special types of layers that work well on images” (math works on numbers) • Pixel = 3 colour channels (R, G, B) • Pixel intensity ∈[0,255] • Image has width w and height h • Therefore image is w x h x 3 numbers
- 33. 34 This is VGGNet – don’t panic, we’ll break it down piece by piece Example Architecture
- 34. 35 This is VGGNet – don’t panic, we’ll break it down piece by piece Example Architecture
- 37. New Layer Type: Convolutional Layer 38 • 2-d weighted average when multiply kernel over pixel patches • We slide the kernel over all pixels of the image (handle borders) • Kernel starts off with “random” values and network updates (learns) the kernel values (using backpropagation) to try minimize loss • Kernels shared across the whole image (parameter
- 38. Many Kernels = Many “Activation Maps” = Volume 39http://cs231n.github.io/convolutional-networks/
- 39. New Layer Type: Convolutional Layer 40
- 40. Convolutions 41 https://github.com/fchollet/keras/blob/master/examples/conv_filter_visuali zation.py
- 41. Convolutions 42
- 42. Convolutions 43
- 43. Convolutions 44
- 44. Convolution Learn Hierarchical Features 45
- 45. New Layer Type: Max Pooling 47
- 46. New Layer Type: Max Pooling • Reduces dimensionality from one layer to next • …by replacing NxN sub-area with max value • Makes network “look” at larger areas of the image at a time • e.g. Instead of identifying fur, identify cat • Reduces overfitting since losing information helps the network generalize 48http://cs231n.github.io/convolutional-networks/
- 47. New Layer Type: Max Pooling 49
- 48. Stack Conv + Pooling Layers and Go Deep 50 Convolution + max pooling + fully connected + softmax
- 49. 51 Stack Conv + Pooling Layers and Go Deep Convolution + max pooling + fully connected + softmax
- 50. 52 Stack These Layers and Go Deep Convolution + max pooling + fully connected + softmax
- 51. 53 Stack These Layers and Go Deep Convolution + max pooling + fully connected + softmax
- 52. 54 Flatten the Final “Bottleneck” layer Convolution + max pooling + fully connected + softmax Flatten the final 7 x 7 x 512 max pooling layer Add fully-connected dense layer on top
- 53. 55 Bringing it all together Convolution + max pooling + fully connected + softmax
- 54. Softmax Convert scores ∈ ℝ to probabilities ∈ [0,1] Final output prediction = highest probability class 56
- 55. Bringing it all together 57 Convolution + max pooling + fully connected + softmax
- 56. We need labelled training data!
- 57. ImageNet 59 http://image-net.org/explore 1000 object categories 1.2 million training images
- 58. ImageNet 60
- 59. ImageNet 61
- 60. ImageNet Top 5 Error Rate 62 Traditional Image Processing Methods AlexNet 8 Layers ZFNet 8 Layers GoogLeNe t 22 Layers ResNet 152 Layers SENet Ensamble TSNet Ensamble
- 61. 3. How to use a convolutional neural network
- 62. Using a Pre-Trained ImageNet-Winning CNN 64 • We’ve been looking at “VGGNet” • Oxford Visual Geometry Group (VGG) • ImageNet 2014 Runner-up • Network is 16 layers (deep!) • Easy to fine-tune https://blog.keras.io/building-powerful-image-classification- models-using-very-little-data.html
- 63. Example: Classifying Product Images 65 https://github.com/alexcnwy/CTDL_CNN_TALK_20 170620 Classifying products into 9 categories
- 64. 66 https://blog.keras.io/building-powerful-image-classification-models-using-very-little- data.html Start with Pre-Trained ImageNet Model Consumers vs Producers of Machine Lea
- 65. 67
- 66. “Transfer Learning” is a game changer
- 67. Fine-tuning A CNN To Solve A New Problem • Cut off last layer of pre-trained Imagenet winning CNN • Keep learned network (convolutions) but replace final layer • Can learn to predict new (completely different) classes • Fine-tuning is re-training new final layer - learn for new task 69
- 68. Fine-tuning A CNN To Solve A New Problem 70
- 71. Fine-tuning A CNN To Solve A New Problem • Fix weights in convolutional layers (set trainable=False) • Remove final dense layer that predicts 1000 ImageNet classes • Replace with new dense layer to predict 9 categories 73 88% accuracy in under 2 minutes for classifying products into categories Fine-tuning is awesome! Insert obligatory brain analogy
- 73. Visual Similarity 75 • Chop off last 2 VGG layers • Use dense layer with 4096 activations • Compute nearest neighbours in the space of these activations https://memeburn.com/2017/06/spree-image-search/
- 75. 77 Input Image not seen by model Results Top 10 most “visually similar”
- 76. 78 Final Convolutional Layer = Semantic Vector • The final convolutional layer encodes everything the network needs to make predictions • The dense layer added on top and the softmax layer both have lower dimensionality
- 78. Use a Better Architecture (or all of them!) 80 “Ensambles win” learn a weighted average of many models’ predictions
- 79. cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf There are MANY Computer Vision Tasks
- 80. Long et al. “Fully Convolutional Networks for Semantic Segmentation” CVPR 2015 Noh et al. Learning Deconvolution Network for Semantic Segmentation. IEEE on Computer Vision 2016 Semantic Segmentation
- 82. http://blog.romanofoti.com/style_transfer/ Johnson et al. Perceptual losses for real-time style transfer and super-resolution. 2016 Style Transfer f ( ) =,
- 85. This image is 3.8 kb Super-Resolution
- 87. Image Captioning https://einstein.ai/research/knowing-when-to-look-adaptive-attention-via-a-visual-sentinel-for-image- captioning Karpathy & Li. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), pp. 664–676
- 91. king + woman – man ≈ queen 95 Frome et al. (2013) ‘DeViSE: A Deep Visual-Semantic Embedding Model’, Advances in Neural Information Processing Systems, pp. 2121–2129 CNN + Word2Vec = AWESOME
- 92. DeViSE: A Deep Visual-Semantic Embedding Model XXX 96 Before: Encode labels as 1-hot vector
- 93. DeViSE: A Deep Visual-Semantic Embedding Model XXX 97 After: Encode labels as word2vec vectors (FROM A SEPARATE MODEL) Can look these up for all the nouns in ImageNet 300-d word2vec vectors
- 94. DeViSE: A Deep Visual-Semantic Embedding Model wv(fish)+ wv(net) 98 https://www.youtube.com/watch?v=uv0gmrXSXVg 2 wv* = …get nearest neighbours to wv*
- 95. 5. Case Studies
- 96. Estimating Accident Repair Cost from Photos TODO 100 Prototype for large SA insurer Detect car make & model from registration disk Predict repair cost using learned
- 97. Image & Video Moderation TODO 101 Large international gay dating app with tens of millions of users uploading hundreds-of-thousands of photos per day
- 99. m Counting People TODO Count shoppers, segment on age & gender facial recognition loyalty is next
- 100. Counting Cars TODO
- 101. Detecting Potholes
- 102. Extracting Data from Product Catalogues
- 103. Real-time ATM Video Classification 107
- 105. We are hiring! alex @ numberboost.com
- 106. GET IN TOUCH! Alex Conway alex @ numberboost.com @alxcnwy
Editor's Notes
- SENet = Squeeze and Excitation Net. TSNet = Trimps-Soushen
- Gets 88% accuracy in classifying products into categories