![What is a neuron?
25
• 3 inputs [x1,x2,x3]
• 3 weights [w1,w2,w3]
• 1 bias term b
• Element-wise multiply and sum: inputs & weights
• Add bias term
• Apply activation function f
• Output of neuron is just a number (like the in](https://image.slidesharecdn.com/pyconde17-171025091553/75/Deep-Learning-for-Computer-Vision-PyconDE-2017-24-2048.jpg)
![What is an Activation Function?
26
Sigmoid Tanh ReLU
Nonlinearities … “squashing functions” … transform neuron’s
output
NB: sigmoid output in [0,1]](https://image.slidesharecdn.com/pyconde17-171025091553/75/Deep-Learning-for-Computer-Vision-PyconDE-2017-25-2048.jpg)
![What is a Convolutional Neural
Network?
43
“like a ordinary neural network but with
special types of layers that work well
on images”
(math works on numbers)
• Pixel = 3 colour channels (R, G, B)
• Pixel intensity ∈[0,255]
• Image has width w and height h
• Therefore image is w x h x 3 numbers](https://image.slidesharecdn.com/pyconde17-171025091553/75/Deep-Learning-for-Computer-Vision-PyconDE-2017-42-2048.jpg)
![Softmax
Convert scores ∈ ℝ to probabilities ∈ [0,1]
Final output prediction = highest probability class
67](https://image.slidesharecdn.com/pyconde17-171025091553/75/Deep-Learning-for-Computer-Vision-PyconDE-2017-65-2048.jpg)
More Related Content
Similar to Deep Learning for Computer Vision - PyconDE 2017
![[DSC Europe 25] Marija Vlajkovic & Andrea Radonjanin - Integration of AI tool...](https://cdn.slidesharecdn.com/ss_thumbnails/qf1jrglttoc3bm8s3aop-final-integration-of-ai-tools-251208151905-394f3a6a-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...](https://cdn.slidesharecdn.com/ss_thumbnails/xer2bb6nrdc8pdpev0pc-8-251204082258-7c2fa4a1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Boris Perkovic - Lost in performance.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/uq5hrp7vsuahqkxzifux-1-251204082258-fd2ee09d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...](https://cdn.slidesharecdn.com/ss_thumbnails/7nw2xxixrxqdxvrb5wca-6-251205085714-ab09a2ac-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Max Talanov - Non digital NNs.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/wif8tr3gtua74qvtopke-non-digital-nns-251205090438-26b0eea6-thumbnail.jpg?width=640&height=640&fit=bounds)
Deep Learning for Computer Vision - PyconDE 2017
- 1. Deep Learning for ComputerVision Executive-ML 2017/09/21 Neither Proprietary nor Confidential – Please Distribute ;) Alex Conway alex @ numberboost.com @alxcnwy PyConDE’17 Neither confidential nor proprietary - please distribute ;)
- 2.
- 3. Deep Learning isSexy (for a reason!) 3
- 4. Image Classification 4 http://yann.lecun.com/exdb/mnist/ https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py (99.25% testaccuracy in 192 seconds and 46 lines of code)
- 5.
- 6.
- 7. Image Classification 7 ImageNet Classificationwith Deep Convolutional Neural Networks, Krizhevsky et. Al. Advances in Neural Information Processing Systems 25 (NIPS2012)
- 8.
- 9.
- 10.
- 11. Image Captioning &Visual Attention XXX 11 https://einstein.ai/research/knowing-when-to-look-adaptive- attention-via-a-visual-sentinel-for-image-captioning
- 12.
- 13.
- 14.
- 15.
- 16. 16 Original input Rear Window(1954) Pix2pix output Fully Automated Remastered Painstakingly by Hand https://hackernoon.co m/remastering-classic- films-in-tensorflow- with-pix2pix- f4d551fa0503
- 17.
- 18. Style Season Transfer https://github.com/junyanz/CycleGAN18 I WISH!1! ;)
- 19.
- 20.
- 21. Deep learning isMagicDeep learning is MagicDeep learning is EASY!
- 22. 1. What isa neural network? 2. What is a convolutional neural network? 3. How to use a convolutional neural network 4. More advanced Methods 5. Case studies & applications 22
- 23. 1.What is aneural network?
- 24. What is aneuron? 25 • 3 inputs [x1,x2,x3] • 3 weights [w1,w2,w3] • 1 bias term b • Element-wise multiply and sum: inputs & weights • Add bias term • Apply activation function f • Output of neuron is just a number (like the in
- 25. What is anActivation Function? 26 Sigmoid Tanh ReLU Nonlinearities … “squashing functions” … transform neuron’s output NB: sigmoid output in [0,1]
- 26. So what isa Neural Network?
- 27. What is a 28 Inputsoutputs hidden layer 1 hidden layer 2 hidden layer 3 Note: Outputs of one layer are inputs into the next layer This (non-convolutional) architecture is called a “multi-layered perceptron” Neural Network?(Deep)
- 28.
- 29.
- 30. 1 Layer NeuralNetwork 31https://ml4a.github.io/ml4a/looking_inside_neural_nets/ 0 1 2 3 4 5 6 7 8 9
- 31. 2 Layer NeuralNetwork 32https://ml4a.github.io/ml4a/looking_inside_neural_nets/ 0 1 2 3 4 5 6 7 8 9
- 32.
- 33.
- 34.
- 35.
- 36. How does aneural network learn? 37 • We need labelled examples “training data” • We initialize network weights randomly and initially get random predictions • For each labelled training data point, we calculate the error (cross-entropy error for classification or mean-squared error for regression) between the network’s predictions and the ground-truth labels • Use ‘backpropagation’ (chain rule), to update the network parameters (weights + convolutional filters ) in the opposite direction to the error
- 37. “How much error increases whenwe increase this weight” How does a neural network learn? 38 New weight = Old weight Learning rate- Gradient of Error with respect to Weight ( )x
- 38.
- 39.
- 40. What is aNeural Network? For much more detail, see: 1. Michael Nielson’s Neural Networks & Deep Learning free online book http://neuralnetworksanddeeplearning.com/chap1.html 2. Anrej Karpathy’s CS231n Notes http://neuralnetworksanddeeplearning.com/chap1.html 41
- 41. 2. What isa convolutional neural network?
- 42. What is aConvolutional Neural Network? 43 “like a ordinary neural network but with special types of layers that work well on images” (math works on numbers) • Pixel = 3 colour channels (R, G, B) • Pixel intensity ∈[0,255] • Image has width w and height h • Therefore image is w x h x 3 numbers
- 43. First of allwhy do we need special layers? 44 1. Locality – objects tend to have a local spatial support 2. Translation Invariance – objects can appear anywhere in the image 3. Parameter counts explode in fully connected nets – recall 3 layer 28x28 pixel MNIST net had 13’002 parameters… if we instead had 500x500 pixel images, we’d need 4’000’458 4. We can get better results ;) – Keras MLP (98.4% accuracy = 1.6% error) vs CNN (99.25% = 0.75 error) So more than half the error rate https://www.youtube.com/watch?v=t_TY_5bG9J8
- 44. 45 This is VGGNet– don’t panic, we’ll break it down piece by piece Example Architecture
- 45. 46 This is VGGNet– don’t panic, we’ll break it down piece by piece Example Architecture
- 46.
- 47.
- 48. New Layer Type:Convolutional Layer 49 • 2-d weighted average when multiply kernel over pixel patches • We slide the kernel over all pixels of the image (handle borders) • Kernel starts off with “random” values and network updates (learns) the kernel values (using backpropagation) to try minimize loss • Kernels shared across the whole image (parameter
- 49. Many Kernels =Many “Activation Maps” = Volume 50http://cs231n.github.io/convolutional-networks/
- 50. New Layer Type:Convolutional Layer 51
- 51.
- 52.
- 53.
- 54.
- 55. Convolution Learn HierarchicalFeatures 56
- 56. New Layer Type:Max Pooling 58
- 57. New Layer Type:Max Pooling • Reduces dimensionality from one layer to next • …by replacing NxN sub-area with max value • Makes the network less sensitive to local perturbations • Makes network “look” at larger areas of the image at a time • e.g. Instead of identifying fur, identify cat • Reduces overfitting since losing information helps the network generalize 59http://cs231n.github.io/convolutional-networks/
- 58. New Layer Type:Max Pooling 60
- 59. Stack Conv +Pooling Layers and Go Deep 61 Convolution + max pooling + fully connected + softmax
- 60. 62 Stack Conv +Pooling Layers and Go Deep Convolution + max pooling + fully connected + softmax
- 61. 63 Stack These Layersand Go Deep Convolution + max pooling + fully connected + softmax
- 62. 64 Stack These Layersand Go Deep Convolution + max pooling + fully connected + softmax
- 63. 65 Flatten the Final“Bottleneck” layer Convolution + max pooling + fully connected + softmax Flatten the final 7 x 7 x 512 max pooling layer Add fully-connected dense layer on top
- 64. 66 Bringing it alltogether Convolution + max pooling + fully connected + softmax
- 65. Softmax Convert scores ∈ℝ to probabilities ∈ [0,1] Final output prediction = highest probability class 67
- 66. Bringing it alltogether 68 Convolution + max pooling + fully connected + softmax
- 67. We need labelledtraining data!
- 68. ImageNet 70 http://image-net.org/explore 1000 object categories 1.2million training images
- 69.
- 70.
- 71. ImageNet Top 5Error Rate 73 Traditional Image Processing Methods AlexNet 8 Layers ZFNet 8 Layers GoogLeNe t 22 Layers ResNet 152 Layers SENet Ensamble TSNet Ensamble
- 72. 3. How touse a convolutional neural network
- 73. Using a Pre-TrainedImageNet-Winning CNN 75 • We’ve been looking at “VGGNet” • Oxford Visual Geometry Group (VGG) • ImageNet 2014 Runner-up • Network is 16 layers (deep!) • Trained on 4 GPUs for 2-3 weeks (data split across GPUs) • Easy to fine-tune
- 74. Example: Classifying ProductImages 76 https://github.com/alexcnwy/DeepLearning4Computer Vision Classifying products into 8 categories
- 75.
- 76. 78 https://blog.keras.io/building-powerful-image-classification-models-using-very-little- data.html Start with Pre-TrainedImageNet Model Consumers vs Producers of Machine Lea
- 77.
- 78. …but we don’twant to ImageNet categories… we want to predict something new!
- 79. “Transfer Learning” to therescue!1!
- 80. Fine-tuning A CNNTo Solve A New Problem • Cut off last layer of pre-trained Imagenet winning CNN • Keep learned feature detectors (convolutions) • …BUT replace the dense and softmax layers with new ones! • Can learn to predict new (completely different) classes • Fine-tuning is re-training the new final layer(s) 82
- 81. Fine-tuning A CNNTo Solve A New Problem 83 8
- 82.
- 83. 85 After Fine-Tuning Replace Finallayers of Pre-trained Model
- 84. Fine-tuning A CNNTo Solve A New Problem • Load pre-trained VGG network • Remove final dense & softmax layers that predict 1000 ImageNet classes • Replace with new dense & softmax layer to predict 8 categories 86 96.3% accuracy in under 2 minutes for classifying products into categories (WITH ONLY 3467 TRAINING IMAGES!!1!) Fine-tuning is awesome! Insert obligatory brain analogy
- 85. Brief Error Analysis “Truth”= tee Prediction = shirt “Truth” = jacket Prediction = knitwear
- 87. Visual Similarity 89 • Chopoff VGG dense layers and keep bottleneck features • Add single dense layer with 256 activations • “Predict” to get activation for each image • Compute nearest neighbours in the space of these https://memeburn.com/2017/06/spree-image-search/
- 88.
- 89. 91 Input Image not seenby model Results Top 10 most “visually similar”
- 90. 92 Input Image not seenby model Results Top 10 most “visually similar”
- 91. 93 Final Convolutional Layer= Semantic Vector The final convolutional layer encodes everything the network needs to make predictions
- 92. 94 Let’s Visualize theData with PCA http://setosa.io/ev/principal-component-analysis/
- 93.
- 94. Use a BetterArchitecture (or all of them!) 96 “Ensambles win” learn a weighted average of many models’ predictions
- 95. cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf There are MANYComputer Vision Tasks
- 96. Long et al.“Fully Convolutional Networks for Semantic Segmentation” CVPR 2015 Noh et al. Learning Deconvolution Network for Semantic Segmentation. IEEE on Computer Vision 2016 Semantic Segmentation
- 97.
- 98. http://blog.romanofoti.com/style_transfer/ Johnson et al.Perceptual losses for real-time style transfer and super-resolution. 2016 Style Transfer f ( ) =,
- 99.
- 100.
- 101. This image is 3.8kb Super-Resolution
- 102.
- 103. Image Captioning https://einstein.ai/research/knowing-when-to-look-adaptive-attention-via-a-visual-sentinel-for-image- captioning Karpathy &Li. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), pp. 664–676
- 104.
- 105.
- 106.
- 107. king + woman– man ≈ queen 111 Frome et al. (2013) ‘DeViSE: A Deep Visual-Semantic Embedding Model’, Advances in Neural Information Processing Systems, pp. 2121–2129 CNN + Word2Vec = AWESOME
- 108. DeViSE: A DeepVisual-Semantic Embedding Model XXX 112 Before: Encode labels as 1-hot vector
- 109. DeViSE: A DeepVisual-Semantic Embedding Model XXX 113 After: Encode labels as word2vec vectors (FROM A SEPARATE MODEL) Can look these up for all the nouns in ImageNet 300-d word2vec vectors
- 110. DeViSE: A DeepVisual-Semantic Embedding Model wv(fish)+ wv(net) 114 https://www.youtube.com/watch?v=uv0gmrXSXVg 2 wv* = …get nearest neighbours to wv*
- 112. Multi-Modal Modeling –Mix & Match! Structured data e.g. #beds, #garages, m2, etc. Features extracted from description e.g. with an LSTM Features extracted from photos using convolutional network Price
- 113.
- 114. Estimating Accident RepairCost from Photos TODO 118 Prototype for large S.A. insurer Detect car make & model from license sticker on windshield Predict repair cost using learned model
- 115. Image & VideoModeration TODO 119 Large international gay dating app with tens of millions of users uploading hundreds-of-thousands of photos per day
- 116.
- 117. m Counting People TODO Count shoppers,segment on age & gender facial recognition loyalty is next
- 118.
- 119. “Using the finalmodel that has been trained on survey data, we can estimate per capita consumption expenditure for any location where we have daytime satellite imagery ” http://sustain.stanford.edu/predicting-poverty/ “…features visible in daytime satellite imagery, such as roofing material and distance to urban areas, vary roughly linearly with expenditure”
- 120.
- 121. Extracting Data fromProduct Catalogues
- 122. Real-time ATM VideoClassification 126
- 123.
- 124. We are hiring! (andavailable for hire ;) alex @ numberboost.com
- 125. GET IN TOUCH! AlexConway alex @ numberboost.com @alxcnwy