Download as PDF, PPTX
![Gluon CV: classification, detection, segmentation
https://github.com/dmlc/gluon-cv
[electric_guitar],
with probability 0.671](https://image.slidesharecdn.com/20180606berlinsummit-180606151740/75/Deep-Dive-on-Deep-Learning-June-2018-35-2048.jpg)
Uploaded byJulien SIMON
Deep Dive on Deep Learning (June 2018)
This document provides a summary of a presentation on deep learning concepts, common architectures, Apache MXNet, and infrastructure for deep learning. The agenda includes an overview of deep learning concepts like neural networks and training, common architectures like convolutional neural networks and LSTMs, a demonstration of Apache MXNet's symbolic and imperative APIs, and a discussion of infrastructure for deep learning on AWS like optimized EC2 instances and Amazon SageMaker.
More Related Content
Similar to Deep Dive on Deep Learning (June 2018)
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Deep Dive on Deep Learning (June 2018)
- 1. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AI and Machine Learning @julsimon Deep Dive on Deep Learning
- 2. • Deep Learningconcepts • Common architectures and use cases • Apache MXNet • Infrastructure for Deep Learning • Demos along the way: MXNet, Gluon, Keras, TensorFlow, PyTorch ☺ Agenda
- 3.
- 4. Activation functions i=1 l xi ∗wi = u ”Multiply and Accumulate” Source: Wikipedia The neuron
- 5. x = x11, x12,…. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 6. x = x11, x12,…. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories Total number of predictions Accuracy = Number of correct predictions 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 7. Initially, the networkwill not predict correctly f(X1) = Y’1 A loss function measures the difference between the real label Y1 and the predicted label Y’1 error = loss(Y1, Y’1) For a batch of samples: 𝑖=1 𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒 loss(Yi, Y’i) = batch error The purpose of the training process is to minimize error by gradually adjusting weights. Neural networks
- 8. Training data setTraining Trained neural network Batch size Learning rate Number of epochs Hyper parameters Backpropagation Training
- 9. Stochastic Gradient Descent Imagineyou stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat—this is exactly what gradient descent does. Tim Dettmers University of Lugano 2015 https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ The « step size » depends on the learning rate z=f(x,y)
- 10. Local minima andsaddle points « Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of saddle points? Answering these questions definitively is difficult, but we present evidence strongly suggesting that the answer to all of these questions is no. » « Qualitatively characterizing neural network optimization problems », Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544
- 11. https://medium.com/@julsimon/tumbling-down-the-sgd-rabbit-hole-part-1-740fa402f0d7 SGD works remarkablywell and is still widely used. Adaptative optimizers use a variable learning rate. Some even use a learning rate per dimension (Adam). Optimizers
- 12. Validation data set (alsocalled dev set) Neural network in training Validation accuracy Prediction at the end of each epoch This data set must have the same distribution as real-life samples, or else validation accuracy won’t reflect real-life accuracy. Validation
- 13. Test data setFully trained neural network Test accuracy Prediction at the end of experimentation This data set must have the same distribution as real-life samples, or else test accuracy won’t reflect real-life accuracy. Test
- 14. Training accuracy Loss function Accuracy 100% Epochs Validationaccuracy Loss Best epoch OVERFITTING « Deep Learning ultimately is about finding a minimum that generalizes well, with bonus points for finding one fast and reliably », Sebastian Ruder Early stopping
- 15.
- 16. Convolutional Neural Networks(CNN) Le Cun, 1998: handwritten digit recognition, 32x32 pixels https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
- 17. Source: http://timdettmers.com Extracting featureswith convolution Convolution extracts features automatically. Kernel parameters are learned during the training process.
- 18. Downsampling images withpooling Source: Stanford University Pooling shrinks images while preserving significant information.
- 19.
- 20.
- 21. Text Detection andRecognition https://github.com/Bartzi/stn-ocr MXNet
- 22. Face Detection https://github.com/tornadomeet/mxnet-face https://github.com/deepinsight/insightface https://arxiv.org/abs/1801.07698 FaceRecognition LFW 99.80%+ Megaface 98%+ with a single model MXNet MXNet
- 23.
- 24. Long Short TermMemory Networks (LSTM) • A LSTM neuron computes the output based on the input and a previous state • LSTM networks have memory • They’re great at predicting sequences, e.g. machine translation
- 25.
- 26. GAN: Welcome tothe (un)real world, Neo Generating new ”celebrity” faces https://github.com/tkarras/progressive_growing_of_gans From semantic map to 2048x1024 picture https://tcwang0509.github.io/pix2pixHD/ TF PyTorch
- 27. Wait! There’s more! Modelscan also generate text from text, text from images, text from video, images from text, sound from video, 3D models from 2D images, etc.
- 28.
- 29. Apache MXNet: OpenSource library for Deep Learning Programmable Portable High Performance Near linear scaling across hundreds of GPUs Highly efficient models for mobile and IoT Simple syntax, multiple languages Most Open Best On AWS Optimized for Deep Learning on AWS Accepted into the Apache Incubator
- 30. Input Output 1 11 1 0 1 0 0 0 3 mx. sym. Convol ut i on( dat a, ker nel =( 5, 5) , num_f i l t er =20) mx. sym. Pool i ng( dat a, pool _t ype=" max" , ker nel =( 2, 2) , st r i de=( 2, 2) l st m. l st m_unr ol l ( num_l st m_l ayer , seq_l en, l en, num_hi dden, num_embed) 4 2 2 0 4=Max 1 3 ... 4 0.2 -0.1 ... 0.7 mx. sym. Ful l yConnect ed( dat a, num_hi dden=128) 2 mx. symbol . Embeddi ng( dat a, i nput _di m, out put _di m = k) 0.2 -0.1 ... 0.7 Queen 4 2 2 0 2=Avg Input Weights cos(w, queen ) = cos(w, k i n g) - cos(w, m an ) + cos(w, w om an ) mx. sym. Act i vat i on( dat a, act _t ype=" xxxx" ) " r el u" " t anh" " si gmoi d" " sof t r el u" Neural Art Face Search Image Segmentation Image Caption “ People Riding Bikes” Bicycle, People, Road, Sport Image Labels Image Video Speech Text “ People Riding Bikes” Machine Translation “ Οι άνθρωποι ιππασίας ποδήλατα” Events mx. model . FeedFor war d model . f i t mx. sym. Sof t maxOut put
- 31. • More chancesfor optimization • Language independent • E.g. TensorFlow, Theano, Caffe, MXNet Symbol API • Less flexible • ’Black box’ training PROS CONS C can share memory with D because C is deleted later A = Variable('A') B = Variable('B') C = B * A D = C + 1 f = compile(D) d = f(A=np.ones(10), B=np.ones(10)*2) A B 1 + x Declarative Programming ‘define then run’
- 32. DEMO: Symbol API 1– Fully Connected Neural Network (MNIST) 2 – Convolution Neural Network (MNIST)
- 33. import numpy asnp a = np.ones(10) b = np.ones(10) * 2 c = b * a d = c + 1 • Straightforward and flexible. • Take advantage of language native features (loop, condition, debugger). • E.g. Numpy, PyTorch, MXNet Gluon API • Harder to optimize PROS CONS Imperative Programming ‘define by run’
- 34. DEMO: Gluon API FullyConnected Network (MNIST)
- 35. Gluon CV: classification,detection, segmentation https://github.com/dmlc/gluon-cv [electric_guitar], with probability 0.671
- 36.
- 37.
- 38.
- 39.
- 40. DEMO: Keras-MXNet Convolutional NeuralNetwork (MNIST)
- 41. Infrastructure for DeepLearning
- 42. Amazon EC2 C5instances AVX 512 72 vCPUs “Skylake” 144 GiB memory C5 12 Gbps to EBS 2X vCPUs 2X performance 3X throughput 2.4X memory C4 36 vCPUs “Haswell” 4 Gbps to EBS 60 GiB memory C5: Nex t Ge ne rat ion Compute - Opt imize d Insta nc e s wit h Inte l® Xe on® Sca la ble Proc e ssor AWS Compute opt imize d insta nc e s support t he new Inte l® AV X - 512 a dva nc e d inst ruc t ion set , e na bling you to more effic ie ntly run ve c tor proc e ssing work loa ds wit h single a nd double f loat ing point pre c ision, suc h a s AI/ma c hine le a rning or v ide o proc e ssing. 25% improvement in price/performance over C4
- 43. Faster TensorFlow trainingon C5 https://aws.amazon.com/blogs/machine-learning/faster-training-with-optimized-tensorflow-1-6-on- amazon-ec2-c5-and-p3-instances/
- 44. Amazon EC2 P3Instances • P3.2xlarge, P3.8xlarge, P3.16xlarge • Up to eight NVIDIA Tesla V100 GPUs in a single instance • 40,960 CUDA cores, 5120 Tensor cores • 128GB of GPU memory • 1 PetaFLOPs of computational performance – 14x better than P2 • 300 GB/s GPU-to-GPU communication (NVLink) – 9x better than P2 T h e f a s t e s t , m o s t p o w e r f u l G P U i n s t a n c e s i n t h e c l o u d
- 45. Preconfigured environments toquickly build Deep Learning applications Conda AMI For developers who want pre- installed pip packages of DL frameworks in separate virtual environments. Base AMI For developers who want a clean slate to set up private DL engine repositories or custom builds of DL engines. AMI with source code For developers who want preinstalled DL frameworks and their source code in a shared Python environment. https://aws.amazon.com/machine-learning/amis/ AWS Deep Learning AMI
- 46. Amazon SageMaker Pre-built notebooks for common problems K-MeansClustering Principal Component Analysis Neural Topic Modelling Factorization Machines Linear Learner XGBoost Latent Dirichlet Allocation Image Classification Seq2Seq, And more! ALGORITHMS Apache MXNet TensorFlow Caffe2, CNTK, PyTorch, Torch FRAMEWORKS Set up and manage environments for training Train and tune model (trial and error) Deploy model in production Scale and manage the production environment Built-in, high- performance algorithms Build
- 47. Amazon SageMaker Pre-built notebooks for common problems Built-in,high- performance algorithms One-click training Hyperparameter optimization Build Train Deploy model in production Scale and manage the production environment
- 48. Amazon SageMaker Fully managedhosting with auto-scaling One-click deployment Pre-built notebooks for common problems Built-in, high- performance algorithms One-click training Hyperparameter optimization Build Train Deploy
- 49. Amazon ECR Model Training(on EC2) Model Hosting (on EC2) Trainingdata Modelartifacts Training code Helper code Helper codeInference code GroundTruth Client application Inference code Training code Inference requestInference response Inference Endpoint Amazon SageMaker
- 50. Open Source Containersfor TF and MXNet https://github.com/aws/sagemaker-tensorflow-containers https://github.com/aws/sagemaker-mxnet-containers • Build them and run them on your own machine • Run them directly on a notebook instance (aka local mode) • Customize them and push them to ECR • Run them on SageMaker for training and prediction at scale
- 51. DEMO: SageMaker 1 –Use the built-in algorithm for image classification (CIFAR-10) 2– Bring your own Tensorflow script for image classification (MNIST) 3– Bring your own Gluon script for sentiment analysis (Stanford Sentiment Tree Bank 2) 4 – Build your own Keras-MXNet container (CNN + MNIST) 5 – Build your own PyTorch container (CNN + MNIST)
- 52. Danke schön! Julien Simon PrincipalTechnical Evangelist, AI and Machine Learning @julsimon https://aws.amazon.com/machine-learning https://aws.amazon.com/blogs/ai https://mxnet.incubator.apache.org | https://github.com/apache/incubator-mxnet https://gluon.mxnet.io | https://github.com/gluon-api https://aws.amazon.com/sagemaker https://github.com/awslabs/amazon-sagemaker-examples https://github.com/aws/sagemaker-python-sdk https://github.com/aws/sagemaker-spark https://medium.com/@julsimon https://youtube.com/juliensimonfr https://gitlab.com/juliensimon/dlnotebooks