Uploaded byfawziarahim
31 views

Deep Learning for Computer Vision: Training Neural Network-Part-2

Deep Learning for Computer Vision: Training Neural Network-Part-2

Embed presentation

Download to read offline
Justin Johnson October 7, 2020 Lecture 11: Training Neural Networks (Part 2) Lecture 11 - 1
Justin Johnson October 7, 2020 Reminder: A3 Lecture 11 - 2 Due this Friday 10/9 at 11:59pm EST Reminder: 100% on autograder does not mean 100% score! Some components of each assignment will be hand-graded.
Justin Johnson October 7, 2020 Midterm Lecture 11 - 3 • Monday, October 19 • Will be online via https://crabster.org/ • Exam is 90 minutes • You can take it any time in a 24-hour window • We will have 3-4 “on-call” periods during the 24-hour window where GSIs will answer questions within ~15 minutes • Open note • True / False, multiple choice, short answer • For short answer questions requiring math, either write LaTeX or upload an image with handwritten math
Justin Johnson October 7, 2020 Overview Lecture 10 - 4 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Last Time Today
Justin Johnson October 7, 2020 Last Time: Activation Functions Lecture 11 - 5 Sigmoid tanh ReLU Leaky ReLU Maxout ELU
Justin Johnson October 7, 2020 Last Time: Data Preprocessing Lecture 11 - 6
Justin Johnson October 7, 2020 Last Time: Weight Initialization Lecture 11 - 7 “Just right”: Activations are nicely scaled for all layers! “Xavier” initialization: std = 1/sqrt(Din) Glorot and Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTAT 2010
Justin Johnson October 7, 2020 Last Time: Data Augmentation Lecture 11 - 8 Transform image Load image and label “cat” CNN Compute loss
Justin Johnson October 7, 2020 Last Time: Regularization Lecture 11 - 9 Examples: Batch Normalization Data Augmentation Training: Add randomness Testing: Marginalize out randomness Dropout Stochastic Depth Cutout Fractional pooling DropConnect Mixup
Justin Johnson October 7, 2020 Overview Lecture 10 - 10 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Today
Justin Johnson October 7, 2020 Learning Rate Schedules Lecture 11 - 11
Justin Johnson October 7, 2020 Lecture 11 - 12 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter.
Justin Johnson October 7, 2020 Lecture 11 - 13 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter. Q: Which one of these learning rates is best to use?
Justin Johnson October 7, 2020 Lecture 11 - 14 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter. Q: Which one of these learning rates is best to use? A: All of them! Start with large learning rate and decay over time
Justin Johnson October 7, 2020 Learning Rate Decay: Step Lecture 11 - 15 Reduce learning rate Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90.
Justin Johnson October 7, 2020 Learning Rate Decay: Cosine Lecture 11 - 16 Loshchilov and Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts”, ICLR 2017 Radford et al, “Improving Language Understanding by Generative Pre-Training”, 2018 Feichtenhofer et al, “SlowFast Networks for Video Recognition”, ICCV 2019 Radosavovic et al, “On Network Design Spaces for Visual Recognition”, ICCV 2019 Child at al, “Generating Long Sequences with Sparse Transformers”, arXiv 2019 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% &
Justin Johnson October 7, 2020 Learning Rate Decay: Linear Lecture 11 - 17 Devlin et al, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, NAACL 2018 Liu et al, “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019 Yang et al, “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, NeurIPS 2019 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! &
Justin Johnson October 7, 2020 Learning Rate Decay: Inverse Sqrt Lecture 11 - 18 Vaswani et al, “Attention is all you need”, NIPS 2017 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! & Inverse sqrt: 𝛼! = 𝛼$/ 𝑡
Justin Johnson October 7, 2020 Learning Rate Decay: Constant! Lecture 11 - 19 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! & Inverse sqrt: 𝛼! = 𝛼$/ 𝑡 Constant: 𝛼! = 𝛼$ Brock et al, “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, ICLR 2019 Donahue and Simonyan, “Large Scale Adversarial Representation Learning”, NeurIPS 2019
Justin Johnson October 7, 2020 How long to train? Early Stopping Lecture 11 - 20 Iteration Loss Iteration Accuracy Train Val Stop training here Stop training the model when accuracy on the validation set decreases Or train for a long time, but always keep track of the model snapshot that worked best on val. Always a good idea to do this!
Justin Johnson October 7, 2020 Lecture 11 - 21 Choosing Hyperparameters
Justin Johnson October 7, 2020 Choosing Hyperparameters: Grid Search Lecture 11 - 22 Choose several values for each hyperparameter (Often space choices log-linearly) Example: Weight decay: [1x10-4, 1x10-3, 1x10-2, 1x10-1] Learning rate: [1x10-4, 1x10-3, 1x10-2, 1x10-1] Evaluate all possible choices on this hyperparameter grid
Justin Johnson October 7, 2020 Choosing Hyperparameters: Random Search Lecture 11 - 23 Choose several values for each hyperparameter (Often space choices log-linearly) Example: Weight decay: log-uniform on [1x10-4, 1x10-1] Learning rate: log-uniform on [1x10-4, 1x10-1] Run many different trials
Justin Johnson October 7, 2020 Hyperparameters: Random vs Grid Search Lecture 11 - 24 Important Parameter Important Parameter Unimportant Parameter Unimportant Parameter Grid Layout Random Layout Bergstra and Bengio, “Random Search for Hyper-Parameter Optimization”, JMLR 2012
Justin Johnson October 7, 2020 Choosing Hyperparameters: Random Search Lecture 11 - 25 Radosavovic et al, “On Network Design Spaces for Visual Recognition”, ICCV 2019
Justin Johnson October 7, 2020 Lecture 11 - 26 Choosing Hyperparameters (without tons of GPUs)
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 27 Step 1: Check initial loss Turn off weight decay, sanity check loss at initialization e.g. log(C) for softmax with C classes
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 28 Step 1: Check initial loss Step 2: Overfit a small sample Try to train to 100% training accuracy on a small sample of training data (~5-10 minibatches); fiddle with architecture, learning rate, weight initialization. Turn off regularization. Loss not going down? LR too low, bad initialization Loss explodes to Inf or NaN? LR too high, bad initialization
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 29 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Use the architecture from the previous step, use all training data, turn on small weight decay, find a learning rate that makes the loss drop significantly within ~100 iterations Good learning rates to try: 1e-1, 1e-2, 1e-3, 1e-4
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 30 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Choose a few values of learning rate and weight decay around what worked from Step 3, train a few models for ~1-5 epochs. Good weight decay to try: 1e-4, 1e-5, 0
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 31 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Pick best models from Step 4, train them for longer (~10-20 epochs) without learning rate decay
Justin Johnson October 7, 2020 Choosing Hyperparameters Lecture 11 - 32 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Step 6: Look at learning curves
Justin Johnson October 7, 2020 Look at Learning Curves! Lecture 11 - 33 Losses may be noisy, use a scatter plot and also plot moving average to see trends better
Justin Johnson October 7, 2020 Lecture 11 - 34 Loss time Bad initialization a prime suspect
Justin Johnson October 7, 2020 Lecture 11 - 35 Loss time Loss plateaus: Try learning rate decay
Justin Johnson October 7, 2020 Lecture 11 - 36 Loss time Learning rate step decay Loss was still going down when learning rate dropped, you decayed too early!
Justin Johnson October 7, 2020 Lecture 11 - 37 Accuracy time Train Accuracy still going up, you need to train longer Val
Justin Johnson October 7, 2020 Lecture 11 - 38 Accuracy time Train Huge train / val gap means overfitting! Increase regularization, get more data Val
Justin Johnson October 7, 2020 Lecture 11 - 39 Accuracy time Train No gap between train / val means underfitting: train longer, use a bigger model Val
Justin Johnson October 7, 2020 Lecture 11 - 40 Choosing Hyperparameters Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Step 6: Look at loss curves Step 7: GOTO step 5
Justin Johnson October 7, 2020 Lecture 11 - 41 Hyperparameters to play with: - network architecture - learning rate, its decay schedule, update type - regularization (L2/Dropout strength) neural networks practitioner music = loss function This image by Paolo Guereta is licensed under CC-BY 2.0
Justin Johnson October 7, 2020 Lecture 11 - 42 Cross-validation “command center”
Justin Johnson October 7, 2020 Track ratio of weight update / weight magnitude Lecture 11 - 43 ratio between the updates and values: ~ 0.0002 / 0.02 = 0.01 (about okay) want this to be somewhere around 0.001 or so
Justin Johnson October 7, 2020 Overview Lecture 10 - 44 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training
Justin Johnson October 7, 2020 Model Ensembles Lecture 11 - 45 1. Train multiple independent models 2. At test time average their results (Take average of predicted probability distributions, then choose argmax) Enjoy 2% extra performance
Justin Johnson October 7, 2020 Cyclic learning rate schedules can make this work even better! Model Ensembles: Tips and Tricks Lecture 11 - 46 Instead of training independent models, use multiple snapshots of a single model during training! Loshchilov and Hutter, “SGDR: Stochastic gradient descent with restarts”, arXiv 2016 Huang et al, “Snapshot ensembles: train 1, get M for free”, ICLR 2017 Figures copyright Yixuan Li and Geoff Pleiss, 2017. Reproduced with permission.
Justin Johnson October 7, 2020 Model Ensembles: Tips and Tricks Lecture 11 - 47 Instead of using actual parameter vector, keep a moving average of the parameter vector and use that at test time (Polyak averaging) Polyak and Juditsky, “Acceleration of stochastic approximation by averaging”, SIAM Journal on Control and Optimization, 1992. Karras et al, “Progressive Growing of GANs for Improved Quality, Stability, and Variation”, ICLR 2018 Brock et al, “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, ICLR 2019
Justin Johnson October 7, 2020 Transfer Learning Lecture 11 - 48
Justin Johnson October 7, 2020 Transfer Learning Lecture 11 - 49 “You need a lot of a data if you want to train/use CNNs”
Justin Johnson October 7, 2020 Transfer Learning Lecture 11 - 50 “You need a lot of a data if you want to train/use CNNs” B U S T E D
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 51 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 52 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Classification on Caltech-101
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 53 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Bird Classification on Caltech-UCSD 50.98 56.78 58.75 64.96 40 45 50 55 60 65 70 DPD (Zhang et al, 2013) POOF (Berg & Belhumeur, 2013) AlexNet FC6 + logistic regression AlexNet FC6 + DPD
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 54 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Bird Classification on Caltech-UCSD 50.98 56.78 58.75 64.96 40 45 50 55 60 65 70 DPD (Zhang et al, 2013) POOF (Berg & Belhumeur, 2013) AlexNet FC6 + logistic regression AlexNet FC6 + DPD
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 55 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 2. Use CNN as a feature extractor Image Classification 71.1 64 56.8 80.7 69.9 89.5 73.9 58.4 53.3 74.7 70.8 89 77.2 69 61.8 86.8 73 91.4 50 55 60 65 70 75 80 85 90 95 Objects Scenes Birds Flowers Human Attriburtes Object Attributes Prior State of the art CNN + SVM CNN + Augmentation + SVM Razavian et al, “CNN Features Off-the-Shelf: An Astounding Baseline for Recognition”, CVPR Workshops 2014
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 56 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 2. Use CNN as a feature extractor Image Retrieval: Nearest-Neighbor Razavian et al, “CNN Features Off-the-Shelf: An Astounding Baseline for Recognition”, CVPR Workshops 2014 74.9 67.4 45.4 81.9 89.3 65.9 48.5 64.6 76.3 79.5 68 42.3 84.3 91.1 30 40 50 60 70 80 90 100 Paris Buildings Oxford Buildings Scupltures Scenes Object Instance Prior State of the art CNN + SVM CNN + Augmentation + SVM
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 57 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 3. Bigger dataset: Fine-Tuning
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 58 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 3. Bigger dataset: Fine-Tuning Some tricks: - Train with feature extraction first before fine-tuning - Lower the learning rate: use ~1/10 of LR used in original training - Sometimes freeze lower layers to save computation
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 59 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 3. Bigger dataset: Fine-Tuning Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 44.7 24.1 54.2 29.7 0 10 20 30 40 50 60 VOC 2007 ILSVRC 2013 Object Detection Feature extraction Fine Tuning
Justin Johnson October 7, 2020 Transfer Learning with CNNs: Architecture Matters! Lecture 11 - 60 Improvements in CNN architectures lead to improvements in many downstream tasks thanks to transfer learning!
Justin Johnson October 7, 2020 Transfer Learning with CNNs: Architecture Matters! Lecture 11 - 61 5 15 19 29 36 39 46 0 5 10 15 20 25 30 35 40 45 50 DPM (Pre DL) Fast R-CNN (AlexNet) Fast R-CNN (VGG-16) Faster R-CNN (VGG-16) Faster R-CNN (ResNet-50) Faster R-CNN FPN (ResNet- 101) Mask R-CNN FPN (ResNeXt- 152) Object Detection on COCO Ross Girshick, “The Generalized R-CNN Framework for Object Detection”, ICCV 2017 Tutorial on Instance-Level Visual Recognition
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 62 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) ? ? quite a lot of data (100s to 1000s) ? ?
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 63 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer ? quite a lot of data (100s to 1000s) Finetune a few layers ?
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 64 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer ? quite a lot of data (100s to 1000s) Finetune a few layers Finetune a larger number of layers
Justin Johnson October 7, 2020 Transfer Learning with CNNs Lecture 11 - 65 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer You’re in trouble… Try linear classifier from different stages quite a lot of data (100s to 1000s) Finetune a few layers Finetune a larger number of layers
Justin Johnson October 7, 2020 Lecture 11 - 66 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
Justin Johnson October 7, 2020 Lecture 11 - 67 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. CNN pretrained on ImageNet Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
Justin Johnson October 7, 2020 Lecture 11 - 68 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. CNN pretrained on ImageNet Word vectors pretrained with word2vec Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
Justin Johnson October 7, 2020 Lecture 11 - 69 Transfer learning is pervasive! It’s the norm, not the exception 1. Train CNN on ImageNet 2. Fine-Tune (1) for object detection on Visual Genome 3. Train BERT language model on lots of text 4. Combine (2) and (3), train for joint image / language modeling 5. Fine-tune (5) for image captioning, visual question answering, etc. Zhou et al, “Unified Vision-Language Pre-Training for Image Captioning and VQA”, arXiv 2019
Justin Johnson October 7, 2020 Lecture 11 - 70 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection Training from scratch can work as well as pretraining on ImageNet! … If you train for 3x as long
Justin Johnson October 7, 2020 Lecture 11 - 71 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection Pretraining + Finetuning beats training from scratch when dataset size is very small Collecting more data is more effective than pretraining 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 118K 35K 10K 1K Pretrain + Fine Tune Train From Scratch
Justin Johnson October 7, 2020 Lecture 11 - 72 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection My current view on transfer learning: - Pretrain+finetune makes your training faster, so practically very useful - Training from scratch works well once you have enough data - Lots of work left to be done 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 118K 35K 10K 1K Pretrain + Fine Tune Train From Scratch
Justin Johnson October 7, 2020 Distributed Training Lecture 11 - 73
Justin Johnson October 7, 2020 Beyond individual devices Lecture 11 - 74 Cloud TPU v2 180 TFLOPs 64 GB HBM memory $4.50 / hour (free on Colab!) Cloud TPU v2 Pod 64 TPU-v2 11.5 PFLOPs $384 / hour
Justin Johnson October 7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 75 This image is in the public domain
Justin Johnson October 7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 76 This image is in the public domain Idea #1: Run different layers on different GPUs
Justin Johnson October 7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 77 This image is in the public domain Idea #1: Run different layers on different GPUs Problem: GPUs spend lots of time waiting
Justin Johnson October 7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 78 This image is in the public domain Idea #2: Run parallel branches of model on different GPUs
Justin Johnson October 7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 79 This image is in the public domain Idea #2: Run parallel branches of model on different GPUs Problem: Synchronizing across GPUs is expensive; Need to communicate activations and grad activations
Justin Johnson October 7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 80 Batch of N images N / 2 images N / 2 images
Justin Johnson October 7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 81 Batch of N images N / 2 images N / 2 images Forward: compute loss Forward: compute loss
Justin Johnson October 7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 82 Batch of N images N / 2 images N / 2 images Forward: compute loss Backward: Compute gradient Forward: compute loss Backward: Compute gradient
Justin Johnson October 7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 83 Batch of N images N / 2 images N / 2 images Forward: compute loss Backward: Compute gradient Forward: compute loss Backward: Compute gradient Exchange gradients, sum, update GPUs only communicate once per iteration, and only exchange grad params
Justin Johnson October 7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 84 Batch of N images N / K images N / K images … Scale up to K GPUs (within and across servers) Main mechanism for distributing across many GPUs today
Justin Johnson October 7, 2020 Mixed Model + Data Parallelism Lecture 11 - 85 Batch of N images N / K images N / K images … Model parallelism within each server Data parallelism across K servers Example: https://devblogs.nvidia.com/training-bert-with-gpus/
Justin Johnson October 7, 2020 Mixed Model + Data Parallelism Lecture 11 - 86 Batch of N images N / K images N / K images … Model parallelism within each server Data parallelism across K servers Example: https://devblogs.nvidia.com/training-bert-with-gpus/ Problem: Need to train with very large minibatches!
Justin Johnson October 7, 2020 Large-Batch Training Lecture 11 - 87
Justin Johnson October 7, 2020 Large-Batch Training Lecture 11 - 88 … Suppose we can train a good model with one GPU How to scale up to data- parallel training on K GPUs?
Justin Johnson October 7, 2020 Large-Batch Training Lecture 11 - 89 … Suppose we can train a good model with one GPU How to scale up to data- parallel training on K GPUs? Goal: Train for same number of epochs, but use larger minibatches. We want model to train K times faster!
Justin Johnson October 7, 2020 Large-Batch Training: Scale Learning Rates Lecture 11 - 90 Single-GPU model: batch size N, learning rate ⍺ K-GPU model: batch size KN, learning rate K⍺ Alex Krizhevsky, “One weird trick for parallelizing convolutional neural networks”, arXiv 2014 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017
Justin Johnson October 7, 2020 Large-Batch Training: Learning Rate Warmup Lecture 11 - 91 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 High initial learning rates can make loss explode; linearly increasing learning rate from 0 over the first ~5000 iterations can prevent this
Justin Johnson October 7, 2020 Large-Batch Training: Other Concerns Lecture 11 - 92 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 Be careful with weight decay and momentum, and data shuffling For Batch Normalization, only normalize within a GPU
Justin Johnson October 7, 2020 Large-Batch Training: ImageNet in One Hour! Lecture 11 - 93 ResNet-50 with batch size 8192, distributed across 256 GPUs Trains in one hour! Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017
Justin Johnson October 7, 2020 Large-Batch Training: ImageNet in One Hour! Lecture 11 - 94 Stable training with batch size >8K needs extra trick: Layer-wise Adaptive Learning Rates (LARS) Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 You et al, “Large Batch Training of Convolutional Networks”, arXiv 2017
Justin Johnson October 7, 2020 Large-Batch Training Lecture 11 - 95 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, 2017 Batch size: 8192; 256 P100 GPUs; 1 hour Codreanu et al, “Achieving deep learning training in less than 40 minutes on imagenet-1k”, 2017 Batch size: 12288; 768 Knight’s Landing devices; 39 minutes You et al, “ImageNet training in minutes”, 2017 Batch size: 16000; 1600 Xeon CPUs; 31 minutes Akiba et al, “Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes”, 2017 Batch size: 32768; 1024 P100 GPUs; 15 minutes MLPerf v0.7 (NVIDIA): 1840 A100 GPUs; 46 seconds MLPerf v0.7 (Google): TPU-v3-8192; Batch size 65536; 27 seconds! https://mlperf.org/
Justin Johnson October 7, 2020 Recap Lecture 10 - 96 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Last Time Today
Justin Johnson October 7, 2020 Next Time: Recurrent Neural Networks Lecture 11 - 97

More Related Content

PPTX
Deeplearning
byNimrita Koul
 
PPTX
An Introduction to Deep Learning
bymilad abbasi
 
PPTX
Introduction to Deep Learning
byMehrnaz Faraz
 
PDF
chapter 11 HANDS ON MACHINE LEARNING SCIKIT
byjulianaantunes58
 
PDF
How to make your model happy again @PyData Florence @PyConIT
byAlessia Marcolini
 
PPTX
DeepLearningLecture.pptx
byssuserf07225
 
PPTX
Hyperparameter Tuning
byJon Lederman
 
PPTX
Machine Learning Week 12 SoftMax, Dropout, Optimizers and More.pptx
bylaveezawan1156
 
Deeplearning
byNimrita Koul
 
An Introduction to Deep Learning
bymilad abbasi
 
Introduction to Deep Learning
byMehrnaz Faraz
 
chapter 11 HANDS ON MACHINE LEARNING SCIKIT
byjulianaantunes58
 
How to make your model happy again @PyData Florence @PyConIT
byAlessia Marcolini
 
DeepLearningLecture.pptx
byssuserf07225
 
Hyperparameter Tuning
byJon Lederman
 
Machine Learning Week 12 SoftMax, Dropout, Optimizers and More.pptx
bylaveezawan1156
 

Similar to Deep Learning for Computer Vision: Training Neural Network-Part-2

PPTX
effect of learning rate
byrajshreemuthiah
 
PPTX
Deep learning crash course
byVishwas N
 
PPTX
3. Training Artificial Neural Networks.pptx
bymunwar7
 
PDF
Hands on machine learning with scikit-learn and tensor flow by ahmed yousry
byAhmed Yousry
 
PDF
Tuning learning rate
byJamie (Taka) Wang
 
PPTX
1. Introduction to deep learning.pptx
byKv Sagar
 
PDF
Training Deep Networks with Backprop (D1L4 Insight@DCU Machine Learning Works...
byUniversitat Politècnica de Catalunya
 
PPTX
NITW_Improving Deep Neural Networks (1).pptx
byDrKBManwade
 
PPTX
NITW_Improving Deep Neural Networks.pptx
byssuserd23711
 
PPTX
Introduction to Deep learning and H2O for beginner's
byVidyasagar Bhargava
 
PDF
Separating Hype from Reality in Deep Learning with Sameer Farooqui
byDatabricks
 
PPTX
Leslie Smith's Papers discussion for DL Journal Club
bySujit Pal
 
PDF
Lecture 5: Neural Networks II
bySang Jun Lee
 
PPTX
Nimrita deep learning
byNimrita Koul
 
PPTX
Batch normalization presentation
byOwin Will
 
PPTX
Deep Learning in Recommender Systems - RecSys Summer School 2017
byBalázs Hidasi
 
PDF
Methodology (DLAI D6L2 2017 UPC Deep Learning for Artificial Intelligence)
byUniversitat Politècnica de Catalunya
 
PDF
Module-3_Deep Learning_22ISE74A_ISE_.pdf
byDr. Shivashankar
 
PPTX
Neural network basic and introduction of Deep learning
byTapas Majumdar
 
PPTX
lecture 9 pdddddddddddddddddssdsdnn.pptx
byspeedcomcyber25
 
effect of learning rate
byrajshreemuthiah
 
Deep learning crash course
byVishwas N
 
3. Training Artificial Neural Networks.pptx
bymunwar7
 
Hands on machine learning with scikit-learn and tensor flow by ahmed yousry
byAhmed Yousry
 
Tuning learning rate
byJamie (Taka) Wang
 
1. Introduction to deep learning.pptx
byKv Sagar
 
Training Deep Networks with Backprop (D1L4 Insight@DCU Machine Learning Works...
byUniversitat Politècnica de Catalunya
 
NITW_Improving Deep Neural Networks (1).pptx
byDrKBManwade
 
NITW_Improving Deep Neural Networks.pptx
byssuserd23711
 
Introduction to Deep learning and H2O for beginner's
byVidyasagar Bhargava
 
Separating Hype from Reality in Deep Learning with Sameer Farooqui
byDatabricks
 
Leslie Smith's Papers discussion for DL Journal Club
bySujit Pal
 
Lecture 5: Neural Networks II
bySang Jun Lee
 
Nimrita deep learning
byNimrita Koul
 
Batch normalization presentation
byOwin Will
 
Deep Learning in Recommender Systems - RecSys Summer School 2017
byBalázs Hidasi
 
Methodology (DLAI D6L2 2017 UPC Deep Learning for Artificial Intelligence)
byUniversitat Politècnica de Catalunya
 
Module-3_Deep Learning_22ISE74A_ISE_.pdf
byDr. Shivashankar
 
Neural network basic and introduction of Deep learning
byTapas Majumdar
 
lecture 9 pdddddddddddddddddssdsdnn.pptx
byspeedcomcyber25
 

Recently uploaded

PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PPTX
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
PDF
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
PDF
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
PDF
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PPTX
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
PDF
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
PDF
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
PDF
Security-Fails-in-the-Gaps-Between-Systems.pptx.pdf
byOhhproJunction
 
PDF
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PDF
How to Install XRDP on CentOS 10 .pdf
byGreen Webpage
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
Working Session — Build a Document Understanding Automation Using an OOTB ML ...
byDianaGray10
 
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
Security-Fails-in-the-Gaps-Between-Systems.pptx.pdf
byOhhproJunction
 
Session 1/5: Enhancing Automation with Screenplay & Business Rules
bysuhanisingh58689
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
How to Install XRDP on CentOS 10 .pdf
byGreen Webpage
 

Deep Learning for Computer Vision: Training Neural Network-Part-2

  • 1.
    Justin Johnson October7, 2020 Lecture 11: Training Neural Networks (Part 2) Lecture 11 - 1
  • 2.
    Justin Johnson October7, 2020 Reminder: A3 Lecture 11 - 2 Due this Friday 10/9 at 11:59pm EST Reminder: 100% on autograder does not mean 100% score! Some components of each assignment will be hand-graded.
  • 3.
    Justin Johnson October7, 2020 Midterm Lecture 11 - 3 • Monday, October 19 • Will be online via https://crabster.org/ • Exam is 90 minutes • You can take it any time in a 24-hour window • We will have 3-4 “on-call” periods during the 24-hour window where GSIs will answer questions within ~15 minutes • Open note • True / False, multiple choice, short answer • For short answer questions requiring math, either write LaTeX or upload an image with handwritten math
  • 4.
    Justin Johnson October7, 2020 Overview Lecture 10 - 4 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Last Time Today
  • 5.
    Justin Johnson October7, 2020 Last Time: Activation Functions Lecture 11 - 5 Sigmoid tanh ReLU Leaky ReLU Maxout ELU
  • 6.
    Justin Johnson October7, 2020 Last Time: Data Preprocessing Lecture 11 - 6
  • 7.
    Justin Johnson October7, 2020 Last Time: Weight Initialization Lecture 11 - 7 “Just right”: Activations are nicely scaled for all layers! “Xavier” initialization: std = 1/sqrt(Din) Glorot and Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTAT 2010
  • 8.
    Justin Johnson October7, 2020 Last Time: Data Augmentation Lecture 11 - 8 Transform image Load image and label “cat” CNN Compute loss
  • 9.
    Justin Johnson October7, 2020 Last Time: Regularization Lecture 11 - 9 Examples: Batch Normalization Data Augmentation Training: Add randomness Testing: Marginalize out randomness Dropout Stochastic Depth Cutout Fractional pooling DropConnect Mixup
  • 10.
    Justin Johnson October7, 2020 Overview Lecture 10 - 10 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Today
  • 11.
    Justin Johnson October7, 2020 Learning Rate Schedules Lecture 11 - 11
  • 12.
    Justin Johnson October7, 2020 Lecture 11 - 12 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter.
  • 13.
    Justin Johnson October7, 2020 Lecture 11 - 13 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter. Q: Which one of these learning rates is best to use?
  • 14.
    Justin Johnson October7, 2020 Lecture 11 - 14 SGD, SGD+Momentum, Adagrad, RMSProp, Adam all have learning rate as a hyperparameter. Q: Which one of these learning rates is best to use? A: All of them! Start with large learning rate and decay over time
  • 15.
    Justin Johnson October7, 2020 Learning Rate Decay: Step Lecture 11 - 15 Reduce learning rate Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90.
  • 16.
    Justin Johnson October7, 2020 Learning Rate Decay: Cosine Lecture 11 - 16 Loshchilov and Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts”, ICLR 2017 Radford et al, “Improving Language Understanding by Generative Pre-Training”, 2018 Feichtenhofer et al, “SlowFast Networks for Video Recognition”, ICCV 2019 Radosavovic et al, “On Network Design Spaces for Visual Recognition”, ICCV 2019 Child at al, “Generating Long Sequences with Sparse Transformers”, arXiv 2019 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% &
  • 17.
    Justin Johnson October7, 2020 Learning Rate Decay: Linear Lecture 11 - 17 Devlin et al, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, NAACL 2018 Liu et al, “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019 Yang et al, “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, NeurIPS 2019 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! &
  • 18.
    Justin Johnson October7, 2020 Learning Rate Decay: Inverse Sqrt Lecture 11 - 18 Vaswani et al, “Attention is all you need”, NIPS 2017 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! & Inverse sqrt: 𝛼! = 𝛼$/ 𝑡
  • 19.
    Justin Johnson October7, 2020 Learning Rate Decay: Constant! Lecture 11 - 19 Step: Reduce learning rate at a few fixed points. E.g. for ResNets, multiply LR by 0.1 after epochs 30, 60, and 90. Cosine: 𝛼! = " # 𝛼$ 1 + cos !% & Linear: 𝛼! = 𝛼$ 1 − ! & Inverse sqrt: 𝛼! = 𝛼$/ 𝑡 Constant: 𝛼! = 𝛼$ Brock et al, “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, ICLR 2019 Donahue and Simonyan, “Large Scale Adversarial Representation Learning”, NeurIPS 2019
  • 20.
    Justin Johnson October7, 2020 How long to train? Early Stopping Lecture 11 - 20 Iteration Loss Iteration Accuracy Train Val Stop training here Stop training the model when accuracy on the validation set decreases Or train for a long time, but always keep track of the model snapshot that worked best on val. Always a good idea to do this!
  • 21.
    Justin Johnson October7, 2020 Lecture 11 - 21 Choosing Hyperparameters
  • 22.
    Justin Johnson October7, 2020 Choosing Hyperparameters: Grid Search Lecture 11 - 22 Choose several values for each hyperparameter (Often space choices log-linearly) Example: Weight decay: [1x10-4, 1x10-3, 1x10-2, 1x10-1] Learning rate: [1x10-4, 1x10-3, 1x10-2, 1x10-1] Evaluate all possible choices on this hyperparameter grid
  • 23.
    Justin Johnson October7, 2020 Choosing Hyperparameters: Random Search Lecture 11 - 23 Choose several values for each hyperparameter (Often space choices log-linearly) Example: Weight decay: log-uniform on [1x10-4, 1x10-1] Learning rate: log-uniform on [1x10-4, 1x10-1] Run many different trials
  • 24.
    Justin Johnson October7, 2020 Hyperparameters: Random vs Grid Search Lecture 11 - 24 Important Parameter Important Parameter Unimportant Parameter Unimportant Parameter Grid Layout Random Layout Bergstra and Bengio, “Random Search for Hyper-Parameter Optimization”, JMLR 2012
  • 25.
    Justin Johnson October7, 2020 Choosing Hyperparameters: Random Search Lecture 11 - 25 Radosavovic et al, “On Network Design Spaces for Visual Recognition”, ICCV 2019
  • 26.
    Justin Johnson October7, 2020 Lecture 11 - 26 Choosing Hyperparameters (without tons of GPUs)
  • 27.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 27 Step 1: Check initial loss Turn off weight decay, sanity check loss at initialization e.g. log(C) for softmax with C classes
  • 28.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 28 Step 1: Check initial loss Step 2: Overfit a small sample Try to train to 100% training accuracy on a small sample of training data (~5-10 minibatches); fiddle with architecture, learning rate, weight initialization. Turn off regularization. Loss not going down? LR too low, bad initialization Loss explodes to Inf or NaN? LR too high, bad initialization
  • 29.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 29 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Use the architecture from the previous step, use all training data, turn on small weight decay, find a learning rate that makes the loss drop significantly within ~100 iterations Good learning rates to try: 1e-1, 1e-2, 1e-3, 1e-4
  • 30.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 30 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Choose a few values of learning rate and weight decay around what worked from Step 3, train a few models for ~1-5 epochs. Good weight decay to try: 1e-4, 1e-5, 0
  • 31.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 31 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Pick best models from Step 4, train them for longer (~10-20 epochs) without learning rate decay
  • 32.
    Justin Johnson October7, 2020 Choosing Hyperparameters Lecture 11 - 32 Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Step 6: Look at learning curves
  • 33.
    Justin Johnson October7, 2020 Look at Learning Curves! Lecture 11 - 33 Losses may be noisy, use a scatter plot and also plot moving average to see trends better
  • 34.
    Justin Johnson October7, 2020 Lecture 11 - 34 Loss time Bad initialization a prime suspect
  • 35.
    Justin Johnson October7, 2020 Lecture 11 - 35 Loss time Loss plateaus: Try learning rate decay
  • 36.
    Justin Johnson October7, 2020 Lecture 11 - 36 Loss time Learning rate step decay Loss was still going down when learning rate dropped, you decayed too early!
  • 37.
    Justin Johnson October7, 2020 Lecture 11 - 37 Accuracy time Train Accuracy still going up, you need to train longer Val
  • 38.
    Justin Johnson October7, 2020 Lecture 11 - 38 Accuracy time Train Huge train / val gap means overfitting! Increase regularization, get more data Val
  • 39.
    Justin Johnson October7, 2020 Lecture 11 - 39 Accuracy time Train No gap between train / val means underfitting: train longer, use a bigger model Val
  • 40.
    Justin Johnson October7, 2020 Lecture 11 - 40 Choosing Hyperparameters Step 1: Check initial loss Step 2: Overfit a small sample Step 3: Find LR that makes loss go down Step 4: Coarse grid, train for ~1-5 epochs Step 5: Refine grid, train longer Step 6: Look at loss curves Step 7: GOTO step 5
  • 41.
    Justin Johnson October7, 2020 Lecture 11 - 41 Hyperparameters to play with: - network architecture - learning rate, its decay schedule, update type - regularization (L2/Dropout strength) neural networks practitioner music = loss function This image by Paolo Guereta is licensed under CC-BY 2.0
  • 42.
    Justin Johnson October7, 2020 Lecture 11 - 42 Cross-validation “command center”
  • 43.
    Justin Johnson October7, 2020 Track ratio of weight update / weight magnitude Lecture 11 - 43 ratio between the updates and values: ~ 0.0002 / 0.02 = 0.01 (about okay) want this to be somewhere around 0.001 or so
  • 44.
    Justin Johnson October7, 2020 Overview Lecture 10 - 44 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training
  • 45.
    Justin Johnson October7, 2020 Model Ensembles Lecture 11 - 45 1. Train multiple independent models 2. At test time average their results (Take average of predicted probability distributions, then choose argmax) Enjoy 2% extra performance
  • 46.
    Justin Johnson October7, 2020 Cyclic learning rate schedules can make this work even better! Model Ensembles: Tips and Tricks Lecture 11 - 46 Instead of training independent models, use multiple snapshots of a single model during training! Loshchilov and Hutter, “SGDR: Stochastic gradient descent with restarts”, arXiv 2016 Huang et al, “Snapshot ensembles: train 1, get M for free”, ICLR 2017 Figures copyright Yixuan Li and Geoff Pleiss, 2017. Reproduced with permission.
  • 47.
    Justin Johnson October7, 2020 Model Ensembles: Tips and Tricks Lecture 11 - 47 Instead of using actual parameter vector, keep a moving average of the parameter vector and use that at test time (Polyak averaging) Polyak and Juditsky, “Acceleration of stochastic approximation by averaging”, SIAM Journal on Control and Optimization, 1992. Karras et al, “Progressive Growing of GANs for Improved Quality, Stability, and Variation”, ICLR 2018 Brock et al, “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, ICLR 2019
  • 48.
    Justin Johnson October7, 2020 Transfer Learning Lecture 11 - 48
  • 49.
    Justin Johnson October7, 2020 Transfer Learning Lecture 11 - 49 “You need a lot of a data if you want to train/use CNNs”
  • 50.
    Justin Johnson October7, 2020 Transfer Learning Lecture 11 - 50 “You need a lot of a data if you want to train/use CNNs” B U S T E D
  • 51.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 51 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor
  • 52.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 52 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Classification on Caltech-101
  • 53.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 53 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Bird Classification on Caltech-UCSD 50.98 56.78 58.75 64.96 40 45 50 55 60 65 70 DPD (Zhang et al, 2013) POOF (Berg & Belhumeur, 2013) AlexNet FC6 + logistic regression AlexNet FC6 + DPD
  • 54.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 54 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer Donahue et al, “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition”, ICML 2014 2. Use CNN as a feature extractor Bird Classification on Caltech-UCSD 50.98 56.78 58.75 64.96 40 45 50 55 60 65 70 DPD (Zhang et al, 2013) POOF (Berg & Belhumeur, 2013) AlexNet FC6 + logistic regression AlexNet FC6 + DPD
  • 55.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 55 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 2. Use CNN as a feature extractor Image Classification 71.1 64 56.8 80.7 69.9 89.5 73.9 58.4 53.3 74.7 70.8 89 77.2 69 61.8 86.8 73 91.4 50 55 60 65 70 75 80 85 90 95 Objects Scenes Birds Flowers Human Attriburtes Object Attributes Prior State of the art CNN + SVM CNN + Augmentation + SVM Razavian et al, “CNN Features Off-the-Shelf: An Astounding Baseline for Recognition”, CVPR Workshops 2014
  • 56.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 56 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 2. Use CNN as a feature extractor Image Retrieval: Nearest-Neighbor Razavian et al, “CNN Features Off-the-Shelf: An Astounding Baseline for Recognition”, CVPR Workshops 2014 74.9 67.4 45.4 81.9 89.3 65.9 48.5 64.6 76.3 79.5 68 42.3 84.3 91.1 30 40 50 60 70 80 90 100 Paris Buildings Oxford Buildings Scupltures Scenes Object Instance Prior State of the art CNN + SVM CNN + Augmentation + SVM
  • 57.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 57 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 3. Bigger dataset: Fine-Tuning
  • 58.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 58 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 3. Bigger dataset: Fine-Tuning Some tricks: - Train with feature extraction first before fine-tuning - Lower the learning rate: use ~1/10 of LR used in original training - Sometimes freeze lower layers to save computation
  • 59.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 59 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 1. Train on Imagenet Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 3. Bigger dataset: Fine-Tuning Continue training CNN for new task! Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 Freeze these Remove last layer 2. Use CNN as a feature extractor 44.7 24.1 54.2 29.7 0 10 20 30 40 50 60 VOC 2007 ILSVRC 2013 Object Detection Feature extraction Fine Tuning
  • 60.
    Justin Johnson October7, 2020 Transfer Learning with CNNs: Architecture Matters! Lecture 11 - 60 Improvements in CNN architectures lead to improvements in many downstream tasks thanks to transfer learning!
  • 61.
    Justin Johnson October7, 2020 Transfer Learning with CNNs: Architecture Matters! Lecture 11 - 61 5 15 19 29 36 39 46 0 5 10 15 20 25 30 35 40 45 50 DPM (Pre DL) Fast R-CNN (AlexNet) Fast R-CNN (VGG-16) Faster R-CNN (VGG-16) Faster R-CNN (ResNet-50) Faster R-CNN FPN (ResNet- 101) Mask R-CNN FPN (ResNeXt- 152) Object Detection on COCO Ross Girshick, “The Generalized R-CNN Framework for Object Detection”, ICCV 2017 Tutorial on Instance-Level Visual Recognition
  • 62.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 62 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) ? ? quite a lot of data (100s to 1000s) ? ?
  • 63.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 63 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer ? quite a lot of data (100s to 1000s) Finetune a few layers ?
  • 64.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 64 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer ? quite a lot of data (100s to 1000s) Finetune a few layers Finetune a larger number of layers
  • 65.
    Justin Johnson October7, 2020 Transfer Learning with CNNs Lecture 11 - 65 Image Conv-64 Conv-64 MaxPool Conv-128 Conv-128 MaxPool Conv-256 Conv-256 MaxPool Conv-512 Conv-512 MaxPool Conv-512 Conv-512 MaxPool FC-4096 FC-4096 FC-1000 More generic More specific Dataset similar to ImageNet Dataset very different from ImageNet very little data (10s to 100s) Use Linear Classifier on top layer You’re in trouble… Try linear classifier from different stages quite a lot of data (100s to 1000s) Finetune a few layers Finetune a larger number of layers
  • 66.
    Justin Johnson October7, 2020 Lecture 11 - 66 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
  • 67.
    Justin Johnson October7, 2020 Lecture 11 - 67 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. CNN pretrained on ImageNet Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
  • 68.
    Justin Johnson October7, 2020 Lecture 11 - 68 Girshick, “Fast R-CNN”, ICCV 2015 Figure copyright Ross Girshick, 2015. Reproduced with permission. CNN pretrained on ImageNet Word vectors pretrained with word2vec Karpathy and Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 2015 Object Detection (Fast R-CNN) Transfer learning is pervasive! It’s the norm, not the exception
  • 69.
    Justin Johnson October7, 2020 Lecture 11 - 69 Transfer learning is pervasive! It’s the norm, not the exception 1. Train CNN on ImageNet 2. Fine-Tune (1) for object detection on Visual Genome 3. Train BERT language model on lots of text 4. Combine (2) and (3), train for joint image / language modeling 5. Fine-tune (5) for image captioning, visual question answering, etc. Zhou et al, “Unified Vision-Language Pre-Training for Image Captioning and VQA”, arXiv 2019
  • 70.
    Justin Johnson October7, 2020 Lecture 11 - 70 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection Training from scratch can work as well as pretraining on ImageNet! … If you train for 3x as long
  • 71.
    Justin Johnson October7, 2020 Lecture 11 - 71 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection Pretraining + Finetuning beats training from scratch when dataset size is very small Collecting more data is more effective than pretraining 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 118K 35K 10K 1K Pretrain + Fine Tune Train From Scratch
  • 72.
    Justin Johnson October7, 2020 Lecture 11 - 72 Transfer learning is pervasive! Some very recent results have questioned it He et al, ”Rethinking ImageNet Pre-Training”, ICCV 2019 COCO object detection My current view on transfer learning: - Pretrain+finetune makes your training faster, so practically very useful - Training from scratch works well once you have enough data - Lots of work left to be done 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 118K 35K 10K 1K Pretrain + Fine Tune Train From Scratch
  • 73.
    Justin Johnson October7, 2020 Distributed Training Lecture 11 - 73
  • 74.
    Justin Johnson October7, 2020 Beyond individual devices Lecture 11 - 74 Cloud TPU v2 180 TFLOPs 64 GB HBM memory $4.50 / hour (free on Colab!) Cloud TPU v2 Pod 64 TPU-v2 11.5 PFLOPs $384 / hour
  • 75.
    Justin Johnson October7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 75 This image is in the public domain
  • 76.
    Justin Johnson October7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 76 This image is in the public domain Idea #1: Run different layers on different GPUs
  • 77.
    Justin Johnson October7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 77 This image is in the public domain Idea #1: Run different layers on different GPUs Problem: GPUs spend lots of time waiting
  • 78.
    Justin Johnson October7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 78 This image is in the public domain Idea #2: Run parallel branches of model on different GPUs
  • 79.
    Justin Johnson October7, 2020 Model Parallelism: Split Model Across GPUs Lecture 11 - 79 This image is in the public domain Idea #2: Run parallel branches of model on different GPUs Problem: Synchronizing across GPUs is expensive; Need to communicate activations and grad activations
  • 80.
    Justin Johnson October7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 80 Batch of N images N / 2 images N / 2 images
  • 81.
    Justin Johnson October7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 81 Batch of N images N / 2 images N / 2 images Forward: compute loss Forward: compute loss
  • 82.
    Justin Johnson October7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 82 Batch of N images N / 2 images N / 2 images Forward: compute loss Backward: Compute gradient Forward: compute loss Backward: Compute gradient
  • 83.
    Justin Johnson October7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 83 Batch of N images N / 2 images N / 2 images Forward: compute loss Backward: Compute gradient Forward: compute loss Backward: Compute gradient Exchange gradients, sum, update GPUs only communicate once per iteration, and only exchange grad params
  • 84.
    Justin Johnson October7, 2020 Data Parallelism: Copy Model on each GPU, split data Lecture 11 - 84 Batch of N images N / K images N / K images … Scale up to K GPUs (within and across servers) Main mechanism for distributing across many GPUs today
  • 85.
    Justin Johnson October7, 2020 Mixed Model + Data Parallelism Lecture 11 - 85 Batch of N images N / K images N / K images … Model parallelism within each server Data parallelism across K servers Example: https://devblogs.nvidia.com/training-bert-with-gpus/
  • 86.
    Justin Johnson October7, 2020 Mixed Model + Data Parallelism Lecture 11 - 86 Batch of N images N / K images N / K images … Model parallelism within each server Data parallelism across K servers Example: https://devblogs.nvidia.com/training-bert-with-gpus/ Problem: Need to train with very large minibatches!
  • 87.
    Justin Johnson October7, 2020 Large-Batch Training Lecture 11 - 87
  • 88.
    Justin Johnson October7, 2020 Large-Batch Training Lecture 11 - 88 … Suppose we can train a good model with one GPU How to scale up to data- parallel training on K GPUs?
  • 89.
    Justin Johnson October7, 2020 Large-Batch Training Lecture 11 - 89 … Suppose we can train a good model with one GPU How to scale up to data- parallel training on K GPUs? Goal: Train for same number of epochs, but use larger minibatches. We want model to train K times faster!
  • 90.
    Justin Johnson October7, 2020 Large-Batch Training: Scale Learning Rates Lecture 11 - 90 Single-GPU model: batch size N, learning rate ⍺ K-GPU model: batch size KN, learning rate K⍺ Alex Krizhevsky, “One weird trick for parallelizing convolutional neural networks”, arXiv 2014 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017
  • 91.
    Justin Johnson October7, 2020 Large-Batch Training: Learning Rate Warmup Lecture 11 - 91 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 High initial learning rates can make loss explode; linearly increasing learning rate from 0 over the first ~5000 iterations can prevent this
  • 92.
    Justin Johnson October7, 2020 Large-Batch Training: Other Concerns Lecture 11 - 92 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 Be careful with weight decay and momentum, and data shuffling For Batch Normalization, only normalize within a GPU
  • 93.
    Justin Johnson October7, 2020 Large-Batch Training: ImageNet in One Hour! Lecture 11 - 93 ResNet-50 with batch size 8192, distributed across 256 GPUs Trains in one hour! Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017
  • 94.
    Justin Johnson October7, 2020 Large-Batch Training: ImageNet in One Hour! Lecture 11 - 94 Stable training with batch size >8K needs extra trick: Layer-wise Adaptive Learning Rates (LARS) Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, arXiv 2017 You et al, “Large Batch Training of Convolutional Networks”, arXiv 2017
  • 95.
    Justin Johnson October7, 2020 Large-Batch Training Lecture 11 - 95 Goyal et al, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, 2017 Batch size: 8192; 256 P100 GPUs; 1 hour Codreanu et al, “Achieving deep learning training in less than 40 minutes on imagenet-1k”, 2017 Batch size: 12288; 768 Knight’s Landing devices; 39 minutes You et al, “ImageNet training in minutes”, 2017 Batch size: 16000; 1600 Xeon CPUs; 31 minutes Akiba et al, “Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes”, 2017 Batch size: 32768; 1024 P100 GPUs; 15 minutes MLPerf v0.7 (NVIDIA): 1840 A100 GPUs; 46 seconds MLPerf v0.7 (Google): TPU-v3-8192; Batch size 65536; 27 seconds! https://mlperf.org/
  • 96.
    Justin Johnson October7, 2020 Recap Lecture 10 - 96 1.One time setup Activation functions, data preprocessing, weight initialization, regularization 2.Training dynamics Learning rate schedules; hyperparameter optimization 3.After training Model ensembles, transfer learning, large-batch training Last Time Today
  • 97.
    Justin Johnson October7, 2020 Next Time: Recurrent Neural Networks Lecture 11 - 97