Peerasak C., profile picture
Uploaded byPeerasak C.
PDF, PPTX2,023 views

Deep Learning State of the Art (2019) - MIT by Lex Fridman

The document provides an overview of breakthrough developments in deep learning from 2017 and 2018, highlighting key technologies like BERT for natural language processing, Tesla's autopilot advancements, and techniques in data augmentation such as AutoAugment. It also discusses innovations in training deep networks, GANs for image synthesis, and various frameworks for speeding up training processes. It emphasizes that these advancements represent exciting developments rather than traditional state-of-the-art benchmarks.

Embed presentation

Download as PDF, PPTX
Deep Learning: State of the Art (2019)
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Encoder-Decoder Architecture Sequence-to-Sequence Model - Neural Machine Translation [295] Encoder RNN encodes input sequence into a fixed size vector, and then is passed repeatedly to decoder RNN.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Attention [296] Attention mechanism allows the network to refer back to the input sequence, instead of forcing it to encode all information into one fixed-length vector.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Attention [312]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Attention [312]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Self-Attention [312] "The animal didn't cross the street because it was too tired" More details: http://jalammar.github.io/illustrated-transformer/
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Transformer [312] Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references Word Embeddings (Word2Vec) [297] Word Vector Skip Gram Model:
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Context-Aware Embeddings [312]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references OpenAI Transformer 1. Pre-train a Transformer’s decoder for language modeling 2. Train it on, for example, a sentence classification task [312] 1 2
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references BERT Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references BERT Applications Now you can use BERT: • Create contextualized word embeddings (like ELMo) • Sentence classification • Sentence pair classification • Sentence pair similarity • Multiple choice • Sentence tagging • Question answering
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Tesla Autopilot Hardware v2+ • Specialized NVIDIA Drive PX 2 hardware • Neural network takes all 8 cameras as input • Based on Inception v1 architecture
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Autopilot Reaches 1 Billion Miles (~0.5 Billion on Hardware v2+)
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references AutoML and Neural Architecture Search (NASNet) [300, 301]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references AdaNet [317]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references AutoAugment: RL for Data Augmentation [318] • Show that transfer learning can also be done augmentation policies instead of weights (or with).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Training on Randomized Synthetic Data [319] • number and types of objects • number, types, colors, and scales of distractors • texture on the object of interest, and background photograph • location of the virtual camera with respect to the scene • angle of the camera with respect to the scene • number and locations of point lights Tremblay, Jonathan, et al. "Training deep networks with synthetic data: Bridging the reality gap by domain randomization." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Segmentation Annotation with Polygon-RNN++ • High-resolution polygon segmentation generation with human-in-the-loop [319] Acuna, David, et al. "Efficient Interactive Annotation of Segmentation Datasets With Polygon-RNN++." CVPR 2018.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references DAWNBench: Training Fast and Cheap DAWNBench: • Competition on speed and cost of training and inference that achieves 93% for ImageNet and 94% for CIFAR 10 fast.ai – Training: • ImageNet in 3 hours for $25 • CIFAR10 for $0.26 • Key idea: During training, if you very slowly increase learning rate while decreasing momentum, you can train at extremely high learning rates, thus avoiding over-fitting, and training in far fewer epochs. • Details: http://bit.ly/2H6yv6H [320]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references BigGAN: State of the Art in Image Synthesis • Same GAN techniques, much larger scale • Increase model capacity + increase batch size [321] Brock, Andrew, Jeff Donahue, and Karen Simonyan. "Large scale gan training for high fidelity natural image synthesis." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.edu • AlexNet (2012): First CNN (15.4%) • 8 layers • 61 million parameters • ZFNet (2013): 15.4% to 11.2% • 8 layers • More filters. Denser stride. • VGGNet (2014): 11.2% to 7.3% • Beautifully uniform: 3x3 conv, stride 1, pad 1, 2x2 max pool • 16 layers • 138 million parameters • GoogLeNet (2014): 11.2% to 6.7% • Inception modules • 22 layers • 5 million parameters (throw away fully connected layers) • ResNet (2015): 6.7% to 3.57% • More layers = better performance • 152 layers • CUImage (2016): 3.57% to 2.99% • Ensemble of 6 models • SENet (2017): 2.99% to 2.251% • Squeeze and excitation block: network is allowed to adaptively adjust the weighting of each feature map in the convolutional block. References: [90] Human error (5.1%) surpassed in 2015
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Object Detection / Localization Region-Based Methods | Shown: Faster R-CNN [299]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Object Detection / Localization Single-Shot Methods | Shown: SSD [299]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Object Detection: State of the Art Progress [326]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Semantic Segmentation [175]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references State-of-the-Art: DeepLab v3 • PASCAL VOC Challenge: http://bit.ly/2HdzTEu
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references FCN (Nov 2014) Paper: “Fully Convolutional Networks for Semantic Segmentation” • Repurpose Imagenet pretrained nets • Upsample using deconvolution • Skip connections to improve coarseness of upsampling [173]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references SegNet (Nov 2015) Paper: “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation” • Maxpooling indices transferred to decoder to improve the segmentation resolution. [175]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references Dilated Convolutions (Nov 2015) Paper: “Multi-Scale Context Aggregation by Dilated Convolutions” • Since pooling decreases resolution: • Added “dilated convolution layer” • Still interpolate up from 1/8 of original image size [175]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v1, v2 (Jun 2016) Paper: “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs” • Added fully-connected Conditional Random Fields (CRFs) – as a post-processing step • Smooth segmentation based on the underlying image intensities [179]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references Key Aspects of Segmentation • Fully convolutional networks (FCNs) - replace fully-connected layers with convolutional layers. Classification network is where the biggest gains come from. • Deeper, updated models (ResNet, etc) consistent with ImageNet Challenge object classification tasks. • Conditional Random Fields (CRFs) to capture both local and long-range dependencies within an image to refine the prediction map. • Dilated convolution (aka Atrous convolution) – maintain computational cost, increase resolution of intermediate feature maps • Process at multiple scales and combine the information together [174, 176]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v3 Paper: “Rethinking Atrous Convolution for Semantic Image Segmentation” • Multi-scale processing, without increasing parameters. • Increasing “atrous rate” enlarges the model’s “field-of-view” [179]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v3 trained on CityScapes Tutorial: https://github.com/lexfridman/mit-deep-learning
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references Atari Breakout [85] After 120 Minutes of Training After 10 Minutes of Training After 240 Minutes of Training
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references DQN: Atari Games (2015) [83]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references AlphaGo (2016): Beat Top Human at Go [83]
2019https://deeplearning.mit.eduFor the full updated list of references visit: https://selfdrivingcars.mit.edu/references AlphaGo Zero (2017): Beats AlphaGo [149]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references [324] AlphaZero (Dec 2017) vs StockFish (Chess) & Elmo (Shogi)
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references OpenAI & Dota 2 • Dota 2 as a testbed for the messiness and continuous nature of the real world: teamwork, long time horizons, and hidden information. [325]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references OpenAI & Dota 2 Progress • Aug, 2017: 1v1 bot beats top professional Dota 2 players. • Aug, 2018: OpenAI Five lost two games against top Dota 2 players at The International. “We are looking forward to pushing Five to the next level.” [325]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning Frameworks [327] Deep Learning Framework Power Scores (by Jeff Hale) http://bit.ly/2GBa3tU 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Factors to consider: • Learning curve • Speed of development • Size and passion of community • Number of papers implemented in framework • Likelihood of long-term growth and stability • Ecosystem of tooling
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Deep Learning: 2019 and Beyond • On backpropagation: “My view is throw it all away and start again.” • “The future depends on some graduate student who is deeply suspicious of everything I have said.” - Geoffrey Hinton “Godfather of Deep Learning” [328]
2019https://deeplearning.mit.eduFor the full list of references visit: https://hcai.mit.edu/references Thank You Website: deeplearning.mit.edu • Videos and slides posted on the website • Code posted on GitHub: https://github.com/lexfridman/mit-deep-learning

More Related Content

PPTX
Realtime vs Cloud Firestore
byAppinventiv
 
PDF
Project Tungsten: Bringing Spark Closer to Bare Metal
byDatabricks
 
PDF
Apache Airflow入門 (マーケティングデータ分析基盤技術勉強会)
byTakeshi Mikami
 
PDF
Improving PySpark performance: Spark Performance Beyond the JVM
byHolden Karau
 
PDF
Ingress on Azure Kubernetes Service
byToru Makabe
 
PDF
[DL Hacks]FPGA入門
byDeep Learning JP
 
PDF
Sequence Modelling with Deep Learning
byNatasha Latysheva
 
PDF
Spark 의 핵심은 무엇인가? RDD! (RDD paper review)
byYongho Ha
 
Realtime vs Cloud Firestore
byAppinventiv
 
Project Tungsten: Bringing Spark Closer to Bare Metal
byDatabricks
 
Apache Airflow入門 (マーケティングデータ分析基盤技術勉強会)
byTakeshi Mikami
 
Improving PySpark performance: Spark Performance Beyond the JVM
byHolden Karau
 
Ingress on Azure Kubernetes Service
byToru Makabe
 
[DL Hacks]FPGA入門
byDeep Learning JP
 
Sequence Modelling with Deep Learning
byNatasha Latysheva
 
Spark 의 핵심은 무엇인가? RDD! (RDD paper review)
byYongho Ha
 

What's hot

PDF
Data Management - Full Stack Deep Learning
bySergey Karayev
 
PDF
Apache Solr 検索エンジン入門
byYahoo!デベロッパーネットワーク
 
PDF
Spark SQL
byJoud Khattab
 
PDF
PFNのML/DL基盤を支えるKubernetesにおける自動化 / DevOpsDays Tokyo 2021
byPreferred Networks
 
PDF
PostgreSQL13でのpg_basebackupの改善について（第13回PostgreSQLアンカンファレンス@オンライン）
byNTT DATA Technology & Innovation
 
PDF
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
byEdureka!
 
PDF
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
byMinh Pham
 
PDF
5.6 以前の InnoDB Flushing
byTakanori Sejima
 
PPTX
Apache Spark Architecture
byAlexey Grishchenko
 
PDF
運用が楽になる分散データベース Riak
byTakahiko Sato
 
PPTX
Bert
byAbdallah Bashir
 
PDF
異次元のグラフデータベースNeo4j
by昌桓 李
 
PDF
第31回「今アツい、分散ストレージを語ろう」(2013/11/28 on しすなま!)
bySystem x 部 (生!) : しすなま! @ Lenovo Enterprise Solutions Ltd.
 
PPTX
Unique ID generation in distributed systems
byDave Gardner
 
PDF
PostgreSQLアーキテクチャ入門
byUptime Technologies LLC (JP)
 
PDF
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
PDF
What is new in PostgreSQL 14?
byMydbops
 
PPTX
Data Federation with Apache Spark
byDataWorks Summit
 
PDF
深層学習フレームワークにおけるIntel CPU/富岳向け最適化法
byMITSUNARI Shigeo
 
PDF
Layout lm paper review
bytaeseon ryu
 
Data Management - Full Stack Deep Learning
bySergey Karayev
 
Apache Solr 検索エンジン入門
byYahoo!デベロッパーネットワーク
 
Spark SQL
byJoud Khattab
 
PFNのML/DL基盤を支えるKubernetesにおける自動化 / DevOpsDays Tokyo 2021
byPreferred Networks
 
PostgreSQL13でのpg_basebackupの改善について（第13回PostgreSQLアンカンファレンス@オンライン）
byNTT DATA Technology & Innovation
 
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
byEdureka!
 
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
byMinh Pham
 
5.6 以前の InnoDB Flushing
byTakanori Sejima
 
Apache Spark Architecture
byAlexey Grishchenko
 
運用が楽になる分散データベース Riak
byTakahiko Sato
 
Bert
byAbdallah Bashir
 
異次元のグラフデータベースNeo4j
by昌桓 李
 
第31回「今アツい、分散ストレージを語ろう」(2013/11/28 on しすなま!)
bySystem x 部 (生!) : しすなま! @ Lenovo Enterprise Solutions Ltd.
 
Unique ID generation in distributed systems
byDave Gardner
 
PostgreSQLアーキテクチャ入門
byUptime Technologies LLC (JP)
 
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
What is new in PostgreSQL 14?
byMydbops
 
Data Federation with Apache Spark
byDataWorks Summit
 
深層学習フレームワークにおけるIntel CPU/富岳向け最適化法
byMITSUNARI Shigeo
 
Layout lm paper review
bytaeseon ryu
 

Similar to Deep Learning State of the Art (2019) - MIT by Lex Fridman

PDF
MIT Deep Learning Basics: Introduction and Overview by Lex Fridman
byPeerasak C.
 
PDF
Landscape of AI/ML in 2023
byHyunJoon Jung
 
PDF
Deep Learning and the state of AI / 2016
byGrigory Sapunov
 
PPTX
Transformer Zoo
byGrigory Sapunov
 
PDF
Google Cloud - Google's vision on AI
byBigDataExpo
 
PDF
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
byAI Frontiers
 
PDF
"Large-Scale Deep Learning for Building Intelligent Computer Systems," a Keyn...
byEdge AI and Vision Alliance
 
PDF
building intelligent systems with large scale deep learning
bymustafa sarac
 
PDF
NewMind AI Weekly Chronicles – September ’25 Week II
byNewMind AI
 
PDF
The Past, Present, and Future of Deep Learning | presentation
byCodiste
 
PPTX
Other-Applications-of-Deep-Learning (1).pptx
byamarnath709648
 
PPTX
What is a Transformer basdfdeeflidofdif.pptx
bynnmcfirstyear
 
PPTX
Deep Learning and Recurrent Neural Networks in the Enterprise
byJosh Patterson
 
PDF
20181212 Queensland AI Meetup
byAdam Craven
 
PPTX
Deep Learning and Watson Studio
bySasha Lazarevic
 
PDF
Deep_Learning_Applications_Detailed_Presentation.pdf
bylearnthebest39
 
PDF
TensorFlow London: Cutting edge generative models
bySeldon
 
PDF
Should we be afraid of Transformers?
byDominik Seisser
 
PDF
Alison Lowndes, Artificial Intelligence DevRel, Nvidia – Fueling the Artifici...
byTechsylvania
 
PDF
Deep learning applications - for fun and profit
byRoberto Pereira Silveira
 
MIT Deep Learning Basics: Introduction and Overview by Lex Fridman
byPeerasak C.
 
Landscape of AI/ML in 2023
byHyunJoon Jung
 
Deep Learning and the state of AI / 2016
byGrigory Sapunov
 
Transformer Zoo
byGrigory Sapunov
 
Google Cloud - Google's vision on AI
byBigDataExpo
 
Jeff Dean at AI Frontiers: Trends and Developments in Deep Learning Research
byAI Frontiers
 
"Large-Scale Deep Learning for Building Intelligent Computer Systems," a Keyn...
byEdge AI and Vision Alliance
 
building intelligent systems with large scale deep learning
bymustafa sarac
 
NewMind AI Weekly Chronicles – September ’25 Week II
byNewMind AI
 
The Past, Present, and Future of Deep Learning | presentation
byCodiste
 
Other-Applications-of-Deep-Learning (1).pptx
byamarnath709648
 
What is a Transformer basdfdeeflidofdif.pptx
bynnmcfirstyear
 
Deep Learning and Recurrent Neural Networks in the Enterprise
byJosh Patterson
 
20181212 Queensland AI Meetup
byAdam Craven
 
Deep Learning and Watson Studio
bySasha Lazarevic
 
Deep_Learning_Applications_Detailed_Presentation.pdf
bylearnthebest39
 
TensorFlow London: Cutting edge generative models
bySeldon
 
Should we be afraid of Transformers?
byDominik Seisser
 
Alison Lowndes, Artificial Intelligence DevRel, Nvidia – Fueling the Artifici...
byTechsylvania
 
Deep learning applications - for fun and profit
byRoberto Pereira Silveira
 

More from Peerasak C.

PDF
e-Conomy SEA 2020
byPeerasak C.
 
PDF
Testing Business Ideas by David Bland & Alex Osterwalder
byPeerasak C.
 
PDF
A Roadmap for CrossBorder Data Flows: Future-Proofing Readiness and Cooperati...
byPeerasak C.
 
PDF
General Population Census of the Kingdom of Cambodia 2019
byPeerasak C.
 
PDF
e-Conomy SEA Report 2019
byPeerasak C.
 
PDF
Royal Virtues by Somdet Phra Buddhaghosajahn (P. A. Payutto) translated by Ja...
byPeerasak C.
 
PDF
Artificial Intelligence and Life in 2030. Standford U. Sep.2016
byPeerasak C.
 
PDF
How to start a business: Checklist and Canvas
byPeerasak C.
 
PDF
หนังสือ BlockChain for Government Services
byPeerasak C.
 
PDF
Blockchain for Government Services
byPeerasak C.
 
PDF
TAT The Journey
byPeerasak C.
 
PDF
เสวนาเจาะเกณฑ์ระดมทุนและการลงทุนกับ LiVex
byPeerasak C.
 
PDF
เจาะเทรนด์โลก 2020: Positive Power โดย TCDC
byPeerasak C.
 
PDF
RD Strategies & D2rive
byPeerasak C.
 
PDF
๙ พระสูตร ปฐมโพธิกาล
byPeerasak C.
 
PDF
นิตยสารคิด (Creative Thailand) ฉบับเดือนตุลาคม 2562
byPeerasak C.
 
PDF
คู่มือการใช้ระบบ Biz Portal ส าหรับผู้ยื่นคำขอ (Biz Portal User Manual)
byPeerasak C.
 
PDF
K SME INSPIRED September 2019 Issue 65
byPeerasak C.
 
PDF
The Multiple Effects of Business Planning on New Venture Performance
byPeerasak C.
 
PDF
FREELANCING IN AMERICA 2019
byPeerasak C.
 
e-Conomy SEA 2020
byPeerasak C.
 
Testing Business Ideas by David Bland & Alex Osterwalder
byPeerasak C.
 
A Roadmap for CrossBorder Data Flows: Future-Proofing Readiness and Cooperati...
byPeerasak C.
 
General Population Census of the Kingdom of Cambodia 2019
byPeerasak C.
 
e-Conomy SEA Report 2019
byPeerasak C.
 
Royal Virtues by Somdet Phra Buddhaghosajahn (P. A. Payutto) translated by Ja...
byPeerasak C.
 
Artificial Intelligence and Life in 2030. Standford U. Sep.2016
byPeerasak C.
 
How to start a business: Checklist and Canvas
byPeerasak C.
 
หนังสือ BlockChain for Government Services
byPeerasak C.
 
Blockchain for Government Services
byPeerasak C.
 
TAT The Journey
byPeerasak C.
 
เสวนาเจาะเกณฑ์ระดมทุนและการลงทุนกับ LiVex
byPeerasak C.
 
เจาะเทรนด์โลก 2020: Positive Power โดย TCDC
byPeerasak C.
 
RD Strategies & D2rive
byPeerasak C.
 
๙ พระสูตร ปฐมโพธิกาล
byPeerasak C.
 
นิตยสารคิด (Creative Thailand) ฉบับเดือนตุลาคม 2562
byPeerasak C.
 
คู่มือการใช้ระบบ Biz Portal ส าหรับผู้ยื่นคำขอ (Biz Portal User Manual)
byPeerasak C.
 
K SME INSPIRED September 2019 Issue 65
byPeerasak C.
 
The Multiple Effects of Business Planning on New Venture Performance
byPeerasak C.
 
FREELANCING IN AMERICA 2019
byPeerasak C.
 

Recently uploaded

PDF
FAMILY ASSESSMENT FORMAT - CHN practical
byDr. Sudhadevi Sadanandan
 
PDF
The Drift Principle: When Information Accelerates Faster Than Minds Can Compress
byReality Drift Archive
 
DOCX
Mobile applications Devlopment ReTest year 2025-2026
byDr. Mazin Mohamed alkathiri
 
PPTX
Rectal Surgery in Senior Citiizens .pptx
byJohn Thanakumar
 
PPTX
Limpitlaw "Licensing: From Mindset to Milestones"
byNational Information Standards Organization (NISO)
 
PDF
Models of Teaching - TNTEU - B.Ed I Semester - Teaching and Learning - BD1TL ...
byDr Raja Mohammed T
 
PDF
The Pity of War: Form, Fragment, and the Artificial Echo | Understanding War ...
byNidhi Pandya
 
PPTX
Semester 6 unit 2 Atopic dermatitis.pptx
bysachin7989
 
PDF
APM Wessex Network: DEIB Policy into Practice
byAssociation for Project Management
 
PPTX
2025-2026 History in your Hands Class 4 December 2025 January 2026 .pptx
byEilsONeill
 
PPTX
York "Collaboration for Research Support at U-M Library"
byNational Information Standards Organization (NISO)
 
PPTX
How to Manage Line Discounts in Odoo 18 POS
byCeline George
 
PDF
All Students Workshop 25 Yoga Wellness by LDMMIA
by©LDMMIA, ©Reiki Yoga
 
PDF
Analyzing the data of your initial survey
byThelma Villaflores
 
PPTX
Searching in PubMed andCochrane_Practical Presentation.pptx
bySystematic Reviews Network (SRN)
 
PDF
DHA/HAAD/MOH/DOH OPTOMETRY MCQ PYQ. .pdf
byAnmol Singh
 
PPTX
ICH Harmonization A Global Pathway to Unified Drug Regulation.pptx
byDr. Smita Kumbhar
 
PDF
1ST APPLICATION FOR ANNULMENT (4)8787666.pdf
bykonstantinantountoum1
 
PDF
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
PPTX
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 
FAMILY ASSESSMENT FORMAT - CHN practical
byDr. Sudhadevi Sadanandan
 
The Drift Principle: When Information Accelerates Faster Than Minds Can Compress
byReality Drift Archive
 
Mobile applications Devlopment ReTest year 2025-2026
byDr. Mazin Mohamed alkathiri
 
Rectal Surgery in Senior Citiizens .pptx
byJohn Thanakumar
 
Limpitlaw "Licensing: From Mindset to Milestones"
byNational Information Standards Organization (NISO)
 
Models of Teaching - TNTEU - B.Ed I Semester - Teaching and Learning - BD1TL ...
byDr Raja Mohammed T
 
The Pity of War: Form, Fragment, and the Artificial Echo | Understanding War ...
byNidhi Pandya
 
Semester 6 unit 2 Atopic dermatitis.pptx
bysachin7989
 
APM Wessex Network: DEIB Policy into Practice
byAssociation for Project Management
 
2025-2026 History in your Hands Class 4 December 2025 January 2026 .pptx
byEilsONeill
 
York "Collaboration for Research Support at U-M Library"
byNational Information Standards Organization (NISO)
 
How to Manage Line Discounts in Odoo 18 POS
byCeline George
 
All Students Workshop 25 Yoga Wellness by LDMMIA
by©LDMMIA, ©Reiki Yoga
 
Analyzing the data of your initial survey
byThelma Villaflores
 
Searching in PubMed andCochrane_Practical Presentation.pptx
bySystematic Reviews Network (SRN)
 
DHA/HAAD/MOH/DOH OPTOMETRY MCQ PYQ. .pdf
byAnmol Singh
 
ICH Harmonization A Global Pathway to Unified Drug Regulation.pptx
byDr. Smita Kumbhar
 
1ST APPLICATION FOR ANNULMENT (4)8787666.pdf
bykonstantinantountoum1
 
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 

Deep Learning State of the Art (2019) - MIT by Lex Fridman

  • 1.
  • 2.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 3.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Encoder-Decoder Architecture Sequence-to-Sequence Model - Neural Machine Translation [295] Encoder RNN encodes input sequence into a fixed size vector, and then is passed repeatedly to decoder RNN.
  • 4.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Attention [296] Attention mechanism allows the network to refer back to the input sequence, instead of forcing it to encode all information into one fixed-length vector.
  • 5.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Attention [312]
  • 6.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Attention [312]
  • 7.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Self-Attention [312] "The animal didn't cross the street because it was too tired" More details: http://jalammar.github.io/illustrated-transformer/
  • 8.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Transformer [312] Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
  • 9.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references Word Embeddings (Word2Vec) [297] Word Vector Skip Gram Model:
  • 10.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Context-Aware Embeddings [312]
  • 11.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references OpenAI Transformer 1. Pre-train a Transformer’s decoder for language modeling 2. Train it on, for example, a sentence classification task [312] 1 2
  • 12.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references BERT Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).
  • 13.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references BERT Applications Now you can use BERT: • Create contextualized word embeddings (like ELMo) • Sentence classification • Sentence pair classification • Sentence pair similarity • Multiple choice • Sentence tagging • Question answering
  • 14.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 15.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Tesla Autopilot Hardware v2+ • Specialized NVIDIA Drive PX 2 hardware • Neural network takes all 8 cameras as input • Based on Inception v1 architecture
  • 16.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Autopilot Reaches 1 Billion Miles (~0.5 Billion on Hardware v2+)
  • 17.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 18.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references AutoML and Neural Architecture Search (NASNet) [300, 301]
  • 19.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references AdaNet [317]
  • 20.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 21.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references AutoAugment: RL for Data Augmentation [318] • Show that transfer learning can also be done augmentation policies instead of weights (or with).
  • 22.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 23.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Training on Randomized Synthetic Data [319] • number and types of objects • number, types, colors, and scales of distractors • texture on the object of interest, and background photograph • location of the virtual camera with respect to the scene • angle of the camera with respect to the scene • number and locations of point lights Tremblay, Jonathan, et al. "Training deep networks with synthetic data: Bridging the reality gap by domain randomization." (2018).
  • 24.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 25.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Segmentation Annotation with Polygon-RNN++ • High-resolution polygon segmentation generation with human-in-the-loop [319] Acuna, David, et al. "Efficient Interactive Annotation of Segmentation Datasets With Polygon-RNN++." CVPR 2018.
  • 26.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 27.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references DAWNBench: Training Fast and Cheap DAWNBench: • Competition on speed and cost of training and inference that achieves 93% for ImageNet and 94% for CIFAR 10 fast.ai – Training: • ImageNet in 3 hours for $25 • CIFAR10 for $0.26 • Key idea: During training, if you very slowly increase learning rate while decreasing momentum, you can train at extremely high learning rates, thus avoiding over-fitting, and training in far fewer epochs. • Details: http://bit.ly/2H6yv6H [320]
  • 28.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 29.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references BigGAN: State of the Art in Image Synthesis • Same GAN techniques, much larger scale • Increase model capacity + increase batch size [321] Brock, Andrew, Jeff Donahue, and Karen Simonyan. "Large scale gan training for high fidelity natural image synthesis." (2018).
  • 30.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 31.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
  • 32.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
  • 33.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Video-to-Video Synthesis • Model temporal dynamics by adding a spatio-temporal adversarial objective to the GAN framework [322] Wang, Ting-Chun, et al. "Video-to-video synthesis." (2018).
  • 34.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 35.
    2019https://deeplearning.mit.edu • AlexNet (2012):First CNN (15.4%) • 8 layers • 61 million parameters • ZFNet (2013): 15.4% to 11.2% • 8 layers • More filters. Denser stride. • VGGNet (2014): 11.2% to 7.3% • Beautifully uniform: 3x3 conv, stride 1, pad 1, 2x2 max pool • 16 layers • 138 million parameters • GoogLeNet (2014): 11.2% to 6.7% • Inception modules • 22 layers • 5 million parameters (throw away fully connected layers) • ResNet (2015): 6.7% to 3.57% • More layers = better performance • 152 layers • CUImage (2016): 3.57% to 2.99% • Ensemble of 6 models • SENet (2017): 2.99% to 2.251% • Squeeze and excitation block: network is allowed to adaptively adjust the weighting of each feature map in the convolutional block. References: [90] Human error (5.1%) surpassed in 2015
  • 36.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Object Detection / Localization Region-Based Methods | Shown: Faster R-CNN [299]
  • 37.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Object Detection / Localization Single-Shot Methods | Shown: SSD [299]
  • 38.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Object Detection: State of the Art Progress [326]
  • 39.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Semantic Segmentation [175]
  • 40.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references State-of-the-Art: DeepLab v3 • PASCAL VOC Challenge: http://bit.ly/2HdzTEu
  • 41.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references FCN (Nov 2014) Paper: “Fully Convolutional Networks for Semantic Segmentation” • Repurpose Imagenet pretrained nets • Upsample using deconvolution • Skip connections to improve coarseness of upsampling [173]
  • 42.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references SegNet (Nov 2015) Paper: “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation” • Maxpooling indices transferred to decoder to improve the segmentation resolution. [175]
  • 43.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references Dilated Convolutions (Nov 2015) Paper: “Multi-Scale Context Aggregation by Dilated Convolutions” • Since pooling decreases resolution: • Added “dilated convolution layer” • Still interpolate up from 1/8 of original image size [175]
  • 44.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v1, v2 (Jun 2016) Paper: “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs” • Added fully-connected Conditional Random Fields (CRFs) – as a post-processing step • Smooth segmentation based on the underlying image intensities [179]
  • 45.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references Key Aspects of Segmentation • Fully convolutional networks (FCNs) - replace fully-connected layers with convolutional layers. Classification network is where the biggest gains come from. • Deeper, updated models (ResNet, etc) consistent with ImageNet Challenge object classification tasks. • Conditional Random Fields (CRFs) to capture both local and long-range dependencies within an image to refine the prediction map. • Dilated convolution (aka Atrous convolution) – maintain computational cost, increase resolution of intermediate feature maps • Process at multiple scales and combine the information together [174, 176]
  • 46.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v3 Paper: “Rethinking Atrous Convolution for Semantic Image Segmentation” • Multi-scale processing, without increasing parameters. • Increasing “atrous rate” enlarges the model’s “field-of-view” [179]
  • 47.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references DeepLab v3 trained on CityScapes Tutorial: https://github.com/lexfridman/mit-deep-learning
  • 48.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 49.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references Atari Breakout [85] After 120 Minutes of Training After 10 Minutes of Training After 240 Minutes of Training
  • 50.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references DQN: Atari Games (2015) [83]
  • 51.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references AlphaGo (2016): Beat Top Human at Go [83]
  • 52.
    2019https://deeplearning.mit.eduFor the fullupdated list of references visit: https://selfdrivingcars.mit.edu/references AlphaGo Zero (2017): Beats AlphaGo [149]
  • 53.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references [324] AlphaZero (Dec 2017) vs StockFish (Chess) & Elmo (Shogi)
  • 54.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references OpenAI & Dota 2 • Dota 2 as a testbed for the messiness and continuous nature of the real world: teamwork, long time horizons, and hidden information. [325]
  • 55.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references OpenAI & Dota 2 Progress • Aug, 2017: 1v1 bot beats top professional Dota 2 players. • Aug, 2018: OpenAI Five lost two games against top Dota 2 players at The International. “We are looking forward to pushing Five to the next level.” [325]
  • 56.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: State of the Art* (Breakthrough Developments in 2017 & 2018) • BERT and Natural Language Processing • Tesla Autopilot Hardware v2+: Neural Networks at Scale • AdaNet: AutoML with Ensembles • AutoAugment: Deep RL Data Augmentation • Training Deep Networks with Synthetic Data • Segmentation Annotation with Polygon-RNN++ • DAWNBench: Training Fast and Cheap • BigGAN: State of the Art in Image Synthesis • Video-to-Video Synthesis • Semantic Segmentation • AlphaZero & OpenAI Five • Deep Learning Frameworks * This is not a list of state-of-the-art results on main machine learning benchmark datasets. It’s an overview of exciting recent developments.
  • 57.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning Frameworks [327] Deep Learning Framework Power Scores (by Jeff Hale) http://bit.ly/2GBa3tU 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Factors to consider: • Learning curve • Speed of development • Size and passion of community • Number of papers implemented in framework • Likelihood of long-term growth and stability • Ecosystem of tooling
  • 58.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Deep Learning: 2019 and Beyond • On backpropagation: “My view is throw it all away and start again.” • “The future depends on some graduate student who is deeply suspicious of everything I have said.” - Geoffrey Hinton “Godfather of Deep Learning” [328]
  • 59.
    2019https://deeplearning.mit.eduFor the fulllist of references visit: https://hcai.mit.edu/references Thank You Website: deeplearning.mit.edu • Videos and slides posted on the website • Code posted on GitHub: https://github.com/lexfridman/mit-deep-learning