SlideShare a Scribd company logo
1 of 87
Download to read offline
Deep Learning:
State of the Art (2020)
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning Lecture Series
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
“AI began with an ancient wish to forge the gods.”
- Pamela McCorduck, Machines Who Think, 1979
Visualized here are 3% of the neurons and 0.0001% of the synapses in the brain.
Thalamocortical system visualization via DigiCortex Engine.
Frankenstein (1818)
Ex Machina (2015)
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning & AI in Context of Human History
1700s and beyond: Industrial revolution, steam
engine, mechanized factory systems, machine tools
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Dreams, mathematical foundations, and engineering in reality.
Alan Turing, 1951: “It seems probable that once the machine
thinking method had started, it would not take long to outstrip
our feeble powers. They would be able to converse with each
other to sharpen their wits. At some stage therefore, we should
have to expect the machines to take control."
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Dreams, mathematical foundations, and engineering in reality.
Frank Rosenblatt, Perceptron (1957, 1962): Early description and
engineering of single-layer and multi-layer artificial neural
networks.
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Kasparov vs Deep Blue, 1997
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Lee Sedol vs AlphaGo, 2016
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Robots on four wheels.
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Artificial Intelligence in Context of Human History
Robots on two legs.
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
History of Deep Learning Ideas and Milestones*
• 1943: Neural networks
• 1957-62: Perceptron
• 1970-86: Backpropagation, RBM, RNN
• 1979-98: CNN, MNIST, LSTM, Bidirectional RNN
• 2006: “Deep Learning”, DBN
• 2009: ImageNet + AlexNet
• 2014: GANs
• 2016-17: AlphaGo, AlphaZero
• 2017: 2017-19: Transformers
* Dates are for perspective and not as definitive historical
record of invention or credit
We are here
Perspective:
• Universe created
13.8 billion years ago
• Earth created
4.54 billion years ago
• Modern humans
300,000 years ago
• Civilization
12,000 years ago
• Written record
5,000 years ago
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Turing Award for Deep Learning
• Yann LeCun
• Geoffrey Hinton
• Yoshua Bengio
Turing Award given for:
• “The conceptual and engineering breakthroughs that have made
deep neural networks a critical component of computing.”
• (Also, for popularization in the face of skepticism.)
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Early Key Figures in Deep Learning
(Not a Complete List by Any Means)
• 1943: Walter Pitts and Warren McCulloch
Computational models for neural nets
• 1957, 1962: Frank Rosenblatt
Perceptron (Single-Layer & Multi-Layer)
• 1965: Alexey Ivakhnenko and V. G. Lapa
Learning algorithm for MLP
• 1970: Seppo Linnainmaa
Backpropagation and automatic differentiation
• 1979: Kunihiko Fukushima
Convolutional neural networks
• 1982: John Hopfield
Hopfield networks (recurrent neural networks)
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
People of Deep Learning and Artificial Intelligence
• History of science is a story of both people and ideas.
• Many brilliant people contributed to the development of AI.
Schmidhuber, Jürgen. "Deep learning in
neural networks: An overview." Neural
networks 61 (2015): 85-117
https://arxiv.org/pdf/1404.7828.pdf
My (Lex) hope for the community:
• More respect, open-mindedness, collaboration, credit sharing.
• Less derision, jealousy, stubbornness, academic silos.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Limitations of Deep Learning
• 2019 is the year it became cool
to say that “deep learning” has
limitations.
• Books, articles, lectures, debates,
videos were released that
learning-based methods cannot
do commonsense reasoning.
[3, 4]
Prediction from Rodney Brooks:
“By 2020, the popular press starts having stories that the era of
Deep Learning is over.”
http://rodneybrooks.com/predictions-scorecard-2019-january-01/
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning Research Community is Growing
[2]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning Growth, Celebrations, and Limitations
Hopes for 2020
• Less Hype & Less Anti-Hype: Less tweets on how
there is too much hype in AI and more solid
research in AI.
• Hybrid Research: Less contentious, counter-
productive debates, more open-minded
interdisciplinary collaboration.
• Research topics:
• Reasoning
• Active learning and life-long learning
• Multi-modal and multi-task learning
• Open-domain conversation
• Applications: medical, autonomous vehicles
• Algorithmic ethics
• Robotics
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Competition and Convergence of Deep Learning Libraries
TensorFlow 2.0 and PyTorch 1.3
• Eager execution by default
(imperative programming)
• Keras integration + promotion
• Cleanup (API, etc.)
• TensorFlow.js
• TensorFlow Lite
• TensorFlow Serving
• TorchScript
(graph representation)
• Quantization
• PyTorch Mobile
(experimental)
• TPU support
Python 2 support ended on Jan 1, 2020.
>>> print “Goodbye World”
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Reinforcement Learning Frameworks
• TensorFlow
• OpenAI Baselines
• Stable Baselines – the one I recommend for beginners
• TensorForce
• Dopamine (Google)
• TF-Agents
• TRFL
• RLLib (+ Tune) – great for distributed RL & hyperparameter tuning
• Coach - huge selection of algorithms
• PyTorch
• Horizon
• SLM-Lab
• Misc
• RLgraph
• Keras-RL
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Reinforcement Learning Frameworks
• “Stable Baselines” (OpenAI Baselines Fork)
• A2C, PPO, TRPO, DQN, ACKTR, ACER and DDPG
• Good documentation (and code commenting)
• Easy to get started and use
[5]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning and Deep RL Frameworks
Hopes for 2020
• Framework-agnostic Research: Make it even
easier to translate a trained PyTorch model to
TensorFlow and vice-versa.
• Mature Deep RL frameworks: Converge to fewer,
actively-developed, stable RL frameworks that
are less tied to TensorFlow or PyTorch.
• Abstractions: Build higher and higher
abstractions (i.e. Keras) on top of deep learning
frameworks that empower researchers, scientists,
developers outside of machine learning field.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Transformer
[7, 8]
Vaswani et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
BERT
[9, 10]
Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
BERT Applications
Now you can use BERT:
• Create contextualized word
embeddings (like ELMo)
• Sentence classification
• Sentence pair classification
• Sentence pair similarity
• Multiple choice
• Sentence tagging
• Question answering
[9, 10]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Transformer-Based Language Models
• BERT (Google)
• XLNet (Google/CMU)
• RoBERTa (Facebook)
• DistilBERT (HuggingFace)
• CTRL (Salesforce)
• GPT-2 (OpenAI)
• ALBERT (Google)
• Megatron (NVIDIA)
[12]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Megatron
Shoeybi et al. (NVIDIA)
• Megatron-LM is a 8.3 billion parameter transformer language model with 8-
way model parallelism and 64-way data parallelism trained on 512 GPUs
(NVIDIA Tesla V100)
• Largest transformer model ever trained. 24x the size of BERT and 5.6x the
size of GPT-2.
[13]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
XLNET
Yang et al. (CMU, Google AI)
• Combine bidirectionality of BERT and the relative positional embeddings and the
recurrence mechanism of Transformer-XL.
• XLnet outperforms BERT on 20 tasks, often by a large margin.
• The new model achieves state-of-the-art performance on 18 NLP tasks including question
answering, natural language inference, sentiment analysis & document ranking.
[24, 25]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
ALBERT
Lan et al. (Google Research & Toyota Technological Institute at Chicago)
• Idea: Reduces parameters via cross-layer parameter sharing
• Results: An upgrade to BERT that advances the state-of-the-art performance
on 12 NLP tasks (including SQuAD2.0)
• Code: Open-source TensorFlow implementation, including a number of
ready-to-use ALBERT pre-trained language models
[11]
Machine performance
on the RACE challenge
(SAT-like reading
comprehension).
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Transformers Model Language
They Do Not Understand Language. Far from it (for now).
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
• Key takeaways in the report:
• Coordination during model release between organization is difficult but
possible
• Humans can be convinced by synthetic text
• Machine-based detection is difficult.
• My takeaways
• Conversations on this topic are difficult, because the model of sharing
between ML organizations and experts is mostly non-existent
• Humans are still better at deception (disinformation) and detection in
text and conversation (see Alexa prize slides)
[26]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Transferable Multi-Domain State Generator for Task-
Oriented Dialogue Systems
Wu et al. (Honk Kong UST, Salesforce) – ACL 2019 Outstanding Paper
• Task: Dialogue state tracking
• Problem: Over-dependence on domain
ontology and lack of knowledge sharing
across domains
• Details:
• Share model parameters across domains
• Track slot values mentioned anywhere
in a dialogue history with a context-
enhanced slot gate and copy mechanism
• Don’t require a predefined ontology
• Results: State-of-the-art joint goal
accuracy of 48.62% on MultiWOZ,
a challenging 5-domain human-
human dialogue dataset.
[34]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Explain Yourself! Leveraging Language Models for
Commonsense Reasoning
Rajani et al. (Salesforce)
[15]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Alexa Prize and Open Domain Conversations
• Amazon open-sourced the Topical-Chat dataset.
• Alexa Prize (like the Loebner Prize, etc) are teaching us valuable
lessons (from the Alquist 2.0 team):
• Parts: Break dialogue into small parts
• Tangents: Create an interconnected graph of topics. Be ready to jump
from context to context and back.
• Attention: Not everything that is said is important. E.g.: “You know, I’m
a really terrible cook. But I would like to ask you, what’s your favorite
food?”
• Opinions: Create opinions.
• ML
• Content: ML is okay for generic chitchat, but nothing more for now.
• Classification: ML classifies intent, finds entities or detects sentiment.
• Goal: Goal is to maximize entertainment not information.
[17, 18, 19]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Alexa Prize and Open Domain Conversations
[19]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
code2seq: Generating Sequences from Structured
Representations of Code
Alon et al. (Technion) – ICLR 2019
• Instead of treating source code as a sequence of tokens, code2seq leverages
the syntactic structure of programming languages to better encode source
code as paths in its abstract syntax tree (AST).
[14]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Natural Language Processing:
Hopes for 2020
• Reasoning: Combining (commonsense) reasoning
with language models
• Context: Extending language model context to
thousands of words.
• Dialogue: More focus on open-domain dialogue
• Video: Ideas and successes in self-supervised
learning in visual data.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
OpenAI & Dota 2
• Dota 2 as a testbed for the messiness and continuous nature of
the real world: teamwork, long time horizons, and hidden
information.
[21]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
OpenAI & Dota 2 Progress
• Aug, 2017: 1v1 bot beats top professional Dota 2 players.
• Aug, 2018: OpenAI Five lost two games against top Dota 2 players at The
International. “We are looking forward to pushing Five to the next level.”
• Apr, 2019: OpenAI Five beats OG team (the 2018 world champions)
[21]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
OpenAI & Dota 2 Progress
• The Difference: OpenAI Five’s victories in 2019 as compared to its losses in
2018 are due to: 8x more training compute (training for longer)
• Compute: Current version of OpenAI Five has consumed 800 petaflop/s-days
and experienced about 45,000 years of Dota self-play over 10 realtime months
• Performance: The 2019 version of OpenAI Five has a 99.9% win rate versus
the 2018 version.
[21]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind Quake III Arena Capture the Flag
[23]
• “Billions of people inhabit the planet, each with their own individual goals
and actions, but still capable of coming together through teams,
organisations and societies in impressive displays of collective intelligence.
This is a setting we call multi-agent learning: many individual agents must
act independently, yet learn to interact and cooperate with other agents.
This is an immensely difficult problem - because with co-adapting agents
the world is constantly changing.”
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind Quake III Arena Capture the Flag
• The agents automatically figure out the game rules, important
concepts, behaviors, strategies, etc.
[23]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind Quake III Arena Capture the Flag
• The agents automatically figure out the game rules, important
concepts, behaviors, strategies, etc.
[23]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind AlphaStar
• Dec, 2018: AlphaStar beats MaNa, one of the world’s strongest professional
StarCraft players, 5-0.
• Oct, 2019: AlphaStar reaches Grandmaster level playing the game under
professionally approved conditions (for humans).
[22]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind AlphaStar
[22]
“AlphaStar is an intriguing and unorthodox player – one with the reflexes and
speed of the best pros but strategies and a style that are entirely its own. The
way AlphaStar was trained, with agents competing against each other in a
league, has resulted in gameplay that’s unimaginably unusual; it really makes
you question how much of StarCraft’s diverse possibilities pro players have
really explored.”
- Kelazhur, professional StarCraft II player
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Pluribus: Six-Player No-Limit Texas Hold’em Poker
Brown et al. (CMU, Facebook AI)
• Six-Player No-Limit Texas Hold’em Poker
• Imperfect information
• Multi-agent
• Result: Pluribus won in six-player no-limit Texas Hold’em poker
• Offline: Self-play to generate coarse-grained “blueprint” strategy
• Iterative Monte Carlo CFR (MCCFR) algorithm
• Self-play allows for counterfactual reasoning
• Online: Use search to improve blueprint strategy based on
particular situation
• Abstractions
• Action abstractions: reduce action space
• Information abstraction: reduce decision space based on what
information has been revealed
[28]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Pluribus: Six-Player No-Limit Texas Hold’em Poker
Brown et al. (CMU, Facebook AI)
• Chris Ferguson: “Pluribus is a very hard opponent to play against. It’s really hard to pin
him down on any kind of hand. He’s also very good at making thin value bets on the
river. He’s very good at extracting value out of his good hands.”
• Darren Elias: “Its major strength is its ability to use mixed strategies. That’s the same
thing that humans try to do. It’s a matter of execution for humans — to do this in a
perfectly random way and to do so consistently. Most people just can’t.”
[28]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
OpenAI Rubik’s Cube Manipulation
[20]
• Deep RL: Reinforcement learning approach from OpenAI Five
• ADR: Automatic Domain Randomization (ADR) – generate progressively
more difficult environment as the system learns (alternative for self-play)
• Capacity: Term of “emergent meta-learning” is used to described the fact
that the network is constrained and the ADR process of environment
generation is not
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep RL and Self-Play:
Hopes for 2020
• Robotics: Use of RL methods in manipulation and
real-world interaction tasks.
• Human Behavior: Use of multi-agent self-play to
explore naturally emerging social behaviors as a
way to study equivalent multi-human systems.
• Games: Use RL to assist human experts in
discovering new strategies at games and other
tasks in simulation.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
The Lottery Ticket Hypothesis:
Finding Sparse, Trainable Neural Networks
Frankle et al. (MIT) - Best Paper at ICLR 2019
1. Randomly initialize a neural network.
2. Train the network until it converges.
3. Prune a fraction of the network.
4. Reset the weights of the remaining network to initialization values from step 1
5. Train the pruned, untrained network. Observe convergence and accuracy.
[29, 30]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
The Lottery Ticket Hypothesis:
Finding Sparse, Trainable Neural Networks
Frankle et al. (MIT) - ICLR 2019 Best Paper
• Idea: For every neural network, there is a subnetwork that can achieve the
same accuracy in isolation after training.
• Iterative pruning: Find this subset subset of nodes by iteratively training
network, pruning its smallest-magnitude weights, and re-initializing the
remaining connections to their original values. Iterative vs one-shot is key.
• Inspiring takeaway: There exist architectures that are much more efficient.
Let’s find them!
[29, 30, 31]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Challenging Common Assumptions in the Unsupervised
Learning of Disentangled Representations
Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper
• The goal of disentangled representations is to build models that can capture
explanatory factors in a vector.
• The figure above presents a model with a 10-dimensional representation vector.
• Each of the 10 panels visualizes what information is captured in one of the 10
different coordinates of the representation.
• From the top right and the top middle panel we see that the model has successfully
disentangled floor color, while the two bottom left panels indicate that object color
and size are still entangled.
[32, 33]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Challenging Common Assumptions in the Unsupervised
Learning of Disentangled Representations
Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper
• Proof: Unsupervised learning of disentangled representations without inductive
biases is impossible.
• Takeaway: Inductive biases (assumptions) should be made explicit
• Open problem: Finding good inductive biases for unsupervised model selection that
work across multiple data sets persists is a key open problem.
• Open Experiments: Open source library with implementations of the considered
disentanglement methods and metrics, a standardized training and evaluation
protocol, as well as visualization tools to better understand trained models.
[32, 33]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Double Descent:
Where Bigger Models and More Data Hurt
Nakkiran et al. (Harvard, OpenAI)
• Double Descent Phenomena: As we increase the number of
parameters in a neural network, the test error initially
decreases, increases, and, just as the model is able to fit the
train set, undergoes a second descent.
• Applicable to model size, training time, dataset size.
[36]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Science of Deep Learning and Interesting Directions
Hopes for 2020
• Fundamentals: Exploring fundamentals of model
selection, training dynamics, and representation
characteristics with respect to architecture
characteristics.
• Graph Neural Networks: Exploring use of graph
neural networks for combinatorial optimization,
recommendation systems, etc.
• Bayesian Deep Learning: Exploring Bayesian
neural networks for estimating uncertainty and
online/incremental learning.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
Level 2
Human is Responsible Machine is Responsible
Level 4
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Waymo
• On-road: 20 million miles
• Simulation: 10 billion miles
• Testing & Validation:
20,000 classes of
structured tests
• Initiated testing without a
safety driver
October, 2018: January, 2020:
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Tesla Autopilot
[37]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Tesla Autopilot
[37]
Active Learning Pipeline
(aka Data Engine)
Perform Task
(Perception, Prediction, Planning)
Discover Edge Case
Search for Others Like It
Annotate
Retrain Network
Neural Network
(Version N)
Human helps annotate tricky situations
Human helps design data mining procedures
Human annotates
Neural Network
(Version N+1)
Collaborative Deep Learning
(aka Software 2.0 Engineering)
Retrain Network
Neural Network
(Version N)
Data Engine
for Task 1
Neural Network
(Version N+1)
Data Engine
for Task 2
Data Engine
for Task 3
Data Engine
for Task 100
…
Deploy Updated Neural Network
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Vision vs Lidar
L2 vs L4
• Primarily: Vision Sensors + Deep Learning
• Pros:
• Highest resolution information
• Feasible to collect data at scale and learn
• Roads are designed for human eyes
• Cheap
• Cons:
• Needs a huge amount of data to be accurate
• Less explainable
• Driver must remain vigilant
• Primarily: Lidar + Maps
• Pros:
• Explainable, consistent
• Accurate with less data
• Cons:
• Less amenable to machine learning
• Expensive (for now)
• Safety driver or teleoperation fallback
Example L2 System:
Tesla Autopilot
2+ billion miles
Example L4 System:
Waymo
20+ million miles
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Open Questions for Tesla Autopilot
• Deep learning question:
Problem Difficulty: How difficult is driving? How many edge-
cases does it have? Can it be learned from data?
• Perception (detection, intention modeling, trajectory prediction)
• Action (in a game-theoretic setting)
• Balancing enjoyability and safety
• Human supervision of deep learning system question:
Vigilance: How good can Autopilot get before vigilance
decrements significantly?
• And … will this decrement nullify the safety benefits of automation?
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Open Questions for Waymo
• When we have maps, lidar, and geo-fenced routes:
Problem Difficulty: How difficult is driving? How many edge-
cases does it have? Can it be learned from data?
• Perception (detection, intention modeling, trajectory prediction)
• Action (in a game-theoretic setting)
• Balancing enjoyability and safety
• Simulation question:
How much can be learned from simulation?
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Autonomous Vehicles and AI-Assisted Driving
Hopes for 2020
• Applied deep learning innovation: Life-long learning,
active learning, multi-task learning
• Over-the-air updates: More level 2 systems begin both
data collection and over-the-air software updates.
• Public datasets of edge-cases: More publicly available
datasets of challenging cases.
• Simulators: Improvement of publicly available simulators
(CARLA, NVIDIA DRIVE Constellation, Voyage Deepdrive)
• Less hype: More balanced in-depth reporting (by
journalists and companies) on successes and challenges of
autonomous vehicle development.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
AI in Political Discourse: Andrew Yang
• First presidential candidate to discuss artificial
intelligence extensively as part of his platform
• Proposals
• Department: Create a new executive department –
the Department of Technology – to work with
private industry and Congressional leaders to
monitor technological developments, assess risks,
and create new guidance.
• Focus on AI: The new Department would be based
in Silicon Valley and would initially be focused on
Artificial Intelligence.
• Companies: Create a public-private partnership
between leading tech firms and experts within
government to identify emerging threats and
suggest ways to mitigate those threats while
maximizing the benefit of technological innovation
to society.
[27]
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
American AI Initiative
• In February, 2019, the president signed Executive
Order 13859 announcing the American AI Initiative
• Goals
• Investment in long-term research
• Support research in academia and industry
• Access to federal data
• Promote STEM education
• Develop AI in “a manner consistent with our Nation’s
values, policies, and priorities.”
• AI must also be developed in a way that does not
compromise our American values, civil liberties, or
freedoms.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Tech Leaders Testifying Before Congress
(Ethics of Recommender Systems)
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
DeepMind + Google Research:
Play Store App Discovery
[35]
• Candidate app generation:
LSTM → Transformer → efficient addition attention model
• Candidate app unbiasing:
• The model learns a bias that favors the apps that are shown – and thus installed
– more often.
• To help correct for this bias, impression-to-install rate weighting is introduced.
• Multiple objectives: relevance, popularity, or personal
preferences
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Government, Politics, Policy
Hopes for 2020
• Less fear of AI: More balanced, informed
discussion on the impact of AI in society.
• Experts: Continued conversations by government
officials about AI, privacy, cybersecurity with
experts in academia and industry.
• Recommender system transparency: More open
discussion and publication behind recommender
systems used in industry.
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Online Deep Learning Courses
• Deep Learning
• Fast.ai: Practical Deep Learning for Coders
• Jeremy Howard et al.
• Stanford CS231n: Convolutional Neural Networks for Visual Recognition
• Stanford CS224n: Natural Language Processing with Deep Learning
• Deeplearning.ai (Coursera): Deep Learning
• Andrew Ng
• Reinforcement Learning
• David Silver: Introduction to Reinforcement Learning
• OpenAI: Spinning Up in Deep RL
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Tutorials: Over 200 of the Best Machine Learning, NLP,
and Python Tutorials (by Robbie Allen)
• Link: http://bit.ly/36skFE7
• Topics
• Machine learning
• Activation and Loss Functions
• Bias
• Perceptron
• Regression
• Gradient descent
• Generative learning
• Support vector machines
• Backpropagation
• Deep Learning
• Optimization
• Long Short Term Memory
• Convolutional Neural Networks
• Recurrent Neural Nets (RNNs)
• Reinforcement Learning
• Generative Adversarial Networks
• Multi-task Learning
• NLP
• Word Vectors
• Encoder-Decoder
• TensorFlow
• PyTorch
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning Books
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Outline
• Deep Learning Growth, Celebrations, and Limitations
• Deep Learning and Deep RL Frameworks
• Natural Language Processing
• Deep RL and Self-Play
• Science of Deep Learning and Interesting Directions
• Autonomous Vehicles and AI-Assisted Driving
• Government, Politics, Policy
• Courses, Tutorials, Books
• General Hopes for 2020
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Deep Learning Growth, Celebrations, and Limitations
Hopes for 2020
• Reasoning
• Active learning and life-long learning
• Multi-modal and multi-task learning
• Open-domain conversation
• Applications: medical, autonomous vehicles
• Algorithmic ethics
• Robotics
• Recommender systems
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Hope for 2020: Recipe for Progress (in AI)
Skepticism
Criticism
Perseverance
(Never Give Up)
Open-
Mindedness
Crazy
Hard Work
“The future depends on
some graduate student
who is deeply suspicious
of everything I have said.”
- Geoffrey Hinton
2020https://deeplearning.mit.eduFor the full list of references visit:
http://bit.ly/deeplearn-sota-2020
Thank You
Videos and slides are posted on the website:
deeplearning.mit.edu

More Related Content

What's hot

Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Preferred Networks
 
Algorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyAlgorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyChris Johnson
 
Data Visualisation for Data Science
Data Visualisation for Data ScienceData Visualisation for Data Science
Data Visualisation for Data ScienceChristophe Bontemps
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learningleopauly
 
Recurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRURecurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRUananth
 
Convolutional Neural Network (CNN) - image recognition
Convolutional Neural Network (CNN)  - image recognitionConvolutional Neural Network (CNN)  - image recognition
Convolutional Neural Network (CNN) - image recognitionYUNG-KUEI CHEN
 
Introduction to Keras
Introduction to KerasIntroduction to Keras
Introduction to KerasJohn Ramey
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
Big Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewBig Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewSivashankar Ganapathy
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Larry Guo
 
RNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataRNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataYao-Chieh Hu
 
Prediction of heart disease using machine learning.pptx
Prediction of heart disease using machine learning.pptxPrediction of heart disease using machine learning.pptx
Prediction of heart disease using machine learning.pptxkumari36
 
Uncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial IntelligenceUncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial IntelligenceExperfy
 
Transformers In Vision From Zero to Hero (DLI).pptx
Transformers In Vision From Zero to Hero (DLI).pptxTransformers In Vision From Zero to Hero (DLI).pptx
Transformers In Vision From Zero to Hero (DLI).pptxDeep Learning Italia
 

What's hot (20)

Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural Networks
 
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
Introduction to Graph Neural Networks: Basics and Applications - Katsuhiko Is...
 
Rnn and lstm
Rnn and lstmRnn and lstm
Rnn and lstm
 
Algorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyAlgorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at Spotify
 
Generative models
Generative modelsGenerative models
Generative models
 
Data Visualisation for Data Science
Data Visualisation for Data ScienceData Visualisation for Data Science
Data Visualisation for Data Science
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learning
 
Recurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRURecurrent Neural Networks, LSTM and GRU
Recurrent Neural Networks, LSTM and GRU
 
Cnn
CnnCnn
Cnn
 
rnn BASICS
rnn BASICSrnn BASICS
rnn BASICS
 
Convolutional Neural Network (CNN) - image recognition
Convolutional Neural Network (CNN)  - image recognitionConvolutional Neural Network (CNN)  - image recognition
Convolutional Neural Network (CNN) - image recognition
 
Introduction to Keras
Introduction to KerasIntroduction to Keras
Introduction to Keras
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
Big Data - Applications and Technologies Overview
Big Data - Applications and Technologies OverviewBig Data - Applications and Technologies Overview
Big Data - Applications and Technologies Overview
 
LSTM
LSTMLSTM
LSTM
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10)
 
RNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataRNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential Data
 
Prediction of heart disease using machine learning.pptx
Prediction of heart disease using machine learning.pptxPrediction of heart disease using machine learning.pptx
Prediction of heart disease using machine learning.pptx
 
Uncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial IntelligenceUncertain Knowledge and Reasoning in Artificial Intelligence
Uncertain Knowledge and Reasoning in Artificial Intelligence
 
Transformers In Vision From Zero to Hero (DLI).pptx
Transformers In Vision From Zero to Hero (DLI).pptxTransformers In Vision From Zero to Hero (DLI).pptx
Transformers In Vision From Zero to Hero (DLI).pptx
 

Similar to Deep Learning State of the Art (2020)

Are museums a dial that only goes to 5?
Are museums a dial that only goes to 5? Are museums a dial that only goes to 5?
Are museums a dial that only goes to 5? Michael Edson
 
Connecting the Dots
Connecting the DotsConnecting the Dots
Connecting the DotsAvis Hek
 
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptx
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptxNordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptx
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptxISSIP
 
Incident response: art, science and engineering
Incident response: art, science and engineeringIncident response: art, science and engineering
Incident response: art, science and engineeringMaarten Van Horenbeeck
 
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...Stephen Rhind-Tutt
 
Introduction to the Venice Time Machine
Introduction to the Venice Time MachineIntroduction to the Venice Time Machine
Introduction to the Venice Time MachineGiovanni Colavizza
 
Portsmouth public library evening presentation
Portsmouth public library evening presentationPortsmouth public library evening presentation
Portsmouth public library evening presentationStephen Abram
 
Describing Everything - Open Web standards and classification
Describing Everything - Open Web standards and classificationDescribing Everything - Open Web standards and classification
Describing Everything - Open Web standards and classificationDan Brickley
 
Big data - An Introduction
Big data - An IntroductionBig data - An Introduction
Big data - An IntroductionSpotle.ai
 
Atlas Mapping in the Hybrid Age
Atlas Mapping in the Hybrid AgeAtlas Mapping in the Hybrid Age
Atlas Mapping in the Hybrid AgeAileen Buckley
 
Ntegra 20231003 v3.pptx
Ntegra 20231003 v3.pptxNtegra 20231003 v3.pptx
Ntegra 20231003 v3.pptxISSIP
 
Atlas mapping in the hybrid age
Atlas mapping in the hybrid ageAtlas mapping in the hybrid age
Atlas mapping in the hybrid ageAileen Buckley
 
Dealing with Information Overload
Dealing with Information OverloadDealing with Information Overload
Dealing with Information OverloadJohn Breslin
 
Introduction to Data Science.pptx
Introduction to Data Science.pptxIntroduction to Data Science.pptx
Introduction to Data Science.pptxPerumalPitchandi
 
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & Museums
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & MuseumsALIAOnline Practical Linked (Open) Data for Libraries, Archives & Museums
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & MuseumsJon Voss
 

Similar to Deep Learning State of the Art (2020) (20)

Are museums a dial that only goes to 5?
Are museums a dial that only goes to 5? Are museums a dial that only goes to 5?
Are museums a dial that only goes to 5?
 
Connecting the Dots
Connecting the DotsConnecting the Dots
Connecting the Dots
 
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptx
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptxNordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptx
NordicHouse 20240116 AI Quantum IFTF dfiscussionv7.pptx
 
Future trends in technology
Future trends in technologyFuture trends in technology
Future trends in technology
 
Incident response: art, science and engineering
Incident response: art, science and engineeringIncident response: art, science and engineering
Incident response: art, science and engineering
 
Making data more human
Making data more humanMaking data more human
Making data more human
 
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...
Virtual Reality, Augmented Reality, Mixed Reality, Extended Reality – what yo...
 
Introduction to the Venice Time Machine
Introduction to the Venice Time MachineIntroduction to the Venice Time Machine
Introduction to the Venice Time Machine
 
Portsmouth public library evening presentation
Portsmouth public library evening presentationPortsmouth public library evening presentation
Portsmouth public library evening presentation
 
Describing Everything - Open Web standards and classification
Describing Everything - Open Web standards and classificationDescribing Everything - Open Web standards and classification
Describing Everything - Open Web standards and classification
 
Big data - An Introduction
Big data - An IntroductionBig data - An Introduction
Big data - An Introduction
 
Educational Uses of SL
Educational Uses of SLEducational Uses of SL
Educational Uses of SL
 
Atlas Mapping in the Hybrid Age
Atlas Mapping in the Hybrid AgeAtlas Mapping in the Hybrid Age
Atlas Mapping in the Hybrid Age
 
Ntegra 20231003 v3.pptx
Ntegra 20231003 v3.pptxNtegra 20231003 v3.pptx
Ntegra 20231003 v3.pptx
 
Atlas mapping in the hybrid age
Atlas mapping in the hybrid ageAtlas mapping in the hybrid age
Atlas mapping in the hybrid age
 
Dealing with Information Overload
Dealing with Information OverloadDealing with Information Overload
Dealing with Information Overload
 
Data science general
Data science generalData science general
Data science general
 
Data Science Intro.pptx
Data Science Intro.pptxData Science Intro.pptx
Data Science Intro.pptx
 
Introduction to Data Science.pptx
Introduction to Data Science.pptxIntroduction to Data Science.pptx
Introduction to Data Science.pptx
 
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & Museums
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & MuseumsALIAOnline Practical Linked (Open) Data for Libraries, Archives & Museums
ALIAOnline Practical Linked (Open) Data for Libraries, Archives & Museums
 

More from inside-BigData.com

Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...inside-BigData.com
 
Transforming Private 5G Networks
Transforming Private 5G NetworksTransforming Private 5G Networks
Transforming Private 5G Networksinside-BigData.com
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...inside-BigData.com
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...inside-BigData.com
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...inside-BigData.com
 
HPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural NetworksHPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural Networksinside-BigData.com
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean MonitoringBiohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoringinside-BigData.com
 
Machine Learning for Weather Forecasts
Machine Learning for Weather ForecastsMachine Learning for Weather Forecasts
Machine Learning for Weather Forecastsinside-BigData.com
 
HPC AI Advisory Council Update
HPC AI Advisory Council UpdateHPC AI Advisory Council Update
HPC AI Advisory Council Updateinside-BigData.com
 
Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19inside-BigData.com
 
Energy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic TuningEnergy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic Tuninginside-BigData.com
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODHPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODinside-BigData.com
 
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud AccelerationVersal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Accelerationinside-BigData.com
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance EfficientlyZettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficientlyinside-BigData.com
 
Scaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's EraScaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's Erainside-BigData.com
 
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computingCUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computinginside-BigData.com
 
Introducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi ClusterIntroducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi Clusterinside-BigData.com
 

More from inside-BigData.com (20)

Major Market Shifts in IT
Major Market Shifts in ITMajor Market Shifts in IT
Major Market Shifts in IT
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...
 
Transforming Private 5G Networks
Transforming Private 5G NetworksTransforming Private 5G Networks
Transforming Private 5G Networks
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
 
HPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural NetworksHPC Impact: EDA Telemetry Neural Networks
HPC Impact: EDA Telemetry Neural Networks
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean MonitoringBiohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
 
Machine Learning for Weather Forecasts
Machine Learning for Weather ForecastsMachine Learning for Weather Forecasts
Machine Learning for Weather Forecasts
 
HPC AI Advisory Council Update
HPC AI Advisory Council UpdateHPC AI Advisory Council Update
HPC AI Advisory Council Update
 
Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19Fugaku Supercomputer joins fight against COVID-19
Fugaku Supercomputer joins fight against COVID-19
 
Energy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic TuningEnergy Efficient Computing using Dynamic Tuning
Energy Efficient Computing using Dynamic Tuning
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPODHPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
 
State of ARM-based HPC
State of ARM-based HPCState of ARM-based HPC
State of ARM-based HPC
 
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud AccelerationVersal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Acceleration
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance EfficientlyZettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
 
Scaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's EraScaling TCO in a Post Moore's Era
Scaling TCO in a Post Moore's Era
 
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computingCUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computing
 
Introducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi ClusterIntroducing HPC with a Raspberry Pi Cluster
Introducing HPC with a Raspberry Pi Cluster
 
Overview of HPC Interconnects
Overview of HPC InterconnectsOverview of HPC Interconnects
Overview of HPC Interconnects
 

Recently uploaded

Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 

Recently uploaded (20)

Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 

Deep Learning State of the Art (2020)

  • 1. Deep Learning: State of the Art (2020)
  • 2. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning Lecture Series
  • 3. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 4. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 “AI began with an ancient wish to forge the gods.” - Pamela McCorduck, Machines Who Think, 1979 Visualized here are 3% of the neurons and 0.0001% of the synapses in the brain. Thalamocortical system visualization via DigiCortex Engine. Frankenstein (1818) Ex Machina (2015)
  • 5. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning & AI in Context of Human History 1700s and beyond: Industrial revolution, steam engine, mechanized factory systems, machine tools We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 6. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Dreams, mathematical foundations, and engineering in reality. Alan Turing, 1951: “It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. They would be able to converse with each other to sharpen their wits. At some stage therefore, we should have to expect the machines to take control." We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 7. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Dreams, mathematical foundations, and engineering in reality. Frank Rosenblatt, Perceptron (1957, 1962): Early description and engineering of single-layer and multi-layer artificial neural networks. We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 8. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Kasparov vs Deep Blue, 1997 We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 9. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Lee Sedol vs AlphaGo, 2016 We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 10. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Robots on four wheels. We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 11. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Artificial Intelligence in Context of Human History Robots on two legs. We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 12. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 History of Deep Learning Ideas and Milestones* • 1943: Neural networks • 1957-62: Perceptron • 1970-86: Backpropagation, RBM, RNN • 1979-98: CNN, MNIST, LSTM, Bidirectional RNN • 2006: “Deep Learning”, DBN • 2009: ImageNet + AlexNet • 2014: GANs • 2016-17: AlphaGo, AlphaZero • 2017: 2017-19: Transformers * Dates are for perspective and not as definitive historical record of invention or credit We are here Perspective: • Universe created 13.8 billion years ago • Earth created 4.54 billion years ago • Modern humans 300,000 years ago • Civilization 12,000 years ago • Written record 5,000 years ago
  • 13. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Turing Award for Deep Learning • Yann LeCun • Geoffrey Hinton • Yoshua Bengio Turing Award given for: • “The conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.” • (Also, for popularization in the face of skepticism.)
  • 14. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Early Key Figures in Deep Learning (Not a Complete List by Any Means) • 1943: Walter Pitts and Warren McCulloch Computational models for neural nets • 1957, 1962: Frank Rosenblatt Perceptron (Single-Layer & Multi-Layer) • 1965: Alexey Ivakhnenko and V. G. Lapa Learning algorithm for MLP • 1970: Seppo Linnainmaa Backpropagation and automatic differentiation • 1979: Kunihiko Fukushima Convolutional neural networks • 1982: John Hopfield Hopfield networks (recurrent neural networks)
  • 15. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 People of Deep Learning and Artificial Intelligence • History of science is a story of both people and ideas. • Many brilliant people contributed to the development of AI. Schmidhuber, Jürgen. "Deep learning in neural networks: An overview." Neural networks 61 (2015): 85-117 https://arxiv.org/pdf/1404.7828.pdf My (Lex) hope for the community: • More respect, open-mindedness, collaboration, credit sharing. • Less derision, jealousy, stubbornness, academic silos.
  • 16. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Limitations of Deep Learning • 2019 is the year it became cool to say that “deep learning” has limitations. • Books, articles, lectures, debates, videos were released that learning-based methods cannot do commonsense reasoning. [3, 4] Prediction from Rodney Brooks: “By 2020, the popular press starts having stories that the era of Deep Learning is over.” http://rodneybrooks.com/predictions-scorecard-2019-january-01/
  • 17. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning Research Community is Growing [2]
  • 18. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning Growth, Celebrations, and Limitations Hopes for 2020 • Less Hype & Less Anti-Hype: Less tweets on how there is too much hype in AI and more solid research in AI. • Hybrid Research: Less contentious, counter- productive debates, more open-minded interdisciplinary collaboration. • Research topics: • Reasoning • Active learning and life-long learning • Multi-modal and multi-task learning • Open-domain conversation • Applications: medical, autonomous vehicles • Algorithmic ethics • Robotics
  • 19. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 20. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Competition and Convergence of Deep Learning Libraries TensorFlow 2.0 and PyTorch 1.3 • Eager execution by default (imperative programming) • Keras integration + promotion • Cleanup (API, etc.) • TensorFlow.js • TensorFlow Lite • TensorFlow Serving • TorchScript (graph representation) • Quantization • PyTorch Mobile (experimental) • TPU support Python 2 support ended on Jan 1, 2020. >>> print “Goodbye World”
  • 21. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Reinforcement Learning Frameworks • TensorFlow • OpenAI Baselines • Stable Baselines – the one I recommend for beginners • TensorForce • Dopamine (Google) • TF-Agents • TRFL • RLLib (+ Tune) – great for distributed RL & hyperparameter tuning • Coach - huge selection of algorithms • PyTorch • Horizon • SLM-Lab • Misc • RLgraph • Keras-RL
  • 22. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Reinforcement Learning Frameworks • “Stable Baselines” (OpenAI Baselines Fork) • A2C, PPO, TRPO, DQN, ACKTR, ACER and DDPG • Good documentation (and code commenting) • Easy to get started and use [5]
  • 23. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning and Deep RL Frameworks Hopes for 2020 • Framework-agnostic Research: Make it even easier to translate a trained PyTorch model to TensorFlow and vice-versa. • Mature Deep RL frameworks: Converge to fewer, actively-developed, stable RL frameworks that are less tied to TensorFlow or PyTorch. • Abstractions: Build higher and higher abstractions (i.e. Keras) on top of deep learning frameworks that empower researchers, scientists, developers outside of machine learning field.
  • 24. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 25. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Transformer [7, 8] Vaswani et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
  • 26. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 BERT [9, 10] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).
  • 27. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 BERT Applications Now you can use BERT: • Create contextualized word embeddings (like ELMo) • Sentence classification • Sentence pair classification • Sentence pair similarity • Multiple choice • Sentence tagging • Question answering [9, 10]
  • 28. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Transformer-Based Language Models • BERT (Google) • XLNet (Google/CMU) • RoBERTa (Facebook) • DistilBERT (HuggingFace) • CTRL (Salesforce) • GPT-2 (OpenAI) • ALBERT (Google) • Megatron (NVIDIA) [12]
  • 29. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Megatron Shoeybi et al. (NVIDIA) • Megatron-LM is a 8.3 billion parameter transformer language model with 8- way model parallelism and 64-way data parallelism trained on 512 GPUs (NVIDIA Tesla V100) • Largest transformer model ever trained. 24x the size of BERT and 5.6x the size of GPT-2. [13]
  • 30. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 XLNET Yang et al. (CMU, Google AI) • Combine bidirectionality of BERT and the relative positional embeddings and the recurrence mechanism of Transformer-XL. • XLnet outperforms BERT on 20 tasks, often by a large margin. • The new model achieves state-of-the-art performance on 18 NLP tasks including question answering, natural language inference, sentiment analysis & document ranking. [24, 25]
  • 31. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 ALBERT Lan et al. (Google Research & Toyota Technological Institute at Chicago) • Idea: Reduces parameters via cross-layer parameter sharing • Results: An upgrade to BERT that advances the state-of-the-art performance on 12 NLP tasks (including SQuAD2.0) • Code: Open-source TensorFlow implementation, including a number of ready-to-use ALBERT pre-trained language models [11] Machine performance on the RACE challenge (SAT-like reading comprehension).
  • 32. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020
  • 33. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Transformers Model Language They Do Not Understand Language. Far from it (for now).
  • 34. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 • Key takeaways in the report: • Coordination during model release between organization is difficult but possible • Humans can be convinced by synthetic text • Machine-based detection is difficult. • My takeaways • Conversations on this topic are difficult, because the model of sharing between ML organizations and experts is mostly non-existent • Humans are still better at deception (disinformation) and detection in text and conversation (see Alexa prize slides) [26]
  • 35. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Transferable Multi-Domain State Generator for Task- Oriented Dialogue Systems Wu et al. (Honk Kong UST, Salesforce) – ACL 2019 Outstanding Paper • Task: Dialogue state tracking • Problem: Over-dependence on domain ontology and lack of knowledge sharing across domains • Details: • Share model parameters across domains • Track slot values mentioned anywhere in a dialogue history with a context- enhanced slot gate and copy mechanism • Don’t require a predefined ontology • Results: State-of-the-art joint goal accuracy of 48.62% on MultiWOZ, a challenging 5-domain human- human dialogue dataset. [34]
  • 36. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Explain Yourself! Leveraging Language Models for Commonsense Reasoning Rajani et al. (Salesforce) [15]
  • 37. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Alexa Prize and Open Domain Conversations • Amazon open-sourced the Topical-Chat dataset. • Alexa Prize (like the Loebner Prize, etc) are teaching us valuable lessons (from the Alquist 2.0 team): • Parts: Break dialogue into small parts • Tangents: Create an interconnected graph of topics. Be ready to jump from context to context and back. • Attention: Not everything that is said is important. E.g.: “You know, I’m a really terrible cook. But I would like to ask you, what’s your favorite food?” • Opinions: Create opinions. • ML • Content: ML is okay for generic chitchat, but nothing more for now. • Classification: ML classifies intent, finds entities or detects sentiment. • Goal: Goal is to maximize entertainment not information. [17, 18, 19]
  • 38. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Alexa Prize and Open Domain Conversations [19]
  • 39. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 code2seq: Generating Sequences from Structured Representations of Code Alon et al. (Technion) – ICLR 2019 • Instead of treating source code as a sequence of tokens, code2seq leverages the syntactic structure of programming languages to better encode source code as paths in its abstract syntax tree (AST). [14]
  • 40. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Natural Language Processing: Hopes for 2020 • Reasoning: Combining (commonsense) reasoning with language models • Context: Extending language model context to thousands of words. • Dialogue: More focus on open-domain dialogue • Video: Ideas and successes in self-supervised learning in visual data.
  • 41. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 42. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 OpenAI & Dota 2 • Dota 2 as a testbed for the messiness and continuous nature of the real world: teamwork, long time horizons, and hidden information. [21]
  • 43. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 OpenAI & Dota 2 Progress • Aug, 2017: 1v1 bot beats top professional Dota 2 players. • Aug, 2018: OpenAI Five lost two games against top Dota 2 players at The International. “We are looking forward to pushing Five to the next level.” • Apr, 2019: OpenAI Five beats OG team (the 2018 world champions) [21]
  • 44. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 OpenAI & Dota 2 Progress • The Difference: OpenAI Five’s victories in 2019 as compared to its losses in 2018 are due to: 8x more training compute (training for longer) • Compute: Current version of OpenAI Five has consumed 800 petaflop/s-days and experienced about 45,000 years of Dota self-play over 10 realtime months • Performance: The 2019 version of OpenAI Five has a 99.9% win rate versus the 2018 version. [21]
  • 45. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind Quake III Arena Capture the Flag [23] • “Billions of people inhabit the planet, each with their own individual goals and actions, but still capable of coming together through teams, organisations and societies in impressive displays of collective intelligence. This is a setting we call multi-agent learning: many individual agents must act independently, yet learn to interact and cooperate with other agents. This is an immensely difficult problem - because with co-adapting agents the world is constantly changing.”
  • 46. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind Quake III Arena Capture the Flag • The agents automatically figure out the game rules, important concepts, behaviors, strategies, etc. [23]
  • 47. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind Quake III Arena Capture the Flag • The agents automatically figure out the game rules, important concepts, behaviors, strategies, etc. [23]
  • 48. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind AlphaStar • Dec, 2018: AlphaStar beats MaNa, one of the world’s strongest professional StarCraft players, 5-0. • Oct, 2019: AlphaStar reaches Grandmaster level playing the game under professionally approved conditions (for humans). [22]
  • 49. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind AlphaStar [22] “AlphaStar is an intriguing and unorthodox player – one with the reflexes and speed of the best pros but strategies and a style that are entirely its own. The way AlphaStar was trained, with agents competing against each other in a league, has resulted in gameplay that’s unimaginably unusual; it really makes you question how much of StarCraft’s diverse possibilities pro players have really explored.” - Kelazhur, professional StarCraft II player
  • 50. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Pluribus: Six-Player No-Limit Texas Hold’em Poker Brown et al. (CMU, Facebook AI) • Six-Player No-Limit Texas Hold’em Poker • Imperfect information • Multi-agent • Result: Pluribus won in six-player no-limit Texas Hold’em poker • Offline: Self-play to generate coarse-grained “blueprint” strategy • Iterative Monte Carlo CFR (MCCFR) algorithm • Self-play allows for counterfactual reasoning • Online: Use search to improve blueprint strategy based on particular situation • Abstractions • Action abstractions: reduce action space • Information abstraction: reduce decision space based on what information has been revealed [28]
  • 51. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Pluribus: Six-Player No-Limit Texas Hold’em Poker Brown et al. (CMU, Facebook AI) • Chris Ferguson: “Pluribus is a very hard opponent to play against. It’s really hard to pin him down on any kind of hand. He’s also very good at making thin value bets on the river. He’s very good at extracting value out of his good hands.” • Darren Elias: “Its major strength is its ability to use mixed strategies. That’s the same thing that humans try to do. It’s a matter of execution for humans — to do this in a perfectly random way and to do so consistently. Most people just can’t.” [28]
  • 52. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 OpenAI Rubik’s Cube Manipulation [20] • Deep RL: Reinforcement learning approach from OpenAI Five • ADR: Automatic Domain Randomization (ADR) – generate progressively more difficult environment as the system learns (alternative for self-play) • Capacity: Term of “emergent meta-learning” is used to described the fact that the network is constrained and the ADR process of environment generation is not
  • 53. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep RL and Self-Play: Hopes for 2020 • Robotics: Use of RL methods in manipulation and real-world interaction tasks. • Human Behavior: Use of multi-agent self-play to explore naturally emerging social behaviors as a way to study equivalent multi-human systems. • Games: Use RL to assist human experts in discovering new strategies at games and other tasks in simulation.
  • 54. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 55. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Frankle et al. (MIT) - Best Paper at ICLR 2019 1. Randomly initialize a neural network. 2. Train the network until it converges. 3. Prune a fraction of the network. 4. Reset the weights of the remaining network to initialization values from step 1 5. Train the pruned, untrained network. Observe convergence and accuracy. [29, 30]
  • 56. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Frankle et al. (MIT) - ICLR 2019 Best Paper • Idea: For every neural network, there is a subnetwork that can achieve the same accuracy in isolation after training. • Iterative pruning: Find this subset subset of nodes by iteratively training network, pruning its smallest-magnitude weights, and re-initializing the remaining connections to their original values. Iterative vs one-shot is key. • Inspiring takeaway: There exist architectures that are much more efficient. Let’s find them! [29, 30, 31]
  • 57. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper • The goal of disentangled representations is to build models that can capture explanatory factors in a vector. • The figure above presents a model with a 10-dimensional representation vector. • Each of the 10 panels visualizes what information is captured in one of the 10 different coordinates of the representation. • From the top right and the top middle panel we see that the model has successfully disentangled floor color, while the two bottom left panels indicate that object color and size are still entangled. [32, 33]
  • 58. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper • Proof: Unsupervised learning of disentangled representations without inductive biases is impossible. • Takeaway: Inductive biases (assumptions) should be made explicit • Open problem: Finding good inductive biases for unsupervised model selection that work across multiple data sets persists is a key open problem. • Open Experiments: Open source library with implementations of the considered disentanglement methods and metrics, a standardized training and evaluation protocol, as well as visualization tools to better understand trained models. [32, 33]
  • 59. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Double Descent: Where Bigger Models and More Data Hurt Nakkiran et al. (Harvard, OpenAI) • Double Descent Phenomena: As we increase the number of parameters in a neural network, the test error initially decreases, increases, and, just as the model is able to fit the train set, undergoes a second descent. • Applicable to model size, training time, dataset size. [36]
  • 60. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Science of Deep Learning and Interesting Directions Hopes for 2020 • Fundamentals: Exploring fundamentals of model selection, training dynamics, and representation characteristics with respect to architecture characteristics. • Graph Neural Networks: Exploring use of graph neural networks for combinatorial optimization, recommendation systems, etc. • Bayesian Deep Learning: Exploring Bayesian neural networks for estimating uncertainty and online/incremental learning.
  • 61. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 62. Level 2 Human is Responsible Machine is Responsible Level 4
  • 63. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Waymo • On-road: 20 million miles • Simulation: 10 billion miles • Testing & Validation: 20,000 classes of structured tests • Initiated testing without a safety driver October, 2018: January, 2020:
  • 64. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Tesla Autopilot [37]
  • 65. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Tesla Autopilot [37]
  • 66. Active Learning Pipeline (aka Data Engine) Perform Task (Perception, Prediction, Planning) Discover Edge Case Search for Others Like It Annotate Retrain Network Neural Network (Version N) Human helps annotate tricky situations Human helps design data mining procedures Human annotates Neural Network (Version N+1)
  • 67.
  • 68.
  • 69. Collaborative Deep Learning (aka Software 2.0 Engineering) Retrain Network Neural Network (Version N) Data Engine for Task 1 Neural Network (Version N+1) Data Engine for Task 2 Data Engine for Task 3 Data Engine for Task 100 … Deploy Updated Neural Network
  • 70. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Vision vs Lidar L2 vs L4 • Primarily: Vision Sensors + Deep Learning • Pros: • Highest resolution information • Feasible to collect data at scale and learn • Roads are designed for human eyes • Cheap • Cons: • Needs a huge amount of data to be accurate • Less explainable • Driver must remain vigilant • Primarily: Lidar + Maps • Pros: • Explainable, consistent • Accurate with less data • Cons: • Less amenable to machine learning • Expensive (for now) • Safety driver or teleoperation fallback Example L2 System: Tesla Autopilot 2+ billion miles Example L4 System: Waymo 20+ million miles
  • 71. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Open Questions for Tesla Autopilot • Deep learning question: Problem Difficulty: How difficult is driving? How many edge- cases does it have? Can it be learned from data? • Perception (detection, intention modeling, trajectory prediction) • Action (in a game-theoretic setting) • Balancing enjoyability and safety • Human supervision of deep learning system question: Vigilance: How good can Autopilot get before vigilance decrements significantly? • And … will this decrement nullify the safety benefits of automation?
  • 72. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Open Questions for Waymo • When we have maps, lidar, and geo-fenced routes: Problem Difficulty: How difficult is driving? How many edge- cases does it have? Can it be learned from data? • Perception (detection, intention modeling, trajectory prediction) • Action (in a game-theoretic setting) • Balancing enjoyability and safety • Simulation question: How much can be learned from simulation?
  • 73. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Autonomous Vehicles and AI-Assisted Driving Hopes for 2020 • Applied deep learning innovation: Life-long learning, active learning, multi-task learning • Over-the-air updates: More level 2 systems begin both data collection and over-the-air software updates. • Public datasets of edge-cases: More publicly available datasets of challenging cases. • Simulators: Improvement of publicly available simulators (CARLA, NVIDIA DRIVE Constellation, Voyage Deepdrive) • Less hype: More balanced in-depth reporting (by journalists and companies) on successes and challenges of autonomous vehicle development.
  • 74. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 75. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 AI in Political Discourse: Andrew Yang • First presidential candidate to discuss artificial intelligence extensively as part of his platform • Proposals • Department: Create a new executive department – the Department of Technology – to work with private industry and Congressional leaders to monitor technological developments, assess risks, and create new guidance. • Focus on AI: The new Department would be based in Silicon Valley and would initially be focused on Artificial Intelligence. • Companies: Create a public-private partnership between leading tech firms and experts within government to identify emerging threats and suggest ways to mitigate those threats while maximizing the benefit of technological innovation to society. [27]
  • 76. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 American AI Initiative • In February, 2019, the president signed Executive Order 13859 announcing the American AI Initiative • Goals • Investment in long-term research • Support research in academia and industry • Access to federal data • Promote STEM education • Develop AI in “a manner consistent with our Nation’s values, policies, and priorities.” • AI must also be developed in a way that does not compromise our American values, civil liberties, or freedoms.
  • 77. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Tech Leaders Testifying Before Congress (Ethics of Recommender Systems)
  • 78. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 DeepMind + Google Research: Play Store App Discovery [35] • Candidate app generation: LSTM → Transformer → efficient addition attention model • Candidate app unbiasing: • The model learns a bias that favors the apps that are shown – and thus installed – more often. • To help correct for this bias, impression-to-install rate weighting is introduced. • Multiple objectives: relevance, popularity, or personal preferences
  • 79. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Government, Politics, Policy Hopes for 2020 • Less fear of AI: More balanced, informed discussion on the impact of AI in society. • Experts: Continued conversations by government officials about AI, privacy, cybersecurity with experts in academia and industry. • Recommender system transparency: More open discussion and publication behind recommender systems used in industry.
  • 80. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 81. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Online Deep Learning Courses • Deep Learning • Fast.ai: Practical Deep Learning for Coders • Jeremy Howard et al. • Stanford CS231n: Convolutional Neural Networks for Visual Recognition • Stanford CS224n: Natural Language Processing with Deep Learning • Deeplearning.ai (Coursera): Deep Learning • Andrew Ng • Reinforcement Learning • David Silver: Introduction to Reinforcement Learning • OpenAI: Spinning Up in Deep RL
  • 82. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Tutorials: Over 200 of the Best Machine Learning, NLP, and Python Tutorials (by Robbie Allen) • Link: http://bit.ly/36skFE7 • Topics • Machine learning • Activation and Loss Functions • Bias • Perceptron • Regression • Gradient descent • Generative learning • Support vector machines • Backpropagation • Deep Learning • Optimization • Long Short Term Memory • Convolutional Neural Networks • Recurrent Neural Nets (RNNs) • Reinforcement Learning • Generative Adversarial Networks • Multi-task Learning • NLP • Word Vectors • Encoder-Decoder • TensorFlow • PyTorch
  • 83. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning Books
  • 84. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline • Deep Learning Growth, Celebrations, and Limitations • Deep Learning and Deep RL Frameworks • Natural Language Processing • Deep RL and Self-Play • Science of Deep Learning and Interesting Directions • Autonomous Vehicles and AI-Assisted Driving • Government, Politics, Policy • Courses, Tutorials, Books • General Hopes for 2020
  • 85. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Deep Learning Growth, Celebrations, and Limitations Hopes for 2020 • Reasoning • Active learning and life-long learning • Multi-modal and multi-task learning • Open-domain conversation • Applications: medical, autonomous vehicles • Algorithmic ethics • Robotics • Recommender systems
  • 86. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Hope for 2020: Recipe for Progress (in AI) Skepticism Criticism Perseverance (Never Give Up) Open- Mindedness Crazy Hard Work “The future depends on some graduate student who is deeply suspicious of everything I have said.” - Geoffrey Hinton
  • 87. 2020https://deeplearning.mit.eduFor the full list of references visit: http://bit.ly/deeplearn-sota-2020 Thank You Videos and slides are posted on the website: deeplearning.mit.edu