Lecture 4: Transformers (Full Stack Deep Learning - Spring 2021)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Full Stack Deep Learning - UC Berkeley Spring 2021 - Sergey Karayev, Josh Tobin, Pieter Abbeel
Transfer Learning and Trans...

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision
• Embeddings and Languag...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
• Let's classify birds! We have 10K labeled
images.

...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
5
A LOT of data
Large model
Traditional Machine Learn...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
6
A LOT of data
Large model
Traditional Machine Learn...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
7
A LOT of data
Train on much
less data
Large model
P...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
8
A LOT of data Much less data
Large model
Pretrained...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
9
Imagenet-
pretrained
model
Keep these Replace these

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
10
Imagenet-
pretrained
model
Keep these Replace thes...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transfer Learning
11
https://wandb.ai/wandb/wandb-lightning/reports/Tra...

Full Stack Deep Learning - UC Berkeley Spring 2021
Model Zoos
12
https://pytorch.org/docs/stable/torchvision/models.html h...

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
13

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Converting words to vectors
• In NLP, our inputs are sequences of words...

Full Stack Deep Learning - UC Berkeley Spring 2021
One-hot encoding
16
https://towardsdatascience.com/why-do-we-use-embedd...

Full Stack Deep Learning - UC Berkeley Spring 2021
Problems with one-hot encoding
• Scales poorly with vocabulary size

• ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Map one-hot to dense vectors
• Problem: how do we find the values of th...

Full Stack Deep Learning - UC Berkeley Spring 2021
Solution 1: Learn as part of the task
19

Full Stack Deep Learning - UC Berkeley Spring 2021
Solution 2: Learn a Language Model
• "Pre-train" for your NLP task by l...

Full Stack Deep Learning - UC Berkeley Spring 2021
Solution 2: Learn a Language Model
21
http://jalammar.github.io/illustr...

Full Stack Deep Learning - UC Berkeley Spring 2021
• Slide an N-sized window through the text, forming a dataset of predic...

Full Stack Deep Learning - UC Berkeley Spring 2021
• Slide an N-sized window through the text, forming a dataset of predic...

Full Stack Deep Learning - UC Berkeley Spring 2021
• Slide an N-sized window through the text, forming a dataset of predic...

Full Stack Deep Learning - UC Berkeley Spring 2021
• Look on both sides of the target word, and form multiple samples from...

Full Stack Deep Learning - UC Berkeley Spring 2021
• Binary instead of multi-class (faster training)
Speed up training
26
...

Full Stack Deep Learning - UC Berkeley Spring 2021
Training
27
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
28
http://jalammar.github.io/illustrated-word2vec/
"kin...

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
29
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
30
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Vector math...
31
https://mathinsight.org/vectors_cartesian_coordinates...

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
32
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
33
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
34
http://jalammar.github.io/illustrated-word2vec/

Full Stack Deep Learning - UC Berkeley Spring 2021
Word2Vec (2013)
35
https://ruder.io/nlp-imagenet/

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
36

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Beyond embeddings
• Word2Vec and GloVe embeddings became popular in ~20...

Full Stack Deep Learning - UC Berkeley Spring 2021
Elmo (2018)
• Bidirectional stacked LSTM!
39
https://ruder.io/nlp-image...

Full Stack Deep Learning - UC Berkeley Spring 2021
Elmo (2018)
40

Full Stack Deep Learning - UC Berkeley Spring 2021
Elmo (2018)
41
https://www.slideshare.net/shuntaroy/a-review-of-deep-co...

Full Stack Deep Learning - UC Berkeley Spring 2021
Q&A: SQuAD
• 100K question-answer pairs

• Answers are always spans in
...

Full Stack Deep Learning - UC Berkeley Spring 2021
Natural Language Inference: SNLI
• What relation exists between piece o...

Full Stack Deep Learning - UC Berkeley Spring 2021
Elmo (2018)
44
https://www.slideshare.net/shuntaroy/a-review-of-deep-co...

Full Stack Deep Learning - UC Berkeley Spring 2021
GLUE
45
https://mccormickml.com/2019/11/05/GLUE/

Full Stack Deep Learning - UC Berkeley Spring 2021
GLUE
• 9 tasks, model score is averaged across them
46
https://mccormic...

Full Stack Deep Learning - UC Berkeley Spring 2021
ULMFit (2018)
47
https://humboldt-wi.github.io/blog/research/informatio...

Full Stack Deep Learning - UC Berkeley Spring 2021
ULMFit (2018)
48
https://humboldt-wi.github.io/blog/research/informatio...

Full Stack Deep Learning - UC Berkeley Spring 2021
Pre-training is here!
49

Full Stack Deep Learning - UC Berkeley Spring 2021
Pre-training is here!
50

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
51

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Rise of Transformers
53

Full Stack Deep Learning - UC Berkeley Spring 2021
Rise of Transformers
54

Full Stack Deep Learning - UC Berkeley Spring 2021
Attention is all you need (2017)
• Encoder-decoder with only attention ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Attention is all you need (2017)
56
• For simplicity, can focus just on...

Full Stack Deep Learning - UC Berkeley Spring 2021
Attention is all you need (2017)
• The components:

• (Masked) Self-att...

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• Input: sequence of tensors 
59
http://www.peterb...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• Input: sequence of tensors 
• Output: sequence o...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• Input: sequence of tensors 
• Output: sequence o...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• Input: sequence of tensors 
• Output: sequence o...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
63
http://www.peterbloem.nl/blog/transformers
The ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• SO FAR:

• No learned weights

• Order of the se...

Full Stack Deep Learning - UC Berkeley Spring 2021
Basic self-attention
• SO FAR:

• No learned weights
• Order of the seq...

Full Stack Deep Learning - UC Berkeley Spring 2021
Query, Key, Value
• Every input vector x_i is used in 3 ways:

• Compar...

Full Stack Deep Learning - UC Berkeley Spring 2021
Query, Key, Value
• We can process each input vector to fulfill the thr...

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
68

Full Stack Deep Learning - UC Berkeley Spring 2021
Multi-head attention
• Multiple "heads" of attention just means learnin...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• Self-attention layer -> Layer normalization -> Dense laye...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• Self-attention layer -> Layer normalization -> Dense laye...

Full Stack Deep Learning - UC Berkeley Spring 2021
Layer Normalization
• Neural net layers work best when
input vectors ha...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• Self-attention layer -> Layer normalization -> Dense laye...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• SO FAR:

• Learned query, key, value weights

• Multiple ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• SO FAR:

• Learned query, key, value weights

• Multiple ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
76
http://www.peterbloem.nl/blog/transformers

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer
• Position embedding: just what it sounds!
77
http://www.pe...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer: last trick
• Since the Transformer sees all inputs at once...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformer: last trick
• Since the Transformer sees all inputs at once...

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
80

Full Stack Deep Learning - UC Berkeley Spring 2021
Outline
• Transfer Learning in Computer Vision

• Embeddings and Langua...

Full Stack Deep Learning - UC Berkeley Spring 2021
Attention is all you need (2017)
• Encoder-decoder for translation
82

Full Stack Deep Learning - UC Berkeley Spring 2021
Attention is all you need (2017)
83
http://www.peterbloem.nl/blog/trans...

Full Stack Deep Learning - UC Berkeley Spring 2021
GPT / GPT-2
• Generative Pre-trained Transformer
84
https://docs.google...

Full Stack Deep Learning - UC Berkeley Spring 2021
GPT / GPT-2
• Generative Pre-trained Transformer

• GPT learns to predi...

Full Stack Deep Learning - UC Berkeley Spring 2021
GPT / GPT-2
• Generative Pre-trained Transformer

• GPT learns to predi...

Full Stack Deep Learning - UC Berkeley Spring 2021
GPT / GPT-2
87
117M params 345M params 762M params 1.5B params
Trained ...

Full Stack Deep Learning - UC Berkeley Spring 2021
Talk to Transformer
88
https://talktotransformer.com
1.5B parameter mod...

Full Stack Deep Learning - UC Berkeley Spring 2021
BERT
• Bidirectional Encoder Representations from Transformers

• Encod...

Full Stack Deep Learning - UC Berkeley Spring 2021
BERT
• 340M parameters:

• 24 transformer blocks, embedding dim of 1024...

Full Stack Deep Learning - UC Berkeley Spring 2021
Transformers
91
https://medium.com/huggingface/encoder-decoders-in-tran...

Full Stack Deep Learning - UC Berkeley Spring 2021
Number of parameters
92
https://www.microsoft.com/en-us/research/blog/t...

Full Stack Deep Learning - UC Berkeley Spring 2021
T5: Text-to-Text Transfer Transformer
• Feb 2020

• Evaluated most rece...

Full Stack Deep Learning - UC Berkeley Spring 2021
T5: Text-to-Text Transfer Transformer
94
https://medium.com/@Moscow25/t...

Full Stack Deep Learning - UC Berkeley Spring 2021
95

Full Stack Deep Learning - UC Berkeley Spring 2021
GPT-3
96

Full Stack Deep Learning - UC Berkeley Spring 2021
97

Full Stack Deep Learning - UC Berkeley Spring 2021
98

GSV Breakfast Club - January 2021 - Sergey Karayev
Really, really good text generation
Qspwjefe!cz!nf
Hfofsbufe

GSV Breakfast Club - January 2021 - Sergey Karayev
One problem:
perpetuating unfortunate
things
https://twitter.com/an_ope...

GSV Breakfast Club - January 2021 - Sergey Karayev
Reasonable mental model

GSV Breakfast Club - January 2021 - Sergey Karayev
🤯 Useful for more than text?

GSV Breakfast Club - January 2021 - Sergey Karayev
🤯 Useful for more than text?

GSV Breakfast Club - January 2021 - Sergey Karayev
Image Generation
https://openai.com/blog/dall-e/

Full Stack Deep Learning - UC Berkeley Spring 2021
No sign of slowing down
105

Full Stack Deep Learning - UC Berkeley Spring 2021
106

Full Stack Deep Learning - UC Berkeley Spring 2021
Number of parameters
• The way things are trending, only big
companies ...

Full Stack Deep Learning - UC Berkeley Spring 2021
DistillBERT
108
Knowledge Distillation: a smaller model is
trained to r...

Full Stack Deep Learning - UC Berkeley Spring 2021
Papers and Leaderboards
109
paperswithcode.com

Full Stack Deep Learning - UC Berkeley Spring 2021
Papers and Leaderboards
110
https://github.com/sebastianruder/NLP-progr...

Full Stack Deep Learning - UC Berkeley Spring 2021
Implementations
111
https://github.com/huggingface/transformers

Full Stack Deep Learning - UC Berkeley Spring 2021
Questions?
112

Full Stack Deep Learning - UC Berkeley Spring 2021
113
http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Full Stack Deep Learning - UC Berkeley Spring 2021
Thank you!
114

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 114 Ad

View on Slideshare

1 of 114 Ad