Lecture 3: RNNs - Full Stack Deep Learning - Spring 2021

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Full Stack Deep Learning - UW Spring 2020 - Sergey Karayev - with content by Pieter Abbeel, Josh Tobin
Recurrent Neural Ne...

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectional...

Agenda
1. Sequence Problems
2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectionali...

Sequence
Time series
forecasting Time series Predicted next value
Sentiment
classification
Review text Predicted sentiment...

Types of sequence problems
1. Why RNNs?
(From http://karpathy.github.io/2015/05/21/rnn-eﬀectiveness/)

Full Stack Deep Learning - UW Spring 2020
Questions?
6

Why not use feedforward networks?
1. Why RNNs?
many to many

Why not use feedforward networks?
1. Why RNNs?
Concatenate
many to many

Why not use feedforward networks?
1. Why RNNs?
Concatenate
Fully
connected
Output
many to many

Why not use feedforward networks?
1. Why RNNs?
Concatenate
Fully
connected
Output
Reshape
many to many

Problem 1: variable length inputs
1. Why RNNs?
many to many
? Can deal with this by
padding all sequences to
the max lengt...

Problem 2: memory scaling
1. Why RNNs?
many to many
Memory requirement
scales linearly in number
of timesteps
k-dim featur...

Problem 3: overkill
1. Why RNNs?
many to many
k-dim feature t timesteps
k * t dim
d-dim
k * t * d dim
matrix
This matrix h...

Full Stack Deep Learning - UW Spring 2020
Questions?
14

Agenda
1. Sequence Problems

2. RNNs
3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectionali...

Core idea of RNNs
2. Review of RNNs
Stateful computation
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbV...

Core idea of RNNs
Stateful computation
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69...

The RNN in code
2. Review of RNNs
(Adapted from http://karpathy.github.io/2015/05/21/rnn-eﬀectiveness/)

A look at compute_next_h
2. Review of RNNs
xt
ht−1
( -dim)
nh
( -dim)
nx
ht
Whh
(nh × nh)
Wxh
(nh × nx)
( -dim)
nh
= dim o...

RNNs for many-to-one problems
2. Review of RNNs

RNNs for many-to-one problems
2. Review of RNNs
RNN
0.5
0.2
-0.1
-0.3
0.4
1.2
FC
Input Encoder
State at last
timestep Clas...

Full Stack Deep Learning - UW Spring 2020
Questions?
22

RNNs for one-to-many problems
2. Review of RNNs

RNNs for one-to-many problems
2. Review of RNNs
ConvNet
(e.g.)
0.5
0.2
-0.1
-0.3
0.4
1.2
RNN
“The quick brown
fox jumped o...

RNNs for one-to-many problems
2. Review of RNNs
h0
<latexit sha1_base64="u90zgAiIVkwK7FYP2F5fm926QP8=">AAAB6nicbVBNS8NAEJ3...

2. Review of RNNs
h0
<latexit sha1_base64="u90zgAiIVkwK7FYP2F5fm926QP8=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBiyepaD+gD...

Using RNNs for one-to-many problems
2. Review of RNNs
h0
<latexit sha1_base64="u90zgAiIVkwK7FYP2F5fm926QP8=">AAAB6nicbVBNS...

2. Review of RNNs
h0
<latexit sha1_base64="u90zgAiIVkwK7FYP2F5fm926QP8=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBiyepaD+gD...

2. Review of RNNs
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBiyepaD+gD...

Sequence loss functions
2. Review of RNNs
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3Ur1q/o...

Full Stack Deep Learning - UW Spring 2020
Questions?
31

2. Review of RNNs
RNN
0.5
0.2
-0.1
-0.3
0.4
1.2
RNN
Input Encoder
Hidden
state Decoder Output
Encoder-decoder architecture...

I am a student <s>
Je suis étudiant <s>
RNNs for many-to-many problems
2. Review of RNNs

I am a student <s>
Je suis étudiant <s>
RNNs for many-to-many problems
2. Review of RNNs
All the information in the
input ...

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs
4. Case study: Machine Translation 
(Bidirectionali...

RNN Desiderata
• Goal: handle long sequences

• Connect events from the past to outcomes
in the future

• i.e., Long-term ...

Vanilla RNNs: the reality
• Can’t handle more than 10-20 timesteps

• Longer-term dependencies get lost

• Why? Vanishing ...

Backpropagation through time
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0...

Backpropagation through time
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0G...

Gradients in RNNs: backpropagation
through time
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ3...

Gradients in RNNs: backprapogation
through time
ht
<latexit sha1_base64="9pFHr6U2bFdt9rB9QRXLRwu088U=">AAAB6nicbVBNS8NAEJ...

Back to vanishing gradients
Derivatives of common activation functions
• sigmoid and tanh have deriv-
atives << 1 near sat...

Intro to LSTMs
Recall vanilla RNNs
4. LSTMs

Visualization of vanilla RNNs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Visualization of vanilla RNNs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
As distance from a h...

Visualization of vanilla RNNs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Full Stack Deep Learning - UW Spring 2020
Questions?
47

Intro to LSTMs
Idea: use a compute_next_h that preserves gradients
For more info, see http://colah.github.io/posts/2015-08...

Building up to LSTMs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Main idea: introduce a new
"c...

Building up to LSTMs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Forget gate:
decide what part...

Building up to LSTMs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Input gate:
decide how to upd...

Building up to LSTMs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Now actually update the cell ...

Building up to LSTMs
4. LSTMs
From http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Finally, decide what to outpu...

What about GRUs, etc?
LSTM: A Search Space Odyssey
Greff, et al.
An Empirical Exploration of Recurrent
Neural Network Arch...

What about GRUs, etc?
Our advice
• LSTMs work well for most tasks

• Try GRUs if LSTMs are not
performing well
4. LSTMs

Full Stack Deep Learning - UW Spring 2020
Questions?
56

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectional...

Google’s Neural Machine Translation System:
Bridging the Gap between Human and Machine
Translation
(Wu et al., 2016)

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

Description of the problem
Input
Output
“I am a student”
“Je suis étudiant(e)” (French)
“Soy un estudiante” (Spanish)
我是学⽣...

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

2. Model architecture
RNN
0.5
0.2
-0.1
-0.3
0.4
1.2
RNN
Input Encoder
Hidden
state Decoder Output
Encoder-decoder architec...

2. Model architecture
I am a student <s>
Je suis étudiant <s>
Normal LSTM

2. Model architecture
I am a student <s>
Je suis étudiant <s>
Problem 1:
underfitting
Normal LSTM

2. Model architecture
Solution: stacked LSTM layers
I am a student <s>
Je suis étudiant <s>

2. Model architecture
I am a student <s>
Je suis étudiant <s>
Problem 2: stacked
LSTMs are hard to train
“In our experienc...

2. Model architecture
Solution: residual connections

2. Model architecture
I am a student <s>
Je suis étudiant <s>
Problem 3: Too much
information has to be
encoded in the las...

2. Model architecture
Solution: attention
Key idea: Instead of compressing all past time steps into a
single hidden state,...

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog
0....

2. Model architecture
le renard brun rapide a sauté sur le chien paresseux
The quick brown fox jumped over the lazy dog
0....

2. Model architecture
I am a student <s>
Je suis étudiant <s>
Attention: how to use relevance scores

2. Model architecture
Attention: how to use relevance scores
I am a student <s>
Je suis étudiant <s>
p1
<latexit sha1_base...

2. Model architecture
Attention: how to use relevance scores
I am a student <s>
Je suis étudiant <s>
p1
<latexit sha1_base...

2. Model architecture
Attention Visualization
https://distill.pub/2016/augmented-rnns/#attentional-interfaces

2. Model architecture
Attention Visualization
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html

2. Model architecture
Attention: not just for translation
https://distill.pub/2016/augmented-rnns/#attentional-interfaces

2. Model architecture
https://distill.pub/2016/augmented-rnns/#attentional-interfaces
Attention: not just for translation

Full Stack Deep Learning - UW Spring 2020
Questions?
83

I am a student <s>
Je suis étudiant <s>
Attention
2. Model architecture
Stacked LSTM with attention

I am a student <s>
Je suis étudiant <s>
Attention
2. Model architecture
Stacked LSTM with attention
Problem 4: LSTMs only
...

2. Model architecture
Why is future information important?
The quick brown fox <s>
“quick” could mean a lot of
things: fas...

2. Model architecture
Solution: bidirectionality
Key idea: Use one LSTM to process the sequence in forward
order, the othe...

2. Model architecture
The quick brown fox <s>
Solution: bidirectionality

2. Model architecture
The quick brown fox <s>
Concat
Solution: bidirectionality

I am a student <s>
Je suis étudiant <s>
Attention
2. Model architecture
Stacked biderictional LSTMs with attention

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

3. Loss function
GNMT loss function
Minimize negative log likelihood
L =
N
X
i=1
log P✓(Y i
| Xi
)
<latexit sha1_base64="H...

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

Summary of GNMT approach
• Stacked LSTM encoder-decoder architecture with residual
connections

• Attention enables longer...

Full Stack Deep Learning - UW Spring 2020
Questions?
95

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectional...

Full Stack Deep Learning - UW Spring 2020
The Goal
97
Inputs
Expected Outputs
Lab 2: Line text recognizer

Full Stack Deep Learning - UW Spring 2020
The Idea
98
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
Expected Outputs
t

Full Stack Deep Learning - UW Spring 2020
The idea
99
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t h
Expected Outpu...

Full Stack Deep Learning - UW Spring 2020
The idea
100
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t h e
Expected Ou...

Full Stack Deep Learning - UW Spring 2020
The idea
101
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t h e _
Expected ...

Full Stack Deep Learning - UW Spring 2020
The idea
102
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t h e _ q
Works g...

Full Stack Deep Learning - UW Spring 2020
The problem
103
Lab 2: Line text recognizer
Inputs
Convnet + LSTM
t
What if we a...

Full Stack Deep Learning - UW Spring 2020 104
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t t
The problem

Full Stack Deep Learning - UW Spring 2020 105
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t t h
The problem

Full Stack Deep Learning - UW Spring 2020 106
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t t h e
The problem

Full Stack Deep Learning - UW Spring 2020 107
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t t h e _
The problem

Full Stack Deep Learning - UW Spring 2020 108
Inputs
Lab 2: Line text recognizer
Convnet + LSTM
t t h e _ _
The problem

Full Stack Deep Learning - UW Spring 2020 109
Lab 2: Line text recognizer
t t h e _ _
Misalignment!
The problem

Full Stack Deep Learning - UW Spring 2020 110
Lab 2: Line text recognizer
More generally
https://distill.pub/2017/ctc/
X
Y

Full Stack Deep Learning - UW Spring 2020
Intuition for the solution
111
Lab 2: Line text recognizer
https://distill.pub/2...

Full Stack Deep Learning - UW Spring 2020
CTC Loss
112
Lab 2: Line text recognizer
https://distill.pub/2017/ctc/

Full Stack Deep Learning - UW Spring 2020
For more of the technical details
113
Lab 2: Line text recognizer
https://distil...

Full Stack Deep Learning - UW Spring 2020
We implement this in Lab 3!
114
Lab 2: Line text recognizer

Full Stack Deep Learning - UW Spring 2020
Questions?
115

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectional...

Pros
• Encoder / decoder LSTM architectures can
model arbitrary (one-to-many, many-to-one,
and many-to-many) sequence prob...

OpenAI Dota 2
https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five/

OpenAI Dota 2
https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five/
Brain:
1024-unit
LSTM!

Cons
• Recurrent network training is not as
parallelizable as FC or CNN, due to the
need to go in sequence

• Therefore mu...

Full Stack Deep Learning - UW Spring 2020
Questions?
121

Agenda
1. Sequence Problems

2. RNNs

3. Vanishing gradients and LSTMs

4. Case study: Machine Translation 
(Bidirectional...

Sequence data does not require recurrent models!
• Last week, we saw that a simple convolutional network was able to deal
...

WaveNet: A Generative Model for Raw Audio
(van den Oord et al, 2016)
Approach we’ll discuss
Used in Google Assistant and G...

Key ideas introduced
• Convolutional sequence models

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

Description of the problem
Input Output
“I am a student” Audio waveform of speaking
the text
1. Problem description

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

2. Model architecture
Baseline model architecture
RNN
0.5
0.2
-0.1
-0.3
0.4
1.2
RNN
Input Encoder
Hidden
state Decoder Out...

2. Model architecture
Baseline model architecture
RNN
0.5
0.2
-0.1
-0.3
0.4
1.2
RNN
Input Encoder
Hidden
state Decoder Out...

2. Model architecture
Insight: convolutions can be used for sequence data

2. Model architecture
Insight: convolutions can be used for sequence data

2. Model architecture
Insight: convolutions can be used for sequence data
Causal
convolution

2. Model architecture
Insight: convolutions can be used for sequence data
Causal
convolution
t
X
i=t k
xiwi
<latexit sha1_...

2. Model architecture
Challenge: getting a large receptive field
Receptive
field

2. Model architecture
Solution: dilated convolutions
Receptive
field

2. Model architecture
Other details about the model architecture
1. Gated activation units
Filter Gate

2. Model architecture
Other details about the model architecture
2. Residual and skip connections used throughout

2. Model architecture
Other details about the model architecture
3. Condition every layer on the input representation

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

3. Loss function
WaveNet loss function
Minimize negative log likelihood
L =
N
X
i=1
log P✓(Y i
| Xi
)
<latexit sha1_base64...

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

4. Datasets
Datasets for WaveNet TTS
• Internal Google datasets

• North American English: 24.6 hours of speech

• Mandari...

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

Training WaveNet
5. Training
• WaveNet training can be done in parallel, and
no fancy tricks were needed to train it

• What problem are they trying to solve?

• What model architecture was used?

• What loss function was used?

• What data...

WaveNet inference
6. Inference
• Primary drawback of WaveNet: although
training is parallel, inference is serial

• Follow...

Summary of WaveNet approach
• Another way of dealing with long-term dependencies is to
do away with recurrent networks alt...

Full Stack Deep Learning - UW Spring 2020
Questions?
149

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 149 Ad

View on Slideshare

1 of 149 Ad