This document discusses recurrent neural networks (RNNs) and their training methods. It covers the basic architecture of RNNs, including their ability to process sequential data over time. It then discusses feedforward propagation and backpropagation for training RNNs, including challenges like exploding and vanishing gradients. It introduces techniques like truncated backpropagation through time (BPTT) and mini-batches to help address these issues during training. The document provides code examples to help understand RNN concepts in practice.

1. Interaction Lab. Seoul National University of Science and Technology
핵심 딥러닝 입문
chapter 4. RNN
Jeong Jae-Yeop

2. Interaction Lab., Seoul National University of Science and Technology
■Intro
■Training method
■Code practice
■Conclusion
Agenda
2

3. Intro
Training method
3

4. Interaction Lab., Seoul National University of Science and Technology
■What is RNN?
 Reccurent Neural Network
• Sequence data
• 𝑡 : Time
Intro
4
Input Output
Hidden

5. Interaction Lab., Seoul National University of Science and Technology
■Reccurent architecture
Intro
5

6. Interaction Lab., Seoul National University of Science and Technology
■Activation function
 Hyperbolic tangent
• 𝑥𝑡 : Input
• 𝑊
𝑥 : Input weight
• 𝑏 : Bias
• ℎ𝑡−1 : Previous output
• 𝑊ℎ : Previous output weight
Intro
6

7. Training method
Code practice
7

8. Interaction Lab., Seoul National University of Science and Technology
■Feed forward propagation
 Calculate and store variables sequentially from the input layer to the output layer of the NN
■Backpropagation
 How to calculate gradients for parameters of a NN
Training method

9. Interaction Lab., Seoul National University of Science and Technology
■Feed forward propagation of RNN
 Deep Neural Network
• 𝑈 = 𝑋𝑊 + 𝐵
• 𝑌 = 𝑓(𝑈)
 RNN
• 𝑈(𝑡)
= 𝑋(𝑡)
𝑊 + 𝑌(𝑡−1)
𝑉 + 𝐵
• 𝑌(𝑡)
= 𝑓(𝑈(𝑡)
)
Training method
9

10. Interaction Lab., Seoul National University of Science and Technology
■Feed forward propagation of RNN
Training method
10
Input(t) 행렬 곱
행렬 곱
+
Activation
function Next layer
Next point
Weight
Weight
Bias
Output

11. Interaction Lab., Seoul National University of Science and Technology
■Feed forward propagation of RNN
Training method
11
𝑈(𝑡)
= 𝑥𝑡𝑊𝑥ℎ + ℎ𝑡−1𝑊ℎℎ + 𝑏ℎ

12. Interaction Lab., Seoul National University of Science and Technology
■Backpropagation of RNN
Training method
12

13. Interaction Lab., Seoul National University of Science and Technology
■Backpropagation of RNN
 We have to update parameters 𝑊𝑥ℎ, 𝑊ℎℎ, 𝑏
Training method
13
𝑑ℎ𝑡−1

14. Interaction Lab., Seoul National University of Science and Technology
■BPTT (Backpropagation Through Time)
 As the time scale of time series data increases, the computing resources consumed by
BPTT also increase
 As the time scale increases, the gradient of backpropagation becomes unstable
Training method
14

15. Interaction Lab., Seoul National University of Science and Technology
■Truncated BPTT
 Data must be entered in order
 Cut the backpropagation connection to an appropriate length
Training method
15

16. Interaction Lab., Seoul National University of Science and Technology
■Truncated BPTT using mini-batch
 Mini-batch : 2
 1,000 data : 500 / 500
Training method
16

17. Interaction Lab., Seoul National University of Science and Technology
■Binary addition
 5 = 1 × 22 + 0 × 21 + 1 × 20 ∶ 101
 36 = 1 × 25 + 0 × 24 + 0 × 23 + 0 × 22 +0 × 21 +0 × 20 ∶ 100100
 Input : two randomly selected binary numbers
 Label : sum of two numbers
 Link
Code practice
17

18. Interaction Lab., Seoul National University of Science and Technology
■Disadvantage of RNN
 Gradient vanishing and Gradient exploding
• LSTM and GRU
• Gradient clipping
Conclusion

19. Q&A
19