RNN, LSTM code run

1. Min-Seo Kim
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: kms39273@naver.com

2. 1
Previous work
• A deep learning structure designed for analyzing sequential data.
• O(2) reflects both past and current information using h(1) and x(2).
• U: Input layer to hidden layer.
• W: Hidden layer at time t to hidden layer at time t+1.
• V: Hidden layer to output layer.
Recurrent Neural Networks (RNN)

3. 2
Previous work
• RNNs are models that are flexible in terms of the length of input and output values, allowing for the
construction of RNNs in various structures depending on the form of the input and output.
Recurrent Neural Networks (RNN)

4. 3
Previous work
Recurrent Neural Networks (RNN)

5. 4
Previous work
• As the time steps in a vanilla RNN increase, there arises a problem of long-term dependencies, where
information from earlier time steps is not sufficiently transmitted to later stages.
• If important information for prediction is located at the beginning, it becomes impossible to predict effectively.
• Example 1: "I grew up in France and want to be a plumber who is the best in the world and I speak
fluent French."
Long Short-Term Memory (LSTM)

6. 5
Previous work
Long Short-Term Memory (LSTM)
• The core idea of LSTM is to store the information from previous steps in
a memory cell and pass it forward.
• It determines how much of the past information to forget based on the
current information, multiplies it accordingly, and then adds the current
information to this result to pass it on to the next time step.

7. 6
Baseline
torch.nn.RNN

8. 7
Baseline
torch.nn.LSTM

9. 8
Datasets / Code
Import & gpu setting
• Set gpu to learning RNN, LSTM

10. 9
Datasets / Code
Set params
• Set parameters to fit MNIST dataset

11. 10
Datasets / Code
Get data
• Download MNIST dataset

12. 11
Datasets / Code
Net
• Rnn, LSTM implementation

13. 12
Datasets / Code
Train setting
• Train setting

14. 13
Datasets / Code
Train setting
• Train setting

15. 14
Previous
training

16. 15
Methodology
Softmax Function
Applying Softmax in PyTorch
• Converts k-dimensional input vectors into probabilities for each class
• Sum of the output probabilities equals 1

17. 16
Methodology
Log Softmax
Applying Log Softmax in PyTorch
• Applies log to the softmax result
• Addresses vanishing gradients problem and improves numerical stability

18. 17
Methodology
NLLLoss (Negative Log Likelihood Loss)
Cross Entropy Loss
• Calculates the loss for the log softmax output
• Encourages the model to assign high probabilities to the correct labels
• nn.LogSoftmax + nn.NLLLoss = nn.CrossEntropyLoss
• Directly inputs logits, internally computes softmax and NLLLoss