Udacity 강좌와 여러 Reference 를 참고하여 만든 개인 공부자료 입니다. 아직 미흡한 점이 많지만 여러분들의 의견을 받아 개선해 나가도록 하겠습니다.

1. Recurrent Neural Network
RTSS JUN YOUNG PARK

2. Question
◦ How can the model train sequenced data ?
◦ Can’t understand with only a part of data.
◦ Consider previous part of data by using ‘Recurrent Neural Networks(RNN)’
널 사랑해
저도요!
사
!?

3. RNN Cell
RNN
𝑦𝑦𝑡𝑡
𝑥𝑥𝑡𝑡
𝑊𝑊ℎ𝑦𝑦
𝑊𝑊ℎℎ
◦ Previous state influences current state.
◦ For all ℎ𝑡𝑡, Weight(Params) are shared
𝑦𝑦𝑡𝑡 = 𝑊𝑊ℎ𝑦𝑦ℎ𝑡𝑡
ℎ𝑡𝑡 = 𝑓𝑓𝑊𝑊 ℎ𝑡𝑡−1, 𝑥𝑥𝑡𝑡
ℎ𝑡𝑡 = tanh 𝑊𝑊ℎℎℎ𝑡𝑡−1 + 𝑊𝑊𝑥𝑥𝑥 𝑥𝑥𝑡𝑡
𝑊𝑊ℎℎ
𝑊𝑊𝑥𝑥𝑥
INPUT
ℎ𝑡𝑡
RNN
𝑦𝑦𝑡𝑡−1
𝑥𝑥𝑡𝑡−1
𝑊𝑊ℎ𝑦𝑦
𝑊𝑊ℎℎ
𝑊𝑊𝑥𝑥𝑥
INPUT
ℎ𝑡𝑡−1
Output Layer
Hidden Layer
Input Layer 0
1
0
0
‘e’ – One hot encoded

4. Recurrent Neural Network
◦ Unsupervised Learning – Find pattern and structure from ‘unlabeled’ dataset.
RNN
𝑦𝑦1
𝑥𝑥1
RNN
𝑦𝑦2
𝑥𝑥2
RNN
𝑦𝑦𝑡𝑡−1
𝑥𝑥𝑡𝑡−1
RNN
𝑦𝑦𝑡𝑡
𝑥𝑥𝑡𝑡
…
‘H’ ‘E’ ‘L’ ‘L’
H E L O H E L O
ℎ𝑡𝑡
𝑊𝑊ℎ𝑦𝑦
𝑊𝑊𝑥𝑥𝑥
𝑊𝑊ℎℎ
ℎ1 ℎ2 ℎ𝑡𝑡−1
𝑊𝑊ℎℎ
𝑊𝑊ℎ𝑦𝑦
𝑊𝑊𝑥𝑥𝑥
𝑊𝑊ℎ𝑦𝑦𝑊𝑊ℎ𝑦𝑦
𝑊𝑊ℎℎ𝑊𝑊ℎℎ
𝑊𝑊𝑥𝑥𝑥𝑊𝑊𝑥𝑥𝑥
0
H E L O H E L O

5. ※ LSTM Cell
• Used for avoiding vanish gradient in back propagation
𝑓𝑓𝑡𝑡 ∶ 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑤𝑤𝑤𝑤𝑤𝑤𝑤 0.
𝑖𝑖𝑡𝑡 ∗ �𝐶𝐶𝑡𝑡 : 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖

6. Build RNN Model with TF

7. Build RNN
Model with TF
Make dictionary of characters for sample text.Make
Set parametersSet
Convert sample text to index listConvert
Build hidden layer (Vanilla or LSTM)Build
Run session for training and testRun

8. Make Dictionary
set(string) : return set of characters (Unordered, Non-duplicated)

9. Set Parameters
If the input has multiple strings, Modify this value

10. Convert to index list

11. Build Model
Can also use BasicRNNCell(), To make wider & deeper model, Add ‘rnn.MultiRNNCell([cell] * n, …)’
Set initial state to ‘0’
To input various length of the sequence, Add ‘sequence_length = [L_string1, … ,L_string2]’
Reshape output of RNN Cell to make fully connected layer.
Build fully connected layer

12. Build Model
1. Get sequence loss
2. Get mean of sequence loss
3. Set optimizer to minimize sequence loss

13. Run Session
Get each character element from the prediction result array
np.squeeze() : Remove 1st dimensional entries from the array.
- Ex) [1, 1, 2, 3] -> [2, 3]

14. Result
Low loss makes better sentence

15. Future plan
◦ Review lectures for deep learning (Logistic Classifier, DNN, CNN, RNN)
◦ Make image dataset arranger.
◦ Study data preprocessing.
◦ Implement deep learning models without framework (ref : 밑바닥부터 시작하는 딥러닝)