2. About this Section
• Preprocessing data
• Work with NLTK + word2vec
• Introduce Recurrent Neural Networks(RNN)
• Introduce Long Short-Term Memory
Networks(LSTM)
21. About this Section
• Preprocessing data
• Work with NLTK + word2vec
• Introduce Recurrent Neural Networks(RNN)
• Introduce Long Short-Term Memory
Networks(LSTM)
22. The problem with forward NN
22
已知這些關係
文字預測 James has cat and it like to drink ___ .
圖像表現
23. The problem with forward NN
23
改變這些關係
文字預測 James has cat and it like to drink ___ .
圖像表現
34. About this Section
• Preprocessing data
• Work with NLTK + word2vec
• Introduce Recurrent Neural Networks(RNN)
• Introduce Long Short-Term Memory
Networks(LSTM)
38. What is LSTM ?
38
LSTM 有五個值,四個動作,三個門
• Cell state: This is the internal cell state (that is, memory) of an LSTM cell
• Hidden state: This is the external hidden state used to calculate predictions
• Input gate: This determines how much of the current input is read into the cell state
• Forget gate: This determines how much of the previous cell state is sent into the current cell state
• Output gate: This determines how much of the cell state is output into the hidden state
表過去經驗
表預測結果
• Discard old information(that, forget something…)
• Store which new information in the cell state
• Updated the old cell state Ct-1, into the new cell state Ct
• Output information
39. How does LSTM works ?
39
• Input gate: This determines how much of the current input is read into the cell state (0~1)
• Forget gate: This determines how much of the previous cell state is sent into the current cell state (0~1
• Output gate: This determines how much of the cell state is output into the hidden state (0~1)
42. Discard old information(that, forget something…)
42
• Forget gate: This determines how much of the previous cell state is sent into the current cell state (0~
43. Store which new information in the cell state
43
• Input gate: This determines how much of the current input is read into the cell state (0~1)
44. Updated the old cell state Ct-1, into the new cell state Ct
44