The document discusses Long Short-Term Memory (LSTM) networks, which were invented to address issues with vanilla recurrent neural networks like vanishing and exploding gradients. It explains the key components of an LSTM unit: the memory pipe, forget valve, new memory valve, generation of new output, and output valve. Diagrams are included to illustrate how an LSTM unit works by controlling what information to store, forget, or output from the memory pipe over time through these valves.

1. Understanding LSTM
and
its Diagrams
Shi Yan
-
Slides written by Park JeeHyun
22 DEC 2017

2. Understanding LSTM and its diagrams
• Theoretically the naively connected neural network,
so called recurrent neural network, can work.
But in practice, it suffers from two problems:
• vanishing gradient and
• exploding gradient, which make it unusable.
• Then later, LSTM (long short term memory) was invented
to solve this issue by explicitly introducing a memory
unit, called the cell into the network.

3. Understanding LSTM and its diagrams

4. Understanding LSTM and its diagrams
• Memory pipe
• Forget valve
• New memory valve
• Generate new output
• Output valve

5. Memory pipe
• An element-wise multiplication
• if you multiply the old memory C_t-1 with a vector that is close to 0, that m
eans you want to forget most of the old memory.
• You let the old memory goes through, if your forget valve equals 1.
• A piece-wise summation
• New memory and the old memory will merge by this operation.
• How much new memory should be added to the old memory is
controlled by another valve, below.

6. Memory pipe : forget valve
• An element-wise multiplication
• if you multiply the old memory C_t-1 with a vector that is close to 0, that m
eans you want to forget most of the old memory.
• You let the old memory goes through, if your forget valve equals 1.

7. Memory pipe : new memory valve
• A piece-wise summation
• New memory and the old memory will merge by this operation.
• How much new memory should be added to the old memory is
controlled by another valve, below.

8. Generate new output : output valve
• This step has an output valve that is controlled by the new
memory, the previous output h_t-1, the input X_t and a bias
vector.
• This valve controls how much new memory should output to
the next LSTM unit.

9. Annex : LSTM diagram

10. Annex : LSTM diagram

11. References
• “Understanding LSTM and its diagrams” by Shi Yan
• https://medium.com/mlreview/understanding-lstm-and-its-
diagrams-37e2f46f1714
• “Understanding LSTM Networks” from colah's blog
• http://colah.github.io/posts/2015-08-Understanding-LSTMs/