The document discusses word embeddings and various RNN techniques used in sequence modeling, including GRU and LSTM architectures. It details applications such as text classification, machine translation, and speech recognition, while also addressing the limitations of RNNs like the vanishing gradient problem. The document emphasizes the importance of attention mechanisms for improving long-term dependencies in sequence data.

1. Word Embeddings and RNN
techniques

2. Contents
• Word Embeddings
• Application of Sequence modelling
• RNN
• GRU
• LSTM
• Attention Mechanism

3. Word Embeddings
• Continuous Bag of Words (CBOW)- Predict center word from (bag of)
context words
• Skip-grams- Predict context (”outside”) words (position independent)
given center word
• Glove : Co-occurrence statistics in a form of word co occurrence
matrix X

4. Applications of Sequence Modelling
• Text Classification
• Language Modeling
• Speech Recognition
• Caption Generation
• Machine Translation
• Document Summarization
• Music Generation
• Sentiment Classification
• Handwriting Recognition

5. RNN
• Sequence Modeling i.e. Exhibit dynamic temporal behavior for a time
sequence
• A recurrent neural network (RNN) is a class of artificial neural network
where connections between units form a directed graph along a
sequence.

6. Why RNN
• Traditional Method - Model is usually conditioned on window of n
words.
• Variable width inputs.
• Bidirectional RNNs
• Shared Parameters.

7. Different types of RNNs

8. General Model

9. Machine Translation Model

10. Drawbacks of RNN
• The vanishing/exploding gradient problem
• Example - Jane walked into the room. John walked in too. It was late
in the day. Jane said hi to ____
• RNNs aren't good in long term dependencies.

11. Gated Recurrent Units (GRU)
• Update gate
• Reset gate
• New memory content
• Final memory at time step combines current and previous time steps:

12. Long Short Term Memory (LSTM)
• Input gate (current cell matters)
• Forget (gate 0, forget past)
• Output (how much cell is exposed)
• New memory cell
• Final memory cell
• Final hidden state

13. Attention mechanism