This document provides an overview of sequence models and recurrent neural networks. It discusses different types of RNNs including LSTMs and GRUs. It also covers attention mechanisms, self-attention, and transformers which use self-attention to relate different positions of a sequence to compute a representation of the sequence.