The document outlines preprocessing steps for building a language model, including splitting input text into sentences and words using specific tokenizers. It also specifies to discard sentences with fewer than 'n' tokens and to add start and end tokens to each sentence. These steps are critical for the preparation of data before model training.

1. programming assignment 2
Preprocessing Before building the language model, the following preprocessing steps should be
performed:
1. Input text should be split into sentences using an existing sentence boundary detector (e.g.
sentence tokenizer from NLTK, spaCy, etc.).
2. Each sentence should be split into words using an existing word tokenizer (word tokenizer
from NLTK, spaCy, etc.).
3. Discard sentences with fewer than n tokens.
4. Add start of sentence token, , and end of sentence token, to each sentence