Token: In computing , a token is a categorized block of text, usually consisting of indivisible characters known as lexemes . A token can look like anything: English, gibberish symbols, anything; it just needs to be a useful part of the structured text.
Delimiter: A delimiter is a character used to separate items of data stored on a computer. It is used to tell computers to finish processing one piece of data and move on to the next one. Most delimiters are characters that will not be used in the data, such as spaces or commas . It must also be consistent throughout the data.