This document discusses the key concepts of a lexical analyzer in compiler construction. It defines important terms like alphabets, strings, languages, and special symbols. It explains that a lexical analyzer scans source code character by character, using pointers to identify tokens when it encounters whitespace, operators, or other special symbols based on the longest match rule. The analyzer uses input buffering, with begin and forward pointers to track the scanned input and identify lexemes.

1. Compiler Construction
By Noor Wali Khan
UOCH
Continue
From
Previous
Lecture

2. Lexical Analyzer
• Specifications of Tokens
• Let us understand how the language theory undertakes the following terms:
• Alphabets
• Any finite set of symbols {0,1} is a set of binary alphabets,
{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F} is a set of Hexadecimal alphabets, {a-z, A-Z} is
a set of English language alphabets.
• Strings
• Any finite sequence of alphabets is called a string. Length of the string is the
total number of occurrence of alphabets, e.g., the length of the string Noor is
4 and is denoted by |Noor| = 4.
• A string having no alphabets, i.e. a string of zero length is known as an empty
string and is denoted by ε (epsilon).

3. Lexical Analyzer
• Special Symbols
• A typical high-level language contains the following symbols:

4. Lexical Analyzer
• Language
• A language is considered as a finite set of strings over some finite set of
alphabets.
• Computer languages are considered as finite sets, and mathematically set
operations can be performed on them.
• Finite languages can be described by means of regular expressions.
• Longest Match Rule
• When the lexical analyzer read the source-code, it scans the code letter by
letter; and when it encounters a whitespace, operator symbol, or special
symbols, it decides that a word is completed.

5. Lexical Analyzer
• Input Buffering
• The lexical analyzer scans the input from left to right one character at a time.
It uses two pointers begin ptr(bp) and forward to keep track of the pointer of
the input scanned.

6. Lexical Analyzer
• Input Buffering
• Initially both the pointers point to the first character of the input string as
shown below:

7. Lexical Analyzer
• The forward ptr moves ahead to search for end of lexeme.
• As soon as the blank space is encountered, it indicates
end of lexeme.
• In above example as soon as ptr (fp) encounters a blank
space the lexeme “int” is identified.
• The fp will be moved ahead at white space, when fp
encounters white space, it ignore and moves ahead. then
both the begin ptr(bp) and forward ptr(fp) are set at next
token.