The document discusses the lexical analysis phase of a compiler. In lexical analysis, the source code is divided into tokens. Common token types include keywords, identifiers, and special symbols. Lexical analyzers perform pattern matching and techniques used for lexical analysis can also be applied to other areas like query languages. Lex is a tool that can generate an automaton recognizer for regular expressions to specify lexical analyzers. The role of a lexical analyzer is to read input characters and produce a sequence of tokens for the parser to use in syntax analysis.

1. Subject: Automata Theory And Compiler Design
Topic: Lexical Analysis Phase Of Compiler
Guided By: Presented By:
Sindhu G Yashaswini S(1KT21IS046)
Dept of ISE.

2. INTRODUCTION:
 The scanning/lexical analysis phase of a
compiler performs the task of reading the source
program as a file of characters and dividing up
into tokens.
 Usually implemented as subroutine or co-routine of
parser.
 Front end of compiler.
 Each token is a sequence of characters that represents a
unit of information in the source program.

3. EXAMPLES OF TOKENS:
 Keywords which are fixed string of letters .eg: “if”,
“while”.
 Identifiers which are user-defined strings composed of
letters and numbers.
 Special symbols like arithmetic symbols.

4. APPLICATIONS:
 Scanners perform pattern matching process.
 The techniques used to implement lexical analyzers can
also be applied to other areas such as query languages
and information retrieval systems.
 Since pattern directed programming is widely useful,
pattern action language called Lex for specifying lexical
analyzers.
 In lex , patterns are specified by regular expressions, and
a compiler for lex can generate an efficient finite-
automaton recognizer for the regular expression

5. THE ROLE OF LEXICALANALYZER:
 Lexical analyzer is the first phase of a compiler.
 Its main task is to read input characters and produce as output a
sequence of tokens that parser uses for syntax analysis.

6.  If the lexical analyzer is located as a separate pass in the compiler it can
need an intermediate file to locate its output, from which the parser
would then takes its input.
 The lexical analyzer also interacts with the symbol table while passing
tokens to the parser. Whenever a token is discovered, the lexical
analyzer returns a representation for that token to the parser.
 If the token is a simple construct including parenthesis, comma, or a
colon, then it returns an integer program. If the token is a more complex
items including an identifier or another token with a value, the value is
also passed to the parser.

7. ISSUES IN LEXICALANALYSIS:
 There are several reasons for separating the analysis
phase of compiling into lexical analysis and parsing:
 It leads to simpler design of the parser as the unnecessary
tokens can be eliminated by scanner.
 Efficiency of the process of compilation is improved.
The lexical analysis phase is most time consuming phase
in compilation.
 buffering to improve the speed of compilation.