This document discusses language processing development tools (LPDTs) that focus on generating the analysis phase of language processors. It specifically describes Lexical Analyzers and the LEX tool. LEX accepts an input specification with regular expressions that define the tokens/terminals of a language. It then generates a lexical analyzer that tokenizes the input stream. The analyzer identifies lexical units, classifies them, and builds descriptor tokens containing a class code and number. It also performs semantic actions like entering identifiers and numbers into symbol tables.

1. Sardar VallabhBhai Patel Institute
Of Technology
 Name: RAJAN
SHAH
 Topic: LEX
TOOL

2. Language Processor Development
Tools
 The analysis phase of language processor has
a std form irrespective of its purpose, the
source text is subjected to lexical, syntax,
semantic analysis and the results of analysis
are represented in an IR (Intermediate
Representation).
 LPDT’s focus on generation of the analysis
phase of language processors.

3. LEXICAL ANALYZER
 Lexical analysis identifies the lexical units in the
source statement.
 It then classifies them into different lexical
classes, e.g. id’s, constants etc. and enter them
into different tables.
 Lexical analysis builds a descriptor, called a
TOKEN, for each lexical unit.
 A Token contains two fields: class code, and
number in class.
 Token is depicted as : Code#no, e.g. Id#2.

4. LEX
 Lexical analyzers tokenize input streams
 Tokens are the terminals of a language
 English: words, punctuation marks, …
 Programming language: Identifiers, operators,
keywords, …
 Regular expressions define terminals/tokens

5. Continue...
 LEX accepts an i/p specification which
consists of following two part:
1. Specification of strings representing the
lexical units in L, e.g. Id’s and constants.
2. Specification of semantic actions aimed at
building an IR. The semantic actions makes
new entries in the tables and build tokens for
the lexical units.

6. LEX Program
 %{
letter [A-Za-z]
digit [0-9]
}%
 This component defines the symbol used in
specifying the string of L. It defines the symbol
‘letter’ to stand for any upper/lower case letter.

7. Continue...
 %%
begin {return (BEGIN); }
end {return (END); }
“:=” {return (ASGOP); }
{letter} ({letter} ¦ {digit}) * {yylval=enter_id();
return (ID); }
{digit}+ {yylval=enter_num();
return (num); }
%%

8. Continue...
 enter_id()
{ /* enters the id in the symbol table and
returns the entry number. */ }
 enter_num()
{ /* enters the number in the constants table
and returns entry number */ }

9. Overview of LEX
 Lexical Analyzer lex.yy.c
 lex.yy.c a.out
 Input tokens
Lex
C compiler
a.out

10. LEX with YACC

11. THANKY
OU