LEX is a tool that allows users to specify a lexical analyzer by defining patterns for tokens using regular expressions. The LEX compiler transforms these patterns into a transition diagram and generates C code. It takes a LEX source program as input, compiles it to produce lex.yy.c, which is then compiled with a C compiler to generate an executable that takes an input stream and returns a sequence of tokens. LEX programs have declarations, translation rules that map patterns to actions, and optional auxiliary functions. The actions are fragments of C code that execute when a pattern is matched.

1. V.Anusuya,AP(SG)/CSE
1/21/2020 V.Anusuya,AP(SG)/CSE 1

2. LEX
 LEX is a tool that allows one to specify a Lexical
Analyzer by specifying RE to describe patterns for
tokens.
 Input Notation-Lex language(Specification)
 Lex Compiler-Transforms Input patterns into a
Transition diagram and generates code in a file called
lex.yy.c
1/21/2020 V.Anusuya,AP(SG)/CSE 2

3. Lexical Analyzer Generator - Lex
Lexical Compiler
Lex Source program
lex.l
lex.yy.c
C
compiler
lex.yy.c a.out
a.outInput stream Sequence
of tokens
1/21/2020 V.Anusuya,AP(SG)/CSE 3

4. Structure of Lex programs
Lex program has the following form:
declarations
%%
translation rules
%%
auxiliary functions
Pattern {Action}
1/21/2020 V.Anusuya,AP(SG)/CSE 4

5.  The declarations section includes declarations of
variables, manifest constants (identifiers declared to
stand for a constant, e.g., the name of a token), and
regular definitions.
 The translation rules each have the form
 Pattern { Action }
 pattern is a regular expression
 Action-Fragment of code written in C.
 Third Section-holds whatever additional functions are used in the
actions.
 Alternatively, these functions can be compiled separately and loaded
with the lexical analyser.
1/21/2020 V.Anusuya,AP(SG)/CSE 5

6.  The lexical analyzer returns a single value, the token
name, to the parser, but uses the shared, integer
variable yylval to pass additional information about
the lexeme found, if needed.
1/21/2020 V.Anusuya,AP(SG)/CSE 6

7. Sample Lex Program:
digit [0-9]
letter [a-zA-Z]
%%
{letter}({letter}|{digit})* printf(“id: %sn”, yytext);
n printf(“new linen”);
%%
main() {
yylex();
}
1/21/2020 V.Anusuya,AP(SG)/CSE 7

8.  Lex Predefined Variables
 yytext -- a string containing the lexeme
 yyleng -- the length of the lexeme
 yyin -- the input stream pointer
 yyout -- the output stream pointer
1/21/2020 V.Anusuya,AP(SG)/CSE 8

9.  Lex Library Routines
 yylex() - The default main() contains a call of yylex()
 yymore() - return the next token
 yyless(n) - retain the first n characters in yytext
 yywarp()-is called whenever Lex reaches an end-of-
file.By default yywarp() always returns 1
1/21/2020 V.Anusuya,AP(SG)/CSE 9

10. Program: LEX Specification
%%
{letter}({letter}|{digit})* printf("id: %sn", yytext);
n printf("new linen");
%%
int main(int argc,char **argv)
{
if (argc > 1)
{
FILE *file;
file = fopen(argv[1],"r");
if(!file)
{
printf("could not open %s n",argv[1]);
exit(0);
}
yyin = file;
}
1/21/2020 V.Anusuya,AP(SG)/CSE 10

11. yylex();
fclose(yyin);
return 0;
}
int yywrap()
{
return 1;
}
1/21/2020 V.Anusuya,AP(SG)/CSE 11

12. Input File
Sam.c
void main()
{
int a=5,b;
char c[]="GOOD";
b=a+10;
printf("%d",b);
}
1/21/2020 V.Anusuya,AP(SG)/CSE 12

13. Output
id: c
[]="id: GOOD
";new line
id: b
=id: a
+10;new line
id: printf
("%id: d
",id: b
);new line
}new line
new line
1/21/2020 V.Anusuya,AP(SG)/CSE 13

14.  Conflict Resolution in Lex
 There are two rules that Lex uses to decide on the
proper lexeme to select, when several prefixes of the
input match one or more patterns:
 1. Always prefer a longer prefix to a shorter prefix.
 2. If the longest possible prefix matches two or more
patterns, prefer the pattern listed first in the Lex
program.
1/21/2020 V.Anusuya,AP(SG)/CSE 14

15. The Lookahead Operator
 Lex automatically reads one character ahead of the last
character that forms the selected lexeme, and then
retracts the input so only the lexeme itself is consumed
from the input.
 However, sometimes, we want a certain pattern to be
matched to the input only when it is followed by a certain
other characters. If so, we may use the slash in a pattern
to indicate the end of the part of the pattern that
matches the lexeme.
 What follows / is additional pattern that must be
matched before we can decide that the token in question
was seen, but what matches this second pattern is not
part of the lexeme.
1/21/2020 V.Anusuya,AP(SG)/CSE 15

16. %{
/* program to recognize a c program */
int COMMENT=0;
%}
identifier [a-zA-Z][a-zA-Z0-9]*
%%
#.* { printf("n%s is a PREPROCESSOR
DIRECTIVE",yytext);}
int | float | char |double |while |for |do |if |break |
continue |void | switch |case |long |struct |const |
typedef | return |else | goto {printf("nt%s is a
KEYWORD",yytext);}
"/*" {COMMENT = 1;}
"*/" {COMMENT = 0;}
1/21/2020 V.Anusuya,AP(SG)/CSE 16

17. {identifier}(
{if(!COMMENT)printf("nnFUNCTIONnt%s",yytext);
}
"{" {if(!COMMENT) printf("n BLOCK BEGINS");}
"}" {if(!COMMENT) printf("n BLOCK ENDS");}
{identifier}([[0-9]*])? {if(!COMMENT) printf("n %s
IDENTIFIER",yytext);}
"a"|"n"|"b"|"t"|"t"|"b"|"a" printf("n%stis
Escape sequences",yytext);
""%d""|"%s"|"%c"|"%f"|"%e" printf("n%stis a format
specifier",yytext);
".*" {if(!COMMENT) printf("nt%s is a
STRING",yytext);}
1/21/2020 V.Anusuya,AP(SG)/CSE 17

18. [0-9]+ {if(!COMMENT) printf("nt%s is a
NUMBER",yytext);}
"=" {if(!COMMENT)printf("nt%s is an ASSIGNMENT
OPERATOR",yytext);}
"+"|"-"|"*"|"/"|"%" {printf("n %s is an arithmetic
operator",yytext);}
"<="|">="|"<"|"=="|">" {if(!COMMENT) printf("nt%s is
a RELATIONAL OPERATOR",yytext);}
%%
1/21/2020 V.Anusuya,AP(SG)/CSE 18

19. int main(int argc,char **argv)
{
if (argc > 1)
{
FILE *file;
file = fopen(argv[1],"r");
if(!file)
{
printf("could not open %s n",argv[1]);
exit(0);
}
1/21/2020 V.Anusuya,AP(SG)/CSE 19

20. yyin = file;
}
yylex();
fclose(yyin);
return 0;
}
int yywrap()
{
return 1;
}
1/21/2020 V.Anusuya,AP(SG)/CSE 20