Your SlideShare is downloading. ×
Lex tool manual
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Lex tool manual

5,240
views

Published on


1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total Views
5,240
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
180
Comments
1
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lexical Analyzer Generator Lex (Flex in recent implementation) Samy Said Mohamed Eshaish Pre-Masters student, Department of Computer Science 2012-2013 Compiler Design 2Contents:-
  • 2. 1- Lex  What is Lex  Structure of Lex Program  Lex Predefined Variables  Lex Library Routines2- Set up the Tool  Install on Windows  Install on Linux (Ubuntu)3- Running the Tool  Run on Windows  Run on Linux (Ubuntu)4- Decaf Example  Example Code  Run The Example5- References  References1. Lex What is Lex?
  • 3. • The main job of a lexical analyzer (scanner) is to break up an input stream into tokens (tokenize input streams). • Ex :- a = b + c * d; ID ASSIGN ID PLUS ID MULT ID SEMI • Lex is an utility to help you rapidly generate your scanners Structure of Lex Program • Lex source is separated into three sections by %% delimiters • The general format of Lex source is • The absolute minimum Lex program is thus  Definitions • Declarations of ordinary C variables, constants and Libraries. %{
  • 4. #include <math.h> #include <stdio.h> #include <stdlib.h> %} • flex definitions :- name definition Digit [0-9] (Regular Definition) Operators  “[]^-?.*+|()$/{}%<> • If they are to be used as text characters, an escape should be used $ = “$” = “” • Every character but blank, tab (t), newline (n) and the list above is always a text character Translation Rules • The form of rules are: Pattern { action } • The actions are C/C++ code. [0-9]+ { return(Integer); } // RE {DIGIT}+ { return(Integer); } // RD User Subroutines Section • You can use your Lex routines in the same ways you use routines in other programming languages (Create functions, identifiers).
  • 5. • The section where main() is placed Lex Predefined Variables • yytext -- a string containing the lexeme • yyleng -- the length of the lexeme • yyin -- the input stream pointer • the default input of default main() is stdin • yyout -- the output stream pointer • the default output of default main() is stdout. Lex Library Routines • yylex() – The default main() contains a call of yylex()
  • 6. • yymore() – return the next token • yyless(n) – retain the first n characters in yytext • yywarp() – is called whenever Lex reaches an end-of-file – The default yywarp() always returns 12. Set up the Tool
  • 7.  Windows 1. Download the Windows version of FLEX 2. Download a C/C++ Compiler DevCPP or Code::Blocks 3. Install all. Its recommended to install in folders WITHOUT spaces in their names. I use C: GNUWin32 for FLEX and C: DevCPP 4. Now add the BIN folders of both folders into your PATH variable. In case you dont know how to go about this, see how to do it on Windows XP ,Windows Vista and Windows 7. You will have to add ; C:GNUWin32bin;C:Dev-Cpp to the end of your PATH 5. Open up a CMD prompt and type in the following C:flex --version flex version 2.5.4 C:>gcc --version gcc (GCC) 3.4.2 (mingw-special) Copyright (C) 2004 Free Software Foundation, Inc. this is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY Or FITNESS FOR A PARTICULAR PURPOSE.
  • 8.  Linux (Ubuntu)NOTE: You have to install m4 (general purpose macro processor) before installing Flex:- • Downloading m4: Run the command below, wget ftp://ftp.gnu.org/gnu/m4/m4-1.4.10.tar.gz • Extracting files from the downloaded package: tar -xvzf m4-1.4.10.tar.gz Now, enter the directory where the package is extracted. cd m4-1.4.10 • Configuring m4: ./configure • Compiling m4: make Note: check for any error message. • Installing m4: As root (for privileges on destination directory), run the following. sudo make install Note: check for any error messages.
  • 9. • Downloading Flex (The fast lexical analyzer): Run the command below, wget http://prdownloads.sourceforge.net/flex/flex-.5.33.tar.gz?download Note: Replace 2.5.33 with your version number.• Extracting files from the downloaded package: tar -xvzf flex-2.5.33.tar.gz Now, enter the directory where the package is extracted. cd flex-2.5.33• Configuring flex before installation: PATH=$PATH: /usr/local/m4 NOTE: Replace /usr/local/m4/bin with the location of m4 installation directory. Now, configure the source code before installation. ./configure• Compiling flex: make Note: check for any error message.• Installing flex: As root (for privileges on destination directory), run the following.sudo make install
  • 10. Note: check for any error messages.3. Running the Tool Windows • Open CMD • First Go to Directory Contains Files With CD Command • To run Lex on a source file, type  flex (lex source file.l) Command • It produces a file named lex.yy.c which is a C program for the lexical analyzer. • To compile lex.yy.c, type  gcc lex.yy.c Command • To run the lexical analyzer program, type  a.exe < input file > output file Command Linux (Ubuntu) Open Terminal First Go to Directory Contains Files With CD Command To run Lex on a source file, type  flex (lex source file.l) Command It produces a file named lex.yy.c which is a C program for the lexical analyzer.
  • 11.  To compile lex.yy.c, type  gcc lex.yy.c Command To run the lexical analyzer program, type  ./a.out < input file > output file Command4. Decaf Example CODE //Definitions Section %{ /* need this for the call to atof() below */ #include <math.h> #include <stdio.h> #include <stdlib.h> %} DIGIT [0-9] LITTER [a-zA-Z] C [//] S [] L [n] ID {LITTER}({LITTER}|{DIGIT}|_)* double {DIGIT}+(.{DIGIT}+)?([Ee][+-]?{DIGIT}+)? hex (0[xX])({DIGIT}|[a-fA-F])* String ("({LITTER}|{S})*") Sym [~!@#$%^&*()_+|><""{}.,] Comment (({LITTER}|{DIGIT}|{Sym}|{S}|{L})*) Comment2 ({C}+({LITTER}|{DIGIT}|{Sym}|{S})*{L}) %% //Rule Section {DIGIT}+ { printf( "An integer: %s (%d)n", yytext,atoi( yytext ) ); }
  • 12. {double} { printf( "A double: %s (%g)n", yytext, atof( yytext ) ); }{hex} { printf( "A Hexadecimal: %s n", yytext ); }{String} { printf( "A String: %s n", yytext ); }"/*"{Comment}+"*/" /* eat up one-line comments */{Comment2} /* eat up one-line comments */void|int|double|bool|string|class|interface|null|this|extends|implements|for|while|if|else|return|break|new|NewArray|Print|ReadInteger|ReadLine|true|false { printf( "A keyword: %sn", yytext ); }{ID} { printf( "An identifier: %sn", yytext ); }"+"|"-"|"*"|"/"|"="|"=="|"<"|"<="|">"|">="|"!="|"&&"|"||"|"!"|";"|","|"."|"["|"]"|"("|")"|"{"|"}" { printf( "An operator: %sn", yytext ); }"{"[^}n]*"}" /* eat up one-line comments */[ tn]+ /* eat up whitespace */. {printf( "Unrecognized character: %sn", yytext ); }%% //User Code Sectionmain( argc, argv )int argc;char **argv; { ++argv, --argc; /* skip over program name */ if ( argc > 0 ) yyin = fopen( argv[0], "r" ); else yyin = stdin; yylex();
  • 13. } int yywrap(){ return 1; } Run the CODE • Save file with name “decaf.l” • Open CMD or Terminal then go to directory contains file with cd command • Type this command  flex decaf.l • Type this command  gcc lex.yy.c • Type this command  a.exe<source prog file .txt> output file.txt (Windows) or ./ a.out <source prog file .txt> output file.txt (Ubunto)5. References • lex & yacc 2nd_edition John R. Levine, Tony Mason, Doug Brown, O’reilly • Mastering Regular Expressions by Jeffrey E.F.Friedl O’Reilly