Lexical analyzer


Published on

This Presentation is about the Topic Lexical Analyzer a phase in Structure of compiler (Phases of Compiler).

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Lexical analyzer

  1. 1. Topics: Lexical Analyzer  Specification of tokens  Recognition of Tokens  Data structures Involved in Lexical Analysis. 
  2. 2. Lexical Analysis It involves………  The Role of the Lexical Analyzer  Tokens , Patterns, and Lexeme  Attributes for Tokens
  3. 3. Role of Lexical Analyzer  As the first phase of a compiler, the main task of the lexical analyzer is to read the input characters of the source program, group them into lexemes, and produce as output a sequence of tokens for each lexeme in the source program.
  4. 4.  The stream of tokens is sent to the parser for syntax analysis. It is common for the lexical analyzer to interact with the symbol table as well. When the lexical analyzer discovers a lexeme constituting an identifier, it needs to enter that lexeme into the symbol table
  5. 5. The call, suggested by the getNextToken command, causes the lexical analyzer to read characters from its input until it can identify the next lexeme and produce for it the next token, which it returns to the parser.
  6. 6. Sometimes, lexical analyzers are divided into a cascade of two processes:  a) Scanning consists of the simple processes that do not require tokenization of the input, such as deletion of comments and compaction of consecutive whitespace characters into one.  b) Lexical analysis proper is the more complex portion, where the scanner produces the sequence of tokens as output.
  7. 7. Token : When discussing lexical analysis, we use three related but distinct terms:  A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit,  e.g., a particular keyword, or a sequence of input characters denoting an identifier. The token names are the input symbols that the parser processes. In what follows, we shall generally write the name of a token in boldface. We will often refer to a token by its token name.
  8. 8. Patterns  A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is a more complex structure that is matched by many strings.
  9. 9. One token for each keyword. The pattern for a keyword is the same as the keyword itself. 2. One token representing all identifiers. 3. One or more tokens representing constants, such as numbers and literal strings . 4. Tokens for each punctuation symbol, such as left and right parentheses, comma, and semicolon. 1.
  10. 10. Lexemes:  A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.
  11. 11. Strings and Languages .  Operations on Languages  Regular Expressions . . .  Regular Definitions . . . .  Extensions of Regular Expressions 
  12. 12. Specification of Tokens   Regular expressions are an important notation for specifying lexeme patterns. While they cannot express all possible patterns those types of patterns that we actually need for tokens.
  13. 13. expressions to automata that perform the recognition of the specified token patterns, they are very effective in specifying those types of patterns that we actually need for tokens. we shall see how these expressions are used in a lexical-analyzer generator.  how to build the lexical analyzer by converting regular expressions to automata that perform the recognition of the specified tokens. 
  14. 14. Strings and Languages An alphabet is any finite set of symbols. Typical examples of symbols are letters, digits, and punctuation. The set {a, 1 } is the binary alphabet. ASCII is an important example of an alphabet ; it is used in many software systems.Uni-code, which includes approximately 100,000 characters from alphabets around the world, is another important example of an alphabet.
  15. 15. A string over an alphabet is a finite sequence of symbols drawn from that alphabet. In language theory, the terms "sentence" and "word" are often used as synonyms for "string."  The length of a string s, usually written | s| , is the number of occurrences of symbols in s. For example, banana is a string of length six.  The empty string, denoted t, is the string of length zero.  A language is any countable set of strings over some fixed alphabet. 
  16. 16. Terms for Parts of Strings  A prefix of string S is any string obtained by removing zero or more symbols from the end of s. For example, ban, banana, and E are prefixes of banana.  A suffix of string s is any string obtained by removing zero or more symbols from the beginning of s. For example, nana, banana, and E are suffixes of banana.
  17. 17. A substring of s is obtained by deleting any prefix and any suffix from s. For instance, banana, nan, and E are substrings of banana.  The proper prefixes, suffixes, and substrings of a string s are those, prefixes, suffixes, and substrings, respectively, of s that are not E or not equal to s itself.  A subsequence of s is any string formed by deleting zero or more not necessarily consecutive positions of s. For example, baan is a subsequence of banana. 
  18. 18. If x and y are strings, then the concatenation of x and y, denoted xy, is the string formed by appending y to x. For example, if x = dog and y = house, then xy = doghouse. The empty string is the identity under concatenation;  This definition is very broad. Abstract languages like 0, the empty set, or {t} , the set containing only the empty string, are languages under this definition. 
  19. 19. Operations on Languages In lexical analysis, the most important operations on languages are union, concatenation, and closure, which are defined formally. Union is the familiar operation on sets. The concatenation of languages is all strings formed by taking a string from the first language and a string from the second language, in all possible ways , and concatenating them.
  20. 20.  The (Kleene) closure of a language L, denoted L*, is the set of strings you get by concatenating L zero or more times. Note that L0 , the "concatenation of L zero times," is defined to be {ɛ} , and inductively, L ͥ is L ͥ ˉ ˡ L. Finally, the positive closure, denoted L + ,is the same as the Kleene closure, but without the term L0 .  That is, ɛ will not be in L + unless it is in L itself. 
  21. 21. Regular Language  The set of regular languages over an alphabet is defined recursively as below. Any language belonging to this set is a regular language over.
  22. 22. Definition of Set of Regular Languages :  Basis Clause: Ø, {ʌ} and {a} for any symbol a ϵ are regular languages.  Inductive Clause: If Lr and Ls are regular languages, then Lr Ls , LrLs and Lr* are regular languages.  Extremal Clause: Nothing is a regular language unless it is obtained from the above two clauses.
  23. 23. Regular expression  Regular expressions are used to denote regular languages. They can represent regular languages and operations on them succinctly. The set of regular expressions over an alphabet ∑ is defined recursively as below. Any element of that set is a regular expression.  Basis Clause: Ø, ʌ , and a are regular expressions corresponding to languages Ø , {ʌ} and {a}, respectively, where a is an element of
  24. 24.  Inductive Clause:  If r and s are regular expressions corresponding to languages Lr and Ls , then ( r + s ) , ( rs ) and ( r*) are regular expressions corresponding to languages Lr Ls , LrLs and Lr* , respectively.  Extremal Clause:  Nothing is a regular expression unless it is obtained from the above two clauses.
  25. 25. Rules    is a regular expression that denotes { }, the set containing empty string. If a is a symbol in , then a is a regular expression that denotes {a}, the set containing the string a. Suppose r and s are regular expressions denoting the language L(r) and L(s), then  (r) |(s) is a regular expression denoting L(r) L(s).  (r)(s) is regular expression denoting L (r) L(s).  (r) * is a regular expression denoting (L (r) )*.  (r) is a regular expression denoting L (r).
  26. 26. Example of Regular Expressions  Example: Let = { a, b }  The regular expression a|b denotes the set {a, b}  The regular expression (a|b)(a|b) denotes {aa, ab, ba, bb} the set of all strings of a’s and b’s of length two. Another regular expression for this same set is aa| ab| ba| bb.  The regular language a* denotes that the set of all strings of zero or more a’s i.e… {ϵ,a, aa, aaa, aaaa, ……..}
  27. 27. Precedence Conventions The unary operator * has the highest precedence and is left associative.  Concatenation has the second highest precedence and is left associative.  | has the lowest precedence and is left associative.  (a)|(b)*(c) a|b*c 
  28. 28.  The regular expression (a|b)* denotes the set of all strings containing zero or more instances of an a or b, that is the set of all strings of a’s and b’s. Another regular expression for this set is (a*b*)*  The regular expression a |a*b denotes the set containing the string a and all strings consisting of zero or more a’s followed by a b.
  29. 29. Properties of Regular Expression
  30. 30. where each di is a distinct name, and each ri is a regular expression over the symbols in {d1,d2,…,di-1}, i.e.,  Note that each di can depend on all the previous d's.  Note also that each di can not depend on following d's. This is an important difference between regular definitions and productions. 
  31. 31. Regular Definitions  These look like the productions of a context free grammar we saw previously, but there are differences. Let Σ be an alphabet, then a regular definition is a sequence of definitions d1 → r1 d2 → r2 ... dn → rn
  32. 32. Example: C identifiers can be described by the following regular definition letter_ → A | B | ... | Z | a | b | ... | z | _ digit → 0 | 1 | ... | 9 CId → letter_ ( letter_ | digit)* Example: Unsigned numbers digit → 0| 1 | 2 | … | 9 digit → digit digit* optional_fractional → digit | ϵ optional_exponent → (E ( + | - | ϵ ) digit ) | ϵ num → digit optional_fraction optional_exponent
  33. 33. Extensions of Regular Expressions  Many extensions have been added to regular expressions to enhance their ability to specify string patterns. Here we mention a few notational extensions that were first incorporated into unix utilities such as Lex that are particularly useful in the specification lexical analyzers .  One or more instances. The unary, postfix operator + represents the positive closure of a regular expression and its language. That is , if r is a regular expression, then (r)+denotes the language (L( r) ) +. The operator + has the same precedence and associativity as the operator *. Two useful algebraic laws, r* = r+ I ϵ and r+ = rr* = r*r relate the Kleene closure and positive closure.
  34. 34. Zero or one instance. The unary postfix operator ? means "zero or one occurrence ." That is , r? is equivalent to r lϵ , or put another way, L(r?) = L(r) U {ϵ}. The ? operator has the same precedence and associativity as * and +. Example  digit → 0| 1 | 2 | … | 9 digit → digit digit+ optional_fractional → ( . digit ) ? optional_exponent → (E ( + | - ) ? digit ) ? num → digit optional_fraction optional_exponent
  35. 35. Character classes. A regular expression a1 la2|···| an , where the ai ' s are each symbols of the alphabet , can be replaced by the shorthand [a la2 ··· an] . More importantly, when a 1 , a2 ,.·· ,an form a logical se-quence, e.g., consecutive uppercase letters, lowercase letters, or digits, we can replace them by a I -an , that is , just the first and last separated by a hyphen. Thus, [abc] is shorthand for al b i c, and [a- z] is shorthand for al b l··· l z . Example [A-Z a-z][A-Z a-z 0-9]* 
  36. 36. Lexical analyzer  Also called a scanner or tokenizer  Converts stream of characters into a stream of tokens  Tokens are:  Keywords such as for, while, and class.  Special characters such as +, -, (, and <  Variable name occurrences  Constant occurrences such as 1, 0, true.
  37. 37. Comparison with Lexical Analysis Phase Input Output Lexer Sequence of characters Sequence of tokens Parser Sequence of tokens Parse tree
  38. 38. The role of lexical analyzer Source program token Lexical Analyzer Parser getNextToken Symbol table To semantic analysis
  39. 39. What are Tokens For?  Parser relies on token classification  e.g., How to handle reserved keywords? As an identifier or a separate keyword for each? Output of the lexer is a stream of tokens which is input to the parser.  The lexer usually discards “uninteresting” tokens that do not contribute to parsing   Examples: white space, comments
  40. 40. Recognition of Tokens Task of recognition of token in a lexical analyzer  Isolate the lexeme for the next token in the input buffer  Produce as output a pair consisting of the appropriate token and attribute-value, such as <id,pointer to table entry> , using the translation table given in the Fig in next page
  41. 41. Recognition of Tokens Task of recognition of token in a lexical analyzer Regular expression if id Token < relop if id Attributevalue Pointer to table entry LT
  42. 42. Recognition of tokens  Starting point is the language grammar to understand the tokens: stmt -> if expr then stmt | if expr then stmt else stmt |Ɛ expr -> term relop term | term term -> id | number
  43. 43. Recognition of tokens (cont.) is to formalize the patterns:  The next step digit -> [0-9] Digits -> digit+ number -> digit(.digits)? (E[+-]? Digit)? letter -> [A-Za-z_] id -> letter (letter|digit)* If -> if Then -> then Else -> else Relop -> < | > | <= | >= | = | <>  We also need to handle whitespaces: ws -> (blank | tab | newline)+ The terminals of the grammar, which are if, then, else, relop, id, and number, are the names of tokens as used by the lexical analyzer. The lexical analyzer also has the job of stripping out whitespace, by recognizing the "token" ws .
  44. 44. Tokens, their patterns, and attribute values Compiler Construction
  45. 45. LEXICAL ANALYSIS Recognition of Tokens Method to recognize the token  Use Transition Diagram
  46. 46. Diagrams for Tokens  Transition Diagrams (TD) are used to represent the tokens  Each Transition Diagram has:  States: Represented by Circles  Actions: Represented by Arrows between states  Start State: Beginning of a pattern (Arrowhead)  Final State(s): Circles)  End of pattern (Concentric Deterministic - No need to choose between 2 different actions
  47. 47. Recognition of Tokens Transition Diagram(Stylized flowchart)  Depict the actions that take place when a lexical analyzer is called by the parser to get the next token start 0 Start state > = Accepting state 6 7 other * 8 Notes: Here we use „*‟ to indicate states on which input retraction must take place return(relop,GE) return(relop,GT)
  48. 48. if it is necessary to retract the forward pointer one position ( i.e., the lexeme does not include the symbol that got us to the accepting state) , then we shall additionally place a *near that accepting state.
  49. 49. Ex :RELOP = < | <= | = | <> | > | >= We begin in state 0, the start state < as the first input symbol, then among the lexemes that match the pattern for relop we can only be looking at <, <>, or <=.
  50. 50. Recognition of Identifiers  Ex2: ID = letter(letter | digit) * Transition Diagram: letter or digit letter start 9 # other 10 11 return(id) # indicates input retraction Compiler Construction
  51. 51. Install the reserved words in the symbol table initially. A field of the symboltable entry indicates that these strings are never ordinary identi-fiers, and tells which token they represent . We have supposed that this method is in use in Fig . When we find an identifier, a call to install ID places it in the symbol table if it is not already there and returns a pointer to the symbol-table entry for the lexeme found Of course, any identifier not in the symbol table during lexical analysis cannot be a reserved word, so its token is id.
  52. 52.  So far we have seen the transition diagrams for identifiers and the relational operators.  What remains are:  Whitespace  Numbers
  53. 53. Recognizing white spaces: We also want the laxer to remove whitespace so we define a new token . { ws → ( blank | tab | newline ) + } where blank, tab, and newline are symbols used to represent the corresponding ascii characters.
  54. 54. Transition diagram for white space:
  55. 55. The “delim” in the diagram represents any of the whitespace characters, say space, tab, and newline.  The final star is there because we needed to find a non-whitespace character in order to know when the whitespace ends and this character begins the next token.  There is no action performed at the accepting state. Indeed the lexer does not return to the parser, but starts again from its beginning as it still must find the next token. 
  56. 56. Transition Diagram for Numbers : Accepting float e.g. 12.31E4 Accepting integer e.g. 12 Accepting float e.g. 12.31
  57. 57. Explaination with example :  Beginning in state 12, if we see a digit, we go to state 13. In that state, we can read any number of additional digits . However, if we see anything but a digit or a dot , we have seen a number in the form of an integer; 12 is an example . Accepting integer e.g. 12
  58. 58. If we instead see a dot in state 13, then we have an "optional fraction ."  State 14 is entered, and we look for one or more additional digits; state 15 is used for that purpose. 
  59. 59.  If we see an E, then we have an "optional exponent ," whose recognition is the job of states 16 through 19.
  60. 60. Architecture of a TransitionDiagram- Based Lexical Analyzer :  Each state is represented by a piece of code . We may imagine a variable state holding the number of the current state for a transition diagram. A switch based on the value of state takes us to code for each of the possible states, where we find the action of that state. Often, the code for a state is itself a switch statement or multi way branch that determines the next state by reading and examining the next input character.
  61. 61. Transition diagram for relop:
  62. 62. Sketch of implementation of relop transition diagram
  63. 63. getRelop ( ), a C++ function whose job is to simulate the transition diagram and return an object of type TOKEN , that is , a pair consisting of the token name (which must be relop in this case) and an attribute value .  getRelop ( ) first creates a new object ret Token and initializes its first component to RELOP , the symbolic code for token relop 
  64. 64. We see the typical behavior of a state in case 0, the case where the current state is 0. A function nextChar ( ) obtains the next character from the input and assigns it to local variable c. We then check c for the three characters we expect to find , making the state transition dictated by the transition diagram .  For example, if the next input character is =, we go to state 5. 
  65. 65.  If the next input character is not one that can begin a comparison operator, then a function fail ( ) is called, and It should reset the forward pointer to lexemeBegin.
  66. 66.  Because state 8 bears a *, we must retract the input pointer one position (i.e. , put c back on the input stream) . That task is accomplished by the function retract ( ). Since state 8 represents the recognition of lexeme >=, we set the second component of the returned object , which we suppose is named attribute, to GT, the code for this operator.
  67. 67. “The symbol table is a data structure used by all phases of the compiler to keep track of user defined symbols and keywords.”
  68. 68.  In computer science, a symbol table is a data structure used by a language translator such as a compiler or interpreter, where each identifier in a program's source code is associated with information relating to its declaration or appearance in the source, such as its type, scope level and sometimes its location.
  69. 69.  During early phases (lexical and syntax analysis) symbols are discovered and put into the symbol table.  During later phases symbols are looked up to validate their usage.
  70. 70. Uses   An object file will contain a symbol table of the identifiers it contains that are externally visible. During the linking of different object files, a linker will use these symbol tables to resolve any unresolved references. A symbol table may only exist during the translation process, or it may be embedded in the output of that process for later exploitation, for example, during an interactive debugging session, or as a resource for formatting a diagnostic report during or after execution of a program.
  71. 71.   While reverse engineering an executable, many tools refer to the symbol table to check what address has been assigned to global variables & Non-functions if the symbol table has been stripped or cleaned out before being converted into executable tools will find it harder to determining address or understand anything about the program. At that time of accessing variable & allocating memory dynamically a compiler should perform many works & as such, the extended stack model requires the symbol table.
  72. 72. Typical symbol table activities: Add a new name  Add information for a name  Access information for a name     determine if a name is present in the table remove a name revert to a previous usage for a name (close a scope).
  73. 73. Many possible Implementations Linear List  Sorted List  Hash Table  Tree Structure 
  74. 74. Typical information fields: print value  kind (e.g. reserved, type_id, var_id, func_id, etc.)  block number/level number  type  initial value  base address etc. 