Theory of Computation: Lecture 39

CS 5000: Theory of Computation Lecture 39 Vladimir Kulyukin Department of Computer Science Utah State Universitywww.youtube.com/vkedco

Outline ● Review ● Stages of Compilation ● Syntactic Analysis ● Tokenization ● Parsingwww.youtube.com/vkedco

Review: CFL Membership & CYK Algorithm ● Problem: Given a CFG G = (V, T, P, S) and a string x in T*, determine if x is in L(G)? ● Cocke-Younger-Kasami (CYK) algorithm takes a CFG in CNF and a string and returns true or false, depending on whether x is or is not in L(G) ● The CYK algorithm runs in O(n3), where |x|=nwww.youtube.com/vkedco

CYK Algorithm: Basic Insight A * xij iff A  BC, A B * xik, and C * x(i+k)(j-k), for B C some k, 1 ≤ k < j i i+k-1 i+k i+j j-k k Xijwww.youtube.com/vkedco

Three Stages of Compilation ● Syntactic Analysis: The source program is processed to determine its conformity to the language grammar and its structure ● Contextual Analysis: The output of the syntactic analysis (a parse tree) is checked for its conformity to the language’s contextual constraints ● Code Generation: The checked parse tree is used to generate the target code, e.g. Java byte code or assembly or some other target languagewww.youtube.com/vkedco

Syntactic Analysiswww.youtube.com/vkedco

Components of Syntactic Analsysis ● Syntactic Analysis consists of Tokenization and Parsing ● Tokenization – We have to define a set of FA’s (regular expressions) to tokenize input statements (primitive instructions) ● Parsing – We have to define a CFG to map tokenized input statements (primitive instructions) into parse trees.www.youtube.com/vkedco

Tokenizationwww.youtube.com/vkedco

Two Basic Design Principles ● Zero Token Ambiguity: Each sequence of non-white- space characters must be mapped to at most one token ● Zero Statement (Instruction) Ambiguity: Each sequence of tokens recognized in between the beginning of a line and a newline character must have at most one parse treewww.youtube.com/vkedco

Tokenization of Programming Language Lwww.youtube.com/vkedco

Tokenization: Input Variables (InputVarToken) ● Input variables are tokens of the form X1, X2, X3, etc. In general, an input variable is Xk, where k is a natural number greater than 0. An NFA is as follows: [0 – 9] X [1 – 9]www.youtube.com/vkedco

Tokenization: Output Variables (OutputVarToken) ● L has only one output variable: Y. Here is an NFA: Ywww.youtube.com/vkedco

Tokenization: Local Variables (LocalVarToken) ● Local variables are tokens of the form Z1, Z2, Z3, etc. In general, a local variable is Zk, where k is a natural number greater than 0. An NFA is as follows: [0 – 9] Z [1 – 9]www.youtube.com/vkedco

Tokenization: Labels ● There are two places where a label can occur in a primitive instruction: at the beginning of a line and at the end of a line ● At the beginning of a line a label is bracketed; at the end of a line it is not ● Furthermore, labels that start with A, B, C, D are non-exit labels; labels that start with E are exit labelswww.youtube.com/vkedco

Tokenization: Non-Exit Non-Bracketed Labels (NELblToken) ● Non-exit labels that occur at the end of a line are tokens of the form Λ1, Λ 2, Λ3, etc. In general, a label is Λk, where k is a natural number greater than 0 and Λ is in {A, B, C, D}. An NFA is as follows: [0 – 9] A,B,C,D [1 – 9]www.youtube.com/vkedco

Tokenization: Non-Exit Bracketed Labels (NEBrLblToken) ● Non-exit labels that occur at the end of a line are tokens of the form [Λ1], [Λ2], [Λ3], etc. In general, a label is [Λk], where k is a natural number greater than 0 and Λ is in {A, B, C, D}. An NFA is as follows: [0 – 9] [ A,B,C,D [1 – 9] ]www.youtube.com/vkedco

Tokenization: Exit Non-Bracketed Label (ELblToken) ● Every L program has a unique exit label (E1). If the exit label occurs at the end of a line, it is not bracketed. An NFA is as follows: E 1www.youtube.com/vkedco

Tokenization: Exit Bracketed Label (EBrLblToken) ● Every L program has a unique exit label (E1). If the exit label occurs at the beginning of a line is it bracketed. An NFA is as follows: [ E 1 ]www.youtube.com/vkedco

Tokenization: Operators ● There are four operator tokens in L: <=, +, -, != . Here is possible NFAs for operators: < = AssignOperToken ! = NotEqOperToken + PlusOperToken - MinusOperTokenwww.youtube.com/vkedco

Tokenization: Keywords ● L has two keywords: IF and GOTO. Two possible NFAs: I F IFToken G O T O GOTOTokenwww.youtube.com/vkedco

Tokenization: Literals ● L has 2 literals: 0 and 1. Two possible NFAs: 0 ZeroLitToken 1 OneLitTokenwww.youtube.com/vkedco

Complete List of Tokens 1. InputVarToken 2. OutputVarToken 3. LocalVarToken 4. NELblToken 5. ELblToken 6. NEBrLblToken 7. EBrLblToken 8. AssignOperToken 9. NotEqOperToken 10. PlusOperToken 11. MinusOperToken 12. IFToken 13. GOTOToken 14. ZeroLitToken 15. OneLitTokenwww.youtube.com/vkedco

Tokenization Algorithm: Outline ● Read in a line of text ● Partition the line into substrings on white space ● Run each substring through all possible NFAs ● Each substring can be recognized by at most one NFA ● If a substring is not recognized by an NFA, report an error; otherwise, create an appropriate token, depending on what NFA recognized the substring ● The output is a sequence of tokenswww.youtube.com/vkedco

Tokenization Algorithm: Details ● Activate all Lazy NFAs for token recognition ● Read the file character by character; when a non-white-space character is read, go into the token recognition mode ● In the token recognition mode, when a character is read, feed it to every NFA so that all NFAs that recognize it make their transitions; if no NFA can transition, fail ● When a white-space character is read, switch off the token recognition mode and check if any NFAs accepted the sequence of non-white space characters – if yes, construct the appropriate token and reset each NFA back to its start state – If none of the NFAs accepted or more than one accepted, failwww.youtube.com/vkedco

Tokenization Example ● Let us tokenize the following L program: [A1] X1 <= X1 – 1 Y <= Y + 1 IF X1 != 0 GOTO A1www.youtube.com/vkedco

Tokenization Example: Line 1 ● [A1] X1 <= X1 – 1 ● White space partitioning gives us the following substrings: “[A1]”, “X1”, “<=“, “X1”, “-”, “1” ● “[A1]” is recognized by the Non-Exit Bracketed Label NFA; so create NEBrLblToken(“A1”) ● “X1” is recognized by the Input Variable NFA; so create InputVarToken(“X1”) ● “<=“ is recognized by the Assignment Operator NFA; so create AssignOperToken(“<=“) ● “X1” is recognized by the InputVariable NFA; so create InputVarToken(“X1”) ● “-” is recognized by the Minus Operator NFA; so create MinusOperToken(“-”) ● “1” is recognized by the One Literal NFA; so create OneLitToken(“1”) ● The output is: – <NEBrLblToken(“A1”), InputVarToken(“X1”), AssignOperToken(“<=“), InputVarToken(“X1”), MinusOperToken(“-”), OneLitToken(“1”)>www.youtube.com/vkedco

Tokenization Example: Line 1 ● The line [A1] X1 <= X1 – 1 gives us the following sequences of tokens: NEBrLblToken InputVarToken AssigOperToken InputVarToken MinusOperToken OneLitToken “A1” “X1” “<=“ “X1” “-” “1”www.youtube.com/vkedco

Tokenization Example: Line 2 ● The line Y <= Y + 1 gives us the following sequences of tokens: OutputVarToken AssigOperToken OutputVarToken PlusOperToken OneLitToken “Y” “<=“ “Y” “+” “1”www.youtube.com/vkedco

Tokenization Example: Line 3 ● The line IF X1 != 0 GOTO A1 gives us the following sequences of tokens: IFToken InputVarToken NotEqOperToken ZeroLitToken GOTOToken NELblToken “IF” “X1” “!=“ “0” “” “A1”www.youtube.com/vkedco

Parsingwww.youtube.com/vkedco

Recursive Descent Parsing ● Recursive Descent Parsing is an algorithm that should be considered for any unambiguous CF grammar ● All programming languages are specified either with unambiguous CF grammars or with ambiguous CF grammars where ambiguity can be easily handled ● The basic step in designing an RDP parser is to design a parsing procedure parseN for every non-terminal symbol N in the grammarwww.youtube.com/vkedco

Parsing Programming Language Lwww.youtube.com/vkedco

Developing Recursive-Descent Parser for L ● To develop a recursive-descent parser for L we need to accomplish three tasks: – Develop a CFG G for L – Derive a set of RD parsing procedures from G – Implement the rules in a programming language (Java, C/C+ +, C#, Structured COBOL , etc.)www.youtube.com/vkedco

A CFG Grammar for Lwww.youtube.com/vkedco

A CFG Grammar for L • LInstruct  LblStmnt | Stmnt – L instruction can be a labeled statement or an unlabeled statement • LblStmnt  BrLBL Stmnt – A labeled statement consists of a bracketed label followed by a statement • BrLBL  NEBrLblToken | EBrLblToken – A bracketed label is either a non-exit bracketed label token or an exit bracketed label token • Stmnt  Incrmnt | Decrmnt | NOP | CDisp – A statement can be a increment statement, a decrement statement, a no-op statement, and a conditional dispatch statementwww.youtube.com/vkedco

A CFG Grammar for L ● Incrmnt  VarToken AssignOperToken VarToken PlusOperToken OneLitToken – Note: this rule is simplified, because, technically speaking, VarToken is not present in the list of tokens. So, we have to write additional productions of the form: VarToken  InputVarToken | OutputVarToken | LocalVarToken ● Decrmnt  VarToken AssignOperToken VarToken MinusOperToken OneLitToken ● NOP  VarToken AssignOperToken VarToken ● CDisp  IFToken VarToken NotEqOperToken ZeroLitToken GOTOToken DispLBL ● DispLBL  NELblToken | ELblTokenwww.youtube.com/vkedco

Top-Level CFG Productions ● LProgram  LInstructSEQ – To recognize a L Program is to recognize a sequence of L instructions ● LInstructSEQ  ε – A sequence of L instructions can be empty ● LInstructSEQ  LInstruct LInstructSEQ – A non-empty sequence of L instructions starts with an L instructions and is followed by a sequence of L instructionswww.youtube.com/vkedco

Recursive-Descent Parsing Procedureswww.youtube.com/vkedco

Parsing Procedures for L ● Let us agree that each parsing procedure returns a ParseTree data structure (the base class) ● Consider the first rule in our grammar: LProgram  LInstructSEQ ● ParseTree parseLProgram(input, start_pos) { ParseTree progTree = parseLInstructSEQ(input, start_pos); return progTree; }www.youtube.com/vkedco

Implementation Notes ● ParseTree can be implemented as a base class ● All other parse trees corresponding to each non-terminal, e.g. LProgram, LInstructSEQ, LInstruct, etc., can be implemented as derived classes (sub-classes of ParseTree)www.youtube.com/vkedco

ParseLinstructSEQ Procedure ● There are 2 productions: LInstructSEQ  ε | LInstructSEQ  LInstruct | LInstructSEQ ● ParseTree parseLInstructSEQ(input, start_pos) { if ( input is empty ) return the empty LInstructSEQ; else { ParseTree firstIns = parseLInstruct(input, start_pos); ParseTree restInstructs = parseLInstructSEQ(input, firstIns.getNextPos()); return new LInstructSEQ(firstInstruct, restInstructs); } }www.youtube.com/vkedco

ParseLInstruct Procedure ● Two productions for LInstruct: LInstruct  LblStmnt | Stmnt ● ParseTree parseLInstruct(input, start_pos) { ParseTree lblSt = parseLblStmnt(input, start_pos); if ( lblSt == null ) return parseStmnt(input, start_pos); else return lblSt; }www.youtube.com/vkedco

ParseLblStmnt Procedure ● G has one production for LblStmnt: LblStmnt  BrLBL Stmnt ● ParseTree parseLblStmnt(input, start_pos) { ParseTree brLbl = parseBrLbl(inut, start_pos); if ( brLbl == null ) return null; else { ParseTree stmnt = parseStmnt(input, brLbl.getNextPos(); if ( stmnt == null ) return null; else return new LblStmnt(brLbl, stmnt); }www.youtube.com/vkedco

ParseLbl Procedure ● G has two productions for BrLbl: BrLBL  NEBrLblToken | EBrLblToken ● Note that both right-hand sides consist of tokens Remember that tokens are terminals to the parser ● So, in this case, instead of parsing we have to make sure that these terminals are in the inputwww.youtube.com/vkedco

ParseLbl Procedure ParseTree parseLbl(input, start_pos) { if (input[start_pos] == NEBrLblToken ) return new Lbl(input[start_pos]); else if (input[start_pos] == EBrLblToken) return new Lbl(input[start_pos]); else return null; }www.youtube.com/vkedco

ParseIncrmnt Procedure ● The rest of the parsing procedures can be derived in a similar fashion ● There is one rule for Incrmnt: Incrmnt  VarToken AssignOperToken VarToken PlusOperToken OneLitToken ● This rule does not require any parsing; it requires only matching of tokenswww.youtube.com/vkedco

ParseIncrmnt Procedure ParseTree parseIncrmnt(input, start_pos) { if ( input[start_pos] != VarToken ) return null; else if ( input[start_pos+1] != AssignOperToken ) return null; else if ( input[start_pos+2] != VarToken) return null; else if ( input[start_pos+3] != PlusOperToken) return null; else if ( input[start_pos+4] != OneLitToken) return null; else return new Incrmnt(VarToken, AssignOperToken, VarToken, PlusOperToken, OneLitToken); }www.youtube.com/vkedco

Parsing Example ● Let us parse the following L program: [A1] X1 <= X1 – 1 Y <= Y + 1 IF X1 != 0 GOTO A1www.youtube.com/vkedco

Parsing Example: Line 1 Tokenized ● The line [A1] X1 <= X1 – 1 gives us the following sequences of tokens: NEBrLblToken InputVarToken AssigOperToken InputVarToken MinusOperToken OneLitToken “A1” “X1” “<=“ “X1” “-” “1”www.youtube.com/vkedco

Parsing Example: Line 1 ParseTree LInstruct LblStmnt BrLbl Stmnt Decmnt NEBrLblToken InputVarToken AssignOperToken InputVarToken MinusOperToken OneLitToken “[A1]” “X1” “<=“ “X1” “-” “1”www.youtube.com/vkedco

Parsing Example: Line 2 Tokenized ● The line Y <= Y + 1 gives us the following sequences of tokens: OutputVarToken AssigOperToken OutputVarToken PlusOperToken OneLitToken “Y” “<=“ “Y” “+” “1”www.youtube.com/vkedco

Parsing Example: Line 2 ParseTree LInstruct Stmnt Incmnt OutputVarToken AssignOperToken OutputVarToken PlusOperToken OneLitToken “Y” “<=“ “Y” “+” “1”www.youtube.com/vkedco

Parsing Example: Line 3 Tokenized ● The line IF X1 != 0 GOTO A1 gives us the following sequences of tokens: IFToken InputVarToken NotEqOperToken ZeroLitToken GOTOToken NELblToken “IF” “X1” “!=“ “0” “GOTO” “A1”www.youtube.com/vkedco

Parsing Example: Line 3 ParseTree LInstruct Stmnt CDisp IFToken InputVarToken NotEqOperToken ZeroLitToken GOTOToken NELblToken “IF” “X1“ “!=” “0” “GOTO” “A1”www.youtube.com/vkedco

Parsing Example: LProgram ParseTree LProgram LInstructSEQ LInstruct LInstruct LInstruct “[A1] X1 <= X1 – 1” “Y <= Y + 1” “IF X1 != 0 GOTO A1”www.youtube.com/vkedco

References & Reading Suggestions ● Hopcroft and Ullman. Introduction to Automata Theory, Languages, and Computation, Narosa Publishing House ● Moll, Arbib, and Kfoury. An Introduction to Formal Language Theory ● Davis, Weyuker, Sigal. Computability, Complexity, and Languages, 2nd Edition, Academic Press ● Brooks Webber. Formal Language: A Practical Introduction, Franklin, Beedle & Associates, Incwww.youtube.com/vkedco

Feedback Errors, comments to vladimir dot kulyukin at gmail dot comwww.youtube.com/vkedco

http://vkedco.blogspot.com	203
http://www.vkedco.blogspot.com	43
http://vkedco.blogspot.in	10
http://vkedco.blogspot.com.au	3
http://vkedco.blogspot.co.uk	2
http://translate.googleusercontent.com	1
http://vkedco.blogspot.com.br	1
http://www.vkedco.blogspot.in	1
http://vkedco.blogspot.fr	1
http://vkedco.blogspot.jp	1
http://vkedco.blogspot.it	1

Theory of Computation: Lecture 39

by Vladimir Kulyukin on Dec 09, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

11 Embeds 267

Statistics

Theory of Computation: Lecture 39 — Presentation Transcript