Compiler Components and their Generators - Traditional Parsing Algorithms

2,665 views

Published on

Presentation slides for lecture 12 of course IN4303 on Compiler Construction at TU Delft.

Published in: Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,665
On SlideShare
0
From Embeds
0
Number of Embeds
679
Actions
Shares
0
Downloads
123
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • linguistic theory\n\nNoam Chomsky\n
  • linguistic theory\n\nNoam Chomsky\n
  • linguistic theory\n\nNoam Chomsky\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • \n
  • \n
  • \n
  • \n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • restrictions on production rules => grammar classes\n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • computer science: lexical syntax\n\ncan write this as a regular grammar\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Compiler Components and their Generators - Traditional Parsing Algorithms

    1. 1. Compiler Components & GeneratorsTraditional Parsing AlgorithmsGuido Wachsmuth Delft Course IN4303 University of Technology Compiler Construction Challenge the future
    2. 2. Recap: Lexical Analysislessons learnedWhat are the formalisms to describe regular languages? • regular grammars • regular expressions • finite state automataWhy are these formalisms equivalent? • constructive proofsHow can we generate compiler tools from that? • implement DFAs • generate transition tables Traditional Parsing Algorithms 2
    3. 3. Overviewtoday’s lecture Lexical Analysis 3
    4. 4. Overviewtoday’s lecturethe exam Lexical Analysis 3
    5. 5. Overviewtoday’s lecturethe examassignment 3 Lexical Analysis 3
    6. 6. Overviewtoday’s lecturethe examassignment 3efficient parsing algorithms • predictive parsing • LR parsing Lexical Analysis 3
    7. 7. Overviewtoday’s lecturethe examassignment 3efficient parsing algorithms • predictive parsing • LR parsinggrammar classes • LL(k) grammars • LR(k) grammars Lexical Analysis 3
    8. 8. Ithe exam Lexical Analysis 4
    9. 9. The examtopics • introduction • declarative language definition (theory) • compiling imperative & object-oriented languages • compiler components & generatorsstyle • theory in praxis • understanding over facts • open questions • closed book Traditional Parsing Algorithms 5
    10. 10. IIassignment Lexical Analysis 6
    11. 11. MiniJava CompilerscheduleLab 1 explore Java Bytecode & Jasmin first generator versionLab 2 & 3 generator for statements & expressions generator for classes, fields & methodsLab 4 liveness analysissubmission due Jan 9, 23:59 Traditional Parsing Algorithms 7
    12. 12. MiniJava Compilercode generationclass Main { public static void main(String[] args) { System.out.println(42) ; }} Traditional Parsing Algorithms 8
    13. 13. Recap: Code Generationto-jbc: Nil() -> [ ACONST_NULL() ]to-jbc: NoVal() -> [ NOP() ]to-jbc: Seq(es) -> <mapconcat(to-jbc)> es to-jbc: Int(i) -> [ LDC(Int(i))to-jbc: String(s) -> [ LDC(String(s)) to-jbc: Bop(op, e1, e2) -> <concat> [ <to-jbc> e1, <to-jbc> e2, <to-jbc> op ] to-jbc: PLUS() -> [ IADD() ]to-jbc: MINUS() -> [ ISUB() ]to-jbc: MUL() -> [ IMUL() ]to-jbc: DIV() -> [ IDIV() ]to-jbc: Assign(lhs, e) -> <concat> [ <to-jbc> e, <lhs-to-jbc> lhs ] to-jbc: Var(x) -> [ ILOAD(x) ] where <type-of> Var(x) => INT()to-jbc: Var(x) -> [ ALOAD(x) ] where <type-of> Var(x) => STRING()lhs-to-jbc: Var(x) -> [ ISTORE(x) ] where <type-of> Var(x) => INT()lhs-to-jbc: Var(x) -> [ ASTORE(x) ] where <type-of> Var(x) => STRING() Traditional Parsing Algorithms 9
    14. 14. MiniJava Compilerliveness analysis.method public static main([Ljava/lang/String;)V .limit locals 3 .limit stack 4 .var 1 is x I from startX to endX .var 2 is y Z from startY to endY ... .end method Traditional Parsing Algorithms 10
    15. 15. MiniJava Compilersubmissionprocedure ensure compiler works add 5 positive test cases include your Net ID in project name export project as zip file ensure editor works when imported submit zip file via Blackboardsubmission due Jan 9, 23:59 delay costs you 25 points plus 1 point per hour or part thereof Traditional Parsing Algorithms 11
    16. 16. Assignment: MiniJava Editorgradescode generation 60 points classes 10 points fields & methods 15 points statements 15 points expressions 20 pointsliveness analysis 30+5 points maximum stack size 5 bonus pointstest cases 10 points Traditional Parsing Algorithms 12
    17. 17. IIIpredictive parsing Lexical Analysis 13
    18. 18. Recap: A Theory of Languageformal languages Traditional Parsing Algorithms 14
    19. 19. Recap: A Theory of Languageformal languagesvocabulary Σ finite, nonempty set of elements (words, letters) alphabet Traditional Parsing Algorithms 14
    20. 20. Recap: A Theory of Languageformal languagesvocabulary Σ finite, nonempty set of elements (words, letters) alphabetstring over Σ finite sequence of elements chosen from Σ word, sentence, utterance Traditional Parsing Algorithms 14
    21. 21. Recap: A Theory of Languageformal languagesvocabulary Σ finite, nonempty set of elements (words, letters) alphabetstring over Σ finite sequence of elements chosen from Σ word, sentence, utteranceformal language λ set of strings over a vocabulary Σ λ ⊆ Σ* Traditional Parsing Algorithms 14
    22. 22. Recap: A Theory of Languageformal grammars Traditional Parsing Algorithms 15
    23. 23. Recap: A Theory of Languageformal grammarsformal grammar G = (N, Σ, P, S) nonterminal symbols N terminal symbols Σ production rules P ⊆ (N∪Σ)* N (N∪Σ)* × (N∪Σ)* start symbol S∈N Traditional Parsing Algorithms 15
    24. 24. Recap: A Theory of Languageformal grammarsformal grammar G = (N, Σ, P, S) nonterminal symbols N nonterminal symbol terminal symbols Σ production rules P ⊆ (N∪Σ)* N (N∪Σ)* × (N∪Σ)* start symbol S∈N Traditional Parsing Algorithms 15
    25. 25. Recap: A Theory of Languageformal grammarsformal grammar G = (N, Σ, P, S) nonterminal symbols N terminal symbols Σ production rules P ⊆ (N∪Σ)* N (N∪Σ)* × (N∪Σ)* start symbol S∈N context Traditional Parsing Algorithms 15
    26. 26. Recap: A Theory of Languageformal grammarsformal grammar G = (N, Σ, P, S) nonterminal symbols N terminal symbols Σ production rules P ⊆ (N∪Σ)* N (N∪Σ)* × (N∪Σ)* start symbol S∈N replacement Traditional Parsing Algorithms 15
    27. 27. Recap: A Theory of Languageformal grammarsformal grammar G = (N, Σ, P, S) nonterminal symbols N terminal symbols Σ production rules P ⊆ (N∪Σ)* N (N∪Σ)* × (N∪Σ)* start symbol S∈Ngrammar classes type-0, unrestricted type-1, context-sensitive: (a A c, a b c) type-2, context-free: P ⊆ N × (N∪Σ)* type-3, regular: (A, x) or (A, xB) Traditional Parsing Algorithms 15
    28. 28. Recap: A Theory of Languageformal languages Traditional Parsing Algorithms 16
    29. 29. Recap: A Theory of Languageformal languagesformal grammar G = (N, Σ, P, S) Traditional Parsing Algorithms 16
    30. 30. Recap: A Theory of Languageformal languagesformal grammar G = (N, Σ, P, S)derivation relation G ⊆ (N∪Σ)* × (N∪Σ)* w G w’ ∃(p, q)∈P: ∃u,v∈(N∪Σ)*: w=u p v ∧ w’=u q v Traditional Parsing Algorithms 16
    31. 31. Recap: A Theory of Languageformal languagesformal grammar G = (N, Σ, P, S)derivation relation G ⊆ (N∪Σ)* × (N∪Σ)* w G w’ ∃(p, q)∈P: ∃u,v∈(N∪Σ)*: w=u p v ∧ w’=u q vformal language L(G) ⊆ Σ* L(G) = {w∈Σ* | S G * w} Traditional Parsing Algorithms 16
    32. 32. Recap: A Theory of Languageformal languagesformal grammar G = (N, Σ, P, S)derivation relation G ⊆ (N∪Σ)* × (N∪Σ)* w G w’ ∃(p, q)∈P: ∃u,v∈(N∪Σ)*: w=u p v ∧ w’=u q vformal language L(G) ⊆ Σ* L(G) = {w∈Σ* | S G * w}classes of formal languages Traditional Parsing Algorithms 16
    33. 33. Predictive parsingrecursive descentExp → “while” Exp “do” Exppublic void parseExp() { consume(WHILE); parseExp(); consume(DO); parseExp();} Traditional Parsing Algorithms 17
    34. 34. Predictive parsinglook aheadExp → “while” Exp “do” ExpExp → “if” Exp “then” Exp “else” Exppublic void parseExp() { switch current() { case WHILE: consume(WHILE); parseExp(); ...; break; case IF : consume(IF); parseExp(); ...; break; default : error(); }} Traditional Parsing Algorithms 18
    35. 35. Predictive parsingparse tablerows T1 T2 T3 ... • nonterminal symbols N N1 N1 →... N1 →... • symbol to parse N2 N2 →...columns N3 N3 →... N3 →... N4 N4 →... • terminal symbols Σ k N5 N5 →... • look ahead k N6 N6 →... N6 →...entries N7 N7 →... • production rules P N8 N8 →... N8 →... N8 →... • possible conflicts ... Traditional Parsing Algorithms 19
    36. 36. Predictive parsingautomaton input t1 … tn $ parse table S stack $ Traditional Parsing Algorithms 20
    37. 37. Predictive parsingautomaton x … $ x … $ Traditional Parsing Algorithms 21
    38. 38. Predictive parsingautomaton … $ … $ Traditional Parsing Algorithms 21
    39. 39. Predictive parsingautomaton x … $ x Xi Xi Xi → Y1... Yk … $ Traditional Parsing Algorithms 22
    40. 40. Predictive parsingautomaton x … $ Y1 x … Yk Xi Xi → Y1... Yk … $ Traditional Parsing Algorithms 22
    41. 41. Predictive parsingfilling the tableentry (X, w)∈P at row X and column T T∈FIRST(w) nullable(w) ∧ T∈FOLLOW(X) Traditional Parsing Algorithms 23
    42. 42. Predictive parsingnullablenullable(X) (X, ε) ∈ P nullable(X) (X0, X1 … Xk)∈P ∧ nullable(X1) ∧ … ∧ nullable(Xk) nullable(X0)nullable(w) nullable(ε) nullable(X1 … Xk) = nullable(X1) ∧ … ∧ nullable(Xk) Traditional Parsing Algorithms 24
    43. 43. Predictive parsingfirst setsFIRST(X) X∈Σ : FIRST(X) = {X} (X0, X1 … Xi … Xk)∈P ∧ nullable(X1 … Xi) FIRST(X0) ⊇ FIRST(Xi+1)FIRST(w) FIRST(ε) = {} ¬nullable(X) FIRST(Xw) = FIRST(X) nullable(X) FIRST(Xw) = FIRST(X) ∪ FIRST(w) Traditional Parsing Algorithms 25
    44. 44. Predictive parsingfollow setsFOLLOW(X) (X0, X1 … Xi … Xk)∈P ∧ nullable(Xi+1 … Xk) FOLLOW(Xi) ⊇ FOLLOW(X0) (X0, X1 … Xi … Xk)∈P FOLLOW(Xi) ⊇ FIRST(Xi+1 … Xk) Traditional Parsing Algorithms 26
    45. 45. Predictive parsingencoding precedenceExp → NumExp → “(” Exp “)”Exp → Exp “*” ExpExp → Exp “+” ExpFact → NumFact → “(” Exp “)”Term → Term “*” FactTerm → FactExp → Exp “+” TermExp → Term Traditional Parsing Algorithms 27
    46. 46. Predictive parsingeliminating left recursionTerm → Term “*” FactTerm → FactExp → Exp “+” TermExp → TermTerm’ → “*” Fact Term’Term’ →Term → Fact Term’Exp’ → “+” Term Exp’Exp’ →Exp → Term Exp’ Traditional Parsing Algorithms 28
    47. 47. Predictive parsingleft factoringExp → “if” Exp “then” Exp “else” ExpExp → “if” Exp “then” ExpExp → “if” Exp “then” Exp ElseElse → “else” ExpElse → Traditional Parsing Algorithms 29
    48. 48. Grammar classes context-free grammars Traditional Parsing Algorithms 30
    49. 49. Grammar classes context-free grammars LL(0) Traditional Parsing Algorithms 30
    50. 50. Grammar classes context-free grammars LL(1) LL(0) Traditional Parsing Algorithms 30
    51. 51. Grammar classes context-free grammars LL(k) LL(1) LL(0) Traditional Parsing Algorithms 30
    52. 52. coffee break Traditional Parsing Algorithms 31
    53. 53. IVLR parsing Lexical Analysis 32
    54. 54. LR parsingideaproblems with LL parsing • predicting right rule • left recursionLR parsing • see whole left-hand side of a rule • look ahead • shift or reduce Traditional Parsing Algorithms 33
    55. 55. LR parsingexample stack input$ 7 * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    56. 56. LR parsingexample$ 7 * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    57. 57. LR parsingexample$ 7 * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    58. 58. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    59. 59. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    60. 60. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $ Traditional Parsing Algorithms 34
    61. 61. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $ Traditional Parsing Algorithms 34
    62. 62. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $ Traditional Parsing Algorithms 34
    63. 63. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $ Traditional Parsing Algorithms 34
    64. 64. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $ Traditional Parsing Algorithms 34
    65. 65. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $ Traditional Parsing Algorithms 34
    66. 66. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $ Traditional Parsing Algorithms 34
    67. 67. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $ Traditional Parsing Algorithms 34
    68. 68. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $$ E + E * E $ Traditional Parsing Algorithms 34
    69. 69. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $$ E + E * E $$ E + E $ Traditional Parsing Algorithms 34
    70. 70. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $$ E + E * E $$ E + E $$ E $ Traditional Parsing Algorithms 34
    71. 71. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $$ E + E * E $$ E + E $$ E $ Traditional Parsing Algorithms 34
    72. 72. LR parsingexample$ 7 * 3 + 7 * 3 $$ E * 3 + 7 * 3 $$ E * E + 7 * 3 $$ E + 7 * 3 $$ E + E * 3 $$ E + E * E $$ E + E $$ E $$ S Traditional Parsing Algorithms 34
    73. 73. LR parsingparse tablerows T1 ... N1 ... • states of a DFA 1 s3 2 g5columns 3 r1 • topmost stack symbol 4 r2 • Σ, N 5entries 6 g1 • reduce, goto state 7 s1 • shift, goto state 8 • goto state ... Traditional Parsing Algorithms 35
    74. 74. Grammar classes context-free grammars LL(k) LL(1) LL(0) Traditional Parsing Algorithms 36
    75. 75. Grammar classes context-free grammars LL(k) LL(1) LL(0) LR(0) Traditional Parsing Algorithms 36
    76. 76. Grammar classes context-free grammars LL(k) LL(1) LR(1) LL(0) LR(0) Traditional Parsing Algorithms 36
    77. 77. Grammar classes context-free grammars LL(k) LR(k) LL(1) LR(1) LL(0) LR(0) Traditional Parsing Algorithms 36
    78. 78. Grammar classes context-free grammars LL(k) LR(k) LL(1) LR(1) SLR LL(0) LR(0) Traditional Parsing Algorithms 36
    79. 79. Grammar classes context-free grammars LL(k) LR(k) LL(1) LR(1) LALR(1) SLR LL(0) LR(0) Traditional Parsing Algorithms 36
    80. 80. Vsummary Lexical Analysis 37
    81. 81. Summarylessons learned Traditional Parsing Algorithms 38
    82. 82. Summarylessons learnedHow can we parse context-free languages effectively? • predictive parsing algorithms • LR parsing algorithms Traditional Parsing Algorithms 38
    83. 83. Summarylessons learnedHow can we parse context-free languages effectively? • predictive parsing algorithms • LR parsing algorithmsWhich grammar classes are supported by these algorithms? • LL(k) grammars, LL(k) languages • LR(k) grammars, LR(k) languages Traditional Parsing Algorithms 38
    84. 84. Summarylessons learnedHow can we parse context-free languages effectively? • predictive parsing algorithms • LR parsing algorithmsWhich grammar classes are supported by these algorithms? • LL(k) grammars, LL(k) languages • LR(k) grammars, LR(k) languagesHow can we generate compiler tools from that? • implement automata • generate parse tables Traditional Parsing Algorithms 38
    85. 85. Summarylessons learnedHow can we parse context-free languages effectively? • predictive parsing algorithms • LR parsing algorithmsWhich grammar classes are supported by these algorithms? • LL(k) grammars, LL(k) languages • LR(k) grammars, LR(k) languagesHow can we generate compiler tools from that? • implement automata • generate parse tables Traditional Parsing Algorithms 38
    86. 86. Literaturelearn more Traditional Parsing Algorithms 39
    87. 87. Literaturelearn moreformal languages Noam Chomsky: Three models for the description of language. 1956 J. E. Hopcroft, R. Motwani, J. D. Ullman: Introduction to Automata Theory, Languages, and Computation. 2006 Traditional Parsing Algorithms 39
    88. 88. Literaturelearn moreformal languages Noam Chomsky: Three models for the description of language. 1956 J. E. Hopcroft, R. Motwani, J. D. Ullman: Introduction to Automata Theory, Languages, and Computation. 2006syntactical analysis Andrew W. Appel, Jens Palsberg: Modern Compiler Implementation in Java, 2nd edition. 2002 Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman, Monica S. Lam: Compilers: Principles, Techniques, and Tools, 2nd edition. 2006 Traditional Parsing Algorithms 39
    89. 89. Outlookcoming nextlectures • Lecture 13: SDF inside • Lecture 14: Static analysis • Lecture 15: Beyond grammarwareQuestion & Answer Jan 5 • 10 questions, submit & voteLab Dec 3 • explore Java Bytecode • explore Jasmin • start generator Traditional Parsing Algorithms 40
    90. 90. questions Lexical Analysis 41
    91. 91. credits Lexical Analysis 42
    92. 92. PicturescopyrightsSlide 1: Book Scanner by Ben Woosley, some rights reservedSlide 5: Report card by Carosaurus, some rights reservedSlides 7, 12: Students, TU Delft, Media Solutions, all rights reservedSlides 8, 10: Italian Java book cover by unknown artist, some rights reservedSlide 11: Envelopes by benchilada, some rights reservedSlides 14-16: Noam Chomsky by Fellowsisters, some rights reservedSlide 17, 18, 27-29: Tiger by Bernard Landgraf, some rights reservedSlide 31: West Cornwall Pasty Co. by Dominica Williamson, some rights reservedSlide 40: Ostsee by Mario Thiel, some rights reservedSlide 41: Questions by Oberazzi, some rights reservedSlide 42: Too Much Credit by Andres Rueda, some rights reserved Traditional Parsing Algorithms 43

    ×