Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Syntax Definition

1,114 views

Published on

Slides for lecture on syntax definition for course on Compiler Construction at TU Delft

Published in: Software
  • Be the first to comment

Syntax Definition

  1. 1. IN4303 2016-2017 Compiler Construction Syntax Definition language specification Guido Wachsmuth, Eelco Visser
  2. 2. Syntax Definition 2 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp parser
  3. 3. Syntax Definition 3 3 * +7 21 parser Const Mul Const 213 7 Const Add
  4. 4. Syntax Definition 4 3 * +7 21 parser Add( Mul( Const(3) , Const(7) ) , Const(21) )
  5. 5. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 5 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  6. 6. Syntax Definition 6 syntax definition regular expressions Backus-Naur Form Extended BNF SDF3 lexical syntax context-free syntax abstract syntax disambiguation Spoofax architecture testing editors syntax processing pretty-printing discovery more editor services
  7. 7. Syntax Definition 7 syntax definition
  8. 8. Syntax Definition 8 software languages
  9. 9. Syntax Definition 9 finite models
  10. 10. Syntax Definition 10 Philosophy Linguistics lexicology grammar morphology syntax phonology semantics Interdisciplinary Computer Science syntax semantics
  11. 11. Syntax Definition 11 lexical syntax words made from letters irrelevant structure literals 42 "foo" true identifiers x foo keywords if while operators + * whitespace // FIXME regular expressions
 context-free syntax sentences made from words relevant structure context-free grammars Backus-Naur Form Extended Backus-Naur Form computer science syntax
  12. 12. Syntax Definition 12 syntax definition regular expressions
  13. 13. Syntax Definition 13 basics strings "nil" character classes [a-zA-Z] combinators concatenation E1 E2 option E? repetition (zero or more) E* repetition (one or more) E+ alternative E1 | E2 regular expressions
  14. 14. Syntax Definition 14 /* factorial function */ let var x := 0 function fact(n : int) : int = if n < 1 then 1 else (n * fact(n - 1)) in for i := 1 to 3 do ( x := x + fact(i); printint(x); print(" ") ) end Tiger the lecture language
  15. 15. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. Tiger lexical syntax
  16. 16. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* Tiger lexical syntax
  17. 17. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* A comment may appear between any two tokens. Comments start with /* and end with */ and may be nested. Tiger lexical syntax
  18. 18. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* A comment may appear between any two tokens. Comments start with /* and end with */ and may be nested. Tiger lexical syntax Pumping Lemma
  19. 19. Syntax Definition 16 syntax definition Backus-Naur Form
  20. 20. Syntax Definition 17 basics strings “nil” symbols <s> combinators concatenation E1 E2 production <s> ::= E1 | … | En Backus-Naur Form
  21. 21. Syntax Definition 18 <exp> ::= <num> | <exp> “+” <exp> | <exp> “-“ <exp> | <exp> “*” <exp> | <exp> “/“ <exp> | “(“ <exp> “)” | <id> “(“ <args> “)” <args> ::= | <argp> <argp> ::= <exp> | <argp> “,” <exp>
  22. 22. Syntax Definition 19 <exp> ::= <num> | <exp> “+” <exp> | <exp> “-“ <exp> | <exp> “*” <exp> | <exp> “/“ <exp> | “(“ <exp> “)” Tiger context-free syntax in BNF
  23. 23. Syntax Definition 20 syntax definition Extended Backus-Naur Form
  24. 24. Syntax Definition 21 basics strings "nil" symbols s combinators concatenation E1,E2 option [E] repetition (zero or more) {E} alternative E1 | E2 production s = E1 | … | En ; Extended Backus-Naur Form
  25. 25. Syntax Definition 22 <exp> ::= <num> | <exp> “+” <exp> | <exp> “-“ <exp> | <exp> “*” <exp> | <exp> “/“ <exp> | “(“ <exp> “)” | <id> “(“ [<args>] “)” <args> ::= <exp> {“,” <exp>}
  26. 26. Syntax Definition 23 exp = num | exp , "+" , exp | exp , "-" , exp | exp , "*" , exp | exp , "/" , exp | "(" , exp , ")" ; Tiger context-free syntax
  27. 27. Syntax Definition 24 SDF3 Syntax Definition Formalism 3
  28. 28. Syntax Definition 25 SDF3 lexical syntax
  29. 29. Syntax Definition 26 basics strings "nil" character classes [a-zA-Z] complements ~[a-zA-Z] sorts S combinators concatenation E1 E2 option E? repetition (zero or more) E* with separator {E s}* repetition (one or more) E+ with separator {E s}+ alternative E1 | E2 production S = E SDF3 lexical syntax
  30. 30. Syntax Definition 27 module Literals lexical syntax ID = [a-zA-Z] [a-zA-Z0-9_]* INT = "-"? [0-9]+ STRING = """ StringChar* """ StringChar = ~["n] StringChar = """ StringChar = "" Tiger lexical syntax
  31. 31. Syntax Definition 28 module Whitespace lexical syntax LAYOUT = [ tnr] LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = [*] Tiger lexical syntax
  32. 32. Syntax Definition 29 SDF3 context-free syntax
  33. 33. Syntax Definition 30 basics strings "nil" sorts S combinators concatenation E1 E2 option E? repetition (zero or more) E* with separator {E s}* repetition (one or more) E+ with separator {E s}+ production S = E SDF3 context-free syntax
  34. 34. Syntax Definition 31 <exp> ::= <num> <exp> ::= <exp> “+” <exp> <exp> ::= <exp> “-“ <exp> <exp> ::= <exp> “*” <exp> <exp> ::= <exp> “/“ <exp> <exp> ::= “(“ <exp> “)” <exp> ::= <id> “(“ {<exp> “,”}* “)”
  35. 35. Syntax Definition 32 module Expressions context-free syntax Exp = LValue LValue = ID LValue = LValue "." ID LValue = LValue "[" Exp "]" Exp = "nil" Exp = INT Exp = STRING Exp = ID "(" {Exp ","}* ")" Exp = TypeId "{" {InitField ","}* "}" Exp = TypeId "[" Exp "]" "of" Exp InitField = ID "=" Exp Tiger context-free syntax x x.f1 x[i] nil 42 "foo" sqr(x) aty [x] of 42 rty {f1 = "foo", f2 = 42}
  36. 36. Syntax Definition 33 context-free syntax Exp = Exp "+" Exp Exp = Exp "-" Exp Exp = Exp "*" Exp Exp = Exp "/" Exp Exp = Exp "=" Exp Exp = Exp "<>" Exp Exp = Exp ">" Exp Exp = Exp "<" Exp Exp = Exp ">=" Exp Exp = Exp "<=" Exp Exp = Exp "&" Exp Exp = Exp "|" Exp Tiger context-free syntax
  37. 37. Syntax Definition 34 SDF3 abstract syntax
  38. 38. Syntax Definition 35 tree leaf parent node children are trees term constant constructor arguments are terms ATerms trees - terms Const Mul Const 213 7 Const Add Add( Mul( Const(3) , Const(7) ) , Const(21) )
  39. 39. Syntax Definition 36 context-free syntax Exp.Add = Exp "+" Exp Exp.Sub = Exp "-" Exp Exp.Mul = Exp "*" Exp Exp.Div = Exp "/" Exp Exp.Eq = Exp "=" Exp Exp.Neq = Exp "<>" Exp Exp.Gt = Exp ">" Exp Exp.Lt = Exp "<" Exp Exp.Gte = Exp ">=" Exp Exp.Lte = Exp "<=" Exp Exp.And = Exp "&" Exp Exp.Or = Exp "|" Exp Tiger context-free syntax Const Mul Const 213 7 Const Add Add( Mul( Const(3) , Const(7) ) , Const(21) )
  40. 40. Syntax Definition 37 Tiger* abstract syntax : algebraic signature signature constructors Uminus : Exp -> Exp Power : Exp * Exp -> Exp Times : Exp * Exp -> Exp Divide : Exp * Exp -> Exp Plus : Exp * Exp -> Exp Minus : Exp * Exp -> Exp CPlus : Exp * Exp -> Exp CMinus : Exp * Exp -> Exp Eq : Exp * Exp -> Exp Neq : Exp * Exp -> Exp Gt : Exp * Exp -> Exp Lt : Exp * Exp -> Exp Geq : Exp * Exp -> Exp Leq : Exp * Exp -> Exp True : Exp False : Exp And : Exp * Exp -> Exp Or : Exp * Exp -> Exp
  41. 41. Syntax Definition 38 SDF3 disambiguation
  42. 42. Syntax Definition 39 lexical syntax ID = [a-zA-Z] [a-zA-Z0-9_]* ID = "nil" {reject} INT = "-"? [0-9]+ lexical restrictions ID -/- [a-zA-Z0-9_] INT -/- [0-9] Tiger lexical disambiguation
  43. 43. Syntax Definition 40 lexical syntax LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = "*" lexical restrictions EOF -/- ~[] "*" -/- [/] Tiger lexical disambiguation
  44. 44. Syntax Definition 41 lexical syntax LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = CommentEndChar CommentEndChar = "*" lexical restrictions EOF -/- ~[] CommentEndChar -/- [/] Tiger lexical disambiguation
  45. 45. Syntax Definition 42 lexical syntax LAYOUT = [ tnr] LAYOUT = "//" ~[nr]* NewLineEOF LAYOUT = "/*" CommentPart* "*/" context-free restrictions LAYOUT? -/- [ tnr] LAYOUT? -/- [/].[/] LAYOUT? -/- [/].[*] Tiger context-free disambiguation
  46. 46. Syntax Definition 43 context-free syntax Exp.Add = Exp "+" Exp Exp.Sub = Exp "-" Exp Exp.Mul = Exp "*" Exp Exp.Div = Exp "/" Exp Exp.Eq = Exp "=" Exp Exp.Neq = Exp "<>" Exp Exp.Gt = Exp ">" Exp Exp.Lt = Exp "<" Exp Exp.Gte = Exp ">=" Exp Exp.Lte = Exp "<=" Exp Exp.And = Exp "&" Exp Exp.Or = Exp "|" Exp Tiger context-free syntax
  47. 47. Syntax Definition 44 context-free syntax Exp.Add = Exp "+" Exp {left} Exp.Sub = Exp "-" Exp {left} Exp.Mul = Exp "*" Exp {left} Exp.Div = Exp "/" Exp {left} Exp.Eq = Exp "=" Exp {non-assoc} Exp.Neq = Exp "<>" Exp {non-assoc} Exp.Gt = Exp ">" Exp {non-assoc} Exp.Lt = Exp "<" Exp {non-assoc} Exp.Gte = Exp ">=" Exp {non-assoc} Exp.Lte = Exp "<=" Exp {non-assoc} Exp.And = Exp "&" Exp {left} Exp.Or = Exp "|" Exp {left} context-free priorities { left: Exp.Mul Exp.Div } > { left: Exp.Add Exp.Sub } > { non-assoc: Exp.Eq Exp.Neq Exp.Gt Exp.Lt Exp.Gte Exp.Lte } > Exp.And > Exp.Or Tiger context-free disambiguation
  48. 48. Syntax Definition 45 Spoofax architecture
  49. 49. Syntax Definition 46 source code
  50. 50. Syntax Definition 46 source code parse
  51. 51. Syntax Definition 46 source code parse check
  52. 52. Syntax Definition 46 source code parse check
  53. 53. Syntax Definition 46 source code parse generate machine code check
  54. 54. Syntax Definition Data Programs Language 47 Eclipse Fact.java 42 Java Java int
  55. 55. Syntax Definition 48 language definition parse generate language implementation check
  56. 56. Syntax Definition syntax definition 49 parse generate parse table check generic parser
  57. 57. Syntax Definition Data Programs Language 50 Java Fact.java 42 Language Spoofax SDF3 Java int SDF3
  58. 58. Syntax Definition 51 context-free syntax Grammar.Lexical = < lexical syntax <Production*> > Grammar.Contextfree = < context-free syntax <Production*> > Production.SdfProduction = <<Symbol> = <Symbol*> <Attributes>> Production.SdfProductionWithCons = <<SortCons> = <Symbol*> <Attributes>> SortCons.SortCons = <<Symbol>.<Constructor>> context-free syntax SDF3 in SDF3
  59. 59. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 52 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  60. 60. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 53 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  61. 61. Syntax Definition 54 Spoofax testing
  62. 62. Syntax Definition 55 module example-suite language Tiger start symbol Start test name [[...]] parse succeeds test another name [[...]] parse fails test suites
  63. 63. Syntax Definition 56 test cases syntax Language valid program test valid program [[...]] parse succeeds
  64. 64. Syntax Definition 57 test cases syntax Language test invalid program [[...]] parse fails invalid program
  65. 65. Syntax Definition 58 module syntax/identifiers language Tiger start symbol Id test single lower case [[x]] parse succeeds test single upper case [[X]] parse succeeds test single digit [[1]] parse fails test single lc digit [[x1]] parse succeeds test single digit lc [[1x]] parse fails test single uc digit [[X1]] parse succeeds test single digit uc [[1X]] parse fails test double digit [[11]] parse fails test cases syntax
  66. 66. Syntax Definition Language 59 test cases syntax
  67. 67. Syntax Definition 60 module precedence language Tiger start symbol Exp test parentheses [[(42)]] parse to [[42]] test left-associative addition [[21 + 14 + 7]] parse to [[(21 + 14) + 7]] test precedence multiplication [[3 * 7 + 21]] parse to [[(3 * 7) + 21]] test cases ambiguities Note : this does not actually work in Tiger, since () is a sequencing construct; but works fine in Mini-Java
  68. 68. Syntax Definition 61 module precedence language Tiger start symbol Exp test plus/times priority [[ x + 4 * 5 ]] parse to Plus(Var("x"), Times(Int("4"), Int("5"))) test plus/times sequence [[ (x + 4) * 5 ]] parse to Times( Seq([Plus(Var("x"), Int("4"))]) , Int("5") ) test cases ambiguities
  69. 69. Syntax Definition 62 Spoofax editors
  70. 70. Syntax Definition syntax definition 63 parse generate parse table check generic parser
  71. 71. Syntax Definition 64 source code parse generate machine code check
  72. 72. Syntax Definition 65 module Tiger.main imports ... language General properties name: Tiger id: org.metaborg.cube.tiger extends: Root description: "Spoofax-generated editor for the Tiger language" url: http://metaborg.org extensions: tig table: include/Tiger.tbl start symbols: Start provider: include/Tiger.ctree observer: editor-analyze (multifile) on save: editor-save editor specification
  73. 73. Syntax Definition 66 syntax processing
  74. 74. Syntax Definition 67 3 * +7 213 * +7 21 3 * +7 21 Exp ExpExp Exp Exp scanner parser language processors scanners & parsers
  75. 75. Syntax Definition 68 language processors scannerless parsers 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp parser
  76. 76. Syntax Definition 69 syntax processing pretty-printing
  77. 77. Syntax Definition 70 language processors compilation by transformation backend frontend parse desugar analyse normalise generate optimise format
  78. 78. Syntax Definition 71 language processors pretty-printers 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp pretty- printer
  79. 79. Syntax Definition 72 from ASTs to text keywords layout: spaces, line breaks, indentation specification partially defined in grammar missing layout language processors pretty-printing
  80. 80. Syntax Definition 73 SDF3 production rules context-free syntax Exp.Nil = "nil" Exp.IntC = INT Exp.StringC = STRING Exp.Call = ID "(" {Exp ","}* ")" Exp.ArrayI = TypeId "[" Exp "]" "of" Exp Exp.RecordI = TypeId "{" {InitField ","}* "}" InitField.FieldI = ID "=" Exp
  81. 81. Syntax Definition 74 SDF3 templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID> ( <{Exp ","}*> )> Exp.ArrayI = <<TypeId> [ <Exp> ] of <Exp>> Exp.RecordI = <<TypeId> { <{InitField ","}*> }> InitField.FieldI = <<ID> = <Exp>>
  82. 82. Syntax Definition 75 SDF3 formatted templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID>(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId>[<Exp>] of <Exp>> Exp.RecordI = < <TypeId> { <{InitField ",n"}*> }> InitField.FieldI = <<ID> = <Exp>>
  83. 83. Syntax Definition 76 box layout basic boxes _1 “foo” KW [ “foo” ]
  84. 84. Syntax Definition 77 box layout horizontal boxes B B B H hs=x [ ]B B Bhs=x
  85. 85. Syntax Definition 78 box layout vertical boxes V hs=x is=i [ ]B B Bvs=y is=i B B B
  86. 86. Syntax Definition 79 syntax processing discovery
  87. 87. Syntax Definition 80 discovery syntactic code completion
  88. 88. Syntax Definition 81 SDF3 formatted templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID>(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId>[<Exp>] of <Exp>> Exp.RecordI = < <TypeId> { <{InitField ",n"}*> }> InitField.FieldI = <<ID> = <Exp>>
  89. 89. Syntax Definition 82 SDF3 generated completion templates module src-gen/completions/Expressions-esv completions completion template Exp : "nil" = "nil" completion template Exp : "ID()" = <ID:ID> "(" <:Exp> ")" completion template Exp : "TypeId[Exp] of Exp" = <TypeId:TypeId> "[" <Exp:Exp> "] of " <Exp:Exp> completion template Exp : "TypeId { }" = <TypeId:TypeId> " {nt" (cursor) "n}" (blank) completion template InitField : "ID = Exp" = <ID:ID> " = " <Exp:Exp>
  90. 90. Syntax Definition 83 SDF3 improved templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID; text="m">(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId; text="type">[<Exp; text="size">] of <Exp; text="value">> Exp.RecordI = < <TypeId; text="type"> { <{InitField ",n"}*> }> InitField.FieldI = <<ID; text="var"> = <Exp; text="value">>
  91. 91. Syntax Definition 84 syntax processing more editor services
  92. 92. Syntax Definition 85 editor servicesThe Spoofax Language Workbench
  93. 93. Syntax Definition 86 Spoofax generated highlighting rules module libspoofax/color/default imports libspoofax/color/colors colorer // Default, token-based highlighting keyword : 127 0 85 bold identifier : default string : blue number : darkgreen var : 139 69 19 italic operator : 0 0 128 layout : 63 127 95 italic
  94. 94. Syntax Definition 87 Spoofax customised highlighting rules module Tiger-Colorer imports Tiger-Colorer.generated colorer TU Delft colours TUDlavender = 123 160 201 colorer token-based highlighting layout : TUDlavender StrConst : darkgreen TypeId : blue
  95. 95. Syntax Definition 88 Except where otherwise noted, this work is licensed under
  96. 96. Syntax Definition 89 attribution slide title author license 1 The Pine, Saint Tropez Paul Signac public domain 2-4, 43, 45, 46, 59, 60, 63, 64, 66, 67 PICOL icons Melih Bilgil CC BY 3.0 8 Programming language textbooks K.lee public domain 9 Latin Grammar Anthony Nelzin 14, 15, 18, 21 Tiger Bernard Landgraf CC BY-SA 3.0

×