Successfully reported this slideshow.

Syntax Definition

4,475 views

Published on

Presentation slides from lecture 3 of course IN4303 on Compiler Construction at TU Delft.

Published in: Education, Technology
  • Be the first to comment

Syntax Definition

  1. 1. IN4303 2015/16 Compiler Construction Language Specification syntax definition Guido Wachsmuth, Eduardo de Souza Amorim
  2. 2. Syntax Definition 2 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp parser
  3. 3. Syntax Definition 3 3 * +7 21 parser Const Mul Const 213 7 Const Add
  4. 4. Syntax Definition 4 3 * +7 21 parser Add( Mul( Const(3) , Const(7) ) , Const(21) )
  5. 5. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 5 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  6. 6. Syntax Definition 6 syntax definition regular expressions Backus-Naur Form Extended BNF SDF3 lexical syntax context-free syntax abstract syntax disambiguation Spoofax architecture testing editors syntax processing pretty-printing discovery more editor services
  7. 7. Syntax Definition 7 syntax definition
  8. 8. Syntax Definition 8 software languages
  9. 9. Syntax Definition 9 finite models
  10. 10. Syntax Definition 10 Philosophy Linguistics lexicology grammar morphology syntax phonology semantics Interdisciplinary Computer Science syntax semantics
  11. 11. Syntax Definition 11 lexical syntax words made from letters irrelevant structure literals 42 "foo" true identifiers x foo keywords if while operators + * whitespace // FIXME regular expressions
 context-free syntax sentences made from words relevant structure context-free grammars Backus-Naur Form Extended Backus-Naur Form computer science syntax
  12. 12. Syntax Definition 12 syntax definition regular expressions
  13. 13. Syntax Definition 13 basics strings "nil" character classes [a-zA-Z] combinators concatenation E1 E2 option E? repetition (zero or more) E* repetition (one or more) E+ alternative E1 | E2 regular expressions
  14. 14. Syntax Definition 14 /* factorial function */ let var x := 0 function fact(n : int) : int = if n < 1 then 1 else (n * fact(n - 1)) in for i := 1 to 3 do ( x := x + fact(i); printint(x); print(" ") ) end Tiger the lecture language
  15. 15. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. Tiger lexical syntax
  16. 16. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* Tiger lexical syntax
  17. 17. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* A comment may appear between any two tokens. Comments start with /* and end with */ and may be nested. Tiger lexical syntax
  18. 18. Syntax Definition 15 An identifier is a sequence of letters, digits, and under-scores, starting with a letter. Uppercase letters are distinguished from lowercase. [a-zA-Z][a-zA-Z0-9_]* A comment may appear between any two tokens. Comments start with /* and end with */ and may be nested. Tiger lexical syntax Pumping Lemma
  19. 19. Syntax Definition 16 syntax definition Backus-Naur Form
  20. 20. Syntax Definition 17 basics strings nil symbols <s> combinators concatenation E1 E2 production <s> ::= E1 | … | En Backus-Naur Form
  21. 21. Syntax Definition 18 <exp> ::= <num> | <exp> + <exp> | <exp> - <exp> | <exp> * <exp> | <exp> / <exp> | ( <exp> ) Tiger context-free syntax
  22. 22. Syntax Definition 19 syntax definition Extended Backus-Naur Form
  23. 23. Syntax Definition 20 basics strings "nil" symbols s combinators concatenation E1,E2 option [E] repetition (zero or more) {E} alternative E1 | E2 production s = E1 | … | En ; Extended Backus-Naur Form
  24. 24. Syntax Definition 21 exp = num | exp , "+" , exp | exp , "-" , exp | exp , "*" , exp | exp , "/" , exp | "(" , exp , ")" ; Tiger context-free syntax
  25. 25. Syntax Definition 22 SDF3 lexical syntax
  26. 26. Syntax Definition 23 basics strings "nil" character classes [a-zA-Z] complements ~[a-zA-Z] sorts S combinators concatenation E1 E2 option E? repetition (zero or more) E* with separator {E s}* repetition (one or more) E+ with separator {E s}+ alternative E1 | E2 production S = E SDF3 lexical syntax
  27. 27. Syntax Definition 24 module Literals lexical syntax ID = [a-zA-Z] [a-zA-Z0-9_]* INT = "-"? [0-9]+ STRING = """ StringChar* """ StringChar = ~["n] StringChar = """ StringChar = "" Tiger lexical syntax
  28. 28. Syntax Definition 25 module Whitespace lexical syntax LAYOUT = [ tnr] LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = [*] Tiger lexical syntax
  29. 29. Syntax Definition 26 SDF3 context-free syntax
  30. 30. Syntax Definition 27 basics strings "nil" sorts S combinators concatenation E1 E2 option E? repetition (zero or more) E* with separator {E s}* repetition (one or more) E+ with separator {E s}+ production S = E SDF3 context-free syntax
  31. 31. Syntax Definition 28 module Expressions context-free syntax Exp = LValue LValue = ID LValue = LValue "." ID LValue = LValue "[" Exp "]" Exp = "nil" Exp = INT Exp = STRING Exp = ID "(" {Exp ","}* ")" Exp = TypeId "{" {InitField ","}* "}" Exp = TypeId "[" Exp "]" "of" Exp InitField = ID "=" Exp Tiger context-free syntax x x.f1 x[i] nil 42 "foo" sqr(x) aty [x] of 42 rty {f1 = "foo", f2 = 42}
  32. 32. Syntax Definition 29 context-free syntax Exp = Exp "+" Exp Exp = Exp "-" Exp Exp = Exp "*" Exp Exp = Exp "/" Exp Exp = Exp "=" Exp Exp = Exp "<>" Exp Exp = Exp ">" Exp Exp = Exp "<" Exp Exp = Exp ">=" Exp Exp = Exp "<=" Exp Exp = Exp "&" Exp Exp = Exp "|" Exp Tiger context-free syntax
  33. 33. Syntax Definition 30 SDF3 abstract syntax
  34. 34. Syntax Definition 31 tree leaf parent node children are trees term constant constructor arguments are terms ATerms trees - terms Const Mul Const 213 7 Const Add Add( Mul( Const(3) , Const(7) ) , Const(21) )
  35. 35. Syntax Definition 32 context-free syntax Exp.Add = Exp "+" Exp Exp.Sub = Exp "-" Exp Exp.Mul = Exp "*" Exp Exp.Div = Exp "/" Exp Exp.Eq = Exp "=" Exp Exp.Neq = Exp "<>" Exp Exp.Gt = Exp ">" Exp Exp.Lt = Exp "<" Exp Exp.Gte = Exp ">=" Exp Exp.Lte = Exp "<=" Exp Exp.And = Exp "&" Exp Exp.Or = Exp "|" Exp Tiger context-free syntax Const Mul Const 213 7 Const Add Add( Mul( Const(3) , Const(7) ) , Const(21) )
  36. 36. Syntax Definition 33 SDF3 disambiguation
  37. 37. Syntax Definition 34 lexical syntax ID = [a-zA-Z] [a-zA-Z0-9_]* ID = "nil" {reject} INT = "-"? [0-9]+ lexical restrictions ID -/- [a-zA-Z0-9_] INT -/- [0-9] Tiger lexical disambiguation
  38. 38. Syntax Definition 35 lexical syntax LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = "*" lexical restrictions EOF -/- ~[] "*" -/- ~[/] Tiger lexical disambiguation
  39. 39. Syntax Definition 36 lexical syntax LAYOUT = "//" ~[nr]* NewLineEOF NewLineEOF = [nr] | EOF EOF = LAYOUT = "/*" CommentPart* "*/" CommentPart = ~[*] CommentPart = CommentEndChar CommentEndChar = "*" lexical restrictions EOF -/- ~[] CommentEndChar -/- ~[/] Tiger lexical disambiguation
  40. 40. Syntax Definition 37 lexical syntax LAYOUT = [ tnr] LAYOUT = "//" ~[nr]* NewLineEOF LAYOUT = "/*" CommentPart* "*/" context-free restrictions LAYOUT? -/- [ tnr] LAYOUT? -/- [/].[/] LAYOUT? -/- [/].[*] Tiger context-free disambiguation
  41. 41. Syntax Definition 38 context-free syntax Exp.Add = Exp "+" Exp Exp.Sub = Exp "-" Exp Exp.Mul = Exp "*" Exp Exp.Div = Exp "/" Exp Exp.Eq = Exp "=" Exp Exp.Neq = Exp "<>" Exp Exp.Gt = Exp ">" Exp Exp.Lt = Exp "<" Exp Exp.Gte = Exp ">=" Exp Exp.Lte = Exp "<=" Exp Exp.And = Exp "&" Exp Exp.Or = Exp "|" Exp Tiger context-free syntax
  42. 42. Syntax Definition 39 context-free syntax Exp.Add = Exp "+" Exp {left} Exp.Sub = Exp "-" Exp {left} Exp.Mul = Exp "*" Exp {left} Exp.Div = Exp "/" Exp {left} Exp.Eq = Exp "=" Exp {non-assoc} Exp.Neq = Exp "<>" Exp {non-assoc} Exp.Gt = Exp ">" Exp {non-assoc} Exp.Lt = Exp "<" Exp {non-assoc} Exp.Gte = Exp ">=" Exp {non-assoc} Exp.Lte = Exp "<=" Exp {non-assoc} Exp.And = Exp "&" Exp {left} Exp.Or = Exp "|" Exp {left} context-free priorities { left: Exp.Mul Exp.Div } > { left: Exp.Add Exp.Sub } > { non-assoc: Exp.Eq Exp.Neq Exp.Gt Exp.Lt Exp.Gte Exp.Lte } > Exp.And > Exp.Or Tiger context-free disambiguation
  43. 43. Syntax Definition 40 Spoofax architecture
  44. 44. Syntax Definition 41 source code
  45. 45. Syntax Definition 41 source code parse
  46. 46. Syntax Definition 41 source code parse check
  47. 47. Syntax Definition 41 source code parse check
  48. 48. Syntax Definition 41 source code parse generate machine code check
  49. 49. Syntax Definition Data Programs Language Processors 42 Eclipse JDT Fact.java 42 Java Java int
  50. 50. Syntax Definition 43 language definition parse generate language implementation check
  51. 51. Syntax Definition syntax definition 44 parse generate parse table check generic parser
  52. 52. Syntax Definition Data Programs Language Processors 45 Java parser Fact.java 42 Language Workbench Spoofax SDF3 Java int SDF3
  53. 53. Syntax Definition 46 context-free syntax Grammar.Lexical = < lexical syntax <Production*> > Grammar.Contextfree = < context-free syntax <Production*> > Production.SdfProduction = <<Symbol> = <Symbol*> <Attributes>> Production.SdfProductionWithCons = <<SortCons> = <Symbol*> <Attributes>> SortCons.SortCons = <<Symbol>.<Constructor>> context-free syntax SDF3 in SDF3
  54. 54. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 47 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  55. 55. Syntax Definition Stratego NaBL TS SDF3 ESV editor SPT tests 48 syntax definition concrete syntax abstract syntax static semantics name binding type system dynamic semantics translation interpretation
  56. 56. Syntax Definition 49 Spoofax testing
  57. 57. Syntax Definition 50 module example-suite language Tiger start symbol Start test name [[...]] parse succeeds test another name [[...]] parse fails test suites
  58. 58. Syntax Definition 51 test cases syntax Language valid program test valid program [[...]] parse succeeds
  59. 59. Syntax Definition 52 test cases syntax Language test invalid program [[...]] parse fails invalid program
  60. 60. Syntax Definition 53 module syntax/identifiers language Tiger start symbol Id test single lower case [[x]] parse succeeds test single upper case [[X]] parse succeeds test single digit [[1]] parse fails test single lc digit [[x1]] parse succeeds test single digit lc [[1x]] parse fails test single uc digit [[X1]] parse succeeds test single digit uc [[1X]] parse fails test double digit [[11]] parse fails test cases syntax
  61. 61. Syntax Definition Language 54 test cases syntax
  62. 62. Syntax Definition 55 module syntax/expressions/precedence language Tiger start symbol Exp test parentheses [[(42)]] parse to [[42]] test left-associative addition [[21 + 14 + 7]] parse to [[(21 + 14) + 7]] test precedence multiplication [[3 * 7 + 21]] parse to [[(3 * 7) + 21]] test cases ambiguities
  63. 63. Syntax Definition 56 Spoofax editors
  64. 64. Syntax Definition syntax definition 57 parse generate parse table check generic parser
  65. 65. Syntax Definition 58 source code parse generate machine code check
  66. 66. Syntax Definition 59 module Tiger.main imports ... language General properties name: Tiger id: org.metaborg.cube.tiger extends: Root description: "Spoofax-generated editor for the Tiger language" url: http://metaborg.org extensions: tig table: include/Tiger.tbl start symbols: Start provider: include/Tiger.ctree observer: editor-analyze (multifile) on save: editor-save editor specification
  67. 67. Syntax Definition 60 syntax processing
  68. 68. Syntax Definition 61 3 * +7 213 * +7 21 3 * +7 21 Exp ExpExp Exp Exp scanner parser language processors scanners & parsers
  69. 69. Syntax Definition 62 language processors scannerless parsers 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp parser
  70. 70. Syntax Definition 63 syntax processing pretty-printing
  71. 71. Syntax Definition 64 language processors compilation by transformation backend frontend parse desugar analyse normalise generate optimise format
  72. 72. Syntax Definition 65 language processors pretty-printers 3 * +7 21 3 * +7 21 Exp ExpExp Exp Exp pretty- printer
  73. 73. Syntax Definition 66 from ASTs to text keywords layout: spaces, line breaks, indentation specification partially defined in grammar missing layout language processors pretty-printing
  74. 74. Syntax Definition 67 SDF3 production rules context-free syntax Exp.Nil = "nil" Exp.IntC = INT Exp.StringC = STRING Exp.Call = ID "(" {Exp ","}* ")" Exp.ArrayI = TypeId "[" Exp "]" "of" Exp Exp.RecordI = TypeId "{" {InitField ","}* "}" InitField.FieldI = ID "=" Exp
  75. 75. Syntax Definition 68 SDF3 templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID> ( <{Exp ","}*> )> Exp.ArrayI = <<TypeId> [ <Exp> ] of <Exp>> Exp.RecordI = <<TypeId> { <{InitField ","}*> }> InitField.FieldI = <<ID> = <Exp>>
  76. 76. Syntax Definition 69 SDF3 formatted templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID>(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId>[<Exp>] of <Exp>> Exp.RecordI = < <TypeId> { <{InitField ",n"}*> }> InitField.FieldI = <<ID> = <Exp>>
  77. 77. Syntax Definition 70 box layout basic boxes _1 “foo” KW [ “foo” ]
  78. 78. Syntax Definition 71 box layout horizontal boxes B B B H hs=x [ ]B B Bhs=x
  79. 79. Syntax Definition 72 box layout vertical boxes V hs=x is=i [ ]B B Bvs=y is=i B B B
  80. 80. Syntax Definition 73 syntax processing discovery
  81. 81. Syntax Definition 74 discovery syntactic code completion
  82. 82. Syntax Definition 75 SDF3 formatted templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID>(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId>[<Exp>] of <Exp>> Exp.RecordI = < <TypeId> { <{InitField ",n"}*> }> InitField.FieldI = <<ID> = <Exp>>
  83. 83. Syntax Definition 76 SDF3 generated completion templates module src-gen/completions/Expressions-esv completions completion template Exp : "nil" = "nil" completion template Exp : "ID()" = <ID:ID> "(" <:Exp> ")" completion template Exp : "TypeId[Exp] of Exp" = <TypeId:TypeId> "[" <Exp:Exp> "] of " <Exp:Exp> completion template Exp : "TypeId { }" = <TypeId:TypeId> " {nt" (cursor) "n}" (blank) completion template InitField : "ID = Exp" = <ID:ID> " = " <Exp:Exp>
  84. 84. Syntax Definition 77 SDF3 improved templates context-free syntax Exp.Nil = <nil> Exp.IntC = INT Exp.StringC = STRING Exp.Call = <<ID; text="m">(<{Exp ", "}*>)> Exp.ArrayI = <<TypeId; text="type">[<Exp; text="size">] of <Exp; text="value">> Exp.RecordI = < <TypeId; text="type"> { <{InitField ",n"}*> }> InitField.FieldI = <<ID; text="var"> = <Exp; text="value">>
  85. 85. Syntax Definition 78 syntax processing more editor services
  86. 86. Syntax Definition 79 editor servicesThe Spoofax Language Workbench
  87. 87. Syntax Definition 80 Spoofax generated highlighting rules module Tiger-Colorer.generated colorer Default, token-based highlighting keyword : 127 0 85 bold identifier : default string : blue number : darkgreen var : 255 0 100 italic operator : 0 0 128 layout : 100 100 0 italic colorer System colors darkred = 128 0 0 red = 255 0 0 darkgreen = 0 128 0 green = 0 255 0 darkblue = 0 0 128 …
  88. 88. Syntax Definition 81 Spoofax customised highlighting rules module Tiger-Colorer imports Tiger-Colorer.generated colorer TU Delft colours TUDlavender = 123 160 201 colorer token-based highlighting layout : TUDlavender StrConst : darkgreen TypeId : blue
  89. 89. Syntax Definition 82 SDF3 generated folding rules module Tiger-Folding.generated folding Default folding definitions Exp Dec.TypeDec Type.RecordTy
  90. 90. Syntax Definition 83 Spoofax customised folding rules module Tiger-Folding imports Tiger-Folding.generated folding Dec* Dec.FunDec
  91. 91. Syntax Definition 84 Except where otherwise noted, this work is licensed under
  92. 92. Syntax Definition 85 attribution slide title author license 1 The Pine, Saint Tropez Paul Signac public domain 2-4, 43, 45, 46, 59, 60, 63, 64, 66, 67 PICOL icons Melih Bilgil CC BY 3.0 8 Programming language textbooks K.lee public domain 9 Latin Grammar Anthony Nelzin 14, 15, 18, 21 Tiger Bernard Landgraf CC BY-SA 3.0

×