The document introduces a convention-based syntax notation designed for both humans and parser generators. Key aspects include using conventions to simplify EBNF notation, defining rules without delimiters, inferring token sets from rules, and representing abstract syntax trees with variables. The approach was motivated by limitations of existing formalisms. Examples are provided for microsyntax, macrosyntax, and representing abstract syntax trees. Future work discussed expanding the approach to support additional language features.

1. Convention-Based Syntactic Descriptions Ray Toal Derek Smith Loyola Marymount University CSIE 2009, Los Angeles 2009-04-02

2. Outline Introduction

3. Goals and Objectives

4. Motivation and Challenges

5. Approach

6. Conventions

7. Summary

8. Future Work

9. Objectives Design a improved syntax formalism for programming languages, that can BOTH be used as a concise, formal description, and

10. be used as input to a parser generator Formalism must also be understandable to users of existing notations so we're basically using EBNF

11. ... with some regex notation and custom extensions

12. Motivation Few programming language specifications care to even separate microsyntax and macrosyntax ID -> LETTER (LETTER | DIGIT | '_')*

13. STMT -> 'while' EXP 'do' BLOCK Existing parser generator input languages have too much markup

14. Idea: Try to adapt convention over configuration to reduce markup requirements!

15. Challenges Existing formalisms are necessarily parser generator independent: e.g., don't want to commit to LL or LR Solution: Allow rich EBNF extensions to make LL a viable option Existing generators allow code to be run during parse Solution: Restrict “generation” to AST nodes only.

16. Microsyntax Example For a little C-like language: LETTER -> <L> DIGIT -> <Nd> CHAR -> [ˆ<Cc>&quot;\] | '\' [ˆ<Cc>] ID -> LETTER (LETTER | DIGIT | '_')* KEYWORD -> 'var' | 'fun' | 'read' | 'write' | 'while' | 'do' | 'end' NUMLIT -> DIGIT+ ('.' DIGIT+)? ([Ee] [+-]? DIGIT+)? STRLIT -> '&quot;' CHAR* '&quot;' SKIP -> <Zs> | #09 | #0A | #0D | '//' [ˆ#0A#0D]* [#0A#0D]

17. Microsyntax Rules use ->, and no delimiters needed

18. Rules must be non-recursive, and RHS can only use symbols from previous rules

19. Later rules take precedence In our example 'while' is a KEYWORD, not an ID

20. Means we don't need '-' meta operator SKIP predefined and must be last

21. Token set inferred

22. Quoting Object language forms are quoted

23. Five quoting mechanisms Codepoint: #0A, ##2029, ###0001D1CF

24. Category: <L>, <Nd>, <C>, <Zl>

25. String: 'while', ';'

26. One-of: [aeiou], [0-9A-Fa-f], [<L><Nd>_]

27. One-not-of: [^<Zs><C>], [^#0A#0D] No need for escaping: can always use #, and even reposition ']', '^', and '-' in [...]

28. Operators Whitespace between expressions

29. e 1 | e 2 or

30. e? optional

31. e* zero or more

32. e+ one or more

33. e^n exactly n e's

34. (…) grouping

35. Macrosyntax Example PROGRAM => BLOCK BLOCK => (DEC ';')* (STMT ';')+ DEC => 'var' ID ('=' EXP)? | 'fun' ID '(' IDLIST? ')' '=' EXP STMT => ID '=' EXP | 'read' IDLIST | 'write' EXPLIST | 'while' EXP 'do' BLOCK 'end' IDLIST => ID (',' ID)* EXPLIST => EXP (',' EXP)* EXP => TERM ([+-] TERM)* TERM => FACTOR ([*/] FACTOR)* FACTOR => NUMLIT | STRLIT | ID | CALL | '(' EXP ')' CALL => ID '(' EXPLIST? ')'

36. Macrosyntax Rules use =>

37. Means that SKIP* can appear before and after any token

38. Maximal munch assumed for tokenization

39. No delimiters needed between rules

40. First rule is the start symbol

41. Recursion is fine

42. Tokens can be introduced here, too!

43. Abstract Syntax var y; fun half(x) = x / 2; while x - (5 * x) do write half(10.4), x + 2; read x; end; (Program (Block (Var y) (Fun half x (/ (Ref x) (Numlit 2))) (While (- (Ref x) (* (Numlit 5) (Ref x))) (Block (Write (Call half (Numlit 10.4)) (+ (Ref x)(Numlit 2))) (Read x)))))

44. Abstract Syntax Goal: We want to define the AST for a given macrosyntax phrase with minimal markup (no code)

45. Ideas AST markup are declarative node expressions

46. Last expression encountered is the “value”

47. Some rules don't even need AST expressions

48. Can have variables which are implicitly list valued, with special syntax for reassignment

49. Abstract Syntax PROGRAM => b:BLOCK {Program b} BLOCK => (d:DEC ';')* (s:STMT ';')+ {Block d s} DEC => 'var' i:ID ('=' e:EXP)? {Var i e} | 'fun' i:ID '(' p:IDLIST? ')' '=' e:EXP {Fun i p e} IDLIST => i:ID (',' i:ID)* The value of IDLIST is not an AST node; it's just a list since the last thing evaluated was stored in i Value of d is a list of values from each DEC Value of PROGRAM is an AST node with root 'Program; We're purposely using the same variable twice

50. Abstract Syntax EXP => t1:TERM (o=[+-] t2=TERM t1={o t1 t2})* TERM => f1:FACTOR (o=[*/] f2=FACTOR f1={o f1 f2})* FACTOR => n:NUMLIT {Numlit n} | s:STRLIT {Strlit s} | i:ID {Ref i} | c:CALL | '(' e:EXP ')' Each time we iterate through the ([*/] FACTOR)* syntax element, the vaues of the variables o and f1 are reassigned. Here o refers to the Variable because it Is lowercase It's okay that some of the alternatives produce AST nodes and some do not

51. Summary of Conventions Rules found automatically, no delimiters

52. Maximal munch assumed

53. SKIP is just another rule

54. Token set inferred

55. => implies SKIP* separators

56. AST variables lowercase; nodes capitalized

57. Value of abstract syntax object is value of last object encountered in L-R parse

58. More The specification written in itself is in the paper

59. C, JSON, other specifications on web (http://xlg.cs.lmu.edu/ssd/)

60. Tool development ongoing

61. Summary Introduced syntax notation suitable for both humans and parser generators

62. Added custom features to EBNF

63. Defined conventions to simplify the notation

64. Provided examples of use

65. Related Work Krahn, Rumpe V ö kel (2007) Integrated definition of concrete and abstract syntax Van Wyk and Schwerdfeger (2007) Context-aware scanning LR generators (e.g. AnaGram, SableCC) LALR parser, AST nodes introduced with '='

66. Future Work Lookahead (for macrosyntax only) 'if' EXP (@2 'else' 'if' ...)* ('else' ...)?

67. Full syntactic lookahead? Microsyntax lookahead (or greedy qualifiers) '/*' ([^*] | *(?!/))* '*/' '/*' [^]*? '*/' Alternatives to maximal munch? (for Java >>)

68. Ultimate convention: AST nodes automatically generated according to syntax category