This document discusses syntactic editor services including formatting, syntax coloring, and syntactic completion. It describes how syntactic completion can be provided generically based on a syntax definition. The document also discusses how context-free grammars can be extended with templates to specify formatting layout when pretty-printing abstract syntax trees to text. Templates are used to insert whitespace, line breaks, and indentation to produce readable output.

1. Lecture 4: Syntactic Editor Services
CS4200 Compiler Construction
Eelco Visser
TU Delft
September 2019

2. Syntactic editor services
- more interpretations of syntax definitions
Formatting specification
- how to map (abstract syntax) trees to text
Syntactic completion
- proposing valid syntactic completions in an editor
!2
This Lecture

3. Reading Material
3

4. !4
The inverse of parsing is unparsing or pretty-printing or
formatting, i.e. mapping a tree representation of a program
to a textual representation. A plain context-free grammar
can be used as specification of an unparser. However, then
it is unclear where the whitespace should go.
This paper extends context-free grammars with templates
that provide hints for layout of program text when
formatting.
https://doi.org/10.1145/2427048.2427056
Based on master’s thesis project of Tobi Vollebregt

5. !5
Syntax definitions cannot be used just for parsing, but
for many other operations. This paper shows how
syntactic completion can be provided generically given
a syntax definition.
https://doi.org/10.1145/2427048.2427056
Part of PhD thesis work Eduardo Amorim

6. !6
The SDF3 syntax
definition formalism is
documented at the
metaborg.org website.
http://www.metaborg.org/en/latest/source/langdev/meta/lang/sdf3/index.html

7. Syntactic Editor Services
7

8. Editor Services
!8

9. Editor Services
!9
Source
Code
Editor
Parse
Feedback
& Operations
Abstract
Syntax
Tree
Feedback
- syntax coloring
- syntax checking
- outline view
Operations
- syntactic completion
- formatting
- abstract syntax tree

10. Language Project Configuration (ESV)
!10
module Main
imports
Syntax
Analysis
language
extensions : tig
//provider : target/metaborg/stratego.ctree
provider : target/metaborg/stratego.jar
provider : target/metaborg/stratego-javastrat.jar
menus
menu: "Transform" (openeditor) (realtime)
action: "Desugar" = editor-desugar (source)
action: "Desugar AST" = editor-desugar-ast (source)

11. Configuration of Syntactic Services (ESV)
!11
module Syntax
imports
libspoofax/color/default
completion/colorer/Tiger-cc-esv
language
table : target/metaborg/sdf-new.tbl
//table : target/metaborg/sdf.tbl
start symbols : Module
line comment : "//"
block comment : "/*" * "*/"
fences : [ ] ( ) { }
menus
menu: "Syntax" (openeditor)
action: "Format" = editor-format (source)
action: "Show parsed AST" = debug-show-aterm (source)
views
outline view: editor-outline (source)
expand to level: 3

12. Syntax Coloring
12

13. Generated Syntax Coloring
!13
// compute the n-th fibonacci number
let function fib(n: int): int =
if n <= 1 then 1
else fib(n - 1) + fib(n - 2)
in fib(10)
end
module libspoofax/color/default
imports
libspoofax/color/colors
colorer // Default, token-based
highlighting
keyword : 127 0 85 bold
identifier : default
string : blue
number : darkgreen
var : 139 69 19 italic
operator : 0 0 128
layout : 63 127 95 italic

14. Customized Syntax Coloring
!14
module Tiger-Colorer
colorer
red = 255 0 0
green = 0 255 0
blue = 0 0 255
TUDlavender = 123 160 201
colorer token-based highlighting
keyword : red
Id : TUDlavender
StrConst : darkgreen
TypeId : blue
layout : green
// compute the n-th fibonacci number
let function fib(n: int): int =
if n <= 1 then 1
else fib(n - 1) + fib(n - 2)
in fib(10)
end

15. From Unparsers
to Pretty-Printers
with Templates
15

16. !16
The inverse of parsing is unparsing or pretty-printing or
formatting, i.e. mapping a tree representation of a program
to a textual representation. A plain context-free grammar
can be used as specification of an unparser. However, then
it is unclear where the whitespace should go.
This paper extends context-free grammars with templates
that provide hints for layout of program text when
formatting.
https://doi.org/10.1145/2427048.2427056
Based on master’s thesis project of Tobi Vollebregt

17. Converting between Text and Tree Representations
!17
parse
format

18. Unparsing: From Abstract Syntax Term to Text
!18
rules
unparse :
Int(x) -> x
unparse :
Plus(e1, e2) -> $[[<unparse> e1] + [<unparse> e2]]
unparse :
Times(e1, e2) -> $[[<unparse> e1] * [<unparse> e2]]
Mod(
Plus(
Int("1")
, Times(
Int("2")
, Plus(Int("3"), Int("4"))
)
)
)
1 + 2 * 3 + 4
take priorities into account!
context-free syntax
Exp.Int = IntConst
Exp.Times = Exp “*" Exp
Exp.Plus = Exp “+" Exp

19. From ASTs to text
- insert keywords
- insert layout: spaces, line breaks, indentation
- insert parentheses to preserve tree structure
Unparser
- derive transformation rules from context-free grammar
- keywords, literals defined in grammar productions
- parentheses determined by priority, associativity rules
- separate all symbols by a space => not pretty, or even readable
Pretty-printer
- introduce spaces, line breaks, and indentation to produce readable text
- doing that manually is tedious
!19
Pretty-Printing

20. Specifying Formatting Layout with Templates
!20
context-free syntax
Exp.Seq = <
(
<{Exp ";n"}*>
)
>
Exp.If = <
if <Exp> then
<Exp>
else
<Exp>
>
Exp.IfThen = <
if <Exp> then
<Exp>
>
Exp.While = <
while <Exp> do
<Exp>
>
Inverse quotation
- template quotes literal text with <>
- anti-quotations insert non-terminals with <>
Layout directives
- whitespace (linebreaks, indentation, spaces) in template
guides formatting
- is interpreted as LAYOUT? for parsing
Formatter generation
- generate rules for mapping AST to text (via box expressions)
Applications
- code generation; pretty-printing generated AST
- syntactic completions
- formatting

21. Templates for Tiger: Binary Expressions
!21
context-free syntax
Exp.Int = IntConst
Exp.Uminus = [- [Exp]]
Exp.Times = [[Exp] * [Exp]] {left}
Exp.Divide = [[Exp] / [Exp]] {left}
Exp.Plus = [[Exp] + [Exp]] {left}
Exp.Minus = [[Exp] - [Exp]] {left}
Exp.Eq = [[Exp] = [Exp]] {non-assoc}
Exp.Neq = [[Exp] <> [Exp]] {non-assoc}
Exp.Gt = [[Exp] > [Exp]] {non-assoc}
Exp.Lt = [[Exp] < [Exp]] {non-assoc}
Exp.Geq = [[Exp] >= [Exp]] {non-assoc}
Exp.Leq = [[Exp] <= [Exp]] {non-assoc}
Exp.And = [[Exp] & [Exp]] {left}
Exp.Or = [[Exp] | [Exp]] {left}
Use [] quotes instead of
<> to avoid clash with
comparison operators

22. Templates for Tiger: Functions
!22
context-free syntax
Dec.FunDecs = <<{FunDec "n"}+>> {longest-match}
FunDec.ProcDec = <
function <Id>(<{FArg ", "}*>) =
<Exp>
>
FunDec.FunDec = <
function <Id>(<{FArg ", "}*>) : <Type> =
<Exp>
>
FArg.FArg = <<Id> : <Type>>
Exp.Call = <<Id>(<{Exp ", "}*>)>
No space after function name in call
Space after comma!
Function declarations
separated by newline
Indent body
of function

23. Templates for Tiger: Bindings and Records
!23
context-free syntax
Exp.Let = <
let
<{Dec "n"}*>
in
<{Exp ";n"}*>
end
>
context-free syntax // records
Type.RecordTy = <
{
<{Field ", n"}*>
}
>
Field.Field = <<Id> : <TypeId>>
Exp.NilExp = <nil>
Exp.Record = <<TypeId>{ <{InitField ", "}*> }>
InitField.InitField = <<Id> = <Exp>>
LValue.FieldVar = <<LValue>.<Id>>
Note spacing / layout in separators

24. Generating Pretty-Print Rules from Template Productions
!24
context-free syntax
FunDec.FunDec = <
function <Id>(<{FArg ", "}*>) : <Type> =
<Exp>
>
rules
prettyprint-Tiger-FunDec :
ProcDec(t1__, t2__, t3__) -> [ H(
[SOpt(HS(), "0")]
, [ S("function ")
, t1__
, S("(")
, t2__'
, S(") =")
]
)
, t3__'
]
with t1__' := <pp-one-Z(prettyprint-Tiger-Id) <+ pp-one-Z(prettyprint-completion-aux)> t1__
with t2__' := <pp-H-list(prettyprint-Tiger-FArg|", ")
<+ pp-one-Z(prettyprint-completion-aux)> t2__
with t3__' := <pp-indent(|"2")> [ <pp-one-Z(prettyprint-Tiger-Exp) <+ pp-one-Z(prettyprint-completion-aux)> t3__ ]
Separation of concerns:
- generated formatter transforms AST to Box
- Box formatter produces text

25. Boxes for Formatting
!25
_1
“foo”
KW [ “foo” ]
literal text, keywords, parameters

26. Horizontal Layout
!26
B B B
H hs=x [ ]B B Bhs=x
hs: horizontal space between boxes

27. Vertical Layout
!27
V hs=x is=i [ ]B B Bvs=y is=i
B
B
B
vs: vertical space between boxes; is: indentation space

28. Tiger Syntax Definition
with Templates
28

29. Tiger Syntax: Composition
!29
module Tiger
imports Whitespace
imports Comments
imports Types
imports Identifiers
imports Bindings
imports Variables
imports Functions
imports Numbers
imports Strings
imports Records
imports Arrays
imports Control-Flow
context-free start-symbols Module
context-free syntax
Module.Mod = Exp
context-free priorities
Exp.Or > Exp.Array > Exp.Assign ,
{Exp.Uminus LValue.FieldVar LValue.Subscript}
> {left : Exp.Times Exp.Divide}

30. Tiger Syntax: Identifiers and Strings
!30
module Identifiers
lexical syntax
Id = [a-zA-Z] [a-zA-Z0-9_]*
lexical restrictions
Id -/- [a-zA-Z0-9_]
lexical syntax
Id = "nil" {reject}
Id = "let" {reject}
Id = … {reject}
module Strings
sorts StrConst
lexical syntax
StrConst = """ StrChar* """
StrChar = ~["n]
StrChar = [] [n]
StrChar = [] [t]
StrChar = [] [^] [A-Z]
StrChar = [] [0-9] [0-9] [0-9]
StrChar = [] ["]
StrChar = [] []
StrChar = [] [ tn]+ []
context-free syntax // records
Exp.String = StrConst

31. Tiger Syntax: Whitespace & Comments
!31
module Whitespace
lexical syntax
LAYOUT = [ tnr]
context-free restrictions
LAYOUT? -/- [ tnr]
module Comments
lexical syntax // multiline comments
CommentChar = [*]
LAYOUT = "/*" InsideComment* "*/"
InsideComment = ~[*]
InsideComment = CommentChar
lexical restrictions
CommentChar -/- [/]
context-free restrictions
LAYOUT? -/- [/].[/]
lexical syntax // single line comments
LAYOUT = "//" ~[nr]* NewLineEOF
NewLineEOF = [nr]
NewLineEOF = EOF
EOF =
// end of file since it cannot be followed by any character
// avoids the need for a newline to close a single line comment
// at the last line of a file
lexical restrictions
EOF -/- ~[]
context-free restrictions
LAYOUT? -/- [/].[*]

32. Tiger Syntax: Numbers
!32
module Numbers
lexical syntax
IntConst = [0-9]+
context-free syntax
Exp.Int = IntConst
Exp.Uminus = [- [Exp]]
Exp.Times = [[Exp] * [Exp]] {left}
Exp.Divide = [[Exp] / [Exp]] {left}
Exp.Plus = [[Exp] + [Exp]] {left}
Exp.Minus = [[Exp] - [Exp]] {left}
Exp.Eq = [[Exp] = [Exp]] {non-assoc}
Exp.Neq = [[Exp] <> [Exp]] {non-assoc}
Exp.Gt = [[Exp] > [Exp]] {non-assoc}
Exp.Lt = [[Exp] < [Exp]] {non-assoc}
Exp.Geq = [[Exp] >= [Exp]] {non-assoc}
Exp.Leq = [[Exp] <= [Exp]] {non-assoc}
Exp.And = [[Exp] & [Exp]] {left}
Exp.Or = [[Exp] | [Exp]] {left}
context-free
priorities
{Exp.Uminus}
> {left :
Exp.Times
Exp.Divide}
> {left :
Exp.Plus
Exp.Minus}
> {non-assoc :
Exp.Eq
Exp.Neq
Exp.Gt
Exp.Lt
Exp.Geq
Exp.Leq}
> Exp.And
> Exp.Or

33. Tiger Syntax: Variables and Functions
!33
module Bindings
imports Control-Flow
imports Identifiers
imports Types
imports Functions
imports Variables
sorts Declarations
context-free syntax
Exp.Let = <
let
<{Dec "n"}*>
in
<{Exp ";n"}*>
end
>
Declarations.Declarations = <
declarations <{Dec "n"}*>
>
module Variables
imports Identifiers
imports Types
sorts Var
context-free syntax
Dec.VarDec = <var <Id> : <Type> := <Exp>>
Dec.VarDecNoType = <var <Id> := <Exp>>
Var.Var = Id
LValue = Var
Exp = LValue
Exp.Assign = <<LValue> := <Exp>>
module Functions
imports Identifiers
imports Types
context-free syntax
Dec.FunDecs = <<{FunDec "n"}+>> {longest-match}
FunDec.ProcDec = <
function <Id>(<{FArg ", "}*>) =
<Exp>
>
FunDec.FunDec = <
function <Id>(<{FArg ", "}*>) : <Type> =
<Exp>
>
FArg.FArg = <<Id> : <Type>>
Exp.Call = <<Id>(<{Exp ", "}*>)>

34. Tiger Syntax: Records, Arrays, Types
!34
module Records
imports Base
imports Identifiers
imports Types
context-free syntax // records
Type.RecordTy = <
{
<{Field ", n"}*>
}
>
Field.Field = <<Id> : <TypeId>>
Exp.NilExp = <nil>
Exp.Record = <<TypeId>{ <{InitField ", "}*> }>
InitField.InitField = <<Id> = <Exp>>
LValue.FieldVar = <<LValue>.<Id>> module Arrays
imports Types
context-free syntax // arrays
Type.ArrayTy = <array of <TypeId>>
Exp.Array = <<TypeId>[<Exp>] of <Exp>>
LValue.Subscript = <<LValue>[<Index>]>
Index = Exp
module Types
imports Identifiers
imports Bindings
sorts Type
context-free syntax // type declarations
Dec.TypeDecs = <<{TypeDec "n"}+>> {longest-match}
TypeDec.TypeDec = <type <Id> = <Type>>
context-free syntax // type expressions
Type = TypeId
TypeId.Tid = Id
sorts Ty
context-free syntax // semantic types
Ty.INT = <INT>
Ty.STRING = <STRING>
Ty.NIL = <NIL>
Ty.UNIT = <UNIT>
Ty.NAME = <NAME <Id>>
Ty.RECORD = <RECORD <Id>>
Ty.ARRAY = <ARRAY <Ty> <Id>>
Ty.FUN = <FUN ( <{Ty ","}*> ) <Ty>>

35. Syntactic Completion
35

36. !36
Syntax definitions cannot be used just for parsing, but
for many other operations. This paper shows how
syntactic completion can be provided generically given
a syntax definition.
https://doi.org/10.1145/2427048.2427056
Part of PhD thesis work Eduardo Amorim

37. !37
S. Amann, S. Proksch, S. Nadi, and M. Mezini. A study of
visual studio usage in practice. In SANER, 2016.

38. !38
Syntactic
Completion
Semantic
Completion

39. Problems
39
Ad-hoc: re-implement for each language / IDE
Incomplete: not all programs reachableUnsound: propose invalid constructs

40. Sound and Complete
Syntactic Code Completion
from Syntax Definition
40

41. Explicit Placeholders
41

42. !42
Incomplete
programs

43. !43
Incomplete
programs
Make incompleteness explicit using placeholders!

44. 44

45. !45

46. !46

47. !47

48. !48
4
2
3
1

49. Deriving Syntactic Completion from Syntax Definition
!49
context-free syntax // regular production
Statement.If = "if" "(" Exp ")" Statement "else" Statement

50. Placeholders as Language Constructs
!50
context-free syntax // placeholder rule
Statement.Statement-Plhdr = "$Statement"
context-free syntax // regular production
Statement.If = "if" "(" Exp ")" Statement "else" Statement

51. Calculate Placeholder Expansions
!51
context-free syntax // regular production
Statement.If = "if" "(" Exp ")" Statement "else" Statement
Stm-Plhdr
If
Exp-Plhdr Stm-Plhdr
Stm-Plhdr
context-free syntax // placeholder rule
Statement.Statement-Plhdr = “$Statement"
rules
rewrite-placeholder:
Statement-Plhdr() -> If(Exp-Plhdr(), Statement-Plhdr(),
Statement-Plhdr())

52. Calculate Placeholder Expansions
!52
context-free syntax // regular production
Statement.If = "if" "(" Exp ")" Statement "else" Statement
context-free syntax // placeholder rule
Statement.Statement-Plhdr = “$Statement"
rules
rewrite-placeholder:
Statement-Plhdr() -> If(Exp-Plhdr(), Statement-Plhdr(),
Statement-Plhdr())

53. !53
Incomplete
programs
Complete
programs
Expand
placeholder
Expand/overwrite
placeholders
Correct
programs

54. !54
Complete
programs
How to expand a complete program?

55. !55
Manually insert a placeholder?
How to expand a complete program?

56. Inferring Placeholders
56

57. Placeholder Inference
!57
Optional Parent
List of Statements
Add Optional Element
No source region
Add first or last element
Add Element to List

58. Placeholder Inference: Optional
ClassDecl
class “A” {NoParent ConsMethod
[…]
}NilField
58

59. Placeholder Inference: Optional
59
ClassDecl
class “A” {NoParent ConsMethod
[…]
}NilField

60. Placeholder Inference: Optional
60
ClassDecl
class “A” {NoParent ConsMethod
[…]
}NilField

61. Placeholder Inference: Optional
61

62. Placeholder Inference - Lists
62
[…]
[…]
Method
Cons
VarDecl
AssignInt “x”
VarRef
“x”
Add
Int
21
Cons
Nil
{ return
[…]
int ;
Int
=
21

63. Placeholder Inference - Lists
63
[…]
[…]
Method
Cons
VarDecl
AssignInt “x”
VarRef
“x”
Add
Int
21
Cons
Nil
{ return
[…]
int ;
Int
=
21

64. Placeholder Inference - Lists
64
[…]
[…]
Method
Cons
VarDecl
AssignInt “x”
VarRef
“x”
Add
Int
21
Cons
Nil
{ return
[…]
int ;
Int
=
21

65. Placeholder Inference - Lists
65
[…]
[…]
Method
Cons
VarDecl
AssignInt “x”
VarRef
“x”
Add
Int
21
Cons
Nil
{ return
[…]
int ;
Int
=
21

66. Completion Proposals for Inferred Placeholders
66

67. !67
Incomplete
programs
Complete
programs
Expand
placeholder
Infer
placeholder
Correct
programs
Infer
placeholder
Expand/overwrite
placeholders

68. 68
Incorrect
programs

69. Error Recovery
69

70. !70

71. Insertion Rules
!71
context-free syntax //regular syntax rules
Statement.VarDecl = <<Type> <ID>;>
Statement.Assign = <<VarRef> = <Exp>;>
// derived insertion rules for placeholders
context-free syntax
Type.Type-Plhdr = {symbol-insertion}
ID.ID-Plhdr = {symbol-insertion}
Statement.Statement-Plhdr = {symbol-insertion}
VarRef.VarRef-Plhdr = {symbol-insertion}
Exp.Exp-Plhdr = {symbol-insertion}
// derived insertion rules for literals
lexical syntax
"=" = {symbol-completion}
";" = {symbol-completion}
Empty productions

72. Apply Insertion Rules at Cursor
72
Proposal nodes
Insertion nodes
[…]
[…]
Stmt*
amb
VarDecl
ClassType
“x”
VarDecl
[…]
Assign
ID-Plhdr ;
Assign
VarRef
“x”
Exp-Plhdr ;
=

73. Limit Search Space
!73
Exp-Plhdr Exp-Plhdr
+
Assign
VarRef
“x”
;
=
[…]
Add
Assign
VarRef
“x”
;=
[…]
Exp-Plhdr
Use the simplest possible expansions

74. Greedy Recovery
!74
[…]
[…]
Stmt*
amb
VarDecl
ClassType
“x”
VarDecl
[…]
Assign
ID-Plhdr ;
Assign
VarRef
“x”
Exp-Plhdr ;
=
Include postfix in recovery proposal

75. Nested Proposal Nodes
75
IntValue Exp-Plhdr+
Assign
VarRef
“x”
;=
[…]
Add
1

76. 76

77. Syntactic Completion
77

78. Incorrect
programs
Incomplete
programs
Complete
programs
Correct
programs
78

79. Incorrect
programs
Incomplete
programs
Complete
programs
Expand
placeholder
Correct
programs
Expand/overwrite
placeholders
79

80. Incorrect
programs
Incomplete
programs
Complete
programs
Expand
placeholder
Infer
placeholder
Correct
programs
Infer
placeholder
Expand/overwrite
placeholders
80

81. Incorrect
programs
Incomplete
programs
Complete
programs
Expand
placeholder
Infer
placeholder
Correct
programs
Infer
placeholder
Expand/overwrite
placeholders
Recover
incomplete
program
Recover
complete
program
81

82. Syntactic Services:
Summary
82

83. Syntax definition = CFG++
- Concrete syntax: notation
- Constructors: abstract syntax / structure
- Associativity & priority: disambiguation
- Templates: formatting
Parsing
- Mapping text to abstract syntax
- Syntax checking
- Permissive grammars: error recovery
!83
Syntactic Services from Syntax Definition

84. Syntax coloring
- ESV: mapping token sorts to colors
Formatting
- Unparsing: mapping from abstract syntax to concrete syntax
- Pretty-printing: derived from templates
- Parenthesization: derived from disambiguation declarations
Completion
- Make incompleteness explicit, part of the structure
- Generating proposals: derived from structure
- Pretty-printing proposals: derived from templates
!84
Syntactic Services from Syntax Definition

85. Next: Transformation
85

86. Except where otherwise noted, this work is licensed under