Presentation about the journey towards creating a new formatter with Xtext's new formatter2 API. Highlights the migration plan, problems, and our experience with the new API.

1. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France
2017-06-21

2. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France, Toulouse
2017-06-21

3. Ghent
2

4. What do we do?
VHDL & SV IDE
3

5. What do we do?
Advanced analysis and linting
4

6. What do we do?
Visualisation
5

7. What do we do?
Visualisation
6

8. What do we do?
Eat cake
7

9. What do we do?
And cookies...
8

10. What do we do?
And occasionally… formatting
9

11. What do we do?
And occasionally… formatting
9

12. Our formatter problems
10

13. Our formatter problems
● Big grammar
○ ~1.2k LoC
○ ~200 rules
10

14. Our formatter problems
● Big grammar
○ ~1.2k LoC
○ ~200 rules
● Hard to fix bugs
10

15. Our formatter problems
● Big grammar
○ ~1.2k LoC
○ ~200 rules
● Hard to fix bugs
● Near impossible to add new features
10

16. Formatter 1.0 vs Formatter 2.0
11
Formatter 1.0 Formatter 2.0
● No access to
○ Node model
○ AST
○ Text
● => Specify everything purely on the grammar
● => Lots of hacks
● Access to
○ Node model
○ AST
○ Grammar
● Can’t take context into account
○ Can’t adjust to the user
● Context aware formatting
○ Conditional formatting
○ Table formatting
● Not customizable ● Very customizable

17. Formatter 2.0 region example
variable _ foo _ : _ bit _ ; _-- barn
variable foo : bit; -- bar
= ISemanticRegion
= IHiddenRegion
_ -- bar n
= IHiddenRegionPart
12
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api

18. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
13

19. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
}
13

20. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
}
}
13

21. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
}
}
13

22. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
fd.declarations.forEach[format]
}
}
13

23. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
fd.declarations.forEach[format]
fd.regionFor.keyword(“;”)
}
}
13

24. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
fd.declarations.forEach[format]
fd.regionFor.keyword(“;”)
.prepend[noSpace highPriority]
}
}
13

25. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) {
interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent]
fd.declarations.forEach[format]
fd.regionFor.keyword(“;”)
.prepend[noSpace highPriority]
.append[setNewlines(1, 1, 2)]
}
}
13

26. Time for a plan
14

27. Time for a plan
1. Make tests compatible
14

28. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
14

29. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
14

30. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
14

31. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
14

32. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
6. Feature parity & extend with new features
14

33. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
6. Feature parity & extend with new features
7. Performance tweaking to same or better level than old
14

34. Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
15

35. Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
15

36. Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
val request = new FormatterRequest()
val parsed = new XtextResource()
parsed.contents.add(parseResult.rootASTElement)
parsed.setParseResult(parseResult)
val access = builder.forNodeModel(resource).create()
request.setTextRegionAccess(access)
val replacements = formatter.format(request)
access.rewriter.renderToString(replacements)
15

37. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline
2. Start out easy: only indentation
16

38. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline
2. Start out easy: only indentation
● Indentation requires newline definitions
16

39. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline
2. Start out easy: only indentation
● Indentation requires newline definitions
● => replace all newlines with… the same amount of newlines
16

40. Actual timeline
3. Fix tests one by one
4. Fix old bugs
17

41. Actual timeline
3. Fix tests one by one
4. Fix old bugs
17

42. Actual timeline
5. Do not introduce new bugs
18

43. Actual timeline
5. Do not introduce new bugs
18

44. Actual timeline
6. Feature parity & extend formatter with new features
19

45. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
19

46. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
19

47. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
19

48. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
19

49. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
19

50. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
19

51. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
19

52. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
○ Formatter tags
19

53. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
○ Formatter tags
Easy once we have alignment
19

54. Alignment?
20

55. Alignment?
20

56. Alignment?
● Replacements are added in random order, applied in correct order
21

57. Alignment?
● Replacements are added in random order, applied in correct order
● Alignment needs to be done on post-formatted input
21

58. Alignment?
● Replacements are added in random order, applied in correct order
● Alignment needs to be done on post-formatted input
● How do I know which region to format?
21

59. Alignment!
22

60. Alignment!
22

61. Alignment!
22

62. Alignment!
22

63. Alignment!
start end
22

64. Alignment!
start end
22

65. Alignment!
Find smallest encapsulating object that
formats region between start and end
start end
22

66. Alignment!
Find smallest encapsulating object that
formats region between start and end
start end
22

67. Alignment!
clk : in std_logic := ‘X’;
23

68. Alignment!
clk : in std_logic := ‘X’;
start end
23

69. PortDeclaration
Alignment!
clk : in std_logic := ‘X’;
start end
name direction type value
end.getSemanticElement()
PortDeclaration
PortList
23

70. PortDeclaration
Alignment!
clk : in std_logic := ‘X’;
start end
name direction type value
offset x
x + 4
end.getSemanticElement()
PortDeclaration
PortList
23

71. val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
Format smallest encapsulating object
24

72. val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
val newAccess = createTextRegionAccess(rule, formatted.text)
val oldIndices = oldAccess.getIndices(start, end)
val length = newAccess.countBetween(oldIndices)
Format smallest encapsulating object
Map unformatted string to formatted string
24

73. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
25

74. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
25

75. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
25

76. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
25

77. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
25

78. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
25

79. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
25

80. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
25

81. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
Find largest encapsulating element on same line
25

82. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
26

83. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
26

84. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
]
26

85. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
r.lineCount > 1
]
26

86. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
r.lineCount > 1 &&
r.replacementText.newlines + 1 == r.lineCount
]
26

87. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
27

88. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
27

89. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
27

90. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
27

91. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
27

92. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
val newRegions = originalRegions.removeRegions(betweenRegions)
27

93. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
val newRegions = originalRegions.removeRegions(betweenRegions)
request.setRegions(newRegions)
27

94. Performance
7. Performance tweaking to same or better level than old
28

95. Performance
7. Performance tweaking to same or better level than old
● Cache ITextRegionAccess in subformats
28

96. Performance
7. Performance tweaking to same or better level than old
● Cache ITextRegionAccess in subformats
● Cache IParser.createAllRules(Grammar)
28

97. Performance
7. Performance tweaking to same or better level than old
● Cache ITextRegionAccess in subformats
● Cache IParser.createAllRules(Grammar)
● Merge all replacements into one big replacement
28

98. Performance
7. Performance tweaking to same or better level than old
emacs/xtend > 25m
29

99. Performance
7. Performance tweaking to same or better level than old
emacs/xtend > 25m
30

100. Biggest problems in hindsight
31

101. Biggest problems in hindsight
● Tests pass, close enough?
31

102. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
31

103. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
31

104. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31

105. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31

106. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31

107. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31

108. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031

109. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031

110. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031

111. Warning to Xtext users
Default ExceptionHandler logs DSL source code to stderr
32

112. Sigasi future work
33

113. Sigasi future work
● Smooth out remaining bugs
33

114. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
33

115. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
33

116. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
● Performance
33

117. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
● Performance
● Upstream features
33

118. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
● Performance
● Upstream features
○ Formatter tags
33

119. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
● Performance
● Upstream features
○ Formatter tags
○ Correct indentation
33

120. Sigasi future work
● Smooth out remaining bugs
● Some more context awareness
● Autowrap
● Performance
● Upstream features
○ Formatter tags
○ Correct indentation
○ Performance optimizations
33

121. Xtext wish list
34

122. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
34

123. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
34

124. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
34

125. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
● Very slow Xtend formatter source file
34

126. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
● Very slow Xtend formatter source file
● Add pre- and post format hooks in format(FormatterRequest)
34

127. Conclusion
35

128. Conclusion
● Context aware
35

129. Conclusion
● Context aware
● Debugable
35

130. Conclusion
● Context aware
● Debugable
● Good performance
35

131. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
35

132. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
35

133. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
35

134. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
35

135. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
35

136. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
○ Very readable
35

137. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
○ Very readable
○ Conditional formatting
35

138. Evaluate the Sessions
-1 0 +1
Sign in and vote at eclipsecon.org