Presentation about the journey towards creating a new formatter with Xtext's new formatter2 API. Highlights the migration plan, problems, and our experience with the new API.
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API
1. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France
2017-06-21
2. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext
formatter with the formatter2 API
ir. Titouan Vervack
EclipseCon France, Toulouse
2017-06-21
15. Our formatter problems
● Big grammar
○ ~1.2k LoC
○ ~200 rules
● Hard to fix bugs
● Near impossible to add new features
10
16. Formatter 1.0 vs Formatter 2.0
11
Formatter 1.0 Formatter 2.0
● No access to
○ Node model
○ AST
○ Text
● => Specify everything purely on the grammar
● => Lots of hacks
● Access to
○ Node model
○ AST
○ Grammar
● Can’t take context into account
○ Can’t adjust to the user
● Context aware formatting
○ Conditional formatting
○ Table formatting
● Not customizable ● Very customizable
17. Formatter 2.0 region example
variable _ foo _ : _ bit _ ; _-- barn
variable foo : bit; -- bar
= ISemanticRegion
= IHiddenRegion
_ -- bar n
= IHiddenRegionPart
12
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
18. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
13
19. Formatting 2.0 usage example
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 {
}
13
28. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
14
29. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
14
30. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
14
31. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
14
32. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
6. Feature parity & extend with new features
14
33. Time for a plan
1. Make tests compatible
2. Start out easy: only indentation
3. Fix tests one by one
4. Fix old bugs
5. Do not introduce new bugs
6. Feature parity & extend with new features
7. Performance tweaking to same or better level than old
14
36. Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
val request = new FormatterRequest()
val parsed = new XtextResource()
parsed.contents.add(parseResult.rootASTElement)
parsed.setParseResult(parseResult)
val access = builder.forNodeModel(resource).create()
request.setTextRegionAccess(access)
val replacements = formatter.format(request)
access.rewriter.renderToString(replacements)
15
47. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
19
48. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
19
49. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
19
50. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
19
51. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
19
52. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
○ Formatter tags
19
53. Actual timeline
6. Feature parity & extend formatter with new features
● Parity:
○ Keyword casing
○ Comment alignment to columns
○ Vertical alignment on keywords “:”, “:=”,...
○ Preserve newline
● New features:
○ Correct indentation
○ Formatter tags
Easy once we have alignment
19
57. Alignment?
● Replacements are added in random order, applied in correct order
● Alignment needs to be done on post-formatted input
21
58. Alignment?
● Replacements are added in random order, applied in correct order
● Alignment needs to be done on post-formatted input
● How do I know which region to format?
21
69. PortDeclaration
Alignment!
clk : in std_logic := ‘X’;
start end
name direction type value
end.getSemanticElement()
PortDeclaration
PortList
23
70. PortDeclaration
Alignment!
clk : in std_logic := ‘X’;
start end
name direction type value
offset x
x + 4
end.getSemanticElement()
PortDeclaration
PortList
23
71. val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
Format smallest encapsulating object
24
72. val rule = portDecl.getParserRule()
val oldAccess = createTextRegionAccess(rule, portDecl.text)
val formatted = portDecl.format()
Alignment!
val newAccess = createTextRegionAccess(rule, formatted.text)
val oldIndices = oldAccess.getIndices(start, end)
val length = newAccess.countBetween(oldIndices)
Format smallest encapsulating object
Map unformatted string to formatted string
24
73. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
25
74. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
25
75. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
25
76. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
25
77. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
25
78. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
25
81. Align on first parameter?
→return (
→→foo((5,
→→·····6),
→→····3));
→return (5,
→········foo((5,
→·············6),
→············3));
→return foo((5,
→············6),
→···········3));
→return foo(
→→(5,
→→·6),
→→3));
Find largest encapsulating element on same line
25
83. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
26
84. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
]
26
85. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
r.lineCount > 1
]
26
86. Actual timeline
6. Feature parity & extend formatter with new features
Correct indentation
val replacements = formatter.format(request)
replacements
.filter[ r|
r.lineCount > 1 &&
r.replacementText.newlines + 1 == r.lineCount
]
26
88. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
27
89. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
27
90. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
27
91. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
27
92. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
val newRegions = originalRegions.removeRegions(betweenRegions)
27
93. Actual timeline
6. Feature parity & extend formatter with new features
Formatter tags
-- @formatter:off Turns formatter off
-- @formatter:on Turns formatter on
val originalRegions = request.getRegions() ?: document.region
val betweenRegions = document.findRegionsBetween(off, on)
.addAll(document.findRegionsBetween(off, EOF))
val newRegions = originalRegions.removeRegions(betweenRegions)
request.setRegions(newRegions)
27
96. Performance
7. Performance tweaking to same or better level than old
● Cache ITextRegionAccess in subformats
● Cache IParser.createAllRules(Grammar)
28
97. Performance
7. Performance tweaking to same or better level than old
● Cache ITextRegionAccess in subformats
● Cache IParser.createAllRules(Grammar)
● Merge all replacements into one big replacement
28
102. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
31
103. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
31
104. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31
105. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31
106. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31
107. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
31
108. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031
109. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031
110. Biggest problems in hindsight
● Tests pass, close enough?
○ Nope, old testset isn’t nearly enough
● Comments are an enormous pain: eat newlines, add spaces, eat
spaces, fake empty lines
● Alignment interactions with normal and first param alignments and
comments
3031
111. Warning to Xtext users
Default ExceptionHandler logs DSL source code to stderr
32
122. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
34
123. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
34
124. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
34
125. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
● Very slow Xtend formatter source file
34
126. Xtext wish list
● Accept my PRs (tags, correct indent, performance)
● Fix/discuss my issues
● Fix Xtend formatter issues
● Very slow Xtend formatter source file
● Add pre- and post format hooks in format(FormatterRequest)
34
133. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
35
134. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
35
135. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
35
136. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
○ Very readable
35
137. Conclusion
● Context aware
● Debugable
● Good performance
● Very customizable
● User friendly
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
● Happy with the switch
○ New features
○ Very readable
○ Conditional formatting
35