Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API

578 views

Published on

Presentation about the journey towards creating a new formatter with Xtext's new formatter2 API. Highlights the migration plan, problems, and our experience with the new API.

Published in: Software
  • Be the first to comment

  • Be the first to like this

EclipseCon France 2017 - Xtending Our Vhdl Xtext Formatter With The Formatter2 API

  1. 1. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog. Xtending our VHDL Xtext formatter with the formatter2 API ir. Titouan Vervack EclipseCon France 2017-06-21
  2. 2. redefine.digital.design: Helping you deal with complexity in VHDL and Verilog. Xtending our VHDL Xtext formatter with the formatter2 API ir. Titouan Vervack EclipseCon France, Toulouse 2017-06-21
  3. 3. Ghent 2
  4. 4. What do we do? VHDL & SV IDE 3
  5. 5. What do we do? Advanced analysis and linting 4
  6. 6. What do we do? Visualisation 5
  7. 7. What do we do? Visualisation 6
  8. 8. What do we do? Eat cake 7
  9. 9. What do we do? And cookies... 8
  10. 10. What do we do? And occasionally… formatting 9
  11. 11. What do we do? And occasionally… formatting 9
  12. 12. Our formatter problems 10
  13. 13. Our formatter problems ● Big grammar ○ ~1.2k LoC ○ ~200 rules 10
  14. 14. Our formatter problems ● Big grammar ○ ~1.2k LoC ○ ~200 rules ● Hard to fix bugs 10
  15. 15. Our formatter problems ● Big grammar ○ ~1.2k LoC ○ ~200 rules ● Hard to fix bugs ● Near impossible to add new features 10
  16. 16. Formatter 1.0 vs Formatter 2.0 11 Formatter 1.0 Formatter 2.0 ● No access to ○ Node model ○ AST ○ Text ● => Specify everything purely on the grammar ● => Lots of hacks ● Access to ○ Node model ○ AST ○ Grammar ● Can’t take context into account ○ Can’t adjust to the user ● Context aware formatting ○ Conditional formatting ○ Table formatting ● Not customizable ● Very customizable
  17. 17. Formatter 2.0 region example variable _ foo _ : _ bit _ ; _-- barn variable foo : bit; -- bar = ISemanticRegion = IHiddenRegion _ -- bar n = IHiddenRegionPart 12 https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
  18. 18. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api 13
  19. 19. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { } 13
  20. 20. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { } } 13
  21. 21. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] } } 13
  22. 22. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] } } 13
  23. 23. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) } } 13
  24. 24. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) .prepend[noSpace highPriority] } } 13
  25. 25. Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) .prepend[noSpace highPriority] .append[setNewlines(1, 1, 2)] } } 13
  26. 26. Time for a plan 14
  27. 27. Time for a plan 1. Make tests compatible 14
  28. 28. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 14
  29. 29. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 14
  30. 30. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 14
  31. 31. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs 14
  32. 32. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs 6. Feature parity & extend with new features 14
  33. 33. Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs 6. Feature parity & extend with new features 7. Performance tweaking to same or better level than old 14
  34. 34. Actual timeline formatter.format(node, node.offset, node.length).formattedText 1. Make tests compatible 15
  35. 35. Actual timeline formatter.format(node, node.offset, node.length).formattedText 1. Make tests compatible 15
  36. 36. Actual timeline formatter.format(node, node.offset, node.length).formattedText 1. Make tests compatible val request = new FormatterRequest() val parsed = new XtextResource() parsed.contents.add(parseResult.rootASTElement) parsed.setParseResult(parseResult) val access = builder.forNodeModel(resource).create() request.setTextRegionAccess(access) val replacements = formatter.format(request) access.rewriter.renderToString(replacements) 15
  37. 37. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent] Actual timeline 2. Start out easy: only indentation 16
  38. 38. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent] Actual timeline 2. Start out easy: only indentation ● Indentation requires newline definitions 16
  39. 39. interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent] Actual timeline 2. Start out easy: only indentation ● Indentation requires newline definitions ● => replace all newlines with… the same amount of newlines 16
  40. 40. Actual timeline 3. Fix tests one by one 4. Fix old bugs 17
  41. 41. Actual timeline 3. Fix tests one by one 4. Fix old bugs 17
  42. 42. Actual timeline 5. Do not introduce new bugs 18
  43. 43. Actual timeline 5. Do not introduce new bugs 18
  44. 44. Actual timeline 6. Feature parity & extend formatter with new features 19
  45. 45. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: 19
  46. 46. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing 19
  47. 47. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns 19
  48. 48. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... 19
  49. 49. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline 19
  50. 50. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline ● New features: 19
  51. 51. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline ● New features: ○ Correct indentation 19
  52. 52. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline ● New features: ○ Correct indentation ○ Formatter tags 19
  53. 53. Actual timeline 6. Feature parity & extend formatter with new features ● Parity: ○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline ● New features: ○ Correct indentation ○ Formatter tags Easy once we have alignment 19
  54. 54. Alignment? 20
  55. 55. Alignment? 20
  56. 56. Alignment? ● Replacements are added in random order, applied in correct order 21
  57. 57. Alignment? ● Replacements are added in random order, applied in correct order ● Alignment needs to be done on post-formatted input 21
  58. 58. Alignment? ● Replacements are added in random order, applied in correct order ● Alignment needs to be done on post-formatted input ● How do I know which region to format? 21
  59. 59. Alignment! 22
  60. 60. Alignment! 22
  61. 61. Alignment! 22
  62. 62. Alignment! 22
  63. 63. Alignment! start end 22
  64. 64. Alignment! start end 22
  65. 65. Alignment! Find smallest encapsulating object that formats region between start and end start end 22
  66. 66. Alignment! Find smallest encapsulating object that formats region between start and end start end 22
  67. 67. Alignment! clk : in std_logic := ‘X’; 23
  68. 68. Alignment! clk : in std_logic := ‘X’; start end 23
  69. 69. PortDeclaration Alignment! clk : in std_logic := ‘X’; start end name direction type value end.getSemanticElement() PortDeclaration PortList 23
  70. 70. PortDeclaration Alignment! clk : in std_logic := ‘X’; start end name direction type value offset x x + 4 end.getSemanticElement() PortDeclaration PortList 23
  71. 71. val rule = portDecl.getParserRule() val oldAccess = createTextRegionAccess(rule, portDecl.text) val formatted = portDecl.format() Alignment! Format smallest encapsulating object 24
  72. 72. val rule = portDecl.getParserRule() val oldAccess = createTextRegionAccess(rule, portDecl.text) val formatted = portDecl.format() Alignment! val newAccess = createTextRegionAccess(rule, formatted.text) val oldIndices = oldAccess.getIndices(start, end) val length = newAccess.countBetween(oldIndices) Format smallest encapsulating object Map unformatted string to formatted string 24
  73. 73. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); 25
  74. 74. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); 25
  75. 75. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); 25
  76. 76. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); 25
  77. 77. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); 25
  78. 78. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); 25
  79. 79. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3)); 25
  80. 80. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3)); 25
  81. 81. Align on first parameter? →return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3)); Find largest encapsulating element on same line 25
  82. 82. Actual timeline 6. Feature parity & extend formatter with new features Correct indentation 26
  83. 83. Actual timeline 6. Feature parity & extend formatter with new features Correct indentation val replacements = formatter.format(request) 26
  84. 84. Actual timeline 6. Feature parity & extend formatter with new features Correct indentation val replacements = formatter.format(request) replacements .filter[ r| ] 26
  85. 85. Actual timeline 6. Feature parity & extend formatter with new features Correct indentation val replacements = formatter.format(request) replacements .filter[ r| r.lineCount > 1 ] 26
  86. 86. Actual timeline 6. Feature parity & extend formatter with new features Correct indentation val replacements = formatter.format(request) replacements .filter[ r| r.lineCount > 1 && r.replacementText.newlines + 1 == r.lineCount ] 26
  87. 87. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags 27
  88. 88. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on 27
  89. 89. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on val originalRegions = request.getRegions() ?: document.region 27
  90. 90. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) 27
  91. 91. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF)) 27
  92. 92. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF)) val newRegions = originalRegions.removeRegions(betweenRegions) 27
  93. 93. Actual timeline 6. Feature parity & extend formatter with new features Formatter tags -- @formatter:off Turns formatter off -- @formatter:on Turns formatter on val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF)) val newRegions = originalRegions.removeRegions(betweenRegions) request.setRegions(newRegions) 27
  94. 94. Performance 7. Performance tweaking to same or better level than old 28
  95. 95. Performance 7. Performance tweaking to same or better level than old ● Cache ITextRegionAccess in subformats 28
  96. 96. Performance 7. Performance tweaking to same or better level than old ● Cache ITextRegionAccess in subformats ● Cache IParser.createAllRules(Grammar) 28
  97. 97. Performance 7. Performance tweaking to same or better level than old ● Cache ITextRegionAccess in subformats ● Cache IParser.createAllRules(Grammar) ● Merge all replacements into one big replacement 28
  98. 98. Performance 7. Performance tweaking to same or better level than old emacs/xtend > 25m 29
  99. 99. Performance 7. Performance tweaking to same or better level than old emacs/xtend > 25m 30
  100. 100. Biggest problems in hindsight 31
  101. 101. Biggest problems in hindsight ● Tests pass, close enough? 31
  102. 102. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough 31
  103. 103. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines 31
  104. 104. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 31
  105. 105. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 31
  106. 106. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 31
  107. 107. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 31
  108. 108. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 3031
  109. 109. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 3031
  110. 110. Biggest problems in hindsight ● Tests pass, close enough? ○ Nope, old testset isn’t nearly enough ● Comments are an enormous pain: eat newlines, add spaces, eat spaces, fake empty lines ● Alignment interactions with normal and first param alignments and comments 3031
  111. 111. Warning to Xtext users Default ExceptionHandler logs DSL source code to stderr 32
  112. 112. Sigasi future work 33
  113. 113. Sigasi future work ● Smooth out remaining bugs 33
  114. 114. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness 33
  115. 115. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap 33
  116. 116. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap ● Performance 33
  117. 117. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap ● Performance ● Upstream features 33
  118. 118. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap ● Performance ● Upstream features ○ Formatter tags 33
  119. 119. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap ● Performance ● Upstream features ○ Formatter tags ○ Correct indentation 33
  120. 120. Sigasi future work ● Smooth out remaining bugs ● Some more context awareness ● Autowrap ● Performance ● Upstream features ○ Formatter tags ○ Correct indentation ○ Performance optimizations 33
  121. 121. Xtext wish list 34
  122. 122. Xtext wish list ● Accept my PRs (tags, correct indent, performance) 34
  123. 123. Xtext wish list ● Accept my PRs (tags, correct indent, performance) ● Fix/discuss my issues 34
  124. 124. Xtext wish list ● Accept my PRs (tags, correct indent, performance) ● Fix/discuss my issues ● Fix Xtend formatter issues 34
  125. 125. Xtext wish list ● Accept my PRs (tags, correct indent, performance) ● Fix/discuss my issues ● Fix Xtend formatter issues ● Very slow Xtend formatter source file 34
  126. 126. Xtext wish list ● Accept my PRs (tags, correct indent, performance) ● Fix/discuss my issues ● Fix Xtend formatter issues ● Very slow Xtend formatter source file ● Add pre- and post format hooks in format(FormatterRequest) 34
  127. 127. Conclusion 35
  128. 128. Conclusion ● Context aware 35
  129. 129. Conclusion ● Context aware ● Debugable 35
  130. 130. Conclusion ● Context aware ● Debugable ● Good performance 35
  131. 131. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable 35
  132. 132. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly 35
  133. 133. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly ○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend 35
  134. 134. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly ○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend ● Happy with the switch 35
  135. 135. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly ○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend ● Happy with the switch ○ New features 35
  136. 136. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly ○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend ● Happy with the switch ○ New features ○ Very readable 35
  137. 137. Conclusion ● Context aware ● Debugable ● Good performance ● Very customizable ● User friendly ○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend ● Happy with the switch ○ New features ○ Very readable ○ Conditional formatting 35
  138. 138. Evaluate the Sessions -1 0 +1 Sign in and vote at eclipsecon.org

×