This document discusses building a Python-like language with Xtext that is indentation aware. It covers parsing indentation aware languages, tokenizing and splitting tokens, defining indentation tokens, adjusting the Xtext workflow, marking tokens as splittable, and formatting indentation in the language. Key challenges discussed are parsing, content assist, formatting, folding and outline for indentation aware languages. The document provides code examples and recap of defining block structures with open and close tokens.

1. B U I L D I N G A P Y T H O N I D E
W I T H X T E X T
S E B A S T I A N Z A R N E K O W

2. T H I S I S N O T A B O U T

3. T H I S I S A B O U T

4. A N D A N T L R

5. A N D
[nrt ]

6. I N D E N TAT I O N A WA R E
Block:
OPEN
stmt+=Statement+
CLOSE;

7. C H A L L E N G E S
I N D E N TA T I O N A WA R E L A N G U A G E S

8. PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
C H A L L E N G E S
I N D E N TA T I O N A WA R E L A N G U A G E S

9. PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S

10. PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
Stateless, Upfront Lexing
Reentrant Lexing
CommonToken Oddities
Token Types
Only Basic Logic in Terminal Rules

11. S O M E P Y T H O N E S Q U E C O D E
if name==“Sebastian”:
println “Hello”

12. S O M E P Y T H O N E S Q U E C O D E
if.name==“Sebastian”:n
....println.“Hello”n

13. PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i a
n ” : n . . . . p r i n t l n . “ H
e l l o ” n

14. PA R S I N G - T O K E N I Z I N G
i f . n a m
e = = “ S e
b a s t i a
n ” : n . .
. . p r i n
t l n . “ H
e l l o ” n
CharStream
o.a.IntStream
AntlrStringStream

15. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
AntlrStringStream

16. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1

17. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
TokenSource
Lexer
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1

18. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
TokenStream
- 1
TokenSource
Lexer
o.a.IntStream

19. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G

20. Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
9
9
9
9
- 1
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n

21. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
9
9
9
9
- 1
TokenSource
TokenStream
?
PA R S I N G - T O K E N I Z I N G
Parser

22. R E C A P
Block:
OPEN
stmt+=Statement+
CLOSE;
IfStatement:
‘if’ cond=Condition ‘:’
then=Block;

23. R E C A P
if name==“Sebastian”:
println “Hello”

24. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
n OPEN . . . .
9
9
9
9
- 1
1 3 9
n CLOSE1 4
PA R S I N G - T O K E N S P L I T T I N G

25. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
9
9
9
9
- 1
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G

26. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1

27. if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
PA R S I N G - T O K E N S P L I T T I N G
1 3 9
1 4
TokenStream
Parser
SplittingTokenSource
TokenSource
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Lexer

28. L E T ’ S D O T H I S
I N D E N TA T I O N A WA R E L A N G U A G E S

29. Block:
OPEN
stmt+=Statement+
CLOSE;
1. D E F I N E I N D E N TAT I O N
T O K E N S

30. 1. D E F I N E I N D E N TAT I O N
T O K E N S
terminal OPEN: ‘synthetic:OPEN’;
terminal CLOSE: ‘synthetic:CLOSE’;

31. fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {}
..
fragment =
parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {}
2. A D J U S T M W E 2
W O R K F L O W
1. GENERATE SPLITTING TOKEN SOURCE
2. DISABLE PARTIAL PARSING

32. public class MyDslTokenSource extends AbstractIndentationTokenSource {
public MyDslTokenSource(TokenSource delegate) {
super(delegate);
}
@Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
}
3. M A R K T O K E N S A S
S P L I T TA B L E

33. 3. M A R K T O K E N S A S
S P L I T TA B L E
protected Token createEndToken(int offset) {
CommonToken result = new CommonToken(getEndTokenType());
result.setText("");
result.setChannel(Token.DEFAULT_CHANNEL);
result.setStartIndex(offset);
result.setStopIndex(offset-1);
return result;
}
public class AbstractIndentationTokenSource
extends AbstractSplittingTokenSource {
…
…
}

34. @Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
3. M A R K T O K E N S A S
S P L I T TA B L E

35. @Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
3. M A R K T O K E N S A S
S P L I T TA B L E

36. @Override
protected boolean shouldEmitPendingEndTokens() {
return false;
}
4. M A R K T O K E N S A S
S P L I T TA B L E A G A I N
1. Configure Content-Assist Parser
2. Almost the same, but:

37. CharStream
o.a.IntStream TokenSource
Lexer
AntlrStringStream
P U T T I N G E V E R T H I N G T O G E T H E R
I N D E N TA T I O N A WA R E L A N G U A G E S
TokenStream Parser
AbstractSplittingTokenSource
AbstractIndentationTokenSource

38. O N E M O R E T H I N G
I N D E N TA T I O N A WA R E L A N G U A G E S

39. def void format(Block block,
extension IFormattableDocument document) {
val open = block.regionForRuleCallTo(OPENRule)
open.prepend[ newLineAndIncreaseIndentation ]
val close = block.regionForRuleCallTo(CLOSERule)
if (block.isLast)
close.append[ decreaseIndentation ]
else
close.append[ newLineAndDecreaseIndentation ]
}
def void format(Block block,
extension IFormattableDocument document) {
val open = block.regionForRuleCallTo(OPENRule)
open.prepend[ newLineAndIncreaseIndentation ]
val close = block.regionForRuleCallTo(CLOSERule)
if (block.isLast)
close.append[ decreaseIndentation ]
else
close.append[ newLineAndDecreaseIndentation ]
}
5. F O R M AT T I N G
1. Indent Blocks
2. Adjust NewLines
3. Remove superfluous spaces

40. PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
D O N E
I N D E N TA T I O N A WA R E L A N G U A G E S

41. Q & A