Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
B U I L D I N G A P Y T H O N I D E
W I T H X T E X T
S E B A S T I A N Z A R N E K O W
T H I S I S N O T A B O U T
T H I S I S A B O U T
A N D A N T L R
A N D
[nrt ]
I N D E N TAT I O N A WA R E
Block:
OPEN
stmt+=Statement+
CLOSE;
C H A L L E N G E S
I N D E N TA T I O N A WA R E L A N G U A G E S
PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
C H A L L E N G E S
I N D E N TA T I...
PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
Stateless, Upfront Lexing
Reentrant Lex...
S O M E P Y T H O N E S Q U E C O D E
if name==“Sebastian”:
println “Hello”
S O M E P Y T H O N E S Q U E C O D E
if.name==“Sebastian”:n
....println.“Hello”n
PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i a

n ” : n . . . . p r i n t l n . “ H
e l l o ” n
PA R S I N G - T O K E N I Z I N G
i f . n a m
e = = “ S e
b a s t i a
n ” : n . .
. . p r i n
t l n . “ H
e l l o ” n
Cha...
PA R S I N G - T O K E N I Z I N G
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
CharStream
o.a.IntStream...
PA R S I N G - T O K E N I Z I N G
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
CharStream
o.a.IntStream...
PA R S I N G - T O K E N I Z I N G
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
CharStream
o.a.IntStream...
PA R S I N G - T O K E N I Z I N G
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
1 2
9
1 0
7
8
6
9
1 0
9
...
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Parser
TokenStream
TokenSou...
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
9
9
9
9
- 1
if

.

name

==

“Sebastian”

:

n....

prin...
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
9
9
9
9
- 1
TokenSource
TokenStream
?
PA R S I N G - T O K ...
R E C A P
Block:
OPEN
stmt+=Statement+
CLOSE;
IfStatement:
‘if’ cond=Condition ‘:’

then=Block;
R E C A P
if name==“Sebastian”:
println “Hello”
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
n OPEN . . . .
9
9
9
9
- 1
1 3 9
n CLOSE1 4
PA R S I N G - ...
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
9
9
9
9
- 1
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T ...
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
1 2
9
...
if

.

name

==

“Sebastian”

:

n....

println

.

“Hello”

n
PA R S I N G - T O K E N S P L I T T I N G
1 3 9
1 4
TokenS...
L E T ’ S D O T H I S
I N D E N TA T I O N A WA R E L A N G U A G E S
Block:
OPEN
stmt+=Statement+
CLOSE;
1. D E F I N E I N D E N TAT I O N 

T O K E N S
1. D E F I N E I N D E N TAT I O N 

T O K E N S
terminal OPEN: ‘synthetic:OPEN’;
terminal CLOSE: ‘synthetic:CLOSE’;
fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {}
..
fragment =

parser.antlr.ex.ca.ContentAssistParserGeneratorFrag...
public class MyDslTokenSource extends AbstractIndentationTokenSource {
public MyDslTokenSource(TokenSource delegate) {
sup...
3. M A R K T O K E N S A S
S P L I T TA B L E
protected Token createEndToken(int offset) {
CommonToken result = new Common...
@Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@...
@Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@...
@Override
protected boolean shouldEmitPendingEndTokens() {
return false;
}
4. M A R K T O K E N S A S
S P L I T TA B L E A...
CharStream
o.a.IntStream TokenSource
Lexer
AntlrStringStream
P U T T I N G E V E R T H I N G T O G E T H E R
I N D E N TA ...
O N E M O R E T H I N G
I N D E N TA T I O N A WA R E L A N G U A G E S
def void format(Block block,

extension IFormattableDocument document) {

val open = block.regionForRuleCallTo(OPENRule)
o...
PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
D O N E
I N D E N TA T I O N A WA R ...
Q & A
Upcoming SlideShare
Loading in …5
×

Building a Python IDE with Xtext

1,229 views

Published on

Introduction on indentation aware languages with Xtext.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Building a Python IDE with Xtext

  1. 1. B U I L D I N G A P Y T H O N I D E W I T H X T E X T S E B A S T I A N Z A R N E K O W
  2. 2. T H I S I S N O T A B O U T
  3. 3. T H I S I S A B O U T
  4. 4. A N D A N T L R
  5. 5. A N D [nrt ]
  6. 6. I N D E N TAT I O N A WA R E Block: OPEN stmt+=Statement+ CLOSE;
  7. 7. C H A L L E N G E S I N D E N TA T I O N A WA R E L A N G U A G E S
  8. 8. PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E C H A L L E N G E S I N D E N TA T I O N A WA R E L A N G U A G E S
  9. 9. PA R S I N G I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
  10. 10. PA R S I N G I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S Stateless, Upfront Lexing Reentrant Lexing CommonToken Oddities Token Types Only Basic Logic in Terminal Rules
  11. 11. S O M E P Y T H O N E S Q U E C O D E if name==“Sebastian”: println “Hello”
  12. 12. S O M E P Y T H O N E S Q U E C O D E if.name==“Sebastian”:n ....println.“Hello”n
  13. 13. PA R S I N G - T O K E N I Z I N G i f . n a m e = = “ S e b a s t i a
 n ” : n . . . . p r i n t l n . “ H e l l o ” n
  14. 14. PA R S I N G - T O K E N I Z I N G i f . n a m e = = “ S e b a s t i a n ” : n . . . . p r i n t l n . “ H e l l o ” n CharStream o.a.IntStream AntlrStringStream
  15. 15. PA R S I N G - T O K E N I Z I N G if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n CharStream o.a.IntStream AntlrStringStream
  16. 16. PA R S I N G - T O K E N I Z I N G if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n CharStream o.a.IntStream AntlrStringStream 1 2 9 1 0 7 8 6 9 1 0 9 9 8 - 1
  17. 17. PA R S I N G - T O K E N I Z I N G if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n CharStream o.a.IntStream TokenSource Lexer AntlrStringStream 1 2 9 1 0 7 8 6 9 1 0 9 9 8 - 1
  18. 18. PA R S I N G - T O K E N I Z I N G if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n 1 2 9 1 0 7 8 6 9 1 0 9 9 8 TokenStream - 1 TokenSource Lexer o.a.IntStream
  19. 19. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n 1 2 9 1 0 7 8 6 9 1 0 9 9 8 - 1 Parser TokenStream TokenSource PA R S I N G - T O K E N I Z I N G
  20. 20. Parser TokenStream TokenSource PA R S I N G - T O K E N I Z I N G 9 9 9 9 - 1 if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n
  21. 21. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n 9 9 9 9 - 1 TokenSource TokenStream ? PA R S I N G - T O K E N I Z I N G Parser
  22. 22. R E C A P Block: OPEN stmt+=Statement+ CLOSE; IfStatement: ‘if’ cond=Condition ‘:’
 then=Block;
  23. 23. R E C A P if name==“Sebastian”: println “Hello”
  24. 24. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n n OPEN . . . . 9 9 9 9 - 1 1 3 9 n CLOSE1 4 PA R S I N G - T O K E N S P L I T T I N G
  25. 25. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n 9 9 9 9 - 1 1 3 9 1 4 PA R S I N G - T O K E N S P L I T T I N G
  26. 26. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n 1 3 9 1 4 PA R S I N G - T O K E N S P L I T T I N G 1 2 9 1 0 7 8 6 9 1 0 9 9 8 - 1
  27. 27. if
 .
 name
 ==
 “Sebastian”
 :
 n....
 println
 .
 “Hello”
 n PA R S I N G - T O K E N S P L I T T I N G 1 3 9 1 4 TokenStream Parser SplittingTokenSource TokenSource 1 2 9 1 0 7 8 6 9 1 0 9 9 8 - 1 Lexer
  28. 28. L E T ’ S D O T H I S I N D E N TA T I O N A WA R E L A N G U A G E S
  29. 29. Block: OPEN stmt+=Statement+ CLOSE; 1. D E F I N E I N D E N TAT I O N 
 T O K E N S
  30. 30. 1. D E F I N E I N D E N TAT I O N 
 T O K E N S terminal OPEN: ‘synthetic:OPEN’; terminal CLOSE: ‘synthetic:CLOSE’;
  31. 31. fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {} .. fragment =
 parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {} 2. A D J U S T M W E 2 W O R K F L O W 1. GENERATE SPLITTING TOKEN SOURCE 2. DISABLE PARTIAL PARSING
  32. 32. public class MyDslTokenSource extends AbstractIndentationTokenSource { public MyDslTokenSource(TokenSource delegate) { super(delegate); } @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; } @Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; } @Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } } 3. M A R K T O K E N S A S S P L I T TA B L E
  33. 33. 3. M A R K T O K E N S A S S P L I T TA B L E protected Token createEndToken(int offset) { CommonToken result = new CommonToken(getEndTokenType()); result.setText(""); result.setChannel(Token.DEFAULT_CHANNEL); result.setStartIndex(offset); result.setStopIndex(offset-1); return result; } public class AbstractIndentationTokenSource
 extends AbstractSplittingTokenSource {
 … … }
  34. 34. @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; } @Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; } @Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } 3. M A R K T O K E N S A S S P L I T TA B L E
  35. 35. @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; } @Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; } @Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } 3. M A R K T O K E N S A S S P L I T TA B L E
  36. 36. @Override protected boolean shouldEmitPendingEndTokens() { return false; } 4. M A R K T O K E N S A S S P L I T TA B L E A G A I N 1. Configure Content-Assist Parser 2. Almost the same, but:
  37. 37. CharStream o.a.IntStream TokenSource Lexer AntlrStringStream P U T T I N G E V E R T H I N G T O G E T H E R I N D E N TA T I O N A WA R E L A N G U A G E S TokenStream Parser AbstractSplittingTokenSource AbstractIndentationTokenSource
  38. 38. O N E M O R E T H I N G I N D E N TA T I O N A WA R E L A N G U A G E S
  39. 39. def void format(Block block,
 extension IFormattableDocument document) {
 val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] } def void format(Block block,
 extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] } 5. F O R M AT T I N G 1. Indent Blocks 2. Adjust NewLines 3. Remove superfluous spaces
  40. 40. PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E D O N E I N D E N TA T I O N A WA R E L A N G U A G E S
  41. 41. Q & A

×