Dynamic Language Embedding
With Homogeneous Tool Support
PhD Defense
Lukas Renggli
Advisor
Oscar Nierstrasz
1
2
SELECT	
  email	
  FROM	
  users
WHERE	
  username	
  =	
  'lr'
3
SELECT	
  email	
  FROM	
  users
WHERE	
  username	
  =	
  'lr'
Syntax
4
SELECT	
  email	
  FROM	
  users
WHERE	
  username	
  =	
  'lr'
Semantics
General Purpose
Host Language
5
+
SELECT	
  email	
  FROM	
  users
WHERE	
  username	
  =	
  'lr'
General Purpose
Host Language
6
?
SELECT	
  email	
  FROM	
  users
WHERE	
  username	
  =	
  'lr'
SyntaxSQL SemanticsSQL
SyntaxHost SemanticsHost
ToolsHost
7
8
Host Language
External
Language
9
10
Host Language
Internal
Language
11
12
Host Language
Embedded
Language
13
Non-Standard
Host Language
Embedded
Language
14
15
!" #"$%&
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
16
Conventional
Language
17
18
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
Multiple Context
Specific Languages
19
20
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
Homogeneous Code
and Data Abstraction
21
§
§
§
22
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
§
§
§
Homogeneous
Tool Support
23
Thesis
To support seamless integration of context-dependent
languages without breaking the tools, we need
1.	 a host-language grammar that can be changed by
language extensions,
2.	 a first-class language description used by the
development environment, and
3.	 a transformation mechanism of the embedded language
into a common executable representation.
24
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
25
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
26
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
27
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
28
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
29
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
30
31
!" #"$%&
32
Types of
Embedded Languages
33
◦ ◦ ◦
◦ ◦ ●
◦ ● ◦
◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
● ● ●Syntax
Vocabulary
Semantics
34
Host Language ◦ ◦ ◦
◦ ◦ ●
◦ ● ◦
◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
● ● ●Syntax
Vocabulary
Semantics
35
Host Language ◦ ◦ ◦
◦ ◦ ●
Internal Language ◦ ● ◦
◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
● ● ●Syntax
Vocabulary
Semantics
36
Host Language ◦ ◦ ◦
◦ ◦ ●
Internal Language ◦ ● ◦
Pidgin ◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
● ● ●Syntax
Vocabulary
Semantics
37
Host Language ◦ ◦ ◦
◦ ◦ ●
Internal Language ◦ ● ◦
Pidgin ◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
Creole ● ● ●Syntax
Vocabulary
Semantics
38
Host Language ◦ ◦ ◦
Argot ◦ ◦ ●
Internal Language ◦ ● ◦
Pidgin ◦ ● ●
● ◦ ◦
● ◦ ●
● ● ◦
Creole ● ● ●Syntax
Vocabulary
Semantics
39
Host Language ◦ ◦ ◦
Argot ◦ ◦ ●
Internal Language ◦ ● ◦
Pidgin ◦ ● ●
— ● ◦ ◦
— ● ◦ ●
— ● ● ◦
Creole ● ● ●Syntax
Vocabulary
Semantics
40
Pidgin ◦ ● ●
Creole ● ● ●
Argot ◦ ◦ ●
Syntax
Vocabulary
Semantics
41
42
43
44
45
46
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2
47
aBuilder	
  row	
  grow.
aBuilder	
  row	
  fill.
aBuilder	
  column	
  grow.
aBuilder	
  column	
  fill.
aBuilder	
  x:	
  1	
  y:	
  1	
  add:	
  (LabelShape	
  new
	
   text:	
  [	
  :each	
  |	
  each	
  name	
  ];
	
   borderColor:	
  #black;
	
   borderWidth:	
  1;
	
   yourself).
aBuilder	
  x:	
  1	
  y:	
  2	
  w:	
  2	
  h:	
  1	
  add:	
  (RectangleShape	
  new
	
   borderColor:	
  #black;
	
   borderWidth:	
  1;
	
   width:	
  200;
	
   height:	
  100;
	
   yourself)
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2
48
row	
  =	
  grow.
row	
  =	
  fill.
column	
  =	
  grow.
column	
  =	
  fill.
(1	
  ,	
  1)	
  =	
  label
	
   	
   text:	
  [	
  :each	
  |	
  each	
  name	
  ];
	
   	
   borderColor:	
  #black;
	
   	
   borderWidth:	
  1.
(1	
  ,	
  2)	
  -­‐	
  (2	
  ,	
  1)	
  =	
  rectangle
	
   	
   borderColor:	
  #black;
	
   	
   borderWidth:	
  1;
	
   	
   width:	
  200;
	
   	
   height:	
  100.
49
shape	
  {
	
   cols:	
  #grow,	
  #fill;
	
   rows:	
  #grow,	
  #fill;
}
label	
  {
	
   position:	
  1	
  ,	
  1;	
  
	
   text:	
  [	
  :each	
  |	
  each	
  name	
  ];
	
   borderColor:	
  #black;
	
   borderWidth:	
  1;
}
rectangle	
  {	
  
	
   position:	
  1	
  ,	
  2;
	
   colspan:	
  2;
	
   borderColor:	
  #black;
	
   borderWidth:	
  1;
	
   width:	
  200;
	
   height:	
  100;
}
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2
50
51
52
Conventional
Language
53
Conventional
Language
Context
Specific
54
Conventional
Language
Context
Specific
Homogeneous
Code & Data
55
Conventional
Language
Context
Specific
Homogeneous
Code & Data
Homogeneous
Tool Support
56
Conventional
Language
Context
Specific
Homogeneous
Code & Data
Homogeneous
Tool Support
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
57
Renggli et al.ECOOP 2010
58
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
59
MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
Editor
Compiler
Debugger
Syntax
Language 1
60
Editor
Compiler
Debugger
Syntax
Language 1
61
Language 2
Editor
Compiler
Debugger
Syntax
Language 1
62
Language 2
Editor
Compiler
Debugger
Syntax
Language 1
63
Language 2
Language Boxes
64
65
SELECT	
  *	
  FROM	
  users
66
|	
  r	
  |
r	
  :=	
  
^	
  User	
  fromRow:	
  r
.SELECT	
  *	
  FROM	
  users
Language
Scope
Language
Concern
Language
Change
Language
Box
67
Language Scope
Active?
68
Language Scope
69
‣ System
‣ Packages
‣ Classes
‣ Methods
Language Scope
70
Language Concern
Semantics
71
Language Concern
Transformation
72
Language Concern
73
Context Menus
Navigation Search
Code Expansion
Code Completion
Error Correction
Custom Inspector
Refactorings
Code Folding
Highlighting
Language Change
74
Syntax
Language Change
75
Host Language
Language Change
76
Host Language
+ SQL Language
Language Change
77
Host Language
+ SQL Language
+ ...
Host Language
+ SQL Language
+ ...
= Custom Host Language
Language Change
78
79
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
80
Renggli et al.SLE 2009
Language Boxes
Host Language
Dynamic
Grammars
81
Language and Tool Extensions
Language
Change
Language
Concern
Language
Scope
Language
Box
82
Language
Change
= Host Language
δ
83
Language
Change
=
Host Language
Grammarδ
84
Language
Change
=
Grammar
Transformation
85
scanIdentifier
	 self step.	
	 ((currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ]) ifTrue: [
	 	 [ self recordMatch: #IDENTIFIER.
	 	 self step.
	 	 (currentCharacter between: $0 and: $9)
or: [ (currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ] ] ]
		 	 	 whileTrue.
	 ^ self reportLastMatch ]
86
#(	#[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41]
	 #[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112]
	 #[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34]
	 #[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
	 #[0 1 210 0 76 0 81]
	 #[0 1 214 0 76 0 81]
	 #[1 0 173 0 76 0 177 0 81]
	 #[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
	 #[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69]

 #[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34]

 #[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76]
	 #[0 2 50 0 21 0 25 0 26 0 76 0 79]
	 #[1 1 13 0 76 2 85 0 124 1 21 0 125]
	 #[1 2 89 0 17 2 30 0 21 2 30 0 82]
	 #[1 2 93 0 21 2 97 0 82] )
87
Scannerless
Parser Combinator
88
a..z a..z
0..9
ID	
  ::=	
  letter	
  {	
  letter	
  |	
  digit	
  }	
  ;
89
letter
letter digit
sequence
choice
many
ID	
  ::=	
  letter	
  {	
  letter	
  |	
  digit	
  }	
  ;
90
Grammar
Transformation
91
letter
letter digit
sequence
choice
many
letter	
  	
  !	
  	
  letter	
  |	
  "_"
92
letter
digit
sequence
choice
manychoice
_
letter
choice
_
letter	
  	
  !	
  	
  letter	
  |	
  "_"	
  
93
letter digit
sequence
choice
manychoice
_
Optimizations
94
Grammar
Composition
95
Insert	
  grammar	
  fragment
before/after	
  grammar	
  production
as	
  a	
  choice/sequence/replacement.
96
Language Change
Insert	
  SQL	
  grammar
after	
  expression	
  production
as	
  an	
  additional	
  choice
97
Language Change
98
Language Change
Variable
Literal
Parens
Expr:
99
Language Change
Variable
Literal
Parens
Expr:
SQL
Conflicts &
Ambiguities
100
SELECT	
  *	
  FROM	
  users
101
SELECT	
  *	
  FROM	
  users
102
|	
  r	
  |
r	
  :=	
  
^	
  User	
  fromRow:	
  r
.
expr sql|
103
SELECT	
  *	
  FROM	
  users<SQL: >
104
Parsing Expression
Grammars
105
expr sql/
ordered
106
expr sql/
no conflict
107
expr sql/
108
surprise
/ exprsql
109
surprise
expr sql|
110
unordered
expr sql|
111
!sql	
  expr	
  /	
  !expr	
  sql
expr sql$
112
!sql	
  expr	
  /	
  !expr	
  sql	
  /	
  ui
expr sql$
113
!sql	
  expr	
  /	
  !expr	
  sql	
  /	
  ui
Language and Tool Extensions
Language Boxes
Host Language
Dynamic
Grammars
Renggli et al.DYLA 2010
114
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
115
Assignments and Swapping
Asynchronous Messages
Automaton
Brainfuck Language
Functional Pattern Matching
Grammar Definition
Message Pipes
Mondrian
Object Relationships
Positional Arguments
Program Checking
Quasiquoting
Regular Expression
Roman Numbers
SPath Expression
SQL
Schematic Tables
String Interpolation
Transactional Memory
Tuple Space
116
[http://scg.unibe.ch/research/helvetia/examples]
Assignments and Swapping
Asynchronous Messages
Automaton
Brainfuck Language
Functional Pattern Matching
Grammar Definition
Message Pipes
✓Mondrian
Object Relationships
Positional Arguments
Program Checking
Quasiquoting
Regular Expression
Roman Numbers
SPath Expression
✓SQL
Schematic Tables
String Interpolation
Transactional Memory
Tuple Space
117
[http://scg.unibe.ch/research/helvetia/examples]
Assignments and Swapping
Asynchronous Messages
Automaton
Brainfuck Language
Functional Pattern Matching
Grammar Definition
Message Pipes
✓Mondrian
Object Relationships
Positional Arguments
‣ Program Checking
‣ Quasiquoting
Regular Expression
Roman Numbers
SPath Expression
✓SQL
Schematic Tables
String Interpolation
‣ Transactional Memory
Tuple Space
118
[http://scg.unibe.ch/research/helvetia/examples]
apply
hasChanged
hasConflict
Change
object
*
changes
Process
0..1
currentTransaction
do: aBlock
retry: aBlock
checkpoint
abort: anObject
escapeContext
Transaction
previousCopy
workingCopy
ObjectChange
applyBlock
conflictTestBlock
CustomChange
*
Transactional Memory
119
Meta-Programming Facilities
``(`,(aString) asRegex)
120
Domain-Specific
Program Checking
121
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
Renggli et al.
CLSS 2009 Renggli et al.
IWST 2009 Nierstrasz et al.
LNCS 2009 Renggli et al.
TOOLS 2010
122
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
123
To support seamless integration of context-dependent
languages without breaking the tools, we need
1.	 a host-language grammar that can be changed by
language extensions,
2.	 a first-class language description used by the
development environment, and
3.	 a transformation mechanism of the embedded
language into a common executable representation.
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
124
To support seamless integration of context-dependent
languages without breaking the tools, we need
1.	 a host-language grammar that can be changed by
language extensions,
2.	 a first-class language description used by the
development environment, and
3.	 a transformation mechanism of the embedded
language into a common executable representation.
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
125
To support seamless integration of context-dependent
languages without breaking the tools, we need
1.	 a host-language grammar that can be changed by
language extensions,
2.	 a first-class language description used by the
development environment, and
3.	 a transformation mechanism of the embedded
language into a common executable representation.
Language Boxes
Host Language
Dynamic
Grammars
Language and Tool Extensions
126
To support seamless integration of context-dependent
languages without breaking the tools, we need
1.	 a host-language grammar that can be changed by
language extensions,
2.	 a first-class language description used by the
development environment, and
3.	 a transformation mechanism of the embedded
language into a common executable representation.
Language Boxes
Host Language
Dynamic
Grammars
127
Language and Tool Extensions

Dynamic Language Embedding With Homogeneous Tool Support