Dynamic Language Embedding With Homogeneous Tool Support

1,976 views

Published on

Domain-specific languages (DSLs) are increasingly used as embedded languages within general-purpose host languages. DSLs provide a compact, dedicated syntax for specifying parts of an application related to specialized domains. Unfortunately, such language extensions typically do not integrate well with existing development tools. Editors, compilers and debuggers are either unaware of the extensions, or must be adapted at a non-trivial cost. Furthermore, these embedded languages typically conflict with the grammar of the host language and make it difficult to write hybrid code; few mechanisms exist to control the scope and usage of multiple tightly interconnected embedded languages. In this dissertation we present Helvetia, a novel approach to embed languages into an existing host language by leveraging the underlying representation of the host language used by these tools. We introduce Language Boxes, an approach that offers a simple, modular mechanism to encapsulate (i) compositional changes to the host language, (ii) transformations to address various concerns such as compilation and syntax highlighting, and (iii) scoping rules to control visibility of fine-grained language changes. We describe the design and implementation of Helvetia and Language Boxes, discuss the required infrastructure of a host language enabling language embedding, and validate our approach by case studies that demonstrate different ways to extend or adapt the host language syntax and semantics.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,976
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Dynamic Language Embedding With Homogeneous Tool Support

  1. 1. Dynamic Language Embedding With Homogeneous Tool Support PhD Defense Lukas Renggli Advisor Oscar Nierstrasz 1
  2. 2. 2 SELECT  email  FROM  users WHERE  username  =  'lr'
  3. 3. 3 SELECT  email  FROM  users WHERE  username  =  'lr' Syntax
  4. 4. 4 SELECT  email  FROM  users WHERE  username  =  'lr' Semantics
  5. 5. General Purpose Host Language 5 + SELECT  email  FROM  users WHERE  username  =  'lr'
  6. 6. General Purpose Host Language 6 ? SELECT  email  FROM  users WHERE  username  =  'lr' SyntaxSQL SemanticsSQL SyntaxHost SemanticsHost ToolsHost
  7. 7. 7
  8. 8. 8 Host Language External Language
  9. 9. 9
  10. 10. 10 Host Language Internal Language
  11. 11. 11
  12. 12. 12 Host Language Embedded Language
  13. 13. 13 Non-Standard Host Language Embedded Language
  14. 14. 14
  15. 15. 15 !" #"$%&
  16. 16. MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de 16
  17. 17. Conventional Language 17
  18. 18. 18 MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
  19. 19. Multiple Context Specific Languages 19
  20. 20. 20 MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
  21. 21. Homogeneous Code and Data Abstraction 21
  22. 22. § § § 22 MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
  23. 23. § § § Homogeneous Tool Support 23
  24. 24. Thesis To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation. 24
  25. 25. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 25
  26. 26. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 26
  27. 27. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 27
  28. 28. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 28
  29. 29. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 29
  30. 30. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 30
  31. 31. 31 !" #"$%&
  32. 32. 32 Types of Embedded Languages
  33. 33. 33 ◦ ◦ ◦ ◦ ◦ ● ◦ ● ◦ ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ ● ● ●Syntax Vocabulary Semantics
  34. 34. 34 Host Language ◦ ◦ ◦ ◦ ◦ ● ◦ ● ◦ ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ ● ● ●Syntax Vocabulary Semantics
  35. 35. 35 Host Language ◦ ◦ ◦ ◦ ◦ ● Internal Language ◦ ● ◦ ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ ● ● ●Syntax Vocabulary Semantics
  36. 36. 36 Host Language ◦ ◦ ◦ ◦ ◦ ● Internal Language ◦ ● ◦ Pidgin ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ ● ● ●Syntax Vocabulary Semantics
  37. 37. 37 Host Language ◦ ◦ ◦ ◦ ◦ ● Internal Language ◦ ● ◦ Pidgin ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ Creole ● ● ●Syntax Vocabulary Semantics
  38. 38. 38 Host Language ◦ ◦ ◦ Argot ◦ ◦ ● Internal Language ◦ ● ◦ Pidgin ◦ ● ● ● ◦ ◦ ● ◦ ● ● ● ◦ Creole ● ● ●Syntax Vocabulary Semantics
  39. 39. 39 Host Language ◦ ◦ ◦ Argot ◦ ◦ ● Internal Language ◦ ● ◦ Pidgin ◦ ● ● — ● ◦ ◦ — ● ◦ ● — ● ● ◦ Creole ● ● ●Syntax Vocabulary Semantics
  40. 40. 40 Pidgin ◦ ● ● Creole ● ● ● Argot ◦ ◦ ● Syntax Vocabulary Semantics
  41. 41. 41
  42. 42. 42
  43. 43. 43
  44. 44. 44
  45. 45. 45
  46. 46. 46 Package Name x = 1 y = 1 (2, 1) (2, 2)(1, 2) x = 2 y = 2
  47. 47. Package Name x = 1 y = 1 (2, 1) (2, 2)(1, 2) x = 2 y = 2 47 aBuilder  row  grow. aBuilder  row  fill. aBuilder  column  grow. aBuilder  column  fill. aBuilder  x:  1  y:  1  add:  (LabelShape  new   text:  [  :each  |  each  name  ];   borderColor:  #black;   borderWidth:  1;   yourself). aBuilder  x:  1  y:  2  w:  2  h:  1  add:  (RectangleShape  new   borderColor:  #black;   borderWidth:  1;   width:  200;   height:  100;   yourself)
  48. 48. Package Name x = 1 y = 1 (2, 1) (2, 2)(1, 2) x = 2 y = 2 48 row  =  grow. row  =  fill. column  =  grow. column  =  fill. (1  ,  1)  =  label     text:  [  :each  |  each  name  ];     borderColor:  #black;     borderWidth:  1. (1  ,  2)  -­‐  (2  ,  1)  =  rectangle     borderColor:  #black;     borderWidth:  1;     width:  200;     height:  100.
  49. 49. 49
  50. 50. shape  {   cols:  #grow,  #fill;   rows:  #grow,  #fill; } label  {   position:  1  ,  1;     text:  [  :each  |  each  name  ];   borderColor:  #black;   borderWidth:  1; } rectangle  {     position:  1  ,  2;   colspan:  2;   borderColor:  #black;   borderWidth:  1;   width:  200;   height:  100; } Package Name x = 1 y = 1 (2, 1) (2, 2)(1, 2) x = 2 y = 2 50
  51. 51. 51
  52. 52. 52 Conventional Language
  53. 53. 53 Conventional Language Context Specific
  54. 54. 54 Conventional Language Context Specific Homogeneous Code & Data
  55. 55. 55 Conventional Language Context Specific Homogeneous Code & Data Homogeneous Tool Support
  56. 56. 56 Conventional Language Context Specific Homogeneous Code & Data Homogeneous Tool Support
  57. 57. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 57 Renggli et al.ECOOP 2010
  58. 58. 58 MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
  59. 59. 59 MarcoZanoli,cc-by-sa,2.5,www.wikipedia.de
  60. 60. Editor Compiler Debugger Syntax Language 1 60
  61. 61. Editor Compiler Debugger Syntax Language 1 61 Language 2
  62. 62. Editor Compiler Debugger Syntax Language 1 62 Language 2
  63. 63. Editor Compiler Debugger Syntax Language 1 63 Language 2
  64. 64. Language Boxes 64
  65. 65. 65 SELECT  *  FROM  users
  66. 66. 66 |  r  | r  :=   ^  User  fromRow:  r .SELECT  *  FROM  users
  67. 67. Language Scope Language Concern Language Change Language Box 67
  68. 68. Language Scope Active? 68
  69. 69. Language Scope 69
  70. 70. ‣ System ‣ Packages ‣ Classes ‣ Methods Language Scope 70
  71. 71. Language Concern Semantics 71
  72. 72. Language Concern Transformation 72
  73. 73. Language Concern 73 Context Menus Navigation Search Code Expansion Code Completion Error Correction Custom Inspector Refactorings Code Folding Highlighting
  74. 74. Language Change 74 Syntax
  75. 75. Language Change 75 Host Language
  76. 76. Language Change 76 Host Language + SQL Language
  77. 77. Language Change 77 Host Language + SQL Language + ...
  78. 78. Host Language + SQL Language + ... = Custom Host Language Language Change 78
  79. 79. 79
  80. 80. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 80 Renggli et al.SLE 2009
  81. 81. Language Boxes Host Language Dynamic Grammars 81 Language and Tool Extensions
  82. 82. Language Change Language Concern Language Scope Language Box 82
  83. 83. Language Change = Host Language δ 83
  84. 84. Language Change = Host Language Grammarδ 84
  85. 85. Language Change = Grammar Transformation 85
  86. 86. scanIdentifier self step. ((currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ]) ifTrue: [ [ self recordMatch: #IDENTIFIER. self step. (currentCharacter between: $0 and: $9) or: [ (currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ] ] ] whileTrue. ^ self reportLastMatch ] 86
  87. 87. #( #[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41] #[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112] #[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34] #[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[0 1 210 0 76 0 81] #[0 1 214 0 76 0 81] #[1 0 173 0 76 0 177 0 81] #[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69] #[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34] #[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76] #[0 2 50 0 21 0 25 0 26 0 76 0 79] #[1 1 13 0 76 2 85 0 124 1 21 0 125] #[1 2 89 0 17 2 30 0 21 2 30 0 82] #[1 2 93 0 21 2 97 0 82] ) 87
  88. 88. Scannerless Parser Combinator 88
  89. 89. a..z a..z 0..9 ID  ::=  letter  {  letter  |  digit  }  ; 89
  90. 90. letter letter digit sequence choice many ID  ::=  letter  {  letter  |  digit  }  ; 90
  91. 91. Grammar Transformation 91
  92. 92. letter letter digit sequence choice many letter    !    letter  |  "_" 92
  93. 93. letter digit sequence choice manychoice _ letter choice _ letter    !    letter  |  "_"   93
  94. 94. letter digit sequence choice manychoice _ Optimizations 94
  95. 95. Grammar Composition 95
  96. 96. Insert  grammar  fragment before/after  grammar  production as  a  choice/sequence/replacement. 96 Language Change
  97. 97. Insert  SQL  grammar after  expression  production as  an  additional  choice 97 Language Change
  98. 98. 98 Language Change Variable Literal Parens Expr:
  99. 99. 99 Language Change Variable Literal Parens Expr: SQL
  100. 100. Conflicts & Ambiguities 100
  101. 101. SELECT  *  FROM  users 101
  102. 102. SELECT  *  FROM  users 102 |  r  | r  :=   ^  User  fromRow:  r .
  103. 103. expr sql| 103
  104. 104. SELECT  *  FROM  users<SQL: > 104
  105. 105. Parsing Expression Grammars 105
  106. 106. expr sql/ ordered 106
  107. 107. expr sql/ no conflict 107
  108. 108. expr sql/ 108 surprise
  109. 109. / exprsql 109 surprise
  110. 110. expr sql| 110 unordered
  111. 111. expr sql| 111 !sql  expr  /  !expr  sql
  112. 112. expr sql$ 112 !sql  expr  /  !expr  sql  /  ui
  113. 113. expr sql$ 113 !sql  expr  /  !expr  sql  /  ui
  114. 114. Language and Tool Extensions Language Boxes Host Language Dynamic Grammars Renggli et al.DYLA 2010 114
  115. 115. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 115
  116. 116. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes Mondrian Object Relationships Positional Arguments Program Checking Quasiquoting Regular Expression Roman Numbers SPath Expression SQL Schematic Tables String Interpolation Transactional Memory Tuple Space 116 [http://scg.unibe.ch/research/helvetia/examples]
  117. 117. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes ✓Mondrian Object Relationships Positional Arguments Program Checking Quasiquoting Regular Expression Roman Numbers SPath Expression ✓SQL Schematic Tables String Interpolation Transactional Memory Tuple Space 117 [http://scg.unibe.ch/research/helvetia/examples]
  118. 118. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes ✓Mondrian Object Relationships Positional Arguments ‣ Program Checking ‣ Quasiquoting Regular Expression Roman Numbers SPath Expression ✓SQL Schematic Tables String Interpolation ‣ Transactional Memory Tuple Space 118 [http://scg.unibe.ch/research/helvetia/examples]
  119. 119. apply hasChanged hasConflict Change object * changes Process 0..1 currentTransaction do: aBlock retry: aBlock checkpoint abort: anObject escapeContext Transaction previousCopy workingCopy ObjectChange applyBlock conflictTestBlock CustomChange * Transactional Memory 119
  120. 120. Meta-Programming Facilities ``(`,(aString) asRegex) 120
  121. 121. Domain-Specific Program Checking 121
  122. 122. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions Renggli et al. CLSS 2009 Renggli et al. IWST 2009 Nierstrasz et al. LNCS 2009 Renggli et al. TOOLS 2010 122
  123. 123. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 123 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
  124. 124. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 124 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
  125. 125. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 125 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
  126. 126. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 126 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
  127. 127. Language Boxes Host Language Dynamic Grammars 127 Language and Tool Extensions

×