Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Domains Shape Languages

8 views

Published on

Depending on the purpose of a language, its design will be completely different: the syntax may be textual, graphical, tabular, or a mix of those. It may be highly expressive, allowing users to define their own abstractions, or restricted, simplifying the language for non-programmer end users. The degree of formality may be different, depending on the domain for which the language is built and the assurances one must be able to make about the correctness of programs. Depending on performance requirements, a simple AST interpreter may be good enough for executing programs, or one may need an optimizing generator. I this talk, I revisit a number of languages we have built over the last five years. For each of them, I discuss the core design decisions as well as the drivers in the domain that led to those decisions. The talk includes languages for embedded software, insurance product definition, security analysis, health algorithms, engine control and system engineering. All of the languages discussed have been designed with and for industrial customers and have been implemented on the basis of the open source Jetbrains MPS language workbench.

Published in: Software
  • Be the first to comment

  • Be the first to like this

How Domains Shape Languages

  1. 1. Markus Völter voelter@acm.org www.voelter.de @markusvoelter Domain Requirements Languages How shape Language Design Experience from 10 Years of Building DSLs
  2. 2. Health & Medical AutomoFve Aerospace RoboFcs Finance Embedded Government DOMAINS SoNware Engineering Requirements Engineering SpecificaFon and VerificaFon ImplementaFon and Test Mixed NotaFons and End User Programming Informal -> Semiformal -> Formal Languages + VerificaFon OptmizaFons, Performance and Concurrency Fundamentals: Editors, Type Systems, Trafos User-Friendly IDEs, Tools Safety and Security Architecture Science Methodology Language Engineering
  3. 3. What is a Language 1 How Domain Requirements Shape Languages
  4. 4. Metamodel for Business Logic Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns.
  5. 5. Metamodel for Business Logic { Data Structures Behavioral Rules Expressions ValidaFons Special Types (eg. temporal) ...
  6. 6. Metamodel for Business Logic SemanFcs Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns. Well-defined meaning of this data structure
  7. 7. Metamodel for Business Logic SemanFcs Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns. Well-defined meaning of this data structure IDE Support is possible EvoluFon is possible Portability is possible
  8. 8. Metamodel for Business Logic SemanFcs Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns. Well-defined meaning of this data structure IDE Support is possible EvoluFon is possible Portability is possible Type Checking Solver-IntegraFon Model Checking Contracts
  9. 9. Tech Infrastructure Metamodel for Business Logic SemanFcs Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns. Well-defined meaning of this data structure ExecuFon Engine Technical Pla_orm for correct, efficient and scalable execuFon
  10. 10. Metamodel for Business Logic SemanFcs Clearly defined data structure to express all business-relevant structures, behaviors and non-funcFonal concerns. Well-defined meaning of this data structure Technical Pla_orm for correct, efficient and scalable execuFon Tech Infrastructure generate code, deploy transfer data, interpret
  11. 11. Tech Infrastructure generate code, deploy transfer data, interpret + Code Inspec9on + Debugging + Performance & Op9miza9on + PlaBorm Con- formance GeneraFon InterpretaFon + Turn around Time + Run9me Change
  12. 12. Metamodel for Business Logic SemanFcs Tech Infrastructure generate code, deploy transfer data, interpret
  13. 13. Tech Infrastructure Metamodel for Business Logic SemanFcs Syntax L a n g u a g e generate code, deploy transfer data, interpret
  14. 14. Tech Infrastructure Metamodel for Business Logic SemanFcs Syntax L a n g u a g e generate code, deploy transfer data, interpret Syntax is criFcally important for ProducFvity CommunicaFon and Review Domain Expert IntegraFon Only Buaons and Forms don‘t work!
  15. 15. Tech Infrastructure Metamodel for Business Logic SemanFcs Syntax L a n g u a g e generate code, deploy transfer data, interpret IDE is criFcally important for acceptance IDE
  16. 16. Tech Infrastructure Metamodel for Business Logic SemanFcs Syntax L a n g u a g e generate code, deploy transfer data, interpret IDE is criFcally important for acceptance Learning ExploraFon ProducFvity Tool IntegraFon IDE
  17. 17. Tech Infrastructure Metamodel for Business Logic SemanFcs Syntax L a n g u a g e generate code, deploy transfer data, interpret IDE L a n g u a g e W o r b e n c h Language Workbenches let you efficiently implement languages and IDEs. They are essenFal for the language implementor.
  18. 18. What is Language Design 2 How Domain Requirements Shape Languages
  19. 19. L a n g u a g e W o r b e n c h Requirements from Domain Language ImplementaFon Other Requirements ImplementaFon R D Iterate! Language Design
  20. 20. Language Design Forces Constraints Tradeoffs CommunicaFon Compromises ValidaFon { Done by people. Supported by Tools.
  21. 21. Domain Structure Model Purpose Analyze, Generate User Skills SoNware Engineering PracFces Non FuncFonals Permissions, IP, Sharing Tool CapabiliFes NotaFons, EdiFng, Scale Sep. of Concerns Different Views Get a beaer tool :-) Refactor towards Structure Educate, Put results in context Language Design::Influences
  22. 22. Domain Structure Model Purpose Analyze, Generate User Skills SoNware Engineering PracFces Non FuncFonals Permissions, IP, Sharing Tool CapabiliFes NotaFons, EdiFng, Scale Sep. of Concerns Different Views Get a beaer tool :-) Refactor towards Structure Educate, Put results in context Language Design::Influences Style!
  23. 23. Good Language Design Fitness for purpose EvaluaFon. Balancing the Forces RaFonales. Conceptual Frameworks e.g. CogniFve Dimensions of NotaFons ? AbstracFon gradient Ÿ Closeness of mapping Ÿ Consistency Ÿ Diffuseness/Terseness Ÿ Error-proneness Ÿ Hard mental operaFons Ÿ Hidden dependencies Ÿ Juxtaposability Ÿ Premature commitment Ÿ Progressive evaluaFon Ÿ Role-expressiveness Ÿ Secondary notaFon and escape from formalism Ÿ Viscosity Ÿ Visibility
  24. 24. Language Design::Main Paaerns New Language GPL Extension ExisFng Domain NotaFon (Informal) FormalizaFon Formalized Language Reuse GPL incl. Expressions and TS Add/Embed DS-extensions CompaFble notaFonal style Reduce to GPL Analyze Domain to find AbstracFons Define suitable, new notaFons Rely on exisFng behavioral paradigm Reuse standard expression language Interpret/Generate to one or more GPLs Use exisFng notaFon from domain Clean up and formalize Generate/Interpret ONen import exisFng „models“
  25. 25. Language Design::Granularity Contract Structure? Inheritance? State Machine? Graphical or Textual? ImperaFve? Temporal Types? Error Handling? Syntax f. OpFon Types? vs. vs. ?
  26. 26. Language Great IDE Analyses Refactorings TesFng Debuggers AbstracFons NotaFons Syntax Coloring Code CompleFon Goto DefiniFon Relevant Good Errors Aligned with Processes Write Tests Run them Report Back Animate ExecuFon Simulators GOOD GREAT Language Design::More than Lang
  27. 27. Language Great IDE Analyses Refactorings TesFng Debuggers Language Design::Cross Influences Some Syntaxes can be beaer supported by IDEs More first-class abstracFons make analyses simpler Can only refactor what can be analyzed Syntax to express tests Some abstracFons are easier to debug Good test support may limit need for debugging
  28. 28. Example Domains & Languages 3 How Domain Requirements Shape Languages
  29. 29. Stakeholder integraFon, Scalable Business, Document GeneraFon + CerFficaFon Code Complexity, Frameworks (Autosar), Product Lines ReducFon of Accidental Complexity in Code, Process Conformance (Docs) A powerful language and IDE for exisFng frameworks (Industry Robots, ROS) Precise SpecificaFon and ImplementaFon of Insurance Products („Rules“) MulF-Paradigm Programming, not just Simulink and C Consistent Derived Documents Changing RegulaFons, Fast ImplementaFon, End User Empowerment
  30. 30. Mixed NotaFons and End User Programming Informal -> Semiformal -> Formal Languages + VerificaFon OptmizaFons, Performance and Concurrency Fundamentals: Editors, Type Systems, Trafos User-Friendly IDEs, Tools SimulaFon, ExecuFon, Test Consistent Derived Documents Language vs. Library, Code Gen ObservaFons: 1)  Every domain has different reasons why language engineering is used. 2)  No domain has only one reason – it is always a combina9on 3)  Languages, Editors, Type Systems and good IDEs are always important!
  31. 31. F I N A N C E Customer had „coded“ in Word Needed a real IDE Syntax to be retained if possible
  32. 32. Insurance Programs Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) No reuse No modularity No varibility
  33. 33. Insurance Programs Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) No reuse No modularity No varibility
  34. 34. Insurance Programs Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) No reuse No modularity No varibility
  35. 35. Insurance Programs „Pixelcrap“ Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) Printed, PDF No reuse No modularity No varibility
  36. 36. Insurance Programs „Pixelcrap“ C Code Implement Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) Printed, PDF Developer reads „spec“ Very idiomaFc implementaFon Dev acts as a human compiler and implements it in C No reuse No modularity No varibility
  37. 37. Insurance Programs „Pixelcrap“ C Code Debug Implement Specify/Program Write formal code in a DSL mixed with tables and text No tool support whatsoever No tesFng (except inspecFon) Printed, PDF Developer reads „spec“ Very idiomaFc implementaFon Dev acts as a human compiler and implements it in C Debugging directly in C Search-for-use by text search Don‘t trust the documents – may be outdated! No reuse No modularity No varibility
  38. 38. Insurance Programs Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Specify/Program/Test/Debug
  39. 39. Insurance Programs Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Specify/Program/Test/Debug
  40. 40. Insurance Programs Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Specify/Program/Test/Debug
  41. 41. Insurance Programs Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Specify/Program/Test/Debug
  42. 42. Insurance Programs Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Specify/Program/Test/Debug
  43. 43. Insurance Programs C Code Specify/Program/Test/Debug Write formal code in a DSL mixed with tables and text Now with IDE support and executable tests The same notaFon! Exactly the same C code. Generate
  44. 44. Insurance Programs C Code Generate Incremental Refinement/Refactoring of languages: ParFally automated migraFon of models Add model natural notaFons (insurance-specific, math) Add Support for modularity, reuse, variants SFll exactly the same C code, or improved as needed. Specify/Program/Test/Debug
  45. 45. Small-scale syntacFc decisions
  46. 46. Small-scale syntacFc decisions Tool support via paleaes is important for things that „cannot be typed“.
  47. 47. Structure/Guid. NotaFon Views IDE/Tool Learn/EffecFve + Mixed * Clean L - Text 1 Powerful E
  48. 48. Syntax is the „UI“ of the language. ONen perceived as the most important aspect from the customer‘s perspecFve. ReplicaFng exisFng syntax is oNen a good starFng point, if your tool can do it.
  49. 49. EMBEDDED SOFTWARE „General Purpose DSL“ Based on C Improved Robustness, ProducFvity Designed for end-user extensibility
  50. 50. An extensible set of integrated languages for embedded soNware engineering.
  51. 51. Composable extensions, Diverse notaFons
  52. 52. Different kinds of languages, as illustrated by the different distribuFons of aspect code.
  53. 53. Relying on modular language extension and extension composiFon.
  54. 54. Typical OO paaerns to support extensibility
  55. 55. Design for Extensibility The internal design decisions may not be relevant to the end user, but they are decisive wrt. the ability to modularly extend a language. This may in itself be a goal!
  56. 56. Integrated VerificaFons Bounded Model Checking for C using the CBMC tool.
  57. 57. Formal VerificaFon is simplified if the language has the right abstracFons. ... because no semanFc recovery is required.
  58. 58. A E R O S P A C E C-based satellite on-board soNware Standardized Architecture Lots of infrastructure code Lots of documentaFon
  59. 59. Further domain-specific extensions to C. Developed by end-user lang engineer.
  60. 60. Infrastructure Specifics in C
  61. 61. Infrastructure Specific in C
  62. 62. Generated DocumentaFon
  63. 63. An exisFng meta model / standard is oNen a great starFng point for a DSL. There is more to generate/derive than just executable soNware.
  64. 64. „ “I am comminng myself to develop the next satellite‘s on board soNware in one year instead of the usual 50.
  65. 65. The more infrastructure you have to deal with, the more can be generated. Remote communicaFon, protocols, buffering/serializaFon, mulF-pla_orm interfaces...
  66. 66. AUTOMOTIVE SpecificaFon of FuncFons and Interfaces And the Data communicated betw. Them Incrementally more specific
  67. 67. Start with a few ontological data definiFons
  68. 68. Define components that produce or consume the data items (allocate responsibiliFes).
  69. 69. Establish refinement relaFonships between data items.
  70. 70. Only then do you start defining types and value constraints.
  71. 71. Then decompose hierarchical system structures.
  72. 72. ... recursively.
  73. 73. Workflow may impact the language. Not defining a type for a variable is unintuiFve for the soNware engineer. But it is the way a systems engineer works. Make sure you accomodate this.
  74. 74. HEALTH/MEDICINE SoNware Medical Devices Accessible to Doctors Robustness/Correctness Required To be FDA-cerFfied
  75. 75. Semi-Graphical Expressions
  76. 76. Prose-like call syntax Extension funcFon can be called in dot-notaFon, perfectly suitable for developers.
  77. 77. Prose-like call syntax For non-programmers, a more prose-like notaFon is helpful. NoFce the prose-call facility is a modular extension of the expression language.
  78. 78. Once again, be creaFve in the notaFons you use to accomodate end users. They might not like your fancy higher-order, lambda-based pure funcFonal language.
  79. 79. Test + Monitoring Aspects System Model MS T & V Model MT ensures LWB System Implementa9on IS T&V Implementa9on IT ensures transform transform TS TT RTE Established approaches for developing reliable and safe systems applied to languages.
  80. 80. Test + Monitoring Aspects System Model MS T & V Model MT ensures LWB Behavior Constraints monitors System Implementa9on IS T&V Implementa9on IT ensures transform TS TT RTE Established approaches for developing reliable and safe systems applied to languages.
  81. 81. Test + Monitoring Aspects System Model MS T & V Model MT ensures LWB Behavior Constraints monitors System Implementa9on IS T&V Implementa9on IT ensures transform TS TT RTE During TesFng Established approaches for developing reliable and safe systems applied to languages.
  82. 82. Test + Monitoring Aspects System Model MS T & V Model MT ensures LWB Behavior Constraints monitors System Implementa9on IS T&V Implementa9on IT ensures transform TS TT RTE During Regular OperaFon Established approaches for developing reliable and safe systems applied to languages.
  83. 83. Redundancy in the Tool Chain The basic idea is that the system model and the test model are disFnct, and are transformed separately.
  84. 84. Redundancy in the Tool Chain A number of addiFonal steps must be taken to address possible error scenarios; many rely on redundancy.
  85. 85. Integrated VerificaFons Integrated SMT solver checks for logical and arithmeFc inconsistencies in decision tables. Solver also for decision trees, state machines and protocol/contract checking in other projects.
  86. 86. Non-FuncFonal Concerns can have a huge impact on language design and transformaFon architecture. Safety, Security, Performance
  87. 87. End Users vs. SemanFcs Estsblished NotaFons End-user Happyness Expressive Formalism VerificaFon feasible Ask user for X ... ... reply aYer 2 minutes otherwise Some kind of decision tree/diagram ? S1 S2 S3 /entry Ask aNer 2 min on Reply [data.isValid] State Machine Asynchronicity (User InteracFon) Time-based decisions (Fmeouts) Checking for completeness and other criteria (model checking) No problem for semanFc issues. But „strange“ way of expressing things. Hard to get accepted.
  88. 88. GeneraFng „nice“ reports Start with this ... ... derive this. Editable. Analyzable. Read-only. Traceback. Configurable.
  89. 89. GeneraFng „nice“ reports Read-only. Traceback. Configurable. { Generate parFally. Avoid some details. Mark ExcepFonal Paths. Show only Happy Path. Highlight Safety-Checks. ...
  90. 90. Generate derived arFfacts Review the DSL sources Pair programming Independent Development Levels of Domain Expert IntegraFon User sophisFcaFon Development over Fme.
  91. 91. SemanFcs First Language Design! Robust, reusable, embeddable. Established paradigms and semanFcs, but custom language Completely Custom LWB must support language composiFon!
  92. 92. E X P R E S S I O N S Reusable Expression Language Embedded in > 10 DSLs What is a good way to do opFon types?
  93. 93. Basic Expressions and types OpFon Types
  94. 94. What is a good way to do opFon types? Good == Safe Checkable Concise Non-Surprising Implementable ?can the value only be used in the then part? does type checking work? how writable and readable is it? is the syntax surprising or obvious? is it implementable with MPS and easily extensible for language extension developers? StarFng Point: a first-class expression ... Safe Checkable Concise Non-Surprising Implementable
  95. 95. Problem: what about nesFng? one has specify names for the value otherwise the val is ambiguous Safe Checkable Concise Non-Surprising Implementable SoluFon: two styles for opFonal names Safe Checkable Concise Non-Surprising Implementable We opted for the 2nd version, because IDE support is beaer: write post-facto we already had a framework for the as notaFon
  96. 96. NesFng: one can now write this Safe Checkable Concise Non-Surprising Implementable This is ugly! Beaer: support comma-separated mulFple tests Safe Checkable Concise Non-Surprising Implementable
  97. 97. NesFng: one can now write this Safe Checkable Concise Non-Surprising Implementable This is ugly! Beaer: support comma-separated mulFple tests Safe Checkable Concise Non-Surprising Implementable But: why not just use an if?
  98. 98. Why not just use an if? Safe Checkable Concise Non-Surprising Implementable Problem: how to analyze this? SoluFon: enfore some to be topmost expression Safe Checkable Concise Non-Surprising Implementable
  99. 99. Use an opFon as bool? Safe Checkable Concise Non-Surprising Implementable Discarded: too much magic. Use the expression in the then part? Safe Checkable Concise Non-Surprising Implementable Challenging to implement with MPS because - one cannot change the type of a node (x) depending on its context. - alternaFvely, the x in the then part must be a different lang concept with negaFve consequences for scoping. Adopted, accepFng the scoping challenge.
  100. 100. Final issue: comma in expressions Safe Checkable Concise Non-Surprising Implementable SoluFon: allow && but not other operators Safe Checkable Concise Non-Surprising Implementable
  101. 101. Final, adopted soluFon: Safe Checkable Concise Non-Surprising Implementable We accept the tradeoff between end-user syntax and the somewhat more complicated implementaFon (scoping).
  102. 102. Many concerns affect language design; tradeoffs are oNen required. Specifically, if the language is expected to be extended or embedded, since then the customers‘ language engineers are also stakeholders.
  103. 103. How to deal with numbers? ? Seems to be obvious: why think about it? We have int and real. Why bother? But many domains have to deal with -  numbers bigger than int or long -  fixed precision (e.g., currencies) -  numbers with specific ranges -  or other constraints on the values. This should be part of the language!
  104. 104. Ranges and Constraints
  105. 105. Modifying Precision
  106. 106. Modifying Limits
  107. 107. Ranges and Operators Ranges are checked staFcally by the TS. Other constraints are checked at runFme. (Later maybe with a solver).
  108. 108. Even „obvious“ issues may be worth revisiFng for DSLs. „Normal people“ don‘t really care about integer and real, they just want to work with numbers.
  109. 109. Science High-Performance CompuFng OpFmized Math on Specialized H/W Parallel CompuFng High-Throughput
  110. 110. Math computaFon with sliding windows
  111. 111. Metamodel for Business Logic SemanFcs Syntax L a n g u a g e IDE Different Data Layouts OpFmizing f. Registers Minimizing Mem Accs CPU GPU OpFmizing Generators.
  112. 112. „ “ANer making an algorithm efficient (for efficient execuFon), we cannot recognize the science or math anymore. We cannot maintain the code.
  113. 113. Metamodel for Business Logic SemanFcs Syntax L a n g u a g e IDE Different Data Layouts OpFmizing f. Registers Minimizing Mem Accs CPU GPU AbstracFons that contain enough semanFcs to enable generators! NotaFons that make „the power“ accessible to scienFtsts.
  114. 114. Meta 4 How Domain Requirements Shape Languages
  115. 115. MPS is bootstrapped, i.e. it is mostly built with itself. The MPS team and us are eaFng our own dogfood. Java Expressions and Statements (BL) Refactorings Inten9ons Ac9ons Structure Type System Transforma9on Editor f(x) Dataflow ImperaFve/OO DeclaraFve
  116. 116. MPS is bootstrapped, i.e. it is mostly built with itself. The MPS team and us are eaFng our own dogfood. Java Expressions and Statements (BL) Refactorings Inten9ons Ac9ons Structure Type System Transforma9on Editor Interpreter Math Tables Diagrams f(x) IncA Dataflow Pa]erns ImperaFve/OO DeclaraFve Tree
  117. 117. Interpreter DeclaraFve type mappings Type Guards Concise syntax, especially for recursive calls Composable (just as languages) AbstracFons for environments and stacks
  118. 118. Program Analysis (IncA) Restricted, DeclaraFve Language Transformed to Paaerns Executed incrementally Very fast!
  119. 119. New Editor NotaFons New primiFve editor cells for math, diagram, tables More query-oriented than default MPS notaFons NotaFon independent from Language
  120. 120. Java as an imperaFve base language is problemaFc Because of side effects, it is very hard to analyze and opFmize Java Expressions and Statements (BL) Refactorings Inten9ons Ac9ons Structure Type System Transforma9on Editor Interpreter Math Tables Diagrams f(x) IncA Dataflow Pa]erns ImperaFve/OO DeclaraFve
  121. 121. Java as an imperaFve base language is problemaFc Because of side effects, it is very hard to analyze and opFmize f(x) Refactorings Inten9ons Ac9ons Structure Type System Transforma9on Editor Interpreter Math Tables Diagrams IncA Dataflow Pa]erns ImperaFve/OO DeclaraFve We are invesFgaFng how to get a more funcFonal core into MPS
  122. 122. Summing up 5 How Domain Requirements Shape Languages
  123. 123. All languages shown in this talk are built with the open source JetBrains MPS language workbench.
  124. 124. Lots of further reading is available.
  125. 125. NotaFon is crucial for users. Be open to requirements and be creaFve. Rely on a solid semanFc foundaFon. Otherwise you cannot deliver the promises of DSLs. SimulaFon, VerificaFon and TesFng the models is a major part of the value-add of a DSL. Try to incorporate SE best pracFces (modularity, SoC) to keep overall complexity in check. Use a powerful language workbench to make sure it does not unduly constrain the design.
  126. 126. A personal note. 1999 UML Modeling with RaFonal Rose and „Skeleton Code“ generaFon. Classes, Stereotypes, Tagged Values, Simple OCL constraints.
  127. 127. A personal note. 2016 MPS with full code generaFon. Mixed notaFons, mulFple integrated languages, type checking and inferencing, data flow checking, refactoring, version control, diff/ merge, non-trivial transformaFons, languages for non-programmers. Modeling 2.0 „ “I cannot understand how developers sFll deal with modeling 1.0 – and then complain that it does not work. Try 2.0!

×