DSL Best Practices

MD*
MD*
Best Practices
MarkusVoelter
MD*
Independent/itemis AG
www.voelter.devoelter@acm.org

About
Markus Voelter
Independent Consultant

About
Markus Voelter
Software Architecture
DSLs & MDSD
ProductLines

About
Markus Voelter
http://www.voelter.de
voelter@acm.org
skype: schogglad

Read the
Paper!
http://www.voelter.de/data/articles/DSLBestPractices-Website.pdf


DesigningLanguage

the

Sourcesforthe Language

Technical DSLs
extractedfrom
… framework
… library
… pattern (language)
… architecture
Language Sources

Business DSLs
extractedfrom
Language Sources

Domain Analysis
Language Sources

Domain Analysis
Language Sources
Not just software!

Limit Expressiveness

Configuration
select
…fromconfigspace
… Properties
… Feature Models

Customization
compose
… creatively
… Vocabulary
… Sentences
… Box and Line

Precision
preciselywhat
… facts
… declarative
… domainexpertscan!

AlgorithmicCompleteness
formallyhow
… automation
… execution
… in model processors
… developers can!

Use a 3GLifnecessary
Generate APIs, Hooks

Do you want
users
to build their own
abstractions
with the language?

DSL is a compromise:
… Domain Abstractions
… Reuse, Modularization, …
… All Data for Generation
… DSL Tool influences
Viewpoints?

Notation, Notation, Notation

Domain Users
caredeeply
Notation, …
aboutnotation!
„UI“ forthelanguage

Textual
Graphical
Form-Based
Spreadsheet
Notation, …

Parts
Notation, …

Convertible
Notation, …

Embedded
Notation, …

Notation, …
Tool
Specific!

Graphical vs. Textual

Graphical
Sequence/Flow
Grapical vs. Textual

Graphical
Relationships

Graphical
Timing

Textual
In all othercases!

Textual
Real simple…
…CVS/SVN Integration
… Diff/Merge
… Buildautomation
… Model Migration

Text + Visualization

Text + Visualization
… problem-specific
… anwers specific questions
… highlight specific aspects
… several different
visualizations

DSL Semantics

?
Whatdoesit all
mean?
DSL Semantics

DSL Semantics

DSL Semantics
Documentation

Viewpoints

Well-defined
DependenciesandConnectionPoints
Viewpoints

Try toavoid
overlapandtheneedforsynchronization.
Viewpoints

Viewpoints
Tool
Specific!

Partitioning

scale
MD* tools do not
Partitioning
arbitrarily!

Partitions
Partitioning
… separate resources
… != logicalstructure
… unitsof check in/out … processable separately
… unit of IP

X
cross
Partition
references
Partitioning
… lazy
… byproxy
… byname w/ linker

Partition
Partitioning
… not transparent
… partoflanguage design
… referencableelements
… „includepath“

Partitioning
Tool
Specific!

Evolution

Whatto do withexisting
models
Evolution
ifthe
language
changes?

MightRequire…
Evolution
… configmanagement
… version tag in models
… changetracking
… migration M2M

Think about…
Evolution
… backwardcompatibility
… deprecation
… instrumentation
… viewpoints + partitions

Evolution
Tool
Specific!

The Fallacyof
GenericLanguages

FallacyofGeneric Lang.

Youcan model
everything
with

Youcan model
everything
with
somehow!

Problem
Shoehorningdomainabstractionsintothe
genericlanguage

Problem
Sidetrackedbyexistingabstractionsandnotations

Very LimitedTool Support!
Notations/ Abstractionsextensiblevia Profiles

Meta Model
Complexity!

Wherearestandardsuseful?

Peoplehavetolearnunderlying
concepts anyway.

Is UML witha profilestill a standardlanguage?

On whichmetaleveldo I wanttostandardize?
M2 (UML), M3 (MOF)?

Isn‘t a DSLbased on MOFas „standard“ as a profilebased on UML?

Introduce an
intermediate
language

separate
DSL
viewpoints
UML + DSL
DS Model

BPM
is to analysts
what
UML
is to developers.

Butdon‘treinventthewheeleither.

Build your own languageinspiredbyexisting language

Learnfrom 3GLs

DSL != 3GL
Learnfrom 3GLs
But:

Namespaces
VisibilityEncapsulation
Scoping
Specialization
Cohesion
Coupling
abstract
…
Wecan learn:
Learnfrom 3GLs

Read thisBook:
Learnfrom 3GLs
Concepts, TechniquesandModels of Computer Programming
by Peter Van Roy and SeifHaridi

Type Systems

Knownfrom3GLs
Type Systems
Classes & Objects

Examples:
Components
& Instances
Type Systems
Data Types
& Instances
Config Table Defs
& ConfigTables

Viewpoint B
(Types)
Type Systems
instanceof
Viewpoint A
(Instances)

MetaMeta Model (M3)
instanceof
Viewpoint B
(Types)
MetaModel (M2)
Type Systems
instanceof
Viewpoint A
(Instances)
instanceof
Model (M1)

Support for ReuseandVariations

There‘smoreto
reuse
Reuse andVariations
than
partitions

The language
must provide
explicit
Reuse andVariations
support!
(wecanlearnthatfrom OO)

Specialization
overriding, overwriting
Leaving Holes
Reuse andVariations
for variant tofill in
InjectStuff
severalplacesat a time?

Specialization
Inheritance
Leaving Holes
Reuse andVariations
Template Method Pattern
InjectStuff
Aspect Orientation

Reuse andVariations
Makesureyou
delineate
the API!

Reuse andVariations
Domain Users
might not
understand this!

Who are 1st Class Citizens?

Big Language
Who are 1st Class Citizens
withmanyconcepts!

Small Language
withfew, but powefulconcepts!

Big Language
Easierfor Business Users
Concepts easy to find
COBOL style

Small Language
Technical DSLs
Conceptsharderto find
More expressive
Lisp Style

Do not
mix
Reuse andVariations
thetwostyles!

Teamwork Support

Versioning
Lock
Check in/out
Diff/compare
Merge
Branch
Tag
Teamwork Support

Versioning
Lock
Check in/out
Diff/compare
Merge
Branch
Tag
Teamwork Support
On the
Level ofthe
Concrete Syntax!

Versioning
Lock
Check in/out
Diff/compare
Merge
Branch
Tag
Teamwork Support
Together
withmanually
written Source!

Repository
vs.
Teamwork Support
File-Based

Repository
… Element-Specific
… Real-Time
… Oftengoodfor Business DSLs
Teamwork Support

File-Based
… like SCMs
… integrates wellwithmanuallywrittencode
… Technical DSLs
Teamwork Support

File-Based
… integratesvery well with (real) textual DSLs
Teamwork Support

Teamwork Support
Tool
Specific!

ToolingMatters!

The Language
is not
ToolingMatters!
enough!

Teamwork
Navigation
Overviews
Searching
Quick-find
Find-references
Show usage
Refactoring
Debugging
Code Completion
Syntax Highlighting
The Language
is not
ToolingMatters!
enough!

The Language
is not
ToolingMatters!
enough!
Thisis also
trueforthe
Meta
Developers!

Users shouldbeable
toworkwithandstore
wrong
or
ToolingMatters!
incomplete
models!

Users shouldbeable
toworkwithandstore
wrong
or
ToolingMatters!
incomplete
models!
Temporarily.
Noprocessing!

ToolingMatters!
Nightly
Build!


ProcessingModels

the

Interpretation
Generation
vs.

Interpretation
Interpretation vs. Generation

Generation

Generation
resultingcode
canbeeasily
inspected

Generation
resultingcode
canbeeasily
debugged

Generation
resultingcode
canbe
optimized
andmore
efficient

Generation
Templates canbe
derived
fromexistingcode

Generation
workaround
limitations
oftargetlanguage

Generation
nochanges
totargetenvironment
(leavesnotrace)

Generation
reuse
runtime infrastructure
(garbage collection, monitoring…)

Interpretation
faster
turnaround
noregeneration
test
build
deploy

Interpretation
for platform indepenence
an interpreter might be
less porting
effort

Combinations

Rich Domain SpecificPlatform

Rich Platform

Rich Platform
Grownwiththe DSL!

Extreme Case
Rich Platform
populates

Checks Firstand Separate

Language Structure
is not enough.
Checks firstand separate
Youneedconstraints.
Boolean expressionsthatvalidate
the model beyondstructure.

Model
G
… complex
Constraints
Code

Model
G
G‘
… complex
… duplication
Constraints
Constraints
Code
Code‘

separate phase
Model
firstclasscitizens
muchbetter.
Constraints
G
G‘
check asmany
aspossible.
makeittight.
Code
Code‘

different constraints
atdifferenttimes
orfor
different partitions
ofthe model
partition-local: editor, on-save
global: batch, on-request

check early.
Model
moresemantics.
Constraints
bettermessages.
T
Model‘
T
Model‘
G
Code

Model
Constraints 1
constraints 1 ok
T
implies
Model‘
constraints 2 ok
Constraints 2
implies
T
constraints 3 ok
Model‘
Constraints 3
G
Code

ERROR
WARNING
INFO

IntegratingGeneratedandManuallyWritten Code

Ifat all possible,
Do not modify
Integrated Gen/Man Code
Generated Code!

ProtectedRegions
are a badideabecause
generatedcode
is not a
throwaway
anymore.

ProtectedRegions
… needto check in
… sedimentofgeneratedcode

Better Approach:
Hooks
in thegeneratedcode

Better Approach:
Hooks
in thegeneratedcode
extension points, base class, abstract methods & subclassing, empty callback methods, delegate, implement interfaces, #include, reflection, AOP, design patterns, partial classes

ControlManuallyWritten Code

After codegeneration
how do youmakesure
developersfollow
Control Manual Code
all therequired
procedures?

procedures?
… subclass
… overwrite
… naming
conventions
Control Manual Code

Compiler Errors
are not enough.
Control Manual Code
wrong
abstraction
level!

generatechecks
againstthecodebase
Control Manual Code
evaluatedbytheIDE

if (false) { GeneratedBaseClass x = new ManualSubclass(); }
Control Manual Code

Care aboutgeneratedcode
Andrew Vargas

Generated Code a
Throwaway
Product?
Care aboutGenerated Code

Generated Code a
Throwaway
Product?
Yes.
Can beregenerated.

But:
must be…
… written (templates)
… understood
… debugged

But:
must be…
… written (templates)
… understood
… debugged
… extended
… programmedagainst
ifyoudon‘tgenerate100%:

Care!
… indent
… usegoodnames
… document
… modularize

Make CodeTrue to Model

Analyses on the model
canverify all kindsof
properties
aboutthe
Code truetothe Model
system.

Analyses on the model
canverify all kindsof
properties
aboutthe
system.
Iffthecodeis
true
tothe model

Use a clever
programming model
thatdoes not allow
violations.

generatethe
configuration
for
architectureanalysis
tools.
SoftwareTomographySonarJ
Structure 101
… andthelike

Viewpoint-AwareProcessing

whenandhow do you
and
validate
process
Viewpoint-Aware Processing
eachviewpoint?

whenandhow do you
and
validate
process
eachviewpoint?
Roles?
Process?

Generate in phases:
type
developer
implementmanualcode
deployment
integrator
run on targetsystem
behaviour
runtime
interpretstatemachine

Also consider
partitions
in thiscontext!

Overall ConfigurationViewpoint

Teamwork
Partitions
Viewpoints
Multiple Targets
Overall ConfigurationViewpt.
Multiple Configurations

Teamwork
Partitions
Viewpoints
Multiple Targets
Multiple Configurations
large ormany
models

Putthisoverall
configurationinto
a model:
configuration
viewpoint

Care About Templates

Templates
… importantasset
Care about Templates
… containplatform
knowledge
… tendtogrow
morecomplex

Care!
… modularize
… functions
… naming
… polymorphism
… aspects
… refactor

Indentforthetemplates
andthenuse a
beautifier
forthe
generatedcode.

Indentforthetemplates
andthenuse a
beautifier
forthe
generatedcode.
exceptifyouuse a langauage
withsemanticwhitespace!

GoodPlatform.
Fewer Templates.
Less Care.

Model-2-Model
ToSimplify Generators

Reducingtemplate
complexity
isimportant.
M2M toSimplify Generators

Reducingtempalte
complexity
isimportant.
Separation ofConcerns
isthewaytogo.

Insteadofputting
complexlogic
intothetemplates
putitinto an M2M
thatruns
beforecode gen.

In-language
reduction rules:

Model-2-Model
For Simulation andProof

Many
usefulformalisms
alreadyexist.
M2M for Simulation andProof

Many
usefulformalisms
alreadyexist.
Simulation
Proofs
Properties

Use an M2M
forthisifpossible.
OftentheinputisXML
so youactually
„generatecode“

Cascading

?
PIM?
Cascading
PSM?
PSSM?

BottomUp
Cascading
Works Better!

Cascading
M2T
Code + othertargetplatformartifacts

Cascading
M2M
M2M
M2T
M2T

M2M
Cascading
M2M
M2M
M2T
M2T

MODELS READ ONLY!
M2M
Cascading
M2M
M2M
M2T
M2T

Allowfor
Adaptations

EconomiesofScale
Reuse!
AllowforAdaptations

EconomiesofScale
Reuse!
AllowforAdaptations
But:
Adaptations!

EconomiesofScale
Reuse!
AllowforAdaptations
But:
Adaptations!
Unexpected

Annotations on Models
n/v pairs
AllowforAdaptations
tobeused in generators
Adaptations on Templates
Template-AO
Factories/Polymorphism

AllowforAdaptations
Becarefulto
delineate
the API!

Annotation Models

M2M
Annotation Models
M2M
M2T
M2T

M2M
Annotation Models
M2M
M2M‘
M2T
M2T

Annotation Model
references
elements in base model.
Annotation Models
Transformation
takes additional
informationinto
account.

Makesurethe
annotation model
onlycaptures
Annotation Models
Exceptions
fromthedefault
in thetemplates.

ClassifyBehaviour

?
Action
Semantics
ClassifyBehaviour
Languages
useful?

?
Action
Semantics
ClassifyBehaviour
Languages
useful! But…

Classify!
… statebased
… businessrules
ClassifyBehaviour
… mathematics

Classify!
… statebased
… businessrules
ClassifyBehaviour
… mathematics
… or a specific DSL

Classify!
… statebased
… businessrules
ClassifyBehaviour
… mathematics
… or a specific DSL
… 3GL code

Don‘tforget
Testing

Don‘t Forget Testing
Limited Expressiveness.
Reduced Need For Tests.

Constraint Checks.
A Form of Test.

Better Testing
Generator
Example
Models
Code
Based On
Binary
Test Cases
(hand written)
Tests

Better Testing
Generator
Example
Models
Code
Based On
Models and Language serve as meaningful „spec“ for what to test
Binary
Test Cases
(hand written)
Tests

Testing Generators
Generator
Reference
Model
Code

Testing Generators
Generator
Reference
Model
Code
Based On
Binary
Reference
Test Cases
Tests

TestingTransformations
M2M
ResultModel
Reference
Model

TestingTransformations
M2M
ResultModel
Reference
Model
Based On
Reference
Constraints
Tests

TestingMetware
Reference
Model
… maintaned!
… bymetaware
Reference
Test Cases
developers
Reference
Constraints

This tests
only the generators
Generator
Model
Code
Tests
Generator
Test Code

This tests not
the model:
self fulfilling!
Generator
Model
Code
Tests
Generator
Test Code

Separate test models
and generated test code
Model
Generator
Code
based on
tests
Test Model
(Test Language)
Generator
Test Code

Separate test models
and generate Mocks
Model
Generator
Code
Mocks
based on
tests
Generator
Test Model
(Test Language)
Generator
Test Code

Auto-Derived Test Models
work sometimes.
Model
Generator
Code
automatically derived
tests
Test Model
(Test Language)
Generator
Test Code


ProcessOrganization

the

Iterate!

Waterfallisbad!
WithorWithout MD*
Iterate!

Iterate!
Concepts

Iterate!
Concepts
Language

Iterate!
Examples
Concepts
Language

Generator+Tests
Iterate!
Examples
Concepts
Language

Generator+Tests
Editor Beautification
Iterate!
Examples
Concepts
Language

Co-Evolve Language andConcepts

Co-Evolve Language & Domain

Building a language
requires
Formalization

Building a language
requires
Formalization
requiresyouto
thinkanddecide
aboutthedomain.

requiresfrequent
Evolution!

and flexible, agile
Tooling!

Documentationis still necessary

The DSL and the „programs“ are documentation.
Documentation still necessary

The DSL and the „programs“ are documentation.
Not Quite!

Language Definition
is not a
Teaching Tool!

Tutorials
… Concepts
… Howtouse Language
… Howtointegrate
manualcode
Example-Driven!

Language Definition
captures
the WHAT
but not
the WHY

Rationales
… whytheconcepts?
… whywegenerate
whatwegenerate
… targetplatform de-
cisionsandidioms
Grammar as Reference!

Different Media

Reviews

Reviews
In mostcases,
peoplecan still
makemistakes.

DSL programs
aremoreconcise, so
Reviews
reviewsare
moreefficient.

Repeated „Mistakes“?
Add constraint check.
Reviews

Repeated „Mistakes“?
Add constraint check.
Reviews
Ormaybethelanguage
iswronganditis
not a mistake?

Letpeople do what
they‘regoodat

MD* hasseveral
clearlydefined
Let‘em do whatthey‘regoodat
Roles

Tech Experts
… evaluatetechnologies
… digdeep
… tune
… createtemplates
… spreadknowledge

Language Designer
… works w/ domain expert
abstractions, notations
… adds modularization, in-
heritance, „abstract“, etc.
… works w/ architect
generators, interpreters
… requires „metapeople“

App Developer
… caresaboutappdomain
… uses DSLs + metaware
… isisolatedfrom
technology, does not
have to care (that much)

Flip side:
Youactuallyneedpeople
whoaregoodatthis!

Domain Users
Programming?

Precision
!=
Programming
Domain Users Programming?

Precision
!=
Programming
DomainUsers
Programmers
Programming = Precision + X

Precision
… Scientists
… Insurance Mathematicians
… Logisticians
… Medical Doctors

If Domain Users
don‘tgetit
itmighthintat a
problemwith
thelanguage!
… orthedocumentation!

Things learned
from 3GLs
often not
intuitively
understandable
for domain people.

Domain Users
vs. Experts

Creatingthe Language
vs.
Domain Users vs. Experts
Usingthe Language

Creating: Domain Expert
Using: Domain User

Creating: Domain Expert
… senior
… complete
… bigpicture
… deep
… precise, formal
… guru

Using: Domain User
… not senior
… narrower
… shallow

Can Domain Users
understand
whattheExperts
putinto
thelanguage?
Verifyearlyandoften!

MetawareasProduct

Product:
… releaseschedule
… incrementaldevelopmt
MetawareasProduct
… requirementsmgt.
… issuetracking
… documentation
… supportstaff

Exhange People
… makemetadevelopers
develop real apps
MetawareasProduct
… letappdevelopers
developmetaware

Compatible
Organization

MD* requires
cross-project
CompatibleOrganization
work.

MD* requires
cross-project
work.
A strictproject-focused
organizationdoes
not work

Make
room & budget
forcross-cutting
work.
Open Source?

Forget PublishedCase Studies

How do youknow
ifitworks
foryoursituation?
Forget Published Case Studies

How do youknow
ifitworks
foryoursituation?
Do not judgeby
published
casestudies!

How do youknow
ifitworks
foryoursituation?
Do not judgeby
published
casestudies!
(yes, theyareworthreading
but don‘tdecidebased on them)

Instead:
Prototype!
… meaningful
… 4-8 personweeks
… incremental
… externalhelp?



Challenges

Mixing Notations
Challenges

Mixing Notations
Language Modularity
Challenges

Mixing Notations
Language Modularity
MetawareRefactoring
Challenges

Mixing Notations
Language Modularity
MetawareRefactoring
Model/Code Refactoring
Challenges

Mixing Notations
Language Modularity
MetawareRefactoring
Challenges
Automatic Model Migration

Mixing Notations
Language Modularity
MetawareRefactoring
Challenges
Model Debugging

Mixing Notations
Language Modularity
MetawareRefactoring
Challenges
Model Debugging
Interpretation -> Code Gen

Mixing Notations
Language Modularity
MetawareRefactoring
Challenges
Model Debugging
Interpretation -> Code Gen
Cartridges

THE END.
Thankyou.
Questions?

DSL Best Practices

0 comments

Notes on slide 1

3 Favorites