Slides from a guest lecture I gave at TU/E in October 2005.

1. Generative Software Development
overview and examples
Eelco Visser
visser@acm.org
Universiteit Utrecht, Faculty of Science,
Department of Information and Computing Sciences,
Center for Software Technology
October 27, 2005

2. Generative Software Development
Generative software development is a system-family
approach, which focuses on automating the creation of
system-family members: a given system can be
automatically generated from a specification written in
one or more domain-specific languages.
K. Czarnecki. Overview of Generative Software Development. In J.-P. Bantre
et al. (Eds.): Unconventional Programming Paradigms (UPP) 2004, Mont
Saint-Michel, France, LNCS 3566, pp. 313-328, 2005

3. Outline
• Software system families
• Commonality and variability
• Domain engineering and application engineering
• Feature modeling
• Domain-specific languages
• Examples
• Generative programming
• Problem space
• Solution space
• Mapping
• Realizing generative software development
• Overview of techniques
• Example: assimilating embedded DSLs
• Risks
• Summary

4. Software System Families
Current research and practical experience suggest that
achieving significant progress with respect to software
reuse requires a paradigm shift towards modeling and
developing software system families rather than
individual systems.
Czarnecki 2005

5. Examples of Software System Families
• Compilers
• program representation, parsing, code generation, ...
• Web applications
• processing http requests, session management
• E-commerce systems
• products, user accounts, payment
• Office applications
• text processing
• Web browsers
• html rendering
• TV software

6. System Family Engineering
System-family engineering (product-line engineering)
seeks to exploit the commonalities among systems from
a given problem domain while managing the variabilities
among them in a systematic way.
Czarnecki 2005

7. Commonality and Variability
• E-commerce systems
• Commonalities: products, user accounts, payment, ...
• Variabilities: types of products, shipment methods, ...
• Web applications
• Commonalities: processing http requests, session management
• Variabilities: application workflow, page layout, ...
• Web browsers
• Commonalities: render HTML
• Variabilities: plugins for email, chatting, ...

8. Commonality and Variability
• Compilers
• Commonalities: program representation, parsing, code
generation, ...
• Variabilities: syntax and semantics of language
• Office applications
• Commonalities: text processing
• Variabilities: spellchecking, grammarchecking, ...
• TV software
• Commonalities: ?
• Variabilities: ?

9. System Family Engineering Process
system family engineering
=
domain engineering
+
application engineering
reusable assets
Domain engineering Application engineering
Domain analysis new requirements System requirements
Domain design System derivation
Domain implementation System tailoring

10. Domain Engineering = Development for Reuse
Development of reusable assets
• Reusable components, generic architectures, documentation
Software engineering process for a class of systems
• Domain analysis
• scope of family
• identify commonalities and variabilities
• Domain design
• common architecture
• plan creation of systems from reusable assets
• Domain implementation
• implementation of reusable assets

11. Feature Modeling
A method and notation to elicit and represent common and
variable features of the systems in a system family
• And: all subfeatures must be selected
• Alternative: only one subfeature can be selected
• Or: one or more can be selected
• Mandatory: features that are required
• Optional: features that are optional
Additional information
• constraints
• binding times
• default attribute values, default features
• stakeholders, priorities, etc.

12. Feature Modeling: Example
e-shop
payment taxCalculation shipping
creditCard debitCard electronicCheque addressResolution
country postalCode streetAddress

13. Feature Modeling: Alternative Notations
• Feature diagrams can be expressed as context-free grammars
• Feature selection is sentence generated by the grammar
e-shop : payment taxCalculation [shipping]
payment : paymentMethod+
paymentMethod : creditCard | debitCard | electronicCheque
taxCalculation : addressResolution
addressResolution : country | postalCode | streetAddres
• Expressivity of feature diagrams is restricted
• Features and other constraints can be expressed by means of
propositional formulae
See also D. Batory. Feature Models, Grammars, and Propositional Formulas. In
H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS 3714, pp. 7–20, 2005

14. Application Engineering = Development with Reuse
Build concrete applications using reusable assets
• Requirements = delta of generic system requirements
• System derivation: manual or automated construction of
system from reusable assets
Feedback to domain engineering
• Refine and adapt requirements
• Tailor reusable assets to application

15. Features vs Software Modules
Concrete features
• map to individual components
Crosscutting features
• can be implemented using aspect-oriented techniques
• Examples: logging, synchronization
Abstract features
• Also known as: non-functional requirements
• map to configuration of components and/or aspects
Grouping features
• Also known as: variation points
• map to common interface and plug-compatible components

16. Generative Software Development
Generative software development is a system-family
approach, which focuses on automating the creation of
system-family members: a given system can be
automatically generated from a specification written in
one or more domain-specific languages.
Czarnecki 2005

17. General Purpose Languages
Expressivity of general purpose languages is restricted to
• while(cond) { x = foo(y, z); }
• variable binding: storing values
• function calls: computing values
• control-flow: combining and choosing computations
This is a rather powerful and generic approach
• any computation can be expressed (Turing completeness)
• modularization facilities allow further abstraction
But it does not match application domains very well

18. Advantages of Domain-Specific Languages
• Domain-specific abstractions
• pre-defined abstractions for concepts in application domain
• Domain-specific concrete syntax
• natural notation for a domain
• Domain-specific error checking
• find and report errors based on domain knowledge and notation
• Domain-specific optimizations
• opportunities for optimizations based on domain knowledge
• Domain-specific tool support
• tools can be aware of domain

19. Domain-Specific Languages: Examples
Successful domain-specific languages exist for many domains
• Numerical computation
• Database queries
• Text search and analysis
• Symbolic computation
• Document transformation
• Workflow
• ...

20. Domain-Specific Languages: Arithmetic
x = c * y + z
• Notation and semantics based on long tradition in
mathematics
• General purpose language notation would require
x = plus(times(c, y), z)
x = z.plus(y.times(c))
x = c.times(y).plus(z)
• Compiler knows a lot about arithmetic
• optimizations (partial evaluation, strength reduction, ...)
• automatic coercions (int to real)
• exceptions

21. Domain-Specific Languages:
A = c * B + C

22. Domain-Specific Languages: Matrix Computation
A = c * B + C
• Now the elements of the assignment are matrices
• The expression ‘means’
for i = lb to ub do
A[i] = c * B[i] + C[i]

23. Domain-Specific Languages: Matrix Computation
A = c * B + C
• Now the elements of the assignment are matrices
• The expression ‘means’
for i = lb to ub do
A[i] = c * B[i] + C[i]
• General-purpose languages (C, C++, Java, ...) don’t
understand matrices
• ‘Mathematical’ languages such as Matlab and Octave do
• Matrix notation allows
• concise programs
• compiler may apply optimizations

24. Domain-Specific Languages: Database Queries
SELECT NAME
FROM ORDERS, INCLUDES, SUPPLIES
WHERE CUST = ’Chaudron’
AND ORDERS.O# = INCLUDUES.O#
AND INCLUDES.ITEM = SUPPLIES.ITEM
• SQL: notation for querying relational databases
• Abstracts from physical data structures of database
• Based on small set of relational algebra operators
• Allows efficient implementation of queries
• Queries can be checked against database schema

25. Domain-Specific Languages: Regular Expressions
^([a-z]+)/([A-Z][^/]+)/(.*)
• A regular expression describes or matches a set of strings,
according to certain syntax rules
• Used by many text editors and utilities to search and
manipulate bodies of text based on certain patterns.
• Many programming languages support regular expressions for
string manipulation.
• Notation abstracts from underlying finite automata

26. Domain-Specific Languages: Context-free Grammars
<postal-address> ::= <name-part> <street-address> <zip-part>
<personal-part> ::= <first-name> | <initial> "."
<name-part> ::= <personal-part> <last-name> [<jr-part>] <EOL>
| <personal-part> <name-part>
<street-address> ::= [<apt>] <house-num> <street-name> <EOL>
<zip-part> ::= <town-name> "," <state-code> <ZIP-code> <EOL>
• Describes syntax of a formal language
• Used to derive parsers that automatically recognize texts
• Writing parsers manually is complex and error-prone

27. Domain-Specific Languages: Rewrite Systems
|[ if 0 then s1 else s2 ]| -> |[ s2 ]|
|[ i + j ]| -> |[ k ]| where <add>(i,j) => k
• Rewrite rule replaces a matching term with another term
• Exhaustive application of rules reduces term to normal form
• Applications: symbolic computation, program transformation,

28. Domain-Specific Languages: Path Queries
/bookstore/book[price>35]/title
• XPATH path queries address parts of an XML document
• used in document transformation with XSLT

29. Domain-Specific Languages: Workflow

30. Domain-Specific Languages: Observations
• Domain-specific languages are everywhere
• DSL are sucessful when they cover a well-defined domain and
are based on solid foundations
• DSLs typically do not cover entire systems, but rather
well-defined parts/aspects of systems

31. Generative Software Development
Generative software development is a system-family
approach, which focuses on automating the creation of
system-family members: a given system can be
automatically generated from a specification written in
one or more domain-specific languages.
Czarnecki 2005

32. Generative Programming: Mapping Problems to Solutions
mapping
Problem space Solution space
Problem Space
• Set of domain specific abstractions
• Used to specify desired system family member
Solution Space
• Implementation oriented abstractions
• Can be instantiated to implement domain abstractions

33. Problem Space and Solution Space: Example
Problem Space
• Regular expressions
Solution Space
• Finite automata
• Expressed as transition table in code
Mapping
• Lexical analyzer generator (flex)

34. Mapping: Configuration View
Problem space
• domain-specific concepts and their features
• constraints: illegal feature combinations, default settings,
default dependencies
Specification
• selection of features for a specific system
Solution space
• implementation components, composed to create system
Mapping
• feature configuration mapped to component configuration
• mapping defined by construction rules and optimizations
Separation of spaces allows freedom of implementation

35. Configuration Example: Apache’s httpd.conf
Software composition
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
Configuration settings
<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>
Binding-time: configuration is processed at start-up time

36. Mapping: Transformational View
Problem space
• domain concepts captured by domain-specific language
Specification
• DSL program
Solution space
• implementation language
• components/libraries (‘run-time system’)
Mapping
• transformation from DSL to implementation language

37. Configuration vs Transformation
• Two views on mappings provide different perspectives
• Feature configuration defines a DSL with components as
implementation language

38. Combining Mappings
chaining
multiple problem spaces multiple solution spaces
alternative problem spaces alternative solution spaces

39. Combining Mappings
language
text search
definition
regexp
automaton embed
DFA SDF
sdf2pp sdf2sig pgen
codegen pp table signature SLR(0) table
parse
code sglr

40. Realizing Generative Software Development
Requires techniques/technologies for creating
• Domain-specific languages
• Mappings
• Solution components

41. Realizing Domain-Specific Languages
Domain-specific languages come in many shapes and forms
• Textual languages
• Embedded in a programming language (MetaBorg)
• Graphical languages
• Wizards, GUIs

42. Realizing Mappings
Product line configurators
• map configuration file + components to system
Generators, implemented using
• Template and frame processors
• Transformation systems
• Multi-staged programming
• Program specialization
• Built-in meta-programming facility
Interpreters
• run-time interpretation of DSL program

43. Realizing Solution Space
• Functions or classes
• Generic components
• Component models (JavaBeans, ActiveX, Corba)
• Aspect-oriented programming (AspectJ)

44. Combining Techniques
Generative software development can be realized using various
combinations of techniques
Example scenario:
• Textual embedded language
• ‘Assimilation’ by program transformation
• Library for basic functionality of target programs
Example scenario:
• Graphical model language
• Graph transformations to map to aspect-oriented programs
• Aspects ‘inject’ code from model into base application

45. Relation To Other Fields
Problem Space Configuration Knowledge Solution Space
System Families, Product Lines
Feature modeling, feature interactions
Components
!
Generators/Transformers
¨
Architectures
Domain-specific languages
©
Aspect-oriented programming
¨
§
Aspect-oriented modeling
¦
¥
¥
¤
£
¢
Generic programming
¢
¢
¡
60

46. Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
Examples
• Java + Swing Userinterface Language: Expressions reflect
hierarchy of Swing GUI objects
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations

47. Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
Concrete syntax for objects
Examples
syntactic abstractions for libraries (APIs)
• Java + Swing Userinterface Language: Expressions reflect
transformations to assimilate (and optimize)
hierarchy of Swing GUI objects
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations

48. Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
local to global transformation
Examples
generation of event handler class
• Java + Swing Userinterface regexp compiles
caching of Language: Expressions reflect
hierarchy of Swing GUI objects
adding class variable declarations
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations

49. Example: Embedding Regular Expressions
re := str | alpha | re1 |re2 | re1 re2 | re* | re+
e := /re/ e | /re;f / x
t := regexp regexps
/re/e tests whether regular expression re matches value of e

50. Example: Embedding Regular Expressions
re := str | alpha | re1 |re2 | re1 re2 | re* | re+
e := /re/ e | /re;f / x
t := regexp regexps
/re/e tests whether regular expression re matches value of e
function isEmail(x : string) : int
begin
return / alpha+ @ (alpha+ .)+ alpha+ / x;
end

51. Example: Embedding Regular Expressions
re := str | alpha | re1 |re2 | re1 re2 | re* | re+
e := /re/ e | /re;f / x
t := regexp regexps
/re/e tests whether regular expression re matches value of e
function isEmail(x : string) : int
begin
return / alpha+ @ (alpha+ .)+ alpha+ / x;
end
/re;f/e tests if regular expression re matches a prefix of e, and
the suffix satisfies predicate f

52. Assimilating Regular Expression Matching
function match(x : string) : int
/ (a | b) c / x;
Desugar =
ReMch + ReStr + ReAlt + ReSeq + ReKle + once-AddDef

53. Assimilating Regular Expression Matching
function match(x : string) : int
/ (a | b) c / x;
Rewrite rule
L : t1 - t2 where s
replace fragment matching t1 with t2
Desugar =
ReMch + ReStr + ReAlt + ReSeq + ReKle + once-AddDef
ReMch :
|[ /re/ e ]| -
|[ begin
var x : string; x := e;
return /re; isEmpty/ x;
end ]|
where new = x

54. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / (a | b) c; isEmpty / z;
end
Desugar =
ReMch + ReStr + ReAlt + ReSeq + ReKle + once-AddDef
ReMch :
|[ /re/ e ]| -
|[ begin
var x : string; x := e;
return /re; isEmpty/ x;
end ]|
where new = x

55. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / (a | b) c; isEmpty / z;
end
ReSeq :
|[ /re1 re2 ; f/ x ]| - |[ /re1 ; g/ x ]|
where new = g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| - |[ def def* ]| )

56. Assimilating Regular Expression Matching
Dynamic rules
rules( L : t1 - t2 )
defines new rewrite rule
inherits variable bindings from definition context
function match(x : string) : int
begin
var z : string; z := x; return / (a | b) c; isEmpty / z;
end
AddDef :
|[ def* ]| - |[ function k(a : string) : int /c; isEmpty/ a; def* ]|
ReSeq :
|[ /re1 re2 ; f/ x ]| - |[ /re1 ; g/ x ]|
where new = g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| - |[ def def* ]| )

57. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / (a | b) ; k / z;
end
AddDef :
|[ def* ]| - |[ function k(a : string) : int /c; isEmpty/ a; def* ]|
ReSeq :
|[ /re1 re2 ; f/ x ]| - |[ /re1 ; g/ x ]|
where new = g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| - |[ def def* ]| )

58. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / (a | b) ; k / z;
end
ReAlt :
|[ / (re1 | re2) ; f / x ]| - |[ (/re1;f/x) | (/re2;f/x) ]|

59. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return (/ a ; k / z) | (/ b ; k / z);
end
ReAlt :
|[ / (re1 | re2) ; f / x ]| - |[ (/re1;f/x) | (/re2;f/x) ]|

60. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return (/ a ; k / z) | (/ b ; k / z);
end
ReStr :
|[ /str ; f/ x ]| - |[ isPref(str, x) f(getSuf(str, x)) ]|

61. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x;
return (isPref(a,z) k(getSuf(a,z)))
| (isPref(b,z) k(getSuf(b,z)))
end
ReStr :
|[ /str ; f/ x ]| - |[ isPref(str, x) f(getSuf(str, x)) ]|

62. Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x;
return (isPref(a,z) k(getSuf(a,z)))
| (isPref(b,z) k(getSuf(b,z)))
endAddDef :
|[ def* ]| - |[ function k(a : string) : int /c; isEmpty/ a; def* ]|
Desugar =
ReMch + ReStr + ReAlt + ReSeq + ReKle + once-AddDef

63. Assimilating Regular Expression Matching
function k(a : string) : int
begin
return /c; isEmpty/ a;
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref(a,z) k(getSuf(a,z)))
| (isPref(b,z) k(getSuf(b,z)))
endAddDef :
|[ def* ]| - |[ function k(a : string) : int /c; isEmpty/ a; def* ]|
Desugar =
ReMch + ReStr + ReAlt + ReSeq + ReKle + once-AddDef

64. Assimilating Regular Expression Matching
function k(a : string) : int
begin
return /c; isEmpty/ a;
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref(a,z) k(getSuf(a,z)))
| (isPref(b,z) k(getSuf(b,z)))
end
ReStr :
|[ /str ; f/ x ]| - |[ isPref(str, x) f(getSuf(str, x)) ]|

65. Assimilating Regular Expression Matching
function k(a : string) : int
begin
return isPref(c, a) isEmpty(getSuf(c, a));
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref(a,z) k(getSuf(a,z)))
| (isPref(b,z) k(getSuf(b,z)))
end
ReStr :
|[ /str ; f/ x ]| - |[ isPref(str, x) f(getSuf(str, x)) ]|

66. Risks of Domain-Specific Languages
• Design and implementation effort
• implementing complete toolset for language is major effort
• Documentation
• tutorials, reference manual, implementation documentation
• Training
• need programmers that are skilled in using the language
• Maintenance
• of the language
• of DSL programs
• How does it fit in the existing work flow
• tool chain
• DSLs need abstraction/modularity features too
• DSL programs become large; reuse of parts needed

67. Conclusion
Summary
• Generative software development is encapsulation of domain
knowledge in generators and target components
• Domain-specific languages are the interface to this knowledge
• DSL programs can be seen as a configuration for the ‘solution
space’ components
Challenges
• Finding good formalizations and encapsulations of new
application domains
• A platform for cheap construction of DSLs and generators
• Integration of generators in workflow/toolchain

68. References
• K. Czarnecki. Overview of Generative Software Development.
In J.-P. Bantre et al. (Eds.): Unconventional Programming
Paradigms (UPP) 2004, Mont Saint-Michel, France, LNCS 3566,
pp. 313-328, 2005
• D. Batory. Feature Models, Grammars, and Propositional
Formulas. In H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS
3714, pp. 7–20, 2005
• M. Bravenboer and E. Visser. Concrete Syntax for Objects.
Domain-Specific Language Embedding and Assimilation
without Restrictions. In D. C. Schmidt, editor, Proceedings of the
19th ACM SIGPLAN Conference on Object-Oriented Programing,
Systems, Languages, and Applications (OOPSLA’04), pages
365–383, Vancouver, Canada, October 2004. ACM Press.
• E. Visser. Transformations for Abstractions. In J. Krinke and G.
Antoniol, editors, Fifth IEEE International Workshop on Source
Code Analysis and Manipulation (SCAM’05), Budapest, Hungary,
October 2005. IEEE Computer Society Press. (Keynote paper)

69. Further Studying
At Master Program Software Technology; Universiteit Utrecht
• Seminar on Software Generation and Configuration
• Master course on Program Transformation