1. Photo (cc) Horia Varlan
Virtual Separation of Concerns
Toward preprocessors 2.0
Christian KƤstner
2. Christian KƤstner: Virtual Separation of Concerns 2
Software Product Lines & Features
ā¢ Set of related software systems (variants)
in a domain
ā¢ Generated from common code base
ā¢ Variants distinguished in terms of features
3. Christian KƤstner: Virtual Separation of Concerns 3
Conditional Compilation with
Preprocessors
ā Annotated code static int _rep_queue_filedone(...
DB_ENV *dbenv;
fragments REP *rep;
__rep_fileinfo_args *rfp; {
ā Variants with and #ifndef HAVE_QUEUE
COMPQUIET(rep, NULL);
without features COMPQUIET(rfp, NULL);
return (__db_no_queue_am(dbenv
#else
db_pgno_t first, last;
ā Common for product- u_int32_t flags;
int empty, ret, t_ret;
line implementation #ifdef DIAGNOSTIC
DB_MSGBUF mb;
in industry #endif
...
Excerpt from Oracleās Berkeley DB
Preprocessors in C, C++, Java ME, Pascal, Erlang, C#, Visual Basic, ā¦
GPP, GNU M4, SPLET, Frames/XVCL, Gears, pure::variants
4. Christian KƤstner: Virtual Separation of Concerns 4
Objections / Criticism
Designed in the 70th and hardly evolved since
ā#ifdef considered harmfulā
ā#ifdef hellā
āmaintenance becomes a āhit or missā processā
āis difficult to determine if the code being
viewed is actually compiled into the systemā
āincomprehensible source textsā
āprogramming errors are easy āCPP makes maintenance difficultā
to make and difficult to detectā
āsource code rapidly
āpreprocessor diagnostics are poorā becomes a mazeā
6. Christian KƤstner: Virtual Separation of Concerns 6
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
7. Christian KƤstner: Virtual Separation of Concerns 7
Problem 1: Lack of Separation of Concerns
ā¢ Scattered and tangled source code
ā¢ Global search to understand a feature
ā¢ Deleting obsolete feature as challenge
ā¢ Difficult distributed development
ā¢ Difficult reuse
Beispiel Berkeley DB:
Feature Sperren mit 1835 LOC
in 28 von 300 Dateien
an 155 Stellen
8. Christian KƤstner: Virtual Separation of Concerns 8
Problem 2: Obfuscation
ā¢ Mixture of two class Stack {
class Stack {
void push(Object o
languages void push(Object o
#ifdef SYNC
#ifdef SYNC
, Transaction txn
(C und #ifdef, ā¦) , Transaction txn
#endif
#endif
) {
ā¢ Control flow hard to ) {
if (o==null
if (o==null
#ifdef SYNC
#ifdef SYNC
follow || txn==null
|| txn==null
#endif
#endif
ā¢ Additional line breaks ) return;
) return;
#ifdef SYNC
#ifdef SYNC
destroy code layout Lock l=txn.lock(o);
Lock l=txn.lock(o);
#endif
#endif
ā¢ Long annotations elementData[size++] = o;
elementData[size++] = o;
#ifdef SYNC
#ifdef SYNC
difficult to recognize l.unlock();
l.unlock();
#endif
#endif
fireStackChanged();
fireStackChanged();
}
}
}
}
10. Christian KƤstner: Virtual Separation of Concerns 10
Problem 3: Error-Prone
ā¢ Syntax Errors ā¢ Type Errors
static int _rep_queue_filedone(...) #ifdef TABLES
DB_ENV *dbenv; class Table {
REP *rep; void insert(Object data,
__rep_fileinfo_args *rfp; { Txn txn) {
#ifndef HAVE_QUEUE storage.set(data,
COMPQUIET(rep, NULL); txn.getLock());
COMPQUIET(rfp, NULL); }
return (__db_no_queue_am(dbenv)); }
#else #endif
db_pgno_t first, last; class Storage {
u_int32_t flags; #ifdef WRITE
int empty, ret, t_ret; boolean set(ā¦) { ... }
#ifdef DIAGNOSTIC #endif
DB_MSGBUF mb; }
#endif
// over 100 lines of additional code
}
#endif
11. Christian KƤstner: Virtual Separation of Concerns 11
Problem 3: Error-Prone
ā¢ Syntax Errors ā¢ Type Errors
static int _rep_queue_filedone(...) #ifdef TABLES
DB_ENV *dbenv; class Table {
REP *rep; void insert(Object data,
__rep_fileinfo_args *rfp; { Txn txn) {
#ifndef HAVE_QUEUE storage.set(data,
COMPQUIET(rep, NULL); txn.getLock());
COMPQUIET(rfp, NULL); }
return (__db_no_queue_am(dbenv)); }
#else #endif
db_pgno_t first, last; class Storage {
u_int32_t flags; #ifdef WRITE
int empty, ret, t_ret; boolean set(ā¦) { ... }
#ifdef DIAGNOSTIC #endif
DB_MSGBUF mb; }
#endif
// over 100 lines of additional code
}
#endif
12. Christian KƤstner: Virtual Separation of Concerns 12
Problem 3: Error-Prone
ā¢ Syntax Errors ā¢ Type Errors
static int _rep_queue_filedone(...) #ifdef TABLES
DB_ENV *dbenv; class Table {
REP *rep; void insert(Object data,
__rep_fileinfo_args *rfp; { Txn txn) {
#ifndef HAVE_QUEUE storage.set(data,
COMPQUIET(rep, NULL); txn.getLock());
COMPQUIET(rfp, NULL); }
return (__db_no_queue_am(dbenv)); }
#else #endif
db_pgno_t first, last; class Storage {
u_int32_t flags; #ifdef WRITE
int empty, ret, t_ret; boolean set(ā¦) { ... }
#ifdef DIAGNOSTIC #endif
DB_MSGBUF mb; }
#endif
// over 100 lines of additional code
}
#endif
13. Christian KƤstner: Virtual Separation of Concerns 13
Advantages
ā¢ Easy to use
ā¢ āannotate and removeā
ā¢ Already part of many languages / simple tools
ā¢ Most developers already familiar
ā¢ Flexible / Expressive
ā¢ Language-independent / Uniform
ā¢ Low Adoption Barrier
ā¢ esp. with legacy code
14. Christian KƤstner: Virtual Separation of Concerns 14
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
17. Christian KƤstner: Virtual Separation of Concerns 17
View on Feature Eval
ā¢ Shows all files containing Eval code
ā¢ Shows context information (kursiv)
ā¢ Hides code without annotations
18. Christian KƤstner: Virtual Separation of Concerns 18
View on the Variant āAdd and Evalā
ā¢ Entirely hides file Power
ā¢ Hides method print in Add ([ā¦])
ā¢ Code without annotation remains visible
19. Christian KƤstner: Virtual Separation of Concerns 19
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
21. Christian KƤstner: Virtual Separation of Concerns 21
Alternative Visual Representations
ā NetBeans ā Spotlight
22. Christian KƤstner: Virtual Separation of Concerns 22
Scaling Visual Representations
ā¢ Focus on few features at a time
ā¢ Repeating colors / manual assignment
sufficient
ā¢ Analysis of 4 Java ME and 40 C programs:
ā¢ 96 % pages of source code with ā¤ 3 colors
ā¢ 99 % pages of source code with ā¤ 7 colors
23. Christian KƤstner: Virtual Separation of Concerns 23
Program Comprehension: An Experiment
ā¢ #ifdef vs. colors
ā¢ 43 subjects in 2
groups
ā¢ S1-2: search tasks
faster with colors
(43% & 23%)
ā¢ M1-3: maintenance
tasks same perform.
ā¢ M4: maintenance
task with red back-
ground color -37%
ā¢ No influence on correctness
ā¢ Subjects prefer colors
24. Christian KƤstner: Virtual Separation of Concerns 24
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
25. Christian KƤstner: Virtual Separation of Concerns 25
[TOOLSā09]
Disciplined Annotations
class Foo { }
ā¢ Syntax errors caused by
annotations on plain text class Foo { }
ā¢ Solution: Use underlying artifact structure
ā¢ Enforce disciplined annotations on entire classes,
methods, or statements
ā¢ Annotations of isolated brackets or ClassDeclaration
Name=C
keywords regarded as undisciplined MethodDeclaration
Name=m
ReturnType Parameter
class C { Type-void Name=p
Block
void m(int p) {
s1();
MethodInvk MethodInvk
s2(p, true); Name=s1 Name=s2
}
} Parameter Parameter
Name=p Name=true
26. Christian KƤstner: Virtual Separation of Concerns 26
Variant Generation by AST Transformation
ClassDeclaration ClassDeclaration
Name=C Name=C
MethodDeclaration
MethodDeclaration
Name=m
Name=m
ReturnType Parameter
Block
Type-void Name=p ReturnType Parameter
Block
Type-void Name=p
MethodInvk MethodInvk
Name=s1 Name=s2
variant MethodInvk
Name=s1
Parameter Parameter generation
Name=p Name=true
print /
parsing unparse
class C { class C {
void m(int p) { void m(int p) {
s1(); s1();
s2(p, true); }
} }
}
Variant generation cannot cause syntax errors.
27. Christian KƤstner: Virtual Separation of Concerns 27
Expressiveness of Disciplined Annotations
ā¢ Not all annotations are possible
ā¢ Sometimes workarounds required
ā¢ Experience: not a significant restriction in
practice
ā¢ ~90% of all annotations in 40 C programs is
disciplined already
high flexibility no flexibility
unrestricted disciplined annotations classes no
annotations (underlying structure) only annotations
on any character
28. Christian KƤstner: Virtual Separation of Concerns 28
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
29. Christian KƤstner: Virtual Separation of Concerns 29
[ASEā08]
Product-Line-Aware Type System
class Database {
Storage storage;
ā¢ Base: Type correct without void insert(Object data
Txn txn) {
annotations (for tool support) storage.set(data,
txn.getLock())
ā¢ Detecting all pairs: }
}
ā¢ Method invocation and class Storage {
#ifdef WRITE
method declaration boolean set(ā¦) { ... }
ā¢ Reference and declaration #endif
}
of class, variable, etc.
ā¢ ā¦ Related Work:
ā¢ Comparison of annotations for ā¢ SafeGen
ā¢ fmp2rsm
each pair ā¢ Safe Composition
ā¢ FFJPL
Declaration annotated + Error in some ā¢ Conditional
Reference not annotated variants Methods
30. Christian KƤstner: Virtual Separation of Concerns 30
Type Checking in the Context
of a Feature Model
Ā¬Read
ā¢ Handling dependencies class Database {
between features void insert(ā¦) {
storage.set(ā¦);
ā¢ Can all variants with
}
}
class Storage {
reference reach a boolean set(ā¦) { ... }
corresp. declaration? }
Write
FM = API ā§ ( API ā
ā¢ Implementation: Propositional (read āØ write))
logic and SAT solvers:
In all variants in which code fragment A is
included also code fragment B is included:
FM ā ( AT (a ) ā AT (b))
31. Christian KƤstner: Virtual Separation of Concerns 31
[ASEā08]
Formalization: CFJ
ā On top of Featherweight Java
ā Theorem: Generation preserves typing
ā Formal proof with Coq
33. Christian KƤstner: Virtual Separation of Concerns 33
[GPCEā09]
Automated Refactorings
ā¢ Feature Module ā Annotationen
class Stack {
class Stack { class Stack {
class Stack {
int[] data;
int[] data; int[] data;
int[] data;
Base
void push(int o) { ā¦ }
void push(int o) { ā¦ } #ifdef UNDO
#ifdef UNDO
int pop() { /*...*/ }
int pop() { /*...*/ } int backup;
int backup;
}
} void undo() { /*...*/ }
void undo() { /*...*/ }
#endif
#endif
refines class Stack {
refines class Stack { #ifdef TOP
#ifdef TOP
Automated Transformationtop() { /*...*/ }
Top
int top() { /*...*/ }
int top() { /*...*/ } int top() { /*...*/ }
int
}
} #endif
#endif
void push(int o) {
void push(int o) {
refines class Stack {
refines class Stack { #ifdef UNDO
#ifdef UNDO
int backup;
int backup; backup = top();
backup = top();
void undo() { /*...*/ }
void undo() { /*...*/ } #endif
#endif
Undo
void push(int o) {
void push(int o) { /*...*/
/*...*/
backup = top();
backup = top(); }
}
original(v);
original(v); int pop() { /*...*/ }
int pop() { /*...*/ }
}
} }
}
34. Christian KƤstner: Virtual Separation of Concerns 34
Agenda
ā¢ Problems and Advantages of Preprocessors
ā¢ 4 Improvements
ā¢ Views
ā¢ Visual Representation
ā¢ Disciplined Annotations
ā¢ Product-Line-Aware Type System
ā¢ Summary and Perspective
35. Christian KƤstner: Virtual Separation of Concerns 35
Summary
ā¢ Problems: ā¢ Improvements:
ā¢ Separation of Concerns ā¢ Views
ā¢ Obfuscation ā¢ Visual representation
ā¢ Error-proneness ā¢ Disciplined annotations
ā¢ Product-line-aware type
system
ā¢ Retained advantages:
ā¢ Easy to use
ā¢ Flexible
ā¢ Language-independent
ā¢ Low adoption barrier
āVirtual Separation of Concernsā
36. Christian KƤstner: Virtual Separation of Concerns 36
Summary
ā¢ Problems: ā¢ Improvements:
ā¢ Separation of Concerns ā¢ Views
ā¢ Obfuscation ā¢ Visual representation
ā¢ Error-proneness ā¢ Disciplined annotations
ā¢ Product-line-aware type
system
ā¢ Retained advantages:
ā¢ Easy to use
ā¢ Flexible
ā¢ Language-independent
ā¢ Low adoption barrier
āVirtual Separation of Concernsā
37. Christian KƤstner: Virtual Separation of Concerns 37
Summary
ā¢ Problems: ā¢ Improvements:
ā¢ Separation of Concerns ā¢ Views
ā¢ Obfuscation ā¢ Visual representation
ā¢ Error-proneness ā¢ Disciplined annotations
ā¢ Product-line-aware type
system
ā¢ Retained advantages:
ā¢ Easy to use ā¢ Remaining Problems:
ā¢ Flexible ā¢ Separate Compilation
ā¢ Language-independent ā¢ Black-box Reuse
ā¢ Low adoption barrier
āVirtual Separation of Concernsā
38. Christian KƤstner: Virtual Separation of Concerns 38
Perspective
ā¢ Preprocessors common in practice,
despite strong criticism
ā¢ Neglected by researchers
ā¢ Tool support can address most problems
ā¢ Interim solution (with migration path) or
serious alternative?
ā¢ Give preprocessors a second chance!