The document discusses generic programming in D and related concepts. It defines generic programming as finding the most general algorithm representation with the narrowest requirements and widest guarantees. This allows algorithms to be reused for different data types. It provides examples of implementing generic minimum and argminimum functions in D. The document also discusses how generic programming relates to generative programming, with code generation at compile-time using D's capabilities for compile-time evaluation and code injection via mixins.

1. Generic Programming Galore Using D
Andrei Alexandrescu, PhD
Research Scientist
Facebook
c 2011 Andrei Alexandrescu 1 / 33

2. 37% off
c 2011 Andrei Alexandrescu 2 / 33

3. Generic Programming
c 2011 Andrei Alexandrescu 3 / 33

4. What is Generic Programming?
c 2011 Andrei Alexandrescu 4 / 33

5. What is Generic Programming?
• Find most general algorithm representation
◦ Narrowest requirements
◦ Widest guarantees
◦ Big-O() encapsulation should be a crime
◦ No need to regress to hand-written code
c 2011 Andrei Alexandrescu 4 / 33

6. What is Generic Programming?
• Find most general algorithm representation
◦ Narrowest requirements
◦ Widest guarantees
◦ Big-O() encapsulation should be a crime
◦ No need to regress to hand-written code
• Define types to implement said requirements
c 2011 Andrei Alexandrescu 4 / 33

7. What is Generic Programming?
• Find most general algorithm representation
◦ Narrowest requirements
◦ Widest guarantees
◦ Big-O() encapsulation should be a crime
◦ No need to regress to hand-written code
• Define types to implement said requirements
• Leverage the algorithm for ultimate reuse
c 2011 Andrei Alexandrescu 4 / 33

8. What is Generic Programming?
• Find most general algorithm representation
◦ Narrowest requirements
◦ Widest guarantees
◦ Big-O() encapsulation should be a crime
◦ No need to regress to hand-written code
• Define types to implement said requirements
• Leverage the algorithm for ultimate reuse
• ...
• Profit!
c 2011 Andrei Alexandrescu 4 / 33

9. What is Generic Programming?
• Find most general algorithm representation
◦ Narrowest requirements
◦ Widest guarantees
◦ Big-O() encapsulation should be a crime
◦ No need to regress to hand-written code
• Define types to implement said requirements
• Leverage the algorithm for ultimate reuse
• ...
• Profit!
´
• Arguably one of the noblest endeavors of our metier
c 2011 Andrei Alexandrescu 4 / 33

10. Generic Programming
• “Write once, instantiate anywhere”
◦ Specialized implementations welcome
• Prefers static type information and static expansion
(macro style)
• Fosters strong mathematical underpinnings
◦ Minimizing assumptions common theme in math
• Tenuous relationship with binary interfaces
• Starts with algorithms, not interfaces or objects
• “Premature encapsulation is the root of some
derangement”
c 2011 Andrei Alexandrescu 5 / 33

11. Warmup: the “dream min”
• Should work at efficiency comparable to hand-written
code
• Take variadic arguments: min(a, b, ...)
◦ Avoid nonsensical calls min() and min(a)
• Work for all ordered types and conversions
• Decline incompatible types, without prejudice
c 2011 Andrei Alexandrescu 6 / 33

12. First prototype
auto min(L, R)(L lhs, R rhs) {
return rhs < lhs ? rhs : lhs;
}
auto a = min(x, y);
• This function is as efficient as any specialized,
handwritten version (true genericity)
• Deduction for argument types and result type
c 2011 Andrei Alexandrescu 7 / 33

13. Variadic arguments
auto min(T...)(T x) {
static if (x.length > 2)
return min(min(x[0], x[1]), x[2 .. $]);
else
return x[0] > x[1] ? x[1] : x[0];
}
...
auto m = min(a + b, 100, c);
• x is not an array
• This is not classic recursion
c 2011 Andrei Alexandrescu 8 / 33

14. Reject nonsensical calls
auto min(T...)(T x) if (x.length > 1) {
...
}
• Rejects calls with 0 or 1 arguments
• Allows other overloads to take over, e.g. min element
over a collection
• More work needed
◦ Only accept types with a valid intersection
◦ Only accept types comparable with ”<”
c 2011 Andrei Alexandrescu 9 / 33

15. Common type
• Task: Given a list of types, find the common type of all
template CommonType(T...)
{
static if (T.length == 1)
alias T[0] CommonType;
else
static if (is(typeof(1 ? T[0].init : T[1].init) U))
alias CommonType!(U, T[2 .. $]) CommonType;
else
alias void CommonType;
}
// Usage
static assert(is(CommonType!(int, short, long) == long));
c 2011 Andrei Alexandrescu 10 / 33

16. Using CommonType
auto min(T...)(T x)
if (x.length > 1
&& is(typeof(CommonType!T.init < CommonType!T.init)
== bool))
{
static if (x.length > 2)
return min(min(x[0], x[1]), x[2 .. $]);
else
return x[0] > x[1] ? x[1] : x[0];
}
c 2011 Andrei Alexandrescu 11 / 33

17. How about min over many elements?
auto min(R)(R range)
if (isInputRange!R &&
is(typeof(range.front < range.front) == bool))
{
auto result = range.front;
range.popFront();
foreach (e; range) {
if (e < result) result = e;
}
return result;
}
auto m = [ 1, 5, 2, 0, 7, 9 ].min();
• Works over anything that can be iterated
c 2011 Andrei Alexandrescu 12 / 33

18. How about argmin now?
auto argmin(alias fun, R)(R r)
if (isInputRange!R &&
is(typeof(fun(r.front) < fun(r.front)) == bool))
{
auto result = r.front;
auto cache = fun(result);
r.popFront();
foreach (e; r) {
auto cand = fun(e);
if (cand > cache) continue;
result = e;
cache = cand;
}
return result;
}
c 2011 Andrei Alexandrescu 13 / 33

19. argmin
auto s = ["abc", "a", "xz"];
auto m = s.argmin!((x) => x.length);
• Works on anything iterable and any predicate
• Predicate is passed by alias
• No loss of efficiency
c 2011 Andrei Alexandrescu 14 / 33

20. The Generative Connection
c 2011 Andrei Alexandrescu 15 / 33

21. Generative programming
• In brief: code that generates code
• Generic programming often requires algorithm
specialization
• Specification often present in a DSL
c 2011 Andrei Alexandrescu 16 / 33

22. Embedded DSLs
Force into host language’s syntax?
c 2011 Andrei Alexandrescu 17 / 33

23. Embedded DSLs
• Formatted printing?
c 2011 Andrei Alexandrescu 18 / 33

24. Embedded DSLs
• Formatted printing?
• Regular expressions?
c 2011 Andrei Alexandrescu 18 / 33

25. Embedded DSLs
• Formatted printing?
• Regular expressions?
• EBNF?
c 2011 Andrei Alexandrescu 18 / 33

26. Embedded DSLs
• Formatted printing?
• Regular expressions?
• EBNF?
• PEG?
c 2011 Andrei Alexandrescu 18 / 33

27. Embedded DSLs
• Formatted printing?
• Regular expressions?
• EBNF?
• PEG?
• SQL?
c 2011 Andrei Alexandrescu 18 / 33

28. Embedded DSLs
• Formatted printing?
• Regular expressions?
• EBNF?
• PEG?
• SQL?
• . . . Pasta for everyone!
c 2011 Andrei Alexandrescu 18 / 33

29. Embedded DSLs
Here: use with native grammar
Process during compilation
Generate D code accordingly
c 2011 Andrei Alexandrescu 19 / 33

30. Compile-Time Evaluation
• A large subset of D available for compile-time evaluation
ulong factorial(uint n) {
ulong result = 1;
foreach (i; 2 .. n) result *= i;
return result;
}
...
auto f1 = factorial(10); // run-time
static f2 = factorial(10); // compile-time
c 2011 Andrei Alexandrescu 20 / 33

31. Code injection with mixin
mixin("writeln("hello, world");");
mixin(generateSomeCode());
• Not as glamorous as AST manipulation but darn
effective
• Easy to understand and debug
• Now we have compile-time evaluation AND mixin. . .
c 2011 Andrei Alexandrescu 21 / 33

32. Wait a minute!
c 2011 Andrei Alexandrescu 22 / 33

33. Example: bitfields in library
struct A {
int a;
mixin(bitfields!(
uint, "x", 2,
int, "y", 3,
uint, "z", 2,
bool, "flag", 1));
}
A obj;
obj.x = 2;
obj.z = obj.x;
c 2011 Andrei Alexandrescu 23 / 33

34. Parser
import pegged.grammar; // by Philippe Sigaud
mixin(grammar("
Expr < Factor AddExpr*
AddExpr < (’+’/’-’) Factor
Factor < Primary MulExpr*
MulExpr < (’*’/’/’) Primary
Primary < Parens / Number / Variable
/ ’-’ Primary
Parens < ’(’ Expr ’)’
Number <~ [0-9]+
Variable <- Identifier
"));
c 2011 Andrei Alexandrescu 24 / 33

35. Usage
// Parsing at compile-time:
static parseTree1 = Expr.parse(
"1 + 2 - (3*x-5)*6");
pragma(msg, parseTree1.capture);
// Parsing at run-time:
auto parseTree2 = Expr.parse(readln());
writeln(parseTree2.capture);
c 2011 Andrei Alexandrescu 25 / 33

36. Scaling up
1000 lines of D grammar →
3000 lines D parser
c 2011 Andrei Alexandrescu 26 / 33

37. Highly integrated lex+yacc
c 2011 Andrei Alexandrescu 27 / 33

38. What about regexen?
c 2011 Andrei Alexandrescu 28 / 33

39. Compile-time regular expressions
• GSoC project by Dmitry Olshansky: FReD
• Fully UTF capable, no special casing for ASCII
• Two modes sharing the same backend:
auto r1 = regex("^.*/([^/]+)/?$");
static r2 = ctRegex!("^.*/([^/]+)/?$");
• Run-time version uses intrinsics in a few places
• Static version generates specialized automaton, then
compiles it
c 2011 Andrei Alexandrescu 29 / 33

41. Summary
c 2011 Andrei Alexandrescu 31 / 33

42. Summary
• Generic/generative programming—long unattained
promise
• D’s angle
◦ Powerful base language
◦ Type manipulation
◦ Compile-time evaluation
◦ Code generation
c 2011 Andrei Alexandrescu 32 / 33

43. Grill the Speaker!
More about ranges? • Thought the talk was
boring • Intriguing! • He lost me at mixin •
Awesome • Went over my head • Meh • I wonder
how I can implement binary search • What’s for
dinner? • Accent is bothersome • Real
programmers use C • Must. . . control. . . fist. . .
of. . . death. . .
c 2011 Andrei Alexandrescu 33 / 33