Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
D	
  in	
  real	
  life
By:	
  Liran	
  Zvibel,	
  Weka.IO	
  CTO
2D	
  in	
  real	
  life
• Defining	
  the	
  future	
  of	
  software	
  defined,	
  scale	
  out	
  storage	
  for	
  th...
• Extremely	
  reliable	
  
• High	
  performance	
  data	
  path	
  
• Complicated	
  “control	
  plane”/“Management	
  c...
• C	
  codebase	
  
• A	
  lot	
  of	
  auto-­‐generated	
  code	
  from	
  XML	
  for	
  RPC,	
  clustering	
  code	
  an...
• Data-­‐path	
  in	
  C++,	
  control	
  path	
  in	
  Python	
  (or	
  Java,	
  C#,	
  Scala,	
  etc)	
  
- Deciding	
  ...
6D	
  in	
  real	
  life
“Often, programming teams will resort to a hybrid approach, where they
will mix Python and C++, t...
• Created	
  ’99	
  by	
  Walter	
  Bright,	
  Andrei	
  Alexandrescu,	
  as	
  a	
  C++	
  replacement	
  
• Work	
  on	
...
• Smaller	
  community	
  
• Stability	
  reminds	
  C++	
  of	
  the	
  early	
  2000s	
  
• Binary	
  compatible	
  to	
...
• range	
  is	
  the	
  D	
  equivalent	
  of	
  an	
  iterable	
  	
  
• InputRange	
  requires	
  3	
  basic	
  methods:...
Output copy(Input, Output)(ref Input source,
ref Output sink)
if (isInputRange!Input &&
isOutputRange!(Output, ElementType...
Symbol[] getSymbols(string filename)
{
return executeShell("nm -S " ~ filename)
.output
.splitLines
.map!” a.split()"
.fil...
bool binarySearch(T)(T[] input, T value) {
   while (!input.empty) {
      auto i = input.length / 2;
      auto mid = inp...
• It	
  supports	
  raw	
  pointers	
  if	
  required	
  
• Has	
  native	
  C	
  ABI	
  so	
  linking	
  with	
  existing...
• It	
  supports	
  Garbage	
  Collection	
  
- this	
  is	
  the	
  default,	
  but	
  you	
  can	
  opt-­‐out,	
  do	
  ...
@StatsCategory(“fs_driver_user"):
@RateQuery @StatUnit("Ops/Sec") {
@("Number of read operations per second")
IncrementalS...
• It	
  has	
  modules,	
  packages	
  and	
  imports	
  system	
  
• Native	
  Array,	
  Tuple	
  and	
  Associative	
  A...
auto dumpField(U)(U field) {
static if (__traits(hasMember, field, "toJSON")) {
return field.toJSON();
} else static if (i...
JSONValue structToJSON(T)(ref T obj)
if (is(T == struct) || is(T == class)) {
string[string] dummy;
auto jv = JSONValue(du...
struct RobinHashTable(K, V,
size_t maxLength) {
static if (NumEntries <
ushort.max-1) {
alias CellIdx = ushort;
} else {
a...
• All	
  variables	
  are	
  initialized	
  to	
  default	
  values	
  
• Contract	
  programming	
  
• Casts	
  are	
  ex...
• static	
  if	
  
• scope(exit/success/failure)	
  statement	
  
• Unit	
  testing	
  support	
  
• -­‐debug	
  and	
  -­...
void testing(T...)(T values) {
   writeln("Called with ", values.length, " arguments.");
   // Access each index and each ...
string makeHammingWeightsTable(string name, uint max = 255) {
    string result = "immutable ubyte[%s] %s = [ ".format(max...
Questions?
Peta	
  	
  
Exa	
  
Zetta	
  	
  	
  
Yotta	
  
Xenna	
  
Weka	
  (1030)
• Tracing	
  system	
  
- space	
  and	
  cpu	
  efficient	
  
- keep	
  track	
  of	
  func	
  enter/exit	
  
- Easy	
  w...
Upcoming SlideShare
Loading in …5
×

Introduction to D programming language at Weka.IO

2,643 views

Published on

Introduction and basic use cases of the D programming language at Weka.IO, a stealth mode storage startup.

Published in: Technology
  • Be the first to comment

Introduction to D programming language at Weka.IO

  1. 1. 1 D  in  real  life By:  Liran  Zvibel,  Weka.IO  CTO
  2. 2. 2D  in  real  life • Defining  the  future  of  software  defined,  scale  out  storage  for  the   cloud-­‐based  datacenter   • VC  backed  company  (NVP,  Gemini),  largest  round-­‐A  of  2014   • Large,  strong  engineering  team,  many  XIV  veterans   • Founded  by  Omri  Palmon,  Maor  Ben-­‐Dayan,  Liran  Zvibel About  Weka.IO
  3. 3. • Extremely  reliable   • High  performance  data  path   • Complicated  “control  plane”/“Management  code”   • Distributed  nature  due  to  HA  requirements   • Low  level  interaction  with  HW  devices   • Some  kernel-­‐level  code,  some  assembly   • Language  has  to  be  efficient  to  program,  and  fit  for   large  projects 3 Storage  system  requirements D  in  real  life
  4. 4. • C  codebase   • A  lot  of  auto-­‐generated  code  from  XML  for  RPC,  clustering  code  and   external  APIs  (CLI,  GUI)   - Requires  a  complicated  build  process   - Difficult  to  understand  where  “magic”  code  came  from   • Our  own  implementation  of  Classes/polymorphism  and  templates 4 What  did  we  do  at  XIV? D  in  real  life
  5. 5. • Data-­‐path  in  C++,  control  path  in  Python  (or  Java,  C#,  Scala,  etc)   - Deciding  whether  to  run  in  the  same  process  or  via  RPC,  both  options  have   downsides   • All  in  C++   - Outside  the  scope  of  this  talk,  C++  has  many  downsides  writing  very  large   projects,  many  pitfalls   • Google  Go   - Can’t  disable  GC,  no  generic  programming,  inheritance,  no  C  ABI.  Not  performant   • Mozilla  Rust   - Currently  a  moving  target 5 Some  other  possible  options D  in  real  life
  6. 6. 6D  in  real  life “Often, programming teams will resort to a hybrid approach, where they will mix Python and C++, trying to get the productivity of Python and the performance of C++.The frequency of this approach indicates that there is a large unmet need in the programming language department.” “D intends to fill that need. It combines the ability to do low-level manipulation of the machine with the latest technologies in building reliable, maintainable, portable, high-level code. D has moved well ahead of any other language in its abilities to support and integrate multiple paradigms like imperative, OOP, and generic programming.” — Walter Bright, Co-designer of D
  7. 7. • Created  ’99  by  Walter  Bright,  Andrei  Alexandrescu,  as  a  C++  replacement   • Work  on  D2  started  2007,  considered  stable  in  late  2010.   • Modern  programming  language  with  native  efficiency,  but  leveraging  20   years  of  experience  making  mistakes  while  designing  C++  and  Java.   • Combines  the  power  of  C++,  but  with  the  usability  of  languages  like   Python,  Java  and  C#.   • Has  several  compilers  :  dmd,  gdc,  ldc 7 What  is  D? D  in  real  life
  8. 8. • Smaller  community   • Stability  reminds  C++  of  the  early  2000s   • Binary  compatible  to  C,  no  source  compatibility   - We  must  “translate”  h  files  to  d   - Inability  to  inline  c  code  (that  would  otherwise  be  inlined  in  C++)   • Compilation  time  grows  if  many  templates  are  instantiated  across   different  compilation  units   • Missing  keyword  arguments  when  calling  functions  (a  la  python) 8 D  down  sides D  in  real  life
  9. 9. • range  is  the  D  equivalent  of  an  iterable     • InputRange  requires  3  basic  methods:   - bool empty; - T front; - void popFront(); • foreach(elem; range) • foreach(k, v; dict) • foreach(idx, ref e; array) 9 Ranges  and  Algorithms D  in  real  life
  10. 10. Output copy(Input, Output)(ref Input source, ref Output sink) if (isInputRange!Input && isOutputRange!(Output, ElementType!Input)) { foreach(c; source) sink.put(c); return sink; } 10 Simple  Algorithm D  in  real  life
  11. 11. Symbol[] getSymbols(string filename) { return executeShell("nm -S " ~ filename) .output .splitLines .map!” a.split()" .filter!"4 == a.length" .map!((x) => new Symbol(x)) .filter!`a.typeFits(["d", "D", "v", "V"])` .array; } 11 UFCS  and  Pipeline  programming D  in  real  life
  12. 12. bool binarySearch(T)(T[] input, T value) {    while (!input.empty) {       auto i = input.length / 2;       auto mid = input[i];       if (mid > value) input = input[0 .. i];       else if (mid < value) input = input[i + 1 .. $];       else return true;    }    return false; } 12 Slices D  in  real  life
  13. 13. • It  supports  raw  pointers  if  required   • Has  native  C  ABI  so  linking  with  existing  libraries  is  easier  than  C++   • Supports  static  local  variables   • Has  Templates  (Generic  programming)   - They  are  much  easier  and  flexible  than  in  C++   - D  supports  compile  time  execution  —  CTFE   • Supports  Class  hierarchies  and  polymorphism   • Statically  typed,  but  with  very  strong  type  inference   • Supports  operator  overloading     • Inline  assembler  support 13 D  is  similar  to  C++ D  in  real  life
  14. 14. • It  supports  Garbage  Collection   - this  is  the  default,  but  you  can  opt-­‐out,  do  pointer  arithmetic   - Compiler  helps  you  verify  GC  is  indeed  not  being  used   - Allows  prototyping  in  a  high  level  way,  then  optimizing  GC  out   where  needed   • It  has  inner-­‐classes     • It  has  delegates   • Vast  support  for  attributes   • Very  strong  type  introspection/reflection  allowing  “duck  typing” 14 Python  that  compiles? D  in  real  life
  15. 15. @StatsCategory(“fs_driver_user"): @RateQuery @StatUnit("Ops/Sec") { @("Number of read operations per second") IncrementalStat READS; @("Number of write operations per second") IncrementalStat WRITES; } @StatUnit("Microseconds", 0.1) { @("Average latency of READ operations") @MeasuredQuery!`{READS}` TimerStat READ_LATENCY; @("Average latency of WRITE operations") @MeasuredQuery!`{WRITES}` TimerStat WRITE_LATENCY } 15 Attributes  usage D  in  real  life
  16. 16. • It  has  modules,  packages  and  imports  system   • Native  Array,  Tuple  and  Associative  Array  types  with  bounds  checking   • Supports  nested  functions  and  closures   - Also  very  strong  anonymous  functions/delegates  (lambda  with  a  closure)   • Has  pure  (enforced!)  functional  programming  subset   - Higher  order  functions:  reduce,  map,  filter,  compose,  curry   - supports  lazy  evaluation  of  function  parameters   16 Python  that  compiles? D  in  real  life
  17. 17. auto dumpField(U)(U field) { static if (__traits(hasMember, field, "toJSON")) { return field.toJSON(); } else static if (is(typeof({auto x = JSONValue(field);}))) { return field; } else static if (is(typeof(structToJSON!U))) { return structToJSON(field); } else static if (isArray!U) { return field.map!(dumpField!(typeof(field[0]))).array; } else static if (is(typeof({text(field);}))) { return text(field); } else { return "<unjsonable>"; } } 17 Introspection D  in  real  life
  18. 18. JSONValue structToJSON(T)(ref T obj) if (is(T == struct) || is(T == class)) { string[string] dummy; auto jv = JSONValue(dummy); foreach(i, U; typeof(obj.tupleof)) { enum NAME = __traits(identifier, obj.tupleof[i]); jv.object[NAME] = dumpField(obj.tupleof[i]); } return jv; } 18 Introspection D  in  real  life
  19. 19. struct RobinHashTable(K, V, size_t maxLength) { static if (NumEntries < ushort.max-1) { alias CellIdx = ushort; } else { alias CellIdx = uint; } static if (K.sizeof % 8 < 7) { align(8) struct KV { align(1): K k; ubyte cellData; align(8): V v; } } 19 Compile  time  magic D  in  real  life else { align(8) struct KV { K k; align(8): V v; align(1): ubyte cellData; } }
  20. 20. • All  variables  are  initialized  to  default  values   • Contract  programming   • Casts  are  explicit,  with  special  keyword   • Exceptions  support   • mixin  support   • Concurrent  programming  support   - native  thread  local  storage   - native  locks  and  message  passing  mechanisms 20 D  is  for  real  SW  engineers D  in  real  life
  21. 21. • static  if   • scope(exit/success/failure)  statement   • Unit  testing  support   • -­‐debug  and  -­‐release  default  support   • Versioned  compilation  (special  debug  case)   • @safe/@trusted/@system  code   • Documentation  comments 21 D  is  for  real  SW  engineers D  in  real  life
  22. 22. void testing(T...)(T values) {    writeln("Called with ", values.length, " arguments.");    // Access each index and each value    foreach (i, value; values) {       writeln(i, ": ", typeid(T[i]), " ", value);    } } void main() { testing(5, "hello", 4.2); } Called with 3 arguments. 0: int 5 1: immutable(char)[] hello 2: double 4.2 22 (Hetero)  Variadic  functions D  in  real  life
  23. 23. string makeHammingWeightsTable(string name, uint max = 255) {     string result = "immutable ubyte[%s] %s = [ ".format(max+1,name);     foreach (b; 0 .. max + 1) {         result ~= to!string(bitsSet(b)) ~ ", ";     }     return result ~ "];"; } mixin(makeHammingWeightsTable("hwTable")); unittest {    assert(hwTable[10] == 2);    assert(hwTable[0] == 0);    assert(hwTable[255] == 8); } 23 Mixin,  CTFE,  Unittests D  in  real  life
  24. 24. Questions? Peta     Exa   Zetta       Yotta   Xenna   Weka  (1030)
  25. 25. • Tracing  system   - space  and  cpu  efficient   - keep  track  of  func  enter/exit   - Easy  way  to  keep  logs   • RPC   - Very  efficient   - Easy  to  use  and  declare   • CLI   • Statistics   • Events  (Telemetry)     25 Infrastructure D  in  real  life

×