Writing a Cassandra   client in Perl
Stefano Rodighierostefano.rodighiero@gmail.com
@larsen
• High Write Throughput• Large Tables• Reports• Analytics
Cassandra
Bigtable: A distributed storage system for  structured data, 2006
Dynamo: Amazonshighly available keyvalue       store, 2007
Dynamopartitioning and  replication
ColumnFamilydata model similar   to Bigtables
High availability
Incremental scalability
Eventually consistent
Tunable tradeoffsbetween consistency    and latency
Minimal administration
(not quite so)
No SPOFp2p distribution model
Writing
Writinginsert(key,	  {column_name	  =>	  value})☺
Writing☺          R
Writing☺          R
Writing☺               17%          17%                            R                17%   17%
Writing☺          R
Writing☺          R
Writing a Cassandra   client in Perl
Talking with a server
Cassandra::CassandraClient Thrift::BinaryProtocol Thrift::FramedTransport  Thrift::Socket
Cassandra::CassandraClient Thrift::BinaryProtocol Thrift::FramedTransport  Thrift::Socket
Cassandra::CassandraClient   The Transport layer provides                             a simple abstraction for Thrift::Bin...
Cassandra::CassandraClient   The Protocol abstraction                             defines a mechanism to Thrift::BinaryProt...
Cassandra::CassandraClient   "High level" interface: login,                             reading & writing data, …
1) Install Cassandra2) Install Thrift
thrift	  -­‐gen	  perl	  cassandra.thrift
Login
thrift  service	  Cassandra	  {  	  	  #	  auth	  methods  	  	  void	  login(1:	  required	  AuthenticationRequest	  auth...
thrift thrift  service	  Cassandra	  {  	  	  #	  auth	  methods     struct	  AuthenticationRequest	  {  	  	  void	  logi...
my	  $ks	  =	  TestIPW;my	  $socket	  	  	  	  =	  Thrift::Socket-­‐>new($server_name,	  $server_port);my	  $transport	  =...
my	  $ks	  =	  TestIPW;my	  $socket	  	  	  	  =	  Thrift::Socket-­‐>new($server_name,	  $server_port);my	  $transport	  =...
thriftmy	  $ks	  =	  TestIPW;              service	  Cassandra	  {my	  $socket	  	  	  	  =	  Thrift::Socket-­‐>new($serve...
Reading
thrift  list<ColumnOrSuperColumn>	  get_slice(  	  	  1:required	  binary	  key,  	  	  2:required	  ColumnParent	  column...
thrift thrift  list<ColumnOrSuperColumn>	  get_slice(  	  	  1:required	  binary	  key,  	  	  2:required	  ColumnParent	 ...
thrift thrift  list<ColumnOrSuperColumn>	  get_slice(  	  	  1:required	  binary	  key,  	  	  2:required	  ColumnParent	 ...
thrift thrift  list<ColumnOrSuperColumn>	  get_slice(  	  	  1:required	  binary	  key,      struct	  SliceRange	  {  	  	...
thrift thrift  list<ColumnOrSuperColumn>	  get_slice(      enum	  ConsistencyLevel	  {      	  	  	  	  ONE	  =	  1,  	  	...
package	  Cassandra::ConsistencyLevel;use	  constant	  ONE	  	  	  	  	  	  	  	  	  	  =>	  1;use	  constant	  QUORUM	  	...
package	  Cassandra::ConsistencyLevel;use	  constant	  ONE	  	  	  	  	  	  	  	  	  	  =>	  1;use	  constant	  QUORUM	  	...
my	  $cf	  =	  20121011.larsen.tweets;my	  $columnParent	  =	  Cassandra::ColumnParent-­‐>new({column_family	  =>	  $cf});...
my	  $cf	  =	  20121011.larsen.tweets;my	  $columnParent	  =	  Cassandra::ColumnParent-­‐>new({column_family	  =>	  $cf});...
thrift                    list<ColumnOrSuperColumn>	  get_slice(my	  $cf	  =	  20121011.larsen.tweets;                  	 ...
Writing
thrift  void insert(1:required binary key,·              2:required ColumnParent column_parent,              3:required Co...
thrift thrift   struct	  Column	  {  void insert(1:required binary key,·   	  	  	  1:	  required	  binary	  name, column_...
eval	  {	  	  	  	  my	  $cf	  =	  TestCF;	  	  	  	  my	  $key	  =	  bar;	  	  	  	  my	  $columnParent	  =	  Cassandra::...
eval	  {	  	  	  	  my	  $cf	  =	  TestCF;                                                                           thrif...
CPAN?
☹
Net::Cassandra       ✘Net::Cassandra::Easy   ✘  Cassandra::Lite      ✘AnyEvent::Cassandra    ✘
☺
Questions?
Grazie! ☺Stefano Rodighierostefano.rodighiero@gmail.com           @larsen
Writing a cassandra Client with perl
Writing a cassandra Client with perl
Writing a cassandra Client with perl
Writing a cassandra Client with perl
Upcoming SlideShare
Loading in …5
×

Writing a cassandra Client with perl

2,253 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,253
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Writing a cassandra Client with perl

  1. 1. Writing a Cassandra client in Perl
  2. 2. Stefano Rodighierostefano.rodighiero@gmail.com
  3. 3. @larsen
  4. 4. • High Write Throughput• Large Tables• Reports• Analytics
  5. 5. Cassandra
  6. 6. Bigtable: A distributed storage system for structured data, 2006
  7. 7. Dynamo: Amazonshighly available keyvalue store, 2007
  8. 8. Dynamopartitioning and replication
  9. 9. ColumnFamilydata model similar to Bigtables
  10. 10. High availability
  11. 11. Incremental scalability
  12. 12. Eventually consistent
  13. 13. Tunable tradeoffsbetween consistency and latency
  14. 14. Minimal administration
  15. 15. (not quite so)
  16. 16. No SPOFp2p distribution model
  17. 17. Writing
  18. 18. Writinginsert(key,  {column_name  =>  value})☺
  19. 19. Writing☺ R
  20. 20. Writing☺ R
  21. 21. Writing☺ 17% 17% R 17% 17%
  22. 22. Writing☺ R
  23. 23. Writing☺ R
  24. 24. Writing a Cassandra client in Perl
  25. 25. Talking with a server
  26. 26. Cassandra::CassandraClient Thrift::BinaryProtocol Thrift::FramedTransport Thrift::Socket
  27. 27. Cassandra::CassandraClient Thrift::BinaryProtocol Thrift::FramedTransport Thrift::Socket
  28. 28. Cassandra::CassandraClient The Transport layer provides a simple abstraction for Thrift::BinaryProtocol reading/writing from/to the network. This enables Thrift Thrift::FramedTransport to decouple the underlying transport from the rest of the system (serialization/ deserialization, for instance).
  29. 29. Cassandra::CassandraClient The Protocol abstraction defines a mechanism to Thrift::BinaryProtocol map in-memory data structures to a wire-format. In other words, a protocol specifies how datatypes use the underlying Transport to encode/decode themselves.
  30. 30. Cassandra::CassandraClient "High level" interface: login, reading & writing data, …
  31. 31. 1) Install Cassandra2) Install Thrift
  32. 32. thrift  -­‐gen  perl  cassandra.thrift
  33. 33. Login
  34. 34. thrift service  Cassandra  {    #  auth  methods    void  login(1:  required  AuthenticationRequest  auth_request)  i        throws  (1:AuthenticationException  authnx,                          2:AuthorizationException  authzx),    …Perl (generated code) package  Cassandra::CassandraClient; … sub  login{    my  $self  =  shift;    my  $auth_request  =  shift;    $self-­‐>send_login($auth_request);    $self-­‐>recv_login(); }
  35. 35. thrift thrift service  Cassandra  {    #  auth  methods struct  AuthenticationRequest  {    void  login(1:  required  AuthenticationRequest  auth_request)  i        1:  required  map<string,  string>  credentials        throws  (1:AuthenticationException  authnx,   }                        2:AuthorizationException  authzx),    …Perl (generated code)Perl (generated code) package  Cassandra::CassandraClient; … package  Cassandra::AuthenticationRequest; use  base  qw(Class::Accessor); sub  login{ Cassandra::AuthenticationRequest-­‐>mk_accessors(  qw(  credentials  )  );    my  $self  =  shift;    my  $auth_request  =  shift; sub  new  {    …    $self-­‐>send_login($auth_request); }    $self-­‐>recv_login(); }
  36. 36. my  $ks  =  TestIPW;my  $socket        =  Thrift::Socket-­‐>new($server_name,  $server_port);my  $transport  =  Thrift::FramedTransport-­‐>new($socket,  1024,  1024);my  $protocol    =  Thrift::BinaryProtocol-­‐>new($transport);$client              =  Cassandra::CassandraClient-­‐>new(  $protocol  );eval  {        $transport-­‐>open;        my  $auth  =  Cassandra::AuthenticationRequest-­‐>new(            {  credentials  =>  {}  }        );        $client-­‐>login($auth);        $client-­‐>set_keyspace($ks);};if  (  $@  )  {        die  Dumper(  $@  );}
  37. 37. my  $ks  =  TestIPW;my  $socket        =  Thrift::Socket-­‐>new($server_name,  $server_port);my  $transport  =  Thrift::FramedTransport-­‐>new($socket,  1024,  1024);my  $protocol    =  Thrift::BinaryProtocol-­‐>new($transport);$client              =  Cassandra::CassandraClient-­‐>new(  $protocol  );eval  {        $transport-­‐>open;        my  $auth  =  Cassandra::AuthenticationRequest-­‐>new(            {  credentials  =>  {}  }        ); Cassandra::CassandraClient        $client-­‐>login($auth); Thrift::BinaryProtocol Thrift::FramedTransport        $client-­‐>set_keyspace($ks);};if  (  $@  )  { Thrift::Socket        die  Dumper(  $@  );}
  38. 38. thriftmy  $ks  =  TestIPW; service  Cassandra  {my  $socket        =  Thrift::Socket-­‐>new($server_name,  $server_port);    #  auth  methods    void  login(1:  required  AuthenticationRequest  auth_request)  imy  $transport  =  Thrift::FramedTransport-­‐>new($socket,  1024,  1024);        throws  (1:AuthenticationException  authnx,  my  $protocol    =  Thrift::BinaryProtocol-­‐>new($transport);                        2:AuthorizationException  authzx),$client              =  Cassandra::CassandraClient-­‐>new(  $protocol  );    …eval  {        $transport-­‐>open;        my  $auth  =  Cassandra::AuthenticationRequest-­‐>new(            {  credentials  =>  {}  }        );        $client-­‐>login($auth);        $client-­‐>set_keyspace($ks);};if  (  $@  )  {        die  Dumper(  $@  );}
  39. 39. Reading
  40. 40. thrift list<ColumnOrSuperColumn>  get_slice(    1:required  binary  key,    2:required  ColumnParent  column_parent,    3:required  SlicePredicate  predicate,    4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)  throws  (1:InvalidRequestException  ire,                    2:UnavailableException  ue,                    3:TimedOutException  te)Perl (generated code) package  Cassandra::CassandraClient; … sub  get_slice{    my  $self  =  shift;    my  $key  =  shift;    my  $column_parent  =  shift;    my  $predicate  =  shift;    my  $consistency_level  =  shift;    $self-­‐>send_get_slice($key,  $column_parent,  $predicate,  $consistency_level);    return  $self-­‐>recv_get_slice(); }
  41. 41. thrift thrift list<ColumnOrSuperColumn>  get_slice(    1:required  binary  key,    2:required  ColumnParent  column_parent, struct  ColumnParent  {    3:required  SlicePredicate  predicate,        3:  required  string  column_family,    4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)        4:  optional  binary  super_column,  throws  (1:InvalidRequestException  ire,   }                  2:UnavailableException  ue,                    3:TimedOutException  te)Perl (generated code)Perl (generated code) package  Cassandra::CassandraClient; … sub  get_slice{    my  $self  =  shift; package  Cassandra::ColumnParent;    my  $key  =  shift; use  base  qw(Class::Accessor);    my  $column_parent  =  shift; Cassandra::ColumnParent-­‐>mk_accessors(      my  $predicate  =  shift;    qw(  column_family  super_column  )      my  $consistency_level  =  shift; );    $self-­‐>send_get_slice($key,  $column_parent,  $predicate,  $consistency_level);    return  $self-­‐>recv_get_slice(); }
  42. 42. thrift thrift list<ColumnOrSuperColumn>  get_slice(    1:required  binary  key,    2:required  ColumnParent  column_parent, struct  SlicePredicate  {    3:required  SlicePredicate  predicate,        1:  optional  list<binary>  column_names,    4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)        2:  optional  SliceRange      slice_range,  throws  (1:InvalidRequestException  ire,   }                  2:UnavailableException  ue,                    3:TimedOutException  te)Perl (generated code)Perl (generated code) package  Cassandra::CassandraClient; … sub  get_slice{    my  $self  =  shift; package  Cassandra::SlicePredicate;    my  $key  =  shift; use  base  qw(Class::Accessor);    my  $column_parent  =  shift; Cassandra::SlicePredicate-­‐>mk_accessors(      my  $predicate  =  shift;    qw(  column_names  slice_range  )      my  $consistency_level  =  shift; );    $self-­‐>send_get_slice($key,  $column_parent,  $predicate,  $consistency_level);    return  $self-­‐>recv_get_slice(); }
  43. 43. thrift thrift list<ColumnOrSuperColumn>  get_slice(    1:required  binary  key, struct  SliceRange  {    2:required  ColumnParent  column_parent,        1:  required  binary  start,    3:required  SlicePredicate  predicate,        2:  required  binary  finish,    4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)        3:  required  bool  reversed=0,  throws  (1:InvalidRequestException  ire,          4:  required  i32  count=100,                  2:UnavailableException  ue,   }                  3:TimedOutException  te)Perl (generated code)Perl (generated code) package  Cassandra::CassandraClient; … sub  get_slice{    my  $self  =  shift; package  Cassandra::SliceRange;    my  $key  =  shift; use  base  qw(Class::Accessor);    my  $column_parent  =  shift; Cassandra::SliceRange-­‐>mk_accessors(      my  $predicate  =  shift;    qw(  start  finish  reversed  count  )      my  $consistency_level  =  shift; );    $self-­‐>send_get_slice($key,  $column_parent,  $predicate,  $consistency_level);    return  $self-­‐>recv_get_slice(); }
  44. 44. thrift thrift list<ColumnOrSuperColumn>  get_slice( enum  ConsistencyLevel  {        ONE  =  1,    1:required  binary  key,        QUORUM  =  2,    2:required  ColumnParent  column_parent,        LOCAL_QUORUM  =  3,    3:required  SlicePredicate  predicate,        EACH_QUORUM  =  4,    4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)        ALL  =  5,  throws  (1:InvalidRequestException  ire,          ANY  =  6,                  2:UnavailableException  ue,          TWO  =  7,                  3:TimedOutException  te)        THREE  =  8, }Perl (generated code)Perl (generated code) package  Cassandra::CassandraClient; … package  Cassandra::ConsistencyLevel; sub  get_slice{ use  constant  ONE                    =>  1;    my  $self  =  shift; use  constant  QUORUM              =>  2;    my  $key  =  shift; use  constant  LOCAL_QUORUM  =>  3;    my  $column_parent  =  shift; use  constant  EACH_QUORUM    =>  4;    my  $predicate  =  shift; use  constant  ALL                    =>  5;    my  $consistency_level  =  shift; use  constant  ANY                    =>  6;    $self-­‐>send_get_slice($key,  $column_parent,  $predicate,  $consistency_level); use  constant  TWO                    =>  7;    return  $self-­‐>recv_get_slice(); use  constant  THREE                =>  8; }
  45. 45. package  Cassandra::ConsistencyLevel;use  constant  ONE                    =>  1;use  constant  QUORUM              =>  2;use  constant  LOCAL_QUORUM  =>  3;use  constant  EACH_QUORUM    =>  4;use  constant  ALL                    =>  5;use  constant  ANY                    =>  6;use  constant  TWO                    =>  7;use  constant  THREE                =>  8;
  46. 46. package  Cassandra::ConsistencyLevel;use  constant  ONE                    =>  1;use  constant  QUORUM              =>  2;use  constant  LOCAL_QUORUM  =>  3;use  constant  EACH_QUORUM    =>  4;use  constant  ALL                    =>  5;use  constant  ANY                    =>  6;use  constant  TWO                    =>  7;use  constant  THREE                =>  8;
  47. 47. my  $cf  =  20121011.larsen.tweets;my  $columnParent  =  Cassandra::ColumnParent-­‐>new({column_family  =>  $cf});my  $sliceRange  =      Cassandra::SliceRange-­‐>new({          start    =>  ,        finish  =>      });my  $predicate  =      Cassandra::SlicePredicate-­‐>new({        slice_range  =>  $sliceRange    });$res  =  $client-­‐>get_slice(    $key,      $columnParent,      $predicate,      Cassandra::ConsistencyLevel::ONE);
  48. 48. my  $cf  =  20121011.larsen.tweets;my  $columnParent  =  Cassandra::ColumnParent-­‐>new({column_family  =>  $cf});my  $sliceRange  =      Cassandra::SliceRange-­‐>new({          start    =>  ,        finish  =>      });my  $predicate  =      Cassandra::SlicePredicate-­‐>new({        slice_range  =>  $sliceRange    }); thrift$res  =  $client-­‐>get_slice(    $key,      $columnParent,   struct  SliceRange  {    $predicate,          1:  required  binary  start,        2:  required  binary  finish,    Cassandra::ConsistencyLevel::ONE);        3:  required  bool  reversed=0,        4:  required  i32  count=100, }
  49. 49. thrift list<ColumnOrSuperColumn>  get_slice(my  $cf  =  20121011.larsen.tweets;    1:required  binary  key,    2:required  ColumnParent  column_parent,my  $columnParent  =  Cassandra::ColumnParent-­‐>new({column_family  =>  $cf});    3:required  SlicePredicate  predicate,my  $sliceRange  =      4:required  ConsistencyLevel  consistency_level=ConsistencyLevel.ONE)  throws  (1:InvalidRequestException  ire,      Cassandra::SliceRange-­‐>new({                    2:UnavailableException  ue,          start    =>  ,                  3:TimedOutException  te)        finish  =>      });my  $predicate  =      Cassandra::SlicePredicate-­‐>new({        slice_range  =>  $sliceRange    });$res  =  $client-­‐>get_slice(    $key,      $columnParent,      $predicate,      Cassandra::ConsistencyLevel::ONE);
  50. 50. Writing
  51. 51. thrift void insert(1:required binary key,· 2:required ColumnParent column_parent, 3:required Column column, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue, 3:TimedOutException te)Perl (generated code) sub  insert{    my  $self  =  shift;    my  $key  =  shift;    my  $column_parent  =  shift;    my  $column  =  shift;    my  $consistency_level  =  shift;    $self-­‐>send_insert(  $key,  $column_parent,  $column,  $consistency_level);    $self-­‐>recv_insert(); }
  52. 52. thrift thrift struct  Column  { void insert(1:required binary key,·      1:  required  binary  name, column_parent, 2:required ColumnParent 3:required Column column,      2:  optional  binary  value, 4:required ConsistencyLevel consistency_level=ConsistencyLevel.ONE) throws (1:InvalidRequestException ire, 2:UnavailableException ue,      3:  optional  i64  timestamp, 3:TimedOutException te)      4:  optional  i32  ttl, }Perl (generated code)Perl (generated code) sub  insert{    my  $self  =  shift;    my  $key  =  shift;    my  $column_parent  =  shift; package  Cassandra::Column;    my  $column  =  shift; use  base  qw(Class::Accessor);    my  $consistency_level  =  shift; Cassandra::Column-­‐>mk_accessors(  qw(  name  value  timestamp  ttl  )  );    $self-­‐>send_insert(  $key,  $column_parent,  $column,  $consistency_level);    $self-­‐>recv_insert(); }
  53. 53. eval  {        my  $cf  =  TestCF;        my  $key  =  bar;        my  $columnParent  =  Cassandra::ColumnParent-­‐>new({            column_family  =>  $cf        });        my  $ts  =  sprintf  "%.6f",  Time::HiRes::time;        $ts  =~  s/.//;        say  $ts;        my  $column  =  Cassandra::Column-­‐>new({            name            =>  test,              value          =>  a  test  value,              timestamp  =>  $ts        });        $client-­‐>insert(              $key,  $columnParent,  $column,  Cassandra::ConsistencyLevel::ONE);};if  (  $@  )  {        die  Dumper(  $@  );}
  54. 54. eval  {        my  $cf  =  TestCF; thrift        my  $key  =  bar; struct  Column  {        my  $columnParent  =  Cassandra::ColumnParent-­‐>new({      1:  required  binary  name,            column_family  =>  $cf      2:  optional  binary  value,        });      3:  optional  i64  timestamp,      4:  optional  i32  ttl,        my  $ts  =  sprintf  "%.6f",  Time::HiRes::time; }        $ts  =~  s/.//;        say  $ts;        my  $column  =  Cassandra::Column-­‐>new({            name            =>  test,              value          =>  a  test  value,              timestamp  =>  $ts        });        $client-­‐>insert(              $key,  $columnParent,  $column,  Cassandra::ConsistencyLevel::ONE);};if  (  $@  )  {        die  Dumper(  $@  );}
  55. 55. CPAN?
  56. 56.
  57. 57. Net::Cassandra ✘Net::Cassandra::Easy ✘ Cassandra::Lite ✘AnyEvent::Cassandra ✘
  58. 58.
  59. 59. Questions?
  60. 60. Grazie! ☺Stefano Rodighierostefano.rodighiero@gmail.com @larsen

×