Key-Value Storage Systems …  and beyond … with Python
Who the hell are you? + + = Ian Lewis Company: BeProud Tags: #python #django #redbull #mercurial Twitter: IanMLewis HP: ht...
Pro: Fast Simple Con: Can't easily store complex relational data Can't do complex queries Why?
What?
Memcached <ul><li>Made by this guy  </li></ul><ul><li>Not Persistent </li></ul><ul><li>It's a cache so values go away </li...
python-memcached >>>  import  memcache  >>> client = memcache.Client([ &quot;127.0.0.1:11211&quot; ])  >>> client. set ( &...
Tokyo-Cabinet/Tyrant <ul><li>Made by Mikio Hirabayashi for Mixi </li></ul><ul><li>Persistent Key-Value Store </li></ul><ul...
python-tokyotyrant/pytyrant >>>  import  pytyrant  >>> t = pytyrant.PyTyrant. open ( '127.0.0.1' ,  1978 )  >>> t[ '__test...
B+ Trees
pytc >>>  import  pytc >>> db = pytc.BDB( 'bdb.db' , pytc.BDBOWRITER | pytc.BDBOCREAT) >>> db[ 'niku' ] =  'umai' >>> db[ ...
= ?
redis <ul><li>Re -mote  Di -ctionary  S -erver </li></ul><ul><li>Cool new KVS </li></ul><ul><li>Data structures: Lists, Se...
So I can't eat it?
>>>  from  redis  import  Redis, ConnectionError, ResponseError >>> r = Redis(db= 9 ) >>> r[ 'a' ] =  24 . 0 >>> r[ 'a' ] ...
<ul><li>Made at Facebook for their Inbox search </li></ul><ul><li>Distributed Key-Value Store </li></ul><ul><li>Influenced...
Cassandra
pycassa import  pycassa CLIENT = pycassa.connect_thread_local(framed_transport= True ) USER = pycassa.ColumnFamily(CLIENT,...
Questions!
Upcoming SlideShare
Loading in …5
×

Key Value Storage Systems ... and Beyond ... with Python

3,586 views

Published on

Presentation for PyCon Asia Pacific 2010

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
  • Hi Ian,

    Many thanks for the excellent presentation.

    I learned about some new libraries from it and will try them out.

    Best wishes,

    Atish Chandra Lahiri
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
3,586
On SlideShare
0
From Embeds
0
Number of Embeds
203
Actions
Shares
0
Downloads
26
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Key Value Storage Systems ... and Beyond ... with Python

  1. 1. Key-Value Storage Systems … and beyond … with Python
  2. 2. Who the hell are you? + + = Ian Lewis Company: BeProud Tags: #python #django #redbull #mercurial Twitter: IanMLewis HP: http://www.ianlewis.org/
  3. 3. Pro: Fast Simple Con: Can't easily store complex relational data Can't do complex queries Why?
  4. 4. What?
  5. 5. Memcached <ul><li>Made by this guy </li></ul><ul><li>Not Persistent </li></ul><ul><li>It's a cache so values go away </li></ul><ul><li>Not useful as a database </li></ul><ul><li>If you make websites and aren't using it you should be </li></ul>
  6. 6. python-memcached >>> import memcache >>> client = memcache.Client([ &quot;127.0.0.1:11211&quot; ]) >>> client. set ( &quot;test&quot; , 1 ) True >>> client. add ( &quot;test&quot; , 1 ) False >>> client. add ( &quot;test2&quot; , 1 ) True >>> client. set ( &quot;test&quot; , 2 ) True >>> client. incr ( &quot;test&quot; ) 3 >>>
  7. 7. Tokyo-Cabinet/Tyrant <ul><li>Made by Mikio Hirabayashi for Mixi </li></ul><ul><li>Persistent Key-Value Store </li></ul><ul><li>Mmapped file </li></ul><ul><li>Supports hash-table, b+trees, fixed-length array </li></ul><ul><li>Replication </li></ul>
  8. 8. python-tokyotyrant/pytyrant >>> import pytyrant >>> t = pytyrant.PyTyrant. open ( '127.0.0.1' , 1978 ) >>> t[ '__test_key__' ] = 'foo' >>> t. concat ( '__test_key__' , 'bar' ) >>> print t[ '__test_key__' ] foobar >>> del t[ '__test_key__' ] PY-
  9. 9. B+ Trees
  10. 10. pytc >>> import pytc >>> db = pytc.BDB( 'bdb.db' , pytc.BDBOWRITER | pytc.BDBOCREAT) >>> db[ 'niku' ] = 'umai' >>> db[ 'niku' ] 'umai' >>> db[ 'ra-men' ] = 'kuitai' >>> db[ 'ra-men' ] 'kuitai' >>> for key in db: >>> print 'key:' , key, ' value:' , db[key] 'key: niki value: umai' 'key: ra-men value: kuitai'
  11. 11. = ?
  12. 12. redis <ul><li>Re -mote Di -ctionary S -erver </li></ul><ul><li>Cool new KVS </li></ul><ul><li>Data structures: Lists, Sets, Sorted Sets </li></ul><ul><li>Queries: unions, intersection, complement (diff) </li></ul><ul><li>In Memory (Persisted in background) </li></ul><ul><li>Replication </li></ul>
  13. 13. So I can't eat it?
  14. 14. >>> from redis import Redis, ConnectionError, ResponseError >>> r = Redis(db= 9 ) >>> r[ 'a' ] = 24 . 0 >>> r[ 'a' ] Decimal( &quot;24.0&quot; ) >>> r = Redis(db= 9 , float_fn= float ) >>> r[ 'a' ] 24 . 0 >>> del r[ 'a' ] >>> print r.get( 'a' ) # r['a'] will raise KeyError None redis-py
  15. 15. <ul><li>Made at Facebook for their Inbox search </li></ul><ul><li>Distributed Key-Value Store </li></ul><ul><li>Influenced by Apache Dynamo </li></ul><ul><li>Values are columnar and stored together in column families and super columns </li></ul><ul><li>Eventually consistent </li></ul><ul><li>Consistency is user definable (Read servers, Write servers, Quorum, Full) </li></ul>Cassandra
  16. 16. Cassandra
  17. 17. pycassa import pycassa CLIENT = pycassa.connect_thread_local(framed_transport= True ) USER = pycassa.ColumnFamily(CLIENT, 'Twissandra' , 'User' , dict_class=OrderedDict) ... TWEET = pycassa.ColumnFamily(CLIENT, 'Twissandra' , 'Tweet' , dict_class=OrderedDict) TIMELINE = pycassa.ColumnFamily(CLIENT, 'Twissandra' , 'Timeline' , dict_class=OrderedDict) ... timeline = TIMELINE.get( str (user_id), column_start=start, column_count=limit, column_reversed= True ) tweets = TWEET.multiget(timeline.values())
  18. 18. Questions!

×