Successfully reported this slideshow.
Your SlideShare is downloading. ×

Redis And python at pycon_2011

Loading in …3

Check these out next

1 of 65 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (20)


Similar to Redis And python at pycon_2011 (20)

Recently uploaded (20)


Redis And python at pycon_2011

  1. 1. PyCon India, 2011 Sunil Arora Redis And Python
  2. 2. Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?
  3. 3. About Me <ul><li>I tweet at Sunil Arora / @_sunil_ </li></ul><ul><li>I work at ShopSocially </li></ul><ul><li>I blog at </li></ul>
  4. 4. Today's talk <ul><li>What is Redis </li></ul>How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis
  5. 5. Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez
  6. 6. Redis – Brief History <ul><li>Released in March 2009 (Open Source, BSD licensed) </li></ul><ul><li>VMWare hired Salvatore in March, 2010 </li></ul><ul><li>Then Pieter Noordhuis (key contributor) was hired </li></ul>
  7. 7. What is Redis ? Its between lot of stuff, so difficult to categorize it precisely
  8. 8. What is Redis ? <ul><li>I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well </li></ul><ul><li>-Salvatore Sanfilippo </li></ul>Picture by herzogbr
  9. 9. Redis is... Remote Data Structure Server
  10. 10. Redis is... <ul><li>Supports rich data types of computer science </li></ul><ul><ul><li>- Strings, Lists, Sets, Sorted Sets, Hashes... </li></ul></ul><ul><li>Rich sets of primitives (commands) to manipulate these types </li></ul><ul><li>Predictive complexity measurements </li></ul>
  11. 11. A few fundamentals <ul><li>Written in C (no external dependency) </li></ul><ul><li>Uses memory as main storage </li></ul><ul><li>Uses disk for persistence </li></ul><ul><li>Single Threaded </li></ul><ul><li>Every command performs atomic execution </li></ul>
  12. 12. Performance <ul><li>Screamingly fast performance </li></ul><ul><li>~50K read/write operations per seconds </li></ul><ul><li>100K+ read/write ops per second on a regular EC2 instance </li></ul>
  13. 13. Installation <ul><li>$ git clone </li></ul><ul><li>$ cd redis </li></ul><ul><li>$ make </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>.......... </li></ul><ul><li>.......... </li></ul><ul><li>$./src/redis-cli </li></ul><ul><li>Redis> PING </li></ul><ul><li>PONG </li></ul>
  14. 14. Python Libraries <ul><ul><li>Redis-py - Python client library </li></ul></ul><ul><ul><li>Txredisapi - An asynchronous Python client for the Redis database, based on Twisted. </li></ul></ul><ul><ul><li>Redisco - ORM for redis along the lines Ohm for Ruby </li></ul></ul>
  15. 15. redis-py <ul><li>The most popular python client library </li></ul><ul><li>Andy McCurdy ( [email_address] ) </li></ul><ul><li>Github: </li></ul><ul><ul><li>$easy_install redis OR </li></ul></ul><ul><ul><li>$pip install redis </li></ul></ul><ul><li>Optional </li></ul><ul><ul><li>$easy_install hiredis </li></ul></ul><ul><ul><li>$pip install hiredis </li></ul></ul>
  16. 16. Lets get started... <ul><li>$ cd redis </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>......... </li></ul><ul><li>>>> from redis import Redis </li></ul><ul><li>>>> redis_client = Redis() </li></ul><ul><li>>>> redis_client.keys() </li></ul><ul><li>>>> help(redis_client) </li></ul>
  17. 17. Redis Keys <ul><li>Not binary safe. </li></ul><ul><li>Should not contain space or newline character </li></ul><ul><li>A few rules about keys: </li></ul><ul><ul><li>Too long keys are not a good idea </li></ul></ul><ul><ul><li>Too short keys is also not a good idea </li></ul></ul><ul><ul><li>“ object-type:id:field” can be a nice idea, i.e. “user:1001:name” </li></ul></ul>
  18. 18. Operations on Keys <ul><li>KEYS </li></ul><ul><li>EXISTS </li></ul><ul><li>DEL </li></ul><ul><li>EXPIRE </li></ul><ul><li>OBJECT </li></ul><ul><li>PERSIST </li></ul><ul><li>RANDOMKEY </li></ul><ul><li>RENAME </li></ul><ul><li>TYPE </li></ul><ul><li>TTL </li></ul><ul><li>EXPIREAT </li></ul><ul><li>MOVE </li></ul>
  19. 19. Lets play with keys <ul><li>>>>redis_client.keys() </li></ul><ul><li>>>>redis_client.exists('key') </li></ul><ul><li>>>>redis_client.delete('key') </li></ul><ul><li>>>>redis_client.type('key') </li></ul><ul><li>>>>...... </li></ul><ul><li>>>>...... </li></ul>
  20. 20. Data Structures <ul><li>Strings </li></ul><ul><li>Lists </li></ul><ul><li>Sets </li></ul><ul><li>Sorted Sets </li></ul><ul><li>Hashes </li></ul>
  21. 21. Strings <ul><li>SET </li></ul><ul><li>GET </li></ul><ul><li>MSET </li></ul><ul><li>MGET </li></ul><ul><li>SETEX </li></ul><ul><li>GETSET </li></ul><ul><li>SETNX </li></ul><ul><li>INCR </li></ul><ul><li>INCRBY </li></ul><ul><li>DECR </li></ul><ul><li>DECRBY </li></ul>
  22. 22. Strings – with redis client <ul><li>>>> redis_client.set('key', 'value') </li></ul><ul><li>>>> redis_client.get('key') </li></ul><ul><li>>>> redis_client.delete('key') </li></ul>
  23. 23. Fetch multiple keys at once <ul><li>mget/mset </li></ul><ul><li>redis_client.mset({'key1': 'val1', 'key2': 'val2'}) </li></ul><ul><li>redis_client.mget('key1', 'key2',......) </li></ul>
  24. 24. Expiration <ul><li>Set a value with expire </li></ul><ul><li>>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs </li></ul><ul><li>>>>redis_client.expire('key', 2) </li></ul><ul><li>>>>redis_client.get('key') </li></ul><ul><li>>>>None </li></ul>
  25. 25. Expire Semantics <ul><li>Key with expiry known as volatile keys </li></ul><ul><li>Whenever a volatile key is modified, its expiry is reset </li></ul><ul><ul><li>- To avoid inconsistency in following cases </li></ul></ul><ul><ul><ul><li>Replication </li></ul></ul></ul><ul><ul><ul><li>Append Only Log </li></ul></ul></ul>
  26. 26. Uses To store transient states in your web application
  27. 27. Uses Who is online?
  28. 28. Uses Redis as LRU cache (
  29. 29. Atomic Increments <ul><li>>>>help(redis_client.incr) </li></ul><ul><li>>>>help(redis_client.decr) </li></ul><ul><li>>>> </li></ul><ul><li>>>> redis_client.incr('counter', 1) </li></ul><ul><li>>>>1 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>2 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>3 </li></ul>
  30. 30. Uses High Speed counters (views/clicks/votes/likes..)
  31. 31. Uses API Rate Limiting
  32. 32. Uses Generating unique IDs
  33. 33. Lists <ul><li>Ordered list of binarysafe strings </li></ul><ul><li>Doubly linked list </li></ul><ul><li>Memory footprint optimized for smaller list </li></ul><ul><li>O(1) insertion/deletion at both ends </li></ul>
  34. 34. Lists - operations <ul><li>LPUSH </li></ul><ul><li>RPUSH </li></ul><ul><li>LSET </li></ul><ul><li>LRANGE </li></ul><ul><li>LPOP </li></ul><ul><li>BLPOP </li></ul><ul><li>BRPOP </li></ul><ul><li>BRPOPLPUSH </li></ul><ul><li>LINSERT </li></ul><ul><li>RPOP </li></ul><ul><li>RPOPLPUSH </li></ul><ul><li>LPUSHX </li></ul><ul><li>RPUSHX </li></ul>
  35. 35. Uses Web apps are full of lists :)
  36. 36. Uses Capped List
  37. 37. Uses Real time message Queue Background Worker queues (Resque, Celery)
  38. 38. Uses Social Activity Streams or notifications
  39. 39. Sets An unordered collection of distinct byte strings Nothing different from the data type in python
  40. 40. Sets - Operations <ul><li>SADD </li></ul><ul><li>SCARD </li></ul><ul><li>SREM </li></ul><ul><li>SISMEMBER </li></ul><ul><li>SMEMBERS </li></ul><ul><li>SPOP </li></ul><ul><li>SRANDMEMBER </li></ul>
  41. 41. Operations between Sets <ul><li>SMOVE </li></ul><ul><li>SUNION </li></ul><ul><li>SDIFF </li></ul><ul><li>SINTER </li></ul>
  42. 42. Sets - Operations <ul><li>SDIFFSTORE </li></ul><ul><li>SINTERSTORE </li></ul><ul><li>SUNIONSTORE </li></ul>
  43. 43. Sets - Uses <ul><li>Picking random items from a set using SRANDMEMBER </li></ul><ul><li>Ex. </li></ul><ul><ul><li>Picking a random article from daily news </li></ul></ul><ul><ul><li>Pick a random Ad for serving </li></ul></ul><ul><ul><li>Pick a random option for A/B </li></ul></ul>Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
  44. 44. Sets - Uses <ul><li>To model Relations in social graph </li></ul><ul><li>Ex. </li></ul><ul><ul><li>>>>redis_client.sadd('friends:john', 'jenny', 'maria') </li></ul></ul><ul><ul><li>>>>redis_client.sadd('friends:ben', 'maria', 'kate') </li></ul></ul><ul><ul><li>>>>redis_client.sinter('friends:john', 'friends:ben') </li></ul></ul>
  45. 45. Relations (friend/followers) <ul><li>Common followlist for A and B </li></ul><ul><ul><li>>>> sinter('users:A:follows', 'users:B:follows') </li></ul></ul><ul><li>Unique to B compared to C </li></ul><ul><ul><li>>>>sdiff('users:B:follows', 'users:C:follows') </li></ul></ul><ul><li>Mutual relationship (friend as well as follower) </li></ul><ul><ul><li>>>>sinter('users:B:followers', 'users:B:friends') </li></ul></ul><ul><li>Who does not follow me back ? </li></ul><ul><ul><li>>>>sdiff('users:B:friends', 'users:B:followers') </li></ul></ul><ul><li>Who am I not following back? </li></ul><ul><ul><li>>>>sdiff('users:B:followers', 'user:B:friends') </li></ul></ul>
  46. 46. Which of my friend's are online right now? SINTER online_people my_friends
  47. 47. Which of my friends are online right now? <ul><li>>>>RENAME online:fresh online:stale #every minute </li></ul><ul><li>>>>SUNIONSTORE online:fresh online:stale </li></ul><ul><li>>>>#friend jack connects </li></ul><ul><li>>>>SADD online:fresh 'jack' </li></ul><ul><li>>>>SUNIONSTORE online online:fresh online:stale </li></ul><ul><li>>>>#who is online ? </li></ul><ul><li>>>>SINTER online friends </li></ul>
  48. 48. Sorted Sets Ordered sets on the basis of score
  49. 49. Sorted Sets <ul><li>ZADD </li></ul><ul><li>ZCARD </li></ul><ul><li>ZCOUNT </li></ul><ul><li>ZINCRBY </li></ul><ul><li>ZINTERSTORE </li></ul><ul><li>ZRANGE </li></ul><ul><li>ZRANGEBYSCORE </li></ul><ul><li>ZRANK </li></ul><ul><li>ZREM </li></ul><ul><li>ZREMRANGEBYRANK </li></ul><ul><li>ZREMRANGEBYSCORE </li></ul><ul><li>ZREVRANGE </li></ul><ul><li>ZREVRANGEBYSCORE </li></ul><ul><li>ZSCORE </li></ul><ul><li>ZUNIONSTORE </li></ul>
  50. 50. Sorted Sets – Use Case To build index on your dataset
  51. 51. Uses - Realtime Leaderboards <ul><li>>>>zincrby('leaderboard', 'john', 2) </li></ul><ul><li>>>>zincrby('leaderboard', 'jack', 5) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 1) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 10) </li></ul><ul><li>.... </li></ul><ul><li>>>>zrange('leaderboard', 0, -1, withscores = True) </li></ul>
  52. 52. Uses - Realtime Leaderboards <ul><li>Most download resources </li></ul><ul><li>Most popular articles on the website </li></ul><ul><li>Weekly popular list </li></ul><ul><li>Most downloaded stuff between date1 and date2 </li></ul><ul><li>..... </li></ul><ul><li>..... </li></ul>
  53. 53. AutoComplete with Sored Sets
  54. 54. Hashes <ul><li>Equivalent to Python dictionary or Ruby hash or Java hashmap </li></ul><ul><li>Operations </li></ul><ul><ul><li>hmset users:1 {'username': 'jim', 'score': 23} </li></ul></ul><ul><ul><li>hgetall users:1 </li></ul></ul><ul><ul><li>Hincrby </li></ul></ul><ul><li>Useful for storing structured data </li></ul><ul><ul><li>hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'} </li></ul></ul><ul><ul><li>Expire user:1:preferences <duration> </li></ul></ul><ul><ul><li>Hmgetall user:1:preferences #returns entire dict </li></ul></ul>
  55. 55. Redis Transactions <ul><li>Using Multi/Watch </li></ul><ul><li>>>>p = redis_client.pipeline() </li></ul><ul><li>>>>p.lpush('a', 1) </li></ul><ul><li>>>>p.ltrim('a', 0, 100) </li></ul><ul><li>>>>p.execute </li></ul><ul><li>Limitation </li></ul><ul><ul><li>Since commands are queued, Can't read values </li></ul></ul><ul><ul><li>No conditional execution </li></ul></ul><ul><ul><li>If not high write contention scenario, WATCH can be used </li></ul></ul><ul><li>Use Redis Scripting to write your own primitive </li></ul>
  56. 56. Designing with Redis <ul><li>Data layout design on basis of Query </li></ul><ul><li>No Query Optimizer </li></ul><ul><li>Manually build indexes </li></ul>
  57. 57. Durability <ul><li>Snapshotting mode </li></ul><ul><ul><li>Binary dump every x secs or y ops </li></ul></ul><ul><li>Append Only File (AOF) </li></ul><ul><ul><li>Every command is written to a file </li></ul></ul><ul><ul><li>On restart/crash, commands replayed </li></ul></ul><ul><ul><li>Fsync on every new command </li></ul></ul><ul><ul><li>Fsync every second </li></ul></ul><ul><ul><li>OS decide to </li></ul></ul><ul><li>Replication </li></ul>
  58. 58. Publish/Subscribe <ul><li>A simple and efficient implementation of publish/subscribe messaging paradigm </li></ul><ul><li>Client can subscribe/psubscribe to receive messages on channels (key patterns) </li></ul>
  59. 59. Publish/Subscribe <ul><li>PSUBSCRIBE </li></ul><ul><li>PUBLISH </li></ul><ul><li>PUNSUBSCRIBE </li></ul><ul><li>SUBSCRIBE </li></ul><ul><li>UNSUBSCRIBE </li></ul>
  60. 60. Uses Many to Many message passing
  61. 61. Uses Web Chat
  62. 62. References <ul><li>Redis Website ( ) </li></ul><ul><li>SimonWillison Tutorial ( ) </li></ul><ul><li>Redis from ground up ( ) </li></ul><ul><li>Redis under the hood ( ) </li></ul><ul><li>Tumbler and Redis ( </li></ul>
  63. 63. References <ul><li>Redis at disqus ( </li></ul><ul><li>Redis at Craiglist </li></ul><ul><li>Redis at Bump </li></ul><ul><li> </li></ul><ul><li>Redis: AK 47 of postrelational database </li></ul><ul><li> </li></ul>
  64. 64. Questions
  65. 65. Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT