Your SlideShare is downloading. ×
0
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Redis And python at pycon_2011
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Redis And python at pycon_2011

17,433

Published on

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

Published in: Technology
1 Comment
43 Likes
Statistics
Notes
No Downloads
Views
Total Views
17,433
On Slideshare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
463
Comments
1
Likes
43
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. PyCon India, 2011 Sunil Arora Redis And Python
  • 2. Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?
  • 3. About Me <ul><li>I tweet at Sunil Arora / @_sunil_ </li></ul><ul><li>I work at ShopSocially </li></ul><ul><li>I blog at http://sunilarora.org </li></ul>
  • 4. Today's talk <ul><li>What is Redis </li></ul>How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis
  • 5. Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com
  • 6. Redis – Brief History <ul><li>Released in March 2009 (Open Source, BSD licensed) </li></ul><ul><li>VMWare hired Salvatore in March, 2010 </li></ul><ul><li>Then Pieter Noordhuis (key contributor) was hired </li></ul>
  • 7. What is Redis ? Its between lot of stuff, so difficult to categorize it precisely
  • 8. What is Redis ? <ul><li>I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well </li></ul><ul><li>-Salvatore Sanfilippo </li></ul>Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
  • 9. Redis is... Remote Data Structure Server
  • 10. Redis is... <ul><li>Supports rich data types of computer science </li></ul><ul><ul><li>- Strings, Lists, Sets, Sorted Sets, Hashes... </li></ul></ul><ul><li>Rich sets of primitives (commands) to manipulate these types </li></ul><ul><li>Predictive complexity measurements </li></ul>
  • 11. A few fundamentals <ul><li>Written in C (no external dependency) </li></ul><ul><li>Uses memory as main storage </li></ul><ul><li>Uses disk for persistence </li></ul><ul><li>Single Threaded </li></ul><ul><li>Every command performs atomic execution </li></ul>
  • 12. Performance <ul><li>Screamingly fast performance </li></ul><ul><li>~50K read/write operations per seconds </li></ul><ul><li>100K+ read/write ops per second on a regular EC2 instance </li></ul>
  • 13. Installation <ul><li>$ git clone http://github.com/antirez/redis </li></ul><ul><li>$ cd redis </li></ul><ul><li>$ make </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>.......... </li></ul><ul><li>.......... </li></ul><ul><li>$./src/redis-cli </li></ul><ul><li>Redis> PING </li></ul><ul><li>PONG </li></ul>
  • 14. Python Libraries <ul><ul><li>Redis-py - Python client library </li></ul></ul><ul><ul><li>Txredisapi - An asynchronous Python client for the Redis database, based on Twisted. </li></ul></ul><ul><ul><li>Redisco - ORM for redis along the lines Ohm for Ruby </li></ul></ul>
  • 15. redis-py <ul><li>The most popular python client library </li></ul><ul><li>Andy McCurdy ( [email_address] ) </li></ul><ul><li>Github: http://github.com/andymccurdy/redis-py </li></ul><ul><ul><li>$easy_install redis OR </li></ul></ul><ul><ul><li>$pip install redis </li></ul></ul><ul><li>Optional </li></ul><ul><ul><li>$easy_install hiredis </li></ul></ul><ul><ul><li>$pip install hiredis </li></ul></ul>
  • 16. Lets get started... <ul><li>$ cd redis </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>......... </li></ul><ul><li>>>> from redis import Redis </li></ul><ul><li>>>> redis_client = Redis() </li></ul><ul><li>>>> redis_client.keys() </li></ul><ul><li>>>> help(redis_client) </li></ul>
  • 17. Redis Keys <ul><li>Not binary safe. </li></ul><ul><li>Should not contain space or newline character </li></ul><ul><li>A few rules about keys: </li></ul><ul><ul><li>Too long keys are not a good idea </li></ul></ul><ul><ul><li>Too short keys is also not a good idea </li></ul></ul><ul><ul><li>“ object-type:id:field” can be a nice idea, i.e. “user:1001:name” </li></ul></ul>
  • 18. Operations on Keys <ul><li>KEYS </li></ul><ul><li>EXISTS </li></ul><ul><li>DEL </li></ul><ul><li>EXPIRE </li></ul><ul><li>OBJECT </li></ul><ul><li>PERSIST </li></ul><ul><li>RANDOMKEY </li></ul><ul><li>RENAME </li></ul><ul><li>TYPE </li></ul><ul><li>TTL </li></ul><ul><li>EXPIREAT </li></ul><ul><li>MOVE </li></ul>
  • 19. Lets play with keys <ul><li>>>>redis_client.keys() </li></ul><ul><li>>>>redis_client.exists('key') </li></ul><ul><li>>>>redis_client.delete('key') </li></ul><ul><li>>>>redis_client.type('key') </li></ul><ul><li>>>>...... </li></ul><ul><li>>>>...... </li></ul>
  • 20. Data Structures <ul><li>Strings </li></ul><ul><li>Lists </li></ul><ul><li>Sets </li></ul><ul><li>Sorted Sets </li></ul><ul><li>Hashes </li></ul>
  • 21. Strings <ul><li>SET </li></ul><ul><li>GET </li></ul><ul><li>MSET </li></ul><ul><li>MGET </li></ul><ul><li>SETEX </li></ul><ul><li>GETSET </li></ul><ul><li>SETNX </li></ul><ul><li>INCR </li></ul><ul><li>INCRBY </li></ul><ul><li>DECR </li></ul><ul><li>DECRBY </li></ul>
  • 22. Strings – with redis client <ul><li>>>> redis_client.set('key', 'value') </li></ul><ul><li>>>> redis_client.get('key') </li></ul><ul><li>>>> redis_client.delete('key') </li></ul>
  • 23. Fetch multiple keys at once <ul><li>mget/mset </li></ul><ul><li>redis_client.mset({'key1': 'val1', 'key2': 'val2'}) </li></ul><ul><li>redis_client.mget('key1', 'key2',......) </li></ul>
  • 24. Expiration <ul><li>Set a value with expire </li></ul><ul><li>>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs </li></ul><ul><li>>>>redis_client.expire('key', 2) </li></ul><ul><li>>>>redis_client.get('key') </li></ul><ul><li>>>>None </li></ul>
  • 25. Expire Semantics <ul><li>Key with expiry known as volatile keys </li></ul><ul><li>Whenever a volatile key is modified, its expiry is reset </li></ul><ul><ul><li>- To avoid inconsistency in following cases </li></ul></ul><ul><ul><ul><li>Replication </li></ul></ul></ul><ul><ul><ul><li>Append Only Log </li></ul></ul></ul>
  • 26. Uses To store transient states in your web application
  • 27. Uses Who is online?
  • 28. Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
  • 29. Atomic Increments <ul><li>>>>help(redis_client.incr) </li></ul><ul><li>>>>help(redis_client.decr) </li></ul><ul><li>>>> </li></ul><ul><li>>>> redis_client.incr('counter', 1) </li></ul><ul><li>>>>1 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>2 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>3 </li></ul>
  • 30. Uses High Speed counters (views/clicks/votes/likes..)
  • 31. Uses API Rate Limiting
  • 32. Uses Generating unique IDs
  • 33. Lists <ul><li>Ordered list of binarysafe strings </li></ul><ul><li>Doubly linked list </li></ul><ul><li>Memory footprint optimized for smaller list </li></ul><ul><li>O(1) insertion/deletion at both ends </li></ul>
  • 34. Lists - operations <ul><li>LPUSH </li></ul><ul><li>RPUSH </li></ul><ul><li>LSET </li></ul><ul><li>LRANGE </li></ul><ul><li>LPOP </li></ul><ul><li>BLPOP </li></ul><ul><li>BRPOP </li></ul><ul><li>BRPOPLPUSH </li></ul><ul><li>LINSERT </li></ul><ul><li>RPOP </li></ul><ul><li>RPOPLPUSH </li></ul><ul><li>LPUSHX </li></ul><ul><li>RPUSHX </li></ul>
  • 35. Uses Web apps are full of lists :)
  • 36. Uses Capped List
  • 37. Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque
  • 38. Uses Social Activity Streams or notifications
  • 39. Sets An unordered collection of distinct byte strings Nothing different from the data type in python
  • 40. Sets - Operations <ul><li>SADD </li></ul><ul><li>SCARD </li></ul><ul><li>SREM </li></ul><ul><li>SISMEMBER </li></ul><ul><li>SMEMBERS </li></ul><ul><li>SPOP </li></ul><ul><li>SRANDMEMBER </li></ul>
  • 41. Operations between Sets <ul><li>SMOVE </li></ul><ul><li>SUNION </li></ul><ul><li>SDIFF </li></ul><ul><li>SINTER </li></ul>
  • 42. Sets - Operations <ul><li>SDIFFSTORE </li></ul><ul><li>SINTERSTORE </li></ul><ul><li>SUNIONSTORE </li></ul>
  • 43. Sets - Uses <ul><li>Picking random items from a set using SRANDMEMBER </li></ul><ul><li>Ex. </li></ul><ul><ul><li>Picking a random article from daily news </li></ul></ul><ul><ul><li>Pick a random Ad for serving </li></ul></ul><ul><ul><li>Pick a random option for A/B </li></ul></ul>Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
  • 44. Sets - Uses <ul><li>To model Relations in social graph </li></ul><ul><li>Ex. </li></ul><ul><ul><li>>>>redis_client.sadd('friends:john', 'jenny', 'maria') </li></ul></ul><ul><ul><li>>>>redis_client.sadd('friends:ben', 'maria', 'kate') </li></ul></ul><ul><ul><li>>>>redis_client.sinter('friends:john', 'friends:ben') </li></ul></ul>
  • 45. Relations (friend/followers) <ul><li>Common followlist for A and B </li></ul><ul><ul><li>>>> sinter('users:A:follows', 'users:B:follows') </li></ul></ul><ul><li>Unique to B compared to C </li></ul><ul><ul><li>>>>sdiff('users:B:follows', 'users:C:follows') </li></ul></ul><ul><li>Mutual relationship (friend as well as follower) </li></ul><ul><ul><li>>>>sinter('users:B:followers', 'users:B:friends') </li></ul></ul><ul><li>Who does not follow me back ? </li></ul><ul><ul><li>>>>sdiff('users:B:friends', 'users:B:followers') </li></ul></ul><ul><li>Who am I not following back? </li></ul><ul><ul><li>>>>sdiff('users:B:followers', 'user:B:friends') </li></ul></ul>
  • 46. Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends
  • 47. Which of my friends are online right now? <ul><li>>>>RENAME online:fresh online:stale #every minute </li></ul><ul><li>>>>SUNIONSTORE online:fresh online:stale </li></ul><ul><li>>>>#friend jack connects </li></ul><ul><li>>>>SADD online:fresh 'jack' </li></ul><ul><li>>>>SUNIONSTORE online online:fresh online:stale </li></ul><ul><li>>>>#who is online ? </li></ul><ul><li>>>>SINTER online friends </li></ul>
  • 48. Sorted Sets Ordered sets on the basis of score
  • 49. Sorted Sets <ul><li>ZADD </li></ul><ul><li>ZCARD </li></ul><ul><li>ZCOUNT </li></ul><ul><li>ZINCRBY </li></ul><ul><li>ZINTERSTORE </li></ul><ul><li>ZRANGE </li></ul><ul><li>ZRANGEBYSCORE </li></ul><ul><li>ZRANK </li></ul><ul><li>ZREM </li></ul><ul><li>ZREMRANGEBYRANK </li></ul><ul><li>ZREMRANGEBYSCORE </li></ul><ul><li>ZREVRANGE </li></ul><ul><li>ZREVRANGEBYSCORE </li></ul><ul><li>ZSCORE </li></ul><ul><li>ZUNIONSTORE </li></ul>
  • 50. Sorted Sets – Use Case To build index on your dataset
  • 51. Uses - Realtime Leaderboards <ul><li>>>>zincrby('leaderboard', 'john', 2) </li></ul><ul><li>>>>zincrby('leaderboard', 'jack', 5) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 1) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 10) </li></ul><ul><li>.... </li></ul><ul><li>>>>zrange('leaderboard', 0, -1, withscores = True) </li></ul>
  • 52. Uses - Realtime Leaderboards <ul><li>Most download resources </li></ul><ul><li>Most popular articles on the website </li></ul><ul><li>Weekly popular list </li></ul><ul><li>Most downloaded stuff between date1 and date2 </li></ul><ul><li>..... </li></ul><ul><li>..... </li></ul>
  • 53. AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html
  • 54. Hashes <ul><li>Equivalent to Python dictionary or Ruby hash or Java hashmap </li></ul><ul><li>Operations </li></ul><ul><ul><li>hmset users:1 {'username': 'jim', 'score': 23} </li></ul></ul><ul><ul><li>hgetall users:1 </li></ul></ul><ul><ul><li>Hincrby </li></ul></ul><ul><li>Useful for storing structured data </li></ul><ul><ul><li>hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'} </li></ul></ul><ul><ul><li>Expire user:1:preferences <duration> </li></ul></ul><ul><ul><li>Hmgetall user:1:preferences #returns entire dict </li></ul></ul>
  • 55. Redis Transactions <ul><li>Using Multi/Watch </li></ul><ul><li>>>>p = redis_client.pipeline() </li></ul><ul><li>>>>p.lpush('a', 1) </li></ul><ul><li>>>>p.ltrim('a', 0, 100) </li></ul><ul><li>>>>p.execute </li></ul><ul><li>Limitation </li></ul><ul><ul><li>Since commands are queued, Can't read values </li></ul></ul><ul><ul><li>No conditional execution </li></ul></ul><ul><ul><li>If not high write contention scenario, WATCH can be used </li></ul></ul><ul><li>Use Redis Scripting to write your own primitive </li></ul>
  • 56. Designing with Redis <ul><li>Data layout design on basis of Query </li></ul><ul><li>No Query Optimizer </li></ul><ul><li>Manually build indexes </li></ul>
  • 57. Durability <ul><li>Snapshotting mode </li></ul><ul><ul><li>Binary dump every x secs or y ops </li></ul></ul><ul><li>Append Only File (AOF) </li></ul><ul><ul><li>Every command is written to a file </li></ul></ul><ul><ul><li>On restart/crash, commands replayed </li></ul></ul><ul><ul><li>Fsync on every new command </li></ul></ul><ul><ul><li>Fsync every second </li></ul></ul><ul><ul><li>OS decide to </li></ul></ul><ul><li>Replication </li></ul>
  • 58. Publish/Subscribe <ul><li>A simple and efficient implementation of publish/subscribe messaging paradigm </li></ul><ul><li>Client can subscribe/psubscribe to receive messages on channels (key patterns) </li></ul>
  • 59. Publish/Subscribe <ul><li>PSUBSCRIBE </li></ul><ul><li>PUBLISH </li></ul><ul><li>PUNSUBSCRIBE </li></ul><ul><li>SUBSCRIBE </li></ul><ul><li>UNSUBSCRIBE </li></ul>
  • 60. Uses Many to Many message passing
  • 61. Uses Web Chat
  • 62. References <ul><li>Redis Website ( http://redis.io ) </li></ul><ul><li>SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ ) </li></ul><ul><li>Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 ) </li></ul><ul><li>Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html ) </li></ul><ul><li>Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications) </li></ul>
  • 63. References <ul><li>Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/) </li></ul><ul><li>Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/ </li></ul><ul><li>Redis at Bump </li></ul><ul><li>http://devblog.bu.mp/how-we-use-redis-at-bump </li></ul><ul><li>Redis: AK 47 of postrelational database </li></ul><ul><li>http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases </li></ul>
  • 64. Questions
  • 65. Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com

×