1.
PyCon India, 2011 Sunil Arora Redis And Python

2.
Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?

3.
About Me <ul><li>I tweet at Sunil Arora / @_sunil_ </li></ul><ul><li>I work at ShopSocially </li></ul><ul><li>I blog at http://sunilarora.org </li></ul>

4.
Today's talk <ul><li>What is Redis </li></ul>How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis

5.
Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com

6.
Redis – Brief History <ul><li>Released in March 2009 (Open Source, BSD licensed) </li></ul><ul><li>VMWare hired Salvatore in March, 2010 </li></ul><ul><li>Then Pieter Noordhuis (key contributor) was hired </li></ul>

7.
What is Redis ? Its between lot of stuff, so difficult to categorize it precisely

8.
What is Redis ? <ul><li>I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well </li></ul><ul><li>-Salvatore Sanfilippo </li></ul>Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/

9.
Redis is... Remote Data Structure Server

10.
Redis is... <ul><li>Supports rich data types of computer science </li></ul><ul><ul><li>- Strings, Lists, Sets, Sorted Sets, Hashes... </li></ul></ul><ul><li>Rich sets of primitives (commands) to manipulate these types </li></ul><ul><li>Predictive complexity measurements </li></ul>

11.
A few fundamentals <ul><li>Written in C (no external dependency) </li></ul><ul><li>Uses memory as main storage </li></ul><ul><li>Uses disk for persistence </li></ul><ul><li>Single Threaded </li></ul><ul><li>Every command performs atomic execution </li></ul>

12.
Performance <ul><li>Screamingly fast performance </li></ul><ul><li>~50K read/write operations per seconds </li></ul><ul><li>100K+ read/write ops per second on a regular EC2 instance </li></ul>

13.
Installation <ul><li>$ git clone http://github.com/antirez/redis </li></ul><ul><li>$ cd redis </li></ul><ul><li>$ make </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>.......... </li></ul><ul><li>.......... </li></ul><ul><li>$./src/redis-cli </li></ul><ul><li>Redis> PING </li></ul><ul><li>PONG </li></ul>

14.
Python Libraries <ul><ul><li>Redis-py - Python client library </li></ul></ul><ul><ul><li>Txredisapi - An asynchronous Python client for the Redis database, based on Twisted. </li></ul></ul><ul><ul><li>Redisco - ORM for redis along the lines Ohm for Ruby </li></ul></ul>

15.
redis-py <ul><li>The most popular python client library </li></ul><ul><li>Andy McCurdy ( [email_address] ) </li></ul><ul><li>Github: http://github.com/andymccurdy/redis-py </li></ul><ul><ul><li>$easy_install redis OR </li></ul></ul><ul><ul><li>$pip install redis </li></ul></ul><ul><li>Optional </li></ul><ul><ul><li>$easy_install hiredis </li></ul></ul><ul><ul><li>$pip install hiredis </li></ul></ul>

16.
Lets get started... <ul><li>$ cd redis </li></ul><ul><li>$ ./src/redis-server </li></ul><ul><li>......... </li></ul><ul><li>>>> from redis import Redis </li></ul><ul><li>>>> redis_client = Redis() </li></ul><ul><li>>>> redis_client.keys() </li></ul><ul><li>>>> help(redis_client) </li></ul>

17.
Redis Keys <ul><li>Not binary safe. </li></ul><ul><li>Should not contain space or newline character </li></ul><ul><li>A few rules about keys: </li></ul><ul><ul><li>Too long keys are not a good idea </li></ul></ul><ul><ul><li>Too short keys is also not a good idea </li></ul></ul><ul><ul><li>“ object-type:id:field” can be a nice idea, i.e. “user:1001:name” </li></ul></ul>

18.
Operations on Keys <ul><li>KEYS </li></ul><ul><li>EXISTS </li></ul><ul><li>DEL </li></ul><ul><li>EXPIRE </li></ul><ul><li>OBJECT </li></ul><ul><li>PERSIST </li></ul><ul><li>RANDOMKEY </li></ul><ul><li>RENAME </li></ul><ul><li>TYPE </li></ul><ul><li>TTL </li></ul><ul><li>EXPIREAT </li></ul><ul><li>MOVE </li></ul>

19.
Lets play with keys <ul><li>>>>redis_client.keys() </li></ul><ul><li>>>>redis_client.exists('key') </li></ul><ul><li>>>>redis_client.delete('key') </li></ul><ul><li>>>>redis_client.type('key') </li></ul><ul><li>>>>...... </li></ul><ul><li>>>>...... </li></ul>

20.
Data Structures <ul><li>Strings </li></ul><ul><li>Lists </li></ul><ul><li>Sets </li></ul><ul><li>Sorted Sets </li></ul><ul><li>Hashes </li></ul>

21.
Strings <ul><li>SET </li></ul><ul><li>GET </li></ul><ul><li>MSET </li></ul><ul><li>MGET </li></ul><ul><li>SETEX </li></ul><ul><li>GETSET </li></ul><ul><li>SETNX </li></ul><ul><li>INCR </li></ul><ul><li>INCRBY </li></ul><ul><li>DECR </li></ul><ul><li>DECRBY </li></ul>

22.
Strings – with redis client <ul><li>>>> redis_client.set('key', 'value') </li></ul><ul><li>>>> redis_client.get('key') </li></ul><ul><li>>>> redis_client.delete('key') </li></ul>

23.
Fetch multiple keys at once <ul><li>mget/mset </li></ul><ul><li>redis_client.mset({'key1': 'val1', 'key2': 'val2'}) </li></ul><ul><li>redis_client.mget('key1', 'key2',......) </li></ul>

24.
Expiration <ul><li>Set a value with expire </li></ul><ul><li>>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs </li></ul><ul><li>>>>redis_client.expire('key', 2) </li></ul><ul><li>>>>redis_client.get('key') </li></ul><ul><li>>>>None </li></ul>

25.
Expire Semantics <ul><li>Key with expiry known as volatile keys </li></ul><ul><li>Whenever a volatile key is modified, its expiry is reset </li></ul><ul><ul><li>- To avoid inconsistency in following cases </li></ul></ul><ul><ul><ul><li>Replication </li></ul></ul></ul><ul><ul><ul><li>Append Only Log </li></ul></ul></ul>

26.
Uses To store transient states in your web application

27.
Uses Who is online?

28.
Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)

29.
Atomic Increments <ul><li>>>>help(redis_client.incr) </li></ul><ul><li>>>>help(redis_client.decr) </li></ul><ul><li>>>> </li></ul><ul><li>>>> redis_client.incr('counter', 1) </li></ul><ul><li>>>>1 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>2 </li></ul><ul><li>>>> redis_client.incr('counter') </li></ul><ul><li>>>>3 </li></ul>

30.
Uses High Speed counters (views/clicks/votes/likes..)

31.
Uses API Rate Limiting

32.
Uses Generating unique IDs

33.
Lists <ul><li>Ordered list of binarysafe strings </li></ul><ul><li>Doubly linked list </li></ul><ul><li>Memory footprint optimized for smaller list </li></ul><ul><li>O(1) insertion/deletion at both ends </li></ul>

34.
Lists - operations <ul><li>LPUSH </li></ul><ul><li>RPUSH </li></ul><ul><li>LSET </li></ul><ul><li>LRANGE </li></ul><ul><li>LPOP </li></ul><ul><li>BLPOP </li></ul><ul><li>BRPOP </li></ul><ul><li>BRPOPLPUSH </li></ul><ul><li>LINSERT </li></ul><ul><li>RPOP </li></ul><ul><li>RPOPLPUSH </li></ul><ul><li>LPUSHX </li></ul><ul><li>RPUSHX </li></ul>

35.
Uses Web apps are full of lists :)

36.
Uses Capped List

37.
Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque

38.
Uses Social Activity Streams or notifications

39.
Sets An unordered collection of distinct byte strings Nothing different from the data type in python

40.
Sets - Operations <ul><li>SADD </li></ul><ul><li>SCARD </li></ul><ul><li>SREM </li></ul><ul><li>SISMEMBER </li></ul><ul><li>SMEMBERS </li></ul><ul><li>SPOP </li></ul><ul><li>SRANDMEMBER </li></ul>

41.
Operations between Sets <ul><li>SMOVE </li></ul><ul><li>SUNION </li></ul><ul><li>SDIFF </li></ul><ul><li>SINTER </li></ul>

42.
Sets - Operations <ul><li>SDIFFSTORE </li></ul><ul><li>SINTERSTORE </li></ul><ul><li>SUNIONSTORE </li></ul>

43.
Sets - Uses <ul><li>Picking random items from a set using SRANDMEMBER </li></ul><ul><li>Ex. </li></ul><ul><ul><li>Picking a random article from daily news </li></ul></ul><ul><ul><li>Pick a random Ad for serving </li></ul></ul><ul><ul><li>Pick a random option for A/B </li></ul></ul>Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”

44.
Sets - Uses <ul><li>To model Relations in social graph </li></ul><ul><li>Ex. </li></ul><ul><ul><li>>>>redis_client.sadd('friends:john', 'jenny', 'maria') </li></ul></ul><ul><ul><li>>>>redis_client.sadd('friends:ben', 'maria', 'kate') </li></ul></ul><ul><ul><li>>>>redis_client.sinter('friends:john', 'friends:ben') </li></ul></ul>

45.
Relations (friend/followers) <ul><li>Common followlist for A and B </li></ul><ul><ul><li>>>> sinter('users:A:follows', 'users:B:follows') </li></ul></ul><ul><li>Unique to B compared to C </li></ul><ul><ul><li>>>>sdiff('users:B:follows', 'users:C:follows') </li></ul></ul><ul><li>Mutual relationship (friend as well as follower) </li></ul><ul><ul><li>>>>sinter('users:B:followers', 'users:B:friends') </li></ul></ul><ul><li>Who does not follow me back ? </li></ul><ul><ul><li>>>>sdiff('users:B:friends', 'users:B:followers') </li></ul></ul><ul><li>Who am I not following back? </li></ul><ul><ul><li>>>>sdiff('users:B:followers', 'user:B:friends') </li></ul></ul>

46.
Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends

47.
Which of my friends are online right now? <ul><li>>>>RENAME online:fresh online:stale #every minute </li></ul><ul><li>>>>SUNIONSTORE online:fresh online:stale </li></ul><ul><li>>>>#friend jack connects </li></ul><ul><li>>>>SADD online:fresh 'jack' </li></ul><ul><li>>>>SUNIONSTORE online online:fresh online:stale </li></ul><ul><li>>>>#who is online ? </li></ul><ul><li>>>>SINTER online friends </li></ul>

48.
Sorted Sets Ordered sets on the basis of score

49.
Sorted Sets <ul><li>ZADD </li></ul><ul><li>ZCARD </li></ul><ul><li>ZCOUNT </li></ul><ul><li>ZINCRBY </li></ul><ul><li>ZINTERSTORE </li></ul><ul><li>ZRANGE </li></ul><ul><li>ZRANGEBYSCORE </li></ul><ul><li>ZRANK </li></ul><ul><li>ZREM </li></ul><ul><li>ZREMRANGEBYRANK </li></ul><ul><li>ZREMRANGEBYSCORE </li></ul><ul><li>ZREVRANGE </li></ul><ul><li>ZREVRANGEBYSCORE </li></ul><ul><li>ZSCORE </li></ul><ul><li>ZUNIONSTORE </li></ul>

50.
Sorted Sets – Use Case To build index on your dataset

51.
Uses - Realtime Leaderboards <ul><li>>>>zincrby('leaderboard', 'john', 2) </li></ul><ul><li>>>>zincrby('leaderboard', 'jack', 5) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 1) </li></ul><ul><li>>>>zincrby('leaderboard', 'kate', 10) </li></ul><ul><li>.... </li></ul><ul><li>>>>zrange('leaderboard', 0, -1, withscores = True) </li></ul>

52.
Uses - Realtime Leaderboards <ul><li>Most download resources </li></ul><ul><li>Most popular articles on the website </li></ul><ul><li>Weekly popular list </li></ul><ul><li>Most downloaded stuff between date1 and date2 </li></ul><ul><li>..... </li></ul><ul><li>..... </li></ul>

53.
AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html

54.
Hashes <ul><li>Equivalent to Python dictionary or Ruby hash or Java hashmap </li></ul><ul><li>Operations </li></ul><ul><ul><li>hmset users:1 {'username': 'jim', 'score': 23} </li></ul></ul><ul><ul><li>hgetall users:1 </li></ul></ul><ul><ul><li>Hincrby </li></ul></ul><ul><li>Useful for storing structured data </li></ul><ul><ul><li>hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'} </li></ul></ul><ul><ul><li>Expire user:1:preferences <duration> </li></ul></ul><ul><ul><li>Hmgetall user:1:preferences #returns entire dict </li></ul></ul>

55.
Redis Transactions <ul><li>Using Multi/Watch </li></ul><ul><li>>>>p = redis_client.pipeline() </li></ul><ul><li>>>>p.lpush('a', 1) </li></ul><ul><li>>>>p.ltrim('a', 0, 100) </li></ul><ul><li>>>>p.execute </li></ul><ul><li>Limitation </li></ul><ul><ul><li>Since commands are queued, Can't read values </li></ul></ul><ul><ul><li>No conditional execution </li></ul></ul><ul><ul><li>If not high write contention scenario, WATCH can be used </li></ul></ul><ul><li>Use Redis Scripting to write your own primitive </li></ul>

56.
Designing with Redis <ul><li>Data layout design on basis of Query </li></ul><ul><li>No Query Optimizer </li></ul><ul><li>Manually build indexes </li></ul>

57.
Durability <ul><li>Snapshotting mode </li></ul><ul><ul><li>Binary dump every x secs or y ops </li></ul></ul><ul><li>Append Only File (AOF) </li></ul><ul><ul><li>Every command is written to a file </li></ul></ul><ul><ul><li>On restart/crash, commands replayed </li></ul></ul><ul><ul><li>Fsync on every new command </li></ul></ul><ul><ul><li>Fsync every second </li></ul></ul><ul><ul><li>OS decide to </li></ul></ul><ul><li>Replication </li></ul>

58.
Publish/Subscribe <ul><li>A simple and efficient implementation of publish/subscribe messaging paradigm </li></ul><ul><li>Client can subscribe/psubscribe to receive messages on channels (key patterns) </li></ul>

59.
Publish/Subscribe <ul><li>PSUBSCRIBE </li></ul><ul><li>PUBLISH </li></ul><ul><li>PUNSUBSCRIBE </li></ul><ul><li>SUBSCRIBE </li></ul><ul><li>UNSUBSCRIBE </li></ul>

60.
Uses Many to Many message passing

61.
Uses Web Chat

62.
References <ul><li>Redis Website ( http://redis.io ) </li></ul><ul><li>SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ ) </li></ul><ul><li>Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 ) </li></ul><ul><li>Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html ) </li></ul><ul><li>Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications) </li></ul>

63.
References <ul><li>Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/) </li></ul><ul><li>Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/ </li></ul><ul><li>Redis at Bump </li></ul><ul><li>http://devblog.bu.mp/how-we-use-redis-at-bump </li></ul><ul><li>Redis: AK 47 of postrelational database </li></ul><ul><li>http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases </li></ul>

64.
Questions

65.
Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com