Redis And python at pycon_2011

PyCon India, 2011 Sunil Arora Redis And Python

Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?

About Me
I tweet at Sunil Arora / @_sunil_
I work at ShopSocially
I blog at http://sunilarora.org

Today's talk
What is Redis
How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis

Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com

Redis – Brief History
Released in March 2009 (Open Source, BSD licensed)
VMWare hired Salvatore in March, 2010
Then Pieter Noordhuis (key contributor) was hired

What is Redis ? Its between lot of stuff, so difficult to categorize it precisely

What is Redis ?
I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well
-Salvatore Sanfilippo
Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/

Redis is... Remote Data Structure Server

Redis is...
Supports rich data types of computer science
- Strings, Lists, Sets, Sorted Sets, Hashes...
Rich sets of primitives (commands) to manipulate these types
Predictive complexity measurements

A few fundamentals
Written in C (no external dependency)
Uses memory as main storage
Uses disk for persistence
Single Threaded
Every command performs atomic execution

Performance
Screamingly fast performance
~50K read/write operations per seconds
100K+ read/write ops per second on a regular EC2 instance

Installation
$ git clone http://github.com/antirez/redis
$ cd redis
$ make
$ ./src/redis-server
..........
..........
$./src/redis-cli
Redis> PING
PONG

Python Libraries
Redis-py - Python client library
Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
Redisco - ORM for redis along the lines Ohm for Ruby

redis-py
The most popular python client library
Andy McCurdy ( [email_address] )
Github: http://github.com/andymccurdy/redis-py
$easy_install redis OR
$pip install redis
Optional
$easy_install hiredis
$pip install hiredis

Lets get started...
$ cd redis
$ ./src/redis-server
.........
>>> from redis import Redis
>>> redis_client = Redis()
>>> redis_client.keys()
>>> help(redis_client)

Redis Keys
Not binary safe.
Should not contain space or newline character
A few rules about keys:
Too long keys are not a good idea
Too short keys is also not a good idea
“ object-type:id:field” can be a nice idea, i.e. “user:1001:name”

Operations on Keys
KEYS
EXISTS
DEL
EXPIRE
OBJECT
PERSIST
RANDOMKEY
RENAME
TYPE
TTL
EXPIREAT
MOVE

Lets play with keys
>>>redis_client.keys()
>>>redis_client.exists('key')
>>>redis_client.delete('key')
>>>redis_client.type('key')
>>>......
>>>......

Data Structures
Strings
Lists
Sets
Sorted Sets
Hashes

Strings
SET
GET
MSET
MGET
SETEX
GETSET
SETNX
INCR
INCRBY
DECR
DECRBY

Strings – with redis client
>>> redis_client.set('key', 'value')
>>> redis_client.get('key')
>>> redis_client.delete('key')

Fetch multiple keys at once
mget/mset
redis_client.mset({'key1': 'val1', 'key2': 'val2'})
redis_client.mget('key1', 'key2',......)

Expiration
Set a value with expire
>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
>>>redis_client.expire('key', 2)
>>>redis_client.get('key')
>>>None

Expire Semantics
Key with expiry known as volatile keys
Whenever a volatile key is modified, its expiry is reset
- To avoid inconsistency in following cases
Replication
Append Only Log

Uses To store transient states in your web application

Uses Who is online?

Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)

Atomic Increments
>>>help(redis_client.incr)
>>>help(redis_client.decr)
>>>
>>> redis_client.incr('counter', 1)
>>>1
>>> redis_client.incr('counter')
>>>2
>>> redis_client.incr('counter')
>>>3

Uses High Speed counters (views/clicks/votes/likes..)

Uses API Rate Limiting

Uses Generating unique IDs

Lists
Ordered list of binarysafe strings
Doubly linked list
Memory footprint optimized for smaller list
O(1) insertion/deletion at both ends

Lists - operations
LPUSH
RPUSH
LSET
LRANGE
LPOP
BLPOP
BRPOP
BRPOPLPUSH
LINSERT
RPOP
RPOPLPUSH
LPUSHX
RPUSHX

Uses Web apps are full of lists :)

Uses Capped List

Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque

Uses Social Activity Streams or notifications

Sets An unordered collection of distinct byte strings Nothing different from the data type in python

Sets - Operations
SADD
SCARD
SREM
SISMEMBER
SMEMBERS
SPOP
SRANDMEMBER

Operations between Sets
SMOVE
SUNION
SDIFF
SINTER

Sets - Operations
SDIFFSTORE
SINTERSTORE
SUNIONSTORE

Sets - Uses
Picking random items from a set using SRANDMEMBER
Ex.
Picking a random article from daily news
Pick a random Ad for serving
Pick a random option for A/B
Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”

Sets - Uses
To model Relations in social graph
Ex.
>>>redis_client.sadd('friends:john', 'jenny', 'maria')
>>>redis_client.sadd('friends:ben', 'maria', 'kate')
>>>redis_client.sinter('friends:john', 'friends:ben')

Relations (friend/followers)
Common followlist for A and B
>>> sinter('users:A:follows', 'users:B:follows')
Unique to B compared to C
>>>sdiff('users:B:follows', 'users:C:follows')
Mutual relationship (friend as well as follower)
>>>sinter('users:B:followers', 'users:B:friends')
Who does not follow me back ?
>>>sdiff('users:B:friends', 'users:B:followers')
Who am I not following back?
>>>sdiff('users:B:followers', 'user:B:friends')

Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends

Which of my friends are online right now?
>>>RENAME online:fresh online:stale #every minute
>>>SUNIONSTORE online:fresh online:stale
>>>#friend jack connects
>>>SADD online:fresh 'jack'
>>>SUNIONSTORE online online:fresh online:stale
>>>#who is online ?
>>>SINTER online friends

Sorted Sets Ordered sets on the basis of score

Sorted Sets
ZADD
ZCARD
ZCOUNT
ZINCRBY
ZINTERSTORE
ZRANGE
ZRANGEBYSCORE
ZRANK
ZREM
ZREMRANGEBYRANK
ZREMRANGEBYSCORE
ZREVRANGE
ZREVRANGEBYSCORE
ZSCORE
ZUNIONSTORE

Sorted Sets – Use Case To build index on your dataset

Uses - Realtime Leaderboards
>>>zincrby('leaderboard', 'john', 2)
>>>zincrby('leaderboard', 'jack', 5)
>>>zincrby('leaderboard', 'kate', 1)
>>>zincrby('leaderboard', 'kate', 10)
....
>>>zrange('leaderboard', 0, -1, withscores = True)

Uses - Realtime Leaderboards
Most download resources
Most popular articles on the website
Weekly popular list
Most downloaded stuff between date1 and date2
.....
.....

AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html

Hashes
Equivalent to Python dictionary or Ruby hash or Java hashmap
Operations
hmset users:1 {'username': 'jim', 'score': 23}
hgetall users:1
Hincrby
Useful for storing structured data
hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
Expire user:1:preferences <duration>
Hmgetall user:1:preferences #returns entire dict

Redis Transactions
Using Multi/Watch
>>>p = redis_client.pipeline()
>>>p.lpush('a', 1)
>>>p.ltrim('a', 0, 100)
>>>p.execute
Limitation
Since commands are queued, Can't read values
No conditional execution
If not high write contention scenario, WATCH can be used
Use Redis Scripting to write your own primitive

Designing with Redis
Data layout design on basis of Query
No Query Optimizer
Manually build indexes

Durability
Snapshotting mode
Binary dump every x secs or y ops
Append Only File (AOF)
Every command is written to a file
On restart/crash, commands replayed
Fsync on every new command
Fsync every second
OS decide to
Replication

Publish/Subscribe
A simple and efficient implementation of publish/subscribe messaging paradigm
Client can subscribe/psubscribe to receive messages on channels (key patterns)

Publish/Subscribe
PSUBSCRIBE
PUBLISH
PUNSUBSCRIBE
SUBSCRIBE
UNSUBSCRIBE

Uses Many to Many message passing

Uses Web Chat

References
Redis Website ( http://redis.io )
SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ )
Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 )
Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html )
Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)

References
Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
Redis at Bump
http://devblog.bu.mp/how-we-use-redis-at-bump
Redis: AK 47 of postrelational database
http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases

Questions

Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com

http://blog.nosqlfan.com	1938
http://sunilarora.org	1099
http://www.mypython.info	1011
http://simple-is-better.com	462
http://www.pythoner.info	387
http://feed.feedsky.com	61
http://www.webguo.com	60
http://www.kuqin.com	47
http://paper.li	21
http://xianguo.com	20
http://www.simple-is-better.com	16
http://voilethill.blogspot.in	15
http://voilethill.blogspot.com	14
http://www.zhuaxia.com	13
http://voilethill.blogspot.fr	8
http://posterous.com	8
http://abtasty.com	7
http://reader.youdao.com	6
http://voilethill.blogspot.ch	5
http://voilethill.blogspot.jp	3
http://trunk.ly	3
http://www.docshut.com	2
http://voilethill.blogspot.tw	2
http://voilethill.blogspot.hk	2
http://voilethill.blogspot.gr	2
http://www.pythonpub.com	2
http://pythonspace.webfactional.com	2
http://www.pydn.cn	2
https://twitter.com	2
http://us-w1.rockmelt.com	2
http://prlog.ru	1
http://sunilarora.posthaven.com	1
http://voilethill.blogspot.ru	1
http://voilethill.blogspot.kr	1
http://voilethill.blogspot.ca	1
http://www.slideshare.net	1
http://xnny.net	1
http://eatsalt.blog.163.com	1
http://voilethill.blogspot.nl	1
http://dev.toeska.cl	1
http://cache.baidu.com	1
http://webcache.googleusercontent.com	1
http://translate.googleusercontent.com	1
http://localhost	1
http://voilethill.blogspot.ro	1

Redis And python at pycon_2011

by sunilar0ra

Accessibility

Categories

Upload Details

Usage Rights

45 Embeds 5,237

Statistics

Redis And python at pycon_2011 Presentation Transcript