Redis And python at pycon_2011
Upcoming SlideShare
Loading in...5
×
 

Redis And python at pycon_2011

on

  • 16,843 views

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

Statistics

Views

Total Views
16,843
Views on SlideShare
10,656
Embed Views
6,187

Actions

Likes
41
Downloads
426
Comments
1

56 Embeds 6,187

http://blog.nosqlfan.com 2226
http://sunilarora.org 1677
http://www.mypython.info 1011
http://simple-is-better.com 474
http://www.pythoner.info 387
http://feed.feedsky.com 61
http://www.webguo.com 60
http://www.kuqin.com 48
http://localhost 26
http://voilethill.blogspot.com 25
http://paper.li 21
http://xianguo.com 20
http://voilethill.blogspot.in 17
http://www.simple-is-better.com 16
http://www.zhuaxia.com 13
https://twitter.com 10
http://voilethill.blogspot.fr 9
http://posterous.com 8
http://voilethill.blogspot.kr 8
http://abtasty.com 7
http://reader.youdao.com 6
http://voilethill.blogspot.ch 5
http://voilethill.blogspot.hk 5
http://trunk.ly 3
http://voilethill.blogspot.jp 3
http://www.docshut.com 3
http://voilethill.blogspot.tw 2
http://voilethill.blogspot.gr 2
http://us-w1.rockmelt.com 2
http://www.pydn.cn 2
http://www.pythonpub.com 2
http://prlog.ru 2
http://pythonspace.webfactional.com 2
http://dev1.localhost 2
http://cache.baiducontent.com 1
http://www.uplook.cn 1
http://www.schoox.com 1
http://www.r66r.net 1
http://xue.uplook.cn 1
http://pythontip.sinaapp.com 1
http://www.techgig.com 1
http://droot.github.io 1
http://pluto 1
http://voilethill.blogspot.co.uk 1
http://voilethill.blogspot.ro 1
http://sunilarora.posthaven.com 1
http://eatsalt.blog.163.com 1
http://cache.baidu.com 1
http://webcache.googleusercontent.com 1
http://xnny.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Redis And python at pycon_2011 Redis And python at pycon_2011 Presentation Transcript

  • PyCon India, 2011 Sunil Arora Redis And Python
  • Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?
  • About Me
    • I tweet at Sunil Arora / @_sunil_
    • I work at ShopSocially
    • I blog at http://sunilarora.org
  • Today's talk
    • What is Redis
    How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis
  • Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com
  • Redis – Brief History
    • Released in March 2009 (Open Source, BSD licensed)
    • VMWare hired Salvatore in March, 2010
    • Then Pieter Noordhuis (key contributor) was hired
  • What is Redis ? Its between lot of stuff, so difficult to categorize it precisely
  • What is Redis ?
    • I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well
    • -Salvatore Sanfilippo
    Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
  • Redis is... Remote Data Structure Server
  • Redis is...
    • Supports rich data types of computer science
      • - Strings, Lists, Sets, Sorted Sets, Hashes...
    • Rich sets of primitives (commands) to manipulate these types
    • Predictive complexity measurements
  • A few fundamentals
    • Written in C (no external dependency)
    • Uses memory as main storage
    • Uses disk for persistence
    • Single Threaded
    • Every command performs atomic execution
  • Performance
    • Screamingly fast performance
    • ~50K read/write operations per seconds
    • 100K+ read/write ops per second on a regular EC2 instance
  • Installation
    • $ git clone http://github.com/antirez/redis
    • $ cd redis
    • $ make
    • $ ./src/redis-server
    • ..........
    • ..........
    • $./src/redis-cli
    • Redis> PING
    • PONG
  • Python Libraries
      • Redis-py - Python client library
      • Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
      • Redisco - ORM for redis along the lines Ohm for Ruby
  • redis-py
    • The most popular python client library
    • Andy McCurdy ( [email_address] )
    • Github: http://github.com/andymccurdy/redis-py
      • $easy_install redis OR
      • $pip install redis
    • Optional
      • $easy_install hiredis
      • $pip install hiredis
  • Lets get started...
    • $ cd redis
    • $ ./src/redis-server
    • .........
    • >>> from redis import Redis
    • >>> redis_client = Redis()
    • >>> redis_client.keys()
    • >>> help(redis_client)
  • Redis Keys
    • Not binary safe.
    • Should not contain space or newline character
    • A few rules about keys:
      • Too long keys are not a good idea
      • Too short keys is also not a good idea
      • “ object-type:id:field” can be a nice idea, i.e. “user:1001:name”
  • Operations on Keys
    • KEYS
    • EXISTS
    • DEL
    • EXPIRE
    • OBJECT
    • PERSIST
    • RANDOMKEY
    • RENAME
    • TYPE
    • TTL
    • EXPIREAT
    • MOVE
  • Lets play with keys
    • >>>redis_client.keys()
    • >>>redis_client.exists('key')
    • >>>redis_client.delete('key')
    • >>>redis_client.type('key')
    • >>>......
    • >>>......
  • Data Structures
    • Strings
    • Lists
    • Sets
    • Sorted Sets
    • Hashes
  • Strings
    • SET
    • GET
    • MSET
    • MGET
    • SETEX
    • GETSET
    • SETNX
    • INCR
    • INCRBY
    • DECR
    • DECRBY
  • Strings – with redis client
    • >>> redis_client.set('key', 'value')
    • >>> redis_client.get('key')
    • >>> redis_client.delete('key')
  • Fetch multiple keys at once
    • mget/mset
    • redis_client.mset({'key1': 'val1', 'key2': 'val2'})
    • redis_client.mget('key1', 'key2',......)
  • Expiration
    • Set a value with expire
    • >>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
    • >>>redis_client.expire('key', 2)
    • >>>redis_client.get('key')
    • >>>None
  • Expire Semantics
    • Key with expiry known as volatile keys
    • Whenever a volatile key is modified, its expiry is reset
      • - To avoid inconsistency in following cases
        • Replication
        • Append Only Log
  • Uses To store transient states in your web application
  • Uses Who is online?
  • Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
  • Atomic Increments
    • >>>help(redis_client.incr)
    • >>>help(redis_client.decr)
    • >>>
    • >>> redis_client.incr('counter', 1)
    • >>>1
    • >>> redis_client.incr('counter')
    • >>>2
    • >>> redis_client.incr('counter')
    • >>>3
  • Uses High Speed counters (views/clicks/votes/likes..)
  • Uses API Rate Limiting
  • Uses Generating unique IDs
  • Lists
    • Ordered list of binarysafe strings
    • Doubly linked list
    • Memory footprint optimized for smaller list
    • O(1) insertion/deletion at both ends
  • Lists - operations
    • LPUSH
    • RPUSH
    • LSET
    • LRANGE
    • LPOP
    • BLPOP
    • BRPOP
    • BRPOPLPUSH
    • LINSERT
    • RPOP
    • RPOPLPUSH
    • LPUSHX
    • RPUSHX
  • Uses Web apps are full of lists :)
  • Uses Capped List
  • Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque
  • Uses Social Activity Streams or notifications
  • Sets An unordered collection of distinct byte strings Nothing different from the data type in python
  • Sets - Operations
    • SADD
    • SCARD
    • SREM
    • SISMEMBER
    • SMEMBERS
    • SPOP
    • SRANDMEMBER
  • Operations between Sets
    • SMOVE
    • SUNION
    • SDIFF
    • SINTER
  • Sets - Operations
    • SDIFFSTORE
    • SINTERSTORE
    • SUNIONSTORE
  • Sets - Uses
    • Picking random items from a set using SRANDMEMBER
    • Ex.
      • Picking a random article from daily news
      • Pick a random Ad for serving
      • Pick a random option for A/B
    Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
  • Sets - Uses
    • To model Relations in social graph
    • Ex.
      • >>>redis_client.sadd('friends:john', 'jenny', 'maria')
      • >>>redis_client.sadd('friends:ben', 'maria', 'kate')
      • >>>redis_client.sinter('friends:john', 'friends:ben')
  • Relations (friend/followers)
    • Common followlist for A and B
      • >>> sinter('users:A:follows', 'users:B:follows')
    • Unique to B compared to C
      • >>>sdiff('users:B:follows', 'users:C:follows')
    • Mutual relationship (friend as well as follower)
      • >>>sinter('users:B:followers', 'users:B:friends')
    • Who does not follow me back ?
      • >>>sdiff('users:B:friends', 'users:B:followers')
    • Who am I not following back?
      • >>>sdiff('users:B:followers', 'user:B:friends')
  • Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends
  • Which of my friends are online right now?
    • >>>RENAME online:fresh online:stale #every minute
    • >>>SUNIONSTORE online:fresh online:stale
    • >>>#friend jack connects
    • >>>SADD online:fresh 'jack'
    • >>>SUNIONSTORE online online:fresh online:stale
    • >>>#who is online ?
    • >>>SINTER online friends
  • Sorted Sets Ordered sets on the basis of score
  • Sorted Sets
    • ZADD
    • ZCARD
    • ZCOUNT
    • ZINCRBY
    • ZINTERSTORE
    • ZRANGE
    • ZRANGEBYSCORE
    • ZRANK
    • ZREM
    • ZREMRANGEBYRANK
    • ZREMRANGEBYSCORE
    • ZREVRANGE
    • ZREVRANGEBYSCORE
    • ZSCORE
    • ZUNIONSTORE
  • Sorted Sets – Use Case To build index on your dataset
  • Uses - Realtime Leaderboards
    • >>>zincrby('leaderboard', 'john', 2)
    • >>>zincrby('leaderboard', 'jack', 5)
    • >>>zincrby('leaderboard', 'kate', 1)
    • >>>zincrby('leaderboard', 'kate', 10)
    • ....
    • >>>zrange('leaderboard', 0, -1, withscores = True)
  • Uses - Realtime Leaderboards
    • Most download resources
    • Most popular articles on the website
    • Weekly popular list
    • Most downloaded stuff between date1 and date2
    • .....
    • .....
  • AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html
  • Hashes
    • Equivalent to Python dictionary or Ruby hash or Java hashmap
    • Operations
      • hmset users:1 {'username': 'jim', 'score': 23}
      • hgetall users:1
      • Hincrby
    • Useful for storing structured data
      • hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
      • Expire user:1:preferences <duration>
      • Hmgetall user:1:preferences #returns entire dict
  • Redis Transactions
    • Using Multi/Watch
    • >>>p = redis_client.pipeline()
    • >>>p.lpush('a', 1)
    • >>>p.ltrim('a', 0, 100)
    • >>>p.execute
    • Limitation
      • Since commands are queued, Can't read values
      • No conditional execution
      • If not high write contention scenario, WATCH can be used
    • Use Redis Scripting to write your own primitive
  • Designing with Redis
    • Data layout design on basis of Query
    • No Query Optimizer
    • Manually build indexes
  • Durability
    • Snapshotting mode
      • Binary dump every x secs or y ops
    • Append Only File (AOF)
      • Every command is written to a file
      • On restart/crash, commands replayed
      • Fsync on every new command
      • Fsync every second
      • OS decide to
    • Replication
  • Publish/Subscribe
    • A simple and efficient implementation of publish/subscribe messaging paradigm
    • Client can subscribe/psubscribe to receive messages on channels (key patterns)
  • Publish/Subscribe
    • PSUBSCRIBE
    • PUBLISH
    • PUNSUBSCRIBE
    • SUBSCRIBE
    • UNSUBSCRIBE
  • Uses Many to Many message passing
  • Uses Web Chat
  • References
    • Redis Website ( http://redis.io )
    • SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ )
    • Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 )
    • Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html )
    • Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)
  • References
    • Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
    • Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
    • Redis at Bump
    • http://devblog.bu.mp/how-we-use-redis-at-bump
    • Redis: AK 47 of postrelational database
    • http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases
  • Questions
  • Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com