Redis And python at pycon_2011
Upcoming SlideShare
Loading in...5
×
 

Redis And python at pycon_2011

on

  • 16,652 views

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

Statistics

Views

Total Views
16,652
Slideshare-icon Views on SlideShare
10,480
Embed Views
6,172

Actions

Likes
41
Downloads
423
Comments
1

55 Embeds 6,172

http://blog.nosqlfan.com 2215
http://sunilarora.org 1674
http://www.mypython.info 1011
http://simple-is-better.com 474
http://www.pythoner.info 387
http://feed.feedsky.com 61
http://www.webguo.com 60
http://www.kuqin.com 48
http://localhost 26
http://voilethill.blogspot.com 25
http://paper.li 21
http://xianguo.com 20
http://voilethill.blogspot.in 17
http://www.simple-is-better.com 16
http://www.zhuaxia.com 13
https://twitter.com 10
http://voilethill.blogspot.fr 9
http://posterous.com 8
http://voilethill.blogspot.kr 8
http://abtasty.com 7
http://reader.youdao.com 6
http://voilethill.blogspot.ch 5
http://voilethill.blogspot.hk 5
http://voilethill.blogspot.jp 3
http://trunk.ly 3
http://www.docshut.com 3
http://www.pydn.cn 2
http://prlog.ru 2
http://dev1.localhost 2
http://voilethill.blogspot.tw 2
http://voilethill.blogspot.gr 2
http://us-w1.rockmelt.com 2
http://pythonspace.webfactional.com 2
http://www.pythonpub.com 2
http://droot.github.io 1
http://voilethill.blogspot.ro 1
http://pythontip.sinaapp.com 1
http://xue.uplook.cn 1
http://www.uplook.cn 1
http://www.r66r.net 1
http://cache.baiducontent.com 1
http://voilethill.blogspot.co.uk 1
http://www.schoox.com 1
http://www.techgig.com 1
http://sunilarora.posthaven.com 1
http://eatsalt.blog.163.com 1
http://cache.baidu.com 1
http://webcache.googleusercontent.com 1
http://xnny.net 1
http://dev.toeska.cl 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Redis And python at pycon_2011 Redis And python at pycon_2011 Presentation Transcript

    • PyCon India, 2011 Sunil Arora Redis And Python
    • Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?
    • About Me
      • I tweet at Sunil Arora / @_sunil_
      • I work at ShopSocially
      • I blog at http://sunilarora.org
    • Today's talk
      • What is Redis
      How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis
    • Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com
    • Redis – Brief History
      • Released in March 2009 (Open Source, BSD licensed)
      • VMWare hired Salvatore in March, 2010
      • Then Pieter Noordhuis (key contributor) was hired
    • What is Redis ? Its between lot of stuff, so difficult to categorize it precisely
    • What is Redis ?
      • I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well
      • -Salvatore Sanfilippo
      Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
    • Redis is... Remote Data Structure Server
    • Redis is...
      • Supports rich data types of computer science
        • - Strings, Lists, Sets, Sorted Sets, Hashes...
      • Rich sets of primitives (commands) to manipulate these types
      • Predictive complexity measurements
    • A few fundamentals
      • Written in C (no external dependency)
      • Uses memory as main storage
      • Uses disk for persistence
      • Single Threaded
      • Every command performs atomic execution
    • Performance
      • Screamingly fast performance
      • ~50K read/write operations per seconds
      • 100K+ read/write ops per second on a regular EC2 instance
    • Installation
      • $ git clone http://github.com/antirez/redis
      • $ cd redis
      • $ make
      • $ ./src/redis-server
      • ..........
      • ..........
      • $./src/redis-cli
      • Redis> PING
      • PONG
    • Python Libraries
        • Redis-py - Python client library
        • Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
        • Redisco - ORM for redis along the lines Ohm for Ruby
    • redis-py
      • The most popular python client library
      • Andy McCurdy ( [email_address] )
      • Github: http://github.com/andymccurdy/redis-py
        • $easy_install redis OR
        • $pip install redis
      • Optional
        • $easy_install hiredis
        • $pip install hiredis
    • Lets get started...
      • $ cd redis
      • $ ./src/redis-server
      • .........
      • >>> from redis import Redis
      • >>> redis_client = Redis()
      • >>> redis_client.keys()
      • >>> help(redis_client)
    • Redis Keys
      • Not binary safe.
      • Should not contain space or newline character
      • A few rules about keys:
        • Too long keys are not a good idea
        • Too short keys is also not a good idea
        • “ object-type:id:field” can be a nice idea, i.e. “user:1001:name”
    • Operations on Keys
      • KEYS
      • EXISTS
      • DEL
      • EXPIRE
      • OBJECT
      • PERSIST
      • RANDOMKEY
      • RENAME
      • TYPE
      • TTL
      • EXPIREAT
      • MOVE
    • Lets play with keys
      • >>>redis_client.keys()
      • >>>redis_client.exists('key')
      • >>>redis_client.delete('key')
      • >>>redis_client.type('key')
      • >>>......
      • >>>......
    • Data Structures
      • Strings
      • Lists
      • Sets
      • Sorted Sets
      • Hashes
    • Strings
      • SET
      • GET
      • MSET
      • MGET
      • SETEX
      • GETSET
      • SETNX
      • INCR
      • INCRBY
      • DECR
      • DECRBY
    • Strings – with redis client
      • >>> redis_client.set('key', 'value')
      • >>> redis_client.get('key')
      • >>> redis_client.delete('key')
    • Fetch multiple keys at once
      • mget/mset
      • redis_client.mset({'key1': 'val1', 'key2': 'val2'})
      • redis_client.mget('key1', 'key2',......)
    • Expiration
      • Set a value with expire
      • >>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
      • >>>redis_client.expire('key', 2)
      • >>>redis_client.get('key')
      • >>>None
    • Expire Semantics
      • Key with expiry known as volatile keys
      • Whenever a volatile key is modified, its expiry is reset
        • - To avoid inconsistency in following cases
          • Replication
          • Append Only Log
    • Uses To store transient states in your web application
    • Uses Who is online?
    • Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
    • Atomic Increments
      • >>>help(redis_client.incr)
      • >>>help(redis_client.decr)
      • >>>
      • >>> redis_client.incr('counter', 1)
      • >>>1
      • >>> redis_client.incr('counter')
      • >>>2
      • >>> redis_client.incr('counter')
      • >>>3
    • Uses High Speed counters (views/clicks/votes/likes..)
    • Uses API Rate Limiting
    • Uses Generating unique IDs
    • Lists
      • Ordered list of binarysafe strings
      • Doubly linked list
      • Memory footprint optimized for smaller list
      • O(1) insertion/deletion at both ends
    • Lists - operations
      • LPUSH
      • RPUSH
      • LSET
      • LRANGE
      • LPOP
      • BLPOP
      • BRPOP
      • BRPOPLPUSH
      • LINSERT
      • RPOP
      • RPOPLPUSH
      • LPUSHX
      • RPUSHX
    • Uses Web apps are full of lists :)
    • Uses Capped List
    • Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque
    • Uses Social Activity Streams or notifications
    • Sets An unordered collection of distinct byte strings Nothing different from the data type in python
    • Sets - Operations
      • SADD
      • SCARD
      • SREM
      • SISMEMBER
      • SMEMBERS
      • SPOP
      • SRANDMEMBER
    • Operations between Sets
      • SMOVE
      • SUNION
      • SDIFF
      • SINTER
    • Sets - Operations
      • SDIFFSTORE
      • SINTERSTORE
      • SUNIONSTORE
    • Sets - Uses
      • Picking random items from a set using SRANDMEMBER
      • Ex.
        • Picking a random article from daily news
        • Pick a random Ad for serving
        • Pick a random option for A/B
      Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
    • Sets - Uses
      • To model Relations in social graph
      • Ex.
        • >>>redis_client.sadd('friends:john', 'jenny', 'maria')
        • >>>redis_client.sadd('friends:ben', 'maria', 'kate')
        • >>>redis_client.sinter('friends:john', 'friends:ben')
    • Relations (friend/followers)
      • Common followlist for A and B
        • >>> sinter('users:A:follows', 'users:B:follows')
      • Unique to B compared to C
        • >>>sdiff('users:B:follows', 'users:C:follows')
      • Mutual relationship (friend as well as follower)
        • >>>sinter('users:B:followers', 'users:B:friends')
      • Who does not follow me back ?
        • >>>sdiff('users:B:friends', 'users:B:followers')
      • Who am I not following back?
        • >>>sdiff('users:B:followers', 'user:B:friends')
    • Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends
    • Which of my friends are online right now?
      • >>>RENAME online:fresh online:stale #every minute
      • >>>SUNIONSTORE online:fresh online:stale
      • >>>#friend jack connects
      • >>>SADD online:fresh 'jack'
      • >>>SUNIONSTORE online online:fresh online:stale
      • >>>#who is online ?
      • >>>SINTER online friends
    • Sorted Sets Ordered sets on the basis of score
    • Sorted Sets
      • ZADD
      • ZCARD
      • ZCOUNT
      • ZINCRBY
      • ZINTERSTORE
      • ZRANGE
      • ZRANGEBYSCORE
      • ZRANK
      • ZREM
      • ZREMRANGEBYRANK
      • ZREMRANGEBYSCORE
      • ZREVRANGE
      • ZREVRANGEBYSCORE
      • ZSCORE
      • ZUNIONSTORE
    • Sorted Sets – Use Case To build index on your dataset
    • Uses - Realtime Leaderboards
      • >>>zincrby('leaderboard', 'john', 2)
      • >>>zincrby('leaderboard', 'jack', 5)
      • >>>zincrby('leaderboard', 'kate', 1)
      • >>>zincrby('leaderboard', 'kate', 10)
      • ....
      • >>>zrange('leaderboard', 0, -1, withscores = True)
    • Uses - Realtime Leaderboards
      • Most download resources
      • Most popular articles on the website
      • Weekly popular list
      • Most downloaded stuff between date1 and date2
      • .....
      • .....
    • AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html
    • Hashes
      • Equivalent to Python dictionary or Ruby hash or Java hashmap
      • Operations
        • hmset users:1 {'username': 'jim', 'score': 23}
        • hgetall users:1
        • Hincrby
      • Useful for storing structured data
        • hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
        • Expire user:1:preferences <duration>
        • Hmgetall user:1:preferences #returns entire dict
    • Redis Transactions
      • Using Multi/Watch
      • >>>p = redis_client.pipeline()
      • >>>p.lpush('a', 1)
      • >>>p.ltrim('a', 0, 100)
      • >>>p.execute
      • Limitation
        • Since commands are queued, Can't read values
        • No conditional execution
        • If not high write contention scenario, WATCH can be used
      • Use Redis Scripting to write your own primitive
    • Designing with Redis
      • Data layout design on basis of Query
      • No Query Optimizer
      • Manually build indexes
    • Durability
      • Snapshotting mode
        • Binary dump every x secs or y ops
      • Append Only File (AOF)
        • Every command is written to a file
        • On restart/crash, commands replayed
        • Fsync on every new command
        • Fsync every second
        • OS decide to
      • Replication
    • Publish/Subscribe
      • A simple and efficient implementation of publish/subscribe messaging paradigm
      • Client can subscribe/psubscribe to receive messages on channels (key patterns)
    • Publish/Subscribe
      • PSUBSCRIBE
      • PUBLISH
      • PUNSUBSCRIBE
      • SUBSCRIBE
      • UNSUBSCRIBE
    • Uses Many to Many message passing
    • Uses Web Chat
    • References
      • Redis Website ( http://redis.io )
      • SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ )
      • Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 )
      • Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html )
      • Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)
    • References
      • Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
      • Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
      • Redis at Bump
      • http://devblog.bu.mp/how-we-use-redis-at-bump
      • Redis: AK 47 of postrelational database
      • http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases
    • Questions
    • Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com