Redis And python at pycon_2011

  • 16,611 views
Uploaded on

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
16,611
On Slideshare
0
From Embeds
0
Number of Embeds
27

Actions

Shares
Downloads
439
Comments
1
Likes
42

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. PyCon India, 2011 Sunil Arora Redis And Python
  • 2. Raising Hands... How many of you have used Redis before ? How many of you have got a laptop ?
  • 3. About Me
    • I tweet at Sunil Arora / @_sunil_
    • I work at ShopSocially
    • I blog at http://sunilarora.org
  • 4. Today's talk
    • What is Redis
    How it works and what you can do with it Real life use-cases Real life use-cases Some hand-on with Redis
  • 5. Redis - Brief History Initially written to improve performance of Web Analytics product LLOOGG out of his startup Salvatore Sanfilippo @antirez http://antirez.com
  • 6. Redis – Brief History
    • Released in March 2009 (Open Source, BSD licensed)
    • VMWare hired Salvatore in March, 2010
    • Then Pieter Noordhuis (key contributor) was hired
  • 7. What is Redis ? Its between lot of stuff, so difficult to categorize it precisely
  • 8. What is Redis ?
    • I see Redis definitely more as a flexible tool that as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well
    • -Salvatore Sanfilippo
    Picture by herzogbr http://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
  • 9. Redis is... Remote Data Structure Server
  • 10. Redis is...
    • Supports rich data types of computer science
      • - Strings, Lists, Sets, Sorted Sets, Hashes...
    • Rich sets of primitives (commands) to manipulate these types
    • Predictive complexity measurements
  • 11. A few fundamentals
    • Written in C (no external dependency)
    • Uses memory as main storage
    • Uses disk for persistence
    • Single Threaded
    • Every command performs atomic execution
  • 12. Performance
    • Screamingly fast performance
    • ~50K read/write operations per seconds
    • 100K+ read/write ops per second on a regular EC2 instance
  • 13. Installation
    • $ git clone http://github.com/antirez/redis
    • $ cd redis
    • $ make
    • $ ./src/redis-server
    • ..........
    • ..........
    • $./src/redis-cli
    • Redis> PING
    • PONG
  • 14. Python Libraries
      • Redis-py - Python client library
      • Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
      • Redisco - ORM for redis along the lines Ohm for Ruby
  • 15. redis-py
    • The most popular python client library
    • Andy McCurdy ( [email_address] )
    • Github: http://github.com/andymccurdy/redis-py
      • $easy_install redis OR
      • $pip install redis
    • Optional
      • $easy_install hiredis
      • $pip install hiredis
  • 16. Lets get started...
    • $ cd redis
    • $ ./src/redis-server
    • .........
    • >>> from redis import Redis
    • >>> redis_client = Redis()
    • >>> redis_client.keys()
    • >>> help(redis_client)
  • 17. Redis Keys
    • Not binary safe.
    • Should not contain space or newline character
    • A few rules about keys:
      • Too long keys are not a good idea
      • Too short keys is also not a good idea
      • “ object-type:id:field” can be a nice idea, i.e. “user:1001:name”
  • 18. Operations on Keys
    • KEYS
    • EXISTS
    • DEL
    • EXPIRE
    • OBJECT
    • PERSIST
    • RANDOMKEY
    • RENAME
    • TYPE
    • TTL
    • EXPIREAT
    • MOVE
  • 19. Lets play with keys
    • >>>redis_client.keys()
    • >>>redis_client.exists('key')
    • >>>redis_client.delete('key')
    • >>>redis_client.type('key')
    • >>>......
    • >>>......
  • 20. Data Structures
    • Strings
    • Lists
    • Sets
    • Sorted Sets
    • Hashes
  • 21. Strings
    • SET
    • GET
    • MSET
    • MGET
    • SETEX
    • GETSET
    • SETNX
    • INCR
    • INCRBY
    • DECR
    • DECRBY
  • 22. Strings – with redis client
    • >>> redis_client.set('key', 'value')
    • >>> redis_client.get('key')
    • >>> redis_client.delete('key')
  • 23. Fetch multiple keys at once
    • mget/mset
    • redis_client.mset({'key1': 'val1', 'key2': 'val2'})
    • redis_client.mget('key1', 'key2',......)
  • 24. Expiration
    • Set a value with expire
    • >>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
    • >>>redis_client.expire('key', 2)
    • >>>redis_client.get('key')
    • >>>None
  • 25. Expire Semantics
    • Key with expiry known as volatile keys
    • Whenever a volatile key is modified, its expiry is reset
      • - To avoid inconsistency in following cases
        • Replication
        • Append Only Log
  • 26. Uses To store transient states in your web application
  • 27. Uses Who is online?
  • 28. Uses Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
  • 29. Atomic Increments
    • >>>help(redis_client.incr)
    • >>>help(redis_client.decr)
    • >>>
    • >>> redis_client.incr('counter', 1)
    • >>>1
    • >>> redis_client.incr('counter')
    • >>>2
    • >>> redis_client.incr('counter')
    • >>>3
  • 30. Uses High Speed counters (views/clicks/votes/likes..)
  • 31. Uses API Rate Limiting
  • 32. Uses Generating unique IDs
  • 33. Lists
    • Ordered list of binarysafe strings
    • Doubly linked list
    • Memory footprint optimized for smaller list
    • O(1) insertion/deletion at both ends
  • 34. Lists - operations
    • LPUSH
    • RPUSH
    • LSET
    • LRANGE
    • LPOP
    • BLPOP
    • BRPOP
    • BRPOPLPUSH
    • LINSERT
    • RPOP
    • RPOPLPUSH
    • LPUSHX
    • RPUSHX
  • 35. Uses Web apps are full of lists :)
  • 36. Uses Capped List
  • 37. Uses Real time message Queue Background Worker queues (Resque, Celery) https://github.com/blog/542-introducing-resque
  • 38. Uses Social Activity Streams or notifications
  • 39. Sets An unordered collection of distinct byte strings Nothing different from the data type in python
  • 40. Sets - Operations
    • SADD
    • SCARD
    • SREM
    • SISMEMBER
    • SMEMBERS
    • SPOP
    • SRANDMEMBER
  • 41. Operations between Sets
    • SMOVE
    • SUNION
    • SDIFF
    • SINTER
  • 42. Sets - Operations
    • SDIFFSTORE
    • SINTERSTORE
    • SUNIONSTORE
  • 43. Sets - Uses
    • Picking random items from a set using SRANDMEMBER
    • Ex.
      • Picking a random article from daily news
      • Pick a random Ad for serving
      • Pick a random option for A/B
    Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
  • 44. Sets - Uses
    • To model Relations in social graph
    • Ex.
      • >>>redis_client.sadd('friends:john', 'jenny', 'maria')
      • >>>redis_client.sadd('friends:ben', 'maria', 'kate')
      • >>>redis_client.sinter('friends:john', 'friends:ben')
  • 45. Relations (friend/followers)
    • Common followlist for A and B
      • >>> sinter('users:A:follows', 'users:B:follows')
    • Unique to B compared to C
      • >>>sdiff('users:B:follows', 'users:C:follows')
    • Mutual relationship (friend as well as follower)
      • >>>sinter('users:B:followers', 'users:B:friends')
    • Who does not follow me back ?
      • >>>sdiff('users:B:friends', 'users:B:followers')
    • Who am I not following back?
      • >>>sdiff('users:B:followers', 'user:B:friends')
  • 46. Which of my friend's are online right now? http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/ SINTER online_people my_friends
  • 47. Which of my friends are online right now?
    • >>>RENAME online:fresh online:stale #every minute
    • >>>SUNIONSTORE online:fresh online:stale
    • >>>#friend jack connects
    • >>>SADD online:fresh 'jack'
    • >>>SUNIONSTORE online online:fresh online:stale
    • >>>#who is online ?
    • >>>SINTER online friends
  • 48. Sorted Sets Ordered sets on the basis of score
  • 49. Sorted Sets
    • ZADD
    • ZCARD
    • ZCOUNT
    • ZINCRBY
    • ZINTERSTORE
    • ZRANGE
    • ZRANGEBYSCORE
    • ZRANK
    • ZREM
    • ZREMRANGEBYRANK
    • ZREMRANGEBYSCORE
    • ZREVRANGE
    • ZREVRANGEBYSCORE
    • ZSCORE
    • ZUNIONSTORE
  • 50. Sorted Sets – Use Case To build index on your dataset
  • 51. Uses - Realtime Leaderboards
    • >>>zincrby('leaderboard', 'john', 2)
    • >>>zincrby('leaderboard', 'jack', 5)
    • >>>zincrby('leaderboard', 'kate', 1)
    • >>>zincrby('leaderboard', 'kate', 10)
    • ....
    • >>>zrange('leaderboard', 0, -1, withscores = True)
  • 52. Uses - Realtime Leaderboards
    • Most download resources
    • Most popular articles on the website
    • Weekly popular list
    • Most downloaded stuff between date1 and date2
    • .....
    • .....
  • 53. AutoComplete with Sored Sets http://antirez.com/post/autocomplete-with-redis.html
  • 54. Hashes
    • Equivalent to Python dictionary or Ruby hash or Java hashmap
    • Operations
      • hmset users:1 {'username': 'jim', 'score': 23}
      • hgetall users:1
      • Hincrby
    • Useful for storing structured data
      • hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
      • Expire user:1:preferences <duration>
      • Hmgetall user:1:preferences #returns entire dict
  • 55. Redis Transactions
    • Using Multi/Watch
    • >>>p = redis_client.pipeline()
    • >>>p.lpush('a', 1)
    • >>>p.ltrim('a', 0, 100)
    • >>>p.execute
    • Limitation
      • Since commands are queued, Can't read values
      • No conditional execution
      • If not high write contention scenario, WATCH can be used
    • Use Redis Scripting to write your own primitive
  • 56. Designing with Redis
    • Data layout design on basis of Query
    • No Query Optimizer
    • Manually build indexes
  • 57. Durability
    • Snapshotting mode
      • Binary dump every x secs or y ops
    • Append Only File (AOF)
      • Every command is written to a file
      • On restart/crash, commands replayed
      • Fsync on every new command
      • Fsync every second
      • OS decide to
    • Replication
  • 58. Publish/Subscribe
    • A simple and efficient implementation of publish/subscribe messaging paradigm
    • Client can subscribe/psubscribe to receive messages on channels (key patterns)
  • 59. Publish/Subscribe
    • PSUBSCRIBE
    • PUBLISH
    • PUNSUBSCRIBE
    • SUBSCRIBE
    • UNSUBSCRIBE
  • 60. Uses Many to Many message passing
  • 61. Uses Web Chat
  • 62. References
    • Redis Website ( http://redis.io )
    • SimonWillison Tutorial ( http://simonwillison.net/static/2010/redis-tutorial/ )
    • Redis from ground up ( http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19 )
    • Redis under the hood ( http://pauladamsmith.com/articles/redis-under-the-hood.html )
    • Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)
  • 63. References
    • Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
    • Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
    • Redis at Bump
    • http://devblog.bu.mp/how-we-use-redis-at-bump
    • Redis: AK 47 of postrelational database
    • http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases
  • 64. Questions
  • 65. Thanks Contact details Twitter: @_sunil_ Email: arora.sunil AT gmail.com