Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kicking ass with redis


Published on

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here
  • Shameless plug: just yesterday, I published a Ruby gem for bit-wise operations over sparse bitmaps in Redis using natural syntax that lets you store millions of users without wasting precious memory:

    Memory consumption depends on how the set bits are spread out but for 1 bit it uses just around 20kb, for 8 bits from 30kb to 160kb (using the default bitmap chunk size).

    users[128_000_000] = true

    Uses 20kb as opposed to 122MB.

    Btw. syntax is also pretty nice:

    result = (active_users & premium_users & (google_search | ~google_adwords)

    (The above will be optimized down to just three BITOPs).
    Are you sure you want to  Yes  No
    Your message goes here
  • @dvirsky Thank you for your reply! What is the best practice if you want to sort by strings? For example, want to list your users in alphabetical order?
    Are you sure you want to  Yes  No
    Your message goes here
  • @SzabolcsCsermk If you use a ZSET to index strings, you can't do range queries on it, because the hash values are not lexically ordered as the original strings.

    but if you use a ZSET to index numeric value, say a user's age. So your ZSET looks like {userId=>age, userId=>age, ...} you can use ZRANGE to get the the top N youngest users, or ZRANGEBYSCORE to get all users within an age group.
    Are you sure you want to  Yes  No
    Your message goes here
  • What do you mean about 'Ranges with ZRANGE, ZRANGEBYSCORE on numeric values only'?

    Could you show an example?
    Are you sure you want to  Yes  No
    Your message goes here

Kicking ass with redis

  1. 1. Kicking Ass WithRedis for real world problemsDvir Volk, Chief Architect, (@dvirsky)
  2. 2. O HAI! I CAN HAS REDIS?Extremely Quick introduction to Redis● Key => Data Structure server● In memory, with persistence● Extremely fast and versatile● Rapidly growing (, Craigslist, Youporn ....)● Open Source, awesome community● Used as the primary data source in ○ Relational Data ○ Queueing ○ Caching ○ Machine Learning ○ Text Processing and search ○ Geo Stuff
  3. 3. Key => { Data Structures } "Im a Plain Text String!" Strings/Blobs/Bitmaps Key1 Val1 Hash Tables (objects!) Key2 Val 2 Key C B B A C Linked Lists A B C D Sets Sorted Sets A: 0.1 B: 0.3 C: 500 D: 500
  4. 4. Redis is like Lego for Data● Yes, It can be used as a simple KV store.● But to really Use it, you need to think of it as a tool set.● You have a nail - redis is a hammer building toolkit.● That can make almost any kind of hammer.● Learning how to efficiently model your problem is the Zen of Redis.● Here are a few examples...
  5. 5. Pattern 1: Simple, Fast, Object StoreOur problem:● Very fast object store that scales up well.● High write throughput.● Atomic manipulation of object members.Possible use cases:● Online user data (session, game state)● Social Feed● Shopping Cart● Anything, really...
  6. 6. Storing users as HASHes email name John users:1 Password aebc65feae8b id 1 email name Jane users:2 Password aebc65ab117b id 2
  7. 7. Redis Pattern 1● Each object is saved as a HASH.● Hash objects are { key=> string/number }● No JSON & friends serialization overhead.● Complex members and relations are stored as separate HASHes.● Atomic set / increment / getset members.● Use INCR for centralized incremental ids.● Load objects with HGETALL / HMGET
  8. 8. Objects as Hashesclass User(RedisObject): > INCR users:id def __init__(email, name, password): (integer) 1 = email = name > HMSET "users:1" self.password = password "email" "" = self.createId() "name" "John" "password" "1234"user = User(, John, OK1234) > HGETALL "users:1" { "email": "", ... }
  9. 9. Performance with growing data
  10. 10. Pattern 2: Object IndexingThe problem:● We want to index the objects we saved by various criteria.● We want to rank and sort them quickly.● We want to be able to update an index quickly.Use cases:● Tagging● Real-Time score tables● Social Feed Views
  11. 11. Indexing with Sorted Sets k:users:email email user:1 => 1789708973 name Johnusers:1 score 300 user:2 => 2361572523 id 1 .... email k:users:score name Jane user:2 => 250users:2 score 250 user:1 => 300 id 2 user:3 => 300
  12. 12. Redis Pattern● Indexes are sorted sets (ZSETs)● Access by value O(1), by score O(log(N)). plus ranges.● Sorted Sets map { value => score (double) }● So we map { objectId => score }● For numerical members, the value is the score● For string members, the score is a hash of the string.● Fetching is done with ZRANGEBYSCORE● Ranges with ZRANGE / ZRANGEBYSCORE on numeric values only (or very short strings)● Deleting is done with ZREM● Intersecting keys is possible with ZINTERSTORE● Each class objects have a special sorted set for ids.
  13. 13. Automatic Keys for objectsclass User(RedisObject): > ZADD k:users:email 238927659283691 "1" 1 _keySpec = KeySpec( UnorderedKey(email), > ZADD k:users:name 9283498696113 "1" UnorderedKey(name), 1 OrderedNumericalKey(points) > ZADD k:users:points 300 "1" ) .... 1 > ZREVRANGE k:users:points 0 20 withscores#creating the users - now with points 1) "1"user = User(, John, 2) "300"1234, points = 300) > ZRANGEBYSCORE k:users:email 238927659283691 238927659283691#saving auto-indexes 1) "1" redis> HGETALL users:1 { .. }#range query on rankusers = User.getByRank(0,20)#get by nameusers = User.get(name = John)
  14. 14. Pattern 3: Unique Value CounterThe problem:● We want an efficient way to measure cardinality of a set of objects over time.● We may want it in real time.● We dont want huge overhead.Use Cases:● Daily/Monthly Unique users● Split by OS / country / whatever● Real Time online users counter
  15. 15. Bitmaps to the rescue
  16. 16. Redis Pattern● Redis strings can be treated as bitmaps.● We keep a bitmap for each time slot.● We use BITSET offset=<object id>● the size of a bitmap is max_id/8 bytes● Cardinality per slot with BITCOUNT (2.6)● Fast bitwise operations - OR / AND / XOR between time slots with BITOP● Aggregate and save results periodically.● Requires sequential object ids - or mapping of (see incremental ids)
  17. 17. Counter API (with redis internals)counter = BitmapCounter(uniques, timeResolutions=(RES_DAY,))#sampling current userscounter.add(userId)> BITSET uniques:day:1339891200 <userId> 1#Getting the unique user count for todaycounter.getCount(time.time())> BITCOUNT uniques:day:1339891200 #Getting the the weekly unique users in the past weektimePoints = [now() - 86400*i for i in xrange(7, 0, -1)]counter.aggregateCounts(timePoints, counter.OP_TOTAL)> BITOP OR tmp_key uniques:day:1339891200 uniques:day:1339804800 ....> BITCOUNT tmp_key   
  18. 18. Pattern 4: Geo resolvingThe Problem:● Resolve lat,lon to real locations● Find locations of a certain class (restaurants) near me● IP2Location searchUse Cases:● Find a users City, ZIP code, Country, etc.● Find the users location by IP
  19. 19. A bit about geohashing● Converts (lat,lon) into a single 64 bit hash (and back)● The closer points are, their common prefix is generally bigger.● Trimming more lower bits describes a larger bounding box.● example: ○ Tel Aviv (32.0667, 34.7667) => 14326455945304181035 ○ Netanya (32.3336, 34.8578) => 14326502174498709381● We can use geohash as scores in sorted sets.● There are drawbacks such as special cases near lat/lon 0.
  20. 20. Redis Pattern● Lets index cities in a sorted set: ○ { cityId => geohash(lat,lon) }● We convert the users {lat,lon} into a goehash too.● Using ZRANGEBYSCORE we find the N larger and N smaller elements in the set: ○ ZRANGEBYSCORE <user_hash> +inf 0 8 ○ ZREVRANGEBYSCORE <user_hash> -inf 0 8● We use the scores as lat,lons again to find distance.● We find the closest city, and load it.● We can save bounding rects for more precision.● The same can be done for ZIP codes, venues, etc.● IP Ranges are indexed on a sorted set, too.
  21. 21. Other interesting use cases● Distributed Queue ○ Workers use blocking pop (BLPOP) on a list. ○ Whenever someone pushes a task to the list (RPUSH) it will be popped by exactly one worker.● Push notifications / IM ○ Use redis PubSub objects as messaging channels between users. ○ Combine with WebSocket to push messages to Web Browsers, a-la googletalk.● Machine learning ○ Use redis sorted sets as a fast storage for feature vectors, frequency counts, probabilities, etc. ○ Intersecting sorted sets can yield SUM(scores) - think log(P(a)) + log (P(b))
  22. 22. Get the sourcesImplementations of most of the examples in thisslideshow: resolving library: redis at