• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Kicking ass with redis
 

Kicking ass with redis

on

  • 23,179 views

 

Statistics

Views

Total Views
23,179
Views on SlideShare
21,468
Embed Views
1,711

Actions

Likes
62
Downloads
264
Comments
4

21 Embeds 1,711

http://blog.nosqlfan.com 927
http://www.scoop.it 267
http://feed.feedsky.com 185
http://max-makarochkin.blogspot.ru 176
http://xianguo.com 48
http://max-makarochkin.blogspot.com 44
http://www.imarks.me 24
http://reader.youdao.com 11
https://twitter.com 10
http://www.zhuaxia.com 3
http://max-makarochkin.blogspot.co.il 2
http://max-makarochkin.blogspot.ca 2
http://www.uplook.cn 2
http://us-w1.rockmelt.com 2
http://reader.googleusercontent.com 2
http://xue.uplook.cn 1
http://max-makarochkin.blogspot.jp 1
http://max-makarochkin.blogspot.it 1
http://translate.googleusercontent.com 1
http://www.steampdf.com 1
http://max-makarochkin.blogspot.tw 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

14 of 4 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Shameless plug: just yesterday, I published a Ruby gem for bit-wise operations over sparse bitmaps in Redis using natural syntax that lets you store millions of users without wasting precious memory:

    Memory consumption depends on how the set bits are spread out but for 1 bit it uses just around 20kb, for 8 bits from 30kb to 160kb (using the default bitmap chunk size).

    users[128_000_000] = true

    Uses 20kb as opposed to 122MB.

    Btw. syntax is also pretty nice:

    result = (active_users & premium_users & (google_search | ~google_adwords)

    (The above will be optimized down to just three BITOPs).

    http://github.com/bilus/redis-bitops
    Are you sure you want to
    Your message goes here
    Processing…
  • @dvirsky Thank you for your reply! What is the best practice if you want to sort by strings? For example, want to list your users in alphabetical order?
    Are you sure you want to
    Your message goes here
    Processing…
  • @SzabolcsCsermk If you use a ZSET to index strings, you can't do range queries on it, because the hash values are not lexically ordered as the original strings.

    but if you use a ZSET to index numeric value, say a user's age. So your ZSET looks like {userId=>age, userId=>age, ...} you can use ZRANGE to get the the top N youngest users, or ZRANGEBYSCORE to get all users within an age group.
    Are you sure you want to
    Your message goes here
    Processing…
  • What do you mean about 'Ranges with ZRANGE, ZRANGEBYSCORE on numeric values only'?

    Could you show an example?
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Kicking ass with redis Kicking ass with redis Presentation Transcript

    • Kicking Ass WithRedis for real world problemsDvir Volk, Chief Architect, Everything.me (@dvirsky)
    • O HAI! I CAN HAS REDIS?Extremely Quick introduction to Redis● Key => Data Structure server● In memory, with persistence● Extremely fast and versatile● Rapidly growing (Instagr.am, Craigslist, Youporn ....)● Open Source, awesome community● Used as the primary data source in Everything.me: ○ Relational Data ○ Queueing ○ Caching ○ Machine Learning ○ Text Processing and search ○ Geo Stuff
    • Key => { Data Structures } "Im a Plain Text String!" Strings/Blobs/Bitmaps Key1 Val1 Hash Tables (objects!) Key2 Val 2 Key C B B A C Linked Lists A B C D Sets Sorted Sets A: 0.1 B: 0.3 C: 500 D: 500
    • Redis is like Lego for Data● Yes, It can be used as a simple KV store.● But to really Use it, you need to think of it as a tool set.● You have a nail - redis is a hammer building toolkit.● That can make almost any kind of hammer.● Learning how to efficiently model your problem is the Zen of Redis.● Here are a few examples...
    • Pattern 1: Simple, Fast, Object StoreOur problem:● Very fast object store that scales up well.● High write throughput.● Atomic manipulation of object members.Possible use cases:● Online user data (session, game state)● Social Feed● Shopping Cart● Anything, really...
    • Storing users as HASHes email john@domain.com name John users:1 Password aebc65feae8b id 1 email Jane@domain.com name Jane users:2 Password aebc65ab117b id 2
    • Redis Pattern 1● Each object is saved as a HASH.● Hash objects are { key=> string/number }● No JSON & friends serialization overhead.● Complex members and relations are stored as separate HASHes.● Atomic set / increment / getset members.● Use INCR for centralized incremental ids.● Load objects with HGETALL / HMGET
    • Objects as Hashesclass User(RedisObject): > INCR users:id def __init__(email, name, password): (integer) 1 self.email = email self.name = name > HMSET "users:1" self.password = password "email" "user@domain.com" self.id = self.createId() "name" "John" "password" "1234"user = User(user@domain.com, John, OK1234) > HGETALL "users:1" { "email": "user@domain.com", ... }user.save()
    • Performance with growing data
    • Pattern 2: Object IndexingThe problem:● We want to index the objects we saved by various criteria.● We want to rank and sort them quickly.● We want to be able to update an index quickly.Use cases:● Tagging● Real-Time score tables● Social Feed Views
    • Indexing with Sorted Sets k:users:email email john@domain.com user:1 => 1789708973 name Johnusers:1 score 300 user:2 => 2361572523 id 1 .... email Jane@domain.com k:users:score name Jane user:2 => 250users:2 score 250 user:1 => 300 id 2 user:3 => 300
    • Redis Pattern● Indexes are sorted sets (ZSETs)● Access by value O(1), by score O(log(N)). plus ranges.● Sorted Sets map { value => score (double) }● So we map { objectId => score }● For numerical members, the value is the score● For string members, the score is a hash of the string.● Fetching is done with ZRANGEBYSCORE● Ranges with ZRANGE / ZRANGEBYSCORE on numeric values only (or very short strings)● Deleting is done with ZREM● Intersecting keys is possible with ZINTERSTORE● Each class objects have a special sorted set for ids.
    • Automatic Keys for objectsclass User(RedisObject): > ZADD k:users:email 238927659283691 "1" 1 _keySpec = KeySpec( UnorderedKey(email), > ZADD k:users:name 9283498696113 "1" UnorderedKey(name), 1 OrderedNumericalKey(points) > ZADD k:users:points 300 "1" ) .... 1 > ZREVRANGE k:users:points 0 20 withscores#creating the users - now with points 1) "1"user = User(user@domain.com, John, 2) "300"1234, points = 300) > ZRANGEBYSCORE k:users:email 238927659283691 238927659283691#saving auto-indexes 1) "1"user.save() redis 127.0.0.1:6379> HGETALL users:1 { .. }#range query on rankusers = User.getByRank(0,20)#get by nameusers = User.get(name = John)
    • Pattern 3: Unique Value CounterThe problem:● We want an efficient way to measure cardinality of a set of objects over time.● We may want it in real time.● We dont want huge overhead.Use Cases:● Daily/Monthly Unique users● Split by OS / country / whatever● Real Time online users counter
    • Bitmaps to the rescue
    • Redis Pattern● Redis strings can be treated as bitmaps.● We keep a bitmap for each time slot.● We use BITSET offset=<object id>● the size of a bitmap is max_id/8 bytes● Cardinality per slot with BITCOUNT (2.6)● Fast bitwise operations - OR / AND / XOR between time slots with BITOP● Aggregate and save results periodically.● Requires sequential object ids - or mapping of (see incremental ids)
    • Counter API (with redis internals)counter = BitmapCounter(uniques, timeResolutions=(RES_DAY,))#sampling current userscounter.add(userId)> BITSET uniques:day:1339891200 <userId> 1#Getting the unique user count for todaycounter.getCount(time.time())> BITCOUNT uniques:day:1339891200 #Getting the the weekly unique users in the past weektimePoints = [now() - 86400*i for i in xrange(7, 0, -1)]counter.aggregateCounts(timePoints, counter.OP_TOTAL)> BITOP OR tmp_key uniques:day:1339891200 uniques:day:1339804800 ....> BITCOUNT tmp_key   
    • Pattern 4: Geo resolvingThe Problem:● Resolve lat,lon to real locations● Find locations of a certain class (restaurants) near me● IP2Location searchUse Cases:● Find a users City, ZIP code, Country, etc.● Find the users location by IP
    • A bit about geohashing● Converts (lat,lon) into a single 64 bit hash (and back)● The closer points are, their common prefix is generally bigger.● Trimming more lower bits describes a larger bounding box.● example: ○ Tel Aviv (32.0667, 34.7667) => 14326455945304181035 ○ Netanya (32.3336, 34.8578) => 14326502174498709381● We can use geohash as scores in sorted sets.● There are drawbacks such as special cases near lat/lon 0.
    • Redis Pattern● Lets index cities in a sorted set: ○ { cityId => geohash(lat,lon) }● We convert the users {lat,lon} into a goehash too.● Using ZRANGEBYSCORE we find the N larger and N smaller elements in the set: ○ ZRANGEBYSCORE <user_hash> +inf 0 8 ○ ZREVRANGEBYSCORE <user_hash> -inf 0 8● We use the scores as lat,lons again to find distance.● We find the closest city, and load it.● We can save bounding rects for more precision.● The same can be done for ZIP codes, venues, etc.● IP Ranges are indexed on a sorted set, too.
    • Other interesting use cases● Distributed Queue ○ Workers use blocking pop (BLPOP) on a list. ○ Whenever someone pushes a task to the list (RPUSH) it will be popped by exactly one worker.● Push notifications / IM ○ Use redis PubSub objects as messaging channels between users. ○ Combine with WebSocket to push messages to Web Browsers, a-la googletalk.● Machine learning ○ Use redis sorted sets as a fast storage for feature vectors, frequency counts, probabilities, etc. ○ Intersecting sorted sets can yield SUM(scores) - think log(P(a)) + log (P(b))
    • Get the sourcesImplementations of most of the examples in thisslideshow:https://github.com/EverythingMe/kickass-redisGeo resolving library:http://github.com/doat/geodisGet redis at http://redis.io