• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Redis in Practice
 

Redis in Practice

on

  • 21,543 views

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn ...

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

Statistics

Views

Total Views
21,543
Views on SlideShare
14,208
Embed Views
7,335

Actions

Likes
109
Downloads
849
Comments
5

32 Embeds 7,335

http://www.lukemelia.com 3455
http://www.cnblogs.com 1763
http://localhost 1523
http://archive.cnblogs.com 319
http://asyncionews.com 132
http://www.fishxl.com 28
http://www.note4u.net 26
http://kimelia.com 15
http://blog.0898888.com 13
http://www.kimelia.com 10
http://cache.baiducontent.com 9
http://www.shaoqun.com 7
http://thinkery.me 5
http://cache.baidu.com 3
http://www.pega.vn 3
http://reader.youdao.com 3
http://pinterest.com 2
https://twimg0-a.akamaihd.net 2
http://www.linkedin.com 2
http://www.docshut.com 2
http://a0.twimg.com 2
http://translate.googleusercontent.com 1
http://m.baidu.com 1
http://www.slashdocs.com 1
http://newsblur.com 1
http://www.techgig.com 1
http://paper.li 1
http://j-reference.blogspot.fr 1
https://si0.twimg.com 1
http://mail.lukemelia.com 1
http://j-reference.blogspot.in 1
http://www.haogongju.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 5 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice work!
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice Work!
    Are you sure you want to
    Your message goes here
    Processing…
  • Redis in Practice
    Are you sure you want to
    Your message goes here
    Processing…
  • thks
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks Luke, nice work!!
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
  • Another 1500 LOC for specs\n
  • \n
  • \n

Redis in Practice Redis in Practice Presentation Transcript

  • Redis in PracticeNoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
  • About Noah & LukeWeplay.comRubyistsRunning Redis in production since mid-2009@noahd1 and @lukemelia on Twitter
  • Pronouncing Redis
  • Tonight’s gameplanFly-by intro to RedisRedis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
  • Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy.Recently hired by VMWare. Great project leader.
  • Describing RedisRemote dictionary server“advanced, fast, persistent key- value database”“Data structures server”“memcached on steroids”
  • In Salvatore’s words “I see Redis definitely more as a flexible toolthan as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
  • QualitiesWritten in CFew dependenciesFastSingle-threadedLots of clients availableInitial release was March 2009
  • FeaturesKey-value store where a value is one of: scalar/string, list, set, sorted sets, hashPersistence: in-memory, snapshots, append-only log, VMReplication: master-slave, configureablePub-SubExpiry“Transaction”-yIn development: Redis Cluster
  • What Redis ain’t (...yet)Big dataSeamless scalingAd-hoc query toolThe only data store you’ll ever need
  • Who’s using Redis?VMWare GroovesharkGithub SuperfeedrCraigslist RavelryEngineYard mediaFAILThe Guardian WeplayForrst More...PostRank
  • Our infrastructureApp server App server Utility server MySQL master Redis master MySQL slave Redis slave
  • Redis in Practice #1View Counts
  • View countspotential for massive amounts of simple writes/readsvaluable data, but not mission criticallots of row contention in SQL
  • Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
  • Redis in Practice #2Distributed Locks
  • Subscribe to a Weplay Calendar Subscribe to “ics”
  • Subscription challengesGenerally static content for long periods of time, but withshort sessions of frequent updatesExpensive to compute on the fly
  • Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
  • Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
  • Redis: Obtained LockADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
  • Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
  • Redis in Practice #3 Presence
  • “Who’s online?”Weplay members told usthey wanted to be ableto see which of theirfriends were online...
  • About sets0 to N elements Adding a value to a set does not require youUnordered to check if the valueNo repeated members exists in the set first
  • Working with Redis sets 1/3# SADD key, member# Adds the specified member to the set stored at keyredis = Redis.newredis.sadd my_set, foo # => trueredis.sadd my_set, bar # => trueredis.sadd my_set, bar # => falseredis.smembers my_set # => ["foo", "bar"]
  • Working with Redis sets 2/3# SUNION key1 key2 ... keyN# Returns the members of a set resulting from the union of all# the sets stored at the specified keys.# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b># Works like SUNION but instead of being returned the resulting# set is stored as dstkey.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sunion set_a, set_b # => ["foo", "baz", "bar"]redis.sunionstore set_ab, set_a, set_bredis.smembers set_ab # => ["foo", "baz", "bar"]
  • Working with Redis sets 3/3# SINTER key1 key2 ... keyN# Returns the members that are present in all# sets stored at the specified keys.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sinter set_a, set_b # => ["bar"]
  • Approach 1/2
  • Approach 2/2
  • Implementation 1/2# Defining the keysdef current_key  key(Time.now.strftime("%M"))enddef keys_in_last_5_minutes  now = Time.now  times = (0..5).collect {|n| now - n.minutes }  times.collect{ |t| key(t.strftime("%M")) }enddef key(minute)  "online_users_minute_#{minute}"end
  • Implementation 2/2# Tracking an Active User, and calculating who’s onlinedef track_user_id(id)  key = current_key  redis.sadd(key, id)enddef online_user_ids  redis.sunion(*keys_in_last_5_minutes)enddef online_friend_ids(interested_user_id)  redis.sunionstore("online_users", *keys_in_last_5_minutes)  redis.sinter("online_users", "user:#{interested_user_id}:friend_ids")end
  • Redis in Practice #4Social Activity Feeds
  • Social Activity Feeds
  • Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
  • Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
  • Large SQL tableGrew quicklyDifficult to maintainDifficult to prune
  • Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
  • Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
  • via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH user:11:feed, 100 Key Value Friend: 11 LTRIM user:11:feed, 0, 50Activity: 100 user:11:feed [100,99,97,96] LPUSH user:22:feed, 100 Actor 1 Friend: 22 LTRIM user:22:feed, 0, 50 user:22:feed [100,99,98] LPUSH user:33:feed, 100 Teammate: 33 LTRIM user:33:feed, 0, 100 user:33:feed [100,99]
  • via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
  • Rendering the Feed
  • Redis in Practice #5Friend Suggestions
  • Friend Suggestionssuggest new connections based onexisting connectionsexpanding the social graph andmirroring real world connections is key
  • The Concept PaulMe George John Ringo
  • The Yoko Factor PaulMe George John Yoko Ringo
  • The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
  • The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
  • ZSETS in Redis“Sorted Sets”Each member of the set has a scoreOrdered by the score at all times
  • Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “donny”zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Mezincrby “user:1:suggestions” 1 “erika” Ted Erikazincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
  • Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Tedzrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
  • Redis in Practice #6 Caching
  • Suitability for cachingExcellent match for managed denormalization ex. friendships, teams, teammatesExcellent match where you would benefit from persistenceand/or replicationHistorically, not a good match for a “generational cache,” inwhich you want to optimize memory use by evicting least-recently used (LRU) keys
  • As an LRU cacheTTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTLRedis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
  • Easy to adaptNamespaced Rack::Session, Rack::Cache, I18n and cacheRedis cache stores for Ruby web frameworks implementation is under 1,000 LOC
  • Contentious benchmarks
  • Anyquestions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com