Successfully reported this slideshow.
Your SlideShare is downloading. ×

Redis in Practice

Ad

Redis in Practice
NoSQL NYC • December 2nd, 2010
    Noah Davis & Luke Melia

Ad

About Noah & Luke

Weplay.com
Rubyists
Running Redis in production since mid-2009
@noahd1 and @lukemelia on Twitter

Ad

Pronouncing Redis

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Upcoming SlideShare
Redis SoCraTes 2014
Redis SoCraTes 2014
Loading in …3
×

Check these out next

1 of 57 Ad
1 of 57 Ad

Redis in Practice

Download to read offline

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

More Related Content

Redis in Practice

  1. 1. Redis in Practice NoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
  2. 2. About Noah & Luke Weplay.com Rubyists Running Redis in production since mid-2009 @noahd1 and @lukemelia on Twitter
  3. 3. Pronouncing Redis
  4. 4. Tonight’s gameplan Fly-by intro to Redis Redis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
  5. 5. Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy. Recently hired by VMWare. Great project leader.
  6. 6. Describing Redis Remote dictionary server “advanced, fast, persistent key- value database” “Data structures server” “memcached on steroids”
  7. 7. In Salvatore’s words “I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
  8. 8. Qualities Written in C Few dependencies Fast Single-threaded Lots of clients available Initial release was March 2009
  9. 9. Features Key-value store where a value is one of: scalar/string, list, set, sorted sets, hash Persistence: in-memory, snapshots, append-only log, VM Replication: master-slave, configureable Pub-Sub Expiry “Transaction”-y In development: Redis Cluster
  10. 10. What Redis ain’t (...yet) Big data Seamless scaling Ad-hoc query tool The only data store you’ll ever need
  11. 11. Who’s using Redis? VMWare Grooveshark Github Superfeedr Craigslist Ravelry EngineYard mediaFAIL The Guardian Weplay Forrst More... PostRank
  12. 12. Our infrastructure App server App server Utility server MySQL master Redis master MySQL slave Redis slave
  13. 13. Redis in Practice #1 View Counts
  14. 14. View counts potential for massive amounts of simple writes/reads valuable data, but not mission critical lots of row contention in SQL
  15. 15. Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
  16. 16. Redis in Practice #2 Distributed Locks
  17. 17. Subscribe to a Weplay Calendar Subscribe to “ics”
  18. 18. Subscription challenges Generally static content for long periods of time, but with short sessions of frequent updates Expensive to compute on the fly
  19. 19. Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
  20. 20. Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
  21. 21. Redis: Obtained Lock ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
  22. 22. Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
  23. 23. Redis in Practice #3 Presence
  24. 24. “Who’s online?” Weplay members told us they wanted to be able to see which of their friends were online...
  25. 25. About sets 0 to N elements Adding a value to a set does not require you Unordered to check if the value No repeated members exists in the set first
  26. 26. Working with Redis sets 1/3 # SADD key, member # Adds the specified member to the set stored at key redis = Redis.new redis.sadd 'my_set', 'foo' # => true redis.sadd 'my_set', 'bar' # => true redis.sadd 'my_set', 'bar' # => false redis.smembers 'my_set' # => ["foo", "bar"]
  27. 27. Working with Redis sets 2/3 # SUNION key1 key2 ... keyN # Returns the members of a set resulting from the union of all # the sets stored at the specified keys. # SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b> # Works like SUNION but instead of being returned the resulting # set is stored as dstkey. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sunion 'set_a', 'set_b' # => ["foo", "baz", "bar"] redis.sunionstore 'set_ab', 'set_a', 'set_b' redis.smembers 'set_ab' # => ["foo", "baz", "bar"]
  28. 28. Working with Redis sets 3/3 # SINTER key1 key2 ... keyN # Returns the members that are present in all # sets stored at the specified keys. redis = Redis.new redis.sadd 'set_a', 'foo' redis.sadd 'set_a', 'bar' redis.sadd 'set_b', 'bar' redis.sadd 'set_b', 'baz' redis.sinter 'set_a', 'set_b' # => ["bar"]
  29. 29. Approach 1/2
  30. 30. Approach 2/2
  31. 31. Implementation 1/2 # Defining the keys def current_key   key(Time.now.strftime("%M")) end def keys_in_last_5_minutes   now = Time.now   times = (0..5).collect {|n| now - n.minutes }   times.collect{ |t| key(t.strftime("%M")) } end def key(minute)   "online_users_minute_#{minute}" end
  32. 32. Implementation 2/2 # Tracking an Active User, and calculating who’s online def track_user_id(id)   key = current_key   redis.sadd(key, id) end def online_user_ids   redis.sunion(*keys_in_last_5_minutes) end def online_friend_ids(interested_user_id)   redis.sunionstore("online_users", *keys_in_last_5_minutes)   redis.sinter("online_users", "user:#{interested_user_id}:friend_ids") end
  33. 33. Redis in Practice #4 Social Activity Feeds
  34. 34. Social Activity Feeds
  35. 35. Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
  36. 36. Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
  37. 37. Large SQL table Grew quickly Difficult to maintain Difficult to prune
  38. 38. Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
  39. 39. Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
  40. 40. via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH 'user:11:feed', '100' Key Value Friend: 11 LTRIM 'user:11:feed', 0, 50 Activity: 100 user:11:feed [100,99,97,96] LPUSH 'user:22:feed', '100' Actor 1 Friend: 22 LTRIM 'user:22:feed', 0, 50 user:22:feed [100,99,98] LPUSH 'user:33:feed', '100' Teammate: 33 LTRIM 'user:33:feed', 0, 100 user:33:feed [100,99]
  41. 41. via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
  42. 42. Rendering the Feed
  43. 43. Redis in Practice #5 Friend Suggestions
  44. 44. Friend Suggestions suggest new connections based on existing connections expanding the social graph and mirroring real world connections is key
  45. 45. The Concept Paul Me George John Ringo
  46. 46. The Yoko Factor Paul Me George John Yoko Ringo
  47. 47. The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
  48. 48. The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
  49. 49. ZSETS in Redis “Sorted Sets” Each member of the set has a score Ordered by the score at all times
  50. 50. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “donny” zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Me zincrby “user:1:suggestions” 1 “erika” Ted Erika zincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
  51. 51. Friend Suggestions in Redis zincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Ted zrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
  52. 52. Redis in Practice #6 Caching
  53. 53. Suitability for caching Excellent match for managed denormalization ex. friendships, teams, teammates Excellent match where you would benefit from persistence and/or replication Historically, not a good match for a “generational cache,” in which you want to optimize memory use by evicting least- recently used (LRU) keys
  54. 54. As an LRU cache TTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTL Redis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
  55. 55. Easy to adapt Namespaced Rack::Session, Rack::Cache, I18n and cache Redis cache stores for Ruby web frameworks implementation is under 1,000 LOC
  56. 56. Contentious benchmarks
  57. 57. Any questions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com

Editor's Notes

  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
  • Another 1500 LOC for specs\n
  • \n
  • \n

×