• Like
Redis in Practice
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Redis in Practice

  • 23,154 views
Published

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn …

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Nice work!
    Are you sure you want to
    Your message goes here
  • Nice Work!
    Are you sure you want to
    Your message goes here
  • Redis in Practice
    Are you sure you want to
    Your message goes here
  • thks
    Are you sure you want to
    Your message goes here
  • Thanks Luke, nice work!!
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
23,154
On SlideShare
0
From Embeds
0
Number of Embeds
11

Actions

Shares
Downloads
956
Comments
5
Likes
115

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
  • Another 1500 LOC for specs\n
  • \n
  • \n

Transcript

  • 1. Redis in PracticeNoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
  • 2. About Noah & LukeWeplay.comRubyistsRunning Redis in production since mid-2009@noahd1 and @lukemelia on Twitter
  • 3. Pronouncing Redis
  • 4. Tonight’s gameplanFly-by intro to RedisRedis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
  • 5. Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy.Recently hired by VMWare. Great project leader.
  • 6. Describing RedisRemote dictionary server“advanced, fast, persistent key- value database”“Data structures server”“memcached on steroids”
  • 7. In Salvatore’s words “I see Redis definitely more as a flexible toolthan as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
  • 8. QualitiesWritten in CFew dependenciesFastSingle-threadedLots of clients availableInitial release was March 2009
  • 9. FeaturesKey-value store where a value is one of: scalar/string, list, set, sorted sets, hashPersistence: in-memory, snapshots, append-only log, VMReplication: master-slave, configureablePub-SubExpiry“Transaction”-yIn development: Redis Cluster
  • 10. What Redis ain’t (...yet)Big dataSeamless scalingAd-hoc query toolThe only data store you’ll ever need
  • 11. Who’s using Redis?VMWare GroovesharkGithub SuperfeedrCraigslist RavelryEngineYard mediaFAILThe Guardian WeplayForrst More...PostRank
  • 12. Our infrastructureApp server App server Utility server MySQL master Redis master MySQL slave Redis slave
  • 13. Redis in Practice #1View Counts
  • 14. View countspotential for massive amounts of simple writes/readsvaluable data, but not mission criticallots of row contention in SQL
  • 15. Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
  • 16. Redis in Practice #2Distributed Locks
  • 17. Subscribe to a Weplay Calendar Subscribe to “ics”
  • 18. Subscription challengesGenerally static content for long periods of time, but withshort sessions of frequent updatesExpensive to compute on the fly
  • 19. Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
  • 20. Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
  • 21. Redis: Obtained LockADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
  • 22. Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
  • 23. Redis in Practice #3 Presence
  • 24. “Who’s online?”Weplay members told usthey wanted to be ableto see which of theirfriends were online...
  • 25. About sets0 to N elements Adding a value to a set does not require youUnordered to check if the valueNo repeated members exists in the set first
  • 26. Working with Redis sets 1/3# SADD key, member# Adds the specified member to the set stored at keyredis = Redis.newredis.sadd my_set, foo # => trueredis.sadd my_set, bar # => trueredis.sadd my_set, bar # => falseredis.smembers my_set # => ["foo", "bar"]
  • 27. Working with Redis sets 2/3# SUNION key1 key2 ... keyN# Returns the members of a set resulting from the union of all# the sets stored at the specified keys.# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b># Works like SUNION but instead of being returned the resulting# set is stored as dstkey.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sunion set_a, set_b # => ["foo", "baz", "bar"]redis.sunionstore set_ab, set_a, set_bredis.smembers set_ab # => ["foo", "baz", "bar"]
  • 28. Working with Redis sets 3/3# SINTER key1 key2 ... keyN# Returns the members that are present in all# sets stored at the specified keys.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sinter set_a, set_b # => ["bar"]
  • 29. Approach 1/2
  • 30. Approach 2/2
  • 31. Implementation 1/2# Defining the keysdef current_key  key(Time.now.strftime("%M"))enddef keys_in_last_5_minutes  now = Time.now  times = (0..5).collect {|n| now - n.minutes }  times.collect{ |t| key(t.strftime("%M")) }enddef key(minute)  "online_users_minute_#{minute}"end
  • 32. Implementation 2/2# Tracking an Active User, and calculating who’s onlinedef track_user_id(id)  key = current_key  redis.sadd(key, id)enddef online_user_ids  redis.sunion(*keys_in_last_5_minutes)enddef online_friend_ids(interested_user_id)  redis.sunionstore("online_users", *keys_in_last_5_minutes)  redis.sinter("online_users", "user:#{interested_user_id}:friend_ids")end
  • 33. Redis in Practice #4Social Activity Feeds
  • 34. Social Activity Feeds
  • 35. Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
  • 36. Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
  • 37. Large SQL tableGrew quicklyDifficult to maintainDifficult to prune
  • 38. Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
  • 39. Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
  • 40. via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH user:11:feed, 100 Key Value Friend: 11 LTRIM user:11:feed, 0, 50Activity: 100 user:11:feed [100,99,97,96] LPUSH user:22:feed, 100 Actor 1 Friend: 22 LTRIM user:22:feed, 0, 50 user:22:feed [100,99,98] LPUSH user:33:feed, 100 Teammate: 33 LTRIM user:33:feed, 0, 100 user:33:feed [100,99]
  • 41. via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
  • 42. Rendering the Feed
  • 43. Redis in Practice #5Friend Suggestions
  • 44. Friend Suggestionssuggest new connections based onexisting connectionsexpanding the social graph andmirroring real world connections is key
  • 45. The Concept PaulMe George John Ringo
  • 46. The Yoko Factor PaulMe George John Yoko Ringo
  • 47. The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
  • 48. The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
  • 49. ZSETS in Redis“Sorted Sets”Each member of the set has a scoreOrdered by the score at all times
  • 50. Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “donny”zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Mezincrby “user:1:suggestions” 1 “erika” Ted Erikazincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
  • 51. Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Tedzrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
  • 52. Redis in Practice #6 Caching
  • 53. Suitability for cachingExcellent match for managed denormalization ex. friendships, teams, teammatesExcellent match where you would benefit from persistenceand/or replicationHistorically, not a good match for a “generational cache,” inwhich you want to optimize memory use by evicting least-recently used (LRU) keys
  • 54. As an LRU cacheTTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTLRedis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
  • 55. Easy to adaptNamespaced Rack::Session, Rack::Cache, I18n and cacheRedis cache stores for Ruby web frameworks implementation is under 1,000 LOC
  • 56. Contentious benchmarks
  • 57. Anyquestions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com