Your SlideShare is downloading. ×
0
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Redis in Practice
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Redis in Practice

25,075

Published on

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn …

Noah Davis & Luke Melia of Weplay share a series of examples of Redis in the real world. In doing so, they cover a survey of Redis' features, approach, history and philosophy. Most examples are drawn from the Weplay team's experience using Redis to power features on Weplay.com, a social site for youth sports.

Published in: Technology
5 Comments
130 Likes
Statistics
Notes
No Downloads
Views
Total Views
25,075
On Slideshare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
1,065
Comments
5
Likes
130
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • http://antirez.com/m/p.php?i=220\nvolatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used.\nvolatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.\nvolatile-random remove a random key among the ones with an expire set.\nallkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.\nallkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.\n\n
  • Another 1500 LOC for specs\n
  • \n
  • \n
  • Transcript

    • 1. Redis in PracticeNoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia
    • 2. About Noah & LukeWeplay.comRubyistsRunning Redis in production since mid-2009@noahd1 and @lukemelia on Twitter
    • 3. Pronouncing Redis
    • 4. Tonight’s gameplanFly-by intro to RedisRedis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching
    • 5. Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy.Recently hired by VMWare. Great project leader.
    • 6. Describing RedisRemote dictionary server“advanced, fast, persistent key- value database”“Data structures server”“memcached on steroids”
    • 7. In Salvatore’s words “I see Redis definitely more as a flexible toolthan as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo
    • 8. QualitiesWritten in CFew dependenciesFastSingle-threadedLots of clients availableInitial release was March 2009
    • 9. FeaturesKey-value store where a value is one of: scalar/string, list, set, sorted sets, hashPersistence: in-memory, snapshots, append-only log, VMReplication: master-slave, configureablePub-SubExpiry“Transaction”-yIn development: Redis Cluster
    • 10. What Redis ain’t (...yet)Big dataSeamless scalingAd-hoc query toolThe only data store you’ll ever need
    • 11. Who’s using Redis?VMWare GroovesharkGithub SuperfeedrCraigslist RavelryEngineYard mediaFAILThe Guardian WeplayForrst More...PostRank
    • 12. Our infrastructureApp server App server Utility server MySQL master Redis master MySQL slave Redis slave
    • 13. Redis in Practice #1View Counts
    • 14. View countspotential for massive amounts of simple writes/readsvaluable data, but not mission criticallots of row contention in SQL
    • 15. Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6
    • 16. Redis in Practice #2Distributed Locks
    • 17. Subscribe to a Weplay Calendar Subscribe to “ics”
    • 18. Subscription challengesGenerally static content for long periods of time, but withshort sessions of frequent updatesExpensive to compute on the fly
    • 19. Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish
    • 20. Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true
    • 21. Redis: Obtained LockADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"
    • 22. Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0
    • 23. Redis in Practice #3 Presence
    • 24. “Who’s online?”Weplay members told usthey wanted to be ableto see which of theirfriends were online...
    • 25. About sets0 to N elements Adding a value to a set does not require youUnordered to check if the valueNo repeated members exists in the set first
    • 26. Working with Redis sets 1/3# SADD key, member# Adds the specified member to the set stored at keyredis = Redis.newredis.sadd my_set, foo # => trueredis.sadd my_set, bar # => trueredis.sadd my_set, bar # => falseredis.smembers my_set # => ["foo", "bar"]
    • 27. Working with Redis sets 2/3# SUNION key1 key2 ... keyN# Returns the members of a set resulting from the union of all# the sets stored at the specified keys.# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b># Works like SUNION but instead of being returned the resulting# set is stored as dstkey.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sunion set_a, set_b # => ["foo", "baz", "bar"]redis.sunionstore set_ab, set_a, set_bredis.smembers set_ab # => ["foo", "baz", "bar"]
    • 28. Working with Redis sets 3/3# SINTER key1 key2 ... keyN# Returns the members that are present in all# sets stored at the specified keys.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sinter set_a, set_b # => ["bar"]
    • 29. Approach 1/2
    • 30. Approach 2/2
    • 31. Implementation 1/2# Defining the keysdef current_key  key(Time.now.strftime("%M"))enddef keys_in_last_5_minutes  now = Time.now  times = (0..5).collect {|n| now - n.minutes }  times.collect{ |t| key(t.strftime("%M")) }enddef key(minute)  "online_users_minute_#{minute}"end
    • 32. Implementation 2/2# Tracking an Active User, and calculating who’s onlinedef track_user_id(id)  key = current_key  redis.sadd(key, id)enddef online_user_ids  redis.sunion(*keys_in_last_5_minutes)enddef online_friend_ids(interested_user_id)  redis.sunionstore("online_users", *keys_in_last_5_minutes)  redis.sinter("online_users", "user:#{interested_user_id}:friend_ids")end
    • 33. Redis in Practice #4Social Activity Feeds
    • 34. Social Activity Feeds
    • 35. Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data
    • 36. Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100
    • 37. Large SQL tableGrew quicklyDifficult to maintainDifficult to prune
    • 38. Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE
    • 39. Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM
    • 40. via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH user:11:feed, 100 Key Value Friend: 11 LTRIM user:11:feed, 0, 50Activity: 100 user:11:feed [100,99,97,96] LPUSH user:22:feed, 100 Actor 1 Friend: 22 LTRIM user:22:feed, 0, 50 user:22:feed [100,99,98] LPUSH user:33:feed, 100 Teammate: 33 LTRIM user:33:feed, 0, 100 user:33:feed [100,99]
    • 41. via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]
    • 42. Rendering the Feed
    • 43. Redis in Practice #5Friend Suggestions
    • 44. Friend Suggestionssuggest new connections based onexisting connectionsexpanding the social graph andmirroring real world connections is key
    • 45. The Concept PaulMe George John Ringo
    • 46. The Yoko Factor PaulMe George John Yoko Ringo
    • 47. The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika
    • 48. The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue
    • 49. ZSETS in Redis“Sorted Sets”Each member of the set has a scoreOrdered by the score at all times
    • 50. Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “donny”zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Mezincrby “user:1:suggestions” 1 “erika” Ted Erikazincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]
    • 51. Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Tedzrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]
    • 52. Redis in Practice #6 Caching
    • 53. Suitability for cachingExcellent match for managed denormalization ex. friendships, teams, teammatesExcellent match where you would benefit from persistenceand/or replicationHistorically, not a good match for a “generational cache,” inwhich you want to optimize memory use by evicting least-recently used (LRU) keys
    • 54. As an LRU cacheTTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTLRedis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random
    • 55. Easy to adaptNamespaced Rack::Session, Rack::Cache, I18n and cacheRedis cache stores for Ruby web frameworks implementation is under 1,000 LOC
    • 56. Contentious benchmarks
    • 57. Anyquestions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com

    ×