1.
Redis in PracticeNoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia

2.
About Noah & LukeWeplay.comRubyistsRunning Redis in production since mid-2009@noahd1 and @lukemelia on Twitter

3.
Pronouncing Redis

4.
Tonight’s gameplanFly-by intro to RedisRedis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching

5.
Salvatore Sanfilippo Original author of Redis. @antirez on Twitter. Lives in Italy.Recently hired by VMWare. Great project leader.

6.
Describing RedisRemote dictionary server“advanced, fast, persistent key- value database”“Data structures server”“memcached on steroids”

7.
In Salvatore’s words “I see Redis definitely more as a flexible toolthan as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo

8.
QualitiesWritten in CFew dependenciesFastSingle-threadedLots of clients availableInitial release was March 2009

9.
FeaturesKey-value store where a value is one of: scalar/string, list, set, sorted sets, hashPersistence: in-memory, snapshots, append-only log, VMReplication: master-slave, configureablePub-SubExpiry“Transaction”-yIn development: Redis Cluster

10.
What Redis ain’t (...yet)Big dataSeamless scalingAd-hoc query toolThe only data store you’ll ever need

11.
Who’s using Redis?VMWare GroovesharkGithub SuperfeedrCraigslist RavelryEngineYard mediaFAILThe Guardian WeplayForrst More...PostRank

12.
Our infrastructureApp server App server Utility server MySQL master Redis master MySQL slave Redis slave

13.
Redis in Practice #1View Counts

14.
View countspotential for massive amounts of simple writes/readsvaluable data, but not mission criticallots of row contention in SQL

15.
Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6

16.
Redis in Practice #2Distributed Locks

17.
Subscribe to a Weplay Calendar Subscribe to “ics”

18.
Subscription challengesGenerally static content for long periods of time, but withshort sessions of frequent updatesExpensive to compute on the fly

19.
Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish

20.
Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true

21.
Redis: Obtained LockADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"

22.
Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0

23.
Redis in Practice #3 Presence

24.
“Who’s online?”Weplay members told usthey wanted to be ableto see which of theirfriends were online...

25.
About sets0 to N elements Adding a value to a set does not require youUnordered to check if the valueNo repeated members exists in the set first

26.
Working with Redis sets 1/3# SADD key, member# Adds the specified member to the set stored at keyredis = Redis.newredis.sadd my_set, foo # => trueredis.sadd my_set, bar # => trueredis.sadd my_set, bar # => falseredis.smembers my_set # => ["foo", "bar"]

27.
Working with Redis sets 2/3# SUNION key1 key2 ... keyN# Returns the members of a set resulting from the union of all# the sets stored at the specified keys.# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b># Works like SUNION but instead of being returned the resulting# set is stored as dstkey.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sunion set_a, set_b # => ["foo", "baz", "bar"]redis.sunionstore set_ab, set_a, set_bredis.smembers set_ab # => ["foo", "baz", "bar"]

28.
Working with Redis sets 3/3# SINTER key1 key2 ... keyN# Returns the members that are present in all# sets stored at the specified keys.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sinter set_a, set_b # => ["bar"]

29.
Approach 1/2

30.
Approach 2/2

31.
Implementation 1/2# Defining the keysdef current_key key(Time.now.strftime("%M"))enddef keys_in_last_5_minutes now = Time.now times = (0..5).collect {|n| now - n.minutes } times.collect{ |t| key(t.strftime("%M")) }enddef key(minute) "online_users_minute_#{minute}"end

32.
Implementation 2/2# Tracking an Active User, and calculating who’s onlinedef track_user_id(id) key = current_key redis.sadd(key, id)enddef online_user_ids redis.sunion(*keys_in_last_5_minutes)enddef online_friend_ids(interested_user_id) redis.sunionstore("online_users", *keys_in_last_5_minutes) redis.sinter("online_users", "user:#{interested_user_id}:friend_ids")end

33.
Redis in Practice #4Social Activity Feeds

34.
Social Activity Feeds

35.
Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data

36.
Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100

37.
Large SQL tableGrew quicklyDifficult to maintainDifficult to prune

38.
Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE

39.
Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM

40.
via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH user:11:feed, 100 Key Value Friend: 11 LTRIM user:11:feed, 0, 50Activity: 100 user:11:feed [100,99,97,96] LPUSH user:22:feed, 100 Actor 1 Friend: 22 LTRIM user:22:feed, 0, 50 user:22:feed [100,99,98] LPUSH user:33:feed, 100 Teammate: 33 LTRIM user:33:feed, 0, 100 user:33:feed [100,99]

41.
via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]

42.
Rendering the Feed

43.
Redis in Practice #5Friend Suggestions

44.
Friend Suggestionssuggest new connections based onexisting connectionsexpanding the social graph andmirroring real world connections is key

45.
The Concept PaulMe George John Ringo

46.
The Yoko Factor PaulMe George John Yoko Ringo

47.
The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika

48.
The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue

49.
ZSETS in Redis“Sorted Sets”Each member of the set has a scoreOrdered by the score at all times

50.
Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “donny”zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Mezincrby “user:1:suggestions” 1 “erika” Ted Erikazincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]

51.
Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Tedzrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]

52.
Redis in Practice #6 Caching

53.
Suitability for cachingExcellent match for managed denormalization ex. friendships, teams, teammatesExcellent match where you would benefit from persistenceand/or replicationHistorically, not a good match for a “generational cache,” inwhich you want to optimize memory use by evicting least-recently used (LRU) keys

54.
As an LRU cacheTTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTLRedis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random

55.
Easy to adaptNamespaced Rack::Session, Rack::Cache, I18n and cacheRedis cache stores for Ruby web frameworks implementation is under 1,000 LOC

56.
Contentious benchmarks

57.
Anyquestions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com