Redis in Practice

Redis in PracticeNoSQL NYC • December 2nd, 2010 Noah Davis & Luke Melia

About Noah & LukeWeplay.comRubyistsRunning Redis in production since mid-2009@noahd1 and @lukemelia on Twitter

Pronouncing Redis

Tonight’s gameplanFly-by intro to RedisRedis in Practice View counts Q&A throughout, please. Global locks We love being Presence interrupted. Social activity feeds Friend suggestions Caching

Salvatore Sanﬁlippo Original author of Redis. @antirez on Twitter. Lives in Italy.Recently hired by VMWare. Great project leader.

Describing RedisRemote dictionary server“advanced, fast, persistent key- value database”“Data structures server”“memcached on steroids”

In Salvatore’s words “I see Redis definitely more as a flexible toolthan as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.” - Salvatore Sanfilippo

QualitiesWritten in CFew dependenciesFastSingle-threadedLots of clients availableInitial release was March 2009

FeaturesKey-value store where a value is one of: scalar/string, list, set, sorted sets, hashPersistence: in-memory, snapshots, append-only log, VMReplication: master-slave, conﬁgureablePub-SubExpiry“Transaction”-yIn development: Redis Cluster

What Redis ain’t (...yet)Big dataSeamless scalingAd-hoc query toolThe only data store you’ll ever need

Who’s using Redis?VMWare GroovesharkGithub SuperfeedrCraigslist RavelryEngineYard mediaFAILThe Guardian WeplayForrst More...PostRank

Our infrastructureApp server App server Utility server MySQL master Redis master MySQL slave Redis slave

Redis in Practice #1View Counts

View countspotential for massive amounts of simple writes/readsvaluable data, but not mission criticallots of row contention in SQL

Redis incrementing $ redis-cli redis> incr "medium:301:views" (integer) 1 redis> incr "medium:301:views" (integer) 2 redis> incrby "medium:301:views" 5 (integer) 7 redis> decr "medium:301:views" (integer) 6

Redis in Practice #2Distributed Locks

Subscribe to a Weplay Calendar Subscribe to “ics”

Subscription challengesGenerally static content for long periods of time, but withshort sessions of frequent updatesExpensive to compute on the ﬂy

Without Redis/Locking CANCEL EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish Contention/Redundancy ADD EVENT BACKGROUND Pubisher QUEUE Queue: Publish ICS Publish

Redis Locking: SetNX redis = Redis.new redis.setnx "locking.key", Time.now + 2.hours => true redis.setnx "locking.key", Time.now + 2.hours => false redis.del "locking.key" => true¨ redis.setnx "locking.key", Time.now + 2.hours => true

Redis: Obtained LockADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 1 BACKGROUND PUBLISHER REDIS QUEUE 5 minutes later: PUBLISH.... Queue Job: Publish.publish_ics("34") DEL "group_34_publish_ics"

Redis: Locked Out CANCEL EVENT REDIS SETNX "group_34_publish_ics" Returns: 0 ADD EVENT REDIS SETNX "group_34_publish_ics" Returns: 0

Redis in Practice #3 Presence

“Who’s online?”Weplay members told usthey wanted to be ableto see which of theirfriends were online...

About sets0 to N elements Adding a value to a set does not require youUnordered to check if the valueNo repeated members exists in the set ﬁrst

Working with Redis sets 1/3# SADD key, member# Adds the specified member to the set stored at keyredis = Redis.newredis.sadd my_set, foo # => trueredis.sadd my_set, bar # => trueredis.sadd my_set, bar # => falseredis.smembers my_set # => ["foo", "bar"]

Working with Redis sets 2/3# SUNION key1 key2 ... keyN# Returns the members of a set resulting from the union of all# the sets stored at the specified keys.# SUNIONSTORE <i>dstkey key1 key2 ... keyN</i></b># Works like SUNION but instead of being returned the resulting# set is stored as dstkey.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sunion set_a, set_b # => ["foo", "baz", "bar"]redis.sunionstore set_ab, set_a, set_bredis.smembers set_ab # => ["foo", "baz", "bar"]

Working with Redis sets 3/3# SINTER key1 key2 ... keyN# Returns the members that are present in all# sets stored at the specified keys.redis = Redis.newredis.sadd set_a, fooredis.sadd set_a, barredis.sadd set_b, barredis.sadd set_b, bazredis.sinter set_a, set_b # => ["bar"]

Approach 1/2

Approach 2/2

Implementation 1/2# Defining the keysdef current_key key(Time.now.strftime("%M"))enddef keys_in_last_5_minutes now = Time.now times = (0..5).collect {|n| now - n.minutes } times.collect{ |t| key(t.strftime("%M")) }enddef key(minute) "online_users_minute_#{minute}"end

Implementation 2/2# Tracking an Active User, and calculating who’s onlinedef track_user_id(id) key = current_key redis.sadd(key, id)enddef online_user_ids redis.sunion(*keys_in_last_5_minutes)enddef online_friend_ids(interested_user_id) redis.sunionstore("online_users", *keys_in_last_5_minutes) redis.sinter("online_users", "user:#{interested_user_id}:friend_ids")end

Redis in Practice #4Social Activity Feeds

Social Activity Feeds

Via SQL, Approach 1/2 Doesn’t scale to more complex social graphs e.g. friends who are SELECT activities.* FROM activities JOIN friendships f1 ON f1.from_id = ‘hidden’ from your feed activities.actor_id JOIN friendships f2 ON f2.to_id = activities.actor_id May still require multiple WHERE f1.to_id = ? OR f2.from_id = ? ORDER BY activities.id DESC queries to grab LIMIT 15 unindexed data

Via SQL, Approach 2/2 user_id 11 activity_id 96 22 96 11 97 22 97 33 97 11 98 22 98 Friend: 11 11 99 Activity: 100 11 100 Actor 1 Friend: 22 INSERTS 22 100 Teammate: 33 33 100

Large SQL tableGrew quicklyDifﬁcult to maintainDifﬁcult to prune

Redis Lists 1/2 redis> lpush "teams" "yankees" (integer) 1 redis> lpush "teams" "redsox" (integer) 2 redis> llen "teams" (integer) 2 redis> lrange "teams" 0 1 1. "redsox" 2. "yankees" redis> lrange "teams" 0 -1 1. "redsox" 2. "yankees" LTRIM, LLEN, LRANGE

Redis Lists 2/2 redis> ltrim "teams" 0 0 OK redis> lrange "teams" 0 -1 1. "redsox" LTRIM

via Redis, 1/2 LPUSH “user:11:feed”, “100” LTRIM “user:11:feed”, 0, 50 LPUSH user:11:feed, 100 Key Value Friend: 11 LTRIM user:11:feed, 0, 50Activity: 100 user:11:feed [100,99,97,96] LPUSH user:22:feed, 100 Actor 1 Friend: 22 LTRIM user:22:feed, 0, 50 user:22:feed [100,99,98] LPUSH user:33:feed, 100 Teammate: 33 LTRIM user:33:feed, 0, 100 user:33:feed [100,99]

via Redis, 2/2 Key Value Friend: 11 user:11:feed [100,99,97,96] Friend: 22 user:22:feed [100,99,98] Activity: 100 Actor 1 user:33:feed [100,99] Teammate: 33 Key Value New York new_york:feed [100,67] Boston boston:feed [100,99]

Rendering the Feed

Redis in Practice #5Friend Suggestions

Friend Suggestionssuggest new connections based onexisting connectionsexpanding the social graph andmirroring real world connections is key

The Concept PaulMe George John Ringo

The Yoko Factor PaulMe George John Yoko Ringo

The Approach 1/2 Sarah 2 Frank Donny Me 2 Ted Erika

The Approach 2/2 Sarah 2 Frank Donny Me Ted 3 Erika Sue

ZSETS in Redis“Sorted Sets”Each member of the set has a scoreOrdered by the score at all times

Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “donny”zincrby “user:1:suggestions” 1 “donny” Sarah Frank Donny Mezincrby “user:1:suggestions” 1 “erika” Ted Erikazincrby “user:1:suggestions” 1 “erika” [ Donny 2 , Erika 2 ]

Friend Suggestions in Rediszincrby “user:1:suggestions” 1 “erika” Sarah Frank Donny Me Tedzrevrange “user:1:suggestions” 0 1 Erika Sue [ Erika 3 , Donny 2 ]

Redis in Practice #6 Caching

Suitability for cachingExcellent match for managed denormalization ex. friendships, teams, teammatesExcellent match where you would beneﬁt from persistenceand/or replicationHistorically, not a good match for a “generational cache,” inwhich you want to optimize memory use by evicting least-recently used (LRU) keys

As an LRU cacheTTL-support since inception, but with unintuitive behavior Writing to volatile key replaced it and cleared the TTLRedis 2.2 changes this behavior and adds key features: ability to write to, and update expiry of volatile keys maxmemory [bytes], maxmemory-policy [policy] policies: volatile-lru, volatile-ttl, volatile-random, allkeys-lru, allkeys-random

Easy to adaptNamespaced Rack::Session, Rack::Cache, I18n and cacheRedis cache stores for Ruby web frameworks implementation is under 1,000 LOC

Contentious benchmarks

Anyquestions? Thanks! Follow us on Twitter: @noahd1 and @lukemelia Tell your friends in youth sports about Weplay.com

http://www.lukemelia.com	1279
http://www.cnblogs.com	1070
http://archive.cnblogs.com	313
http://asyncionews.com	116
http://www.note4u.net	21
http://www.fishxl.com	13
http://blog.0898888.com	10
http://www.shaoqun.com	7
http://kimelia.com	6
http://www.kimelia.com	5
http://cache.baidu.com	3
http://thinkery.me	3
http://www.pega.vn	3
https://twimg0-a.akamaihd.net	2
http://reader.youdao.com	2
http://a0.twimg.com	2
http://translate.googleusercontent.com	1
http://www.haogongju.net	1
http://j-reference.blogspot.in	1
http://mail.lukemelia.com	1
http://www.techgig.com	1
http://paper.li	1

Redis in Practice

by noahd1 on Dec 02, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

22 Embeds 2,861

Statistics

Redis in Practice — Presentation Transcript