Redis for serious Social Media Performance Chris Mordue @cmordue March 22, 2012
Outline What is Redis and what can you really do with it? ● Data structures ● Command processing ● Persistence ● Recipes ● Redis roadmap ●
What is RedisMany things to many people: Cache (Memcache replacement) ● Queueing or Pub/Sub system ● Data structure server ● Advanced Key/Value Store ●What isnt it? Solution to everything ● Easily scalable across ● Object/Graph/Relational DB ● although many structures can be modeled ○Author Salvatore - sponsored by VMware ●
Use cases● Memcached replacement Autocomplete ●● Duplicate detector Twitter ●● FIFO/LIFO Queue digg / Hacker News ●● Priority Queue Social activity feed ●● Distributed hashmap ○ e.g. url shortener● UID generator● Game high scores● Geolocation lookup ○ lat & long -> city● Real-time analytics● Metrics DB● API throttling (rate-limits)
Command processingExecution Basics● IO of requests/responses on other thread(s)● Commands are processed in serial in a single thread ○ If you want to use more CPUs, shard your data client-side● Superfast processing of commandsCommand groups● Request/response● Pipelined: Several requests / responses● Transactions: ○ Traditional: Several requests processed in order atomically ○ Check-And-Set: Request(s), data manipulation in client, more requests● Scripts (v2.6): execute client-defined Lua scripts in the server for more complex actions
Recipes - UID GeneratorObjective: Create unique idsImplementation*: ● incr global:nextDocId ● The response is a unique idPerformance: >100.000/s* from http://redis.io/commands/incr
Recipes - Duplicate DetectorObjective: Determine if an object has already been processedImplementation: ● sadd <objectId> ● If the response is 1, its new. If 0, it already existedGotchas: ● use lru eviction rules to remove old data to keep the data size containedExamples: ● Socialmetrixs twitter-engine
Recipes - MemcachedObjective: Application cacheImplementation: serialize your objects as a string ● get/set ● expire (time to live) ●Performance >= memcacheGotchas: Memcache scales horizontally out of the box, redis does not ●Examples: Integration with Play! Frameworks Cache: ● http://www.playframework.org/modules/redis ○ https://github.com/tkral/play-redis ○ Garantia Data: providing hosted redis/memcached instances ● http://www.garantiadata.com/ ○Image from http://www.garantiadata.com/
Recipes - API Rate LimiterObjective: Recognize when more than E events happen in time T seconds byuser U.Implementation 1*: Implementation 2:FUNCTION LIMIT_API_CALL(ip) FUNCTION LIMIT_API_CALL(ip)ts = CURRENT_UNIX_TIME() ts = CURRENT_UNIX_TIME()keyname = ip+":"+ts keyname = ip+":"+tscurrent = GET(keyname) MULTI current = INCR(keyname, 1)IF current != NULL AND current > 10 THEN EXPIRE(keyname,10) ERROR "too many requests per second" EXECELSE IF current < 10 THEN MULTI PERFORM_API_CALL() INCR(keyname,1) ELSE EXPIRE(keyname,10) ERROR "too many requests per second" END EXEC PERFORM_API_CALL()ENDGotchas: ● make sure you set expires on the keys in a transaction because set or incr removes old expires valuesExamples:* from:http://redis.io/commands/incr
Recipes - Twitter cloneObjective: Duplicate the functionality of twitterImplementation: ● Generate Ids with String: INCR global:nextUserId => 1000Username to id with String: SET username:antirez:uid 1000 ● Followers/Follwings with 2 sets: uid:1000:followers ● Timeline with List: uid:1000:posts ● Tweeting: ○ Create tweet id with String: INCR global:nextPostId => 10343 ○ Store tweet: SET post:10343 "$owner_id|$time|Having fun w/ Retwis" ○ Get Followers list and push tweetId to allow followers timelines: ■ LPUSH uid:$followeruid:posts 10343 ● View posts: ○ Get list of postIds: LRANGE uid:1000:posts 0 50 ○ For each postId, get posts to display: GET post:$postIdFrom: ● http://redis.io/topics/twitter-clone
Recipes - Twitter clone cont.Code: ● https://github.com/SpringSource/spring-data-keyvalue-examples ● http://retwis.antirez.com/ ● http://code.google.com/p/redis/downloads/listPerformance:"On a very slow and loaded server, apache benchmark with 100 parallel clientsissuing 100000 requests measured the average pageview to take 5milliseconds. This means you can serve millions of users every day with just asingle Linux box, and this one was monkey asses slow!" - Salvatore
Redis roadmapv2.6: Server side scripting with Lua (v5.1): http://www.lua.org/manual/5.1/ ●Lots of possibilities with scripts - invent your owncommandsv3.0: ● Cluster support ● In dev for many months
Other RecipesOld cookbook (soon to be updated): http://rediscookbook.org/GeoHashing: https://github.com/doat/geodisAutocomplete: http://patshaughnessy.net/2011/11/29/two-ways-of-using-redis-to-build-a-nosql-autocomplete-search-indexSlides 17-20: http://www.slideshare.net/dvirsky/introduction-to-redis-version-2Digg clone design: http://groups.google.com/group/redis-db/browse_thread/thread/94d0dd1a24bf400e/98a0f5935380a075?lnk=gst&q=digg#98a0f5935380a075Lots of other recipes: http://groups.google.com/group/redis-dbSinatra dashboard for redis: https://github.com/steelThread/redmon
¡Gracias! Chris MordueSoftware Architect @cmordue
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.