Published on

What is Redis and what can you really do with it?
Data structures
Command processing
Redis roadmap

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Redis for serious Social Media Performance Chris Mordue @cmordue March 22, 2012
  2. 2. Outline What is Redis and what can you really do with it? ● Data structures ● Command processing ● Persistence ● Recipes ● Redis roadmap ●
  3. 3. What is RedisMany things to many people: Cache (Memcache replacement) ● Queueing or Pub/Sub system ● Data structure server ● Advanced Key/Value Store ●What isnt it? Solution to everything ● Easily scalable across ● Object/Graph/Relational DB ● although many structures can be modeled ○Author Salvatore - sponsored by VMware ●
  4. 4. Use cases● Memcached replacement Autocomplete ●● Duplicate detector Twitter ●● FIFO/LIFO Queue digg / Hacker News ●● Priority Queue Social activity feed ●● Distributed hashmap ○ e.g. url shortener● UID generator● Game high scores● Geolocation lookup ○ lat & long -> city● Real-time analytics● Metrics DB● API throttling (rate-limits)
  5. 5. Data structuresTypes: Strings (regular or bitmap) ● get/set (& not exist), setbit/getbit, incr[by]/decr[by], append ○ Lists ● push/pop, set, trim, len, range, blocking/non-blocking ○ Hashsets ● get/set, incrby, exists, del, keys, vals, getall ○ Sets ● add/rem, pop, diff/union/inter (& store), move ○● SortedSets ○ add/rem, rank, range/revrange, interstore/unionstore
  6. 6. Command processingExecution Basics● IO of requests/responses on other thread(s)● Commands are processed in serial in a single thread ○ If you want to use more CPUs, shard your data client-side● Superfast processing of commandsCommand groups● Request/response● Pipelined: Several requests / responses● Transactions: ○ Traditional: Several requests processed in order atomically ○ Check-And-Set: Request(s), data manipulation in client, more requests● Scripts (v2.6): execute client-defined Lua scripts in the server for more complex actions
  7. 7. PersistencePeriodic SnapshotsAppend Only File
  8. 8. Recipes - UID GeneratorObjective: Create unique idsImplementation*: ● incr global:nextDocId ● The response is a unique idPerformance: >100.000/s* from
  9. 9. Recipes - Duplicate DetectorObjective: Determine if an object has already been processedImplementation: ● sadd <objectId> ● If the response is 1, its new. If 0, it already existedGotchas: ● use lru eviction rules to remove old data to keep the data size containedExamples: ● Socialmetrixs twitter-engine
  10. 10. Recipes - MemcachedObjective: Application cacheImplementation: serialize your objects as a string ● get/set ● expire (time to live) ●Performance >= memcacheGotchas: Memcache scales horizontally out of the box, redis does not ●Examples: Integration with Play! Frameworks Cache: ● ○ ○ Garantia Data: providing hosted redis/memcached instances ● ○Image from
  11. 11. Recipes - API Rate LimiterObjective: Recognize when more than E events happen in time T seconds byuser U.Implementation 1*: Implementation 2:FUNCTION LIMIT_API_CALL(ip) FUNCTION LIMIT_API_CALL(ip)ts = CURRENT_UNIX_TIME() ts = CURRENT_UNIX_TIME()keyname = ip+":"+ts keyname = ip+":"+tscurrent = GET(keyname) MULTI current = INCR(keyname, 1)IF current != NULL AND current > 10 THEN EXPIRE(keyname,10) ERROR "too many requests per second" EXECELSE IF current < 10 THEN MULTI PERFORM_API_CALL() INCR(keyname,1) ELSE EXPIRE(keyname,10) ERROR "too many requests per second" END EXEC PERFORM_API_CALL()ENDGotchas: ● make sure you set expires on the keys in a transaction because set or incr removes old expires valuesExamples:* from:
  12. 12. Recipes - Twitter cloneObjective: Duplicate the functionality of twitterImplementation: ● Generate Ids with String: INCR global:nextUserId => 1000Username to id with String: SET username:antirez:uid 1000 ● Followers/Follwings with 2 sets: uid:1000:followers ● Timeline with List: uid:1000:posts ● Tweeting: ○ Create tweet id with String: INCR global:nextPostId => 10343 ○ Store tweet: SET post:10343 "$owner_id|$time|Having fun w/ Retwis" ○ Get Followers list and push tweetId to allow followers timelines: ■ LPUSH uid:$followeruid:posts 10343 ● View posts: ○ Get list of postIds: LRANGE uid:1000:posts 0 50 ○ For each postId, get posts to display: GET post:$postIdFrom: ●
  13. 13. Recipes - Twitter clone cont.Code: ● ● ●"On a very slow and loaded server, apache benchmark with 100 parallel clientsissuing 100000 requests measured the average pageview to take 5milliseconds. This means you can serve millions of users every day with just asingle Linux box, and this one was monkey asses slow!" - Salvatore
  14. 14. Redis roadmapv2.6: Server side scripting with Lua (v5.1): ●Lots of possibilities with scripts - invent your owncommandsv3.0: ● Cluster support ● In dev for many months
  15. 15. Other RecipesOld cookbook (soon to be updated): 17-20: clone design: of other recipes: dashboard for redis:
  16. 16. ¡Gracias! Chris MordueSoftware Architect @cmordue