0
Redis for serious Social Media         Performance      Chris Mordue @cmordue           March 22, 2012
Outline     What is Redis and what can you really do with it? ●     Data structures ●     Command processing ●     Persist...
What is RedisMany things to many people:     Cache (Memcache replacement) ●     Queueing or Pub/Sub system ●     Data stru...
Use cases●   Memcached replacement              Autocomplete                                   ●●   Duplicate detector    ...
Data structuresTypes:     Strings (regular or bitmap) ●          get/set (& not exist), setbit/getbit, incr[by]/decr[by], ...
Command processingExecution Basics●   IO of requests/responses on other thread(s)●   Commands are processed in serial in a...
PersistencePeriodic SnapshotsAppend Only File
Recipes - UID GeneratorObjective: Create unique idsImplementation*: ● incr global:nextDocId ● The response is a unique idP...
Recipes - Duplicate DetectorObjective: Determine if an object has already been processedImplementation: ● sadd <objectId> ...
Recipes - MemcachedObjective: Application cacheImplementation:    serialize your objects as a string  ●    get/set  ●    e...
Recipes - API Rate LimiterObjective: Recognize when more than E events happen in time T seconds byuser U.Implementation 1*...
Recipes - Twitter cloneObjective: Duplicate the functionality of twitterImplementation: ● Generate Ids with String: INCR g...
Recipes - Twitter clone cont.Code: ●   https://github.com/SpringSource/spring-data-keyvalue-examples ●   http://retwis.ant...
Redis roadmapv2.6:   Server side scripting with Lua (v5.1): http://www.lua.org/manual/5.1/  ●Lots of possibilities with sc...
Other RecipesOld cookbook (soon to be updated): http://rediscookbook.org/GeoHashing: https://github.com/doat/geodisAutocom...
¡Gracias! Chris MordueSoftware Architect   @cmordue
Upcoming SlideShare
Loading in...5
×

Redis

1,518

Published on


What is Redis and what can you really do with it?
Data structures
Command processing
Persistence
Recipes
Redis roadmap

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,518
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Redis"

  1. 1. Redis for serious Social Media Performance Chris Mordue @cmordue March 22, 2012
  2. 2. Outline What is Redis and what can you really do with it? ● Data structures ● Command processing ● Persistence ● Recipes ● Redis roadmap ●
  3. 3. What is RedisMany things to many people: Cache (Memcache replacement) ● Queueing or Pub/Sub system ● Data structure server ● Advanced Key/Value Store ●What isnt it? Solution to everything ● Easily scalable across ● Object/Graph/Relational DB ● although many structures can be modeled ○Author Salvatore - sponsored by VMware ●
  4. 4. Use cases● Memcached replacement Autocomplete ●● Duplicate detector Twitter ●● FIFO/LIFO Queue digg / Hacker News ●● Priority Queue Social activity feed ●● Distributed hashmap ○ e.g. url shortener● UID generator● Game high scores● Geolocation lookup ○ lat & long -> city● Real-time analytics● Metrics DB● API throttling (rate-limits)
  5. 5. Data structuresTypes: Strings (regular or bitmap) ● get/set (& not exist), setbit/getbit, incr[by]/decr[by], append ○ Lists ● push/pop, set, trim, len, range, blocking/non-blocking ○ Hashsets ● get/set, incrby, exists, del, keys, vals, getall ○ Sets ● add/rem, pop, diff/union/inter (& store), move ○● SortedSets ○ add/rem, rank, range/revrange, interstore/unionstore
  6. 6. Command processingExecution Basics● IO of requests/responses on other thread(s)● Commands are processed in serial in a single thread ○ If you want to use more CPUs, shard your data client-side● Superfast processing of commandsCommand groups● Request/response● Pipelined: Several requests / responses● Transactions: ○ Traditional: Several requests processed in order atomically ○ Check-And-Set: Request(s), data manipulation in client, more requests● Scripts (v2.6): execute client-defined Lua scripts in the server for more complex actions
  7. 7. PersistencePeriodic SnapshotsAppend Only File
  8. 8. Recipes - UID GeneratorObjective: Create unique idsImplementation*: ● incr global:nextDocId ● The response is a unique idPerformance: >100.000/s* from http://redis.io/commands/incr
  9. 9. Recipes - Duplicate DetectorObjective: Determine if an object has already been processedImplementation: ● sadd <objectId> ● If the response is 1, its new. If 0, it already existedGotchas: ● use lru eviction rules to remove old data to keep the data size containedExamples: ● Socialmetrixs twitter-engine
  10. 10. Recipes - MemcachedObjective: Application cacheImplementation: serialize your objects as a string ● get/set ● expire (time to live) ●Performance >= memcacheGotchas: Memcache scales horizontally out of the box, redis does not ●Examples: Integration with Play! Frameworks Cache: ● http://www.playframework.org/modules/redis ○ https://github.com/tkral/play-redis ○ Garantia Data: providing hosted redis/memcached instances ● http://www.garantiadata.com/ ○Image from http://www.garantiadata.com/
  11. 11. Recipes - API Rate LimiterObjective: Recognize when more than E events happen in time T seconds byuser U.Implementation 1*: Implementation 2:FUNCTION LIMIT_API_CALL(ip) FUNCTION LIMIT_API_CALL(ip)ts = CURRENT_UNIX_TIME() ts = CURRENT_UNIX_TIME()keyname = ip+":"+ts keyname = ip+":"+tscurrent = GET(keyname) MULTI current = INCR(keyname, 1)IF current != NULL AND current > 10 THEN EXPIRE(keyname,10) ERROR "too many requests per second" EXECELSE IF current < 10 THEN MULTI PERFORM_API_CALL() INCR(keyname,1) ELSE EXPIRE(keyname,10) ERROR "too many requests per second" END EXEC PERFORM_API_CALL()ENDGotchas: ● make sure you set expires on the keys in a transaction because set or incr removes old expires valuesExamples:* from:http://redis.io/commands/incr
  12. 12. Recipes - Twitter cloneObjective: Duplicate the functionality of twitterImplementation: ● Generate Ids with String: INCR global:nextUserId => 1000Username to id with String: SET username:antirez:uid 1000 ● Followers/Follwings with 2 sets: uid:1000:followers ● Timeline with List: uid:1000:posts ● Tweeting: ○ Create tweet id with String: INCR global:nextPostId => 10343 ○ Store tweet: SET post:10343 "$owner_id|$time|Having fun w/ Retwis" ○ Get Followers list and push tweetId to allow followers timelines: ■ LPUSH uid:$followeruid:posts 10343 ● View posts: ○ Get list of postIds: LRANGE uid:1000:posts 0 50 ○ For each postId, get posts to display: GET post:$postIdFrom: ● http://redis.io/topics/twitter-clone
  13. 13. Recipes - Twitter clone cont.Code: ● https://github.com/SpringSource/spring-data-keyvalue-examples ● http://retwis.antirez.com/ ● http://code.google.com/p/redis/downloads/listPerformance:"On a very slow and loaded server, apache benchmark with 100 parallel clientsissuing 100000 requests measured the average pageview to take 5milliseconds. This means you can serve millions of users every day with just asingle Linux box, and this one was monkey asses slow!" - Salvatore
  14. 14. Redis roadmapv2.6: Server side scripting with Lua (v5.1): http://www.lua.org/manual/5.1/ ●Lots of possibilities with scripts - invent your owncommandsv3.0: ● Cluster support ● In dev for many months
  15. 15. Other RecipesOld cookbook (soon to be updated): http://rediscookbook.org/GeoHashing: https://github.com/doat/geodisAutocomplete: http://patshaughnessy.net/2011/11/29/two-ways-of-using-redis-to-build-a-nosql-autocomplete-search-indexSlides 17-20: http://www.slideshare.net/dvirsky/introduction-to-redis-version-2Digg clone design: http://groups.google.com/group/redis-db/browse_thread/thread/94d0dd1a24bf400e/98a0f5935380a075?lnk=gst&q=digg#98a0f5935380a075Lots of other recipes: http://groups.google.com/group/redis-dbSinatra dashboard for redis: https://github.com/steelThread/redmon
  16. 16. ¡Gracias! Chris MordueSoftware Architect @cmordue
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×