Introduction to redis - version 2


Published on

This is a revised and shorter version of my old "introduction to redis" lecture

Published in: Technology

Introduction to redis - version 2

  1. 1. "Little Server of Awesome" 2011 Dvir Volk Software Architect, DoAT
  2. 2. Redis: Key => DataStructure Server● Memcache-ish in-memory key/value store● But values are complex data types: ○ blobs ○ lists ○ sets ○ sorted sets ○ hash tables● And it has persistence.● Open source; very helpful and friendly community.● Used in the real world: github, craigslist,, engineyard...● Used heavily in doAT as a front-end database, search, geo resolving
  3. 3. Key Features and Cool Stuff● All data is in memory (almost)● All data is eventually persistent● Handles huge workloads easily● Mostly O(1) behavior, Ideal for write-heavy workloads● Support for atomic operations, transactions● Has pub/sub functionality● Single threaded, uses aync. IO● Internal scripting with LUA in alpha
  4. 4. A little benchmarkThis is on my laptop (core i7 @2.2Ghz) ● SET: 187265.92 requests per second ● GET: 185185.17 requests per second ● INCR: 190114.06 requests per second ● LPUSH: 190114.06 requests per second ● LPOP: 187090.73 requests per second ● ● SADD: 186567.16 requests per second ● SPOP: 185873.61 requests per second
  5. 5. Scaling it up ● Master-slave replication out of the box ● Slaves can be made masters on the fly ● Redis-cluster in alpha state. ● Sharding supported by some clients ● Single threaded - run num_cores/2 instances on the same machine
  6. 6. Persistence● All data is synchronized to disk - eventually or immediately● Pick your risk level Vs. performance● Data is either dumped in a forked process, or written as a append-only change-log (AOF)● Append-only mode supports transactional disk writes so you can lose no data (cost: 99% speed loss :) )● You can save the state explicitly, background or blocking● If the data is too big to fit in memory - redis can handle its own swap.
  7. 7. Show me the features!
  8. 8. The basics...Get/Sets - nothing fancy. Keys are strings, anything goes - just quote spaces.redis> SET foo "bar"OKredis> GET foo"bar"You can atomically increment numbersredis> SET bar 337OKredis> INCRBY bar 1000(integer) 1337Getting multiple values at onceredis> MGET foo bar1. "bar"2. "1337"Keys are lazily expiredredis> EXPIRE foo 1(integer) 1redis> GET foo(nil)Be careful with EXPIRE - re-setting a value without re-expiring it will remove theexpiration!
  9. 9. Atomic OperationsGETSET puts a different value inside a key, retriving the old oneredis> SET foo barOKredis> GETSET foo baz"bar"redis> GET foo"baz"SETNX sets a value only if it does not existredis> SETNX foo bar*OK*redis> SETNX foo baz*FAILS*
  10. 10. List operations ● Lists are your ordinary linked lists. ● You can push and pop at both sides, extract range, resize.. ● Random access and ranges at O(N) ● BLPOP: Blocking POP - wait until a list has elements and pop them. Useful for realtime stuff.
  11. 11. Set operations ● Sets are... well, sets of unique values w/ push, pop, etc. ● Sets can be intersected/diffed /unioned server side. ● Can be useful as keys when building complex schemata.
  12. 12. Sorted Sets ● Same as sets, but with score per element ● Ranked ranges, aggregation of scores on INTERSECT ● Can be used as ordered keys in complex schemata ● Think timestamps, inverted index, geohashing, ip ranges
  13. 13. Hashes ● Hash tables as values ● Think of an object store with atomic access to object members redis> HSET foo bar 1 redis> HINCRBY foo bar 1 (integer) 1 (integer) 2 redis> HSET foo baz 2 (integer) 1 redis> HGET foo bar redis> HSET foo foo foo "2" (integer) 1 redis> HKEYS foo redis> HGETALL foo 1. "bar" { 2. "baz" "bar": "1", 3. "foo" "baz": "2", "foo": "foo" }
  14. 14. PubSub - Publish/Subscribe ● Clients can subscribe to channels or patterns and receive notifications when messages are sent to channels. ● Subscribing is O(1), posting messages is O(n) ● Think chats, Comet applications: real-time analytics, twitter
  15. 15. Transactions ● MULTI, ...., EXEC: Easy because of the single thread. ● All commands are executed after EXEC, block and return values for the commands as a list. ● Example:redis> MULTIOKredis> SET "foo" "bar"QUEUEDredis> INCRBY "num" 1QUEUEDredis> EXEC1) OK2) (integer) 1 ● Transactions can be discarded with DISCARD. ● WATCH allows you to lock keys while you are queuing your transaction, and avoid race conditions.
  16. 16. Gotchas, Lessons Learned● 64 bit instances consume much much more RAM.● Master/Slave sync far from perfect.● DO NOT USE THE KEYS COMMAND!!!● vm.overcommit_memory = 1● Use MONITOR to see whats going on
  17. 17. Recipe 1: Social Feed ● Each Users followers are list: user:$ID:followers => { 1, 2, 4 } ● Each Users feed is a sorted set with the time-stamp being the score: user:$ID:feed => { msg1234: 124950132, ... } ● Each message is either a blob or a HASH object with fields: msg1234 => {title, text, ... } ● When a message is added, we: ○ add the message ○ read the posting users follower list ○ add the message id to each followers feed ● When a user reads their feeds, we use ZREVRANGE to get the latest item ids, and HGETALL to fetch the message texts. ● For bonus points, connected users can receive push notifications with PubSub
  18. 18. Recipe 2: Real-Time Analytics ● We want to measure page views, per page, one minute resolution, in real time. ● Each page is represented by a sorted set, where the value is the minute, and the score is the number of hits. ● On each page hit, we do ZINCRBY <page> <current_minute> 1 ● Page views for the last N minutes: ZREVRANGE <page> 0 N ● Group several pages together: ZUNIONSTORE ... AGGREGATE SUM ● Trimming for a specific window: ZREMRANGEBYRANK ● Again, PubSub can be used to push updates to clients
  19. 19. Recipe 3: Processing Queue● Each task can be a blob describing it with json, using SET/GET/DEL.● We keep a list of pending tasks, or multiple lists per task type/priority ("tasks:urgent")● Worker procs can work from any machine over the network.● Workers poll task queues using BRPOP tasks:urgent● atomicity guaranteed!● When a task is pulled it can be marked on a different list/set of "tasks:inprogress". When a task is done it is moved to a "done" or "failed" task list.● A manager can monitor failed tasks and move them around.
  20. 20. Other use case ideas● Geo Resolving with geohashing● Implemented and opened by yours truly● API Key and rate management● Very fast key lookup, rate control counters using INCR● Real time game data● ZSETs for high scores, HASHES for online users, etc● Database Shard Index● map key => database id. Count size with SETS● Comet - no polling ajax● use BLPOP or pub/sub● Machine learning data with automatic persistence.● sorted sets are your vectors!
  21. 21. More resourcesRedis website:http://redis.ioExcellent and more detailed presentation by Simon Willison (abit old): more complex twitter clone: command reference:
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.