Redis Use Patterns (DevconTLV June 2014)


Published on

An introduction to Redis for the SQL practitioner, covering data types and common use cases.

The video of this session can be found at:

Published in: Software, Technology

Redis Use Patterns (DevconTLV June 2014)

  1. 1. Redis Use Patterns An Introduction to the SQL Practitioner @ItamarHaber #DevConTLV 2014
  2. 2. About A Redis Geek and Chief Developers Advocate at .com We provide Redis Cloud – an enterprise-class Redis service for developers (infinitely scalable, highly-available with auto failover, top performing hosted Redis off AWS, Google, Azure & IBM SoftLayer) Get your free t-shirt and sticker outside!
  3. 3. What’s Redis? (REmote DIctionary Server) • Open-source (BSD), in-memory, persist-able key-value advanced datastore • Key-value means something like CREATE TABLE redis ( k VARCHAR(512MB) NOT NULL, v VARCHAR(512MB), PRIMARY KEY (k) ); • 6 data types, 160 commands, blazing fast • Created in 2009 by @antirez (a.k.a Salvatore Sanfilippo) • Source: • Website:
  4. 4. Why Redis? Because It Is Fun! • Simplicity  rich functionality, great flexibility • Performance  easily serves 100K’s of ops/sec • Lightweight  ~ 2MB footprint • Production proven (name dropping)  Twitter Pintrest StackOverflow teowaki many more …
  5. 5. Redis Makes You . . . . THINK! • about how data is stored • about how data is accessed • about efficiency • about performance • about the network • … • Redis is a database construction kit • Beware of Maslow's "Golden" Gavel/Law of Instrument: "If all you have is a hammer, everything looks like a nail"
  6. 6. Pattern: Caching Calls to the DB Motivation: quick responses, reduce load on DBMS How: keep the statement's results using the Redis STRING data type def get_results(sql): hash = result = redis.get(hash) if result is None: result = db.execute(sql) redis.set(hash, result) # or use redis.setex to set a TTL for the key return result
  7. 7. STRINGs • Are the most basic data type • Are binary-safe • Is used for storing: • Strings (duh) – APPEND, GETRANGE, SETRANGE, STRLEN • Integers – INCR, INCRBY, DECR, DECRBY • Floats – INCRBYFLOAT • Bits – SETBIT, GETBIT, BITPOS, BITCOUNT, BITOP
  8. 8. Pattern: Avoiding Calls to the DB Motivation: server-side storage and sharing of data that doesn't need a full-fledged RDBMS, e.g. sessions and shopping carts How: depending on the case, use STRING or HASH to store data in Redis def add_to_cart(session, product, quantity): if quantity > 0: redis.hset('cart:' + session, product, quantity) else: redis.hrem('cart:' + session, product) def get_cart_contents(session): return redis.hgetall('cart:' + session)
  9. 9. The HASH Data Type • Acts as a Redis-within-Redis  contains key-value pairs • Have their own commands: HINCRBY, HINCRBYFLOAT, HLEN, HKEYS, HVALS… • Usually used for aggregation, i.e. keeping related data together for easy fetching/updating (remember that Redis is not a relational database). Example: Using separate keys Using hash aggregation user:1:id  1 user:1 id  1 user:1:fname  Foo fname  Foo user:1:lname  Bar lname  Bar user:1:email  email 
  10. 10. Denormalization • Non relational  no foreign keys, no referential integrity constraints • Thus, data normalization isn't practical • Be prepared to have duplicated data, e.g.: > HSET user:1 country Mordor > HSET user:2 country Mordor … • Tradeoff: Processing Complexity ↔ Data Volume
  11. 11. Pattern: Lists of Items Motivation: keeping track of a sequence, e.g. last viewed profiles How: use Redis' LIST data type def view_product(uid, product): redis.lpush('user:' + uid + ':viewed', product) redis.ltrim('user:' + uid + ':viewed', 0, 9) … def get_last_viewed_products(uid): return redis.lrange('user:' + uid + ':viewed', 0, -1)
  12. 12. Key Points About Key Names • Key names are "limited" to 512MB (also the values btw) • To conserve RAM & CPU, try avoid using unnecessarily_longish_names_for_your_redis_keys because they are more expensive to store and compare (unlike an RDBMS's column names, key names are saved for each key-value pair) • On the other hand, don't be too stringent (e.g 'u:<uid>:r') • Although not mandatory, the convention is to use colons (':') to separate the parts of the key's name • Your schema is your keys' names so keep them in order
  13. 13. Pattern: Queues (apropos the list data type) Motivation: a producer-consumer use case, asynchronous job management, e.g. processing photo uploads def enqueue(queue, item): redis.lpush(queue, item) def dequeue(queue): return redis.rpop(queue) # or use brpop for blocking pop
  14. 14. Is Redis ACID? (mostly) Yes! • Redis is (mostly) single threaded, hence every operation is • Atomic • Consistent • Isolated • WATCH/MULTI/EXEC allow something like transactions (no rollbacks) • Server-side Lua scripts ("stored procedures") also behave like transactions • Durability is configurable and is a tradeoff between efficiency and safety
  15. 15. Pattern: Searching Motivation: finding keys in the database, for example all the users How #1: use a LIST to store key names How #2: the *SCAN commands def do_something_with_all_users(): first = True cursor = 0 while cursor != 0 or first: first = False cursor, data = redis.scan(cursor, 'user:*') do_something(data)
  16. 16. Pattern: Indexing Motivation: Redis doesn't have indices, you need to maintain them How: the SET data type (a collection of unordered unique members) def update_country_idx(country, uid): redis.sadd('country:' + country, uid) def get_users_in_country(country): return redis.smembers('country:' + country)
  17. 17. Pattern: Relationships Motivation: Redis doesn't have foreign keys, you need to maintain them > SADD user:1:friends 3 4 5 // Foo is social and makes friends > SCARD user:1:friends // How many friends does Foo have? > SINTER user:1:friends user:2:friends // Common friends > SDIFF user:1:friends user:2:friends // Exclusive friends > SUNION user:1:friends user:2:friends // All the friends
  18. 18. ZSETs (Sorted Sets) I HAVE CDO IT'S LIKE OCD BUT ALL THE LETTERS ARE IN ALPHABETICAL ORDER AS THEY SHOULD BE • Are just like SETs: • Members are unique • ZADD, ZCARD, ZINCRBY, … • ZSET members have a score that's used for sorting • ZCOUNT, ZRANGE, ZRANGEBYSCORE • When the scores are identical, members are sorted alphabetically • Lexicographical ranges are also supported: • ZLEXCOUNT, ZRANGEBYLEX
  19. 19. Pattern: Sorting Motivation: anything that needs to be sorted How: ZSETs > ZADD friends_count 3 1 1 2 999 3 > ZREVRANGE friends_count 0 -1 3 1 2 Set members (uids) Scores (friends count)
  20. 20. The SORT Command • A command that sorts LISTs, SETs and SORTED SETs • SORT's syntax is the most complex (comparatively) but SQLers should feel right at home with it: SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] [STORE destination] • SORT is also expensive in terms of complexity  O(N+M*log(M)) • BTW, SORT is perhaps the only ad-hoc-like command in Redis
  21. 21. Pattern: Counting Things Motivation: statistics, real-time analytics, dashboards, throttling How #1: use the *INCR commands How #2: use a little bit of BIT* def user_log_login(uid): joined = redis.hget('user:' + uid, 'joined') d0 = datetime.strptime(joined, '%Y-%m-$d') d1 = delta = d1 – d0 redis.setbit('user:' + uid + ':logins', delta, 1) def user_logins_count(uid): return redis.bitcount( 'user:' + uid + ':logins', 0, -1) I love to COUNT(*) data! One datum, two data, three data! HA HA HA HA!
  22. 22. Pattern: Counting Unique Items How #1: SADD items and SCARD for the count Problem: more unique items  more RAM  How #2: the HyperLogLog data structure > PFADD counter item1 item2 item3 … • HLL is a probabilistic data structure that counts (PFCOUNT) unique items • Sacrifices accuracy: standard error of 0.81% • Gains: constant complexity and memory – 12KB per counter • Bonus: HLLs are merge-able with PFMERGE 
  23. 23. Wait, There's More! • There are 107 additional commands that we didn't cover  • Expiration and eviction policies • Publish/Subscribe • Data persistency and durability • Server-side scripting with Lua • Master-Slave(s) replication • High availability with Sentinel • Redis v3 == Cluster (currently in beta) • …
  24. 24. Where Next? • Try the interactive demo and get a free 25MB Redis database in the cloud at • Need help? • RTFM: • Ask the redis-db mailing list • Visit #redis on Freenode IRC • Email me: itamar@ .com