More Related Content


Redis/Lessons learned

  1. Lessons learned: NoSQL/Redis Tit Petric / Jan 2013 Monotek d.o.o.
  2. NoSQL: why it’s here SQL: - Slow query performance - Concurrency / locking - Hard to scale (even harder for writes, storage) Typical problems - Session storage - Statistics (high write to read ratio) - Modifying schema on large data sets Tit Petric / Twitter @titpetric
  3. NoSQL: memcache Memcache (RTV 2008-Present) - Pro: stability, speed - Pro: simple text protocol (added binary to fuck with us) - Love/Hate: Scaling out reads/writes - Con: Persistence - Con: Replication - Con: Key eviction not based on LRU/TTL but slab allocation Tit Petric / Twitter @titpetric
  4. NoSQL: sharedance Sharedance (2009-2011) - Pro: Persistent KV storage - Pro: Simple text protocol (wrote a client in LUA) - Con: Had to patch daemon to handle eviction load (1 key = 1 file, filesystems can’t handle this) - Con: Had to use special ReiserFS filesystem on deployment Tit Petric / Twitter @titpetric
  5. NoSQL: redis Redis (2011-Present) - Pro: stability, speed - Pro: simple text protocol, documentation, clients - Love: Scaling out reads/writes - Pro: Persistence - Pro: Replication - Con/Feature request: key segmentation / grouping. Tit Petric / Twitter @titpetric
  6. NoSQL: uses at RTV Slovenia To-do (2013-?) - Memcached protocol translator to Redis - Look at twemcached, avoid client based sharding - Implement webdis deployment - Redis scripting with LUA Tit Petric / Twitter @titpetric
  7. NoSQL: Redis data types Redis is still a Key/Value store! Only we can have different values: - Strings (essentialy the same as memcache) - Hashes (nested k/v pairs) - Lists (simple arrays) - Sets (unique arrays) - Sorted sets (weighted unique arrays) Tit Petric / Twitter @titpetric
  8. NoSQL: Redis data types LETS SEE SOME EXAMPLES!  Tit Petric / Twitter @titpetric
  9. Redis data types: Strings Limiting Google bot crawl rate “A 503 (Service Unavailable) error will result in fairly frequent retrying. To temporarily suspend crawling, it is recommended to serve a 503 HTTP result code.” Tit Petric / Twitter @titpetric
  10. Redis data types: Strings Limiting Google bot crawl rate SETNX – set a key if it doesn’t exist EXPIRE – expire a key after TTL seconds INCR – increment value by one Tit Petric / Twitter @titpetric
  11. Redis data types: Strings Limiting Google bot crawl rate Tit Petric / Twitter @titpetric
  12. Redis data types: Hashes News ratings HMSET – set multiple hash fields / values HGETALL – get all fields and values HINCRBY – increment integer value of a hash field Tit Petric / Twitter @titpetric
  13. Redis data types: Hashes News ratings Tit Petric / Twitter @titpetric
  14. Redis data types: Hashes News ratings Vote data Why Expire? Race condition. We need HMSETNX Could be better. Tit Petric / Twitter @titpetric
  15. Redis data types: Hashes Other use cases include: - User data, partial data retrieval - select username, realname, birthday from users where id=? - HMGET users:$id username realname birthday - Using SORT (list|set) BY hash values - Don’t use HASHes to store session. Eviction policy works on KEYS not on hash values! Tit Petric / Twitter @titpetric
  16. Redis data types: Lists Any kind of information log (statistics,…) LPUSH – push values to the beginning of the list RPUSH – push values to the end of the list LRANGE – get a range of values LTRIM – trim a list to the specified range LLEN – get the length of the list Tit Petric / Twitter @titpetric
  17. Redis data types: Lists Collecting statistics We can skip the database completely Tit Petric / Twitter @titpetric
  18. Redis data types: Lists Process data Into SQL database Tit Petric / Twitter @titpetric
  19. Redis data types: Lists Well, it’s a way to scale writes to SQL  Processing job can DIE for ages, because: - Back of the envelope calculation for redis memory use: 100M keys use 16 GB ram - Logs get processed in small chunks (200 items), avoiding memory limits. Could increase this by a lot. - We also use sharding so writes are distributed per $table Tit Petric / Twitter @titpetric
  20. Redis data types: Sets Set values are UNIQUE SADD – Add one or more members to a set Perfect use case: set insersection with SINTERSTORE, find duplicates. MySQL is too slow for this, even with indexes… Tit Petric / Twitter @titpetric
  21. Redis data types: Sets SET Intersection in MySQL List1 = first table of data List2 = second table of data Tit Petric / Twitter @titpetric
  22. Redis data types: Sets Bulk transfer MySQL data to redis Via: Tit Petric / Twitter @titpetric
  23. Redis data types: Sets SET Intersection in Redis Much faster, without indexes! 0.118 seconds vs. mysql 1.35 (+0.36 for index) 15x speed increase! Tit Petric / Twitter @titpetric
  24. Redis data types: Sets Other possible uses for sets: • Common friends between A and B • Friend suggestions (You might know…) • People currently online … Tit Petric / Twitter @titpetric
  25. Redis data types: Sets vs. Sorted sets Ok, typical use case in sql> select title, content from news order by stamp desc limit 0,10 #1) Use SORT from redis + HMGET #2) Use sorted sets (ZSET type) Tit Petric / Twitter @titpetric
  26. Redis data types: Sorted sets Sorted sets by time with a PK auto_increment? NO! • Most read news items (sort by views) • Order comments by comment rating • Friends by most friends in common Tit Petric / Twitter @titpetric
  27. Redis data types: Sorted sets Order comments by rating ZINCRBY – increase/decrease score of item ZRANGE – return portion of sorted set, ASC ZREVRANGE – portion of sorted set, DESC Tit Petric / Twitter @titpetric
  28. Redis data types: Sorted sets Sort comments by rating! With pagination! ZRANGE – return portion of sorted set, ASC ZREVRANGE – portion of sorted set, DESC Tit Petric / Twitter @titpetric
  29. Scaling Redis deployment SLAVEOF [host] [port] Starts replicating from [host]:[port], making this instance a slave SLAVEOF NO ONE Promote instance to MASTER role Tit Petric / Twitter @titpetric
  30. Scaling Redis deployment Phpredis client does not implement sharding by itself! But … - Master / Multi-slave scaling is easy to do - Failover for reads is easy, node ejection possible - Client deploys still take time – twemproxy is an option - Twemproxy also provides sharding support, & Memcached  - Want to see what Redis is doing? Issue “MONITOR” command. - Stale data is better than no data, we still consider Redis volatile - FlushDB = rebuild cache, we tolerate data loss Tit Petric / Twitter @titpetric
  31. Redis: Q & A section Questions and answers! Follow me on Twitter: @titpetric Read our tech blog: Tit Petric / Twitter @titpetric