Redis/Lessons learned


Published on

An insight into NoSQL solutions implemented at RTV Slovenia and elsewhere, what problems we are trying to solve and an introduction to solving them with Redis.

Talk given at #wwwh @ Ljubljana, 30.1.2013 by me, Tit Petric

1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Redis/Lessons learned

  1. 1. Lessons learned: NoSQL/Redis Tit Petric / Jan 2013 Monotek d.o.o.
  2. 2. NoSQL: why it’s hereSQL:- Slow query performance- Concurrency / locking- Hard to scale (even harder for writes, storage)Typical problems- Session storage- Statistics (high write to read ratio)- Modifying schema on large data setsTit Petric / Twitter @titpetric
  3. 3. NoSQL: memcacheMemcache (RTV 2008-Present)- Pro: stability, speed- Pro: simple text protocol (added binary to fuck with us)- Love/Hate: Scaling out reads/writes- Con: Persistence- Con: Replication- Con: Key eviction not based on LRU/TTL but slab allocationTit Petric / Twitter @titpetric
  4. 4. NoSQL: sharedanceSharedance (2009-2011)- Pro: Persistent KV storage- Pro: Simple text protocol (wrote a client in LUA)- Con: Had to patch daemon to handle eviction load (1 key = 1 file, filesystems can’t handle this)- Con: Had to use special ReiserFS filesystem on deploymentTit Petric / Twitter @titpetric
  5. 5. NoSQL: redisRedis (2011-Present)- Pro: stability, speed- Pro: simple text protocol, documentation, clients- Love: Scaling out reads/writes- Pro: Persistence- Pro: Replication- Con/Feature request: key segmentation / grouping.Tit Petric / Twitter @titpetric
  6. 6. NoSQL: uses at RTV SloveniaTo-do (2013-?)- Memcached protocol translator to Redis- Look at twemcached, avoid client based sharding- Implement webdis deployment- Redis scripting with LUATit Petric / Twitter @titpetric
  7. 7. NoSQL: Redis data typesRedis is still a Key/Value store!Only we can have different values: - Strings (essentialy the same as memcache) - Hashes (nested k/v pairs) - Lists (simple arrays) - Sets (unique arrays) - Sorted sets (weighted unique arrays)Tit Petric / Twitter @titpetric
  8. 8. NoSQL: Redis data types LETS SEE SOME EXAMPLES! Tit Petric / Twitter @titpetric
  9. 9. Redis data types: StringsLimiting Google bot crawl rate “A 503 (Service Unavailable) error will result in fairly frequent retrying. To temporarily suspend crawling, it is recommended to serve a 503 HTTP result code.”Tit Petric / Twitter @titpetric
  10. 10. Redis data types: StringsLimiting Google bot crawl rateSETNX – set a key if it doesn’t existEXPIRE – expire a key after TTL secondsINCR – increment value by oneTit Petric / Twitter @titpetric
  11. 11. Redis data types: StringsLimiting Google bot crawl rateTit Petric / Twitter @titpetric
  12. 12. Redis data types: HashesNews ratingsHMSET – set multiple hash fields / valuesHGETALL – get all fields and valuesHINCRBY – increment integer value of a hashfieldTit Petric / Twitter @titpetric
  13. 13. Redis data types: HashesNews ratingsTit Petric / Twitter @titpetric
  14. 14. Redis data types: HashesNews ratingsVote data Why Expire? Race condition. We need HMSETNX Could be better.Tit Petric / Twitter @titpetric
  15. 15. Redis data types: HashesOther use cases include:- User data, partial data retrieval - select username, realname, birthday from users where id=? - HMGET users:$id username realname birthday- Using SORT (list|set) BY hash values- Don’t use HASHes to store session. Eviction policy works on KEYS not on hash values!Tit Petric / Twitter @titpetric
  16. 16. Redis data types: ListsAny kind of information log (statistics,…)LPUSH – push values to the beginning of the listRPUSH – push values to the end of the listLRANGE – get a range of valuesLTRIM – trim a list to the specified rangeLLEN – get the length of the listTit Petric / Twitter @titpetric
  17. 17. Redis data types: ListsCollecting statisticsWe can skip the database completelyTit Petric / Twitter @titpetric
  18. 18. Redis data types: ListsProcess dataInto SQLdatabaseTit Petric / Twitter @titpetric
  19. 19. Redis data types: ListsWell, it’s a way to scale writes to SQL Processing job can DIE for ages, because:- Back of the envelope calculation for redis memory use: 100M keys use 16 GB ram- Logs get processed in small chunks (200 items), avoiding memory limits. Could increase this by a lot.- We also use sharding so writes are distributed per $tableTit Petric / Twitter @titpetric
  20. 20. Redis data types: SetsSet values are UNIQUESADD – Add one or more members to a setPerfect use case: set insersection withSINTERSTORE, find duplicates.MySQL is too slow for this, even withindexes…Tit Petric / Twitter @titpetric
  21. 21. Redis data types: SetsSET Intersection in MySQLList1 = first table of dataList2 = second table of dataTit Petric / Twitter @titpetric
  22. 22. Redis data types: SetsBulk transfer MySQL data to redisVia: Petric / Twitter @titpetric
  23. 23. Redis data types: SetsSET Intersection in RedisMuch faster, without indexes!0.118 seconds vs. mysql 1.35 (+0.36 for index)15x speed increase!Tit Petric / Twitter @titpetric
  24. 24. Redis data types: SetsOther possible uses for sets:• Common friends between A and B• Friend suggestions (You might know…)• People currently online …Tit Petric / Twitter @titpetric
  25. 25. Redis data types: Sets vs. Sorted setsOk, typical use case in sql>select title, content from news order bystamp desc limit 0,10#1) Use SORT from redis + HMGET#2) Use sorted sets (ZSET type)Tit Petric / Twitter @titpetric
  26. 26. Redis data types: Sorted setsSorted sets by time with a PKauto_increment? NO!• Most read news items (sort by views)• Order comments by comment rating• Friends by most friends in commonTit Petric / Twitter @titpetric
  27. 27. Redis data types: Sorted setsOrder comments by ratingZINCRBY – increase/decrease score of itemZRANGE – return portion of sorted set, ASCZREVRANGE – portion of sorted set, DESCTit Petric / Twitter @titpetric
  28. 28. Redis data types: Sorted setsSort comments by rating! With pagination!ZRANGE – return portion of sorted set, ASCZREVRANGE – portion of sorted set, DESCTit Petric / Twitter @titpetric
  29. 29. Scaling Redis deploymentSLAVEOF [host] [port]Starts replicating from [host]:[port], making this instance a slaveSLAVEOF NO ONEPromote instance to MASTER roleTit Petric / Twitter @titpetric
  30. 30. Scaling Redis deploymentPhpredis client does not implementsharding by itself! But …- Master / Multi-slave scaling is easy to do- Failover for reads is easy, node ejection possible- Client deploys still take time – twemproxy is an option- Twemproxy also provides sharding support, & Memcached - Want to see what Redis is doing? Issue “MONITOR” command.- Stale data is better than no data, we still consider Redis volatile- FlushDB = rebuild cache, we tolerate data lossTit Petric / Twitter @titpetric
  31. 31. Redis: Q & A section Questions and answers! Follow me on Twitter: @titpetric Read our tech blog: http://foreach.orgTit Petric / Twitter @titpetric