Redis <3 at 8tracks.com
•
4 likes•1,752 views
At 8tracks.com we believe on finding the best tool for the job. In the quest of finding a great persistence solution to power our "Explore" system one of our engineers stumbled upon redis.io. A few years later, we run 25 different Redis instances powering several features for our 10 million users. From Tag Browsing, to Autocomplete Search, persisting Player sessions and our News Feed. Our 4 engineers team eats the redis documentation for breakfast. Come join me in some redis love while I tell you why we're such big fans of zsets, set intersections and some other nice features that help us power 12.5 million hours of streaming per month.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Redis <3 at 8tracks.com
Similar to Redis <3 at 8tracks.com (20)
More from Ivayr Farah Netto
Recently uploaded
Recently uploaded (20)
Redis <3 at 8tracks.com
- 3. 8tracks
- 4. 8tracks
- 9. stats • users: 11,351,961! • public playlists: 1,545,586 (3,5 mi total)! • uploaded tracks: 28,185,644
- 10. not a sysadmin talk
- 12. the right tool for the job http://peopletakingpictureswithipads.tumblr.com/
- 15. engineering effort machine effort
- 16. engineering effort machine effort
- 17. engineering effort machine effort
- 18. engineering effort machine effort
- 19. When development tries to blame operations for the outage http://devopsreactions.tumblr.com/post/84309007419/when-development-tries-to-blame-operations-for-the
- 20. R.T.F.J. = engineer <3 + machine <3
- 22. key-value store • open source • in memory (most of the times) • data-structure server
- 23. data types • strings • lists • sets • sorted sets • hashes
- 24. just like that data structures class
- 25. except it is COOL!
- 26. you can turn this
- 27. into this next next next next head tail
- 28. or this next next next
- 29. things to keep in mind • all in memory (but not restricted to) • no joins or iterating over data (but not restricted to) • you should be able to rebuild data
- 30. why? • sometimes basic data structures is all you need • some problems are hard to fit in a relational way
- 31. redis @ 8tracks = <3 sidekiq feature toggles caching player (moved to couchdb, then to mysql) autocomplete (moved to SOLR, then to ElasticSearch) tag browsing! news feed listening sessions rate limiter
- 34. sidekiq
- 37. 1 SELECT name ! ! ! FROM tags ! ! ! WHERE name ! ! ! LIKE 'ind%' LIMIT 10;! ! 2 SELECT name ! ! ! FROM mixes ! ! ! WHERE name ! ! ! LIKE 'ind%' LIMIT 10;! ! 3 SELECT login ! ! ! FROM users ! ! ! WHERE login ! ! ! LIKE 'ind%' LIMIT 10;! ! 4 SELECT name ! ! FROM artists ! ! WHERE name ! ! LIKE 'ind%' LIMIT 10;!
- 38. you could do 1 (SELECT name FROM tags WHERE name LIKE 'ind%' LIMIT 10)! 2 UNION ALL! 3 (SELECT name FROM mixes WHERE name LIKE 'ind%' LIMIT 10);!
- 39. the problems • cross-type search • I don’t want 10 of each, I want the 10 most relevant ones • can’t select different column counts
- 40. not a fair comparison people don’t use sql databases for search. they use SOLR (or elasticsearch)
- 41. sometimes basic data structures! is all you need
- 42. sorted sets • score • add, remove, or update log(n) • elements in order, fast existence test, fast access to elements in the middle
- 43. breaking it down indie => probably irrelevant relevant i ! ! ! ! ! ! ! ! in ! ! ! ! ! ! ! ! ! ind! ! ! ! ! ! ! ! indi! ! ! ! ! ! ! ! indie
- 44. our collection The indie summer Indiana Industrial
- 45. one zset per n-gram ind [ ] [ ] [ ] indi indie
- 46. which is the same as ZADD ind, <score>, ZADD indi, <score>, ZADD indie, <score>,
- 47. searching 1 ind = $r.zevrange('ind', 0, 10)! ! 2 indie = $r.zevrange('indie', 0, 10)!
- 48. tag browsing
- 49. how?
- 51. a single tag 1 SELECT m.* FROM mixes m! 2 INNER JOIN taggings tg! 3 ON tg.mix_id = m.id! 4 WHERE tg.tag_id = 28! 5 LIMIT 10;! chill
- 52. two tags 1 SELECT m.* FROM mixes m! 2 INNER JOIN taggings tg! 3 ON tg.mix_id = m.id! 4 INNER JOIN taggings tg2! 5 ON tg2.mix_id = m.id! 6 WHERE tg.tag_id = 28! 7 AND tg2.tag_id = 12! 8 LIMIT 10;! chill acoustic
- 53. adding sort 1 SELECT m.* FROM mixes m! 2 INNER JOIN taggings tg! 3 ON tg.mix_id = m.id! 4 INNER JOIN taggings tg2! 5 ON tg2.mix_id = m.id! 6 WHERE tg.tag_id = 28! 7 AND tg2.tag_id = 12! 8 ORDER BY m.first_published_at! 9 LIMIT 10;!
- 56. zinterstore Time complexity: O(N*K)+O(M*log(M)) intersects two sets, then store them in a new different set we’ll get to this
- 57. chill
- 58. acoustic
- 59. chill acoustic +
- 60. chill acoustic 1 sets = ['chill', 'acoustic']! 2 cache_key = 'chill:acoustic:cache'! 3 $r.zinterstore(cache_key, sets, :weights => [1, 1])!
- 61. chill chill acoustic popular 1 sets = ['popular', 'chill', 'acoustic']! 2 cache_key = 'popular:chill:acoustic:cache'! 3 $r.zinterstore(cache_key, sets, :weights => [1, 0, 0])!
- 62. back to the BIG O thing O(N*K)+O(M*log(M)) - N: smallest input - K: number of sorted sets - M: number of elements of the resulting set
- 64. cool stuff worth checking out • pipelining • lua scripting • replication + persistence
- 65. lessons • it consumes a LOT of memory (duuuh) • sentinel for failure recovery • DO NOT USE “keys” in production • duplication is tricky