...viewers start leaving if video doesnt play in 2 seconds...viewers start leaving ifvideo doesnt play in 2seconds...and every second ofadditional delay about 6%more viewers jumping ship! Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs. S. Shunmuga Krishnan, Ramesh K. Sitaraman, 2012
Large ecosystemGlobal audience User growth Performance
Bitmapsgenre:1 -> 010101001genre:2 -> 010010011type:music -> 011000001intersect -> 010000001 (ids: 2 and 9)Good: speed for intersect & memory efficiencyBad: get the ids and the real data. sorting!,paging...
Everything is a setgenre:1 = [1, 2, 4]type:1 = [1, 2, 5]type:2 = [3, 4]Good: Sparse (too many 0s with bitmaps)Complexity O(n+m) (m is sets, n elements inthe smallest set)genre:1 -> 10 elementsvideos -> 100K elementscomplexity: O(10)
Building our own indexesData storedKeeping track of the indexesHow do we find data...redis.call(sort, my_set, BY, v:*->created_at,desc, LIMIT, offset, count, GET, v:*->details)
HoldbacksOld system: not in (id, id, id)first attempt: hash with a list of rulespermutations (too many countries in the world!)CAP -> 10GB. meh~alias matching permutations: 800mb ;)Redis 32bit, even better!CAP is just another set! (well, but a DIFF!)
Hacking RedisVfind: Building our own Redis functionSetlets: Pre-calculated sorted listsMost requests 18~20ms, some cases 100ms(depends on the bigger set)Vfind only gets content to fill 1 page: 15msPaging just showing more: 9msSerialization of jsons: 5msEnough.. for now!
ListsA list is just a sorted set. E.g. a list ofsubscriptions, list of featured content...Is a set!You can apply holdbacks or any other filter.
Many web services Each vertical is a source of truth Logical and operational reasons Everything routed through api.viki.io Oceanus Activities Aphrodite Gaia (Videos) (Behaviour) (Community) (Users)
Queue Centralized queue for events and messages Event-driven web services Messages must be idempotent Message / Events Queue