Talk about how we estimate targeted audience count for newly created segment using probabilistic data structures, like hyperloglog and minhash

1. PROBABILISTIC DATA
STRUCTURES IN REAL LIFE
Valentin Bazarevsky

2. WHO THEY ARE?
Bloom Filter
LogLog Family
MinHash

3. BUSINESS CASE:
ESTIMATE YOUR AUDIENCE

4. SEGMENT BUILDER
15 Tb of transactional data
4h SLA

5. POSSIBLE SOLUTIONS
Brute force (15 TB of transactional data)
Sampling (1 % of users => 1.2 mb / b.o.)
Magic tool (?!)
Estimator
HyperLogLog allows to estimate > 1 000 000 000 sets of unique
elements with 1% error, and requires only 4kb memory
50 000 000 basic operations

6. OOPS…
Supports only Unions
But we need Intersections, Subtractions, Not
operators

7. HYPERLOGLOG INTUITION
00101010101010001111010101101 => a[2] = 0
10010101010100101010101001011 => a[9] = 1
00000101010100101010101110101 => a[0] = 1
01010101010100100101010101010 => a[5] = 1
01010000000000000000000000010 => a[5] = 23

8. INCLUSION-EXCLUSION PRINCIPLE

9. MINHASH
Store only x (8192) smallest hashes in set
Jaccard Distance

10. UNION OF INTERSECTIONS
A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ B)
A - B - C = A - (B ∪ C)

11. NOT OPERATOR
Subtraction

12. I WANT EVERYONE EXCEPT…
A and not B
Not A and Not B

13. CORNER CASES
|(A ∪ not(B)) ∩ C| => |A ∩ C|
|A ∪ not(B)| = |Everything| - |B| + |A ∩ B|
|A ∩ not(B)| => |A| - |A ∩ B|

14. ARCHITECTURE

15. ERROR RATE
Median = 5%
Percentile 75 = 8%