HyperLogLog is dedicated to estimating the number of distinct elements (the cardinality) of very large data ensembles. Using an auxiliary memory of m units (typically, "short bytes''), HyperLogLog performs a single pass over the data and produces an estimate of the cardinality such that the relative accuracy (the standard error) is typically about 1.04/sqrt(m). This improves on the best previously known cardinality estimator, LogLog, whose accuracy can be matched by consuming only 64% of the original memory.

1. HyperLogLog
Eugen Kosteev, SE @ Tubular

2. Paper

3. Math

4. Math

5. Definitions and facts
- harmonic mean:
- if each of a collection of m independent random variables has standard
deviation σ, then their arithmetic mean has standard deviation σ/√m
- the 68–95–99.7 rule

6. Structure
1. The HyperLogLog algorithm
2. Mean value analysis
3. Variance and other stories
4. Discussion

7. Problem statement and naive solution
- given a multiset M, find number of distinct elements
- hash table on M?
- sort(M) + scroll?

8. Issues
- big cardinality of data set, no space to store
- data set stored in distributed environment

9. Examples
- Google search, distinct number of search queries
- traffic monitoring (dos attacks)
- correlation of genomes in human DNA, distinct subwords of fixed size k

10. Constraints
- crucial factor is then to relax the constraint of computing the value of the
cardinality exactly
- allows to apply whole range of probabilistic algorithms
- in 99% practical applications, a tolerance of a few percents on the result
is acceptable

11. Idea of probabilistic counting
- imagine I flip a coin many times and count the number of consecutive
heads before the first tail
- repeat it several times
- Sequence 1: HHHT
- Sequence 2: HT
- Sequence 3: HHT

12. What if?
- what if I say you that I get 1000 sequences and got 2 as maximum index
- what if I say you that I get 10 sequences and got 100 as maximum index
- X ≈ 2k
, X - number of sequences, k - maximum index

13. Prototype
- let h: D → {0, 1}∞
- h(v1
) = 0001001110011...
- h(v2
) = 0100100110011...
- h(v3
) = 0010011010011...
- observe 0p−1
1 patterns, ρ(h(v)) = p, v∈M
- k = maxv∈M
ρ(h(v))
- cardinality(M) ≈ 2k

14. m different hash functions, drawbacks
- complexity = O(Nm)
- it would necessitate a large set (e.g.: 104
to decrease error by 102
) of
independent hashing functions, for which no construction is known

15. Split one problem into m sub-problems
- split M into m buckets
- estimate cardinality of each bucket (X/m)
- compute mean of all estimations
- multiply result by m (get estimation with accuracy σ/√m)

16. HyperLogLog
mZ - harmonic mean
≈ 0.7213

17. Theorem 1

18. Poissonization

19. Edge cases
- small cardinalities (< m)
- large cardinalities (> , t - number of bits in hash function)

20. Final implementation

21. Comparison

22. Types of observables

23. Chart, relative error
n=107
, m=1024, 3%

24. Histogram, relative error
+/- 3σ = 99.7%

25. Data structure
- estimate the cardinality of union of multiple sets. It is natural to combine
multiple HLL’s; simply take the largest count of consecutive leading 0’s
from all the HLL’s
- estimate the overlap of two sets. Since |A ∩ B| = |A| + |B| – |A ∪ B|, the
overlap of two sets can be calculated from the cardinality of each set and
the cardinality of their union

26. Usage
- Elasticsearch (HLL++, “precision_threshold”)
- Redis (HLL++, PF*, 12k + 8 bytes)
- Spark (HLL++, approx_count_distinct, “rsd”)
- ...

27. References
1. http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
2. https://static.googleusercontent.com/media/research.google.com/en//pubs
/archive/40671.pdf
3. http://dblab.kaist.ac.kr/Publication/pdf/ACM90_TODS_v15n2.pdf

28. Thanks! Questions?