This document discusses using HyperLogLog (HLL) in Hive to efficiently estimate the number of unique elements or cardinality in big datasets. It describes how HLL provides fast approximate counting using probabilistic data structures. It covers implementing HLL as user-defined functions in Hive, comparing different open source implementations, and examples of using HLL to estimate unique visitors per day and in a rolling window.

1. HyperLogLog
in Hive How to count
sheep efficiently?
Phillip Capper: Whitecliffs Sheep
@bzamecnik

2. Agenda
● the problem – count distinct elements
● exact counting
● fast approximate counting – using HLL in Hive
● comparing performance and accuracy
● appendix – a bit of theory of probabilistic counting
○ how it works?

3. The problem: count distinct elements
● eg. the number of unique visitors
● each visitor can make a lot of clicks
● typically grouped in various ways
● "set cardinality estimation" problem

4. Small data solutions
● sort the data O(N*log(N)) and skip duplicates O(N)
○ O(N) space
● put data into a hash or tree set and iterate
○ hash set: O(N^2) worst case build, O(N) iteration
○ tree set: O(N*log(N)) build, O(N) iteration
○ both O(N) space
● but: we have big data
Example:
~100M unique values in 5B rows each day
32 bytes per value -> 3 GB unique, 150 GB total

5. Problems with counting big data
● data is partitioned
○ across many machines
○ in time
● we can't sum cardinality of each partition
○ since the subsets are generally not disjoint
○ we would overestimate
count(part1) + count(part1) >= count(part1 ∪ part2)
● we need to merge estimators and then estimate
cardinality
count(estimator(part1) ∪ estimator(part2))

6. SELECT COUNT(DISTINCT user_id)
FROM events;
single reducer!
Exact counting in Hive

7. Exact counting in Hive – subquery
SELECT COUNT(*) FROM (
SELECT 1 FROM events
GROUP BY user_id
) unique_guids;
Or more concisely:
SELECT COUNT(*) FROM (
SELECT DISTINCT user_id
FROM events
) unique_guids;
many reducers
two phases
cannot combine
more aggregations

8. Exact counting in Hive
● hive.optimize.distinct.rewrite
○ allows to rewrite COUNT(DISTINCT) to subquery
○ since Hive 1.2.0

10. Probabilistic counting
● fast results, but approximate
● practical example of using HLL in Hive
● more theory in the appendix

11. ● klout/brickhouse
○ single option
○ no JAR, some tests
○ based on HLL++ from stream-lib (quite fast)
● jdmaturen/hive-hll
○ no options (they are in API, but not implemented!)
○ no JAR, no tests
○ compatible with java-hll, pg-hll, js-hll
● t3rmin4t0r/hive-hll-udf
○ no options, no JAR, no tests
Implementations of HLL as Hive UDFs

12. ● User-Defined Functions
● function registered from a class (loaded from JAR)
● JAR needs to be on HDFS (otherwise it fails)
● you can choose the UDF name at will
● work both in HiveServer2/Beeline and Hive CLI
ADD JAR hdfs:///path/to/the/library.jar;
CREATE TEMPORARY FUNCTION foo_func
AS 'com.example.foo.FooUDF';
● Usage:
SELECT foo_func(...) FROM ...;
UDFs in Hive

13. ● to_hll(value)
○ aggregate values to HLL
○ UDAF (aggregation function)
○ + hash each value
○ optionally can be configured (eg. for precision)
● union_hlls(hll)
○ union multiple HLLs
○ UDAF
● hll_approx_count(hll)
○ estimate cardinality from a HLL
○ UDF
HLL can be stored as binary or string type.
General UDFs API for HLL

14. ● Estimate of total unique visitors:
SELECT hll_approx_count(to_hll(user_id))
FROM events;
● Estimate of total events + unique visitors at once:
SELECT
count(*) AS total_events
hll_approx_count(to_hll(user_id))
AS unique_visitors
FROM events;
Example usage

15. Example usage
● Compute each daily estimator once:
CREATE TABLE daily_user_hll AS
SELECT date, to_hll(user_id) AS users_hll
FROM events
GROUP BY date;
● Then quickly aggregate and estimate:
SELECT hll_approx_count(union_hlls(users_hll))
AS user_count
FROM daily_user_hll
WHERE date BETWEEN '2015-01-01' AND '2015-01-31';

16. https://github.com/klout/brickhouse - Hive UDF
https://github.com/addthis/stream-lib - HLL++
$ git clone https://github.com/klout/brickhouse
disable maven-javadoc-plugin in pom.xml (since it fails)
$ mvn package
$ wget http://central.maven.
org/maven2/com/clearspring/analytics/stream/2.3.0/stream-
2.3.0.jar
$ scp target/brickhouse-0.7.1-SNAPSHOT.jar
stream-2.3.0.jar cluster-host:
cluster-host$ hdfs dfs -copyFromLocal *.jar
/user/me/hive-libs
Brickhouse – installation

17. Brickhouse – usage
ADD JAR /user/zamecnik/lib/brickhouse-0.7.1-15f5e8e.jar;
ADD JAR /user/zamecnik/lib/stream-2.3.0.jar;
CREATE TEMPORARY FUNCTION to_hll AS 'brickhouse.udf.hll.
HyperLogLogUDAF';
CREATE TEMPORARY FUNCTION union_hlls AS 'brickhouse.udf.
hll.UnionHyperLogLogUDAF';
CREATE TEMPORARY FUNCTION hll_approx_count AS 'brickhouse.
udf.hll.EstimateCardinalityUDF';
to_hll(value, [bit_precision])
● bit_precision: 4 to 16 (default 6)

18. Hive-hll usage
ADD JAR /user/zamecnik/lib/hive-hll-0.1-2807db.jar;
CREATE TEMPORARY FUNCTION hll_hash as 'com.kresilas.hll.
HashUDF';
CREATE TEMPORARY FUNCTION to_hll AS 'com.kresilas.hll.
AddAggUDAF';
CREATE TEMPORARY FUNCTION union_hlls AS 'com.kresilas.hll.
UnionAggUDAF';
CREATE TEMPORARY FUNCTION hll_approx_count AS 'com.
kresilas.hll.CardinalityUDF';

19. We have to explicitly hash the value:
SELECT
hll_approx_count(to_hll(hll_hash(user_id)))
FROM events;
Options for creating HLL:
to_hll(x, [log2m, regwidth, expthresh, sparseon])
hardcoded to:
[log2m=11, regwidth=5, expthresh=-1, sparseon=true]
Hive-hll usage

20. Nice things
● HLLs are additive
○ can be computed once
○ various partitions can be merged and estimated for
cardinality later
● we can count multiple unique columns at once
○ no need to subquery
○ we can do wild grouping (by country, browser, …)
● HLLs take only little space

21. Rolling window
-- keep reasonable number of task for month of data
SET mapreduce.input.fileinputformat.split.maxsize=5368709120;
-- keep low number of output files (HLLs are quite small)
SET hive.merge.mapredfiles=true;
-- maximum precision
SET hivevar:hll_precision=16;
-- HLL for each day
CREATE TABLE guids_parquet_hll AS
SELECT
'${year}' AS year,
'${month}' AS month,
day,
to_hll(guid, ${hll_precision}) AS guid_hll
FROM parquet.dump_${year}_${month}
GROUP BY day;

22. -- for each day estimate number of guids 7-days back
CREATE TABLE zamecnik.guids_parquet_rolling_30_day_count
AS
SELECT
`date`,
hll_approx_count(guids_union) AS guid_count
FROM (
SELECT
concat(`year`, '-', `month`, '-', `day`) as `date`,
union_hlls(guid_hll) OVER w AS guids_union
FROM guids_parquet_hll
WINDOW w AS (
ORDER BY `year`, `month`, `day` ROWS 6 PRECEDING
)
) rolling_guids;
Rolling window

23. ● when JARs are not on HDFS the query fails (why?)
● computing on many days of raw clickstream fails in
Beeline (works in Hive CLI), parquet is ok
● HIVE-9073 WINDOW + custom UDAF → NPE
○ fixed in Hive 1.2.0
● DISTRO-631
Pitfalls

24. Approximation error
● Typically < 1-2 %
● Can be controlled by the parameters
● Example: 1 year of guids

25. Appendix – more interesting things

26. ● trade-off: some approximation error for far better
performance and memory consumption
● sketch - streaming & probabilistic algorithm
● KMV - k minimal values
● linear counter
● loglog counter
Probabilistic counting

27. LogLog counter
● run length of initial zeros
● multiple estimators (registers)
● stochastic averaging
○ single hash function
○ multiple buckets
● hash → (register index, run length)

28. Linear counter
m = 20 # size of the register
register = bitarray(m) # register, m bits
def add(value):
h = mmh3.hash(value) % m # select bit index
register[h] = 1 # = max(1, register[h])
def cardinality():
u_n = register.count(0) # number of zeros
v_n = u_n / m # relative number of zeros
n_hat = -m * math.log(v_n) # estimate of the set cardinality
return n_hat

29. ● structure like loglog counter
● harmonic mean to combine registers
● correction for small and large cardinalities
● values needs to be hashed well – murmur3
HyperLogLog (HLL)

30. HLL union
● just take max of each register value
● no loss – same result as HLL of union of streams
● parallelizable
● union preserves error bound, intersection/diff do not

31. Further reading
● very nice explanation of HLL
● Probabilistic Data Structures For Web Analytics And
Data Mining
● Sketch of the Day: HyperLogLog — Cornerstone of a
Big Data Infrastructure
● HyperLogLog in Pure SQL
● Use Subqueries to Count Distinct 50X Faster
● It is possible to combine HLL of different sizes

32. Papers
● HyperLogLog in Practice: Algorithmic Engineering of
a State of The Art Cardinality Estimation Algorithm
● https://github.com/addthis/stream-lib#cardinality

33. Other problems & structures
● set membership – bloom filter
● top-k elements – count-min-sketch, stream-summary