The document discusses audience counting at scale using Spark. It provides context on a customer that collects data on 1 billion user profiles to build models and make predictions. It then discusses motivation, counting fundamentals using HyperLogLog, and how to perform counting with Spark. Specifically, it shows how to transform log data into HyperLogLog summaries, work with the data in DataFrames, and provide examples of counting audiences and segments. It also provides some notes on writing custom Spark aggregation functions and using Spark as an in-memory SQL database for analytics.

1. AUDIENCE COUNTING @ SCALE
Boris Trofimoff
Sigma Software & Collective
@b0ris_1

2. AGENDA
About our customer1
Motivation or Why & What we are counting2
Counting Fundamentals3
Counting with Spark4
Spark Road Notes5

3.  Place where advertisers and end-users meet each user
 Collective Media is a full stack cookie serving company
USERS
 ~1B active user profiles
MODELS
 1000s of models built weekly
PREDICTIONS
100s of billions predictions daily
MODELING AT SCALEVOLUME
Petabytes of data used
VARIETY
Profiles, formats, screens
VELOCITY
100k+ requests per second
20 billions events per day
VERACITY
Robust measurements

4. MOTIVATION

5. HOW AUDIENCE IS CREATED

6. IMPRESSION LOG
AD SITE COOKIE IMPRESSIONS CLICKS SEGMENTS
bmw_X5 forbes.com 13e835610ff0d95 10 1 [a.m, b.rk, c.rh, d.sn, ...]
mercedes_2015 forbes.com 13e8360c8e1233d 5 0 [a.f, b.rk, c.hs, d.mr, ...]
nokia gizmodo.com 13e3c97d526839c 8 0 [a.m, b.tk, c.hs, d.sn, ...]
apple_music reddit.com 1357a253f00c0ac 3 1 [a.m, b.rk, d.sn, e.gh, ...]
nokia cnn.com 13b23555294aced 2 1 [a.f, b.tk, c.rh, d.sn, ...]
apple_music facebook.com 13e8333d16d723d 9 1 [a.m, d.sn, g.gh, s.hr, ...]

7. SEGMENT EXAMPLES
SEGMENT DESCRIPTION
a.m Male
a.f Female
b.tk $75k-$100k annual income
b.rk $100k-$150k annual income
c.hs High School
c.rh College
d.sn Single
d.mr Married

8. BUILDING AUDIENCE PROFILE

9. WHAT WE CAN DO WITH DATA
What is male/female ratio for people who have seen bmw_X5 ad
on forbes.com
Income distribution for people who have seen Apple Music ad
Nokia click distribution across different education levels

10. COUNTING
FUNAMENTALS

11. SQL?
SELECT count(distinct cookie_id)
FROM impressions
WHERE site = 'forbes.com' AND ad = 'bmw_X5' AND segment contains 'a.m'
Infinite combinations
Big Data => Big Latency for Hive,
Impala and Druid

12. CARDINALITY ESTIMATION ALGORITHMS
ACCURACY
MEMORY EFFICIENCY
ESTIMATE LARGE CARDINALITIES
PRACTIALITY
For a fixed amount of memory, the algorithm should provide
as accurate an estimate as possible. Especially for small
cardinalities, the results should be near exact
The algorithm should use the available memory efficiently and
adapt its memory usage to the cardinality
Multisets with cardinalities well beyond 1 billion occur on a
daily basis, and it is important that such large cardinalities can
be estimated with reasonable accuracy
The algorithm should be implementable and maintainable

13. HYPERLOGLOG AND OTHERS

14. AUDIENCE CARDINALITY
APPROXIMATION WITH HLL
Create Audience of people addressed by
unique identifiers (cookies)
Create Audience “Hash Sum” file with fixed
size regardless of audience size
Cardinalities ~ 109
with a typical accuracy of 2%
using 1.5KB of memory.
1.5KB
Create
Audience
Create
Hash

15. HYPERLOGLOG OPERATIONS
trait HyperLogLog {
def add(cookieId: String): Unit
// |A|
def cardinality(): Long
// |A ∪ B|
def merge(other: HyperLogLog): HyperLogLog
// |A ∩ B| = |A| + |B| - |A ∪ B|,
def intersect(other: HyperLogLog): Long
}
∪ ~ merge =
1.5KB 1.5KB 1.5KB
∩ ~ intrsct =
1.5KB 1.5KB
CARDINALITY
| |

16. COUNTING
WITH SPARK

17. IMPRESSION LOG TRANSFORMATION
AD SITE COOKIE IMPRESSIONS CLICKS SEGMENTS
bmw_X5 forbes.com 13e835610ff0d95 10 1 [a.m, b.rk, c.rh, d.sn, ...]
mercedes_2015 forbes.com 13e8360c8e1233d 5 0 [a.f, b.rk, c.hs, d.mr, ...]
nokia gizmodo.com 13e3c97d526839c 8 0 [a.m, b.tk, c.hs, d.sn, ...]
apple_music reddit.com 1357a253f00c0ac 3 1 [a.m, b.rk, d.sn, e.gh, ...]
nokia cnn.com 13b23555294aced 2 1 [a.f, b.tk, c.rh, d.sn, ...]
apple_music facebook.com 13e8333d16d723d 9 1 [a.m, d.sn, g.gh, s.hr, ...]
Splitting original impression log table into
two separate aggregated tables
• by campaign
• by segmentsSegments

18. FROM COOKIES TO HYPERLOGLOG
AD SITE COOKIES HLL IMPRESSIONS CLICKS
bmw_X5 forbes.com HyperLogLog@23sdg4 5468 35
bmw_X5 cnn.com HyperLogLog@84jdg4 8943 29
SEGMENT COOKIES HLL IMPRESSIONS CLICKS
Male HyperLogLog@65xzx2 235468 335
$100k-$150k HyperLogLog@12das1 569473 194

19. DATA FRAMES
> val adImpressions: DataFrame = sqlContext.parquetFile("/aa/${yy-mm-dd}/${hh}/audience")
> adImpressions.printSchema()
// root
// | -- ad: string (nullable = true)
// | -- site: string (nullable = true)
// | -- hll: binary (nullable = true)
// | -- impressions: long (nullable = true)
// | -- clicks: long (nullable = true)
> val segmentImpressions: DataFrame = sqlContext.parquetFile("/aa/${yy-mm-dd}/${hh}/segments")
> segmentImpressions.printSchema()
// root
// | -- segment: string (nullable = true)
// | -- hll: binary (nullable = true)
// | -- impressions: long (nullable = true)
// | -- clicks: long (nullable = true)

20. LET’S COUNT SOMETHING
import org.apache.spark.sql.functions._
import org.apache.spark.sql.HLLFunctions._
val bmwCookies: HyperLogLog = adImpressions
.filter(col("ad") === "bmw_X5")
.select(mergeHll(col("hll")).first() // -- sum(clicks)
val educatedCookies: HyperLogLog = segmentImpressions
.filter(col("segment") in Seq("College", "High School"))
.select(mergeHll(col("hll")).first()
val p = (bmwCookies intersect educatedCookies) / bmwCookies.cardinality()
Percent of college and high school education in BMW campaign?

21. SPARK
ROAD NOTES

22. WRITING OWN SPARK
AGGREGATION FUNCTIONS
case class MergeHLLPartition(child: Expression)
extends AggregateExpression with trees.UnaryNode[Expression] { ... }
case class MergeHLLMerge(child: Expression)
extends AggregateExpression with trees.UnaryNode[Expression] { ... }
case class MergeHLL(child: Expression)
extends PartialAggregate with trees.UnaryNode[Expression] {
override def asPartial: SplitEvaluation = {
val partial = Alias(MergeHLLPartition(child), "PartialMergeHLL")()
SplitEvaluation(
MergeHLLMerge(partial.toAttribute),
partial :: Nil )
}
}
def mergeHLL(e: Column): Column = MergeHLL(e.expr)
define function that will be
applied to each row
in RDD partition
define function that will take
results from different partitions
and merge them together
tell Spark how you want it to
split your computation
across RDD

23. AGGREGATION FUNCTIONS
PROS & CONS
Simple DSL and Native DataFrame look-like functions
Works much faster than solving this problem with Scala transformations on
top of RDD[Row]
Dramatic Performance Speed-Up via mutable state control (10x times)
UDF should be part of private Spark package, risk these interfaces might be
changed/abandoned in the future.

24. SPARK AS IN-MEMORY SQL DATABASE
BATCH-DRIVEN APP LONG-RUNNING APPCHANGE
Create
SparkContext
Run
Calculations
Destloy
SparkContext
Show
Result
Load
Data
Cache it
In memory
Receive
Request
Create
SparkContext
Show
Result
Run
Calculations
~ 500 GB (1 year history)
~40N occupied from ~200N cluster
Response time 1-2 seconds
Destloy
SparkContext

25. REFERENCES
 http://eugenezhulenev.com/blog/2015/07/15/interactive-audience-analytics-
with-spark-and-hyperloglog/
(Especial thanks to Eugene Zhulenev for his brilliant blog post)
 https://github.com/collectivemedia/spark-hyperloglog
 http://research.google.com/pubs/pub40671.html
 https://github.com/AdRoll/cantor
 http://tech.adroll.com/blog/data/2013/07/10/hll-minhash.html

26. THANK YOU!