Talk at the London Data Science Meetup, March 23 2016 http://www.meetup.com/Data-Science-London/events/229755935/ At Schibsted Technology we have been building predictive pipelines in order model our users' attributes and behaviour traits, such as age, gender, interests and (buying) intent etc. Predictive models on the aforementioned properties are fundamental in order to facilitate core parts of the business such as our ad-targeting platform. In this talk I will present the challenges that we had to overtake in order to put together our scalable predictive pipelines, and how we utilized Spark ML features such as User Defined Aggregate Functions (UDAF) in order to achieve this.

1. THE FUTURE IS NOW

2. Scalable Predictive Pipelines
with Spark and Scala
Dimitris Papadopoulos

3. 3
About Schibsted

4. 4
About Schibsted

5. 5
About Schibsted

6. 6
Event Tracking Data

7. 7
Event Tracking Data

8. 8
Event Tracking Data

9. 9
Event Tracking Data

10. 10
Data Science Tasks
Data
Model
Results
Preprocessing

11. 1. Using Spark ML Pipelines
2. Scalable Pipelines
11
Outline

12. 12
Pipeline

13. 13
Pipeline

14. 14
Pipeline

15. 15
Not a pipe

16. 16
Pipeline Stage
● One or more inputs ● Strictly one output

17. 17
Pipeline Stage
● One or more inputs ● Strictly one output
● Closed under concatenation

18. 18
Pipeline Stage
● One or more inputs ● Strictly one output
● Closed under concatenation
● Standalone and runnable
● Spark™ ML inside

19. 19
Spark ML Pipelines

20. 20
Spark ML Pipelines
Using a Pipeline to train a model

21. 21
Spark ML Pipelines
Using a PipelineModel to get predictions

22. 22
Peek inside a Spark pipeline
It’s a Pipeline

23. 23
Peek inside a Spark pipeline
It’s a Pipeline
plain Spark API

24. 24
Peek inside a Spark pipeline
It’s a Pipeline
plain Spark API
From DataFrame to a Model

25. 25
Peek inside a Spark pipeline
Instantiating a Pipeline
Running it!

26. 26
Example Pipeline
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions

27. 27
Example Pipeline
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEvaluator:
computes performance metrics, e.g.
accuracy and area under ROC

28. 28
Scalable Pipelines: pain points
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEvaluator:
computes performance metrics, e.g.
accuracy and area under ROC

29. 29
Scalable Pipelines: pain points
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEvaluator:
computes performance metrics, e.g.
accuracy and area under ROC
Input: 1 day’s / 7 days’ worth of events data.
Larger lookbacks needed for better accuracy.

30. 30
More data for better performance
Performance of three different pipelines,
vs lookback length (1, 7, 30, 45)

31. 31
Scalable Pipelines: pain points
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEvaluator:
computes performance metrics, e.g.
accuracy and area under ROC
What will happen if we try to process
30 days worth of data (e.g. 3B events) ???

32. 32
Scalable Pipelines: pain points
Memory and processing heavy:
● In one use-case, for 7 days lookback (~7 x 100M events) we used to need 20 Spark
executors with 22G of memory each.
Not easily scalable
● As the lookback increases
● As more and more sites are incorporated into our pipelines
Redundant processing
● For K days of lookback, we are repeating processing of K - 2 days worth of data, when we
run the pipeline every day, in a rolling window fashion.
“What will happen if we try to process
30 days worth of data (e.g. 3B events) ???”

33. 33
Saved by Algebra
● The operations (op) along with the corresponding data structures (S) that
we are interested in are monoids.
○ Associative:
■ for all A,B,C in S, (A op B) op C = A op (B op C)
○ Identity element:
■ there exists E in S such that for each A in S, E op A = A op E = A
● Examples:
○ Summation: 1 + 2 + 3 + 4 = (1 + 2) + (3 + 4)
○ String array concatenation: [“foo”] + [“bar”] + [“baz”] = [“foo”, “bar”] + [“baz”]

34. 34
Scalable Pipelines: in monoids fashion
● Split the aggregations in smaller chunks
○ i.e. pre-process events per user and single day (not over the entire lookback)

35. 35
Scalable Pipelines: in monoids fashion
● Split the aggregations in smaller chunks
○ i.e. pre-process events per user and single day (not over the entire lookback)
● Make one (or multiple) day aggregates and combine
○ i.e. aggregate over the pre-preprocessed events per user and day

36. 36
Scalable Pipelines: in monoids fashion
● Split the aggregations in smaller chunks
○ i.e. pre-process events per user and single day (not over the entire lookback)
● Make one (or multiple) day aggregates and combine
○ i.e. aggregate over the pre-preprocessed events per user and day
● It’s like trying to ...eat an elephant: one piece at a time!

37. 37
Scalable Pipelines: building blocks
● Imagine we had a
MapAggregator, for
aggregating maps of [String-
>Double].

38. 38
Scalable Pipelines: building blocks
● Imagine we had a
MapAggregator, for
aggregating maps of [String-
>Double].
● The spec for such an
aggregator implemented in
Scala on Spark could look
like this. :-)

39. 39
Scalable Pipelines: building blocks
● Imagine we had a
MapAggregator, for
aggregating maps of [String-
>Double].
● The spec for such an
aggregator implemented in
Scala on Spark could look
like this. :-)

40. 40
Scalable Pipelines: building blocks
● In Spark we can define our own functions, also known as User Defined
Functions (UDF)
● A UDF takes as arguments one or more columns, and returns some
output.
● It gets executed for each row of the DataFrame.
● It can also be parameterized.
● e.g. val myUDF = udf((myArg: myType) => ...)

41. ● Since Spark 1.5, we can also define our own User Defined Aggregate
Functions (UDAF).
● UDAFs can be used to compute custom calculations over groups of input
data (in contrast, UDFs compute a value looking at a single input row)
● Examples: calculating geometric mean or calculating the product of values for
every group.
● A UDAF maintains an aggregation buffer to store intermediate results for
every group of input data.
● It updates this buffer for every input row.
● Once it has processed all input rows, it generates a result value based on
values of the aggregation buffer.
41
Scalable Pipelines: UDAF

42. 42
Scalable Pipelines: UDAF
A User Defined
Aggregate Function
Implementation of abstract
methods

43. 43
Scalable Pipelines: adding a new stage
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEvaluator:
computes performance metrics, e.g.
accuracy and area under ROC
What will happen if we try to process
30 days worth of data (e.g. 3B events) ???

44. 44
Scalable Pipelines: adding a new stage
EventCoalescer:
collects raw pulse events
(Json) into substantially
fewer files (Parquet)
UserAccountPreprocessor:
harmonises schemas of user
accounts across sites (if
necessary). Provides ground
truth data for training.
EventPreprocessor:
aggregates events per
user and day
GenderPredictor:
creates labels and features,
trains classifier & computes
predictions
GenderPerformanceEval
uator:computes performance
metrics, e.g. accuracy and
area under ROC
EventAggregator:
aggregates pre-processed
events per user over
multiple days (lookback)

45. 45
Scalable Pipelines: Aggregating Events

46. 46
Scalable Pipelines: Aggregating Events
It’s a Transformer

47. 47
Scalable Pipelines: Aggregating Events
It’s a Transformer
DataFrame in , DataFrame out

48. 48
Scalable Pipelines: Aggregating Events
It’s a Transformer
DataFrame in , DataFrame out
Aggregating maps
of feature frequency
counts

49. 49
Scalable Pipelines: closing remarks
● With User Defined Aggregate Functions, we have reduced the workload of
our pipelines by a factor of 20!

50. 50
Scalable Pipelines: closing remarks
● With User Defined Aggregate Functions, we have reduced the workload of
our pipelines by a factor of 20!
● Obvious gains: freeing up resources that can be used for running even
more pipelines, faster, over even more input data

51. 51
Scalable Pipelines: closing remarks
● Needles to say, more factors contribute towards a scalable pipeline:
○ Performance tuning of the Spark cluster
○ Use of a workflow manager (e.g. Luigi) for pipeline orchestration

52. 52
Scalable Pipelines: closing remarks
● Needles to say, more factors contribute towards a scalable pipeline:
○ Performance tuning of the Spark cluster
○ Use of a workflow manager (e.g. Luigi) for pipeline orchestration
● But each one of these is a topic for a separate talk (Carlos? Hint, hint!) :-)

53. 53
Q/A
Thank you!

54. 54
Shameless plug
We are hiring!
Across all our hubs
in London, Oslo, Stockholm, Barcelona
for Data Science, Engineering, UX and Product roles
https://jobs.lever.co/schibsted
spt-recruiters@schibsted.com