1.
Agency for Data Science
Machine learning & AI
Mathematical modelling
Data strategy
Stanimir Dragiev,Tammo Krueger
Towards scalable fraud detection
A journey from scikit-learn to Spark
idalab talks | January 20th 2017

2.
A Journey from
Scikit-learn to
Spark
20.1.2017
Stanimir Dragiev & Tammo Krueger
tammok @ssdpd

3.
2
EUROPE’S LEADING FASHION PLATFORM
15 countries
3 fulfillment centers
+19 million active customers
+€3.0 billion revenue
+160 million visits per month
+1.600 employees in tech
tech.zalando.com
radicalagility.org

4.
3
HOW IT LOOKS

5.
4
TECHNOLOGY HISTORY
~ 30
1600+

6.
5
WHAT WE DO
FRAMEWORK CANDY
PREVIOUS SETUP
REDESIGN
SPARK ODDITIES
OUTLINE

7.
WHAT WE DO

8.
7
TEAM PAYMENT ANALYTICS
Goal: Estimate fraud risk for incoming orders
This requires us to master:
Machine Learning
Production code
A diverse tech stack:
(Spark, Scala, R, SQL, AWS, Jenkins, Docker, Python, sh, ...)

9.
8
MODELING FRAUD PROBABILITIES
Fraud probability pfraud
beyond business rules ...
… but can be modeled via machine learning:
data-driven
unbiased
reproducible
Challenges:
multiple application domains
large variety of models
frequent updates

10.
PREVIOUS SETUP

11.
10
ML ARCHITECTURE

12.
11
DEPLOYMENT OF MODELS

13.
12
SCALABILITY
1
https://blog.zalando.com/en/blog/how-zalando-becoming-online-fashion-platform-europe
2
https://corporate.zalando.com/en/zalando-continues-high-growth-path
Number of orders increase2
Connect all stakeholders in
fashion: online and offline
retailers, advertising agencies,
logistic services, ...
online shop -> fashion platform1
Scalability of our services is now major concern
55.3m41.4m 68.7m
(projected)
2014 20162015

14.
13
PAIN POINTS OF PREVIOUS SETUP
No horizontal scaling of learning (memory bottleneck)
***
Python: fast initial development, but: bad maintenance
***
Too many deployment units for prediction service
***
Coupled learning and data fetching (bottleneck: EXAsol)
***
Learning on in-house cluster ⇝ bottleneck

15.
REDESIGN

16.
15
SPARK IN SCALA ON AWS
• (Nearly) unlimited
computing power
• Cheap data
storage for
unifying input
data
• Type safety
• Real
multithreading
• Seamless
horizontal scaling
• High level APIs
for machine
learning (MLlib)
Spark promises Scala promises AWS promises

17.
16
HORIZONTAL SCALING

18.
17
LEARNING TIME
• Scenario 1: Old solution
• Python-based learning framework
• In-house cluster on a single machine with 10 cores
• Scenario 2: New solution
• Spark-based learning framework
• run on AWS with 1 master and 5 workers
→ Overall learning time drops by factor two

19.
18
DATA SIZE
• Py/slurm
• limited by single node’s memory
• Spark/AWS
• horizontal scaling
• 10x more data points in same time
→ Able to train with more data; higher accuracy

20.
19
SCALA RUNTIME
Real multithreading on the JVM

21.
20
PREDICTION TIME

22.
21
LiNES OF CODE
Language Comment Code
Scala 1192 3322
Python 411 6314
http://cloc.sourceforge.net

23.
SPARK ODDITIES

24.
23
SPARSE DATA
submodels took too long to learn,
sparse and of high dimensionality,
Spark supports sparse data points…

25.
24
SPARSE DATA
submodels took too long to learn,
sparse and of high dimensionality,
Spark supports sparse data points…
but what does the optimizer do:

26.
25
SPARSE DATA: THE CONDENSER

27.
26
OTHER ODDITIES
• Performance of random forest different
• Double saving of random forests
• A lot of (undocumented) settings
• ml (dataframe, pipeline) versus mllib (RDD)
• Distributed learning hard to debug
• balancing of data sets just for linear methods

28.
FRAMEWORK CANDY

29.
28
….
UNIFY DATA SOURCES ON AWS

30.
29
HIERARCHICAL MODELS

31.
30
TYPE SAFETY FOR CONFIGURATION

32.
31
LIFECYCLE OF A MODEL

33.
SUMMARY

34.
33
REDESIGN CONCLUSION
Steep learning curve for Scala
Hard to debug distributed execution
Maturity level of MLlib (as of version 2.0)
Spark & AWS eliminated all pain points
Speedup in learning and execution
Better support for model lifecycle

35.
THANK YOU!
Stanimir Dragiev & Tammo Krueger
https://tech.zalando.com/blog/scalable-fraud-detection-fashion-platform