In this talk we will dive deep into data pre-processing or data preparation part of Data Scientist work. Why data pre-processing is such an important topic to pay attention for aspiring Data Scientists / Machine Learning Engineers? How to process TBs of static and moving, aka streaming, schemaless data? How to ensure horizontal scalability for up to PBs when you expect such growth? We'll give you several insights how Apache Spark, a fast and general engine for large-scale data processing, and Apache Kafka helped us to deal with 80% of Data Scientists’ work. Why do you need such high-caliber tools as Spark or Kafka, when is it viable to use them and how to avoid such tools? What are the pitfalls of distributed processing using Spark and Kafka? How can Google Cloud Platform help and save costs up to 90%? We'll share what we've learned along the (hard) way.

1. MEET
OUR
TEAM
WRITE HERE SOMETHING
#1 Data Science Club
sli.do/exponea
17/18 Summer

2. MEET
OUR
TEAM
WRITE HERE SOMETHING
Batch (Spark) and Streaming (Kafka)
Data Preprocessing
Matus Cimerman

3. What do we do

4. FULL-STACK MARKETING CLOUD

5. SOME NUMBERS
● 150+ employees from 12 countries
● 29 average age
● 8 offices in 5 countries on 4 continents
● 1000%+ growth over the last 2 years

6. AI TEAM
● Recommendations,
● Propensity to buy during the session,
● Optimal time to send an email,
● Tech stack: Python, Gensim, Kubernetes, Spark, Go, TF, ...

7. How do we collect data

8. We don't scrape any websites

9. Arbitrary schema-less JSON objects
Storage: IMF, MongoDB, HDFS
Data Sources
Events
Columnar storage (CSV, SQL DB, …)
Storage: Elastic, HDFS
Products
Static attributes (age, location, …)
Dynamic attributes (# page visits, …)
Storage: MongoDB, IMF,
HDFS (parquet)
Customers

10. Long-term storage (archive)
● Events: Newline delimited JSON
● Customers: Parquet file
● Products: CSV, soon BigTable

11. Why even this talk at all?

12. Data preparation & preprocessing is at least
80% of Data Scientist job

13. ML Algorithms don't eat raw data

14. We expect to growth 10-20x in the upcoming
1-2 years, so scale-out is critical

15. Batch (Spark)

16. Batch processing is done mostly by Apache
Spark jobs, sadly written in Python

17. EASY, 12 LINES OF CODE
+ 500 LOC of boilerplate

18. Batch processing and Spark
things to consider first
1. Try Kappa first.
2. If not possible, try to implement Lambda arch.
3. Combine them all!

19. Streaming (Kafka)

20. Streaming ETL

21. Streaming ETL

22. Streaming ETL

23. EASY, 8 LINES OF CODE
+ 500 LOC of boilerplate

24. Infrastructure

25. Legacy bare-metal

26. Google Cloud Platform for the Win

27. Google Cloud Platform for the win

28. Google Cloud Platform
more moving parts
● Inputs for ML algorithms from Storage.
● Training WORM1 to Storage.
● Kubernetes for orchestration.
● Large scale models deployment, K8s.
● Easy monitoring thanks to Stackdriver.
1Write once read many

29. Lessons learned

30. Unified
PROS
Scale-out
No vendor
lock-in
Python

31. CONS
Distributed
debugging
Cluster size
guessing game
Python

32. Future
● Spark Streaming + for all transformations.
● Batch in Spark for daily data consolidation.
● Kafka Streams for real-time use-cases.

33. That's all folks

34. Matus Cimerman
matus.cimerman@exponea.com
www.exponea.com/internship

35. PALO ALTO, CA
456 University Ave
Palo Alto, CA 94301
+1 (650) 440-7297
PRAGUE, CZ
Rohanské nábřeží 687/29,
186 00 Prague, Czechia
+420 601 372 909
BRATISLAVA, SK
Twin City B, Mlynské Nivy 12,
821 09, Bratislava, Slovakia
+421 948 127 332
WARSAW, PL
Postępu 14, 02-676
Warsaw, PL
+48 603 663 766
MOSCOW, RU
10c1 Kozhevnicheskaya Street
115114, Moscow, RU
+7 (495) 120 26 53
LONDON, UK
41 Corsham Street
London N1 6DR, UK
+44 (0) 203 086 8894
MANCHESTER, UK
1 Spinningfields, Quay Street
Manchester M3 3JE
+44 (0) 203 086 8894
EDINBURGH, UK
20/6 Fountainhall Road
Edinburgh EH9 2NN
+44 (0) 203 086 8894
www.exponea.com