Zalandos AI-driven products and distributed landscape of analytical data marts cannot wait for long-running, hard-to-recover, monolithic batch jobs taking all night to calculate already outdated data. Modern data integration pipelines need to deliver fast and easy to consume data sets in high quality. Based on Spark Streaming and Delta, the central data warehousing team was able to deliver widely-used master data as S3 or Kafka streams and snapshots at the same time. The talk will cover challenges in our fashion data platform and a detailed architectural deep dive about separation of integration from enrichment, providing streams as well as snapshots and feeding the data to distributed data marts. Finally, lessons learned and best practices about Delta's MERGE command, Scala API vs Spark SQL and schema evolution give more insights and guidance for similar use cases.

1. BUILDING
STREAMING
DATA WAREHOUSES
SEBASTIAN HEROLD
2019-11-05

2. 2
# Principal Data Engineer / Architect
# 7y @ Immo-/Scout24
# DataDevOps Manifesto
# Data Platform
# 2y @ Zalando
# ML Productivity
# Streaming DWH
@heroldamus
Sebastian Herold

3. 3
WE BRING FASHION TO PEOPLE
2008-2009
2010
2012-2013
2011
2018
17 markets
9 fulfillment centers
>28M active customers
5.4B revenue 2018
>300M visits/month
>14k employees
>400k product choices
>80% visits from mobile

4. 4
TECH@SCALE
>350 accounts
>100 clusters
>250 teams
>5 data lakes
API
>800 micro services

5. 2019-11-05
BEYOND CLASSICAL ANALYTICS:
ML EVERYWHERE!

6. WHY OUR CENTRAL DWH
DOES NOT SUCCEED
ANYMORE?

7. 7
DB
DB
Early Years Rise of MicroServices ML + Data Mesh Chaos
DB
DWH History
DWH
Kafka
MPP
Data Lake
Data Lake
Data Lake
Data Lake
DWH
DWH
DWH
ML
ML
ML
ML
Data Lake

8. HEAVY INTEGRATION OF
UNSTRUCTURED DATA
INTO RELATIONAL TABLES
DRAWBACKS OF CENTRAL DWH

9. DATASETS ARE NEEDED
DISTRIBUTED
DRAWBACKS OF CENTRAL DWH

10. LOWER LATENCY REQUIRED BY
AI USE-CASES,
OTHER DATA WAREHOUSES,
NEAR-REALTIME USE-CASES
DRAWBACKS OF CENTRAL DWH

11. MULTIPLE TEAMS DO SAME
LOW-LATENCY EVENT INTEGRATION
DRAWBACKS OF CENTRAL DWH

12. HEAVY INTEGRATION OF
UNSTRUCTURED DATA
INTO RELATIONAL TABLES
DATASETS ARE NEEDED
DISTRIBUTED
LOWER LATENCY REQUIRED BY
AI USE-CASES,
OTHER DATA WAREHOUSES,
NEAR-REALTIME USE-CASES
MULTIPLE TEAMS DO SAME
LOW-LATENCY EVENT
INTEGRATION

13. 13
SALES ORDER EXAMPLE
order.created
order_id,
order_date,
items,
...
shipment.created
order_id,
shipping_date,
shipped_items,
...
payment.done
payment_id,
payment_date,
order_id,
...
item.returned
order_id,
return_date,
returned_item,
...
sales-order
order_id,
order_date,
payment_id,
payment_date,
items:
shipped_at,
returned_at,
...
calculated_1,
calculated_2
...

14. 14
ADVANTAGE OF NESTED STRUCTURE
sales-order
order_id,
order_date,
payment_id,
payment_date,
items:
shipped_at,
returned_at,
...
calculated_1,
calculated_2
...
# Contains the whole entity
# Can be streamed as new event
# Can be flattened for analytics
# Easier feature extraction for ML

15. 15
STREAMING ARCHITECTURE
Kafka S3
changes
S3
snapshots
Real-time Analytics
Machine Learning
Stream Processing
DWHs
Machine Learning
Batch Processing

16. 16
CHALLENGES IN STREAMING DATA INTEGRATION
# Integration of forecasts
# Find the right state store
# Cyclic dependencies
# Reprocessing

17. 22
DATABRICKS DELTA FILE FORMAT
S3
stream stream
# now open source
# based on parquet (columnar)
# supports batch and stream
# supports “transactions”
# schema evolution
# scalable metadata handling
# time travelling

18. 23
SQL vs SCALA
# Started with 200 lines of SQL
# Grew fast to 400 lines
# Violated DRY principle
# Hard to unit-test
# Hard to refactor
# Bad support for nested structures

19. 24
LESSONS LEARNED
# Streaming needs different thinking
# DWH ~ Backend Programming
# Don’t start with SQL because it’s easy
# Watch your data formats
# Best works with cross-functional team

20. Thanks
Questions?