Berlin Apache Flink Meetup, May 2016 In this talk we present Zalando's microservices architecture and introduce Saiki – our next generation data integration and distribution platform on AWS. We show why we chose Apache Flink to serve as our stream processing framework and describe how we employ it for our current use cases: business process monitoring and continuous ETL. We then have an outlook on future use cases. By Javier Lopez & Mihail Vieru, Zalando, Zalando SE

1. world of microservices
Flink in Zalando’s

2. Agenda
Zalando’s Microservices Architecture
Saiki - Data Integration and Distribution at Scale
Flink in a Microservices World
Stream Processing Use Cases:
Business Process Monitoring
Continuous ETL
Future Work
2

3. About us
Mihail Vieru
Big Data Engineer,
Business Intelligence
Javier López
Big Data Engineer,
Business Intelligence
3

4. 4

5. 15 countries
4 fulfillment centers
~18 million active customers
2.9 billion € revenue 2015
150,000+ products
10,000+ employees
One of Europe's largest
online fashion retailers

6. Zalando Technology
1100+ TECHNOLOGISTS
Rapidly growing
international team
http://tech.zalando.com
6

7. VINTAGE ARCHITECTURE

8. Classical ETL process
Vintage Business Intelligence
Business Logic
Data Warehouse (DWH)
DatabaseDBA
BI
Business Logic
Database
Business Logic
Database
Business Logic
Database
Dev
8

9. Vintage Business Intelligence
Very stable architecture that is still in use in the oldest
components
Classical ETL process
• Use-case specific
• Usually outputs data into a Data Warehouse
• well structured
• easy to use by the end user (SQL) 9

10. RADICAL AGILITY

11. AUTONOMY
MASTERY
PURPOSE
Radical Agility
11

12. Autonomy
Autonomous teams
• can choose own technology stack
• including persistence layer
• are responsible for operations
• should use isolated AWS accounts
12

13. Supporting autonomy — Microservices
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
13

14. Supporting autonomy — Microservices
Business Logic
Database
Team
A
Business Logic
Database
Team
B
RESTAPI
RESTAPI
Applications communicate using REST APIs
Databases hidden behind the walls of AWS VPC
public Internet
14

15. Supporting autonomy — Microservices
Business Logic
Database
Team
A
Business Logic
Database
Team
B
Classical ETL process is impossible!
RESTAPI
public Internet
RESTAPI
15

16. Supporting autonomy — Microservices
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RESTAPI
AppB
Business
Logic
Database
RESTAPI
AppC
Business
Logic
Database
RESTAPI
AppD
16

17. Supporting autonomy — Microservices
Business Intelligence
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RESTAPI
AppB
Business
Logic
Database
RESTAPI
AppC
Business
Logic
Database
RESTAPI
AppD
17

18. Supporting autonomy — Microservices
App A App B App DApp C BI
Data Warehouse
?
18

19. SAIKI

20. SAIKI
Saiki Data Platform
App A App B App DApp C BI
Data Warehouse
20

21. SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku
21

22. SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku
AWS S3
Tukang
REST API
22

23. SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku Tukang
REST API
AWS S3
23
E.g. Forecast DB

24. Saiki — Data Integration & Distribution
Old Load Process New Load Process
relied on Delta Loads relies on Event Stream
JDBC connection RESTful HTTPS connections
Data Quality could be controlled by BI
independently
trust for Correctness of Data in the
delivery teams
PostgreSQL dependent Independent of the source technology
stack
N to 1 data stream N to M streams, no single data sink
24

25. BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST API
AWS S3
25

26. BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST API
Stream Processing
via Apache Flink
AWS S3
26

27. BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST API
Stream Processing
via Apache Flink Data Lake .
AWS S3
27

28. FLINK IN A
MICROSERVICES
WORLD

29. Current state in BI
29
• Centralized ETL tools
(Pentaho DI/Kettle and Oracle Stored Procedures)
• Reporting data does not arrive in real time
• Data to data science & analytical teams is provided using
Exasol DB
• All data comes from SQL databases

30. Opportunities for Next Gen BI
• Cloud Computing
• Distributed ETL
• Scale
• Access to real time data
• All teams publishing data to central bus (Kafka)
30

31. Opportunities for Next Gen BI
• Hub for Data teams
• Data Lake provides distributed access and fine
grained security
• Data can be transformed (aggregated, joined, etc.)
before delivering it to data teams
• Non structured data
• “General-purpose data processing engines like Flink or
Spark let you define own data types and functions.” -
Fabian Hueske 31

32. The right fit
Stream Processing
32

33. The right fit - Data processing engine (I)
33
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
processing mode micro-batching tuple at a time
temporal processing
support
processing time event time, ingestion time, processing
time
batch processing yes yes
latency seconds sub-second
back pressure handling through manual configuration implicit, through system architecture
state storage distributed dataset in memory distributed key/value store
state access full state scan for each microbatch value lookup by key
state checkpointing yes yes
high availability mode yes yes

34. The right fit - Data processing engine (II)
34
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
event processing guarantee exactly once exactly once
Apache Kafka connector yes yes
Amazon S3 connector yes yes
operator library neutral ++ (split, windowByCount..)
language support Java, Scala, Python Java, Scala, Python
deployment standalone, YARN, Mesos standalone, YARN
support neutral ++ (mailing list, direct contact & support
from data Artisans)
documentation + neutral
maturity ++ neutral

35. Saiki Data Platform
Apache Flink
• true stream processing framework
• process events at a consistently high rate with low latency
• scalable
• great community and on-site support from Berlin/ Europe
• university graduates with Flink skills
https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/
35

36. Flink in AWS - Our appliance
36
MASTER ELB
EC2
Docker
Flink Master
EC2
Docker
Flink Shadow Master
WORKERS ELB
EC2
Docker
Flink Worker
EC2
Docker
Flink Worker
EC2
Docker
Flink Worker

37. USE CASES

38. BUSINESS PROCESS
MONITORING

39. Business Process
A business process is in its simplest form a chain of
correlated events:
Business Events from the whole Zalando platform flow through
Saiki => opportunity to process those streams in near real-time
39
start event completion event
ORDER_CREATED ALL_SHIPMENTS_DONE

40. Near Real-Time Business Process Monitoring
• Check if technically the Zalando platform works
• 1000+ Event Types
• Analyze data on the fly:
• Order velocities
• Delivery velocities
• Control SLAs of correlated events, e.g. parcel sent out
after order
40

41. Architecture BPM
41
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
Stream Processing Tokoh
ZMON
Cfg Service
PUBLICINTERNET
OAUTH
Saiki BPM
Elasticsearch

42. How we use Flink in BPM
• 1 Event Type -> 1 Kafka topic
• Define a granularity level of 1 minute (TumblingWindows)
• Analyze processes with multiple event types (Join &
Union)
• Generation and processing of Complex Events (CEP &
State)
42

43. CONTINUOUS ETL

44. Extract Transform Load (ETL)
Traditional ETL process:
• Batch processing
• No real time
• ETL tools
• Heavy processing on the storage side
44

45. What changed with Radical Agility?
• Data comes in a semi-structured format (JSON payload)
• Data is distributed in separate Kafka topics
• There would be peak times, meaning that the data flow
will increase by several factors
• Data sources number increased by several factors
45

46. Architecture Continuous ETL
46
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
Stream Processing Saiki Continuous ETL
Tukang
Oracle DWH
Orang

47. How we (would) use Flink in Continuous ETL
• Transformation of complex payloads into simple ones for
easier consumption in Oracle DWH
• Combine several topics based on Business Rules (Union,
Join)
• Pre-Aggregate data to improve performance in the
generation of reports (Windows, State)
• Data cleansing
• Data validation
47

48. FUTURE WORK

49. Complex Event Processing for BPM
Cont. example business process:
• Multiple PARCEL_SHIPPED events per order
• Generate complex event ALL_SHIPMENTS_DONE, when
all PARCEL_SHIPPED events received
(CEP lib, State)
49

50. Deployments from other BI Teams
Flink Jobs from other BI Teams
Requirements:
• manage and control deployments
• isolation of data flows
• prevent different jobs from writing to the same sink
• resource management in Flink
• share cluster resources among concurrently running jobs
StreamSQL would significantly lower the entry barrier
50

51. Replace Kafka2Kafka
• Python app
• extracts events from REST API Nakadi Event Bus
• writes them to our Kafka cluster
Idea: Replace Python app with Flink Jobs, which would write
directly to Saiki’s Kafka cluster (under discussion)
51

52. Other Future Topics
• CD integration for Flink appliance
• Monitoring data flows via heartbeats
• Flink <-> Kafka
• Flink <-> Elasticsearch
• New use cases for Real Time Analytics/ BI
• Sales monitoring
52

53. THANK YOU

54. Open Source @ZalandoTech
https://zalando.github.io/
https://tech.zalando.de/blog
https://github.com/zalando/saiki/wiki
54
QUESTIONS?