The document discusses Zalando's microservices architecture and the integration of Apache Flink for real-time data processing and business intelligence. It highlights various use cases including business process monitoring and continuous ETL, emphasizing Flink's advantages over other frameworks. The document concludes that Flink is well-suited for Zalando's next-generation BI platform due to its low latency and scalable stream processing capabilities.

1. FLINK IN
ZALANDO’S
WORLD OF
MICROSERVICES
JAVIER LOPEZ
MIHAIL VIERU
12-09-2016

2. 2
AGENDA
● Zalando’s Microservices Architecture
● Saiki - Data Integration and Distribution at Scale
● Flink in a Microservices World
● Stream Processing Use Cases:
o Business Process Monitoring
o Continuous ETL
● Future Work

3. 3
ABOUT US
Mihail Vieru
Big Data Engineer,
Business Intelligence
Javier López
Big Data Engineer,
Business Intelligence

4. 4

5. 5
One of Europe's largest online fashion retailers
15 countries
~19 million active customers
~3 billion € revenue 2015
1,500 brands
150,000+ products
11,000+ employees in Europe

6. 6
ZALANDO TECHNOLOGY
1300+ TECHNOLOGISTS
Rapidly growing
international team
http://tech.zalando.com

7. VINTAGE ARCHITECTURE

8. 8
VINTAGE BUSINESS INTELLIGENCE
Classical ETL process
Business
Logic
Data Warehouse (DWH)
DatabaseDBA
BI
Business
Logic
Database
Business
Logic
Database
Business
Logic
Database
Dev

9. 9
VINTAGE BUSINESS INTELLIGENCE
DWH Oracle
Exasol

10. RADICAL AGILITY

11. 11
RADICAL AGILITY
AUTONOMY
MASTERY
PURPOSE

12. 12
RADICAL AGILITY - AUTONOMY
Technologies Operations Teams

13. 13
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI

14. 14
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
Team A
Business
Logic
Database
Team B
RESTAPI
RESTAPI
public Internet
Applications communicate using REST APIs
Databases hidden behind the walls of AWS VPC

15. 15
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
Team A
Business
Logic
Database
Team B
RESTAPI
RESTAPI
public Internet
Classical ETL process is impossible!

16. 16
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RESTAPI
AppB
Business
Logic
Database
RESTAPI
AppC
Business
Logic
Database
RESTAPI
AppD
Business Intelligence

17. SAIKI

18. 18
SAIKI DATA PLATFORM
SAIKI
App A App B App DApp C BI
Data Warehouse

19. 19
SAIKI — DATA INTEGRATION & DISTRIBUTION
BI
Data Warehouse E.g. Forecast DB
SAIKI
App A App B App DApp C
Exporter
REST API
Stream Processing
via Apache Flink Data Lake .
AWS S3

20. 20
SAIKI — SUMMARY
Δ J
D
B
C
REST
B
E
F
O
R
E
A
F
T
E
R
Data sources
Technologies
Data sources
Connections
Data sources
Extraction
Data
Delivery

21. FLINK IN A
MICROSERVICES WORLD

22. 22
OPPORTUNITIES FOR NEXT GEN BI
Cloud Computing
- Distributed ETL
- Scale
Access to Real Time Data
- All teams publish data to central event
bus
Hub for Data Teams
- Data Lake provides distributed access
and fine grained security
- Data can be transformed (aggregated,
joined, etc.) before delivering it to data
teams
Semi-Structured Data
“General-purpose data processing engines
like Flink or Spark let you define own data
types and functions.”
- Fabian Hueske,
dataArtisans

23. 23
THE RIGHT FIT
STREAM PROCESSING

24. 24
THE RIGHT FIT — STREAM PROCESSING ENGINE
Candidates:
Storm & Samza ruled out because of batch processing
requirement

25. 25

26. 26
SPARK VS. FLINK DIFFERENCES
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
Processing mode micro-batching tuple at a time
Temporal processing support processing time event time, ingestion time,
processing time
Latency seconds sub-second
Back pressure handling manual configuration implicit, through system
architecture
State access full state scan for each microbatch value lookup by key
Operator library neutral ++ (split, windowByCount..)
Support neutral ++ (mailing list, direct contact &
support from data Artisans)

27. 27
APACHE FLINK
• true stream processing framework
• process events at a consistently high rate with low
latency
• scalable
• great community and on-site support from Berlin/
Europe
• university graduates with Flink skills
https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/

28. 28
FLINK ON AWS - OUR APPLIANCE
MASTER ELB
EC2
Docker
Flink Master
EC2
Docker
Flink Shadow Master
WORKERS ELB
EC2
Docker
Flink Worker
EC2
Docker
Flink Worker
EC2
Docker
Flink Worker

29. USE CASES

30. BUSINESS PROCESS
MONITORING

31. 31
BUSINESS PROCESS
A business process is in its simplest form a chain of
correlated events:
start event completion event
ORDER_CREATE
D
ALL_PARCELS_SHIPPED
Business Events from the whole Zalando platform flow through
Saiki => opportunity to process those streams in near real time

32. 32
REAL-TIME BUSINESS PROCESS MONITORING
• Check if business processes in the Zalando platform work
• Analyze data on the fly:
o Order velocities
o Delivery velocities
o Control SLAs of correlated events, e.g. parcel sent out
after order

33. 33
Saiki BPM
ARCHITECTURE BPM
Cfg Service
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
PUBLICINTERNET
OAUTH
Alert Svc
UI
Elasticsearch
Stream Processing

34. 34
HOW WE USE FLINK IN BPM
• 1000+ Event Types; 1 Event Type -> 1 Kafka topic
• Analyze processes with correlated event types (Join &
Union)
• Enrich data based on business rules
• Sliding Windows (1min to 48hrs) for Platform Snapshots
• State for alert metadata
• Generation and processing of Complex Events (CEP lib)

35. STREAMING ETL

36. 36
Extract Transform Load (ETL)
Traditional ETL process:
• Batch processing
• No real time
• ETL tools
• Heavy processing on the storage side

37. 37
WHAT CHANGED WITH RADICAL AGILITY?
• Data comes in a semi-structured format (JSON payload)
• Data is distributed in separate Kafka topics
• There would be peak times, meaning that the data flow
will increase by several factors
• Data sources number increased by several factors

38. 38
`
Saiki Streaming ETL
ARCHITECTURE STREAMING ETL
Stream Processing
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log Exporter
Oracle DWH
Importer

39. 39
HOW WE (WOULD) USE FLINK IN STREAMING ETL
• Transformation of complex payloads into simple ones for
easier consumption in Oracle DWH
• Combine several topics based on Business Rules (Union,
Join)
• Pre-Aggregate data to improve performance in the
generation of reports (Windows, State)
• Data cleansing
• Data validation

40. FUTURE USE CASES

41. 41
COMPLEX EVENT PROCESSING FOR BPM
Cont. example business process:
• Multiple PARCEL_SHIPPED events per order
• Generate complex event ALL_PARCELS_SHIPPED,
when all PARCEL_SHIPPED events received
(CEP lib, State)

42. 42
DEPLOYMENTS FROM OTHER BI TEAMS
Flink Jobs from other BI Teams
Requirements:
• manage and control deployments
• isolation of data flows
o prevent different jobs from writing to the same sink
• resource management in Flink
o share cluster resources among concurrently running jobs
StreamSQL would significantly lower the entry barrier

43. 43
REPLACE KAFKA2KAFKA COMPONENT
• Python app
• extracts events from REST API Nakadi Event Bus
• writes them to our Kafka cluster
Idea: Create Nakadi consumer/producer to enable stream
processing with Flink to other internal users
(first POC done)

44. 44
OTHER FUTURE TOPICS
• New use cases for Real Time Analytics/ BI
o Sales monitoring
o Price monitoring
• Fraud detection for payments (evaluation)
• Contact customer according to variable event pattern
(evaluation)

45. 45
CONCLUSION
Flink proved to be the right fit for our current stream
processing use cases. It enables us to build Zalando’s Next
Gen BI platform.
https://tech.zalando.de/blog/?tags=Saiki

46. THANK YOU