Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
STREAM
PROCESSING WITH
APACHE FLINK IN
ZALANDO’S WORLD
OF MICROSERVICES
JAVIER LOPEZ
MIHAIL VIERU
2
AGENDA
● Zalando’s Microservices Architecture
● Saiki - Data Integration and Distribution at Scale
● Flink in a Microser...
3
ABOUT US
Mihail Vieru
Big Data Engineer,
Business Intelligence
Javier López
Big Data Engineer,
Business Intelligence
4
5
One of Europe's largest online fashion retailers
15 countries
~19 million active customers
~3 billion € revenue 2015
1,5...
6
ZALANDO TECHNOLOGY
1500+ TECHNOLOGISTS
Rapidly growing
international team
http://tech.zalando.com
VINTAGE ARCHITECTURE
8
VINTAGE BUSINESS INTELLIGENCE
Classical ETL process
Business
Logic
Data Warehouse (DWH)
DatabaseDBA
BI
Business
Logic
Da...
9
VINTAGE BUSINESS INTELLIGENCE
DWH Oracle
Exasol
RADICAL AGILITY
11
RADICAL AGILITY
AUTONOMY
MASTERY
PURPOSE
12
RADICAL AGILITY - AUTONOMY
Technologies Operations Teams
13
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Datab...
14
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
Team A
Business
Logic
Database
Team B
RESTAPI
RESTAPI
public...
15
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
Team A
Business
Logic
Database
Team B
RESTAPI
RESTAPI
public...
16
SUPPORTING AUTONOMY: MICROSERVICES
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RESTAPI
AppB
Business
L...
SAIKI
18
SAIKI DATA PLATFORM
SAIKI
App A App B App DApp C BI
Data Warehouse
19
SAIKI — DATA INTEGRATION & DISTRIBUTION
BI
Data Warehouse E.g. Forecast DB
SAIKI
App A App B App DApp C
Exporter
REST A...
20
SAIKI — SUMMARY
Δ J
D
B
C
REST
B
E
F
O
R
E
A
F
T
E
R
Data sources
Technologies
Data sources
Connections
Data sources
Ex...
FLINK IN A
MICROSERVICES WORLD
22
OPPORTUNITIES FOR NEXT GEN BI
Cloud Computing
- Distributed ETL
- Scale
Access to Real Time Data
- All teams publish da...
23
THE RIGHT FIT
STREAM PROCESSING
24
THE RIGHT FIT — STREAM PROCESSING ENGINE
Candidates:
Storm & Samza ruled out because of batch processing
requirement
25
SPARK VS. FLINK DIFFERENCES
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
Processing mode micro-batching tuple at a ti...
26
SPARK VS. FLINK PERFORMANCE
source: Benchmarking Streaming Computation Engines at Yahoo!
27
APACHE FLINK
• true stream processing framework
• process events at a consistently high rate with low
latency
• scalabl...
28
FLINK ON AWS - OUR APPLIANCE
MASTER ELB
EC2
Docker
Flink Master
EC2
Docker
Flink Shadow Master
WORKERS ELB
EC2
Docker
F...
USE CASES
BUSINESS PROCESS
MONITORING
31
BUSINESS PROCESS
A business process is in its simplest form a chain of
correlated events:
start event completion event
...
32
REAL-TIME BUSINESS PROCESS MONITORING
• Check if business processes in the Zalando platform work
• Analyze data on the ...
33
Saiki BPM
ARCHITECTURE BPM
Cfg Service
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
P...
34
HOW WE USE FLINK IN BPM
• 1000+ Event Types; 1 Event Type -> 1 Kafka topic
• Analyze processes with correlated event ty...
STREAMING ETL
36
Extract Transform Load (ETL)
Traditional ETL process:
• Batch processing
• No real time
• ETL tools
• Heavy processing ...
37
WHAT CHANGED WITH RADICAL AGILITY?
• Data comes in a semi-structured format (JSON payload)
• Data is distributed in sep...
38
`
Saiki Streaming ETL
ARCHITECTURE STREAMING ETL
Stream Processing
App A App B
Nakadi Event Bus
App C
Operational Syste...
39
HOW WE (WOULD) USE FLINK IN STREAMING ETL
• Transformation of complex payloads into simple ones for
easier consumption ...
FUTURE USE CASES
41
COMPLEX EVENT PROCESSING FOR BPM
Cont. example business process:
• Multiple PARCEL_SHIPPED events per order
• Generate ...
42
DEPLOYMENTS FROM OTHER BI TEAMS
Flink Jobs from other BI Teams
Requirements:
• manage and control deployments
• isolati...
43
OTHER FUTURE TOPICS
• New use cases for Real Time Analytics/ BI
o Sales monitoring
o Price monitoring
• Fraud detection...
44
CONCLUSION
Flink proved to be the right fit for our current stream
processing use cases. It enables us to build Zalando...
THANK YOU
Upcoming SlideShare
Loading in …5
×

Stream Processing using Apache Flink in Zalando's World of Microservices - Reactive Summit

2,629 views

Published on

In this talk we present Zalando's microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing for near-real time business intelligence.

Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization.

We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach - Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams.

We no longer live in a world of static data sets, but are instead confronted with an endless stream of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch.

With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL.

Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements.

On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability.

Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.

Published in: Technology
  • Be the first to comment

Stream Processing using Apache Flink in Zalando's World of Microservices - Reactive Summit

  1. 1. STREAM PROCESSING WITH APACHE FLINK IN ZALANDO’S WORLD OF MICROSERVICES JAVIER LOPEZ MIHAIL VIERU
  2. 2. 2 AGENDA ● Zalando’s Microservices Architecture ● Saiki - Data Integration and Distribution at Scale ● Flink in a Microservices World ● Stream Processing Use Cases: o Business Process Monitoring o Continuous ETL ● Future Work
  3. 3. 3 ABOUT US Mihail Vieru Big Data Engineer, Business Intelligence Javier López Big Data Engineer, Business Intelligence
  4. 4. 4
  5. 5. 5 One of Europe's largest online fashion retailers 15 countries ~19 million active customers ~3 billion € revenue 2015 1,500 brands 150,000+ products 11,000+ employees in Europe
  6. 6. 6 ZALANDO TECHNOLOGY 1500+ TECHNOLOGISTS Rapidly growing international team http://tech.zalando.com
  7. 7. VINTAGE ARCHITECTURE
  8. 8. 8 VINTAGE BUSINESS INTELLIGENCE Classical ETL process Business Logic Data Warehouse (DWH) DatabaseDBA BI Business Logic Database Business Logic Database Business Logic Database Dev
  9. 9. 9 VINTAGE BUSINESS INTELLIGENCE DWH Oracle Exasol
  10. 10. RADICAL AGILITY
  11. 11. 11 RADICAL AGILITY AUTONOMY MASTERY PURPOSE
  12. 12. 12 RADICAL AGILITY - AUTONOMY Technologies Operations Teams
  13. 13. 13 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI
  14. 14. 14 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database Team A Business Logic Database Team B RESTAPI RESTAPI public Internet Applications communicate using REST APIs Databases hidden behind the walls of AWS VPC
  15. 15. 15 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database Team A Business Logic Database Team B RESTAPI RESTAPI public Internet Classical ETL process is impossible!
  16. 16. 16 SUPPORTING AUTONOMY: MICROSERVICES Business Logic Database RESTAPI AppA Business Logic Database RESTAPI AppB Business Logic Database RESTAPI AppC Business Logic Database RESTAPI AppD Business Intelligence
  17. 17. SAIKI
  18. 18. 18 SAIKI DATA PLATFORM SAIKI App A App B App DApp C BI Data Warehouse
  19. 19. 19 SAIKI — DATA INTEGRATION & DISTRIBUTION BI Data Warehouse E.g. Forecast DB SAIKI App A App B App DApp C Exporter REST API Stream Processing via Apache Flink Data Lake . AWS S3
  20. 20. 20 SAIKI — SUMMARY Δ J D B C REST B E F O R E A F T E R Data sources Technologies Data sources Connections Data sources Extraction Data Delivery
  21. 21. FLINK IN A MICROSERVICES WORLD
  22. 22. 22 OPPORTUNITIES FOR NEXT GEN BI Cloud Computing - Distributed ETL - Scale Access to Real Time Data - All teams publish data to central event bus Hub for Data Teams - Data Lake provides distributed access and fine grained security - Data can be transformed (aggregated, joined, etc.) before delivering it to data teams Semi-Structured Data “General-purpose data processing engines like Flink or Spark let you define own data types and functions.” - Fabian Hueske, dataArtisans
  23. 23. 23 THE RIGHT FIT STREAM PROCESSING
  24. 24. 24 THE RIGHT FIT — STREAM PROCESSING ENGINE Candidates: Storm & Samza ruled out because of batch processing requirement
  25. 25. 25 SPARK VS. FLINK DIFFERENCES Feature Apache Spark 1.5.2 Apache Flink 0.10.1 Processing mode micro-batching tuple at a time Temporal processing support processing time event time, ingestion time, processing time Latency seconds sub-second Back pressure handling manual configuration implicit, through system architecture State access full state scan for each microbatch value lookup by key Operator library neutral ++ (split, windowByCount..) Support neutral ++ (mailing list, direct contact & support from data Artisans)
  26. 26. 26 SPARK VS. FLINK PERFORMANCE source: Benchmarking Streaming Computation Engines at Yahoo!
  27. 27. 27 APACHE FLINK • true stream processing framework • process events at a consistently high rate with low latency • scalable • great community and on-site support from Berlin/ Europe • university graduates with Flink skills https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/
  28. 28. 28 FLINK ON AWS - OUR APPLIANCE MASTER ELB EC2 Docker Flink Master EC2 Docker Flink Shadow Master WORKERS ELB EC2 Docker Flink Worker EC2 Docker Flink Worker EC2 Docker Flink Worker
  29. 29. USE CASES
  30. 30. BUSINESS PROCESS MONITORING
  31. 31. 31 BUSINESS PROCESS A business process is in its simplest form a chain of correlated events: start event completion event ORDER_CREATED ALL_PARCELS_SHIPPED Business Events from the whole Zalando platform flow through Saiki => opportunity to process those streams in near real time
  32. 32. 32 REAL-TIME BUSINESS PROCESS MONITORING • Check if business processes in the Zalando platform work • Analyze data on the fly: o Order velocities o Delivery velocities o Control SLAs of correlated events, e.g. parcel sent out after order
  33. 33. 33 Saiki BPM ARCHITECTURE BPM Cfg Service App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log PUBLICINTERNET OAUTH Alert Svc UI Elasticsearch Stream Processing
  34. 34. 34 HOW WE USE FLINK IN BPM • 1000+ Event Types; 1 Event Type -> 1 Kafka topic • Analyze processes with correlated event types (Join & Union) • Enrich data based on business rules • Sliding Windows (1min to 48hrs) for Platform Snapshots • State for alert metadata • Generation and processing of Complex Events (CEP lib)
  35. 35. STREAMING ETL
  36. 36. 36 Extract Transform Load (ETL) Traditional ETL process: • Batch processing • No real time • ETL tools • Heavy processing on the storage side
  37. 37. 37 WHAT CHANGED WITH RADICAL AGILITY? • Data comes in a semi-structured format (JSON payload) • Data is distributed in separate Kafka topics • There would be peak times, meaning that the data flow will increase by several factors • Data sources number increased by several factors
  38. 38. 38 ` Saiki Streaming ETL ARCHITECTURE STREAMING ETL Stream Processing App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log Exporter Oracle DWH Importer
  39. 39. 39 HOW WE (WOULD) USE FLINK IN STREAMING ETL • Transformation of complex payloads into simple ones for easier consumption in Oracle DWH • Combine several topics based on Business Rules (Union, Join) • Pre-Aggregate data to improve performance in the generation of reports (Windows, State) • Data cleansing • Data validation
  40. 40. FUTURE USE CASES
  41. 41. 41 COMPLEX EVENT PROCESSING FOR BPM Cont. example business process: • Multiple PARCEL_SHIPPED events per order • Generate complex event ALL_PARCELS_SHIPPED, when all PARCEL_SHIPPED events received (CEP lib, State)
  42. 42. 42 DEPLOYMENTS FROM OTHER BI TEAMS Flink Jobs from other BI Teams Requirements: • manage and control deployments • isolation of data flows o prevent different jobs from writing to the same sink • resource management in Flink o share cluster resources among concurrently running jobs StreamSQL would significantly lower the entry barrier
  43. 43. 43 OTHER FUTURE TOPICS • New use cases for Real Time Analytics/ BI o Sales monitoring o Price monitoring • Fraud detection for payments (evaluation) • Contact customer according to variable event pattern (evaluation)
  44. 44. 44 CONCLUSION Flink proved to be the right fit for our current stream processing use cases. It enables us to build Zalando’s Next Gen BI platform. https://tech.zalando.de/blog/?tags=Saiki
  45. 45. THANK YOU

×