Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
world of microservices
Flink in Zalando’s
Agenda
Zalando’s Microservices Architecture
Saiki - Data Integration and Distribution at Scale
Flink in a Microservices Wo...
About us
Mihail Vieru
Big Data Engineer,
Business Intelligence
Javier López
Big Data Engineer,
Business Intelligence
3
4
15 countries
4 fulfillment centers
~18 million active customers
2.9 billion € revenue 2015
150,000+ products
10,000+ emplo...
Zalando Technology
1100+ TECHNOLOGISTS
Rapidly growing
international team
http://tech.zalando.com
6
VINTAGE ARCHITECTURE
Classical ETL process
Vintage Business Intelligence
Business Logic
Data Warehouse (DWH)
DatabaseDBA
BI
Business Logic
Data...
Vintage Business Intelligence
Very stable architecture that is still in use in the oldest
components
Classical ETL process...
RADICAL AGILITY
AUTONOMY
MASTERY
PURPOSE
Radical Agility
11
Autonomy
Autonomous teams
• can choose own technology stack
• including persistence layer
• are responsible for operations...
Supporting autonomy — Microservices
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Databas...
Supporting autonomy — Microservices
Business Logic
Database
Team
A
Business Logic
Database
Team
B
RESTAPI
RESTAPI
Applicat...
Supporting autonomy — Microservices
Business Logic
Database
Team
A
Business Logic
Database
Team
B
Classical ETL process is...
Supporting autonomy — Microservices
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RESTAPI
AppB
Business
Log...
Supporting autonomy — Microservices
Business Intelligence
Business
Logic
Database
RESTAPI
AppA
Business
Logic
Database
RES...
Supporting autonomy — Microservices
App A App B App DApp C BI
Data Warehouse
?
18
SAIKI
SAIKI
Saiki Data Platform
App A App B App DApp C BI
Data Warehouse
20
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku
21
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku
AWS S3
Tukang
REST API
22
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C BI
Data Warehouse
Buku Tukang
REST API
AWS S3
23
E.g....
Saiki — Data Integration & Distribution
Old Load Process New Load Process
relied on Delta Loads relies on Event Stream
JDB...
BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST A...
BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST A...
BI
Data Warehouse E.g. Forecast DB
SAIKI
Saiki — Data Integration & Distribution
App A App B App DApp C
Buku Tukang
REST A...
FLINK IN A
MICROSERVICES
WORLD
Current state in BI
29
• Centralized ETL tools
(Pentaho DI/Kettle and Oracle Stored Procedures)
• Reporting data does not ...
Opportunities for Next Gen BI
• Cloud Computing
• Distributed ETL
• Scale
• Access to real time data
• All teams publishin...
Opportunities for Next Gen BI
• Hub for Data teams
• Data Lake provides distributed access and fine
grained security
• Dat...
The right fit
Stream Processing
32
The right fit - Data processing engine (I)
33
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
processing mode micro-batchin...
The right fit - Data processing engine (II)
34
Feature Apache Spark 1.5.2 Apache Flink 0.10.1
event processing guarantee e...
Saiki Data Platform
Apache Flink
• true stream processing framework
• process events at a consistently high rate with low ...
Flink in AWS - Our appliance
36
MASTER ELB
EC2
Docker
Flink Master
EC2
Docker
Flink Shadow Master
WORKERS ELB
EC2
Docker
F...
USE CASES
BUSINESS PROCESS
MONITORING
Business Process
A business process is in its simplest form a chain of
correlated events:
Business Events from the whole Z...
Near Real-Time Business Process Monitoring
• Check if technically the Zalando platform works
• 1000+ Event Types
• Analyze...
Architecture BPM
41
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
Stream Processing Tokoh...
How we use Flink in BPM
• 1 Event Type -> 1 Kafka topic
• Define a granularity level of 1 minute (TumblingWindows)
• Analy...
CONTINUOUS ETL
Extract Transform Load (ETL)
Traditional ETL process:
• Batch processing
• No real time
• ETL tools
• Heavy processing on ...
What changed with Radical Agility?
• Data comes in a semi-structured format (JSON payload)
• Data is distributed in separa...
Architecture Continuous ETL
46
App A App B
Nakadi Event Bus
App C
Operational Systems
Kafka2Kafka
Unified Log
Stream Proce...
How we (would) use Flink in Continuous ETL
• Transformation of complex payloads into simple ones for
easier consumption in...
FUTURE WORK
Complex Event Processing for BPM
Cont. example business process:
• Multiple PARCEL_SHIPPED events per order
• Generate com...
Deployments from other BI Teams
Flink Jobs from other BI Teams
Requirements:
• manage and control deployments
• isolation ...
Replace Kafka2Kafka
• Python app
• extracts events from REST API Nakadi Event Bus
• writes them to our Kafka cluster
Idea:...
Other Future Topics
• CD integration for Flink appliance
• Monitoring data flows via heartbeats
• Flink <-> Kafka
• Flink ...
THANK YOU
Open Source @ZalandoTech
https://zalando.github.io/
https://tech.zalando.de/blog
https://github.com/zalando/saiki/wiki
54
...
Upcoming SlideShare
Loading in …5
×

Flink in Zalando's World of Microservices

3,717 views

Published on

Berlin Apache Flink Meetup, May 2016

In this talk we present Zalando's microservices architecture and introduce Saiki – our next generation data integration and distribution platform on AWS. We show why we chose Apache Flink to serve as our stream processing framework and describe how we employ it for our current use cases: business process monitoring and continuous ETL. We then have an outlook on future use cases.

By Javier Lopez & Mihail Vieru, Zalando, Zalando SE

Published in: Technology

Flink in Zalando's World of Microservices

  1. 1. world of microservices Flink in Zalando’s
  2. 2. Agenda Zalando’s Microservices Architecture Saiki - Data Integration and Distribution at Scale Flink in a Microservices World Stream Processing Use Cases: Business Process Monitoring Continuous ETL Future Work 2
  3. 3. About us Mihail Vieru Big Data Engineer, Business Intelligence Javier López Big Data Engineer, Business Intelligence 3
  4. 4. 4
  5. 5. 15 countries 4 fulfillment centers ~18 million active customers 2.9 billion € revenue 2015 150,000+ products 10,000+ employees One of Europe's largest online fashion retailers
  6. 6. Zalando Technology 1100+ TECHNOLOGISTS Rapidly growing international team http://tech.zalando.com 6
  7. 7. VINTAGE ARCHITECTURE
  8. 8. Classical ETL process Vintage Business Intelligence Business Logic Data Warehouse (DWH) DatabaseDBA BI Business Logic Database Business Logic Database Business Logic Database Dev 8
  9. 9. Vintage Business Intelligence Very stable architecture that is still in use in the oldest components Classical ETL process • Use-case specific • Usually outputs data into a Data Warehouse • well structured • easy to use by the end user (SQL) 9
  10. 10. RADICAL AGILITY
  11. 11. AUTONOMY MASTERY PURPOSE Radical Agility 11
  12. 12. Autonomy Autonomous teams • can choose own technology stack • including persistence layer • are responsible for operations • should use isolated AWS accounts 12
  13. 13. Supporting autonomy — Microservices Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI Business Logic Database RESTAPI 13
  14. 14. Supporting autonomy — Microservices Business Logic Database Team A Business Logic Database Team B RESTAPI RESTAPI Applications communicate using REST APIs Databases hidden behind the walls of AWS VPC public Internet 14
  15. 15. Supporting autonomy — Microservices Business Logic Database Team A Business Logic Database Team B Classical ETL process is impossible! RESTAPI public Internet RESTAPI 15
  16. 16. Supporting autonomy — Microservices Business Logic Database RESTAPI AppA Business Logic Database RESTAPI AppB Business Logic Database RESTAPI AppC Business Logic Database RESTAPI AppD 16
  17. 17. Supporting autonomy — Microservices Business Intelligence Business Logic Database RESTAPI AppA Business Logic Database RESTAPI AppB Business Logic Database RESTAPI AppC Business Logic Database RESTAPI AppD 17
  18. 18. Supporting autonomy — Microservices App A App B App DApp C BI Data Warehouse ? 18
  19. 19. SAIKI
  20. 20. SAIKI Saiki Data Platform App A App B App DApp C BI Data Warehouse 20
  21. 21. SAIKI Saiki — Data Integration & Distribution App A App B App DApp C BI Data Warehouse Buku 21
  22. 22. SAIKI Saiki — Data Integration & Distribution App A App B App DApp C BI Data Warehouse Buku AWS S3 Tukang REST API 22
  23. 23. SAIKI Saiki — Data Integration & Distribution App A App B App DApp C BI Data Warehouse Buku Tukang REST API AWS S3 23 E.g. Forecast DB
  24. 24. Saiki — Data Integration & Distribution Old Load Process New Load Process relied on Delta Loads relies on Event Stream JDBC connection RESTful HTTPS connections Data Quality could be controlled by BI independently trust for Correctness of Data in the delivery teams PostgreSQL dependent Independent of the source technology stack N to 1 data stream N to M streams, no single data sink 24
  25. 25. BI Data Warehouse E.g. Forecast DB SAIKI Saiki — Data Integration & Distribution App A App B App DApp C Buku Tukang REST API AWS S3 25
  26. 26. BI Data Warehouse E.g. Forecast DB SAIKI Saiki — Data Integration & Distribution App A App B App DApp C Buku Tukang REST API Stream Processing via Apache Flink AWS S3 26
  27. 27. BI Data Warehouse E.g. Forecast DB SAIKI Saiki — Data Integration & Distribution App A App B App DApp C Buku Tukang REST API Stream Processing via Apache Flink Data Lake . AWS S3 27
  28. 28. FLINK IN A MICROSERVICES WORLD
  29. 29. Current state in BI 29 • Centralized ETL tools (Pentaho DI/Kettle and Oracle Stored Procedures) • Reporting data does not arrive in real time • Data to data science & analytical teams is provided using Exasol DB • All data comes from SQL databases
  30. 30. Opportunities for Next Gen BI • Cloud Computing • Distributed ETL • Scale • Access to real time data • All teams publishing data to central bus (Kafka) 30
  31. 31. Opportunities for Next Gen BI • Hub for Data teams • Data Lake provides distributed access and fine grained security • Data can be transformed (aggregated, joined, etc.) before delivering it to data teams • Non structured data • “General-purpose data processing engines like Flink or Spark let you define own data types and functions.” - Fabian Hueske 31
  32. 32. The right fit Stream Processing 32
  33. 33. The right fit - Data processing engine (I) 33 Feature Apache Spark 1.5.2 Apache Flink 0.10.1 processing mode micro-batching tuple at a time temporal processing support processing time event time, ingestion time, processing time batch processing yes yes latency seconds sub-second back pressure handling through manual configuration implicit, through system architecture state storage distributed dataset in memory distributed key/value store state access full state scan for each microbatch value lookup by key state checkpointing yes yes high availability mode yes yes
  34. 34. The right fit - Data processing engine (II) 34 Feature Apache Spark 1.5.2 Apache Flink 0.10.1 event processing guarantee exactly once exactly once Apache Kafka connector yes yes Amazon S3 connector yes yes operator library neutral ++ (split, windowByCount..) language support Java, Scala, Python Java, Scala, Python deployment standalone, YARN, Mesos standalone, YARN support neutral ++ (mailing list, direct contact & support from data Artisans) documentation + neutral maturity ++ neutral
  35. 35. Saiki Data Platform Apache Flink • true stream processing framework • process events at a consistently high rate with low latency • scalable • great community and on-site support from Berlin/ Europe • university graduates with Flink skills https://tech.zalando.com/blog/apache-showdown-flink-vs.-spark/ 35
  36. 36. Flink in AWS - Our appliance 36 MASTER ELB EC2 Docker Flink Master EC2 Docker Flink Shadow Master WORKERS ELB EC2 Docker Flink Worker EC2 Docker Flink Worker EC2 Docker Flink Worker
  37. 37. USE CASES
  38. 38. BUSINESS PROCESS MONITORING
  39. 39. Business Process A business process is in its simplest form a chain of correlated events: Business Events from the whole Zalando platform flow through Saiki => opportunity to process those streams in near real-time 39 start event completion event ORDER_CREATED ALL_SHIPMENTS_DONE
  40. 40. Near Real-Time Business Process Monitoring • Check if technically the Zalando platform works • 1000+ Event Types • Analyze data on the fly: • Order velocities • Delivery velocities • Control SLAs of correlated events, e.g. parcel sent out after order 40
  41. 41. Architecture BPM 41 App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log Stream Processing Tokoh ZMON Cfg Service PUBLICINTERNET OAUTH Saiki BPM Elasticsearch
  42. 42. How we use Flink in BPM • 1 Event Type -> 1 Kafka topic • Define a granularity level of 1 minute (TumblingWindows) • Analyze processes with multiple event types (Join & Union) • Generation and processing of Complex Events (CEP & State) 42
  43. 43. CONTINUOUS ETL
  44. 44. Extract Transform Load (ETL) Traditional ETL process: • Batch processing • No real time • ETL tools • Heavy processing on the storage side 44
  45. 45. What changed with Radical Agility? • Data comes in a semi-structured format (JSON payload) • Data is distributed in separate Kafka topics • There would be peak times, meaning that the data flow will increase by several factors • Data sources number increased by several factors 45
  46. 46. Architecture Continuous ETL 46 App A App B Nakadi Event Bus App C Operational Systems Kafka2Kafka Unified Log Stream Processing Saiki Continuous ETL Tukang Oracle DWH Orang
  47. 47. How we (would) use Flink in Continuous ETL • Transformation of complex payloads into simple ones for easier consumption in Oracle DWH • Combine several topics based on Business Rules (Union, Join) • Pre-Aggregate data to improve performance in the generation of reports (Windows, State) • Data cleansing • Data validation 47
  48. 48. FUTURE WORK
  49. 49. Complex Event Processing for BPM Cont. example business process: • Multiple PARCEL_SHIPPED events per order • Generate complex event ALL_SHIPMENTS_DONE, when all PARCEL_SHIPPED events received (CEP lib, State) 49
  50. 50. Deployments from other BI Teams Flink Jobs from other BI Teams Requirements: • manage and control deployments • isolation of data flows • prevent different jobs from writing to the same sink • resource management in Flink • share cluster resources among concurrently running jobs StreamSQL would significantly lower the entry barrier 50
  51. 51. Replace Kafka2Kafka • Python app • extracts events from REST API Nakadi Event Bus • writes them to our Kafka cluster Idea: Replace Python app with Flink Jobs, which would write directly to Saiki’s Kafka cluster (under discussion) 51
  52. 52. Other Future Topics • CD integration for Flink appliance • Monitoring data flows via heartbeats • Flink <-> Kafka • Flink <-> Elasticsearch • New use cases for Real Time Analytics/ BI • Sales monitoring 52
  53. 53. THANK YOU
  54. 54. Open Source @ZalandoTech https://zalando.github.io/ https://tech.zalando.de/blog https://github.com/zalando/saiki/wiki 54 QUESTIONS?

×