Микросервисы получают все большую популярность в компаниях по всему миру. Какие организационные и технические проблемы они помогают решать? С какого момента монолиты перестают справляться с растущей нагрузкой на ваш сервис? Почему Zalando -- самый большой онлайн-ретейлер в Европе -- выбрал микросервисы в качестве главной архитектуры для новых проектов? Помогая в решении организационных проблем быстрорастущей компании, микросервисы ставят новые технические задачи, одной из которых, помимо увеличения сложности системы в целом, является проблема безопасного обмена сообщениями между микросервисами, удобной интеграции данных и возможности их корреляции и анализа. Слушатели узнают, как в Zalando решают эту проблему с использованием централизованной шины передачи данных -- Nakadi. Получат представление о тех проблемах, которые их могут поджидать при выборе похожей архитектуры на примере проблем выбора формата передачи данных, системы версионирования формата сообщений и сложностей эксплуатации высоконагруженных кластеров Kafka в облачной системе AWS.

1. События, шины и
интеграция данных
в непростом мире
микросервисов

2. Valentine Gogichashvili
Head of Data Engineering @ZalandoTech
twitter: @valgog
google+: +valgog
email: valentine.gogichashvili@zalando.de

5. 15 countries
4 fulfillment centers
18+ million active customers
~3 billion € revenue
150,000+ products
10,000+ employees
135 million visits per month
One of Europe's largest online
fashion retailers

6. Zalando Technology

7. BERLIN

8. BERLIN
DORTMUND
DUBLIN
HELSINKI
ERFURT
MÖNCHENGLADBACH
HAMBURG

9. Zalando Technology
1500+ Technologists
Rapidly growing
international team
http://tech.zalando.de

10. Good old small world

11. Good old small world
Started as a tiny online shop
Prototyped on Magento (PHP)
Used MySQL as a database
Web Application
Backend
Database

12. REBOOT

13. REBOOT
5½ years ago
• Java
• macro service architecture with SOAP as RPC layer
• PostgreSQL
• Heavy usage of Stored Procedures
• 4 databases + 1 sharded database on 2 shards
• Python for tooling (i.e code deploy automation)

14. REBOOT
Java Web
Frontend
Java Backend
PostgreSQL
Java Backend
PostgreSQL
Java Backend
PostgreSQL 9.0
RC1PostgreSQL 9.0
RC1PostgreSQL 9.0
RC1PostgreSQL
"macro" services

15. REBOOT
• PostgreSQL
• Heavy usage of Stored Procedures
• clean transaction scope
• very clean data
• processing close to data

16. REBOOT
• PostgreSQL
• Java Sproc Wrapper
• complex type mapping
• transparent sharding

17. Live long and prosper…
Very stable architecture that is still in use in the oldest
(vintage) components
We implemented everything ourselves starting from
warehouse and order management and finishing with
Web Shop and Mobile Applications

18. Live long and prosper…
"I want to code in Scala/Clojure/Haskell because it is cool and compact"

19. Live long and prosper…
"I want to code in Scala/Clojure/Haskell because it is cool and compact"
"But nobody will be able to support your code if you leave the company,
everybody should use Java, learn SQL and write Stored Procedures"

20. Live long and prosper…
"I want to code in Scala/Clojure/Haskell because it is cool and compact"
"But nobody will be able to support your code if you leave the company,
everybody should use Java, learn SQL and write Stored Procedures"
"I like the team and people but I am moving on to another company where I
can use cooler technologies!"

21. Radical Agility

22. Radical Agility
● Autonomy
● Purpose
● Mastery

23. Autonomous teams
● can choose own technology stack
● including persistence layer
● are responsible for operations
● should use isolated AWS accounts

24. Autonomy is not Anarchy

25. Supporting autonomy — STUPS

26. Supporting autonomy — STUPS

27. Supporting autonomy — Microservices
Business Logic
Data Storage
Team A Business Logic
Data Storage
Team B
RESTAPI
RESTAPI
Applications communicate using REST APIs
Databases hidden behind the walls of AWS VPC
public Internet

28. Supporting autonomy — Microservices
Business Logic
Data Storage
Team A Business Logic
Data Storage
Team B
RESTAPI
RESTAPI
public Internet
Classical ETL process is impossible!

29. Data integration in
the classical world

30. Data integration in the classical world

31. DBA Business
Intelligence
Developers
Data integration in the classical world

32. Data integration in the classical world
Business Logic
DBA Business
Intelligence
Data Warehouse (DWH)
Database
ETL process
DBA
BI
Dev

33. Data integration in the classical world
Classical ETL process
Business Logic
Data Warehouse (DWH)
DatabaseDBA
BI
Business Logic
Database
Business Logic
Database
Business Logic
Database
Dev

34. Data integration in the classical world
Classical ETL process
• Use-case specific — a lot of manual work
• Usually outputs data into a Data Warehouse
• well structured
• easy to use by the end user (SQL)

35. Data integration in
the world of microservices

36. Data integration in the world of microservices

37. Data integration in the world of microservices
Business
Logic
Database RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
RESTAPI

38. Data integration in the world of microservices
Queues to the
rescue!

39. Data integration in the world of microservices
Queues to the
rescue!
But will be there a
unified bus?

40. Data integration in the world of microservices
Queues to the
rescue!
But will be there a
unified bus?
What about form
and schema?

41. Data integration in the world of microservices
Business
Logic
Database RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
RESTAPI
AppA
AppB
AppC
AppD

42. Business Intelligence
Data integration in the world of microservices
Business
Logic
Database RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
RESTAPI
AppA
AppB
AppC
AppD

43. Business Intelligence
Data integration in the world of microservices
Business
Logic
Database RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
Database
RESTAPI
Business
Logic
RESTAPI
AppA
AppB
AppC
AppD
Data heavy services
(ML and DDDM)

44. Nakadi Event Bus

45. Nakadi Event Bus
• A secured HTTP API
This allows microservices teams to maintain service boundaries, and not
directly depend on any specific message broker technology. Access to the API
can be managed and secured using OAuth scopes;
• An event type registry
Events sent to Nakadi can be defined with a schema and managed via a
registry. Events can be validated before they are distributed to consumers;
• Inbuilt event types
Nakadi has optional support for events describing business processes and
data changes using standard primitives for identity, timestamps, event types,
and causality

46. Nakadi Event Bus
• Schema versioning
Enable schema evolution in the backwards compatible way;
• Low latency event delivery
Once a publisher sends an event using a simple HTTP POST, consumers can be
pushed to via a streaming HTTP connection, allowing near real-time event
processing;
• Built on proven infrastructure
Nakadi uses Apache Kafka as its internal message broker and PostgreSQL as a
backing database for storing metadata and schema information.

47. Nakadi Cluster
Schema
validation
Schema
metadata
store
Event metadata
enrichment
ZooKeeper
config store
Messaging Backend
(Kafka now, Kynesis & others later)
Nakadi Broker
Management of offsets
for high level API

48. Nakadi Event Bus — next steps
Nakadi Cluster

49. Nakadi Event Bus — next steps
Nakadi Cluster
Nakadi UI

50. Nakadi Event Bus — next steps
Nakadi Cluster
(physically separated
per event type)
Nakadi Cluster
(physically separated
per event type)
Nakadi UI

51. Nakadi Event Bus — next steps
Nakadi Cluster
(physically separated
per event type)
Nakadi Cluster
(physically separated
per event type)
Nakadi Schema
Manager
Nakadi UI

52. Nakadi Event Bus — next steps
Nakadi Cluster
(physically separated
per event type)
Nakadi Cluster
(physically separated
per event type)
Nakadi Schema
Manager
Nakadi Schema
Crawler
Nakadi UI

53. Nakadi Event Bus — next steps
Nakadi Cluster
(physically separated
per event type)
Nakadi Cluster
(physically separated
per event type)
Nakadi Schema
Manager
Nakadi Schema
Crawler
Nakadi UI
Nakadi Topology Manager

54. Bubuku
Kafka AWS supervisor

55. Bubuku Kafka AWS supervisor
• IP address discovery using AWS
• Exhibitor discovery using AWS load balancer name
• Rebalance partitions on different events
• React on Exhibitor topology changes
• Automatic Kafka restart in case if broker is considered dead
• Graceful broker termination in case of supervisor stop
• Broker start/stop/restart synchronization across the cluster

56. Open Source at Zalando
https://zalando.github.io/
https://github.com/zalando/zalando-howto-open-source
https://github.com/zalando/nakadi
https://github.com/zalando-incubator/bubuku
STUPS.io for responsible organizations in AWS