1.
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

2.
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* keep simple even. Migrated, disributed

3.
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* messaging platform

4.
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* tackled blockers

5.
@NSilnitsky
@NSilnitsky
Event-driven
architecture
Kafka
Growing
Pains
Large scale
traffic
Hundreds of
developers
Diverse
tech stack
Cluster
overload
DC
awareness

6.
@NSilnitsky
Kafka Broker
You Know
Kafka
Kafka
Consumer
Kafka Broker
Service A Service B
Kafka
Producer
Kafka- based Global Data Mesh

7.
@NSilnitsky
If You Don’t
Know Kafka
Kafka- based Global Data Mesh
Kafka Broker
Kafka Broker
Topic
Partition

8.
@NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A

9.
@NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
Service B
Kafka
Consumer

10.
@NSilnitsky
@NSilnitsky
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka
Growing
Pains
Cluster
overload
DC
awareness

11.
@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Greyhound*
Wraps
Kafka
* open source software
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

12.
@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

13.
@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
val producer = GreyhoundBuilder.resilientProducer(topicName)
Kafka- based Global Data Mesh

14.
@NSilnitsky
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service 🟊
Greyhound
Kafka- based Global Data Mesh
Kafka Broker

15.
@NSilnitsky
Scala NodeJS / Python / Go
Greyhound
Kafka Broker
I’m a container
gRPC + proto
sidecar
JVM library
Kafka- based Global Data Mesh

16.
@NSilnitsky
Kafka Broker
Domain Events
ETL to DWH
Task Scheduling
Async Queue
...
We use it
for...
Kafka- based Global Data Mesh
Infra

17.
@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness

18.
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Kafka Broker
Single
Cluster
Overload
* intra-dc

19.
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Single
Cluster
Overload
A
A
A
A
B
Kafka Cluster (by SLAs)
B
Multi
Cluster
C

20.
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
To which
Cluster?
Kafka Cluster (by SLAs)
B
I want to produce
a domain event.
C

21.
@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
.ClusterA
val producer = GreyhoundBuilder.resilientProducer(topicName,
_.ClusterB
Kafka- based Global Data Mesh
* wix-layer

22.
@NSilnitsky

23.
@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

24.
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Master Active
Master Passive
Read-Only DBs
Http/gRPC
To which
DC?
I want to
process the event
and save the data.

25.
@NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Kafka- based Global Data Mesh
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

26.
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

27.
@NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
A

28.
@NSilnitsky
Greyhound
Batch
Consumer
Greyhound
Producer
Batch #
Replicate !
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
DC1 DC2
How?
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
A
Cross DC
Replication
Service

29.
@NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

30.
@NSilnitsky
Replication
service
Replicate !
DC1 DC2
How?
A
KVStoreReader
In-Memory Map
Key Value
Topic1 AllDCs
Topic2 Custom
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

31.
@NSilnitsky
Kafka- based Global Data Mesh

32.
@NSilnitsky
Kafka- based Global Data Mesh

33.
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

34.
@NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer

35.
@NSilnitsky
main-dc-producer
service
Produce !
DC1 DC2
How?
Master Active
Kafka- based Global Data Mesh
Greyhound
gRPC
producer
producer(topicName, _.toMainDc)

36.
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

37.
@NSilnitsky
DC1 DC2
To where needed
[WIP] Just Produce ❤
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
Master Active

38.
@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Master Active

39.
@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Replication service
* reintroduce decoupling

40.
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
DC1 DC2
DC3 DC4
A
A
A
A
B
B
Just Produce ❤
To a Single Logical
Kafka Cluster.
Kafka Broker
Cluster
DB

41.
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Any
Tech Stack
Greyhound
Anywhere
You Need
developer

42.
@NSilnitsky
developer
github.com/wix/greyhound
Kafka- based Global Data Mesh

43.
@NSilnitsky
@NSilnitsky
Thank
You!
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh

44.
@NSilnitsky
@NSilnitsky
Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh