The document describes Wix's use of Kafka and Greyhound to build a global data mesh. It addresses challenges of large scale traffic, hundreds of developers, and diverse tech stacks. Kafka and Greyhound provide a messaging platform and self-service APIs. Greyhound wraps Kafka for multiple languages. The data mesh supports cross-DC replication and producing to specific DCs. This allows events to be routed to where they are needed across Wix's infrastructure.

1. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

2. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* keep simple even. Migrated, disributed

3. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* messaging platform

4. @NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* tackled blockers

5. @NSilnitsky
@NSilnitsky
Event-driven
architecture
Kafka
Growing
Pains
Large scale
traffic
Hundreds of
developers
Diverse
tech stack
Cluster
overload
DC
awareness

6. @NSilnitsky
Kafka Broker
You Know
Kafka
Kafka
Consumer
Kafka Broker
Service A Service B
Kafka
Producer
Kafka- based Global Data Mesh

7. @NSilnitsky
If You Don’t
Know Kafka
Kafka- based Global Data Mesh
Kafka Broker
Kafka Broker
Topic
Partition

8. @NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A

9. @NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
Service B
Kafka
Consumer

10. @NSilnitsky
@NSilnitsky
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka
Growing
Pains
Cluster
overload
DC
awareness

11. @NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Greyhound*
Wraps
Kafka
* open source software
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

12. @NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer

13. @NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
val producer = GreyhoundBuilder.resilientProducer(topicName)
Kafka- based Global Data Mesh

14. @NSilnitsky
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service 🟊
Greyhound
Kafka- based Global Data Mesh
Kafka Broker

15. @NSilnitsky
Scala NodeJS / Python / Go
Greyhound
Kafka Broker
I’m a container
gRPC + proto
sidecar
JVM library
Kafka- based Global Data Mesh

16. @NSilnitsky
Kafka Broker
Domain Events
ETL to DWH
Task Scheduling
Async Queue
...
We use it
for...
Kafka- based Global Data Mesh
Infra

17. @NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness

18. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Kafka Broker
Single
Cluster
Overload
* intra-dc

19. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Single
Cluster
Overload
A
A
A
A
B
Kafka Cluster (by SLAs)
B
Multi
Cluster
C

20. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
To which
Cluster?
Kafka Cluster (by SLAs)
B
I want to produce
a domain event.
C

21. @NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
.ClusterA
val producer = GreyhoundBuilder.resilientProducer(topicName,
_.ClusterB
Kafka- based Global Data Mesh
* wix-layer

22. @NSilnitsky

23. @NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

24. @NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Master Active
Master Passive
Read-Only DBs
Http/gRPC
To which
DC?
I want to
process the event
and save the data.

25. @NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Kafka- based Global Data Mesh
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

26. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

27. @NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
A

28. @NSilnitsky
Greyhound
Batch
Consumer
Greyhound
Producer
Batch #
Replicate !
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
DC1 DC2
How?
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
A
Cross DC
Replication
Service

29. @NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

30. @NSilnitsky
Replication
service
Replicate !
DC1 DC2
How?
A
KVStoreReader
In-Memory Map
Key Value
Topic1 AllDCs
Topic2 Custom
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh

31. @NSilnitsky
Kafka- based Global Data Mesh

32. @NSilnitsky
Kafka- based Global Data Mesh

33. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

34. @NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer

35. @NSilnitsky
main-dc-producer
service
Produce !
DC1 DC2
How?
Master Active
Kafka- based Global Data Mesh
Greyhound
gRPC
producer
producer(topicName, _.toMainDc)

36. @NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

37. @NSilnitsky
DC1 DC2
To where needed
[WIP] Just Produce ❤
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
Master Active

38. @NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Master Active

39. @NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Replication service
* reintroduce decoupling

40. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
DC1 DC2
DC3 DC4
A
A
A
A
B
B
Just Produce ❤
To a Single Logical
Kafka Cluster.
Kafka Broker
Cluster
DB

41. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Any
Tech Stack
Greyhound
Anywhere
You Need
developer

42. @NSilnitsky
developer
github.com/wix/greyhound
Kafka- based Global Data Mesh

43. @NSilnitsky
@NSilnitsky
Thank
You!
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh

44. @NSilnitsky
@NSilnitsky
Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh