As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Kafka Summit London - Kafka based Global Data Mesh at Wix
1. @NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
4. @NSilnitsky
Kafka- based Global Data Mesh
At Wix
±1B
Unique
visitors
>100M
Websites
Published by
Wix users
(5-7% of all
internet
websites)
±5000
People
Work at Wix
use Wix
platform
every month
5. @NSilnitsky
500B >600
GAs Per Day
~1500
Developers
Daily HTTP
Transactions
18
Data
Centers &
Pops
2
Cloud Providers
[Google/AWS]
Scaling out
Kafka- based Global Data Mesh
7. @NSilnitsky
@NSilnitsky
A Fully Distributed Environment
15B daily business Events
4 Regions @ GCP/AWS
~50k Topics
> 500k Partitions
25% of the data is replicated
+ We assume
x10 scale factor
during 12-24 next
months
31. @NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Kafka- based Global Data Mesh
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster
33. @NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
A
35. @NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
40. @NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer