Successfully reported this slideshow.
Your SlideShare is downloading. ×

Kafka based Global Data Mesh at Wix

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 44 Ad

Kafka based Global Data Mesh at Wix

Download to read offline

As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.

This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various tools and features for adapting to growing scale and insuring high resilience.

As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.

This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various tools and features for adapting to growing scale and insuring high resilience.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Kafka based Global Data Mesh at Wix (20)

Advertisement

More from Natan Silnitsky (16)

Recently uploaded (20)

Advertisement

Kafka based Global Data Mesh at Wix

  1. 1. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh Kafka-based Global Data Mesh at Wix Natan Silnitsky Backend Infra TL, Wix.com natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
  2. 2. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * keep simple even. Migrated, disributed
  3. 3. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * messaging platform
  4. 4. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * tackled blockers
  5. 5. @NSilnitsky @NSilnitsky Event-driven architecture Kafka Growing Pains Large scale traffic Hundreds of developers Diverse tech stack Cluster overload DC awareness
  6. 6. @NSilnitsky Kafka Broker You Know Kafka Kafka Consumer Kafka Broker Service A Service B Kafka Producer Kafka- based Global Data Mesh
  7. 7. @NSilnitsky If You Don’t Know Kafka Kafka- based Global Data Mesh Kafka Broker Kafka Broker Topic Partition
  8. 8. @NSilnitsky Kafka Broker Topic Kafka- based Global Data Mesh Partition Kafka Producer 0 1 2 3 4 5 Service A
  9. 9. @NSilnitsky Kafka Broker Topic Kafka- based Global Data Mesh Partition Kafka Producer 0 1 2 3 4 5 Service A Service B Kafka Consumer
  10. 10. @NSilnitsky @NSilnitsky Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Kafka Growing Pains Cluster overload DC awareness
  11. 11. @NSilnitsky Greyhound Consumer Greyhound Producer Greyhound* Wraps Kafka * open source software Kafka Broker Kafka- based Global Data Mesh Kafka Producer Service A Service B Kafka Consumer
  12. 12. @NSilnitsky Greyhound Consumer Greyhound Producer Kafka Broker Kafka- based Global Data Mesh Kafka Producer Service A Service B Kafka Consumer
  13. 13. @NSilnitsky @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) val producer = GreyhoundBuilder.resilientProducer(topicName) Kafka- based Global Data Mesh
  14. 14. @NSilnitsky ✔ Parallel Consumption ✔ Batch Consumer ✔ Consumer Retry Strategies ✔ Resilient Producer ✔ Context Propagation ✔ Metrics reporting Self Service 🟊 Greyhound Kafka- based Global Data Mesh Kafka Broker
  15. 15. @NSilnitsky Scala NodeJS / Python / Go Greyhound Kafka Broker I’m a container gRPC + proto sidecar JVM library Kafka- based Global Data Mesh
  16. 16. @NSilnitsky Kafka Broker Domain Events ETL to DWH Task Scheduling Async Queue ... We use it for... Kafka- based Global Data Mesh Infra
  17. 17. @NSilnitsky @NSilnitsky Hundreds of developers Event-driven architecture Diverse tech stack Kafka + Greyhound Growing Pains Large scale traffic Cluster Overload DC awareness
  18. 18. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Kafka Broker Single Cluster Overload * intra-dc
  19. 19. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Single Cluster Overload A A A A B Kafka Cluster (by SLAs) B Multi Cluster C
  20. 20. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 A A A A B To which Cluster? Kafka Cluster (by SLAs) B I want to produce a domain event. C
  21. 21. @NSilnitsky @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) .ClusterA val producer = GreyhoundBuilder.resilientProducer(topicName, _.ClusterB Kafka- based Global Data Mesh * wix-layer
  22. 22. @NSilnitsky
  23. 23. @NSilnitsky @NSilnitsky Hundreds of developers Event-driven architecture Diverse tech stack Kafka + Greyhound Growing Pains Large scale traffic Cluster Overload DC awareness + Multi Cluster
  24. 24. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Master Active Master Passive Read-Only DBs Http/gRPC To which DC? I want to process the event and save the data.
  25. 25. @NSilnitsky @NSilnitsky Kafka + Greyhound + Cross DC services Kafka- based Global Data Mesh Growing Pains Hundreds of developers Event-driven architecture Diverse tech stack Large scale traffic Cluster Overload DC awareness + Multi Cluster
  26. 26. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. Replicate across DCs 2. Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
  27. 27. @NSilnitsky Replicate ! DC1 DC2 across DCs Greyhound producer Greyhound consumer Kafka Broker Cluster A Kafka Broker Cluster Kafka- based Global Data Mesh A
  28. 28. @NSilnitsky Greyhound Batch Consumer Greyhound Producer Batch # Replicate ! Kafka Broker Cluster A Kafka Broker Cluster DC1 DC2 How? Greyhound producer Greyhound consumer Kafka- based Global Data Mesh A Cross DC Replication Service
  29. 29. @NSilnitsky Replicate ! DC1 DC2 a Compact Topic for Policy A A Kafka Broker Cluster Kafka Broker Cluster Greyhound producer Greyhound consumer Kafka- based Global Data Mesh
  30. 30. @NSilnitsky Replication service Replicate ! DC1 DC2 How? A KVStoreReader In-Memory Map Key Value Topic1 AllDCs Topic2 Custom A Kafka Broker Cluster Kafka Broker Cluster Greyhound producer Greyhound consumer Kafka- based Global Data Mesh
  31. 31. @NSilnitsky Kafka- based Global Data Mesh
  32. 32. @NSilnitsky Kafka- based Global Data Mesh
  33. 33. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. ✔ Replicate across DCs 2. Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
  34. 34. @NSilnitsky Produce ! DC1 DC2 To main DC Read-Only DB / Master Passive DB Master Active A Kafka Broker Cluster Kafka- based Global Data Mesh Greyhound producer
  35. 35. @NSilnitsky main-dc-producer service Produce ! DC1 DC2 How? Master Active Kafka- based Global Data Mesh Greyhound gRPC producer producer(topicName, _.toMainDc)
  36. 36. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. ✔ Replicate across DCs 2. ✔ Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
  37. 37. @NSilnitsky DC1 DC2 To where needed [WIP] Just Produce ❤ Kafka- based Global Data Mesh Greyhound producer Greyhound consumer Master Active
  38. 38. @NSilnitsky Topology Topic List [consumer group:DC] DC1 DC2 A Kafka Broker Cluster Kafka- based Global Data Mesh Greyhound producer Greyhound consumer To where needed [WIP] Just Produce ❤ Master Active
  39. 39. @NSilnitsky Topology Topic List [consumer group:DC] DC1 DC2 A Kafka Broker Cluster Master Active Kafka- based Global Data Mesh Greyhound producer Greyhound consumer To where needed [WIP] Just Produce ❤ Replication service * reintroduce decoupling
  40. 40. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh DC1 DC2 DC3 DC4 A A A A B B Just Produce ❤ To a Single Logical Kafka Cluster. Kafka Broker Cluster DB
  41. 41. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh Any Tech Stack Greyhound Anywhere You Need developer
  42. 42. @NSilnitsky developer github.com/wix/greyhound Kafka- based Global Data Mesh
  43. 43. @NSilnitsky @NSilnitsky Thank You! natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions? Kafka- based Global Data Mesh
  44. 44. @NSilnitsky @NSilnitsky Q&A natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions? Kafka- based Global Data Mesh

×