Download to read offline
![@NSilnitsky
500B >600
GAs Per Day
~1500
Developers
Daily HTTP
Transactions
18
Data
Centers &
Pops
2
Cloud Providers
[Google/AWS]
Scaling out
Migrating to Multi Cluster Managed Kafka](https://image.slidesharecdn.com/devoxxuk-migratingtomulticlustermanagedkafka-220511055553-9dccfec4/95/Devoxx-UK-Migrating-to-Multi-Cluster-Managed-Kafka-5-638.jpg)
More Related Content
Similar to Devoxx UK - Migrating to Multi Cluster Managed Kafka
Devoxx UK - Migrating to Multi Cluster Managed Kafka
- 1. @NSilnitsky Migrating to MultiCluster Managed Kafka Migrating to a Multi-Cluster Managed Kafka with 0 Downtime Natan Silnitsky Backend Infra TL, Wix.com natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 2. @NSilnitsky Migrating to MultiCluster Managed Kafka
- 3. @NSilnitsky Migrating to MultiCluster Managed Kafka Enable any business, community or person on earth to make their online dream come true. Our mission
- 4. @NSilnitsky Migrating to MultiCluster Managed Kafka At Wix ±1B Unique visitors >100M Websites Published by Wix users (5-7% of all internet websites) ±5000 People Work at Wix use Wix platform every month
- 5. @NSilnitsky 500B >600 GAs PerDay ~1500 Developers Daily HTTP Transactions 18 Data Centers & Pops 2 Cloud Providers [Google/AWS] Scaling out Migrating to Multi Cluster Managed Kafka
- 6. @NSilnitsky Wix has over2300 microservices in production Migrating to Multi Cluster Managed Kafka
- 7. Wix Engineering Locations UkraineIsrael EU Kyiv Tel-Aviv Vilnius Dnipro Be’er Sheva Berlin Haifa Krakow
- 8. @NSilnitsky Kafka in Wix Migratingto Multi Cluster Managed Kafka 2019 1 cluster per region, self hosted 5 K Topics > 45 K Partitions ~ 450 M Messages produced a day Kafka Broker
- 9. @NSilnitsky Migrating to MultiCluster Managed Kafka Kafka in Wix Last Year 2021 1 cluster per region, self hosted 20 K Topics > 200 K Partitions 3 B Messages produced a day
- 10. @NSilnitsky Migrating to MultiCluster Managed Kafka 2021 1 cluster per region, self hosted To multi cluster, managed Kafka platform So, migrate all this 20 K Topics > 200 K Partitions 3 B Messages produced a day overloaded 1. Better Cluster performance & flexibility 2. Transparent version upgrade 3. Easy to add a new Cluster 4. Tiered Storage
- 11. @NSilnitsky Migrating to MultiCluster Managed Kafka Wix wraps Kafka with Greyhound, a Scala/Java high-level SDK. ~2300 Wix microservices Kafka Producer Kafka Consumer Greyhound Producer Greyhound Consumer
- 12. @NSilnitsky Migrating to MultiCluster Managed Kafka Kafka Producer Greyhound Producer Greyhound Consumer Kafka Consumer Kafka Broker Checkout Service example Payments Service
- 13. @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) valproducer = GreyhoundBuilder.resilientProducer(topicName) Migrating to Multi Cluster Managed Kafka
- 14. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 15. @NSilnitsky Kafka Brokers Cluster DC1 DC2 DC3DC4 Single Cluster Overload * intra-dc
- 16. @NSilnitsky DC1 DC2 DC3 DC4 Single Cluster Overload A A A A B KafkaCluster (by SLAs) B Multi Cluster C
- 17. @NSilnitsky DC1 DC2 DC3 DC4 Towhich Cluster? Kafka Cluster (by SLAs) A A A A B B C I want to produce a domain event.
- 18. @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) .ClusterA valproducer = GreyhoundBuilder.resilientProducer(topicName, _.ClusterB The Multi Cluster (Kafka)
- 19. @NSilnitsky Cluster A +DC1 Cluster A + DC2 The Multi Cluster (Kafka)
- 20. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 21. @NSilnitsky Unbalanced brokers Unclear Kafkastrategy Too many partitions Real production impact → → → → Our Starting Point Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 22. @NSilnitsky Migrate on drainedTraffic? Migrations we ❤ Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 23. @NSilnitsky (Blockers) → Specific DCservices → Long time → Not gradual - Edge cases risk Q4 2020: CANCELED Migrate on drained Traffic? Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 24. @NSilnitsky … HAS tobe Seamless & Production-safe. Migrate With Traffic! Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 25. @NSilnitsky Automate, Automate, Automate. Multi Cluster To Single Cluster From overloadedself-hosted managed optimized … HAS to be Seamless & Production-safe.
- 26. Option B The Migration HTTP Service Greyhound Kafka Cluster Self-hosted DataCenter 1 Confluent Cloud @NSilnitsky
- 27. Option B The Migration Kafka Cluster Self-hosted DataCenter 1 Confluent Cloud Replicator service @NSilnitsky
- 28. Option B The Migration DataCenter 1 Topic 1 Partition 5 4 5 5 5 6 5 7 5 8 5 9 6 0 6 1 6 2 6 3 6 4 4 5 6 7 8 9 1 0 0 1 2 3 1. Consume 3. Save offset Mapping 2. Produce Confluent Cloud Replicator service Kafka Cluster Self-hosted @NSilnitsky
- 29. Option B The Migration Greyhound Kafka Cluster Self-hosted DataCenter 1 Replicator service Migration Orchestrator Confluent → @NSilnitsky
- 30. Option B The Migration Greyhound Kafka Cluster Self-hosted DataCenter 1 Replicator service Migration Orchestrator group of consumers Migrate Consumer: 1. Replicate to Confluent Confluent → @NSilnitsky
- 31. Greyhound Kafka Cluster Self-hosted Replicator service Migration Orchestrator Listen to events Option B TheMigration Subscribe Data Center 1 Unsubscribe Migrate Consumer: 1. Replicate to Confluent 2. Unsubscribe from self-hosted Kafka Cluster 3. Subscribe to Confluent 4. Rollback (Seek offset) success! Failure! @NSilnitsky Confluent →
- 32. @NSilnitsky The Migration Best Practices 1.Create a script that checks state by itself and stops if expected state is not reached. 2. Have a rollback readily available. 3. Start with test topics and no impact topics 4. Create custom metrics dashboards that show current state. The Migration
- 33. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 34. @NSilnitsky Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic1 What to Expect when Migrating to Multi Cluster During Nov/Dec 2021
- 35. @NSilnitsky “ Unexpected error from SyncGroup: Theserver experienced an unexpected error when processing the request. Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic 1 What to Expect when Migrating to Multi Cluster During Nov/Dec 2021
- 36. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 37. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic 1 It’s Christmas eve 🎄 2021. “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 38. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=8388608 “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 39. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=8388608 Kafka records start getting DELETED faster than expected (for compact topics too) “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 40. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Restored records from another Data Center Kafka records start getting DELETED faster than expected (for compact topics too) Luckily for us ... “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 41. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Affected Broker versions: 1.1.0, 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1 Fix: Changed dummy value for all topic configs “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 42. @NSilnitsky Replicator service Self Hosted Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic1 What to Expect when Migrating to Multi Cluster Topic 2 Topic N Topic 1 Sharded Consumers ... A B
- 43. @NSilnitsky Greyhound Kafka Cluster Self-hosted Replicator service Migration Orchestrator Unsubscribe What toExpect when Migrating to Multi Cluster We have the infra in place to control it from the outside! So we will soon be able to … 1. Switch Cluster 2. Skip Messages 3. Change processing rate Without GA! Confluent →
- 44. @NSilnitsky Migrating to MultiCluster Managed Kafka We used Greyhound & dedicated orchestration services for an automatic, safe, and gradual migration. Multi Cluster To Single Cluster From overloaded self-hosted Migrations we ❤ managed optimized
- 45.
- 46. @NSilnitsky A Scala/Java high-levelSDK for Apache Kafka. github.com/wix/greyhound Migrating to Multi Cluster Managed Kafka 0.2 is out!
- 47. @NSilnitsky Thank You! Migrating to MultiCluster Managed Kafka natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions?