Whether you know you want to run Apache Kafka in multiple data centers and need practical advice or you are wondering why some organizations even need more than one cluster, this online talk is for you.
In this short session, we’ll discuss the basic patterns of multi-datacenter Kafka architectures, explore some of the use-cases enabled by each architecture and show how Confluent Enterprise products make these patterns easy to implement.
Visit www.confluent.io for more information.
What Are The Drone Anti-jamming Systems Technology?
Common Patterns of Multi Data-Center Architectures with Apache Kafka
1. 1
Patterns of Multi Data-Center
Architectures
Gwen Shapira, Product Manager
@gwenshap
2. 2
What we’ll talk about today
• When is one cluster not enough?
• When is one DC not enough?
• Trade-offs in multi-DC architectures
• Architectures used in common use-cases
8. 8
Main trade-off
Low Latency / High
throughput
• Write to one DC,
replicate later
Strong Consistency
• Wait for multi-DC to
acknowledge writes
9. 9
Operationalizing is hard
• Multi-DC is EXPENSIVE
• Install, configure and upgrade multiple clusters
• Monitor and troubleshoot multiple clusters
• Figure out a multi-DC architecture
• Choose, install, configure replication solution
• Monitor replication
• Failover?
This is complex enough to warrant another talk…
10. 10
Geo-Locality
Why Geo-Locality
Data needs to be close to the users.
And the users are all over the place.
Main Challenges
• Topic names
• Management of replication pipelines
• Managing configuration
• Avoiding “loops”
15. 15
Multi-DC for Legalities
What Legalities
• Similar to geo-localization. but…
• Different countries have different data
storage laws
• But some data needs to be shared
• Laws regarding encryption
• Laws regarding privacy
• Also – legal usually wants failover
Main Challenges
• Avoid copying some data
• Encryption over the wire
• Lineage
• Security, audits
16. 16
SMT to the rescue!
• Simple message transforms.
• Route, filter or modify events with
just a bit of config
• Work with any connector –
specifically, Replicator.
• Pluggable – you can add your own
18. 18
Multi-DC for Cloud Migration
Why Cloud Migration?
• All the cool kids are moving to the cloud
• Likely to be a lengthy process
• Maybe on-prem + multi-clouds forever
• Many apps and teams involved
• Need well known central pipe
Main Challenges
• Most expensive network ever
• Random failure modes
• Many applications, teams, data-stores
• To be honest, Kafka is the easy part
19. 19
At first, this is
no big deal…. App
App
AppApp
DWH
DB
KV
App
DB
DC1 AWS
24. 24
Benefits of Kafka + Replicator for Cloud
Operations
1. Proven architecture
(Watch our online talk with Monsanto!)
2. Non-stop, low latency pipe
3. Cost savings
4. One throat to tune, manage, monitor,
secure and improve
Future-proof
1. Connect lets you explore cloud services
2. Avoid lock-in: ”Kafka is our escape valve”
3. Multi-zone, multi-region, multi-cloud…
4. Microservices ready
5. Streams ready
24
25. 25
Few general lessons
1. Don’t be afraid of many clusters
2. Decide if you need to scale clusters, data-centers, regions or all of above
3. Choose your trade-offs
4. It is better to consume over distance than to produce
5. Unless you stop consuming when you can’t produce
6. Security over the wire:
• SSL encryption for consumers takes LOTS of broker resources
• Maybe consume locally without SSL and produce remotely with SSL
7. Monitor, monitor, monitor. Especially lag
8. Tuning over WAN is different: https://docs.confluent.io/current/multi-dc/replicator-tuning.html
26. 26
Feature Benefit MirrorMaker Confluent Replicator
Data Replication Real-time event streaming between Kafka clusters and data-centers
Schema Replication Integrate with Confluent Schema Registry for multi-dc data quality and governance
Connect Replication Manage data integration across multiple data centers
Flexible topic selection Select topics with white-lists, black-lists and regular expressions
Auto-create topics New topics are automatically detected and replicated. Minimize admin overhead.
Add new partitions New partitions are automatically detected and replicated. Minimize admin overhead.
Configuration Replication
Topic configuration remains synchronized between the two clusters. Avoid configuration
diverging due to human error.
Auto-Scale Scale replication processes as Kafka traffic increases with a single configuration
Active-active replication Redirect events to avoid infinite replication loops in active-active configurations
Aggregate cluster One management point for replicating more than a single cluster
Control Center Integration Manage and monitor replication via Control Center UI
Support transformations Via Connect’s SMT – Lineage, routing, masking, filtering and more
Confluent Replicator is Enterprise-ready MirrorMaker
27. 27
We learned
1. Why you need multiple clusters. Or multiple Data Centers. Or multiple regions.
2. What are the trade-offs involved
3. Few common use-cases and architectures
4. Next week: Data recovery and failover