Many organizations use Apache Kafka® to build data pipelines that span multiple geographically distributed data centers, for use cases ranging from high availability and disaster recovery, to data aggregation and regulatory compliance. The journey from single-cluster deployments to multi-cluster deployments can be daunting, as you need to deal with networking configurations, security models and operational challenges. Geo-replication support for Kafka has come a long way, with both open-source and commercial solutions that support various replication topologies and disaster recovery strategies. So, grab your towel, and join us on this journey as we look at tools, practices, and patterns that can help us build reliable, scalable, secure, global (if not inter-galactic) data pipelines that meet your business needs, and might even save the world from certain destruction.

1. A Hitchhiker's Guide to
Apache Kafka®
Geo-Replication
Sanjana Kaundinya | Senior Software Engineer
Rajini Sivaram | Principal Software Engineer

2. Overview of Kafka Replication

3. Kafka Overview
● Broker - Stores messages in partitions
● Topic - Virtual Group of one or more partitions
● Partitions - Log files on disk with only sequential writes.
Kafka guarantees message ordering in a partition.
Broker
T1
P
P2
P1
C
C
C
P

4. CG1
Kafka Log Offsets
P1
C1
P
C2
4 5 6 7 8 9
0 1 2 3
P2
P3
P4
Partition 1
__consumer_offsets
startOffset CG1 CG2 HW LEO
Produce
● Append Only Log
● Log End Offset
● High Watermark
● Consumer Offsets

5. Why do we Need Replication ?
How can a broker go down?
● Controlled shutdown
● Uncontrolled shutdown
What happens when a broker goes down?
● Durability
● Availability

6. Kafka Replication
● Partition replicas are evenly distributed
● Byte for byte copy of each other
● One replica is a leader and all writes go to the leader
● Leader decides when to commit data
P1
(L)
P2
(L)
P4
P1
P2
P3
P4
P3
(L)
P1
P3
P4
(L)
P2
Replication
Factor = 3

7. How are messages committed ?
● Leader maintains in sync replicas (ISR)
● Failure cases are handled with the use of a leader epoch
● The leader epoch is part of the message(KIP-101)
R1
(L)
R2
(L)
R4
(L)
R2
R1
R3
R4
R3
R3
R3
(L)
R4
R2
P1
(L)

8. Salient Points for Replication
● Intra cluster Replication helps improve durability and
availability for node level failures.
● Offsets are core piece of Kafka producer and consumer
ecosystem.
● Kafka Replication protocol ensures strong consistency
through byte for byte replication and providing
message ordering guarantees.

9. Multi Zone(MZ) HA Kafka Cluster
B B
ZK zk
P C
B
zk
AZ1 AZ2 AZ3
Inter Zone Latency <10 ms
Typical ~3 ms
ZK ZK

10. Why Do We Need To Globally Replicate ?
● Global Availability
● Protection against disasters
○ Natural disaster
○ Cloud provider outage
● Regulatory Compliance
● Aggregate Clusters
● IOT use cases
● Migration from one region to another

11. Differences Among Multi-DC Solutions
Stretched Clusters Connected Clusters

12. Kafka Stretched Clusters

13. Stretched Clusters
● Offset Preserving
● Fast Disaster Recovery
● Automated Client
Failover with No
Custom Code
● Sync or Async
Replication per Topic
with Confluent’s
Multi-Region Clusters
13

14. 3 DC Stretched Cluster

15. 2.5 DC Stretched Cluster

16. Fetch from
Followers
● With KIP-392,
consumers can
read from the
closest replica
● This helps to save
on networking
costs and helps
with overall
latency 16

17. Confluent Multi-Region Clusters (MRC)
Leader
Follower
Observer
● Sync vs Async replication
● Replica placement

18. MRC: Automatic Observer Promotion
Leader
Follower
Observer
observerPromotionPolicy
● under-min-isr
● under-replicated
● leader-is-observer

19. Network Considerations
● Single Kafka Cluster with
bi-directional connectivity
● Cost of cross-DC traffic
● Network Latency: < 50ms
between DCs
○ Sync: client impact
○ Async: durability
impact
● Network partitions
● Replication tuning: buffer
sizes, fetcher threads
19

20. Security Considerations
● Authentication using SSL
or SASL_SSL for
inter-broker connections
● Wire-encryption using
TLS
● Single Kafka Cluster
○ Single account and
access management
for clients
○ ACLs apply across
whole cluster 20

21. Kafka Connected Clusters

22. Clusters can
replicate using
Kafka Connect
● Have two separate
Kafka clusters in use
● Different from a single
stretched cluster
● Offset Translation
● MirrorMaker 2.0 and
Confluent Replicator
Connect based
Replication
22
C

23. Fundamentals of Kafka Connect
● Offset management
● Elastic scalability
● Parallelization
● Task distribution
● Failure & Retries
● Configuration Management
● REST API

24. Multi-Geo Replication Through
MirrorMaker 2
MirrorMaker 2

25. Offset Translation in MirrorMaker 2.0
offset_sync
topic,
partition,
src offset,
matching dest offset
checkpoints
topic,
partition,
group name,
consumer group src offset,
matching dest offset
Consumer
translateOffsets
Destination Cluster

26. Offset Translation in Replicator
26

27. Network Considerations
● Where to run Connect based
clusters?
○ local producer, remote
consumer
● Connectivity from Connect to
source and destination brokers
○ Firewalls
● High Latency networks
○ Kafka batch sizes
○ TCP buffers: OS level and
application level
○ Automatic window scaling
27

28. Security Considerations
● Credentials
○ Source credentials
○ Destination credentials
○ Externalize passwords
● Wire encryption using TLS
● Access control
○ Access to read from
source cluster
○ Access to write to
destination cluster
○ Naming conventions:
prefixed ACLs
28

29. Connecting Clusters
Sans Kafka Connect
● Multi continent
replication without the
an external system
● Offset preserving,
eliminating need for
offset translation
● Has similar use cases as
Kafka Connect based
architectures
Cluster Linking
29

30. Multi-Geographic Deployment Strategies
with Apache Kafka

31. Active-Passive
● One cluster is the
primary, other cluster
is the standby
● The primary cluster is
the only one written to
● Commonly used
topology used for
regulatory compliance
31
Producer
Active DC Passive DC
Consumer Consumer
Replication

32. Active-Active
● Two clusters replicate
to each other
● Records are produced
to both clusters and
seen by clients in both
clusters
● Used for a globally
distributed
architecture, data
needs to be regionally
available 32
Producer
Active DC Active DC
Consumer Consumer
Producer
Replication
Replication

33. Preventing Cyclic Replication in an
Active-Active Setup
How do connected clusters prevent cyclic replication?
● MirrorMaker 2.0 uses alias detection
● Confluent Replicator adds a provenance header to each
record which contains:
○ ID of the origin cluster
○ Name of the topic
○ Timestamp

34. Fan-In AKA
Aggregation
● Multiple clusters write
to one centralized
cluster
● Can aggregate into
one centralized topic
or do this on the
central cluster
● Use cases:
aggregation, analytics,
IOT 34
DC
Producer
Producer
DC
Aggregate
DC DC
Producer
R
e
p
l
i
c
a
t
i
o
n
Replication
Replication
Consumer

35. Fan-Out
● One cluster writes out
to multiple other
clusters
● Only one cluster is
actively produced to
● Use cases: expanded
version of
active-passive setups,
IOT
35
DC
Consumer
DC
Central DC DC
R
e
p
l
i
c
a
t
i
o
n
Replication
Replication
Producer
Consumer
Consumer

36. Disaster Recovery:
Failing Over
● If primary cluster goes
down, all producers
have to be move to the
secondary cluster
● Need to ensure that
consumer applications
can resume where
they last left off
36
R R
R
A - Primary
ZK
R R
R
B - Secondary
ZK
Producer
Replication
Consumer

37. 37
R R
R
B - Secondary
ZK
Disaster Recovery:
Failing Back
● Once the disaster is
mitigated, switch back
to the primary cluster
● Have to ensure client
applications can write
back to the original
cluster
R R
R
A - Primary
ZK
Producer
Consumer
Resume
Replication
Reconciliation

38. Operational
Last point system was
operational
Disaster
Disaster strikes and
system goes down
2
1
Recovery
Begin recovery after
disaster strikes
Normalcy
System back to being
operational
4
3
38
Disaster Recovery: Metrics
Recovery Point
Objective
Recovery Time Objective

39. Which multi-geo deployment to choose?
● It really depends!
● Considerations:
○ Cost
○ Business Requirements
○ Use Case
○ Regulatory Compliance
● Two must haves:
○ Resilient to disasters
○ Security

40. Questions?