Increasingly, organizations are relying on Kafka for mission critical use-cases where high availability and fast recovery times are essential. In particular, enterprise operators need the ability to quickly migrate applications between clusters in order to maintain business continuity during outages. In many cases, out-of-order or missing records are entirely unacceptable. MirrorMaker is a popular tool for replicating topics between clusters, but it has proven inadequate for these enterprise multi-cluster environments. Here we present MirrorMaker 2.0, an upcoming all-new replication engine designed specifically to provide disaster recovery and high availability for Kafka. We describe various replication topologies and recovery strategies using MirrorMaker 2.0 and associated tooling.

1. © Cloudera, Inc. All rights reserved.
DISASTER RECOVERY WITH
MIRRORMAKER 2.0
Ryanne Dolan, 14th May 2019

2. © Cloudera, Inc. All rights reserved.
Kafka
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Clients connect directly to Kafka.
Step 0: Your Kafka today.
Producer

3. © Cloudera, Inc. All rights reserved.
Kafka 1
Kafka 2
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Producers can use VIPs, load balancers,
and health checks to failover transparently.
Step 1: Add a second cluster
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Two independent consumer groups.

4. © Cloudera, Inc. All rights reserved.
Kafka 1
Kafka 2
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Disaster strikes. Can still ingest.
Producer
Consumer
Consumer
Consumer
Consumer
Consumer

5. © Cloudera, Inc. All rights reserved.
More clusters, more
availability.

6. © Cloudera, Inc. All rights reserved.
What about data loss?

7. © Cloudera, Inc. All rights reserved.
Kafka 1
Kafka 2
Gateway
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Records sent to both clusters, but with
different order and offsets.
Step 2: Dual ingest
Producer
R1
R1
R1
R1

8. © Cloudera, Inc. All rights reserved.
Kafka 1
Kafka 2
Gateway
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Step 2: Dual ingest
Producer
R1
Where to resume processing?
R1

9. © Cloudera, Inc. All rights reserved.
Introducing:
MirrorMaker 2.0

10. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Producers send to either cluster.
Consumers can be migrated between
primary and secondary clusters.
Active/standby with MirrorMaker 2.0
Producer
MM2

11. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Data, offset syncs, and
consumer checkpoints.
Active/standby with MirrorMaker 2.0
Producer

12. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Migrate consumers to backup
cluster.
Primary down: migrate consumers.
Producer
MM2

13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved.
primary
secondary
Replicated topics are renamed
according to
ReplicationPolicy.
<source>.<topic>
topic1
topic2
secondary.topic1
secondary.topic2
topic1
topic2
primary.topic1
primary.topic2
Remote topics

14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved.
Checkpoint stream replicates
consumer group state.
topic=primary.topic1
partition=4
group=consumer-group-2
upstreamOffset=100
offset=102
Checkpoints
secondary
primary.checkpoints.internal

15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved.
Offset sync stream maps
offsets between mirrored
clusters.
topic=primary.topic1
partition=4
upstreamOffset=100
downstreamOffset=102
Offset Syncs
secondary
primary.offset-syncs.internal

16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved.
Cross-cluster offset translation
Translate offsets between clusters via RemoteClusterUtils
Map<TopicPartition, Long> newOffsets =
RemoteClusterUtils.translateOffsets(
newClusterProperties, oldClusterName,
consumerGroupId);
consumer.seek(newOffsets);
● offset translation based on checkpoints in new cluster
● no connection to old cluster required

17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved.
MirrorMaker 2.0 configuration file
# mm2.properties
clusters = primary, secondary
primary.bootstrap.servers = …
secondary.bootstrap.servers = …
primary->secondary.enabled = true
secondary->primary.enabled = true

18. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Primary permanently lost? Recover from secondary.
Producer
primary-2

19. © Cloudera, Inc. All rights reserved. 19© Cloudera, Inc. All rights reserved.
MirrorMaker 2.0 configuration file
# mm2.properties
clusters = secondary, primary-2
secondary.bootstrap.servers = …
primary-2.bootstrap.servers = …
primary-2->secondary.enabled = true
secondary->primary-2.enabled = true

20. © Cloudera, Inc. All rights reserved. 20© Cloudera, Inc. All rights reserved.
primary-2
secondary
Lost primary topics can be
recovered from remote topics
on secondary cluster.
topic1
topic2
secondary.topic1
secondary.topic2
secondary.primary.topic1
secondary.primary.topic2
topic1
topic2
primary.topic1
primary.topic2
primary-2.topic1
primary-2.topic2
Recovered topics

21. © Cloudera, Inc. All rights reserved. 21© Cloudera, Inc. All rights reserved.
primary-2
secondary
topic1
topic2
secondary.topic1
secondary.topic2
secondary.primary.topic1
secondary.primary.topic2
topic1
topic2
primary.topic1
primary.topic2
primary-2.topic1
primary-2.topic2
Recovered topics
data from old primary
data from old primary

22. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Producers send to either cluster.
Active/active with MirrorMaker 2.0
Producer
MM2
Consumer
Consumer
Consumer
Consumer
Consumer
Consumers in both clusters.

23. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Cross-cluster consumer groups
Producer
MM2
Consumer
Consumer
Consumer
Consumer
Consumer
R1
R1
R1

24. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Cross-cluster consumer groups
Producer
MM2
Consumer
Consumer
Consumer
Consumer
Consumer
R1
R1
R1
R2
R2
R2
Only one consumer processes
each record.

25. © Cloudera, Inc. All rights reserved.
primary
secondary
Load
balancer
Producer
Producer
Producer
Producer
Producer
Consumer
Consumer
Consumer
Consumer
Consumer
Effectively one big consumer group
Producer
MM2
Consumer
Consumer
Consumer
Consumer
Consumer
R1
R1
R1
R2
R2
R2
Only one consumer processes
each record.

26. © Cloudera, Inc. All rights reserved.
us-eastus-west
Cross-data center replication (XDCR)
us-north
Producer
LB LB LBLBs prefer their
local DC, but can
failover to other
DCs
ConsumerConsumerConsumer
Consumers see
records from all
DCs.

27. © Cloudera, Inc. All rights reserved.
Producers failover to another data center.
us-northus-west us-east
Producer
LB LB LB
ConsumerConsumerConsumer

28. © Cloudera, Inc. All rights reserved.
Entire DC permanently lost? Stand up a new Kafka cluster.
us-northus-west us-east
Producer
LB LB LB
ConsumerConsumerConsumer

29. © Cloudera, Inc. All rights reserved.
New us-west-2 cluster can be recovered from other DCs.
us-northus-west-2 us-east
Producer
LB LB LB
ConsumerConsumer

30. © Cloudera, Inc. All rights reserved.
New cluster back online.
us-northus-west-2 us-east
Producer
LB LB LB
ConsumerConsumerConsumer

31. © Cloudera, Inc. All rights reserved.
Reference architecture: XDCR with active/active AZs
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB

32. © Cloudera, Inc. All rights reserved.
Producers send to nearby DC
DC-1 DC-2
K1 K2 K3 K4
LB LB
P
P

33. © Cloudera, Inc. All rights reserved.
Consumers in each zone. Same group.id.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C

34. © Cloudera, Inc. All rights reserved.
Effectively two big consumer groups.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C

35. © Cloudera, Inc. All rights reserved.
Record processed by one consumer in each DC.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C
P
R1
R1 R1

36. © Cloudera, Inc. All rights reserved.
HA: ingest and process with any single K cluster.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C
P
R2
R2

37. © Cloudera, Inc. All rights reserved.
K1 catches up. Records are not re-processed in AZ-1.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C
R2
P
R2

38. © Cloudera, Inc. All rights reserved.
K3 catches up. Records processed in DC-2 for first time.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C
R2 R2
P
R2

39. © Cloudera, Inc. All rights reserved.
K4 catches up. Records are not re-processed in AZ-4.
DC-1 DC-2
AZ-1 AZ-2 AZ-3 AZ-4
K1 K2 K3 K4
LB LB
C C C C
R2 R2
P
R2

40. © Cloudera, Inc. All rights reserved.
MM2 config
# mm2.properties
clusters = K1, K2, K3, K4
K1.bootstrap.servers = …
# active/active w/in DC-1
K1->K2.enabled = true
K2->K1.enabled = true
# active/active w/in DC-2
K3->K4.enabled = true
K4->K3.enabled = true
# XDCR b/t DCs
K1->K3.enabled = true
K3->K1.enabled = true

41. © Cloudera, Inc. All rights reserved.
Consumer config
• Same consumer group.id across all AZs, DCs.
• DC-1 subscription:
• topic1
• K3.topic1
• K3.K4.topic1
• DC-2 subscription:
• topic1
• K1.topic1
• K1.K2.topic1
Alternatively: .*.topic1

42. © Cloudera, Inc. All rights reserved.
THANK YOU