"As Kafka’s popularity grows, enterprises often find themselves deploying not a single cluster, but several clusters. This is usually because of scale, but can be driven by other needs such as activities in different geographies or compliance requirements. Whenever you have multiple clusters, the ability to move data between clusters is very valuable. It enables very different scenarios such as geo-replication, disaster recovery, hybrid cloud architectures, but also simply allows reusing data between applications. MirrorMaker is the Apache Kafka project’s solution for mirroring clusters and consists of 3 connectors for Connect. This tool allows creating all sorts of mirroring topologies by combining the connectors in different ways. In this session we will demonstrate how to use MirrorMaker by exploring two of the most common mirroring use cases: geo-replication and disaster recovery. For each we’ll explain the combination of connectors to use and highlight the key decision points and configurations that you should carefully consider. At the end of the session you will understand the capabilities of MirrorMaker and the process of building powerful mirroring scenarios with this tool."

1. MirrorMaker:
Beyond the Basics
Mickael Maison
1

2. 2
How to run MirrorMaker in production?
1. Start the connectors 2. Operate in production

3. 1. Hub and Spoke
2. Disaster Recovery
3

4. 4
Review of the Connectors

5. 5
A
MirrorSource

6. 6
MirrorSource
Connector
Kafka Connect
Topics
MirrorSource

7. 7
MirrorSource
Connector
Kafka Connect
Topics Topics
MirrorSource

8. 8
MirrorSource
Connector
Kafka Connect
Topics Topics
Topics
Configs
ACLs
MirrorSource

9. 9
Kafka Connect
Tasks
Partitions Partitions
MirrorSource
MirrorSource
Connector

10. 10
Kafka Connect
Partitions Partitions
MirrorSource
OffsetSyncs
MirrorSource
Connector
Tasks

11. 11
MirrorSource - OffsetSync
42 43 45
44 SourceTask
my-topic-0 my-topic-0
offset-syncs

12. 12
MirrorSource - OffsetSync
SourceTask
my-topic-0 my-topic-0
offset-syncs
0
my-topic-0 42 ➔ 0
Current Offset: 42
42 43 45
44

13. 13
MirrorSource - OffsetSync
SourceTask
my-topic-0 my-topic-0
offset-syncs
0 1
Current Offset: 43
my-topic-0 42 ➔ 0
42 43 45
44

14. 14
MirrorSource - OffsetSync
SourceTask
my-topic-0 my-topic-0
offset-syncs
0 1
Current Offset: 44
my-topic-0 42 ➔ 0
42 43 45
44
Control Record

15. 15
MirrorSource - OffsetSync
1 2
0
SourceTask
my-topic-0 my-topic-0
offset-syncs
Current Offset: 45
my-topic-0 42 ➔ 0
my-topic-0 45 ➔ 2
42 43 45
44

16. 16
MirrorCheckpoint

17. 17
MirrorCheckpoint
MirrorCheckpoint
Connector
Kafka Connect
Groups

18. 18
MirrorCheckpoint
Groups
Kafka Connect
MirrorCheckpoint
Connector

19. 19
MirrorCheckpoint
Tasks
Groups
OffsetSyncs Checkpoints
Kafka Connect
MirrorCheckpoint
Connector

20. 20
MirrorCheckpoint
Groups Groups
OffsetSyncs Checkpoints
Kafka Connect
MirrorCheckpoint
Connector
Tasks

21. 21
MirrorCheckpoint - Offset Translation
offset-syncs
Commits
group1 my-topic-0 42
group2 my-topic-0 48
Commits
my-topic-0 42 ➔ 0
my-topic-0 45 ➔ 2
CheckpointTask

22. 22
MirrorCheckpoint - Offset Translation
offset-syncs
Commits
group1 my-topic-0 42
group2 my-topic-0 48
Checkpoints
my-topic-0 42 ➔ 0
my-topic-0 45 ➔ 2
CheckpointTask
group1 my-topic-0 0

23. 23
MirrorCheckpoint - Offset Translation
offset-syncs
Commits
group1 my-topic-0 42
group2 my-topic-0 48
Commits
my-topic-0 42 ➔ 0
my-topic-0 45 ➔ 2
CheckpointTask
group1 my-topic-0 0
group2 my-topic-0 5

24. 24
MirrorHeartbeat

25. 25
MirrorHeartbeat
MirrorHeartbeat
Connector
Kafka Connect
Task
heartbeats

26. 26

27. 27
Building a Scenario
Which connectors do you need?

28. 28
Building a Scenario
Which connectors do you need?
What features are required?

29. 29
Which connectors do you need?
What features are required?
What monitoring do you need?
Building a Scenario

30. 30
Which connectors do you need?
What features are required?
What monitoring do you need?
What processes do you need to put in place?
Building a Scenario

31. 31
Hub and Spoke

32. 32
Hub and Spoke

33. 33
Hub and Spoke too!

34. 34
Hub and Spoke Scenario
MirrorSource
Kafka Connect
FR UK
mm2-offset-syncs.UK.internal
sales
FR.sales
IT.sales
sales

35. 35
Hub and Spoke - Boilerplate
"name": "FR->UK",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",

36. 36
Hub and Spoke - Boilerplate #2
"source.cluster.alias": "FR",
"target.cluster.alias": "UK",
"source.cluster.bootstrap.servers": "FR-kafka0:9092",
"target.cluster.bootstrap.servers": "UK-kafka0:9092",
"replication.factor": "-1",
"offset-syncs.topic.replication.factor": "-1",

37. 37
Hub and Spoke - Key configs
"sync.topic.con
fi
gs.enabled": "false",
"sync.topic.acls.enabled": "false",
"topics": "sales",
"tasks.max": "10",

38. 38
Hub and Spoke - Monitoring
• Validate initial setup
• MirrorSourceConnector Metrics
• byte-rate/record-count
• record-age-ms
• replication-latency-ms

39. 39
Hub and Spoke - Processes
• Keep monitoring
• No further changes required

40. 40
Disaster Recovery

41. 41
Disaster Recovery Scenario
MirrorSource
Kafka Connect
A B
mm2-offset-syncs.A.internal
sales sales
MirrorCheckpoint
A.checkpoints.internal

42. 42
Disaster Recovery - Source
"name": "A->B - Source",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",

43. 43
Disaster Recovery - Source
"source.cluster.alias": "A",
"target.cluster.alias": "B",
"source.cluster.bootstrap.servers": "A-kafka0:9092",
"target.cluster.bootstrap.servers": "B-kafka0:9092",
"replication.factor": "-1",
"offset-syncs.topic.replication.factor": "-1",

44. 44
Disaster Recovery - Source
"replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy",
"offset-syncs.topic.location": "target",
"topics": "sales.*,inventory.*,"
"tasks.max": "10",

45. 45
Disaster Recovery - Checkpoint
"name": "A->B - Checkpoint",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",

46. 46
Disaster Recovery - Checkpoint
"source.cluster.alias": "A",
"target.cluster.alias": "B",
"source.cluster.bootstrap.servers": "A-kafka0:9092",
"target.cluster.bootstrap.servers": "B-kafka0:9092",
"checkpoints.topic.replication.factor": "-1",

47. 47
Disaster Recovery - Checkpoint
"replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy",
"offset-syncs.topic.location": "target",
"sync.group.offsets.enabled": "true",
"groups": "sales.*,inventory.*,"
"tasks.max": "10"

48. 48
Disaster Recovery - Monitoring
MirrorSourceConnector Metrics
• byte-rate/record-count
• record-age-ms
• replication-latency-ms
MirrorCheckpointConnector Metrics
• checkpoint-latency-ms

49. 49
Disaster Recovery - Processes
• Exercise failover
• Define conditions for failover
• Prepare failback process

50. 50
Recent/upcoming features

51. 51
MirrorMaker Exactly Once - New in 3.5
• exactly.once.source.support on Workers
https://kafka.apache.org/documentation/#connect_exactlyoncesource

52. 52
Offset Management
REST APIs for:
• listing offsets - New in 3.5
• deleting offsets
• updating offsets
https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect

53. 53
Source Cluster Tag - New in 3.5
• add.source.alias.to.metrics (default from 4.0)
kafka.connect.mirror:type=MirrorSourceConnector
source=FR
target=UK
topic=FR.sales
partition=0

54. 54
11 Chapters available in early access
https://www.oreilly.com/library/view/kafka-connect/9781098126520/
https://www.redhat.com/en/engage/build-data-pipelines-20220603

55. 55
https://kafka.apache.org/documentation/#georeplication
https://red.ht/TryKafka
Thank you
https://mas.to/@MickaelMaison
https://twitter.com/MickaelMaison