Download as PDF, PPTX
Uploaded byHostedbyConfluent
MirrorMaker: Beyond the Basics with Mickael Maison
The document provides a comprehensive guide on how to run Apache Kafka's MirrorMaker in production, covering the setup of connectors, operational scenarios like hub and spoke and disaster recovery, and detailed configurations for mirrorsource and mirrorcheckpoint connectors. It includes instructions for monitoring performance and recent features introduced in version 3.5, such as exactly-once source support and offset management APIs. Additionally, it emphasizes the need for ongoing monitoring and processes to ensure smooth operations and failover conditions.
More Related Content
Similar to MirrorMaker: Beyond the Basics with Mickael Maison
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
MirrorMaker: Beyond the Basics with Mickael Maison
- 1.
- 2. 2 How to runMirrorMaker in production? 1. Start the connectors 2. Operate in production
- 3. 1. Hub andSpoke 2. Disaster Recovery 3
- 4. 4 Review of theConnectors
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. 11 MirrorSource - OffsetSync 4243 45 44 SourceTask my-topic-0 my-topic-0 offset-syncs
- 12. 12 MirrorSource - OffsetSync SourceTask my-topic-0my-topic-0 offset-syncs 0 my-topic-0 42 ➔ 0 Current Offset: 42 42 43 45 44
- 13. 13 MirrorSource - OffsetSync SourceTask my-topic-0my-topic-0 offset-syncs 0 1 Current Offset: 43 my-topic-0 42 ➔ 0 42 43 45 44
- 14. 14 MirrorSource - OffsetSync SourceTask my-topic-0my-topic-0 offset-syncs 0 1 Current Offset: 44 my-topic-0 42 ➔ 0 42 43 45 44 Control Record
- 15. 15 MirrorSource - OffsetSync 12 0 SourceTask my-topic-0 my-topic-0 offset-syncs Current Offset: 45 my-topic-0 42 ➔ 0 my-topic-0 45 ➔ 2 42 43 45 44
- 16.
- 17.
- 18.
- 19.
- 20. 20 MirrorCheckpoint Groups Groups OffsetSyncs Checkpoints KafkaConnect MirrorCheckpoint Connector Tasks
- 21. 21 MirrorCheckpoint - OffsetTranslation offset-syncs Commits group1 my-topic-0 42 group2 my-topic-0 48 Commits my-topic-0 42 ➔ 0 my-topic-0 45 ➔ 2 CheckpointTask
- 22. 22 MirrorCheckpoint - OffsetTranslation offset-syncs Commits group1 my-topic-0 42 group2 my-topic-0 48 Checkpoints my-topic-0 42 ➔ 0 my-topic-0 45 ➔ 2 CheckpointTask group1 my-topic-0 0
- 23. 23 MirrorCheckpoint - OffsetTranslation offset-syncs Commits group1 my-topic-0 42 group2 my-topic-0 48 Commits my-topic-0 42 ➔ 0 my-topic-0 45 ➔ 2 CheckpointTask group1 my-topic-0 0 group2 my-topic-0 5
- 24.
- 25.
- 26.
- 27. 27 Building a Scenario Whichconnectors do you need?
- 28. 28 Building a Scenario Whichconnectors do you need? What features are required?
- 29. 29 Which connectors doyou need? What features are required? What monitoring do you need? Building a Scenario
- 30. 30 Which connectors doyou need? What features are required? What monitoring do you need? What processes do you need to put in place? Building a Scenario
- 31.
- 32.
- 33. 33 Hub and Spoketoo!
- 34. 34 Hub and SpokeScenario MirrorSource Kafka Connect FR UK mm2-offset-syncs.UK.internal sales FR.sales IT.sales sales
- 35. 35 Hub and Spoke- Boilerplate "name": "FR->UK", "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
- 36. 36 Hub and Spoke- Boilerplate #2 "source.cluster.alias": "FR", "target.cluster.alias": "UK", "source.cluster.bootstrap.servers": "FR-kafka0:9092", "target.cluster.bootstrap.servers": "UK-kafka0:9092", "replication.factor": "-1", "offset-syncs.topic.replication.factor": "-1",
- 37. 37 Hub and Spoke- Key configs "sync.topic.con fi gs.enabled": "false", "sync.topic.acls.enabled": "false", "topics": "sales", "tasks.max": "10",
- 38. 38 Hub and Spoke- Monitoring • Validate initial setup • MirrorSourceConnector Metrics • byte-rate/record-count • record-age-ms • replication-latency-ms
- 39. 39 Hub and Spoke- Processes • Keep monitoring • No further changes required
- 40.
- 41. 41 Disaster Recovery Scenario MirrorSource KafkaConnect A B mm2-offset-syncs.A.internal sales sales MirrorCheckpoint A.checkpoints.internal
- 42. 42 Disaster Recovery -Source "name": "A->B - Source", "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
- 43. 43 Disaster Recovery -Source "source.cluster.alias": "A", "target.cluster.alias": "B", "source.cluster.bootstrap.servers": "A-kafka0:9092", "target.cluster.bootstrap.servers": "B-kafka0:9092", "replication.factor": "-1", "offset-syncs.topic.replication.factor": "-1",
- 44. 44 Disaster Recovery -Source "replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy", "offset-syncs.topic.location": "target", "topics": "sales.*,inventory.*," "tasks.max": "10",
- 45. 45 Disaster Recovery -Checkpoint "name": "A->B - Checkpoint", "connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
- 46. 46 Disaster Recovery -Checkpoint "source.cluster.alias": "A", "target.cluster.alias": "B", "source.cluster.bootstrap.servers": "A-kafka0:9092", "target.cluster.bootstrap.servers": "B-kafka0:9092", "checkpoints.topic.replication.factor": "-1",
- 47. 47 Disaster Recovery -Checkpoint "replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy", "offset-syncs.topic.location": "target", "sync.group.offsets.enabled": "true", "groups": "sales.*,inventory.*," "tasks.max": "10"
- 48. 48 Disaster Recovery -Monitoring MirrorSourceConnector Metrics • byte-rate/record-count • record-age-ms • replication-latency-ms MirrorCheckpointConnector Metrics • checkpoint-latency-ms
- 49. 49 Disaster Recovery -Processes • Exercise failover • Define conditions for failover • Prepare failback process
- 50.
- 51. 51 MirrorMaker Exactly Once- New in 3.5 • exactly.once.source.support on Workers https://kafka.apache.org/documentation/#connect_exactlyoncesource
- 52. 52 Offset Management REST APIsfor: • listing offsets - New in 3.5 • deleting offsets • updating offsets https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
- 53. 53 Source Cluster Tag- New in 3.5 • add.source.alias.to.metrics (default from 4.0) kafka.connect.mirror:type=MirrorSourceConnector source=FR target=UK topic=FR.sales partition=0
- 54. 54 11 Chapters availablein early access https://www.oreilly.com/library/view/kafka-connect/9781098126520/ https://www.redhat.com/en/engage/build-data-pipelines-20220603
- 55.