A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff Gilmore, AWS
•Download as PPTX, PDF•
0 likes•918 views
From migrations between Apache Kafka clusters to multi-region deployments across datacenters, the introduction of MirrorMaker2 has expanded the possibilities for Apache Kafka deployments and use cases. In this session you will learn about patterns, best practices, and learnings compiled from running MirrorMaker2 in production at every scale.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff Gilmore, AWS
Similar to A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff Gilmore, AWS (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff Gilmore, AWS
- 1. A Look into the Mirror Patterns and Best Practices for MirrorMaker2 © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1 Cliff Gilmore Principal Solutions Architect – Streaming Data
- 2. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2 Agenda • MirrorMaker2 Overview • Live Migration • Active Passive • Hub and Spoke / Aggregation • Active Active • Capacity Planning • Best Practices
- 3. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3 Key MirrorMaker2 Features • Multi-cluster data replication engine based on Kafka Connect framework • Detects new topics and partitions in self-discovery mode • Automatically synchronizes topic configuration & ACLs • Supports “active/active” cluster pairs, as well as any number of active clusters • Provides new metrics including end-to-end replication latency across multiple clusters • Emits the offsets required to migrate consumers between clusters and tooling for offset translation (in Apache Kafka 2.7, offset sync is also automated) • Configuration file for specifying multiple clusters and replication flows in one place
- 4. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4 MirrorMaker2 Core Concepts • Connectors • MirrorSourceConnector • MirrorSinkConnector • MirrorCheckpointConnector • MirrorHeartbeatConnector • Replication Policies • Controls naming of replicated topics • DefaultReplicationPolicy • <source>.<topic> convention • CustomReplicationPolicy • Create your own policy based on preferences (IE, no prefix for migration/active-passive) • Configuration Parameters • topics lets you control which topics are replicated • Groups lets you control which consumer groups have offsets replicated • More configuration for options, connectivity and topology defined at • https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
- 5. Live Migrations with MM2 © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5
- 6. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6 Live Migration Pattern (Initial State) Source Cluster Destination Cluster Consumer Producers
- 7. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7 Live Migration Pattern (Enable Replication) Source Cluster Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Topic Names stay consistent via CustomReplicationPolicy
- 8. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8 Live Migration Pattern (Stop Consumer, Update Offsets) Source Cluster Destination Cluster Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Note: OffsetSync can be Automated with the MirrorMaker2 version part of Apache Kafka 2.7 release RemoteClusterUtils to update offsets
- 9. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9 Live Migration Pattern (Start Consumer on Destination) Source Cluster Destination Cluster Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Consumer
- 10. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10 Live Migration Pattern (Point Producers to Destination) Source Cluster Destination Cluster Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Consumer Producers
- 11. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11 Live Migration Pattern (Shut Down Source/Connect) Destination Cluster Consumer Producers • The source cluster can now be safely shut down • This leaves us with our clients migrated to the new cluster • Very little delay is introduced with just the restart time of each topic’s producers and consumers being the disruption
- 12. Active – Passive Replication with DR Failover/Back © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
- 13. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13 Active-Passive (Normal State) Source Cluster Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Datacenter/Region #1 Datacenter/Region #2 topic1 topic1 region1.topic1
- 14. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14 Active-Passive (Cluster Failure, Bounce clients) Source Cluster Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter/Region #1 Datacenter/Region #2 topic1 region1.topic1 RemoteClusterUtils to update offsets
- 15. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15 Active-Passive (Datacenter Failure) Source Cluster Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter/Region #1 Datacenter/Region #2 Copy of Producers Consumers with same group ID topic1 region1.topic1 RemoteClusterUtils to update offsets
- 16. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16 Active-Passive (Reverse Replication) Source Cluster Destination Cluster Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter/Region #1 Datacenter/Region #2 Copy of Producers Consumers with same group ID topic1 region1.topic1 topic1 region2.topic1
- 17. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17 Active-Passive (Reverse Replication) Source Cluster Destination Cluster Kafka Connect MirrorSinkConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter/Region #1 Datacenter/Region #2 Copy of Producers Consumers with same group ID topic1 region1.topic1 topic1 region2.topic1 Replication for Topics and Offsets Replication for Topics and Offsets Dest Cluster Heartbeats RemoteClusterUtils to update offsets
- 18. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18 Active-Passive (Failback Producer/Consumer) Source Cluster Destination Cluster Kafka Connect MirrorSinkConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter/Region #1 Datacenter/Region #2 topic1 region1.topic1 topic1 region2.topic1 Replication for Topics and Offsets Replication for Topics and Offsets Dest Cluster Heartbeats Consumer Producers
- 19. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19 Active-Passive (Resume Normal Replication) Source Cluster Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Replication for Topics and Offsets Replication for Topics and Offsets Source Cluster Heartbeats Datacenter/Region #1 Datacenter/Region #2 topic1 region2.topic1 topic1 region1.topic1
- 20. Hub and Spoke aka Aggregation Pattern © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
- 21. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21 Hub and Spoke/Aggregation Destination Cluster Consumer Producers Kafka Connect MirrorSourceConnector Replication for Topics Consumer Producers Consumer Producers Consumer Local Datacenter/Region #2 Local Datacenter/Region #1 Local Datacenter/Region #3 Central Datacenter/Region region1.topic1 region2.topic1 region3.topic1
- 22. Active – Active also useful for Cloud Bridging © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
- 23. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23 Active-Active OnPrem Cluster Cloud Cluster Consumers Producers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector Datacenter Cloud Region topic1 cloud.topic1 topic1 datacenter.topic1 Producers Consumers Kafka Connect MirrorSourceConnector MirrorCheckpointConnector MirrorHeartBeatConnector
- 24. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24 Capacity Planning Considerations • Run Kafka Connect in distributed mode • Mainly size based on network throughput • Always consider the impact of a downed worker in capacity planning • Remember to measure impact on source cluster (extra consumer) • Follow general task to CPU core ratio of 8-16 tasks per core • Avoid SMTs and type conversion • Use CPU/Network optimized instances/VMs/machines as disk or high amounts of RAM are not needed • Increase the Heap size to something between 6-12GB
- 25. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25 Best Practices / Observations • Keep partitions per connect task ratio <=10:1 • There can be issues with offset replication when > 10:1 • https://issues.apache.org/jira/browse/KAFKA-12558 • Set consumer.task.id in the connect configuration manually • This allows for client quotas to be set to control replication max throughput for balancing with other producers/consumers • Monitor task status vs Connect REST interface • Monitor JMX metrics to ensure latency and throughput are as expected • You can’t measure consumer lag due to the way MM2 consumes from the source assign() vs subscribe() • Negative lag on replication destinations can occur if replication is pending or behind and offset sync is enabled
- 26. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26 Monitoring Metrics • MirrorSourceConnector • record-count • record-age-ms • record-age-ms-min • record-age-ms-max • record-age-ms-avg • replication-latency-ms • replication-latency-ms-min • replication-latency-ms-max • replication-latency-ms-avg • byte-rate • MirrorCheckpointConnector • checkpoint-latency-ms • checkpoint-latency-ms-min • checkpoint-latency-ms-max • checkpoint-latency-ms-avg
- 27. Thanks! Twitter : @cwgdata LinkedIN: https://www.linkedin.com/in/cliffgilmore © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27