The document discusses the challenges of scaling Kafka MirrorMaker (KMM) pipelines at LinkedIn, which are facing issues with scalability, operability, and fragility due to rapid data growth and complex configurations. It introduces Brooklin MirrorMaker (BMM), which is built on the Brooklin stream ingestion service, designed for higher stability and performance management. The document details BMM's architecture, diagnostics, and performance metrics, highlighting its efficiency in managing data mirroring across multiple data centers.

1. More Data, More Problems:
Scaling Kafka Mirroring Pipelines at LinkedIn
Celia Kung
Software Engineer

2. Agenda
Use Cases & Motivation
Kafka MirrorMaker at LinkedIn
Brooklin MirrorMaker
Future

3. Use Cases & Motivation

4. • Aggregating data from all data centers
• Moving data from offline data stores into
online environments
• Moving data between LinkedIn and external
cloud services
Use Cases

5. Motivation
● Kafka data at LinkedIn continues to grow rapidly

6. Motivation
● Kafka MirrorMaker (KMM) has not scaled well
● KMM is difficult to operate and maintain

7. Kafka MirrorMaker
at LinkedIn

8. Kafka MirrorMaker at LinkedIn
100+ 9pipelines data centers

9. Kafka MirrorMaker at LinkedIn
100+clusters
6K+hosts
1T+messages/day

10. Topology
Datacenter B
aggregate
tracking
tracking
KMM KMM
Datacenter A
aggregate
tracking
tracking
KMM KMM
• Each pipeline:
○ mirrors data from 1 source cluster to 1
destination cluster
○ constitutes its own KMM cluster

11. Topology
Datacenter B
aggregate
tracking
tracking
Datacenter A
aggregate
tracking
tracking
KMM
aggregate
metrics
metrics
aggregate
metrics
metrics
Datacenter C
aggregate
tracking
tracking
aggregate
metrics
metrics
...
KMM KMM
KMM KMM KMM
KMM KMM KMM KMM KMM KMM
KMM KMM KMM KMM KMM KMM
... ... ...

12. KMM does not scale well
● # of KMM clusters = # of data centers x # of Kafka clusters
● More consumer-producer pairs → provision more hardware

13. KMM is difficult to operate
● Static configuration file per KMM cluster
● Changes require deploying to 100+ clusters

14. KMM is fragile
● Poor failure isolation
● Unable to catch up with traffic
● Increased latency

15. Brooklin MirrorMaker

16. Brooklin MirrorMaker
● Optimized for stability and operability
● Built on top of our stream ingestion service, Brooklin
● BMM is in production at LinkedIn today

17. Mirroring pipelines at LinkedIn
100+ 9pipelines data centers

18. Kafka MirrorMaker
100+clusters
6K+hosts
1T+
messages/day

19. Kafka MirrorMaker
100+clusters
6K+hosts
1T+
messages/day

20. Brooklin MirrorMaker
9clusters
<2Khosts
1T+
messages/day

21. Topology
Datacenter B
aggregate
tracking
tracking
BMM
Datacenter A
aggregate
tracking
tracking
BMM
• A single BMM cluster encompasses
multiple pipelines
○ 1 cluster per data center

22. Topology
Datacenter A
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
Datacenter
B
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
Datacenter C
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
...

23. BMM Architecture
BMM

24. BMM is built on Brooklin
DestinationsSources
Data stores
Messaging systems
Microsoft
EventHubs
Data stores
Messaging systems
Microsoft
EventHubs

25. BMM is built on Brooklin
DestinationsSources
Messaging systems
Microsoft
EventHubs
Messaging systems
Microsoft
EventHubs
Data stores Data stores

26. BMM is built on Brooklin

27. BMM is built on Brooklin
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

28. BMM is built on Brooklin
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

29. Dynamic Management
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

30. Creating a pipeline
Brooklin
Engine
Management
Rest API
ZooKeeper
create POST /datastream
name: mm_DC1-tracking_DC2-aggregate-tracking
connectorName: KafkaMirrorMaker
source:
connectionString: kafkassl://DC1-tracking-vip:12345/topicA|topicB
destination:
connectionString: kafkassl://DC2-aggregate-tracking-vip:12345
metadata:
num-streams: 5

31. Creating a pipeline
ZooKeeper
BMM
host
BMM
host
BMM
host
BMM
host
BMM
host

32. Updating a pipeline
Brooklin
Engine
Management
Rest API
ZooKeeper
update PUT /datastream/mm_DC1-tracking_DC2-aggregate-
tracking
name: mm_DC1-tracking_DC2-aggregate-tracking
connectorName: KafkaMirrorMaker
source:
connectionString: kafkassl://DC1-tracking-vip:12345/topicA|topicB|topicC|topicD
destination:
connectionString: kafkassl://DC2-aggregate-tracking-vip:12345
metadata:
num-streams: 10

33. Updating a pipeline
ZooKeeper
BMM
host
BMM
host
BMM
host
BMM
host
BMM
host

34. Dynamic Management
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

35. On-demand Diagnostics
Brooklin
Engine
Diagnostics
Rest API
ZooKeeper
getAllStatus GET /diag?datastream=mm_DC1-tracking_DC2-aggregate-tracking
host1.prod.linkedin.com:
datastream: mm_DC1-tracking_DC2-aggregate-tracking
assignedTopicPartitions: [topicA-0, topicA-3, topicB-0, topicB-2]
autoPausedPartitions: [{topicA-3: {reason: SEND_ERROR, description: failed to produce messages from this
partition}}]
manuallyPausedPartitions: []
host2.prod.linkedin.com:
datastream: mm_DC1-tracking_DC2-aggregate-tracking
assignedTopicPartitions: [topicA-1, topicA-2, topicB-1, topicB-3]
autoPausedPartitions: []
manuallyPausedPartitions: []

36. Error Isolation
● Manually pause and resume mirroring at every level
○ Entire pipeline, topic, topic-partition
● BMM can automatically pause mirroring of partitions
○ Auto-resumes the partitions after configurable duration
● Flow of messages from other partitions continue

37. Processing Loop
while (!shutdown) {
records = consumer.poll();
producer.send(records);
if (timeToCommit) {
producer.flush();
consumer.commit();
}
}

38. Producer flush can be expensive
while (!shutdown) {
records = consumer.poll();
producer.send(records);
if (timeToCommit) {
producer.flush();
consumer.commit();
}
}

39. Long Flush
producer.flush() can take several minutes
200
100
0
flushlatency(seconds)

40. Rebalance Storms
consumers rebalance after max.poll.interval.ms
3
2
1
0
rebalancespersecond

41. Increase max.poll.interval.ms?
● Reduces chances of consumer rebalance
● Risk detecting real failures late

42. Flushless Produce
consumer.poll() → producer.send(records) → producer.flush() → consumer.commit()

43. Flushless Produce
Only commit “safe” acknowledged checkpoints:
consumer.poll() → producer.send(records) → consumer.commit(offsets)

44. Flushless Produce
sp0 consumer producer
checkpoint
manager
o1, o2 o1, o2 o1, o2
o1
o2
Source
Destination
ack(sp0, o2)
dp0
dp1
● Checkpoint manager maintains producer-acknowledged offsets for
each source partition
Source partition sp0
in-flight: [o1]
acked: [o2]
safe checkpoint: --

45. Flushless Produce
sp0 consumer producer
checkpoint
manager
o3, o4 o3, o4 o3, o4
o3
o4
Source
Destination
ack(sp0, o1)
dp0
dp1
● Update safe checkpoint to largest acknowledged offset that is less
than oldest in-flight (if any)
Source partition sp0
in-flight: [o3, o4]
acked: [o1, o2]
safe checkpoint: o2

46. BMM Performance
● Use same consumer/producer configs as KMM
● Single host: 64 GB memory, 24 CPU (12 cores each)
● Metrics:
○ Throughput (output compressed bytes/sec)
○ Memory utilization %
○ CPU utilization %

47. Throughput

48. Memory Utilization

49. CPU Utilization

50. BMM Performance
● BMM is CPU-bound
● Metrics (20 consumer-producer pairs):
○ Throughput: up to 28 MB/s
○ Memory utilization: 70%
○ CPU utilization: 97%

51. Future

52. • 70%+ CPU time spent in decompression & re-
compression
○ GZIPInputStream.read(): ~10%
○ GZIPOutputStream.write(): ~61%
• “Passthrough” mirroring
Performance

53. • Rebalances cause drop in availability
• Kafka low-level consumer
Stability

54. • Auto-scaling: adjust number of consumers
based on throughput needs
Scalability

56. Thank you