Brooklin Mirror Maker (BMM) was created by LinkedIn to address limitations in Kafka Mirror Maker (KMM) for mirroring data between data centers at large scale. BMM is built on LinkedIn's stream ingestion service called Brooklin which provides better operability, fault isolation, and performance optimizations compared to KMM. BMM uses REST APIs for dynamic management and diagnostics of mirroring pipelines. It also includes features like flushless produce and passthrough compression to improve performance. LinkedIn has fully replaced KMM with BMM and plans to open source BMM in the future.

1. Brooklin Mirror Maker
How and Why we moved away from Kafka Mirror Maker
Shun-ping Chiu
Software engineer @ LinkedIn Data Pipelines

2. Agenda
Kafka Mirroring Use Cases
Limitations for Kafka Mirror Maker
Brooklin Mirror Maker
Future Work

3. Mirroring Use Cases

4. ● Aggregating data from all data centers
● Moving data between LinkedIn and external
cloud services
Mirroring
Use Cases

5. Tremendous Kafka Data
● Kafka data at LinkedIn continues to grow rapidly
● We are at 5T messages and 1.4 PB everyday

6. Big Scale to Operate
40+Kafka src clusters
in different DCs
100+pipelines
2Tmessages/day

7. KMM Limitations

8. Kafka Mirror Maker(KMM) Topology
Datacenter B
aggregate
tracking
tracking
KMM KMM
Datacenter A
aggregate
tracking
tracking
KMM KMM
● Each KMM pipeline
○ mirrors data from 1 source cluster to 1
destination cluster
○ constitutes its own KMM cluster

9. Datacenter B
aggregate
tracking
tracking
Datacenter A
aggregate
tracking
tracking
KMM
aggregate
metrics
metrics
aggregate
metrics
metrics
Datacenter C
aggregate
tracking
tracking
aggregate
metrics
metrics
...
KMM KMM
KMM KMM KMM
KMM KMM KMM KMM KMM KMM
KMM KMM KMM KMM KMM KMM
KMM Setup
● # of KMM clusters =
# of data centers x # of Kafka src
clusters
● Need to operate more than 100+ KMM
clusters

10. ● Static configuration file per KMM cluster requires every change
to be deployed
Example - Add a Topic in KMM
● Let’s say we have a pipeline (a KMM cluster) with 100+ hosts
● And 100+ pipelines ?

11. KMM Pain Points
● Hard to operate
○ hard to add new topic
○ difficult to split the pipeline
● One bad partition brings down the pipeline
○ deleted topic
○ ACL issue
● Performance issues
○ Unable to catch up with traffic
○ Increased lag

12. : (
Your Kafka Mirror Maker runs into problems and need to restart. We’re just collecting some error
infos and we will restart for you. (0% completed)

13. Brooklin Mirror Maker

14. Brooklin - Stream Ingestion Service
DestinationsSources
Data stores
Messaging systems
Microsoft
EventHubs
Data stores
Messaging systems
Microsoft
EventHubs

15. BMM is built on Brooklin
DestinationsSources
Data stores
Messaging systems
Microsoft
EventHubs
Data stores
Messaging systems
Microsoft
EventHubs

16. Brooklin Mirror Maker
● Built on top of our stream ingestion service, Brooklin
○ Better operability
○ Fault isolation
○ Performance optimizations
● BMM has fully replaced KMM at LinkedIn today

17. Better Operability

18. KMM vs BMM
Datacenter B
aggregate
tracking
tracking
BMM
Datacenter A
aggregate
tracking
tracking
BMM
Datacenter B
aggregate
tracking
tracking
KMM KMM
Datacenter A
aggregate
tracking
tracking
KMM KMM
● BMM is one cluster per data center

19. BMM Topology
Datacenter A
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
Datacenter B
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
Datacenter C
aggregate
tracking
tracking
BMM
metrics
aggregate
metrics
...
100+KMM clusters
~10BMM clusters

20. Dynamic Management API
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

21. Restful API- Creating a Pipeline
Brooklin
Engine
Management
Rest API
ZooKeeper
create POST /datastream
name: mm_DC1-tracking_DC2-aggregate-tracking
connectorName: KafkaMirrorMaker
source:
connectionString: kafkassl://DC1-tracking-vip:12345/topicA|topicB
destination:
connectionString: kafkassl://DC2-aggregate-tracking-vip:12345
metadata:
taskNums: 5

22. Restful API - Updating a Pipeline
Brooklin
Engine
Management
Rest API
ZooKeeper
update PUT /datastream/mm_DC1-tracking_DC2-aggregate-
tracking
name: mm_DC1-tracking_DC2-aggregate-tracking
connectorName: KafkaMirrorMaker
source:
connectionString: kafkassl://DC1-tracking-vip:12345/topicA|topicB|topicC|topicD
destination:
connectionString: kafkassl://DC2-aggregate-tracking-vip:12345
metadata:
taskNums: 10
^topic*.

23. Pause a Pipeline
● Manually pause and resume mirroring for each pipeline
● BMM can automatically pause mirroring for bad partitions for fault
isolation
○ Flow of messages from healthy partitions continue
○ Auto-resumes the partitions after configurable duration

24. Diagnostic API
Brooklin
Engine
Kafka src
connector
Kafka dest
connector
Management
Rest API
Diagnostics
Rest API
ZooKeeper
Management/
monitoring
portal
SRE/op
dashboards

25. Restful API - On-demand Diagnostics
Brooklin
Engine
Diagnostics
Rest API
ZooKeeper
getAllStatus GET /diag?datastream=mm_DC1-tracking_DC2-aggregate-tracking
host1.prod.linkedin.com:
datastream: mm_DC1-tracking_DC2-aggregate-tracking
assignedTopicPartitions: [topicA-0, topicA-3, topicB-0, topicB-2]
autoPausedPartitions: [{topicA-3: {reason: SEND_ERROR, description: failed to produce messages from this
partition}}]
manuallyPausedPartitions: []
host2.prod.linkedin.com:
datastream: mm_DC1-tracking_DC2-aggregate-tracking
assignedTopicPartitions: [topicA-1, topicA-2, topicB-1, topicB-3]
autoPausedPartitions: []
manuallyPausedPartitions: []

26. Performance Improvements

27. Brooklin Mirroring Pseudocode
while (!shutdown) {
records = consumer.poll();
producer.send(records);
if (timeToCommit) {
producer.flush();
consumer.commit();
}
}
Producer flush can be expensive

28. Flushless Produce
Only commit “safe” acknowledged checkpoints:
consumer.poll() → producer.send(records) → consumer.commit(offsets)
consumer.poll() → producer.send(records) → producer.flush() → consumer.commit()

29. Flushless Produce
sp0 consumer producer
checkpoint
manager
o1, o2 o1, o2 o1, o2
o1
o2
Source
Destination
ack(sp0, o2)
dp0
dp1
● Checkpoint manager maintains producer-acknowledged offsets for
each source partition
Source partition sp0
in-flight: [o1]
acked: [o2]
safe checkpoint: --

30. Flushless Produce
sp0 consumer producer
checkpoint
manager
o3, o4 o3, o4 o3, o4
o3
o4
Source
Destination
ack(sp0, o1)
dp0
dp1
● Update safe checkpoint to largest acknowledged offset that is less
than oldest in-flight (if any)
Source partition sp0
in-flight: [o3, o4]
acked: [o1, o2]
safe checkpoint: o2

31. Manage Performance through Task
● Datastream task
○ Consists of a dedicated kafka consumer and use a share producer pool to
produce the data
○ Performance is controlled by the # of Tasks
○ Tasks are assigned to each host within the BMM cluster
● BMM uses sticky assignment to speeds up the task allocation

32. Sticky Task Assignment
ZooKeeper
BMM
host
BMM
host
BMM
host
BMM
host
Task 1 Task 2 Task 3 Task 4
Task 5 Task 6
ZooKeeper
BMM
host
BMM
host
BMM
host
BMM
host
Task 1 Task 2 Task 3 Task 4
Task 5
BMM
host
Leader
Leader
Task 6

33. BMM Performance Numbers
● Testing environment
○ Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, 12 cores, 64GB RAM
● Performance Metrics with 20 datastream tasks:
○ Throughput: compressed bytes up to 28 MB/s
○ Memory utilization: 70%
○ CPU utilization: ~100%

34. Passthrough Compression
● BMM is CPU bound, 70%+ CPU time is spent in decompression & re-
compression
○ GZIPInputStream.read(): ~10%
○ GZIPOutputStream.write(): ~61%
● “Passthrough” mirroring - skip the decompression & recompression
○ Throughput ~ 100MB/s
○ CPU utilization drops to 50%

35. Future Works

36. ● Better workload distribution - workload
based assignment
● Auto-scaling - adjust number of tasks based
on throughput
Performance
&
Stability

37. Open Source
Expected at EOM, April 2019

38. Questions

39. Thank you