Although LinkedIn data continues to grow rapidly over the years, scaling up to handle the increasing data volume has not been the only challenge in streaming data in near real-time. Supporting the proliferation of new data systems has become yet another huge endeavor for data streaming infrastructure at LinkedIn. Building separate, specialized solutions to move data across heterogeneous systems is not sustainable, as it slows down development and makes the infrastructure unmanageable. This called for a centralized, managed, and extensible solution that can continuously deliver data to nearline applications. We built Brooklin as a managed data streaming service that supports multiple pluggable sources and destinations, which can be data stores or messaging systems. Since 2016, Brooklin has been running in production as a critical piece of LinkedIn's streaming infrastructure, supporting a variety of data movement use cases, such as change data capture (CDC) and data propagation between different systems and environments. We have also leveraged Brooklin for mirroring Kafka data, replacing Kafka MirrorMaker at LinkedIn. In this talk, we will dive deeper into Brooklin's architecture and use cases, as well as our future plans.

1. Dive into Streams with Brooklin
Celia Kung
LinkedIn

2. Outline
Background
Scenarios
Application Use Cases
Architecture
Current and Future

3. Background

4. Nearline Applications
• Require near real-time response
• Thousands of applications at LinkedIn
○ E.g. Live search indices, Notifications

5. Nearline Applications
• Require continuous, low-latency access to data
○ Data could be spread across multiple database systems

6. Heterogeneous Data Systems
Espresso
(LinkedIn’s
document store)
Microsoft
EventHubs

7. Nearline Applications
• Need an easy way to move data to applications
○ App devs should focus on event processing and not on
data access

8. Building the Right Infrastructure
• Build separate, specialized
solutions to stream data from
and to each different system?
○ Slows down development
○ Hard to manage!
Microsoft
EventHubs
...
Streaming
System C
Streaming
System D
Streaming
System A
Streaming
System B
...
Nearline
Applications

9. Need a centralized, managed, and extensible service
to continuously deliver data in near real-time

10. Brooklin

11. Brooklin
• Streams are dynamically
provisioned and individually
configured
• Extensible: Plug-in support for
additional sources/destinations
• Streaming data pipeline service
• Propagates data from many source
types to many destination types
• Multitenant: Can run several
thousand streams simultaneously

12. Kinesis
EventHubs
Kafka
Destinations
Applications
Kinesis
EventHubs
Kafka
Databases
Messaging
Systems
Source
s
Espresso
Pluggable Sources & Destinations

13. Scenarios

14. Change Data Capture
Scenario 1:

15. Capturing Live Updates
1.Member updates her profile to
reflect her recent job change

16. Capturing Live Updates
2.LinkedIn wants to inform her
colleagues of this change

17. Capturing Live Updates
Member DB
Updates
News Feed
Service

18. Capturing Live Updates
Member DB
Updates
News Feed
Service
Search Indices
Service

19. Capturing Live Updates
Member DB
Updates
News Feed
Service
Notifications
Service
Standardization
Service
...
Search Indices
Service

20. Capturing Live Updates
Member DB
Updates
News Feed
Service
Notifications
Service
Standardization
Service
...
Search Indices
Service

21. Capturing Live Updates
Member DB
Updates
News Feed
Service
Notifications
Service
Standardization
Service
...
Search Indices
Service

22. • Isolation: Applications are decoupled
from the sources and don’t compete
for resources with online queries
• Applications can be at different
points in change timelines
• Brooklin can stream database
updates to a change stream
• Data processing applications
consume from change streams
Change Data Capture (CDC)

23. Change Data Capture (CDC)
Messaging System
News Feed
Service
Notifications
Service
Standardization
Service
Search Indices
Service
Member DB
Updates

24. Streaming Bridge
Scenario 2:

25. ● Across…
○ cloud services
○ clusters
○ data centers
Stream Data from X to Y

26. Streaming Bridge
• Data pipe to move data between
different environments
• Enforce policy: Encryption,
Obfuscation, Data formats

27. ● Aggregating data from all data centers into a centralized place
● Moving data between LinkedIn and external cloud services (e.g. Azure)
● Brooklin has replaced Kafka MirrorMaker (KMM) at LinkedIn
○ Issues with KMM: didn’t scale well, difficult to operate and manage, poor
failure isolation
Mirroring Kafka Data

28. Use Brooklin to Mirror Kafka Data
DestinationsSources
Messaging systems
Microsoft
EventHubs
Messaging systems
Microsoft
EventHubs
Databases Databases

29. Kafka
MirrorMaker
Topology
Datacenter B
aggregate
tracking
tracking
Datacenter A
aggregate
tracking
tracking
KMM
aggregate
metrics
metrics
aggregate
metrics
metrics
Datacenter C
aggregate
tracking
tracking
aggregate
metrics
metrics
...
KMM KMM
KMM KMM KMM
KMM KMM KMM KMM KMM KMM
KMM KMM KMM KMM KMM KMM
... ... ...

30. Brooklin Kafka Mirroring Topology
Datacenter A
aggregate
tracking
tracking
Brooklin
metrics
aggregate
metrics
Datacenter B
aggregate
tracking
tracking metrics
aggregate
metrics
Datacenter C
aggregate
tracking
tracking metrics
aggregate
metrics
...Brooklin Brooklin

31. Brooklin Kafka Mirroring
● Optimized for stability and operability
● Manually pause and resume mirroring at every level
○ Entire pipeline, topic, topic-partition
● Can auto-pause partitions facing mirroring issues
○ Auto-resumes the partitions after a configurable duration
● Flow of messages from other partitions is unaffected

32. Application Use Cases

33. Security
Cache
Application Use Cases

34. Security
Cache Search Indices
Application Use Cases

35. Security
Cache Search Indices ETL or Data
warehouse
Application Use Cases

36. Security
Cache Search Indices ETL or Data
warehouse
Materialized Views or
Replication
Application Use Cases

37. Security
Cache Search Indices ETL or Data
warehouse
Materialized Views or
Replication
Repartitioning
Application Use Cases

38. Adjunct Data
Application Use Cases

39. BridgeAdjunct Data
Application Use Cases

40. Bridge Serde, Encryption,
Policy
Adjunct Data
Application Use Cases

41. Bridge Serde, Encryption,
Policy
Standardization,
Notifications …
Adjunct Data
Application Use Cases

42. Architecture

43. Stream updates made to Member Profile
Example:

44. Capturing Live Updates
Member DB
Updates
News Feed
Service

45. ● Scenario: Stream Espresso Member Profile updates into Kafka
○ Source Database: Espresso (Member DB, Profile table)
○ Destination: Kafka
○ Application: News Feed service
Example

46. • Describes the data pipeline
• Mapping between source and
destination
• Holds the configuration for the pipeline
Datastream
Name: MemberProfileChangeStream
Source: MemberDB/ProfileTable
Type: Espresso
Partitions: 8
Destination: ProfileTopic
Type: Kafka
Partitions: 8
Metadata:
Application: News Feed service
Owner: newsfeed@linkedin.com

47. 1. Client makes REST call to create datastream
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
create POST /datastream
News Feed
service

48. 2. Create request goes to any Brooklin instance
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

49. 3. Datastream is written to ZooKeeper
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

50. 4. Leader coordinator is notified of new datastream
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

51. 5. Leader coordinator calculates work distribution
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

52. 6. Leader coordinator writes the assignments to ZK
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

53. 7. ZooKeeper is used to communicate the assignments
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

54. 8. Coordinators hand task assignments to consumers
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

55. 9. Consumers start streaming data from the source
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

56. 10. Consumers propagate data to producers
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

57. 11. Producers write data to the destination
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

58. 12. App consumes from Kafka
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service

59. 13. Destinations can be shared by apps
Brooklin Client
Load Balancer
ZooKeeper
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
(Leader)
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Brooklin Instance
Datastream
Management
Service (DMS) Espresso Consumer
Coordinator
Kafka Producer
Member DB
News Feed
service
News Feed
service
News Feed
service

60. Brooklin Architecture
ZooKeeper
Brooklin Instance
Brooklin Instance
Coordinator
Datastream
Management
Service
(DMS)
ConsumerA
ConsumerB
ProducerX
ProducerY
ProducerZ
Brooklin Instance
Datastream
Management
Service
(DMS)
ConsumerA
ProducerX
ConsumerB
ProducerY
ProducerZ
Coordinator
(Leader)
Brooklin Instance
Coordinator
Datastream
Management
Service
(DMS)
ConsumerA
ConsumerB
ProducerX
ProducerY
ProducerZ
Brooklin Instance

61. Current & Future

62. Current
• Consumers:
○ Espresso
○ Oracle
○ Kafka
○ EventHubs
○ Kinesis
• Producers:
○ Kafka
○ EventHubs
• APIs are standardized to support additional
sources and destinations
• Multitenant: Can power thousands of
datastreams across several source and
destination types
• Guarantees: At-least-once delivery, order is
maintained at partition level
• Kafka mirroring improvements: finer control
of pipelines (pause/auto-pause partitions),
improved latency with flushless-produce
mode
Sources & Destinations Features

63. 38B
messages/day
2K+
datastreams
1K+
unique sources
200+
applications
Brooklin in Production
Brooklin streams with Espresso or Oracle as the source

64. 3T+
messages/day
200+
datastreams
10K+
topics
Brooklin in Production
Brooklin streams mirroring Kafka data

65. Brooklin is now open-source!

66. Future
• Consumers:
○ MySQL
○ Cosmos DB
○ Azure SQL
• Producers:
○ Azure Blob storage
○ Kinesis
○ Cosmos DB
○ Azure SQL
○ Couchbase
• Brooklin auto-scaling
• Passthrough compression
• Read optimizations: Read once, Write multiple
Sources & Destinations Optimizations

67. Thank
you

68. Questions?
: ckung@linkedin.com
: linkedin.com/in/celiakkung/
: gitter.im/linkedin/brooklin
: github.com/linkedin/brooklin