"Processing a lot of data with Kafka means knowing how and when to scale horizontally and vertically. When you’ve exhausted the boundaries of scaling inside a single cluster, replication becomes critical but sometimes standard replication is not enough. New Relic once earned the dubious title of “World’s Largest Kafka Cluster”, and in our journey to break this cluster into dozens of smaller clusters, we needed to route events between clusters and topics based on headers. At the time, this meant we had to do it ourselves. Starting out, our goal was fan out (one-to-many) replication. Since then our needs have expanded to include many-to-one and many-to-many replication. In this talk we'll discuss what bottlenecks we have hit as we scaled out, and what measures we took to remove them, such as: - Replicating data based on Kafka Headers - Connecting to many source and destination Kafka clusters - Managing the replication of Kafka topics of varying traffic - The use of an intermediary Kafka cluster At the end of this talk you will understand how we have scaled replication and routing to support New Relic's ever growing data ingestion, and all the mitigations it took to get us there."

1. Go Big or Go Home
Approaching Kafka Replication at Scale
Julia Holgado

2. Agenda
● New Relic’s cloud migration, focusing on replication between Kafka
clusters
○ Discovering the need for one to many routing
○ What we did to fulfill that need
■ Problems + mitigations
○ Discovering the need for other types of routing
○ Extending out one to many solution to fulfill many to many routing
■ Problems
○ Ongoing improvements

3. 150+ PB
Per month
3 B
Data points per
minute
160+ B
Web requests
per day

4. Starting Architecture

5. Introducing Cells
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

6. Introducing Cells
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Wayfinder
account, data type
cell domain name
Datacenter

7. One To Many Example
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

8. One To Many Example
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

9. One To Many Example
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

10. One To Many Example
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

11. One To Many Example
HTTP
Endpoints
Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

12. One to Many Routing: Requirements
● Isolate the partial or total failure of a destination cell from impacting
other cells
● React to changes in routing without a deploy
● Route based on Kafka headers
● Supports multiple routing strategies

13. “Smart Mirroring”
HTTP
Endpoints Pipeline
Services
Ingest
Tier
Insert
Workers
APIs & UIs
Kafka New Relic
DB
Datacenter

14. Kynapses

15. The Router

16. The Mirror

17. Knowing What to Route

18. Knowing Where to Route

19. One to Many Routing
Router Mirror
dest.topic_name
dest.topic_name
…
● <10 Destination clusters
● <10 Topics

20. Problem: Partition Explosion
Router Mirror
dest.topic_name
dest.topic_name
…

21. Problem: Partition Explosion
Router Mirror
destA.topic_name-0
destB.topic_name-N
…
destB.topic_name-0
destA.topic_name-N
…
destC.topic_name-0
destC.topic_name-N
…

22. Problem: Topics of Varying Size and Traffic
Router Mirror
destA.topic_name-0…N
dest#.topic_name-0…N
destB.high_traffic_topic-0
…2N
destC.high_traffic_topic-0
…2N

23. One to Many Problems: Summary
● Partition explosion
○ More strain on kafka brokers
○ Rebalance storms when managing Mirror instances
● Handling topics of varying size and throughput
○ Cannot steer more resources towards a certain topic

24. Mitigation:“Sharding”
Router Mirror
destA.topic_name-0…N
dest#.topic_name-0…N
destB.high_traffic_topic-0
…2N
destC.high_traffic_topic-0
…2N
Router-Shard Mirror-Shard

25. Sharding Outcomes
● Designate a set of Kynapses instances for particular topics
○ Lessen rebalances on restarts and deploys
○ Scale each shard independently
● Downsides
○ Shards are organized manually

26. Introducing More Cells
Router Mirror
dest.topic_name
dest.topic_name
…
Datacenter

27. Evolving Cells
Kafka
Client
Services

28. Cell Types
“Ingest” cell type “Aggregation” cell type

29. Many to Many Example
Aggregation Cells
Ingest Cells
HTTP
Endpoints

30. Many to Many Example: Aggregation
Aggregation Cells
Ingest Cells
HTTP
Endpoints

31. Many to Many Example: Aggregation
Aggregation Cells
Ingest Cells
HTTP
Endpoints

32. One to Many Example: Sub-account Aggregation

33. One to Many Example: Sub-account Aggregation
Ingest Cells
HTTP
Endpoints

34. One to Many Example: Sub-account Aggregation
Ingest Cells
HTTP
Endpoints

35. Connections
Cell
Type A
Cell
Type B

36. Introducing Routing Cells
Routing cell
Cell
Type A
Cell
Type B

37. Cells + Routing Cells
● 20+ Source clusters
● 20+ Destination Clusters
● 20+ topics
● 3 GB/s through routing cells

38. Problem: Loops

39. Problem: Cost of Intermediary Cluster
Router Mirror
Router
Shard-N
Mirror
Shard-N
Router
Shard-1
Mirror
Shard-1

40. Problem: Central Point
Routing cell
Cell
Type A
Cell
Type B

41. Problem: Central Point
Routing cell
Cell
Type A
Cell
Type B

42. Improvements: Use of an Intermediary Kafka Cluster
● What are other approaches can we take that let us
○ Persist data
○ Backpressure as needed

43. Improvements: WorkAssignment
● Problem: Kynapses cannot distribute itself among the topics it handles

44. Improvements: WorkAssignment
● Goal: Let Kynapses instances assign themselves to a topic, in order to
○ Improve resource distribution; be able to steer more instances to large topics
○ Reduce the amount of consumers each instance spins up
○ Remove operational toil of shards

45. WorkAssignment Algorithms: Simple Modulus
cell_A-topic_name cell_B-topic_name cell_A-other_topic cell_B-other_topic
router-0 hash(router-0) %
hash(cell_A-topic_name)
hash(router-0) %
hash(cell_B-topic_name)
hash(router-0) %
hash(cell_A-other_topic)
hash(router-0) %
hash(cell_B-other_topic)
router-1 hash(router-1) %
hash(cell_A-topic_name)
hash(router-1) %
hash(cell_B-topic_name)
hash(router-1) %
hash(cell_A-other_topic)
hash(router-1) %
hash(cell_B-other_topic)
router-2 hash(router-2) %
hash(cell_A-topic_name)
hash(router-2) %
hash(cell_B-topic_name)
hash(router-2) %
hash(cell_A-other_topic)
hash(router-2) %
hash(cell_B-other_topic)
… … … … …
router-N hash(router-N) %
hash(cell_A-topic_name)
hash(router-N) %
hash(cell_B-topic_name)
hash(router-N) %
hash(cell_A-other_topic)
hash(router-N) %
hash(cell_B-other_topic)

46. WorkAssignment Algorithms: Consistent Hash
router-0 router-1 router-2 router-3 … router-N
hash(cell_name-topic_name)

47. WorkAssignment Algorithms: Weighted Rendezvous
weight: 3
weight: 3
weight: 1
weight: 1

48. WorkAssignment Algorithms: Weighted Rendezvous
weight: 3
weight: 3
weight: 1
weight: 1

49. WorkAssignment Algorithms: Random Latch

50. WorkAssignment Algorithms: Random Latch

51. WorkAssignment Algorithms: Random Latch

52. WorkAssignment: Coordinator-Based Approach
Change in worker
set or task set
Elect a
coordinator

53. WorkAssignment: Coordinator-Based Approach
???

54. Summary
● How New Relic has handled replicating data between many Kafka clusters
○ Redundancy, failure isolation, and our pipeline architecture led us to develop our own
tool
○ We ran into several difficulties with our chosen implementation, particularly
■ Using an intermediate kafka cluster to help separate responsibilities of consuming
from source cluster and producing to destination cluster can result in a large,
difficult to manage cluster
■ Managing the routing of many topics requires more efficient use of service
resources
■ Highlighting the weaknesses of NR’s cellular architecture
○ Our plans moving forward
Julia Holgado
jholgado@newrelic.com