Fan-in Flames: Scaling Kafka to Millions of Producers With Ryanne Dolan | Current 2022
•
0 likes•873 views
Fan-in Flames: Scaling Kafka to Millions of Producers With Ryanne Dolan | Current 2022 At supermassive scale, a perennial problem with Kafka is ""high fan-in"" -- a large number of producers sending records to a small number of brokers. Even a relatively modest amount of data can overwhelm a broker when there are hundreds of thousands of concurrent producer requests. This talk discusses a few real-world applications where high fan-in becomes a problem, and presents a few strategies for dealing with it. These include: fronting Kafka with an ingestion layer; separating brokers into read-only and write-only subsets; implementing specialized partitioning strategies; and scaling across clusters with ""smart clients"".
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Fan-in Flames: Scaling Kafka to Millions of Producers With Ryanne Dolan | Current 2022
Similar to Fan-in Flames: Scaling Kafka to Millions of Producers With Ryanne Dolan | Current 2022 (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Fan-in Flames: Scaling Kafka to Millions of Producers With Ryanne Dolan | Current 2022
- 1. Fan-in Flames Scaling Kafka to Millions of Producers Ryanne Dolan Sr Staff SWE @LinkedIn
- 2. What makes streaming data BIG? 1. Throughput (BPS) 2. Records (RPS) 3. Requests (QPS) 4. ?
- 3. What makes streaming data BIG? 1. Throughput (BPS) 2. Records (RPS) 3. Requests (QPS) 4. Fan-in 5. Fan-out
- 4. "Fan-in" Large number of producers per broker Broker 1 Broker 2 Broker 3
- 5. "Fan-out" Large number of consumer groups per broker Broker 1 Broker 2 Broker 3
- 6. Fan-in and fan-out can overwhelm brokers Even at low throughput!
- 7. Fan-in vs Fan-out Fan-in is more likely to be an issue Fan-in scales with… • # application instances • # containers • # hosts • # edge devices Fan-out scales with… • # applications
- 8. The Consumer Problem We're all listening!
- 9. Fan-out is a throughput multiplier IO ∝ Fin + Fin × Fout 1 MBps per producer 100 producers 10 consumers à 1.1 GBps
- 10. Fan-out can exhaust broker resources Disk IO Network IO Request handling
- 11. Replication … is a fan-out divider Broker 1 Broker 2 Broker 3
- 12. Replication in the Kafka ecosystem Two typical replication strategies Internal Replication • Broker-Broker • Built-in • Synchronous • Requires fetch-from-follower (KIP-392) • Client-side cfg changes • Custom ReplicaSelector and/or rack-awareness • OOTB will not help Mirroring • Cluster-Cluster • External to broker (e.g. MM) • More infra (e.g. Connect) • Asynchronous
- 13. Use internal replication for durability. Use mirroring to reduce fan-out.
- 14. Data pipelines Reduce fan-out by processing records all in one place
- 15. Data pipelines can reduce fan-out One big topic à high fan-out One big topic
- 16. Data pipelines can reduce fan-out Pipelines à smaller topics à less fan-out One big topic Filtered topic Projected topic
- 17. Data pipelines for aggregation Small topics à aggregated topics à less fan-out many small topics aggregated topic 1 many small topics many small topics many small topics aggregated topic 2
- 18. Use case: Logging infrastructure Two requirements Application-specific logs All logs Host-specific logs
- 19. Use case: Logging infrastructure Application, container, and host log events Two reasonable approaches One big topic • All applications send log events to one big topic, which is sent to the cloud • Data pipeline routes records based on application ID, container ID, host ID • Derived topics for each application, container, host • Consumers can process a single application, container, host Many small topics • Each application sends to its own topic • Data pipeline aggregates across containers and hosts, and sends to cloud
- 20. More topics More records More data Less fan-out
- 21. Who needs millions of producers?
- 22. The Producer Problem Applications, Microservices Hosts, Containers Logs, Metrics, Tracing
- 23. This all adds multiplies up!
- 24. Fan-in counteracts batching For a given workload, more producers à less batching à more requests à More stress on brokers!
- 25. High fan-in is detrimental to performance. At extreme scale, fan-in can be more important than BPS, RPS, and QPS!
- 26. Contrived Experiments Simulating massive workloads with constrained brokers 800 MHz 8 cores 40% quota practically 1GHz of power! .4 core per broker
- 27. The Application N Producers and M Consumer per Workload Broker 1 Broker 2
- 28. The Application -- Mirrored N+1 Producers and M+1 Consumer per Workload Broker 1 Broker 2
- 29. The Workload A modest amount of data 10K RPS 10K QPS 300K BPS constant constant constant
- 30. Latency metrics As used here. End-to-End Latency Delay between when a record is created (before send) and when it is ultimately processed by a consumer (after fetch). Send Latency Delay between when a record is created (before send) and when it is written to disk on the last broker (before ACK). Fetch Latency Delay between when a record is written to disk on the last broker (before ACK) and when it is ultimately processed by a consumer (after fetch).
- 31. End-to-End Latency (ms avg) 1000 Producers 1 Consumer 1 Workload 359 555 370 459 419 422 393 396 324 348 309 427 343 317 350 371 343 372 388 351 1 2 3 4 5 6 7 8 9 10 Mirrored Direct
- 32. Send Latency (ms avg) 1000 Producers 1 Consumer 1 Workload 350 552 368 457 417 420 391 394 322 345 145 205 165 166 180 187 175 172 188 185 1 2 3 4 5 6 7 8 9 10 Mirrored Direct
- 33. Fetch Latency (ms avg) 1000 Producers 1 Consumer 1 Workload 5 2 4 4 4 3 6 6 9 10 25 19 27 8 7 15 8 7 26 14 1 2 3 4 5 6 7 8 9 10 Mirrored Direct
- 34. 13% Worse End-to-End Latency Mirroring made latency 1.13x worse, when we'd expect 2x worse!
- 35. End-to-End Latency (ms avg) 1000 Producers 10 Consumers 10 Workloads 6899 8393 Mirrored Direct
- 36. Send Latency (ms avg) 1000 Producers 10 Consumers 10 Workloads 6783 7315 Mirrored Direct
- 37. Fetch Latency (ms avg) 1000 Producers 10 Consumers 10 Workloads Same amount of data! Same number of consumers! 5 1471 Mirrored Direct
- 38. 23% Better End-to-End Latency Additional hop, but less time.
- 39. Producers and consumer compete for resources. Fan-in can quickly overwhelm a broker, even at low throughput. Fan-out magnifies this effect.
- 40. LinkedIn Model Separate local and aggregation clusters Colo-1 Colo-2 Local1 Local2 Agg1 Agg2 Brooklin Brooklin Brooklin Brooklin
- 41. Ingestion layer Producers sharded across multiple ingestion clusters Ingest 1 App 1 Ingest 2 Ingest 3 App 2
- 42. Kafka@Edge Shard workload based on geographical location 1 2 3 4
- 43. Can't we just add more brokers? To a point! Something may be wrong if the amount of data is small relative to the size of the cluster.
- 44. Read vs write sets Separate producers from consumers Broker 1 Broker 2 Broker 3 Write-only broker Read-only brokers
- 45. Partitioning The problem with round-robin sticky partitioning Broker 1 Broker 2 Broker 3
- 46. Smart clients 1. Round-robin among a subset of partitions à only a fraction of clients can send to a given broker 2. Measure latency and avoid slow partitions
- 47. Dealing with Fan-in Some easy strategies Add more brokers Fetch-from-follower can be used to split brokers into read vs write sets Shard workload Divide workload into non- overlapping groups Combine Producers Avoid having many producer clients within the same application
- 48. Dealing with Fan-in Some easy strategies Mirroring Separate producers from consumers Batching Slow down to get faster! Smart partitioning Avoid having multiple producers write to the same partition at the same time