Kafka makes it possible to ingest and process large amounts of data concurrently by decoupling your data streams. This successful approach introduces some challenges when monitoring Kafka pipelines - to understand what’s happening, we need to monitor every single component and how they interact with each other. In this talk, we will take a close look at Kafka’s architecture as well as the key infrastructure, JVM, and system metrics you should monitor for each of its components. Then, we will walk through how to diagnose common Kafka performance anomalies through observing patterns in the metrics for the various components. Finally, we will walk through setting up an open source observability pipeline with OpenTelemetry to enable you to collect and process Kafka metrics at scale.

1. © 2022 New Relic, Inc. All rights reserved.
Key Metrics To Uncover the
Root Cause of Kafka
Performance Anomalies
Anton Rodriguez + Daniel Kim

2. © 2022 New Relic, Inc. All rights reserved.
Event Driven Architecture

3. © 2022 New Relic, Inc. All rights reserved.
How do you debug event driven architecture?

4. © 2022 New Relic, Inc. All rights reserved.
Distributed Tracing

5. © 2022 New Relic, Inc. All rights reserved.
Distributed Tracing

6. © 2022 New Relic, Inc. All rights reserved.
Metrics

7. © 2022 New Relic, Inc. All rights reserved.
Logs for errors

8. © 2022 New Relic, Inc. All rights reserved.
Instrumenting your application
Producer
Producer
Broker
Broker
Broker
consumer
Consumer
Zookeeper

9. © 2022 New Relic, Inc. All rights reserved.
Collect telemetry data
Process telemetry data
Send telemetry data
@learnwdaniel

10. © 2022 New Relic, Inc. All rights reserved.
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specifica
tion/metrics/semantic_conventions/instrumentation/kafka.md

11. Language/Framework
Specific SDKs
Capture Data
Process
(Decorate, Batch,
Encrypt)
Send to Backend
Receiver Processor Exporter
OpenTelemetry Collector
@learnwdaniel

12. © 2022 New Relic, Inc. All rights reserved.
OpenTelemetry Java Agent
https://github.com/open-telemetry/opentelemetry-java-instrumentation

13. © 2022 New Relic, Inc. All rights reserved.
JMX Receiver

14. © 2022 New Relic, Inc. All rights reserved.
KafkaMetricsReceiver

15. © 2022 New Relic, Inc. All rights reserved.
Prometheus Metrics

16. © 2022 New Relic, Inc. All rights reserved.
The Problem: There are way too many metrics

17. © 2022 New Relic, Inc. All rights reserved.
Increasing the ROI of your o11y data

18. - 1000 Metrics
- 800 Traces
- 5500 Logs

19. - 2000 Metrics
- 1600 Traces
- 11000 Logs

20. - 100000 Metrics
- 160000 Traces
- 1100000 Logs

21. Processing, Saving,
Decorating Data
== Money

22. Actionable Insights
Overall Data being Produced
ROI of Observability

23. Language/Framework
Specific SDKs
Capture Data
Process
(Decorate, Batch,
Encrypt)
Send to Backend
Receiver Processor Exporter
OpenTelemetry Collector
@learnwdaniel

24. export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.3
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/READ
ME.md#sampler
Configure sampling for java agent

25. Language/Framework
Specific SDKs
Capture Data
Process
(Decorate, Batch,
Encrypt)
Send to Backend
Receiver Processor Exporter
OpenTelemetry Collector
@learnwdaniel

26. tailsamplingprocessor
Collect
Process
Export
@learnwdaniel

27. Collecting Infrastructure Metrics
Collect
Process
Export
Collector
ECS, Cloudwatch Container
Insights, X-RAY, Firehose

28. k8sattributesprocessor
Collect
Process
Export
Daniel’s Pod A
OpenTelemetry Collector
Metrics Traces Logs Correlated Telemetry
k8s.pod.name
K8s.deployment.name
k8s.cluster.name
Add Metadata
@learnwdaniel

29. © 2022 New Relic, Inc. All rights reserved.
Tuning Kafka Configurations

30. © 2022 New Relic, Inc. All rights reserved.
Measuring Consumer Lag
Producer Broker consumer

31. © 2022 New Relic, Inc. All rights reserved.
Calculating Consumer Lag

32. © 2022 New Relic, Inc. All rights reserved.

33. © 2022 New Relic, Inc. All rights reserved.
Diagram about increasing the # of consumers

34. © 2022 New Relic, Inc. All rights reserved.
Kafka has a lot of configs you can tweak

35. © 2022 New Relic, Inc. All rights reserved.

36. © 2022 New Relic, Inc. All rights reserved.
Kafka Producers
Producer
Daniel’s Topic

37. © 2022 New Relic, Inc. All rights reserved.
Batching

38. ©2008–22 New Relic, Inc. All rights reserved
Producer
Batch 1
5. Fine Tune Kafka Config (Batching/Compressing Messages)
Message 1
Message 2
Message 3
Message 4
Message 5
Message 6
📦 batch.size = 6
Compression

39. ©2008–22 New Relic, Inc. All rights reserved
Producer
Batch 2
5. Fine Tune Kafka Config (Batching/Compressing Messages)
Message 1
Message 2
Message 3
📦 batch.size = 6
⌛linger.ms = 20
📦 batch.size = 6
Compression

40. © 2022 New Relic, Inc. All rights reserved.
What do we measure?

41. © 2022 New Relic, Inc. All rights reserved.
Batch Size
Producer
batch.size = 10kb
batch.size = 16kb

42. © 2022 New Relic, Inc. All rights reserved.
Request Rate
Producer
Average number of requests sent per second

43. © 2022 New Relic, Inc. All rights reserved.
Request Latency Average
Producer
Daniel’s Broker
Producer
Daniel’s Broker
I got it!
sending
message

44. © 2022 New Relic, Inc. All rights reserved.
Kafka Consumers
Consumer
Daniel’s Topic
Consumer

45. © 2022 New Relic, Inc. All rights reserved.
How do we increase throughput?
Consumer
Daniel’s Topic
(How do we get the consumer to read more messages)

46. © 2022 New Relic, Inc. All rights reserved.
Increasing the # of requests
fetch.max.wait.ms - Sets a maximum threshold for time-based batching.
fetch.min.bytes - Sets a minimum threshold for size-based batching.
Consumer
Producer

47. © 2022 New Relic, Inc. All rights reserved.
How do we decrease latency?
Consumer
Daniel’s Topic
How do we get the messages to be consumed faster?

48. © 2022 New Relic, Inc. All rights reserved.
increasing maximum batch sizes
fetch.max.bytes
max.partition.fetch.bytes
Consumer
Consumer

49. © 2022 New Relic, Inc. All rights reserved.
What do we measure?

50. © 2022 New Relic, Inc. All rights reserved.
Records Lag
Producer
Consumer

51. © 2022 New Relic, Inc. All rights reserved.
Record Consumed Rate
Consumer

52. © 2022 New Relic, Inc. All rights reserved.
Fetch rate
Consumer