Unleashing your Kafka Streams Application Metrics!
•
0 likes•199 views
"Kafka Streams has a rich set of metrics for monitoring application health. Through these metrics, you can uncover performance issues, resource allocation concerns, and improve the performance of your application through deployment and configuration changes. Providing dashboards around all of these metrics can be rather challenging. In addition, the vast amount of metrics is extensive. Which metrics are important depends on the type of application you’re building. Let's uncover what you should be monitoring, why you should be monitoring it, and leave you with properly monitored Kafka Streams applications. Not only will you gain an understanding of task-id, sub-topology, and partition-id, but you will also see how to visualize that topology in a dashboard. Explore the new metrics added to Kafka Streams, since 3.0 was released, and go in-depth with the awesome end-to-end latency metrics. Finally, learn how to use these metrics to determine the number of instances an application needs when being deployed. Unleash your Kafka Stream Application metrics making it easier to run your applications effectively."
Recommended
More Related Content
Similar to Unleashing your Kafka Streams Application Metrics!
Similar to Unleashing your Kafka Streams Application Metrics! (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Unleashing your Kafka Streams Application Metrics!
- 1. Unleashing your Kafka Streams Application Metrics Neil Buesing Kinetic Edge Current 2023
- 3. So Many Metrics • Kafka Broker • Network • Machine / Pod • JVM • Kafka Client • Producer • Consumer • Admin • Kafka Streams
- 4. Stress Free Monitoring Katniss (best pair programmer)
- 5. Kafka Streams - GitHub Inquiry (path:**/pom.xml OR path://**/build.gradle) AND kafka-streams AND NOT (org:confluentinc OR org:apache) September 2023 = 8.9K (23K kafka-clients)
- 7. processor task thread JVM task allocation to threads - streams-consumer-group topic partition t1 p0 t1 p1 t2 p1 t3 p1 t2 p0 t3 p0 • Instance (JVM) • Client (Topology) • Thread (Worker) • Task • sub-topology • partition • Processor • State Stores Kafka Streams Architecture state store
- 8. Con fi guration • metrics.recording.level — INFO, DEBUG, TRACE • metrics.sample.window.ms — 30_000 • metrics.num.samples — 2
- 9. Exporting • JMX Reporter + Prometheus Agent • Dashboards Demonstrated with this configuration • No renaming of metrics in extraction rules • Micrometer • renames some metrics — makes dashboards incompatible. • does not export info metrics
- 10. Exporting • JMX Reporter + Prometheus Agent • Dashboards Demonstrated with this configuration • No renaming of metrics in extraction rules • Micrometer • renames some metrics — makes dashboards incompatible. • does not export info metrics private String meterName(MetricName metricName) { String name = METRIC_NAME_PREFIX + metricName.group() + "." + metricName.name(); return name.replaceAll("-metrics", "").replaceAll("-", "."); } currentMetrics.forEach((name, metric) -> { // Filter out non-numeric values // Filter out metrics from groups that include metadata if (!(metric.metricValue() instanceof Number) || APP_INFO.equals(name.group()) || METRICS_COUNT.equals(name.group())) return; 1.11.2
- 12. state store Client Metrics • version • state • num threads • application_id • job (application) • instance (JVM) • client (topology) • thread (worker) processor task thread JVM
- 13. state store Thread Metrics • operation • commit • poll • process • punctuate • ratio • metric • total • rate • latency-{avg|max} • job (application) • instance (JVM) • client (topology) • thread (worker) processor task thread JVM computing rate within grafana from a total metric rate(metric_total{}[$__rate_interval])
- 14. state store Task Metrics processor task thread JVM • operation • commit • process • punctuate • metric • total • rate • job (application) • instance (JVM) • client (topology) • thread (worker) • task • sub-topology • partition KAFKA-9441 (2.6)
- 15. state store Processor Metrics processor task thread JVM • operation • process • total/rate • suppression-emit • total/rate • record-e2e-latency • min/max/avg • job (application) • instance (JVM) • client (topology) • thread (worker) • task • sub-topology • partition • process node
- 16. state store Topic Metrics • operation • consumed • produced • metric • records total • bytes total • job (application) • instance (JVM) • client (topology) • thread (worker) • task • process node • topic processor task thread JVM consumed metrics produced metrics source & sink processors only topic=“topic_name” attribute included
- 17. state store Record Cache Metrics processor task thread JVM • metric • hit-ratio-min • hit-ratio-max • hit-ratio-avg • job (application) • instance (JVM) • thread (worker) • task • subtopology • partition • store name
- 18. state store State Store Metrics processor task thread JVM • operation • all • delete • fetch • get • pre fi x-scan * • put, put-all, put-if-absent • range • remove • restore • fl ush • computed • record-e2e ** • metric • rate • latency-avg (ns) • latency-max (ns) • job (application) • instance (JVM) • thread (worker) • task • subtopology • partition • store type • store name • other • suppression-bu ff er-size • suppression-bu ff er-count • metric • avg • max * KIP 614 / KAFKA-10648 - Pre fi x Scan support to State Stores - 2.8 ** KIP 613 / KAFKA-10054 - e2e latency metrics 2.7 TRACE
- 19. RocksDB Metrics • ~40 Metrics • Property Metrics • INFO • Statistical Metrics • DEBUG* • KIPs • 471 (2.4) • 607 (2.7) • KAFKA-8941 (3.2) ** background-errors block-cache-capacity block-cache-data-hit-ratio block-cache-filter-hit-ratio block-cache-index-hit-ratio block-cache-pinned-usage block-cache-usage bytes-read bytes-read-compaction bytes-written bytes-written-compaction compaction-pending compaction-time ** cur-size-active-mem-table cur-size-all-mem-tables estimate-num-keys estimate-pending-compaction-bytes estimate-table-readers-mem live-sst-files-size memtable-bytes-flushed mem-table-flush-pending memtable-flush-time ** memtable-hit-ratio number-file-errors number-open-files num-deletes-active-mem-table num-deletes-imm-mem-tables num-entries-active-mem-table num-entries-imm-mem-tables num-immutable-mem-table num-live-versions num-running-compactions num-running-flushes size-all-mem-tables total-sst-files-size write-stall-duration * emit as 0 in INFO https://docs.con fl uent.io/platform/current/streams/monitoring.html#rocksdb-metrics https://github.com/facebook/rocksdb/blob/main/include/rocksdb/db.h - “struct Properties”
- 21. Application rekey (user) attach user price aggregate orders fl atmap (line- item) rekey (sku) pickup attach store rekey (order) pickup peek users ktable stores global ktable products ktable fi lter RocksDB sub-topology indicator
- 22. Analytic Applications (Tumbling Window) line-item product-stats aggregate toStream mapValue windowBy TimeWindows .ofSize(30sec) RocksDB
- 23. line-item product-stats aggregate toStream mapValue windowBy TimeWindows .ofSize(30sec) .advanceBy(15sec) RocksDB Analytic Applications (Hopping Window)
- 24. line-item product-stats aggregate toStream mapValue windowBy SlidingWindows .ofSize(30sec) suppress RocksDB InMemory Analytic Applications (Sliding Window)
- 25. line-item product-stats aggregate toStream mapValue windowBy SessionWindows .ofInactivityGap(30sec) RocksDB Analytic Applications (Session Window)
- 26. line-item product-stats aggregate toStream processor mapValue punctuate schedule(15s, wall-clock) Analytic Applications (No Window)
- 27. Dashboards
- 29. Sub-topology Listing name your processors
- 30. Sub-topology Listing prometheus — label_replace label_replace( kafka_streams_stream_processor_node_metrics_process_total{} * on (instance) group_left(application_id) kafka_streams_info{application_id=~"${application}"}, "subtopology", "$1", "task_id", "(.*)_.*" ) prometheus — on / group_left name your processors
- 31. Task Assignment
- 32. Task Assignment
- 37. Threads
- 39. End to End Latency
- 40. State Stores
- 41. Demo
- 42. Recommendations
- 43. • Thread ⬌ Task Assignment • Provide Topology and Task Assignment Dashboards • End-To-End Latency Metrics • Consumed and Produced Metrics • State Store Metrics • DSL could change implementation • ∴ always check metrics after upgrade • Don’t collected too frequently • these demo collections ⥇ what production collections • Verify Documentation Recommendations
- 44. Resources
- 45. • Kinetic Edge Blog https://www.kineticedge.io/insights • GitHub • https://github.com/kineticedge/kafka-streams-dashboards • Sample Applications, dashboards, kraft cluster, zk cluster, “load-balanced” cluster • docker - tra ffi c control, single image to run application (with libraries built in — fast!) • KIPs • KIP-444: Augment metrics for Kafka Streams - 2.4 (& 2.5, 2.6) • KIP-471: Expose RocksDB Metrics in Kafka Streams - 2.4 (& 3.2) • KIP-607: Add Metrics to Kafka Streams to Report Properties of RocksDB - 2.7 • KIP-613: Add end-to-end latency metrics to Streams - 2.6 (& 2.7) • KIP-846: Source/sink node metrics for Consumed/Produced throughput in Streams - 3.3 • KIP-869: Improve Streams State Restoration Visibility - 3.5 (They keep on coming!) • StateUpdated also needs to be enabled (experimental?) Resources restart < 3 seconds