Talk from https://stackconf.eu/talks/how-to-reduce-expenses-on-monitoring-with-victoriametrics/ Given recent economic changes, cost efficiency has become a top priority for many businesses. This is especially important for monitoring because the nature of telemetry data tends to exponential growth. Many monitoring solutions are now switching their focus to optimize costs. The talk will cover open-source instruments from VictoriaMetrics ecosystem for improving monitoring cost-efficiency. Compression optimization. While ZSTD compression and time series specific techniques like delta-encoding are great, there is still room for improvement. I’ll explain what else can be done to reduce disk footprint for long-term storage. Extra compression on data transferring between metrics collectors and TSDB. I’ll explain how VictoriaMetrics collector reduces the traffic volume by 4 times. Pre-computing for telemetry data on collectors, frequently referred to as edge computing. In VictoriaMetrics it is named as streaming aggregation and allows collectors to pre-compute data, reducing its resolution and cardinality before it is pushed to the database. This is especially important for Prometheus-like systems because streaming aggregation is compatible with Prometheus RemoteWrite protocol and can be used with any system which supports it. Cardinality explorer. Interface, which provides useful insights into data stored by the TSDB. It helps to identify the most expensive metrics or labels and see how they have changed in time. Query tracing. This feature provides details about all the stages of query execution, including time spent on index lookups, disk reads, data transfer, computation, and memory expenses. This is similar to SQL EXPLAIN feature, and helps to improve the performance of read queries. Compute efficiency. VictoriaMetrics components for collecting and storing telemetry data consume fewer resources compared to components from Prometheus ecosystem. It may sound like competition or bragging, but this is a real reason why people migrate from Prometheus to VictoriaMetrics – to cut their infrastructure costs by 2-3 times. All the features listed above are open-source and are available for everyone to use. The talk will be mostly concentrated on typical use cases in monitoring and elegant ways to make things more efficient.

1. How to reduce expenses on monitoring
with VictoriaMetrics
Roman Khavronenko | github.com/hagen1778

2. Roman Khavronenko
Co-founder of VictoriaMetrics
Software engineer with experience in distributed systems,
monitoring and high-performance services.
https://github.com/hagen1778
https://twitter.com/hagen1778

3. What this talk is about
1. Best ways for storing and processing metrics
2. Open source tools only
3. For people familiar with Prometheus,
Thanos, Mimir, VictoriaMetrics

9. Expenses!

10. You can either have a faster car…
…or be a smarter driver!

11. What can you get from simple replacing?

13. Prometheus remote-write benchmark

14. Prometheus vs VictoriaMetrics benchmark

15. # the number of nodeexporter instances to scrape
targetsCount: 1000
# how frequently to scrape nodeexporter targets
scrapeInterval: 15s
# rules evaluation interval
# https://awesome-prometheus-alerts.grep.to/rules.html#host-and-hardware-1
queryInterval: 30s
# scrapeConfigUpdatePercent is a churn rate generated once
# per scrapeConfigUpdateInterval
scrapeConfigUpdatePercent: 5
scrapeConfigUpdateInterval: 10m
Prometheus vs VictoriaMetrics benchmark

19. x16 times faster!

20. x1.9 times faster!

21. x1.7 less memory!

22. x2.5 times less!

24. Summary after 7d benchmark (1k nodeexporter targets)
Prometheus:
CPU avg used: 0.79 / 3 cores
Disk occupied: 83.5 GiB
Mem max used: 8.12 GiB / 12 GiB
Read latency avg:
50th - 70.5ms
99th - 7s
VictoriaMetrics:
CPU avg used: 0.76 / 3 cores
Disk occupied: 33 GiB
Mem max used: 4.5 GiB / 12 GiB
Read latency avg:
50th - 4.3ms
99th - 3.6s

25. Data transfer costs

26. Network Data transfer costs

27. x4.5 times less!

28. Improving network compression
1. Increase compression level, trade CPU for network savings:
a. -remoteWrite.vmProtoCompressLevel
2. Increase batch size, trade latency for compression:
a. -remoteWrite.maxBlockSize
b. -remoteWrite.maxRowsPerBlock
c. -remoteWrite.flushInterval
3. Reduce entropy to improve compression:
a. -remoteWrite.significantFigures
b. -remoteWrite.roundDigits

29. How to be smarter about data

30. Keeping only significant figures
instance:cpu_utilization:ratio_avg{instance="foo"} 0.05055757575781
instance:cpu_utilization:ratio_avg{instance="bar"} 0.05058181818236
rules:
- record: instance:cpu_utilization:ratio_avg
expr: avg_over_time(instance:node_cpu_utilization:ratio[5m])

31. Keeping only significant figures
Applying --vm-significant-figures=8 to recording rules
0.05055757575781
0.050557576
changed compression ratio from 1.2B to 0.8B per sample
See more at https://medium.com/victoriametrics-how-to-migrate-data-from-prometheus

32. Understanding the data - query tracing
VictoriaMetrics supports query tracing for detecting bottlenecks during query processing.
This is like EXPLAIN ANALYZE from Postgresql!

33. https://play.victoriametrics.com
Query tracing demo!

34. If query tracing demo didn't work…
Typical query takes 4s to execute… Why?

35. If query tracing demo didn't work…
Let's check the trace!

36. If query tracing demo didn't work…
91% of the time was spent on vmselect while aggregating
9.4k series, 13Mil data samples!

37. How to improve query speed?
1. Add more resources to monitoring.
2. Or… be smarter about data!

38. Cardinality explorer demo!
https://play.victoriametrics.com

39. If cardinality explorer demo didn't work…

40. If cardinality explorer demo didn't work…

41. If cardinality explorer demo didn't work…

42. Cardinality explorer: summary
VictoriaMetrics allows exploring time series cardinality to identify:
● Metric names with the highest number of series
● Labels with the highest number of series
● Values with the highest number of series for the selected label
● label=name pairs with the highest number of series
● Labels with the highest number of unique values
➔ Available built-in in VictoriaMetrics components
➔ Supports specifying Prometheus URL

43. Streaming aggregation vs Recording rules
The number of time series stored in TSDB
is Data-in + Recording Rules results

44. Streaming aggregation vs Recording rules
The number of time series stored in TSDB
is only what needs to be persisted

45. How to use streaming aggregation
- match: "grpc_server_handled_total" # time series selector
interval: "2m" # on 2m interval
outputs: ["total"] # aggregate as counter
without: ["grpc_method"] # group without label
Result:
grpc_server_handled_total:2m_without_grpc_method_total

46. How to use streaming aggregation
https://play.victoriametrics.com

47. Streaming aggregation: summary
1. Aggregate incoming samples in streaming mode before data is written to remote
storage
2. Aggregation is applied to all the metrics received via any supported data
ingestion protocol and/or scraped from Prometheus-compatible targets
3. Statsd alternative
4. Recording rules alternative
5. Reducing the number of stored samples
6. Reducing the number of stored series
7. Compatible with tools supporting Prometheus remote write protocol

48. Complexity penalty

49. Cortex architecture

50. Mimir architecture

51. VictoriaMetrics architecture

52. Complexity penalty
● Complex systems are harder to maintain
● Complex systems are harder to educate about
● Complex systems are more expensive to scale

53. Additional materials
1. Snapshot of Grafana dashboard from the benchmark
2. Benchmark repo for reproducing the test
3. Save network costs with VictoriaMetrics remote write protocol
4. VictoriaMetrics: achieving better compression than Gorilla for time series data
5. Streaming aggregation
6. VictoriaMetrics playground

54. Questions?
● https://github.com/VictoriaMetrics
● https://github.com/hagen1778