VictoriaMetrics 2022 release includes the following new features: - MetricsQL adds features like support for @ modifier to return metric values at timestamps, keep_metric_names modifier, and advanced label filters' propagation. - vmui includes a new cardinality explorer to help determine the source of high cardinality and churn rate, and top queries page to see slow and frequent queries. - vmagent allows fetching targets' responses, filtering targets by URL and labels, and supports new service discovery mechanisms. It also includes multi-tenant support and performance improvements. - Other features include conditional and named relabeling, vmalert improvements, vmctl data migration capabilities, mTLS support, and automatic restore

1. © 2022 VictoriaMetrics
VictoriaMetrics 2022:
new features

2. MetricsQL features
https://docs.victoriametrics.com/MetricsQL.html

3. MetricsQL: support for @ modifier
● m @ timestamp - returns m value at the given timestamp
● m @ end() - returns m value at the end of the selected time range
● m @ (start() + 1h) - returns m value a hour after the beginning of the graph
● sum(foo) @ end() - returns sum(foo) at the end of the selected time range
● The last two features aren’t available in Prometheus yet

4. MetricsQL: keep_metric_names modifier
● rate({__name__=~”http_(requests|errors)_total”}) keep_metric_names - returns series with
http_requests_total and http_errors_total names
● Prometheus returns vector cannot contain metrics with the same labelset error :(
● keep_metric_names modifier works with any MetricsQL function!

5. MetricsQL: advanced label filters’ propagation
rate(http_requests_total{job=”a”}) + rate(http_errors_total)
is automatically optimized to:
rate(http_requests_total{job=”a”}) + rate(http_errors_total{job=”a”})
foo{instance=”a”,status=”success”} / on(instance) sum(foo) by (instance)
is automatically optimized to:
foo{instance=”a”,status=”success”} / on(instance) sum(foo{instance=”a”}) by (instance)
See https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization

6. MetricsQL: automatic label filters’ propagation
q * on(instance) group_left(host) node_info
is automatically optimized to:
q * on(instance) group_left(host) node_info{instance=~”h1|...|hN”}
Where h1, …, hN - unique instance labels returned by q.

7. MetricsQL: support for short numeric constants
● 10KB is automatically expanded to 10000
● 20MiB is automatically expanded to 20*1024*1024
● 1.2G is automatically expanded to 1.2*1000*1000*1000
Examples:
● rate(network_bytes_sent_total) / 1MB - network transfer rate in MBytes/sec
● process_resident_memory_bytes / 1GiB - memory usage in GiBytes
This is a good addition for already supported duration constants:
● sum_over_time(scrape_samples_scraped[1h]) / 1h - average samples’ scrape rate
over the last hour

8. MetricsQL: distributed query tracing!
● An analogue of EXPLAIN ANALYZE from SQL
● Helps determining why the query is slow
● Helps optimizing slow queries
● Helps with query debugging
● https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-traci
ng

9. MetricsQL: distributed query tracing!

10. MetricsQL: new functions
● drop_common_labels(q) - drops common label=value pairs from q results
● tlast_change_over_time(m[d]) - returns the timestamp of the last change for m over the
duration d
● sort_by_label_numeric(q, “label”) and sort_by_label_numeric_desc(q, “label”) - sorts
time series q by the given label using numeric sort
● range_normalize(q) - normalizes every time series returned by q into the range [0 … 1]
● range_linear_regression(q) - linear regression per each time series returned by q. Can
be used for simple predictions
● range_stddev(q) and range_stdvar(q) - returns standard deviation and variance per each
time series returned by q
● range_trim_spikes(phi, q) - trims phi (0…1) percent of the biggest spikes/drops from
every series returned by q

11. vmui features

12. vmui: cardinality explorer!
● Helps determining the source of high cardinality
● Helps determining the source of high churn rate
● https://docs.victoriametrics.com/#cardinality-explorer

13. vmui: cardinality explorer!

14. vmui: top queries
● Helps determining slowest queries
● Helps determining the most frequently executed queries

15. vmui: top queries

16. vmui: significantly improved usability and stability!

17. vmagent features

18. vmagent: fetch target response on behalf of vmagent

19. vmagent: filter targets by url and labels

20. vmagent: /service-discovery page
● Helps understanding why the discovered target has been deleted
● Helps understanding why the discovered target has the given set of labels

21. vmagent: relabel debugging!

22. vmagent: support for absolute __address__

23. vmagent: new service discovery mechanisms
● azure_sd_configs - Azure service discovery
● yandexcloud_sd_configs - Yandex cloud service discovery
● https://docs.victoriametrics.com/sd_configs.html - docs for supported service
discovery mechanisms

24. vmagent: multi-tenant support
● Allow specifying the destination tenant via __tenant_id__ label
● Allow writing multi-tenant data to a single multi-tenant url at vminsert
● https://docs.victoriametrics.com/vmagent.html#multitenancy

25. vmagent: performance improvements
● Optimized CPU and memory usage when discovering and scraping big number of
targets
● Optimized performance for relabeling rules with non-trivial regular expressions

26. relabeling features

27. relabeling: conditional relabeling
- if: ‘{env=~”dev|staging”,team=”qa”}’
action: drop
● Executes the relabeling action only on successful “if” label selector match
● Can be applied to any relabeling action

28. relabeling: named label placeholders
The replacement field may contain named placeholders for any existing labels

29. relabeling: Graphite-style relabeling
● Easier and faster approach for transforming Graphite metric names into Prometheus
metrics
● https://docs.victoriametrics.com/vmagent.html#graphite-relabeling

30. vmalert features

31. vmalert: better integration with Grafana alerts
● Single-node VictoriaMetrics and vmselect allow proxying Grafana requests to
vmalert if -vmalert.proxyURL command-line option is set
● This allows investigating vmalert alerting configs via Grafana UI

32. vmalert: reusable templates for annotations
● https://docs.victoriametrics.com/vmalert.html#reusable-templates

33. vmalert: debugging of alerting rules

34. vmalert: improved compatibility with Prometheus
● Add missing template functions, which are provided by Prometheus -
https://docs.victoriametrics.com/vmalert.html#template-functions
● Improved compatibility with Prometheus Alert Generator Specification -
https://github.com/prometheus/compliance/blob/main/alert_generator/specific
ation.md

35. vmctl features

36. vmctl: migrate all the data between clusters
● vmctl can automatically discover and migrate data for all the tenants from one
VictoriaMetrics cluster to another one -
https://docs.victoriametrics.com/vmctl.html#cluster-to-cluster-migration-mode

37. vmctl: data migration via Prometheus remote_read protocol
● This allows migrating data from systems with Prometheus remote_read protocol
support - Prometheus, Thanos, Cortex and Mimir
● https://docs.victoriametrics.com/vmctl.html#migrating-data-by-remote-read-pr
otocol

38. Enterprise features

39. Enterprise: mTLS support
● mTLS can be set up for communicating between cluster components of
VictoriaMetrics - see
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection

40. Enterprise: vmgateway JWT token enhancements
● Add ability to read JWT token from any HTTP header
● Add ability to verify JWT token signature -
https://docs.victoriametrics.com/vmgateway.html#jwt-signature-verification

41. Enterprise: automatic restore from backups
● Add support for automatic restore from backups by vmbackupmanager -
https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-back
up-via-cli

42. Enterprise: automatic vmstorage discovery
● Automatic vmstorage nodes’ discovery at vminsert and vmselect via dns+srv, http
and files -
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmsto
rage-discovery
Examples:
● -storageNode=dns+srv:vmstorage-hosts-for-vmselect
● -storageNode=file:/path/to/file/with-vmstorage-nodes
● -storageNode=file:http://config-host/vmstorage-nodes

43. Enterprise: multiple retentions
● Support for multiple retentions (per tenant or per arbitrary set of metrics) -
https://docs.victoriametrics.com/#retention-filters and
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters
The following config sets retention to 1 day for tenants with accountID
starting from ‘42', then sets retention to 3 days for time series with label
env="dev" or env="prod" from any tenant, while the rest of tenants
will have 4 weeks retention:
-retentionFilter='{vm_account_id=~"42.*"}:1d'
-retentionFilter='{env=~"dev|staging"}:3d'
-retentionPeriod=4w

44. various enhancements

45. Environment vars can be referred in command-line flags
-someFlag=%{ENV_VAR} => -someFlag=some_value if ENV_VAR=some_value
This works in any shell (and without shell, e.g. inside scratch Docker image)!
Use cases:
● Passing secrets via command-line flags: -metricsAuthKey=%{TOP_SECRET_KEY}
● Explicit configuration by specifying the needed command-line flags and referring
the exact env vars in these flags (easier to debug than implicit config via env vars)
https://docs.victoriametrics.com/#environment-variables

46. Performance improvements
● /api/v1/labels, /api/v1/label/…/values and /api/v1/series work faster
● /api/v1/export/… and /federate work faster
● Improved query performance when replication is enabled
● Improved query performance on hosts with many CPU cores

47. Deduplication and downsampling improvements
● Leave the last sample instead of the first sample on the discrete dedup interval.
This aligns better with Prometheus data model
● Leave the sample with the biggest value out of multiple samples with the same
timestamp

48. Support for metrics push
All the VictoriaMetrics components support automatic metrics push to the configured
remote storage via -pushmetrics.url command-line flag -
https://docs.victoriametrics.com/#push-metrics
Use case: to collect metrics from restricted environments where external scrape isn’t
possible

49. Support for Pushgateway data ingestion format
VictoriaMetrics and vmagent can be used as a drop-in Pushgateway replacement
https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-for
mat

50. VictoriaMetrics cluster: multitenancy enhancements
● Add ability to pass accountID and projectID via labels -
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via
-labels

51. vmbackup / vmrestore: Azure blob storage
● Add ability to make backups to Azure blob storage
● https://docs.victoriametrics.com/vmbackup.html

52. Official MacOS builds
● VictoriaMetrics releases now contain MacOS builds for amd64 and arm64
architectures

53. Official FreeBSD and OpenBSD builds
● The builds are published at
https://github.com/VictoriaMetrics/VictoriaMetrics/releases

54. Raspberry PI optimizations
● Improved performance for arm64
● Allow controlling the frequency of data write to disk via
-inmemoryDataFlushInterval command-line flag - prolongs life for low-end SD
cards

55. LTS release
● v1.79.x is a line of long-term support releases
● LTS line contains only bugfixes without new features
● Major LTS releases are supported for 12 months
● Major LTS releases are created every 6 months

56. The full list of new features
● The full list of new features is available at
https://docs.victoriametrics.com/CHANGELOG.html

57. Questions?