VictoriaMetrics and Grafana Mimir are time series databases with support of mostly the same protocols and APIs. However, they have different architectures and components, which makes the comparison more complicated. In the talk, we'll go through the details of the benchmark where I compared both solutions. We'll see how VictoriaMetrics and Mimir are dealing with identical workloads and how efficient they’re with using the allocated resources. The talk will cover design and architectural details, weak and strong points, trade-offs, and maintenance complexity of both solutions. 

1. Grafana Mimir and VictoriaMetrics
Performance Tests
Roman Khavronenko | github.com/hagen1778

2. Roman Khavronenko
Co-founder of VictoriaMetrics
Software engineer with experience in distributed systems,
monitoring and high-performance services.
https://github.com/hagen1778
https://twitter.com/hagen1778

3. The High Performance
Open Source Time Series Database & Monitoring Solution
120M+ downloads
11k stars in total
200+ contributors
Grammarly, CERN,
Open Cosmos, Wix
Semrush, Roblox

5. Stats:
● Active time series: 1Bil
● Ingestion: 50Mil/s
Resources:
● CPUs: 7000
● RAM: 30TiB
● Replicas: 1500

7. Disclaimer:
Do not simply rely on benchmarks.
Run your own tests.

8. Methodology

9. Differences in architecture

10. Mimir architecture

11. Mimir architecture

12. Mimir architecture

13. Mimir
architecture

14. Mimir allocated resources: grafana/mimir:2.2.0
Component Replicas CPU Mem (GiB)
compactor 1 1.2 2
distributor 5 2 6
ingester 5 4 25
querier 4 2 24
query-frontend 1 2 6
store-gateway 1 1 6
various caches 1* 18
Total 43.2 283

15. VictoriaMetrics architecture

16. VictoriaMetrics architecture

17. VictoriaMetrics architecture

18. VictoriaMetrics resources: VictoriaMetrics:1.80.0-cluster
Component Replicas CPU Mem (GiB)
vminsert 4 2 4
vmselect 2 8 16
vmstorage 10 2 16
Total 44 (+1 CPU) 208 (-80 GiB)

19. Prometheus remote-write benchmark

20. Prometheus remote-write benchmark

21. Prometheus remote-write benchmark

23. Benchmark
Stats:
● Duration: 24h
● Ingestion: 360K samples/s
● Unique time series: 5.5Mil
● Churn rate: 1% each 10 min
● Total number of series over 24h: 13.6Mil
● Total number of samples over 24h: 31Bil

24. Samples per second ingested

25. Mimir disk write/read

26. VictoriaMetrics disk write/read

27. Disk space used: local filesystem

28. Disk space used: object storage

29. Disk space used
Mimir:
Replication: x3
Local FS: 300 GiB
Object storage: 150 GiB
-blocks-storage.tsdb.retention-period
To reduce disk size at local FS
compactor runs compaction jobs at:
2h, 12h and 24h
compactor deduplicates replicated data
VictoriaMetrics:
Replication: x2
Local FS: 50 GiB
Downsampling
-downsampling.period=30d:5m,180d:1h
Retention filters
-retentionFilter='{env="dev"}:3d'
-retentionFilter='{env="staging"}:30d'
-retentionPeriod=1y
No object storage support
Stores all replicated data

30. VictoriaMetrics and Mimir CPU

31. VictoriaMetrics and Mimir Memory

32. Memory utilization for Mimir components

33. Memory utilization for VictoriaMetrics components

34. Mimir Read Query latency

35. VictoriaMetrics Read Query latency

36. Summary after 24h benchmark
Mimir (x3 replication):
CPU avg used: 20 / 43 cores
Mem max used: 120 GiB / 283
GiB
Read latency:
50th - 95ms
99th - 35s
VictoriaMetrics (x2 replication):
CPU avg used: 12 / 44 cores
Mem max used: 19 GiB / 208 GiB
Read latency:
50th - 165ms
99th - 17s

37. Exploring the limits

38. Benchmark - x5 increase in load
Stats:
● Duration: 3h
● Ingestion: 1.8Mil samples/s (x5)
● Unique time series: 29 Mil (x5)
● Churn rate: 1% each 10 min
● Total number of series over 3h: 32 Mil
● Total number of samples over 3h: 19 Bil

39. Samples per second ingested

40. Active time series

41. CPU utilization for VictoriaMetrics components

42. Memory utilization for VictoriaMetrics components

43. Summary after 3h benchmark
VictoriaMetrics (x2 replication):
CPU avg used: 26 / 32 cores
Mem max used: 69 GiB / 208
GiB
Read latency:
50th - 17s
99th - 120s

44. Additional materials
● Benchmark config params
● List of executed alerts on read path
● Mimir helm chart overrides
● VictoriaMetrics helm chart overrides
● Exploring limits benchmark config
● Grafana dashboards snapshots:
○ Prometheus-benchmark writer
○ Mimir and VictoriaMetrics
○ Mimir CPU usage detailed
○ Mimir and VictoriaMetrics: exploring limits
○ VictoriaMetrics cluster dashboard

45. Additional materials

46. Additional materials

47. Additional materials

48. Questions?
● https://victoriametrics.com/blog
● https://victoriametrics.com/blog/mimir-benchmark/
● VictoriaMetrics scaling to 100M samples/s
● The journey of observability infrastructure at Roblox (warn: requires email to share)
● https://github.com/VictoriaMetrics
● https://github.com/hagen1778