Testimonial by: Egor Pronin and I am a DevOps in Wedos, the largest hosting provider in the Czech Republic. We own two private data centers with more than 10k servers. One of my responsibilities from the very first day in the company was monitoring. Since ‘ancient’ times the company had its own Java-written monitoring system. It tracked only the basic server indicators: CPU, RAM, free space on file systems. The monitoring server periodically pinged targets, which served as an indicator of server availability. This simple and old system collected basic metrics and displayed alerts on the monitor in the office of administrators on duty. It operated on a server-agent model, where an agent was installed on each server to collect data and send it to the server every minute. Besides monitoring servers and applications, the system also controlled engineering infrastructure indicators. With the overall development of infrastructure and modernization of our data centers, there arose a need to display data on temperature, humidity and other indicators in server rooms to our administrators. This system is also used in our new data center with oil cooling. There are already much more sensors and detectors, so the monitoring system is still under development. In addition, this system will be connected to the other engineering systems of the data center: lighting, ventilation, air conditioning, and even blinds and gates. After familiarizing myself with the existing monitoring system and its limitations, I decided to use Zabbix, as at that time it was the most suitable tool for our purposes and tasks. However, we quickly abandoned Zabbix due to the company's transition to Kubernetes. In the new conditions, Zabbix no longer met our growing needs and did not fit into the concept of transitioning to Kubernetes. We began looking for a Kubernetes-native solution and stopped on Prometheus. All initial setup of the new monitoring system, as well as my personal acquaintance with the internals of Kubernetes, was done on it. We added targets, wrote alert rules, and experimented with creating our own exporters. However, with time, it became clear that Prometheus could not handle the rapidly growing server park and application autoscaling, causing Out Of Memory (OOM) errors. That's when I got to know about VictoriaMetrics (VM). The transition was smooth: first, we turned off data collection in Prometheus, leaving it only as a storage, and distributed the data collection and discovery tasks to vmagents. But it was painful to see how long Prometheus took to start, sometimes up to an hour and a half. Why bother with configuration and tuning if everything is readily available in VM? Thus, we fully transitioned to monitoring using the VM ecosystem.

1. V I C T O R I A M E T R I C S
&
WEDOS
DECEMBER 2023,
EGOR PRONIN, DEVOPS

2. /Custom Monitoring
System/
01 02
03

3. Indicators
Average temperature in the Server Room behind
the racks (DC1)
Average temperature in the Server Room under
the racks (DC1)
Average humidity in the Server Room under the
racks (DC1)
Average temperature of oil baths (DC2)
/Custom Monitoring System,
Version 2/

4. /Moonshots in oil/
HPE Moonshots in WEDOS DC2 are
cooled by mineral oil only.

5. /Additions:/
lighting
ventilation
air conditioning
blinds
gates

6. /Our Tools/

7. /Our Tools/

8. /Architecture/

9. API
/Core/

10. /Components/

11. Token

12. /Current
Stats/

14. CONTACT
GIT: @krakazyabra
Telegram: @krakazyabra
EGOR PRONIN, DEVOPS

15. THANK
YOU