This document summarizes Coveo's cloud infrastructure, which runs primarily on Kubernetes and uses Prometheus and Thanos for monitoring. It discusses how Prometheus scrapes metrics and Thanos stores and shards the time series data across multiple regions. Grafana is used to query the global Thanos database and display dashboards, while Alertmanager handles alerts. The infrastructure has grown to include 12 Kubernetes clusters across 7 regions managing over 12,500 pods on 370 nodes.

2. C’est qui c’gars là?
Samuel Bégin
Cloud systems developer

3. Mon parcours
Developpeur
full stack
Introduced
to docker
and swarm
First
production
K8S cluster
Engagé chez
Coveo

4. Cloud infrastructure team
Maxime Poirier-Journeault Jérome Heil
Cloud system specialist Cloud system developer
Alexandre Rousseau
Cloud system developer
Dave Robitaille
Cloud system specialist
Martin Ouellet
Cloud infra team lead

5. Where we’re at
● Kubernetes is the main compute
engine
● 95% of the workload runs on Linux
● Everything is deployed autonomously
5
12 K8S clusters
8 Production clusters
12500 Pods
370 Nodes
7 Regions worldwide

6. How to keep track
of all those pieces?

7. Thanos and
Prometheus
Dev

8. Prometheus
• Defied the gods by stealing fire from them
and gave it to humanity in the form of
technology and knowledge
• Generally seen as the author of the human
arts and sciences

9. Prometheus Alone, only goes so far
- Good for the first few millions time series
- Not so redundant
- Costly to have a long retention time
9

10. Thanos
(Thanatos)
• God of non-violent deaths
• Merciless and indiscriminate, hated by
mortals and gods alike
• Also a reference to the “snap of
disintegration” in Avengers

11. High Availability
- Double prometheus instances
- Exact same configuration
- anti-affinity and pod-topology-constraints
- pod-disruption-budget

12. Sharding
- Split scraping configuration
- Exact same configuration
12

13. 13

14. Grafana and the global Thanos Querier
14

15. The lifetime of a time series
1
5

16. Your software exposes a metric
There are a lot of pre-made exporters,
but we also make our own
• Default exporters from kubernetes
• Node exporter in the base AMI
• Custom exporters for Coveo software

17. The metric is scraped
And enters the time series database
• Uses ServiceMonitors
• Most metrics are served on endpoint /metrics
• Accessible through the Sidecar

18. Groups of 3 hours are
uploaded to S3
Prometheus makes a chunk out of 3
hours of data. One chunk is made for
each time series.
Thanos sidecar will upload chunks to
aws S3

19. Downsampling
Thanos compactor will downsample our
time series. This helps with storage and
query speed over large timeranges.

20. Max retention
We configure various retention times for
time series in Thanos.

21. 21

22. 22

23. Grafana and the
global Thanos Frontend
23

24. Comment
profiter de
toute cette
information?

25. Grafana

26. Alertmanager
26

27. Autoscalers
27

28. What’s next?

29. Thank you!
2
9
29