Let's have a look at all the recent features and changes in the Prometheus server and the community. We will introduce the new features and see how you can integrate them in your workflows to get a better Prometheus experience.

1. What’s new in Prometheus and its ecosystem?
Julien Pivotto
November 15, 2022
O11y

2. Who am I
Julien Pivotto
• Prometheus maintainer
• Open Source
• In the monitoring space for 10+ years
O11y 1

3. Intro
• Let’s dive into ~1 year of news
• We can only scratch the surface
• Many thanks to everyone contributing
• Along with the features, many bugfixes were contributed!
O11y 2

4. Recap
O11y

5. What is Prometheus?
• Open Source monitoring solution
• Based on Metrics
• Powerful query language: PromQL
• Pull based
• Service Discovery
O11y 3

6. Prometheus history
• Born in 2012 at SoundCloud
• Open Source in 2015
• Graduated CNCF project in 2018
O11y 4

7. Prometheus
O11y

8. Prometheus UI
• Migration to the new React UI
• Better autocomplete
• Search bars
O11y 5

9. Agent mode
Agent mode is Prometheus without querying capabilities. Service discovery,
Scrapes & Remote Write.
O11y 6

10. Remote Write receivers
• Marked as stable
• Remote Write is a replication protocol for Prometheus metrics
O11y 7

11. New service discoveries
Now supported:
Azure, Consul, DigitalOcean, Docker, Docker Swarm, DNS, EC2, Openstack,
PuppetDB, File SD, GCE, Hetzner, HTTP SD, IONOS, Kubernetes, Kuma, Lightsail,
Linode, Marathon, Nerve, Nomad, Serverset, Triton, Eureka, Scaleway, Uyuni, Vultr,
OVHCloud.
O11y 8

12. HTTP SD
Generic HTTP Service discovery. You can expose targets as JSON.
O11y 9

13. Relabeling
• __scrape_interval__ and __scrape_timeout__ labels can be used to
configure the scrape interval and timeout.
• New relabeling actions: uppercase and lowercase.
O11y 10

14. PromQL
• Trigonometric functions: sin, cos, deg, rad
• sgn, last_over_time, present_over_time, clamp functions
O11y 11

15. PromQL negative offset
The offset keyword now takes negative values, allowing to look ahead.
up offset -10m
O11y 12

16. PromQL @ modifier
Enables you to select vectors at fixed times, regardless of the current step.
rate(http_requests_total[1m])
and
topk(5, rate(http_requests_total[1h] @ end()))
O11y 13

17. Formatting queries
The Prometheus UI now has a feature to format queries.
O11y 14

18. Auto GOMAXPROCS
--enable-feature=auto-gomaxprocs
When set, GOMAXPROCS variable is automatically set to match Linux container
CPU quota.
O11y 15

19. Native histograms
Native histograms are better histograms, more efficient and accurate.
O11y 16

20. Out of order ingestion
Prometheus can ingest metrics out of order.
O11y 17

21. Long Term Support
Prometheus 2.37 LTS is a version of Prometheus that is maintained for 6+ months.
Regular releases are supported for 6 weeks.
O11y 18

22. Prometheus as a Go library
• Many projects use (part of) the Prometheus code base
• We now mirror the codebase for go developers
• End-user v2.40.0 -> go.mod v0.40.0
O11y 19

23. Plugins system
• Prometheus binary is ~110M.
• Mainly due to service discoveries
• Plugins.yml to reduce binary size
• Disabling all but kubernetes SD gains ~45% disk space (down to 70M)
O11y 20

24. Alertmanager
O11y

25. Alertmanager
• Time-based muting for receivers
• Negative matchers
O11y 21

26. Alertmanager new receivers
• AWS SNS
• Telegram
• Discord (coming soon)
O11y 22

27. Exporters
O11y

28. Reaching exporters
• systemd socket activation
• multiple bind addresses
O11y 23

29. Node Exporter
O11y

30. Node Exporter
Some new collectors
(partial list)
• sysctl
• selinux
• DMI
• OS release
O11y 24

31. MySQLd Exporter
Multi target is coming!
O11y 25

32. Blackbox Exporter
• Hostname parameter:
/probe?
target=104.21.60.220
&module=http_2xx
&hostname=prometheus.io
• skip_resolve_phase_with_proxy option
• gRPC health check support
O11y 26

33. Ecosystem updates
O11y

34. Rust client
New official rust client: https:/
/github.com/prometheus/client_rust
O11y 27

35. github.com/prometheus-community
Now 30+ repos
• ecs_exporter
• smartctl_exporter
• systemd_exporter
• json_exporter
• elasticsearch_exporter
• postgres_exporter
• windows_exporter
• PushProx
• helm-charts
O11y 28

36. postgres_exporter
Supports multi target
O11y 29

37. avalanche
Load testing for Prometheus and Remote Write
O11y 30

38. • Prometheus Certified Associate (PCA)
• Hosted by the Linux Foundation Training & Certification
• Launched in September
O11y 31

39. Prometheus Conformance Program
https:/
/github.com/cncf/prometheus-conformance
O11y 32

40. Open Sourced & given to the Prometheus project last week by Julius (Promlabs)
and Chronosphere.
docker run --network=host prom/promlens
• PromQL query builder
• Query explainer
• Metrics explorer
O11y 33

41. Questions?
O11y

42. Julien Pivotto
@roidelapluie
O11y
https:/
/o11y.eu/prometheus-support/
O11y 34