The document provides an in-depth overview of PromQL, the powerful query language of Prometheus, detailing its operators, functions, and data types. It covers topics such as instant and range vectors, time series selectors, and various arithmetic and aggregation operators. Hands-on sessions include building queries, visualizing data in Grafana, and applying these concepts in a real application context.

1. Prometheus:
PromQL Deep Dive
Jeff Hoffer, Developer Experience
github.com/eudaimos

2. Agenda
1. What is PromQL
2. PromQL Operators
3. PromQL Functions
4. Hands On: Building Queries in PromQL
5. Hands On: Visualizing PromQL in Grafana
6. Training on real app
7. What’s next?

3. What is PromQL
• Powerful Query Language of Prometheus
• Provides built in operators and functions
• Vector-based calculations like Excel
• Expressions over time-series vectors

4. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5

5. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5
• Range Vector - set of time series containing a range of data points over time for each series
• e.g. http_request_count[5m] => results in:
• http_request_count{status=“200”}

6. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5
• Range Vector - set of time series containing a range of data points over time for each series
• e.g. http_request_count[5m] => results in:
• http_request_count{status=“200”}
• Scalar - as a literal and as result of an expression
• String - only currently as a literal in an expression

7. Time Series Selectors
• Instant Vector Selectors
• num_nodes
• num_nodes{role=“backend”}

8. Time Series Selectors
• Instant Vector Selectors
• num_nodes
• num_nodes{role=“backend”}
• Range Vector Selectors (s, m, h, d, w, y)
• num_nodes{role=“backend”}[5m]

9. Time Series Selectors
• Instant Vector Selectors
• num_nodes
• num_nodes{role=“backend”}
• Range Vector Selectors (s, m, h, d, w, y)
• num_nodes{role=“backend”}[5m]
• Offset Modifier
• num_nodes{role=“backend”}[5m] offset 1w

10. Operators: Binary
• Arithmetic: +, -, *, /, %, ^
– scalar/scalar
– vector/scalar
– vector/vector

11. Operators: Binary
• Arithmetic: +, -, *, /, %, ^
– scalar/scalar
– vector/scalar
– vector/vector
• Comparison: ==, !=, >, <, >=, <=
– filters results unless bool operator provided (converts 0 or 1)
– scalar/scalar requires bool operator
– vector/scalar & vector/vector drops elements unless bool operator provided

12. Operators: Binary
• Arithmetic: +, -, *, /, %, ^
– scalar/scalar
– vector/scalar
– vector/vector
• Comparison: ==, !=, >, <, >=, <=
– filters results unless bool operator provided (converts 0 or 1)
– scalar/scalar requires bool operator
– vector/scalar & vector/vector drops elements unless bool operator provided
• Logical/Set Binary: only defined between Instant Vectors
– and = intersection between vector1 and vector2
– or = union of vector1 and vector2
– unless = elements of vector1 for which no matches in vector2

13. Operators: Vector Matching
• Label Matching
– ignoring keyword
– on keyword

14. Operators: Vector Matching
• Label Matching
– ignoring keyword
– on keyword
• One-to-one - finds unique pair of entries with all labels
matching

15. Operators: Vector Matching
• Label Matching
– ignoring keyword
– on keyword
• One-to-one - finds unique pair of entries with all labels
matching
• Many-to-one / One-to-many - where each element on a “one”
side can multiple elements on the “many” side
– group_left v group_right determines cardinality
– only used for comparison and arithmetic operations

16. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector

17. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector

18. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector

19. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector
• keep_common (with by) will keep labels that exist in all elements but not listed in the by clause

20. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector
• keep_common (with by) will keep labels that exist in all elements but not listed in the by clause
• topk/bottomk - only subset of original values are returned including original labels - by and without only bucket
the input

21. Functions: Utilities
• time() - number of seconds since Unix
Epoch when the expression is run
• vector(s scalar) - returns a vector
from a scalar
• scalar(v vector) - returns scalar
value of a single sampled vector or NaN

22. Functions: Time-based Instant Vector
• default v=vector(time())
• day_of_month(v)
• day_of_week(v)
• days_in_month(v)
• hour(v)
• minute(v)
• month(v)
• year(v)

23. Functions: Instant Vector
• abs(v)
• absent(v)
• ceil(v)
• clamp_max(v, scalar), clamp_min(v, scalar) - clamps the sample values to have
an upper/lower limit
• count_scalar(v)
• drop_common_labels(v)
• exp(v)
• floor(v), round(v)
• label_replace(v, dst_label string, replacement string, src_label
string, regex string)
• ln(v), log2(v), log10(v)
• sort(v), sort_desc(v)
• sqrt(v)

24. Functions: Range Vector
• changes()ˆ
• delta()˚*, idelta()˚* - diff between first and last in each time series element
• deriv()* - per sec derivative using simple linear regression
• holt_winters(v, sf scalar, tf scalar)* - smooth value for time series based on
range in v
• increase()ˆ - syntactic sugar for rate(v[T]) * (seconds in T)
• irate()ˆ, rate()ˆ - per second instant/avg rate of increase
• predict_linear(v, t scalar)* - predict value at time t using simple linear regression
• resets()ˆ - number of times a counter reset
• <aggregation>_over_time()˚ - aggregate each series of a range vector over time
returning instant vector with per series aggregation results
• ˚returns an instant vector, *should only be used with gauges, ˆshould only be used with
counters

25. Metrics Types
Basic Counters Sampling Counters
counter histogram
gauge summary

26. Metrics Types - Basic Counters
• counter - single numeric metric that only
goes up
• gauge - single numeric metric that
arbitrarily goes up or down

27. Metric Types - Sampling Counters
• histogram - samples observations and
counts them in configurable buckets
• summary - samples observations and
counts them

28. Metric Types - Sampling Counters
• histogram - samples observations and
counts them in configurable buckets
• summary - samples observations and
counts them

29. Metrics Types - Sampling Counters
Histogram!?

30. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken

31. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken
• histograms:
– buckets are configured on client when creating metrics
– time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number
of observations less than or equal to the upper bound of the bucket
– ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function

32. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken
• histograms:
– buckets are configured on client when creating metrics
– time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number
of observations less than or equal to the upper bound of the bucket
– ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function
• summaries:
– quantiles are defined on the client when creating metrics
– time series for each quantile as <name>{…,quantile=“<quantile-upper-bound>”} keeping the
streaming quantile calculation from the client
– are generally not aggregatable

33. Refining Rate
rate(requests[5m])

34. Refining Rate
rate(requests[5m])
sum(rate(requests[5m])) by(service_name)

35. Refining Rate
rate(requests[5m])
sum(rate(requests[5m])) by(service_name)
sum(rate(requests{service_name=“catalogue”}[5m]))
by(instance)

36. Refining Rate
rate(requests[5m])
sum(rate(requests[5m])) by(service_name)
sum(rate(requests{service_name=“catalogue”}[5m]))
by(instance)
request_duration as a histogram

37. Refining Rate
rate(requests[5m])
sum(rate(requests[5m])) by(service_name)
sum(rate(requests{service_name=“catalogue”}[5m]))
by(instance)
request_duration as a histogram
- derive average request duration over a rolling 5
minute period

38. Refining Rate
rate(requests[5m])
sum(rate(requests[5m])) by(service_name)
sum(rate(requests{service_name=“catalogue”}[5m]))
by(instance)
request_duration as a histogram
- derive average request duration over a rolling 5
minute period
rate(request_duration_sum[5m]) /
rate(request_duration_count[5m])

39. RED Monitoring
• (Request) Rate - the number of requests
per second your services are serving
• (Request) Errors - the number of failed
requests per second
• (Request) Duration - distributions of the
amount of time each request takes

40. Training!

41. Join the Weave user group!
meetup.com/pro/Weave/
weave.works/help

42. Other topics
• Kubernetes 101
• Continuous delivery: hooking up my CI/CD
pipeline to Kubernetes
• Network policy for security
We have talks on all these topics in the Weave
user group!

43. Thanks! Questions?
We are hiring!
DX in San Francisco
Engineers in London & SF
weave.works/weave-company/hiring