Report

WeaveworksFollow

Oct. 16, 2017•0 likes•27,492 views

Oct. 16, 2017•0 likes•27,492 views

Report

Technology

- What is PromQL - PromQL operators - PromQL functions - Hands on: Building queries in PromQL - Hands on: Visualizing PromQL in Grafana - Prometheus alerts in PromQL - Hands on: Creating an alert in Prometheus with PromQL

WeaveworksFollow

Monitoring using Prometheus and GrafanaArvind Kumar G.S

VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics

Prometheus OverviewBrian Brazil

Getting Started Monitoring with Prometheus and GrafanaSyah Dwi Prihatmoko

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

Introduction to PrometheusJulien Pivotto

- 1. Prometheus: PromQL Deep Dive Jeff Hoffer, Developer Experience github.com/eudaimos
- 2. Agenda 1. What is PromQL 2. PromQL Operators 3. PromQL Functions 4. Hands On: Building Queries in PromQL 5. Hands On: Visualizing PromQL in Grafana 6. Training on real app 7. What’s next?
- 3. What is PromQL • Powerful Query Language of Prometheus • Provides built in operators and functions • Vector-based calculations like Excel • Expressions over time-series vectors
- 4. Expressions (and sub-expressions) • Instant Vector - set of time series containing single sample for each time series, all sharing same timestamp • e.g. http_request_count => results in: • http_request_count{status=“200”} 20 • http_request_count{status=“404”} 3 • http_request_count{status=“500”} 5
- 5. Expressions (and sub-expressions) • Instant Vector - set of time series containing single sample for each time series, all sharing same timestamp • e.g. http_request_count => results in: • http_request_count{status=“200”} 20 • http_request_count{status=“404”} 3 • http_request_count{status=“500”} 5 • Range Vector - set of time series containing a range of data points over time for each series • e.g. http_request_count[5m] => results in: • http_request_count{status=“200”}
- 6. Expressions (and sub-expressions) • Instant Vector - set of time series containing single sample for each time series, all sharing same timestamp • e.g. http_request_count => results in: • http_request_count{status=“200”} 20 • http_request_count{status=“404”} 3 • http_request_count{status=“500”} 5 • Range Vector - set of time series containing a range of data points over time for each series • e.g. http_request_count[5m] => results in: • http_request_count{status=“200”} • Scalar - as a literal and as result of an expression • String - only currently as a literal in an expression
- 7. Time Series Selectors • Instant Vector Selectors • num_nodes • num_nodes{role=“backend”}
- 8. Time Series Selectors • Instant Vector Selectors • num_nodes • num_nodes{role=“backend”} • Range Vector Selectors (s, m, h, d, w, y) • num_nodes{role=“backend”}[5m]
- 9. Time Series Selectors • Instant Vector Selectors • num_nodes • num_nodes{role=“backend”} • Range Vector Selectors (s, m, h, d, w, y) • num_nodes{role=“backend”}[5m] • Offset Modifier • num_nodes{role=“backend”}[5m] offset 1w
- 10. Operators: Binary • Arithmetic: +, -, *, /, %, ^ – scalar/scalar – vector/scalar – vector/vector
- 11. Operators: Binary • Arithmetic: +, -, *, /, %, ^ – scalar/scalar – vector/scalar – vector/vector • Comparison: ==, !=, >, <, >=, <= – filters results unless bool operator provided (converts 0 or 1) – scalar/scalar requires bool operator – vector/scalar & vector/vector drops elements unless bool operator provided
- 12. Operators: Binary • Arithmetic: +, -, *, /, %, ^ – scalar/scalar – vector/scalar – vector/vector • Comparison: ==, !=, >, <, >=, <= – filters results unless bool operator provided (converts 0 or 1) – scalar/scalar requires bool operator – vector/scalar & vector/vector drops elements unless bool operator provided • Logical/Set Binary: only defined between Instant Vectors – and = intersection between vector1 and vector2 – or = union of vector1 and vector2 – unless = elements of vector1 for which no matches in vector2
- 13. Operators: Vector Matching • Label Matching – ignoring keyword – on keyword
- 14. Operators: Vector Matching • Label Matching – ignoring keyword – on keyword • One-to-one - finds unique pair of entries with all labels matching
- 15. Operators: Vector Matching • Label Matching – ignoring keyword – on keyword • One-to-one - finds unique pair of entries with all labels matching • Many-to-one / One-to-many - where each element on a “one” side can multiple elements on the “many” side – group_left v group_right determines cardinality – only used for comparison and arithmetic operations
- 16. Operators: Aggregation • Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values – sum, avg – min, max – stddev, stdvar – count, count_values* – bottomk*, topk* – quantile* *takes a parameter before the vector
- 17. Operators: Aggregation • Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values – sum, avg – min, max – stddev, stdvar – count, count_values* – bottomk*, topk* – quantile* *takes a parameter before the vector • without clause removes listed labels from resulting vector
- 18. Operators: Aggregation • Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values – sum, avg – min, max – stddev, stdvar – count, count_values* – bottomk*, topk* – quantile* *takes a parameter before the vector • without clause removes listed labels from resulting vector • by clause drops labels not listed from the resulting vector
- 19. Operators: Aggregation • Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values – sum, avg – min, max – stddev, stdvar – count, count_values* – bottomk*, topk* – quantile* *takes a parameter before the vector • without clause removes listed labels from resulting vector • by clause drops labels not listed from the resulting vector • keep_common (with by) will keep labels that exist in all elements but not listed in the by clause
- 20. Operators: Aggregation • Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values – sum, avg – min, max – stddev, stdvar – count, count_values* – bottomk*, topk* – quantile* *takes a parameter before the vector • without clause removes listed labels from resulting vector • by clause drops labels not listed from the resulting vector • keep_common (with by) will keep labels that exist in all elements but not listed in the by clause • topk/bottomk - only subset of original values are returned including original labels - by and without only bucket the input
- 21. Functions: Utilities • time() - number of seconds since Unix Epoch when the expression is run • vector(s scalar) - returns a vector from a scalar • scalar(v vector) - returns scalar value of a single sampled vector or NaN
- 22. Functions: Time-based Instant Vector • default v=vector(time()) • day_of_month(v) • day_of_week(v) • days_in_month(v) • hour(v) • minute(v) • month(v) • year(v)
- 23. Functions: Instant Vector • abs(v) • absent(v) • ceil(v) • clamp_max(v, scalar), clamp_min(v, scalar) - clamps the sample values to have an upper/lower limit • count_scalar(v) • drop_common_labels(v) • exp(v) • floor(v), round(v) • label_replace(v, dst_label string, replacement string, src_label string, regex string) • ln(v), log2(v), log10(v) • sort(v), sort_desc(v) • sqrt(v)
- 24. Functions: Range Vector • changes()ˆ • delta()˚*, idelta()˚* - diff between first and last in each time series element • deriv()* - per sec derivative using simple linear regression • holt_winters(v, sf scalar, tf scalar)* - smooth value for time series based on range in v • increase()ˆ - syntactic sugar for rate(v[T]) * (seconds in T) • irate()ˆ, rate()ˆ - per second instant/avg rate of increase • predict_linear(v, t scalar)* - predict value at time t using simple linear regression • resets()ˆ - number of times a counter reset • <aggregation>_over_time()˚ - aggregate each series of a range vector over time returning instant vector with per series aggregation results • ˚returns an instant vector, *should only be used with gauges, ˆshould only be used with counters
- 25. Metrics Types Basic Counters Sampling Counters counter histogram gauge summary
- 26. Metrics Types - Basic Counters • counter - single numeric metric that only goes up • gauge - single numeric metric that arbitrarily goes up or down
- 27. Metric Types - Sampling Counters • histogram - samples observations and counts them in configurable buckets • summary - samples observations and counts them
- 28. Metric Types - Sampling Counters • histogram - samples observations and counts them in configurable buckets • summary - samples observations and counts them
- 29. Metrics Types - Sampling Counters Histogram!?
- 30. Metric Types - Sampling Counters • both histogram and summary have: – <name>_sum - time series summing the value of all observations – <name>_count - time series counter for the number of observations taken
- 31. Metric Types - Sampling Counters • both histogram and summary have: – <name>_sum - time series summing the value of all observations – <name>_count - time series counter for the number of observations taken • histograms: – buckets are configured on client when creating metrics – time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number of observations less than or equal to the upper bound of the bucket – ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function
- 32. Metric Types - Sampling Counters • both histogram and summary have: – <name>_sum - time series summing the value of all observations – <name>_count - time series counter for the number of observations taken • histograms: – buckets are configured on client when creating metrics – time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number of observations less than or equal to the upper bound of the bucket – ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function • summaries: – quantiles are defined on the client when creating metrics – time series for each quantile as <name>{…,quantile=“<quantile-upper-bound>”} keeping the streaming quantile calculation from the client – are generally not aggregatable
- 33. Refining Rate rate(requests[5m])
- 34. Refining Rate rate(requests[5m]) sum(rate(requests[5m])) by(service_name)
- 35. Refining Rate rate(requests[5m]) sum(rate(requests[5m])) by(service_name) sum(rate(requests{service_name=“catalogue”}[5m])) by(instance)
- 36. Refining Rate rate(requests[5m]) sum(rate(requests[5m])) by(service_name) sum(rate(requests{service_name=“catalogue”}[5m])) by(instance) request_duration as a histogram
- 37. Refining Rate rate(requests[5m]) sum(rate(requests[5m])) by(service_name) sum(rate(requests{service_name=“catalogue”}[5m])) by(instance) request_duration as a histogram - derive average request duration over a rolling 5 minute period
- 38. Refining Rate rate(requests[5m]) sum(rate(requests[5m])) by(service_name) sum(rate(requests{service_name=“catalogue”}[5m])) by(instance) request_duration as a histogram - derive average request duration over a rolling 5 minute period rate(request_duration_sum[5m]) / rate(request_duration_count[5m])
- 39. RED Monitoring • (Request) Rate - the number of requests per second your services are serving • (Request) Errors - the number of failed requests per second • (Request) Duration - distributions of the amount of time each request takes
- 40. Training!
- 41. Join the Weave user group! meetup.com/pro/Weave/ weave.works/help
- 42. Other topics • Kubernetes 101 • Continuous delivery: hooking up my CI/CD pipeline to Kubernetes • Network policy for security We have talks on all these topics in the Weave user group!
- 43. Thanks! Questions? We are hiring! DX in San Francisco Engineers in London & SF weave.works/weave-company/hiring