Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[PromCon2018] Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters

377 views

Published on

Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configured monitoring "packages" hard to build. Simultaneously, it's becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying "Monitoring Mixins" - extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

Published in: Technology
  • Be the first to comment

[PromCon2018] Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters

  1. 1. Prometheus Monitoring Mixins Using Jsonnet to Package Together Dashboards and Alerts Tom Wilkie @tom_wilkie August 2018
  2. 2. sum(container_memory_working_set_bytes{ id=“/“, kubernetes_io_hostname=~”^$Node$” }) / sum(machine_memory_bytes{ kubernetes_io_hostname=~”^$Node$” })
  3. 3. Dashboards and alerts should not be opinionated about labels - this should be configuration.
  4. 4. Want: - configurable - extensible - reusable
  5. 5. Configurable & Extensible
  6. 6. • YAML & JSON • envsubst, m4, jinja etc • Go, Python
  7. 7. from grafanalib.core import * dashboard = Dashboard( title="Frontend Stats", rows=[ Row(panels=[ Graph( title="Frontend QPS", dataSource='My Prometheus', targets=[ Target( expr='sum(irate(nginx_http_requests_total{job="default/ frontend",status=~"1.."}[1m]))', legendFormat="1xx", refId='A', ), ... https://github.com/weaveworks/grafanalib
  8. 8. jsonnet A data templating language for app and tool developers. A simple extension of JSON. http://jsonnet.org/ @sparkprime
  9. 9. Jsonnet Demo
  10. 10. Configurable & Extensible (II)
  11. 11. {  _config+:: {...},  grafanaDashboards+:: {    “dashboard-name.json”: {...},  },  prometheusAlerts+:: [...],  prometheusRules+:: [...], }
  12. 12. { _config+:: { // Selectors are inserted between {} in Prome kubeStateMetricsSelector: 'job="default/kube-state-metrics"', ... }, }
  13. 13. { prometheusAlerts+:: { groups+: [{ name: 'kubernetes-apps', rules: [{ alert: 'KubePodCrashLooping', 'for': '1h', expr: ||| rate(kube_pod_container_status_restarts {%(kubeStateMetricsSelector)s}[15m]) > 0 ||| % $._config, ...
  14. 14. groups: - name: kubernetes-apps rules: - alert: KubePodCrashLooping expr: | rate(kube_pod_container_status_restarts {job=“default/kube-state-metrics”}[15 > 0 for: 1h ...
  15. 15. Reusable
  16. 16. jsonnet-bundler A jsonnet package manager. https://github.com/jsonnet-bundler/jsonnet-bundler
  17. 17. $ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/j $ jb init $ jb install github.com/kubernetes-monitoring/kubernetes- Cloning into 'vendor'... $ $ tree vendor vendor ├── grafonnet │   └── ... └── kubernetes-mixin └── ...
  18. 18. https://goo.gl/JMxwC3 https://goo.gl/JocRm6
  19. 19. …since KubeCon
  20. 20. Application Mixin CoreOS Etcd github.com/coreos/etcd/pull/9640 Hashicorp Consul github.com/kausalco/public/consul-mixin Hashicorp Vault github.com/grapeshot/vault_exporter/vault-mixin Kubernetes github.com/kubernetes-monitoring/kubernetes-mixin
  21. 21. Now up to 9 contributors. @metalmatze became a committer. Contributions from Google, MongoDB, San Diego Supercomputer Centre. Used in all future versions of OpenShift.
  22. 22. But wait! We have new mixins… Prometheus mixin: https://github.com/prometheus/prometheus/pull/4474 node_exporter mixin: https://github.com/prometheus/node_exporter/pull/941 And…
  23. 23. Thanks for listening. Questions? Tom Wilkie @tom_wilkie May 2018 Photo credits: https://unsplash.com/photos/dmkmrNptMpw https://unsplash.com/photos/nlMYrApFE7s

×