Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configured monitoring "packages" hard to build. Simultaneously, it's becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration. In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying "Monitoring Mixins" - extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

1. Prometheus
Monitoring
Mixins
Using Jsonnet to Package Together
Dashboards and Alerts
Tom Wilkie @tom_wilkie
August 2018

4. sum(container_memory_working_set_bytes{
id=“/“,
kubernetes_io_hostname=~”^$Node$”
})
/
sum(machine_memory_bytes{
kubernetes_io_hostname=~”^$Node$”
})

5. Dashboards and alerts should
not be opinionated about labels -
this should be configuration.

8. Want:
- configurable
- extensible
- reusable

9. Configurable &
Extensible

10. • YAML & JSON
• envsubst, m4, jinja etc
• Go, Python

11. from grafanalib.core import *
dashboard = Dashboard(
title="Frontend Stats",
rows=[
Row(panels=[
Graph(
title="Frontend QPS",
dataSource='My Prometheus',
targets=[
Target(
expr='sum(irate(nginx_http_requests_total{job="default/
frontend",status=~"1.."}[1m]))',
legendFormat="1xx",
refId='A',
),
...
https://github.com/weaveworks/grafanalib

12. jsonnet
A data templating language for app and
tool developers.
A simple extension of JSON.
http://jsonnet.org/ @sparkprime

14. Jsonnet Demo

15. Configurable &
Extensible (II)

16. {
_config+:: {...},
grafanaDashboards+:: {
“dashboard-name.json”: {...},
},
prometheusAlerts+:: [...],
prometheusRules+:: [...],
}

17. {
_config+:: {
// Selectors are inserted between {} in Prome
kubeStateMetricsSelector:
'job="default/kube-state-metrics"',
...
},
}

18. {
prometheusAlerts+:: {
groups+: [{
name: 'kubernetes-apps',
rules: [{
alert: 'KubePodCrashLooping',
'for': '1h',
expr: |||
rate(kube_pod_container_status_restarts
{%(kubeStateMetricsSelector)s}[15m])
> 0
||| % $._config,
...

19. groups:
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
expr: |
rate(kube_pod_container_status_restarts
{job=“default/kube-state-metrics”}[15
> 0
for: 1h
...

20. Reusable

22. jsonnet-bundler
A jsonnet package manager.
https://github.com/jsonnet-bundler/jsonnet-bundler

23. $ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/j
$ jb init
$ jb install github.com/kubernetes-monitoring/kubernetes-
Cloning into 'vendor'...
$ $ tree vendor
vendor
├── grafonnet
│ └── ...
└── kubernetes-mixin
└── ...

24. https://goo.gl/JMxwC3
https://goo.gl/JocRm6

25. …since KubeCon

26. Application Mixin
CoreOS Etcd github.com/coreos/etcd/pull/9640
Hashicorp Consul github.com/kausalco/public/consul-mixin
Hashicorp Vault github.com/grapeshot/vault_exporter/vault-mixin
Kubernetes github.com/kubernetes-monitoring/kubernetes-mixin

27. Now up to 9 contributors.
@metalmatze became a committer.
Contributions from Google, MongoDB,
San Diego Supercomputer Centre.
Used in all future versions of OpenShift.

28. But wait! We have new mixins…
Prometheus mixin:
https://github.com/prometheus/prometheus/pull/4474
node_exporter mixin:
https://github.com/prometheus/node_exporter/pull/941
And…

30. Thanks for
listening.
Questions?
Tom Wilkie @tom_wilkie
May 2018
Photo credits:
https://unsplash.com/photos/dmkmrNptMpw
https://unsplash.com/photos/nlMYrApFE7s