Presented at KubeCon Copenhagen in May 2018. Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configure monitoring configuration hard to build. Simultaneously, its becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration. In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” - extensible and customisable combinations of dashboards and alert definitions. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul and Cassandra.

1. Prometheus
Monitoring
Mixins
Using Jsonnet to Package
Together Dashboards and Alerts
Tom Wilkie @tom_wilkie
May 2018

4. sum(container_memory_working_set_bytes{
id=“/“,
kubernetes_io_hostname=~”^$Node$”
})
/
sum(machine_memory_bytes{
kubernetes_io_hostname=~”^$Node$”
})

5. Dashboards and alerts should
not be opinionated about labels -
this should be configuration.

8. Configurable & Extensible
Reusable: easy to install & update
Deployment Agnostic

9. Configurable &
Extensible

10. container_memory_usage_bytes{
namespace="$namespace", pod_name="$pod"}

11. • YAML & JSON
• envsubst, m4, jinja etc
• Go, Python

12. from grafanalib.core import *
dashboard = Dashboard(
title="Frontend Stats",
rows=[
Row(panels=[
Graph(
title="Frontend QPS",
dataSource='My Prometheus',
targets=[
Target(
expr='sum(irate(nginx_http_requests_total{job="de
legendFormat="1xx",
refId='A',
),
...
https://github.com/weaveworks/grafanalib

13. jsonnet
A data templating language for app and
tool developers.
A simple extension of JSON.
http://jsonnet.org/ @sparkprime

15. Jsonnet Demo

16. Configurable &
Extensible (II)

17. {
_config+:: {...},
grafanaDashboards+:: {
“dashboard-name.json”: {...},
},
prometheusAlerts+:: [...],
prometheusRules+:: [...],
}

18. {
_config+:: {
// Selectors are inserted between {} in Prome
kubeStateMetricsSelector:
'job="default/kube-state-metrics"',
...
},
}

19. {
prometheusAlerts+:: {
groups+: [{
name: 'kubernetes-apps',
rules: [{
alert: 'KubePodCrashLooping',
'for': '1h',
expr: |||
rate(kube_pod_container_status_restarts
{%(kubeStateMetricsSelector)s}[15m])
> 0
||| % $._config,
...

20. groups:
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
expr: |
rate(kube_pod_container_status_restarts
{job=“default/kube-state-metrics”}[15
> 0
for: 1h
...

21. Reusable

22. jsonnet-bundler
A jsonnet package manager.
https://github.com/jsonnet-bundler/jsonnet-bundler

23. $ go get github.com/jsonnet-bundler/jsonnet-bundle
$ jb init
$ jb install github.com/kubernetes-monitoring/kube
Cloning into 'vendor'...
$ $ tree vendor
vendor
├── grafonnet
│ └── ...
└── kubernetes-mixin
└── ...

24. $ cat monitoring.jsonnet
local kubernetes = import "kubernetes-mixin/mixin.
kubernetes {
_config+:: {
cadvisorSelector: 'job="cadvisor"',
kubeletSelector: 'job="kubelet"',
kubeStateMetricsSelector: 'job="kube-state-met
nodeExporterSelector: 'job="node-exporter"',
notKubeDnsSelector: 'job!="kube-dns"',
},
}

25. $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((impo
groups:
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
annotations:
message: "{{ $labels.namespace }}/{{ $labels.p
expr: |
rate(kube_pod_container_status_restarts_total
{job="kube-state-metrics"}[15m]) > 0
for: "1h"
labels:
severity: critical
...

26. Deployment Agnostic

27. $ jsonnet -J vendor -S -e
‘std.manifestYamlDoc(
(import "monitoring.jsonnet").prometheusAl
$ jsonnet -J vendor -m dashboards -e
‘(import "monitoring.jsonnet").grafanaDashbo

28. mixins ❤ ksonnet
https://github.com/kausalco/public/prometheus-ksonnet

29. • Prometheus
• Grafana
• Node Exporter
• Kube State Metrics
• Dashboards + alerts in
ConfigMaps

30. Prometheus
Monitoring
Mixins

31. Application Mixin
CoreOS Etcd github.com/coreos/etcd/pull/9640
Hashicorp Consul github.com/kausalco/public/consul-mixin
Hashicorp Vault github.com/grapeshot/vault_exporter/vault-mixin
Kubernetes github.com/kubernetes-monitoring/kubernetes-mixin
Prometheus github.com/kausalco/public/prometheus-ksonnet
Node Exporter TBD

32. Configurable & Extensible
Reusable: easy to install & update
Deployment Agnostic

33. # TODO(twilkie)
#
# - Alerts as a dict, so you can eg add
# URLs to playbook?
# - mixtool to help lint and run mixins.

34. https://goo.gl/JMxwC3
https://goo.gl/JocRm6

35. @davkals@juliusvolz@fredbrancz@sparkprime
Big Thanks To:
@grapeshot_

36. Thanks for listening.
Questions?
Tom Wilkie @tom_wilkie
May 2018
Photo credits:
https://unsplash.com/photos/dmkmrNptMpw
https://unsplash.com/photos/nlMYrApFE7s