The document discusses the use of JSONNet for packaging Prometheus monitoring mixins, encompassing dashboards and alerts that are configurable, extensible, and deployment agnostic. It emphasizes the need for reusable components and provides examples of how to structure monitoring configurations using JSONNet, including setup for Kubernetes monitoring. The document highlights various tools and resources, including Grafana and JSONNet Bundler, necessary for implementing and managing these mixins.

1. Prometheus
Monitoring
Mixins
Using Jsonnet to Package
Together Dashboards and Alerts
Tom Wilkie @tom_wilkie
May 2018

4. sum(container_memory_working_set_bytes{
id=“/“,
kubernetes_io_hostname=~”^$Node$”
})
/
sum(machine_memory_bytes{
kubernetes_io_hostname=~”^$Node$”
})

5. Dashboards and alerts should
not be opinionated about labels -
this should be configuration.

8. Configurable & Extensible
Reusable: easy to install & update
Deployment Agnostic

9. Configurable &
Extensible

10. container_memory_usage_bytes{
namespace="$namespace", pod_name="$pod"}

11. • YAML & JSON
• envsubst, m4, jinja etc
• Go, Python

12. from grafanalib.core import *
dashboard = Dashboard(
title="Frontend Stats",
rows=[
Row(panels=[
Graph(
title="Frontend QPS",
dataSource='My Prometheus',
targets=[
Target(
expr='sum(irate(nginx_http_requests_total{job="de
legendFormat="1xx",
refId='A',
),
...
https://github.com/weaveworks/grafanalib

13. jsonnet
A data templating language for app and
tool developers.
A simple extension of JSON.
http://jsonnet.org/ @sparkprime

15. Jsonnet Demo

16. Configurable &
Extensible (II)

17. {
_config+:: {...},
grafanaDashboards+:: {
“dashboard-name.json”: {...},
},
prometheusAlerts+:: [...],
prometheusRules+:: [...],
}

18. {
_config+:: {
// Selectors are inserted between {} in Prome
kubeStateMetricsSelector:
'job="default/kube-state-metrics"',
...
},
}

19. {
prometheusAlerts+:: {
groups+: [{
name: 'kubernetes-apps',
rules: [{
alert: 'KubePodCrashLooping',
'for': '1h',
expr: |||
rate(kube_pod_container_status_restarts
{%(kubeStateMetricsSelector)s}[15m])
> 0
||| % $._config,
...

20. groups:
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
expr: |
rate(kube_pod_container_status_restarts
{job=“default/kube-state-metrics”}[15
> 0
for: 1h
...

21. Reusable

22. jsonnet-bundler
A jsonnet package manager.
https://github.com/jsonnet-bundler/jsonnet-bundler

23. $ go get github.com/jsonnet-bundler/jsonnet-bundle
$ jb init
$ jb install github.com/kubernetes-monitoring/kube
Cloning into 'vendor'...
$ $ tree vendor
vendor
├── grafonnet
│ └── ...
└── kubernetes-mixin
└── ...

24. $ cat monitoring.jsonnet
local kubernetes = import "kubernetes-mixin/mixin.
kubernetes {
_config+:: {
cadvisorSelector: 'job="cadvisor"',
kubeletSelector: 'job="kubelet"',
kubeStateMetricsSelector: 'job="kube-state-met
nodeExporterSelector: 'job="node-exporter"',
notKubeDnsSelector: 'job!="kube-dns"',
},
}

25. $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((impo
groups:
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
annotations:
message: "{{ $labels.namespace }}/{{ $labels.p
expr: |
rate(kube_pod_container_status_restarts_total
{job="kube-state-metrics"}[15m]) > 0
for: "1h"
labels:
severity: critical
...

26. Deployment Agnostic

27. $ jsonnet -J vendor -S -e
‘std.manifestYamlDoc(
(import "monitoring.jsonnet").prometheusAl
$ jsonnet -J vendor -m dashboards -e
‘(import "monitoring.jsonnet").grafanaDashbo

28. mixins ❤ ksonnet
https://github.com/kausalco/public/prometheus-ksonnet

29. • Prometheus
• Grafana
• Node Exporter
• Kube State Metrics
• Dashboards + alerts in
ConfigMaps

30. Prometheus
Monitoring
Mixins

31. Application Mixin
CoreOS Etcd github.com/coreos/etcd/pull/9640
Hashicorp Consul github.com/kausalco/public/consul-mixin
Hashicorp Vault github.com/grapeshot/vault_exporter/vault-mixin
Kubernetes github.com/kubernetes-monitoring/kubernetes-mixin
Prometheus github.com/kausalco/public/prometheus-ksonnet
Node Exporter TBD

32. Configurable & Extensible
Reusable: easy to install & update
Deployment Agnostic

33. # TODO(twilkie)
#
# - Alerts as a dict, so you can eg add
# URLs to playbook?
# - mixtool to help lint and run mixins.

34. https://goo.gl/JMxwC3
https://goo.gl/JocRm6

35. @davkals@juliusvolz@fredbrancz@sparkprime
Big Thanks To:
@grapeshot_

36. Thanks for listening.
Questions?
Tom Wilkie @tom_wilkie
May 2018
Photo credits:
https://unsplash.com/photos/dmkmrNptMpw
https://unsplash.com/photos/nlMYrApFE7s