1.
Prometheus
& Kubernetes
Lessons Learnt
Tom Wilkie, Weaveworks
8th July 2016

2.
Deploying on
Kubernetes
Service-oriented
monitoring
Alerting on
differences

3.
Deploying on
Kubernetes

4.
Not a new topic!
“Monitoring Kubernetes with Prometheus”
- Brian Brazil
“Prometheus and Kubernetes up and running”
- Fabian Reinartz
Even example config upstream

5.
Kubernetes concepts
Pod
Container
Container
Container
Pod
Container
Container
Container
Pod
Container
Container
Container
Replication Controller / Deployment
Service

6.
Monitoring Pod
Promtheus
Container
AlertManager
Container
Grafana
Container

7.
Monitoring Pod
Prometheus
Container
AlertManager
Container
Grafana Pod
Grafana
Container
Prometheus
ConfigMap
Grafana
ConfigMap

8.
Grafana Pod
Grafana
Container
Grafana
ConfigMap

9.
Grafana Pod
Grafana
Container
SOCKS Proxy
SOCKS
Proxy

10.
Grafana Pod
Grafana
Container
AuthFE
AuthFE
Container

11.
Demo

12.
Deploying on
Kubernetes

13.
Service-oriented
monitoring

14.
When configuring Prometheus:
• map Kubernetes Services to Prometheus Jobs
• map Kubernetes Pods to Prometheus Instances
Using relabelling:
relabel_configs:
- source_labels: [__meta_kubernetes_service_namespace]
action: replace
regex: ^(.+)$
target_label: namespace
replacement: $1
- source_labels: [__meta_kubernetes_service_name]
regex: ^(.+)$
target_label: job
replacement: $1

15.
If you get this
- source_labels: [__meta_kubernetes_role]
action: keep
regex: endpoint

16.
But not all my services export metrics!
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: drop
regex: false
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.scrape: "false"

17.
What about exporters?
Memcached Pod
memcached
Container
memcached-exporter
Container

18.
What about exporters?
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i
action: replace
target_label: __address__
regex: ^(.+)(?::d+);(d+)$
replacement: $1:$2
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.port: "9150"

19.
Service-oriented
monitoring

20.
Alerting on
differences

21.
What I really came
here to talk about…

22.
Kubernetes best-practice
is to version control your
config.
Whats running vs whats
checked in with drift
unless we monitor it.

23.
$ ./kubediff
Usage: kubediff [options] <dir/file>...
Compare yaml files in <dir> to running state
in kubernetes and print the differences.
This is useful to ensure you have applied all
your changes to the appropriate environment.
https://github.com/weaveworks/kubediff

24.
Run it as a service:
This will export the exit code as a Prometheus metric

25.
Get alerts from it:
https://github.com/weaveworks/kubediff
ALERT Kubediff
IF max(command_exit_code{job="kubediff"}) != 0
FOR 2h

26.
Alerting on
differences

27.
Deploying on
Kubernetes
Service-oriented
monitoring
Alerting on
differences

28.
Thanks!
Questions?
@tom_wilkie