SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Prometheus was recently accepted into the Cloud Native Computing Foundation, making it the second project after Kubernetes to be given their blessing and acknowledging that Prometheus and Kubernetes make an awesome combination. In this talk we'll cover common patterns for running Prometheus on Kubernetes, how to monitor services on Kubernetes, and some cool tips and hacks to ensure you get the most out of your Prometheus + Kubernetes deployment.
Prometheus was recently accepted into the Cloud Native Computing Foundation, making it the second project after Kubernetes to be given their blessing and acknowledging that Prometheus and Kubernetes make an awesome combination. In this talk we'll cover common patterns for running Prometheus on Kubernetes, how to monitor services on Kubernetes, and some cool tips and hacks to ensure you get the most out of your Prometheus + Kubernetes deployment.
4.
Not a new topic!
“Monitoring Kubernetes with Prometheus”
- Brian Brazil
“Prometheus and Kubernetes up and running”
- Fabian Reinartz
Even example config upstream
5.
Kubernetes concepts
Pod
Container
Container
Container
Pod
Container
Container
Container
Pod
Container
Container
Container
Replication Controller / Deployment
Service
6.
Monitoring Pod
Promtheus
Container
AlertManager
Container
Grafana
Container
7.
Monitoring Pod
Prometheus
Container
AlertManager
Container
Grafana Pod
Grafana
Container
Prometheus
ConfigMap
Grafana
ConfigMap
14.
When configuring Prometheus:
• map Kubernetes Services to Prometheus Jobs
• map Kubernetes Pods to Prometheus Instances
Using relabelling:
relabel_configs:
- source_labels: [__meta_kubernetes_service_namespace]
action: replace
regex: ^(.+)$
target_label: namespace
replacement: $1
- source_labels: [__meta_kubernetes_service_name]
regex: ^(.+)$
target_label: job
replacement: $1
15.
If you get this
- source_labels: [__meta_kubernetes_role]
action: keep
regex: endpoint
16.
But not all my services export metrics!
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: drop
regex: false
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.scrape: "false"
17.
What about exporters?
Memcached Pod
memcached
Container
memcached-exporter
Container
18.
What about exporters?
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i
action: replace
target_label: __address__
regex: ^(.+)(?::d+);(d+)$
replacement: $1:$2
And add this to your service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io.port: "9150"
22.
Kubernetes best-practice
is to version control your
config.
Whats running vs whats
checked in with drift
unless we monitor it.
23.
$ ./kubediff
Usage: kubediff [options] <dir/file>...
Compare yaml files in <dir> to running state
in kubernetes and print the differences.
This is useful to ensure you have applied all
your changes to the appropriate environment.
https://github.com/weaveworks/kubediff
24.
Run it as a service:
This will export the exit code as a Prometheus metric
25.
Get alerts from it:
https://github.com/weaveworks/kubediff
ALERT Kubediff
IF max(command_exit_code{job="kubediff"}) != 0
FOR 2h