Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Prometheus
& Kubernetes
Lessons Learnt
Tom Wilkie, Weaveworks
8th July 2016
Deploying on
Kubernetes
Service-oriented
monitoring
Alerting on
differences
Deploying on
Kubernetes
Not a new topic!
“Monitoring Kubernetes with Prometheus”
- Brian Brazil
“Prometheus and Kubernetes up and running”
- Fabia...
Kubernetes concepts
Pod
Container
Container
Container
Pod
Container
Container
Container
Pod
Container
Container
Container
...
Monitoring Pod
Promtheus
Container
AlertManager
Container
Grafana
Container
Monitoring Pod
Prometheus
Container
AlertManager
Container
Grafana Pod
Grafana
Container
Prometheus
ConfigMap
Grafana
Config...
Grafana Pod
Grafana
Container
Grafana
ConfigMap
Grafana Pod
Grafana
Container
SOCKS Proxy
SOCKS
Proxy
Grafana Pod
Grafana
Container
AuthFE
AuthFE
Container
Demo
Deploying on
Kubernetes
Service-oriented
monitoring
When configuring Prometheus:
• map Kubernetes Services to Prometheus Jobs
• map Kubernetes Pods to Prometheus Instances
Usi...
If you get this
- source_labels: [__meta_kubernetes_role]
action: keep
regex: endpoint
But not all my services export metrics!
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action...
What about exporters?
Memcached Pod
memcached
Container
memcached-exporter
Container
What about exporters?
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i
action: replace
tar...
Service-oriented
monitoring
Alerting on
differences
What I really came
here to talk about…
Kubernetes best-practice
is to version control your
config.
Whats running vs whats
checked in with drift
unless we monitor ...
$ ./kubediff
Usage: kubediff [options] <dir/file>...
Compare yaml files in <dir> to running state
in kubernetes and print ...
Run it as a service:
This will export the exit code as a Prometheus metric
Get alerts from it:
https://github.com/weaveworks/kubediff
ALERT Kubediff
IF max(command_exit_code{job="kubediff"}) != 0
F...
Alerting on
differences
Deploying on
Kubernetes
Service-oriented
monitoring
Alerting on
differences
Thanks!
Questions?
@tom_wilkie
Kubernetes and Prometheus
Upcoming SlideShare
Loading in …5
×

Kubernetes and Prometheus

7,633 views

Published on

Prometheus was recently accepted into the Cloud Native Computing Foundation, making it the second project after Kubernetes to be given their blessing and acknowledging that Prometheus and Kubernetes make an awesome combination. In this talk we'll cover common patterns for running Prometheus on Kubernetes, how to monitor services on Kubernetes, and some cool tips and hacks to ensure you get the most out of your Prometheus + Kubernetes deployment.

Published in: Software
  • Be the first to comment

Kubernetes and Prometheus

  1. 1. Prometheus & Kubernetes Lessons Learnt Tom Wilkie, Weaveworks 8th July 2016
  2. 2. Deploying on Kubernetes Service-oriented monitoring Alerting on differences
  3. 3. Deploying on Kubernetes
  4. 4. Not a new topic! “Monitoring Kubernetes with Prometheus” - Brian Brazil “Prometheus and Kubernetes up and running” - Fabian Reinartz Even example config upstream
  5. 5. Kubernetes concepts Pod Container Container Container Pod Container Container Container Pod Container Container Container Replication Controller / Deployment Service
  6. 6. Monitoring Pod Promtheus Container AlertManager Container Grafana Container
  7. 7. Monitoring Pod Prometheus Container AlertManager Container Grafana Pod Grafana Container Prometheus ConfigMap Grafana ConfigMap
  8. 8. Grafana Pod Grafana Container Grafana ConfigMap
  9. 9. Grafana Pod Grafana Container SOCKS Proxy SOCKS Proxy
  10. 10. Grafana Pod Grafana Container AuthFE AuthFE Container
  11. 11. Demo
  12. 12. Deploying on Kubernetes
  13. 13. Service-oriented monitoring
  14. 14. When configuring Prometheus: • map Kubernetes Services to Prometheus Jobs • map Kubernetes Pods to Prometheus Instances Using relabelling: relabel_configs: - source_labels: [__meta_kubernetes_service_namespace] action: replace regex: ^(.+)$ target_label: namespace replacement: $1 - source_labels: [__meta_kubernetes_service_name] regex: ^(.+)$ target_label: job replacement: $1
  15. 15. If you get this - source_labels: [__meta_kubernetes_role] action: keep regex: endpoint
  16. 16. But not all my services export metrics! - source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape] action: drop regex: false And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.scrape: "false"
  17. 17. What about exporters? Memcached Pod memcached Container memcached-exporter Container
  18. 18. What about exporters? - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i action: replace target_label: __address__ regex: ^(.+)(?::d+);(d+)$ replacement: $1:$2 And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.port: "9150"
  19. 19. Service-oriented monitoring
  20. 20. Alerting on differences
  21. 21. What I really came here to talk about…
  22. 22. Kubernetes best-practice is to version control your config. Whats running vs whats checked in with drift unless we monitor it.
  23. 23. $ ./kubediff Usage: kubediff [options] <dir/file>... Compare yaml files in <dir> to running state in kubernetes and print the differences. This is useful to ensure you have applied all your changes to the appropriate environment. https://github.com/weaveworks/kubediff
  24. 24. Run it as a service: This will export the exit code as a Prometheus metric
  25. 25. Get alerts from it: https://github.com/weaveworks/kubediff ALERT Kubediff IF max(command_exit_code{job="kubediff"}) != 0 FOR 2h
  26. 26. Alerting on differences
  27. 27. Deploying on Kubernetes Service-oriented monitoring Alerting on differences
  28. 28. Thanks! Questions? @tom_wilkie

×