![If you get this
- source_labels: [__meta_kubernetes_role]
action: keep
regex: endpoint](https://image.slidesharecdn.com/kubernetesandprometheus-160709093102/85/kubernetes-and-prometheus-15-320.jpg?cb=1468056727)
![But not all my services export metrics!
- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]
action...](https://image.slidesharecdn.com/kubernetesandprometheus-160709093102/85/kubernetes-and-prometheus-16-320.jpg?cb=1468056727)
![$ ./kubediff
Usage: kubediff [options] <dir/file>...
Compare yaml files in <dir> to running state
in kubernetes and print ...](https://image.slidesharecdn.com/kubernetesandprometheus-160709093102/85/kubernetes-and-prometheus-23-320.jpg?cb=1468056727)
Just for you: FREE 60-day trial to the world’s largest digital library.
The SlideShare family just got bigger. Enjoy access to millions of ebooks, audiobooks, magazines, and more from Scribd.
Cancel anytime.More Related Content
Related Books
Free with a 14 day trial from Scribd
Kubernetes and Prometheus
- 1. Prometheus & Kubernetes Lessons Learnt Tom Wilkie, Weaveworks 8th July 2016
- 2. Deploying on Kubernetes Service-oriented monitoring Alerting on differences
- 3. Deploying on Kubernetes
- 4. Not a new topic! “Monitoring Kubernetes with Prometheus” - Brian Brazil “Prometheus and Kubernetes up and running” - Fabian Reinartz Even example config upstream
- 5. Kubernetes concepts Pod Container Container Container Pod Container Container Container Pod Container Container Container Replication Controller / Deployment Service
- 6. Monitoring Pod Promtheus Container AlertManager Container Grafana Container
- 7. Monitoring Pod Prometheus Container AlertManager Container Grafana Pod Grafana Container Prometheus ConfigMap Grafana ConfigMap
- 8. Grafana Pod Grafana Container Grafana ConfigMap
- 9. Grafana Pod Grafana Container SOCKS Proxy SOCKS Proxy
- 10. Grafana Pod Grafana Container AuthFE AuthFE Container
- 11. Demo
- 12. Deploying on Kubernetes
- 13. Service-oriented monitoring
- 14. When configuring Prometheus: • map Kubernetes Services to Prometheus Jobs • map Kubernetes Pods to Prometheus Instances Using relabelling: relabel_configs: - source_labels: [__meta_kubernetes_service_namespace] action: replace regex: ^(.+)$ target_label: namespace replacement: $1 - source_labels: [__meta_kubernetes_service_name] regex: ^(.+)$ target_label: job replacement: $1
- 15. If you get this - source_labels: [__meta_kubernetes_role] action: keep regex: endpoint
- 16. But not all my services export metrics! - source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape] action: drop regex: false And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.scrape: "false"
- 17. What about exporters? Memcached Pod memcached Container memcached-exporter Container
- 18. What about exporters? - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i action: replace target_label: __address__ regex: ^(.+)(?::d+);(d+)$ replacement: $1:$2 And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.port: "9150"
- 19. Service-oriented monitoring
- 20. Alerting on differences
- 21. What I really came here to talk about…
- 22. Kubernetes best-practice is to version control your config. Whats running vs whats checked in with drift unless we monitor it.
- 23. $ ./kubediff Usage: kubediff [options] <dir/file>... Compare yaml files in <dir> to running state in kubernetes and print the differences. This is useful to ensure you have applied all your changes to the appropriate environment. https://github.com/weaveworks/kubediff
- 24. Run it as a service: This will export the exit code as a Prometheus metric
- 25. Get alerts from it: https://github.com/weaveworks/kubediff ALERT Kubediff IF max(command_exit_code{job="kubediff"}) != 0 FOR 2h
- 26. Alerting on differences
- 27. Deploying on Kubernetes Service-oriented monitoring Alerting on differences
- 28. Thanks! Questions? @tom_wilkie