Download as PDF, PPTX
![!26
Autodiscover (new in 6.2)
Watch Kubernetes events and react to changes
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: kube-system
config:
- type: docker
containers.ids:
- ${data.kubernetes.container.id}
exclude_lines: ["^s+[-`('.|_]"] # drop asciiart](https://image.slidesharecdn.com/monitoringkubernetesatscalemonicasarbu-180622115832/75/OSDC-2018-Monitoring-Kubernetes-at-Scale-by-Monica-Sarbu-26-2048.jpg)
Uploaded byNETWAYS
OSDC 2018 | Monitoring Kubernetes at Scale by Monica Sarbu
The document discusses monitoring Kubernetes at scale using the Elastic Stack, including the use of various Beats for logging and metrics. It outlines how to monitor Kubernetes clusters and services using Metricbeat and Filebeat, along with the use of metadata processors to enhance logs and metrics collection. Additionally, it introduces deployment strategies, dynamic configuration features like autodiscover, and the importance of curated visualizations for effective monitoring.
More Related Content
Similar to OSDC 2018 | Monitoring Kubernetes at Scale by Monica Sarbu
OSDC 2018 | Monitoring Kubernetes at Scale by Monica Sarbu
- 1. Monica Sarbu Team Lead,Beats and Logstash Monitoring Kubernetes at Scale
- 2.
- 3.
- 4. A single Beatfor each use case Beats family libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat
- 5. Growing Beats community !5 50MCumulativedownloads 3Years
- 6.
- 7.
- 8. !8 With containers architecture, everything isa moving target
- 9.
- 10.
- 11. !11 Monitor all thethings! ✓ Monitor services running in Kubernetes ✓ Collect application logs, metrics, traces
- 12.
- 13. !13 Monitor Kubernetes cluster Viathe Kubernetes module in Metricbeat Node metrics, from kubelet ✓ pod ✓ node ✓ system ✓ container ✓ volume State metrics, from kube-state-metrics ✓ state_container ✓ state_deployment ✓ state_node ✓ state_pod ✓ state_replicated ✓ state_statefulset Kubernetes events ✓ event Kubernetes API server ✓ apiserver
- 14. !14 Monitor services runninginside Kubernetes Metricbeat Filebeat Node n Logs Metrics Nginx
- 15. !15 Collect service logs filebeat.prospectors: -type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch: 9200rn","stream":"stdout","time":"2018-02-11T23:29:19. 236692181Z"} via Docker input in Filebeat
- 16. !16 Metadata processors Enrich eventswith useful metadata to correlate logs, metrics & traces • cloud.availability_zone • cloud.region • cloud.instance_id • cloud.machine_type • cloud.project_id • cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata
- 17. !17 Metadata processors Example { "@timestamp": "2017-11-17T00:53:33.759Z", "message":"2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "project_id": "carlosperez-163008", "provider": "gce" } }, }
- 18. !18 Understand the servicelogs Via Filebeat modules • Apache • Auditd • Icinga • IIS • Kafka • Logstash • MySQL • Nginx • Osquery • Postgresql • Redis • System • Traffic ✓Filebeat modules - Out of the box - Pipeline configuration to parse logs in Elasticsearch Ingest Node - Dashboards included - Come with ML definitions
- 19. !19 Collect service metrics ViaMetricbeat modules Data stores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike • Graphite Queues • Kafka • Redis • RabbitMQ Caches • Memcached Custom apps • JMX/Jolokia • PHP-FPM • Golang Web servers • Apache • Nginx Other • HAProxy • Zookeeper
- 20. !20 Collect APM traces WithElastic APM, previously Opbeat ✓ apm-server based on libbeat ✓Collects traces from agents ✓ Benefits from metadata processors ✓Agents: Python, Node.js, Ruby(Beta), RUM(Beta), Java (Alpha), Go (wip) ✓ Apache 2 license
- 21. !21 Monitor applications instrumentedwith Prometheus Via the Prometheus module in Metricbeat Node 1 Metricbeat Node 2 Metricbeat Node n Metricbeat App App App pull pull pull
- 22.
- 23. !23 Kubernetes deployment Deploy Metricbeatand Filebeat as DaemonSets Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet
- 24. !24 Sample manifests files ToDeploy Filebeat and Metricbeat as DaemonSets $ curl -L -O https:// raw.githubusercontent.com/elastic/beats/ master/deploy/kubernetes/filebeat- kubernetes.yaml $ kubectl create -f filebeat-kubernetes.yaml
- 25.
- 26. !26 Autodiscover (new in6.2) Watch Kubernetes events and react to changes filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: kube-system config: - type: docker containers.ids: - ${data.kubernetes.container.id} exclude_lines: ["^s+[-`('.|_]"] # drop asciiart
- 27. !27 Hints based autodiscover(new in 6.3) Invert control of monitoring settings annotations: co.elastic.logs/module: nginx co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error co.elastic.metrics/module: nginx co.elastic.metrics/hosts: “${data.host}:8080" filebeat.autodiscovery: providers: - type: kubernetes hints.enabled: true
- 28.
- 29.
- 30. !30 High level overview Groupby namespace, service
- 31.
- 32.
- 33.
- 34.