Kubernetes is changing the game in the data centre, but also in the monitoring and troubleshooting landscape. Static tools and vertically scalable TSDBs are no longer fit for the job. Large-scale dynamic infrastructures require scalable dynamic monitoring. This talk presents how the Elastic Stack collects logs, metrics, and APM traces from the applications running in Kubernetes: – Collect application logs, metrics and enhance them with Kubernetes metadata – Collect application metrics from Prometheus endpoints – Collect Kubernetes metrics – Collect application performance traces (APM) – Autodiscover new pods and monitor them based on their type – Control the monitoring via Kubernetes annotations – Use Kibana as a single looking glass to visualize the collected data

1. Monica Sarbu
Team Lead, Beats and Logstash
Monitoring Kubernetes at Scale

2. Monitoring Kubernetes
with Elastic Stack

3. Elastic Stack

4. A single Beat for each use case
Beats family
libbeat
Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat

5. Growing Beats community
!5
50MCumulative downloads
3Years

6. !6

7. Monitoring challenges

8. !8
With containers
architecture,
everything is a moving
target

9. !9
High level
overview

10. !10

11. !11
Monitor all the things!
✓ Monitor services running in
Kubernetes
✓ Collect application logs,
metrics, traces

12. Monitoring with Beats

13. !13
Monitor Kubernetes cluster
Via the Kubernetes module in Metricbeat
Node metrics, from kubelet
✓ pod
✓ node
✓ system
✓ container
✓ volume
State metrics, from kube-state-metrics
✓ state_container
✓ state_deployment
✓ state_node
✓ state_pod
✓ state_replicated
✓ state_statefulset
Kubernetes events
✓ event
Kubernetes API server
✓ apiserver

14. !14
Monitor services running inside Kubernetes
Metricbeat Filebeat
Node n
Logs
Metrics
Nginx

15. !15
Collect service logs
filebeat.prospectors:
- type: docker
containers.ids:
- ‘*’
Parse and ship /var/lib/docker/containers/*/*.log:
{"log":"INFO elasticsearch/client.go:145 Elasticsearch
url:http://elasticsearch:
9200rn","stream":"stdout","time":"2018-02-11T23:29:19.
236692181Z"}
via Docker input in Filebeat

16. !16
Metadata processors
Enrich events with useful metadata to correlate logs, metrics & traces
• cloud.availability_zone
• cloud.region
• cloud.instance_id
• cloud.machine_type
• cloud.project_id
• cloud.provider
• docker.container.id
• docker.container.image
• docker.container.name
• docker.container.labels
• kubernetes.pod.name
• kubernetes.namespace
• kubernetes.labels
• kubernetes.annotations
• kubernetes.container.name
• kubernetes.container.image
add_cloud_metadata add_docker_metadata add_kubernetes_metadata

17. !17
Metadata processors
Example
{
"@timestamp": "2017-11-17T00:53:33.759Z",
"message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0",
"kubernetes": {
"pod": {
"name": "filebeat-vqf85"
},
"container": {
"name": "filebeat"
},
"namespace": "kube-system",
"labels": {
"k8s-app": "filebeat",
"kubernetes.io/cluster-service": "true"
}
},
"meta": {
"cloud": {
"instance_id": "6959555125944564951",
"instance_name": "gke-demo-default-pool-6b42dcb3-z2x7",
"machine_type": "projects/865493543029/machineTypes/n1-standard-1",
"availability_zone": "projects/865493543029/zones/europe-west1-b",
"project_id": "carlosperez-163008",
"provider": "gce"
}
},
}

18. !18
Understand the service logs
Via Filebeat modules
• Apache
• Auditd
• Icinga
• IIS
• Kafka
• Logstash
• MySQL
• Nginx
• Osquery
• Postgresql
• Redis
• System
• Traffic
✓Filebeat modules
- Out of the box
- Pipeline configuration to parse
logs in Elasticsearch Ingest
Node
- Dashboards included
- Come with ML definitions

19. !19
Collect service metrics
Via Metricbeat modules
Data stores
• MySQL
• PostgreSQL
• MongoDB
• Couchbase
• Aerospike
• Graphite
Queues
• Kafka
• Redis
• RabbitMQ
Caches
• Memcached
Custom apps
• JMX/Jolokia
• PHP-FPM
• Golang
Web servers
• Apache
• Nginx
Other
• HAProxy
• Zookeeper

20. !20
Collect APM traces
With Elastic APM, previously Opbeat
✓ apm-server based on libbeat
✓Collects traces from agents
✓ Benefits from metadata
processors
✓Agents: Python, Node.js,
Ruby(Beta), RUM(Beta), Java
(Alpha), Go (wip)
✓ Apache 2 license

21. !21
Monitor applications instrumented with Prometheus
Via the Prometheus module in Metricbeat
Node 1
Metricbeat
Node 2
Metricbeat
Node n
Metricbeat
App App App
pull pull pull

22. Deployment strategies

23. !23
Kubernetes deployment
Deploy Metricbeat and Filebeat as DaemonSets
Node 1
Metricbeat
Filebeat
Node 2
Metricbeat
Filebeat
Node n
Metricbeat
Filebeat
Filebeat
DaemonSet
Metricbeat
DaemonSet

24. !24
Sample manifests files
To Deploy Filebeat and Metricbeat as DaemonSets
$ curl -L -O https://
raw.githubusercontent.com/elastic/beats/
master/deploy/kubernetes/filebeat-
kubernetes.yaml
$ kubectl create -f filebeat-kubernetes.yaml

25. Dynamic configuration

26. !26
Autodiscover (new in 6.2)
Watch Kubernetes events and react to changes
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: kube-system
config:
- type: docker
containers.ids:
- ${data.kubernetes.container.id}
exclude_lines: ["^s+[-`('.|_]"] # drop asciiart

27. !27
Hints based autodiscover (new in 6.3)
Invert control of monitoring settings
annotations:
co.elastic.logs/module: nginx
co.elastic.logs/fileset.stdout: access
co.elastic.logs/fileset.stderr: error
co.elastic.metrics/module: nginx
co.elastic.metrics/hosts: “${data.host}:8080"
filebeat.autodiscovery:
providers:
- type: kubernetes
hints.enabled: true

28. !28

29. Curated visualizations

30. !30
High level overview
Group by namespace,
service

31. !31

32. !32
Live logs

33. !33

34. Questions?