This document provides an agenda and overview of a presentation on using the Elastic Stack to gain observability of a Kubernetes environment. The presentation covers introducing the Elastic Stack and its components, challenges of monitoring and logging Kubernetes, and a demonstration of using Filebeat, Metricbeat, and APM tools to collect logs, metrics, and traces from a Kubernetes cluster.

1. 1
Ted Jung
Consulting Architect
Observable Kubernetes
with Elastic Stack

2. 2
Ted, Jung
Consulting Architect @Elastic

3. 3
• Intro to Elastic Stack
• Kubernetes (challenges, visibility)
• Logging and Monitoring Kubernetes
• APM tracing
• Demo
Agenda

4. 44
Intro to Elastic Stack
● Elastic stack
● Elastic Elasticsearch Service

5. 5
Elastic Stack
Store, Search, & AnalyzeElasticsearch
Visualize & ManageKibana
Beats IngestLogstash
SaaS
Elasticsearch
Service
Self Managed
Elastic Cloud
Enterprise
Standalone
Deployment
The Elastic Stack

6. 6
US West
(Oregon)
US West
(N. Cal)
US East (N.
Virginia)
EU
(Ireland)
EU
(Germany)
S. America
(East)
Asia Pacific
(Sydney)
Asia Pacific
(Tokyo)
Asia Pacific
(Singapore)
Elasticsearch
The Official Hosted Elasticsearch & Kibana Solution
AWS 9Regions
VM or On-Prem VM or On-Prem
The Self Managed Elasticsearch & Kibana Solution

7. 77
Kubernetes
● Challenges
● Visibility

8. 8
Kinds of Kubernetes

9. 9
Container
Runtime
Kubernetes is…
§ Kubernetes is complicated
§ Kubernetes is extremely dynamic, which makes monitoring & logging very challenging
§ All components provide metrics
§ On top of all this there are a ton of different providers

10. 10
Kubernetes Visibility Challenges…

11. 11
Three Pillars of Observability
Elastic brings a holistic solution
for Kubernetes observability in one platform.

12. 1212
Logging
● Cluster level logging
● Service logging (eg, nginx, mysql)
● Custom application logging

13. 13
• Need for a logging solution
– Kubernetes does not have a native solution
– kubectl logs is too hard for large clusters
• Cluster-level logging
– Logs have separate storage and lifecycle independent of nodes, pods and containers
– Kubernetes provides no native storage solution for log data
• Application-level logging
– Complicated
– Packaged applications (eg. nginx)
– Custom applications
Kubernetes Logging

14. 14
kubectl create -f filebeat-kubernetes.yaml
Cluster
Service
Application
Enriched Logging & Better Log Collection
with Filebeat
• Cluster level logging
• Service level logs (eg. Nginx logs)
– Auto-discovery
• Application logs
– Cluster level logging (stdout, stderr)
– Custom log file

15. 15
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.container.image: "nginx"
config:
- module: nginx
access: # For nginx access log
prospector:
type: docker
containers.ids:
- "${data.kubernetes.container.id}"
Filebeat Auto-Discovery

16. 16
• Kubernetes module
• Monitors pods and services
– Cluster, pod & container metrics
– Application metrics through auto-discovery
(eg. Nginx)
• Metrics sources - Cover them ALL
– Kubelet (heapster, cAdvisor)
– kube-state-metric
– Kubernetes events
– Prometheus module (beta)
• Curated infra UI
– Dedicated Kibana app
Comprehensive Metrics Collection with
Metricbeat

17. 17
Metricbeat Modules
Simplify collection and visualization of common metrics
● Aerospike module
● Apache module
● Ceph module
● Couchbase module
● Docker module
● Dropwizard module
● Elasticsearch module
● Etcd module
● Golang module
● Graphite module
● HAProxy module
● HTTP module
● Jolokia module
● Kafka module
● Kibana module
● Kubernetes module
● kvm module
● Logstash module
● Memcached module
● MongoDB module
● Munin module
● MySQL module
● Nginx module
● PHP_FPM module
● PostgreSQL module
● Prometheus module
● RabbitMQ module
● Redis module
● System module
● uwsgi module
● vSphere module
● Windows module
● ZooKeeper module

18. 18
Filebeat Modules
Simplify collection and visualization of common metrics

19. 19
Beats Family

20. 2020
Tracing
● Elastic APM

21. 21
Application Monitoring with Elastic APM
Adding end-user experience and application-level monitoring to the stack
RUM

22. 2222
Observable Kubernetes
Elastic Stack:
Three Pillars of Observability in One Platform
● Logging
● Metrics
● APM Tracing

23. 2323
Demo

24. 24
filebeat
metricbeat
packetbeat
heartbeat
Visualization
indexing
Demo Env
Store

25. 25
Master&WorkerNode 1
Metricbeat
Filebeat
Worker Node 2
Metricbeat
Filebeat
Worker Node 3
Metricbeat
Filebeat
Heartbeat, Packetbeat
Kubernetes Daemonset for Beat

26. 26
Simple deployment templates to gain insights faster
One-click upgrades of the Elastic Stack
Centrally manage all your deployments
Enhanced security and performance for all deployments
Elastic Elasticsearch Service

27. 27
Howto deploy
Create deployment
Cloud(AWS)
Region
Demo Env

28. 28
Demo Env

29. 29
Access URL: http://**.**.**.**:?????/
Demo Application

30. 30