In this session we will learn how to use Elastic Observability to get a better understanding of our Kubernetes clusters and the applications that run on them. Join us to learn new ways to get your data into Elastic with Elastic Agent and discuss some of the best practices to shape up your Kubernetes monitoring solution. Learn about how ingest configuration can be done through Kubernetes manifests with the option to centrally manage it Deep dive into node scope vs cluster scope metrics collection and what you can learn from each. Understand how dynamic workload monitoring (aka autodiscovery) enables application teams to decide what needs to be monitored at workload level Automatic APM instrumentation, to automatically attach an APM Agent and collect APM traces from running Pods. Quick intro into watchers/alerts and how those can help us with issues’ detection. See all these combined with a hands on demo including logs and metrics collection, dynamic workload monitoring, APM instrumentation, and synthetics monitoring for applications. Furthermore, we will convert collected data into actionable observability with alerts and machine learning. Last but not least, learn about our planned next steps and discuss your feedback with us. Speakers: Christos Markou | Senior Software Engineer @Elastic & Miguel Luna | Principal Product Manager @Elastic

1. 1
Elastic Meetup Amsterdam | April 2023
Miguel Luna | Product @Elastic
Christos Markou | Engineering @Elastic
Deep dive into Kubernetes monitoring
with Elastic Observability

2. About us…
Christos Markou
Senior Software Engineer
Miguel Luna
Principal Product Manager
Elastic Cloud Native Observability

3. What does Kubernetes have to do with hipsters?

4. A story about beer Kubernetes

5. Goal: Get to Union Square

6. ⬅️ Queens midtown
tunnel
⬇️ Down 5th Avenue
🛑 Union square hotel
3 STEPS!

7. cab$ take queens-midtown-tunnel
drive down 5th-avenue
stop union square

10. Goal: Get to Union Square

12. TL;DL:
We ❤️ the resilience that
Kubernetes automation brings to
our work.

13. Problem Solved…?

14. Yes! Kubernetes brings challenges (like observing it)
● Dynamic and ephemeral environment
● A new meaning for scale
● Distributed nature of Kubernetes
● Data sprawl across different tools
● Interpreting Kubernetes signals requires expertise
● The rise of managed Kubernetes

15. Observing
with

16. Store, Search, &
Analyze
Visualize &
Manage
Ingest
Elastic Stack
Kibana
Elasticsearch
Beats/Elastic-Agent Logstash
Elastic Stack

17. Getting your K8s data into Elastic
• similar functionality to Beats for log collection and host monitoring
• Elastic Agent has some distinct advantages over Beats
• Easier to deploy and manage
• Easier to configure
• Central management
Elastic Agent

18. Ingest configuration
Type of integration
Shipper to use
Metrics endpoint
Integration specific settings

19. Elastic Integrations
https://docs.elastic.co/en/integrations/redis
https://github.com/elastic/integrations

20. Centrally manage configurations

21. Configuring Elastic Agent (managed by user)
• Standalone Elastic Agents are manually configured and managed
locally on the systems where they are installed.
• They are useful when you are not interested in centrally managing
agents in Fleet, either due to your company’s security requirements,
or because you prefer to use another configuration management
system.

22. From UI to GitOps

23. Elastic Agent on Kubernetes
Filebeat
Daemonset
Filebeat
Daemonset
Filebeat
Daemonset
Node_1 Node_2 Node_3
Elastic Agent
Pod
runs as Deamonset (one Pod per node) on a k8s cluster
Elastic Agent
Pod
Elastic Agent
Pod

24. Inputs
• kubernetes-cluster-metrics (using leaderelection)
• kubernetes-node-metrics (node’s kubelet API)
• system/metrics (from underlying node using system package)
• container-logs (using k8s dynamic provider)
• system-logs (from underlying node using system package)
• uptime monitoring
• redis/metrics (using k8s dynamic provider + hints)
• APM data

25. Dynamic workload discovery
• Conditions based autodiscover
• Hints annotations based autodiscover

26. Conditions based autodiscover
condition: ${kubernetes.labels.app} == ‘redis’

27. Hints annotations based autodiscover

28. APM instrumentation
● An implementation of k8s admission control webhook, that enables
automatic attachment of the Elastic APM agent to application pods.
● The registered MutatingAdmissionWebhook intercepts requests to the
Kubernetes API server and executes the mutating admission control
webhook prior to persistence of the object, but after the request is
authenticated and authorized.
This allows the mutation of the originally submitted request.

29. Your K8s data is in Elastic, now what?
Data collection into one single place, following common schema will
allows us
to convert these data into actionable observability rules:
• Latency
• Resource saturation
• Common errors

30. Alerting (through watchers)
A Watcher is an Elasticsearch feature that you can use to create actions
based on conditions, which are periodically evaluated using queries on your
data. Watches are helpful for analyzing mission-critical and business-critical
streaming data.

31. Alerting (through watchers)
https://github.com/elastic/integrations/blob/main/packages/kubernetes/docs/pod-terminated-oomkilled-alert.md

32. Alerting (through ML)

33. Demo 🤞

34. LET’S KEEP SHAPING ELASTIC
TOGETHER!

35. WE WOULD TO KEEP HEARING FROM YOU
https://discuss.elastic.co/c/beats
https://github.com/elastic/beats
https://discuss.elastic.co/c/elastic-stack/elastic-
agent
https://github.com/elastic/elastic-agent
https://github.com/elastic/integrations

36. Q&A

37. Elastic’s contribution of
Elastic Common Schema (ECS) to
OpenTelemetry (OTel)
April 18, 2023

38. ECS and OTel SemConv* Convergence
ECS
Security Events
Logs Metrics Traces
Resources
OTel SemConv*
Logs Metrics Traces
A schema that includes both Observability and Security
New OTel
common schema
Logs Metrics Traces
Security Events
Resources
Resources
ECS main
contributions
* OTel SemConv = OpenTelemetry Semantic Convention (OTel’s schema definition)

39. How a common schema helps: current state
Reduced visibility and harder root cause analysis
Where are you operationally?
Where are you trending?
Are you meeting business objectives?
Backend
OTel
Agent/S
DK
Elastic
Agents
Infra
Frontend
Dev process
src:10.42.42.42
OR client_ip:10.42.42.42
OR
apache2.access.remote_ip:
10.42.42.42
OR
context.user.ip:10.42.42.42
OR src_ip:10.42.42.42
Example: IP definition of a specific user end point
w/o
COMMON
SCHEMA

40. How a common schema helps: future state
Backend
OTel
Agent/S
DK
Elastic
Agents
Infra
Frontend
Dev process
Example: IP definition of a specific user end point
Where are you operationally?
Where are you trending?
Are you meeting business objectives?
src:10.42.42.42
OR client_ip:10.42.42.42
OR
apache2.access.remote_ip:
10.42.42.42
OR
context.user.ip:10.42.42.42
OR src_ip:10.42.42.42
Where are you operationally?
Where are you trending?
Are you meeting business objectives?
w/
COMMON
SCHEMA
source.ip:10.42.42.42
Simplified visibility and root cause analysis

41. Value of the new common schema
Better visibility and
root cause analysis
for operations and
security teams
Improved
collaboration
between
observability and
security
OTel is the
open standard for
observability and
security telemetry

42. Elastic’s native OpenTelemetry support
OTel
Collector
App
Code
Microservices
OTLP
Agent/SDK
Elastic Observability
Kibana
APM Server
Elasticsearch
OTLP
OR
Agent/SDK
App Code
Microservices
Elastic APM agents and OTel
coexist, delivering full APM
visibility and functionality enabling
customers migrate to an OTEL
NO Elastic based OTel Agent
needed

43. Learn more about this