This document summarizes a presentation about monitoring challenges with Kubernetes and containers. The presentation covered: - Monitoring 101 concepts like categorizing issues and focusing on symptoms to find root causes. - Challenges of monitoring dynamic container infrastructure like increased operational complexity from many ephemeral containers and scaling to monitor thousands of metrics. - Approaches for container monitoring including taking a service-centric rather than host-centric view and using tags to query metrics flexibly across infrastructure attributes. - Features of Datadog for container monitoring like automatic discovery of containers and metadata, off-the-shelf integrations, and dashboards for visualizing metrics and alerts.

1. Monitoring In Motion
Challenges in Monitoring Kubernetes & Containers
Cloud Native SF Meetup
Feb 25, 2016
Ilan Rabinovitch
Director, Community
Datadog

2. About Me
● Long time Datadog user.
● Prior to Datadog built automation and
monitoring tooling at Ooyala and
Edmunds.com
● SCALE and TXLF Co-Founder
Ilan Rabinovitch
Datadog
ilan@datadoghq.com
@irabinovitch

3. Agenda
• Monitoring 101 - Crash Course
• Challenges in Monitoring Dynamic Infrastructure
• Demo Time
• Questions?

4. Monitoring Everything

6. @honest_update on Twitter

7. Quick Overview of Datadog
• Monitoring for modern applications.
• Time series storage of metrics and events.
• Trending, alerting and anomaly detection.
• Hundreds of integrations out of the box.

8. Monitoring 101: Categorization
More at: http://goo.gl/t1Rgcg

10. Monitoring 101: Focus on symptoms
More at: http://goo.gl/t1Rgcg

11. Recurse until you find root cause.
More at: http://goo.gl/t1Rgcg

12. Container Monitoring Challenges

13. https://www.datadoghq.com/docker-adoption/

16. Operational Complexity
•Average containers per host: N
(N=4, 10/2015)
•N-times as many “hosts” to manage
•Affects everything

17. Operational Complexity: Scale
100
instances
400
containers

18. Operational Complexity: Scale
160
metrics per host
640
metrics per host

19. Operational Complexity: Scale
100
instances
64,000
metrics

21. Host Centric vs Service Centric

22. Host Centric vs Service Centric

23. Query Based Monitoring
…
… …

24. •Use tags, labels, etc on your hosts and metrics.
•Pull in existing labels from your infrastructure (Region, Docker Images,
K8S Tags..)
Query Based Monitoring
By using tags, auto-adapt!

25. Where is my application running ?
What’s the total throughput of App X ?
What’s its response time per tag ? (pod, version, DC)
What’s the distribution of 5xx from Nginx per pod ?

26. Auto Discovery

27. Docker API Kubelet API
Monitoring Agent
Container
A O A O
A
O
Application Container
Off-The-Shelf Application (Redis,
PostgreSQL, …)
Containers List
Metadata
Additional Metadata
(Pod names, RC, …)
Config Backend
Integration Configurations
Host Level
Metrics

28. Some Pictures
Dashboards and Metrics Alerts
Sharing

29. Demo time