Downloaded 73 times
Uploaded byDatadog
Monitoring Docker containers - Docker NYC Feb 2015
The document discusses the complexities of monitoring Docker containers at scale, highlighting the evolution of containers, their operational challenges, and the necessity for effective monitoring strategies. It emphasizes the importance of layered monitoring and using tags and queries for efficient oversight of container metrics. Key takeaways include the increased operational complexity introduced by Docker and the need for a structured approach to manage it effectively.
More Related Content
Similar to Monitoring Docker containers - Docker NYC Feb 2015
Monitoring Docker containers - Docker NYC Feb 2015
- 1. Monitoring and Running DockerContainers at Scale Docker NYC Meetup February 25th, 2015
- 2. @alq — CTOat Datadog
- 3. Datadog • Monitoring service •Made for the cloud • Aggregates everything • Support for Docker (since 1.0)
- 4. Goal of thistalk Rethink the monitoring of Docker containers
- 5. Agenda 1.A (very) briefhistory of containers 2.Operational complexity 3.Monitoring Docker effectively 4.Demo
- 6. A brief historyof containers
- 7. Containers in anutshell • Been around for a long time – jails, zones, cgroups • No full-virtualization overhead • Used for runtime isolation (e.g. jails) • Docker is an Escape from Dependency Hell
- 8. Escape from dependencyhell a.out shared libs packages omnibus Docker == ?
- 9. Mini-host or über-process? ProcessContainer Host Spec Source Dockerfile Kickstart On disk .TEXT /var/lib/docker / In memory PID Container ID Hostname In the network Socket veth* eth* Runtime context server core host data center
- 10.
- 11.
- 12. Combinatorial multiplication Hardware OS Off-the-shelf Your Application Hardware Hypervisor Off-the- shelf App OSOS Off-the- shelf App Hardware Hypervisor OS OS A A A A Containers O O O O
- 13. Operational complexity • Averagecontainers per host: N (N=5, 10/2014) • N-times as many “hosts” to manage • Affects – provisioning: prep’ing & building containers – configuration: passing config to containers – orchestration: deciding where/when containers run – monitoring: making sure containers run properly
- 14. Complexity increases with... 1.Number of things to measure 2. Velocity of change
- 15. Number of thingsto measure • 1 Amazon EC2 instance – 10 CloudWatch metrics • 1 operating system (e.g. linux) – 100 metrics •N containers – 100*N metrics •110 + 100*N metrics per instance
- 16. Combinatorial multiplication 100 500instancescontainers Assuming only 5 containers per instance
- 17. Combinatorial multiplication 160 610metrics perhost metrics per host Assuming only 5 containers per instance
- 18. Combinatorial multiplication 100 61,000instancesmetrics Assuming only 5 containers per instance
- 19.
- 20. Aggravating factors • Registry-basedprovisioning – new images as fast as you can git commit • Autonomic orchestration – from imperative to declarative – automated – individual containers don’t matter – e.g. kubernetes, mesos
- 21. A lot more, Alot faster.
- 22. If your monitoringis still centered on individual hosts or instances…
- 23. Host-centric monitoring Monitor Monitor GA P Hypervisor OS OS AA A A Containers O O O O
- 24. A lot morepain, A lot faster.
- 25.
- 26. A new approachto container monitoring
- 27.
- 28. Layers of monitoring Monitor Hypervisor OSOS A A A A Containers O O O O
- 29. Layers of monitoring CloudWatch Infrastructure Monitoring APM Hypervisor OSOS A A A A Containers O O O O
- 30. Layers of monitoring cpu/net/io filesystem dockermem docker cpu db queries web requests app throughput CloudWatch Infrastructure Monitoring APM e.g . Hypervisor OS OS A A A A Containers O O O O
- 31. Layers of monitoring •Access to metrics from all the layers • Amazon CloudWatch, OS metrics, Docker metrics, app metrics in 1 place • Shared timeline
- 32. If monitoring does notcover all layers, pain.
- 33. Tags (a.k.a. labels) You(probably) already use them
- 34. Tags • Monitoring islike Auto-Scaling Groups • Monitoring is like Docker orchestration • From imperative to declarative • Query-based • Queries operate on tags
- 35. Monitoring with tagsand queries “Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… and make sure resident set size < 1GB on c3.xl”
- 36. Monitoring with tagsand queries “Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… and make sure resident set size < 1GB on c3.xl”
- 37. Monitoring with tagsand queries “Monitor all Docker containers running image web” “… in region us-west-2 across all availability zones” “… that use more than 1.5x the average on c3.xl”
- 38. Demo: layers &tags
- 39. Take-aways 1. Docker increasesoperational complexity by an order of magnitude unless… 2. You have layered monitoring, from the instance to the container and to the application, and… 3. You monitor using tags and queries
Editor's Notes
- #3 My name is Alexis. I’m the CTO of Datadog. We monitor cloud-based infrastructures. We have been monitoring containers for a few years now (lxc then docker)
- #4 Datadog is a monitoring service made for cloud environments, such as AWS, Azure, Google Cloud, etc. By that I mean that Datadog understands that your infrastructure can change at any time and deals with it naturally. To be able to monitor effectively, Datadog acts as an aggregator: it aggregates everything, it speaks native Cloudwatch and over 100 different other sources, like databases, web servers, etc.
- #5 My goals for this talk are three-fold. Dive into key Docker metrics Explain operational complexity. In other words I want to take what we have seen on the field and show you where the pain points will be. Rethink monitoring of Docker containers. The old tricks won’t work.
- #6 Here’s what I would like to talk about today. I will start with very brief history of containers and docker. This is a popular topic so I will only focus on operational matters, including key metrics that containers expose. I will focus on the inherent complexity that comes with running fleets of containers. I will illustrate this with what we see out there, in the real world. We have a particular vantage point that gives us good insight into this.
- #8 Containers, as lightweight virtual runtimes have been around for a while without going back all the way to the mainframe. Depending on the operating system, they go by the name of jails, zones, cgroups and are like traditional VMs, without the flexibility but also without the overhead. They were initially designed for security reasons (e.g. jails) but most recently have been used to escape dependency hell.
- #9 Dependency hell is this state where you end up having tens or hundreds of dependencies on shared code. Before shared libraries we had compile-time dependencies to build static executables. Shared libraries were a good idea when the size of a library was commensurate to the amount of RAM available in a machine. Now, obviously, there is a lot less memory pressure. Still, that has remained the default way to build software. Then, packages came: apt, yum, rvm, virtualenv, etc. as a partial solution to have a group of binaries that reliably work together. That proved too slow, having to wait for upstream updates so people started to bundle their code and dependencies into /opt. Then a way to make self-contained packages. And now we are back full-circle to static binaries, when we realized how much baggage we carried in shared code.
- #10 When you look at it a container is a hybrid between a process and a full-blown host. It has a Dockerfile, which is a manifest or a recipe to build the container, much like source code builds a binary and kickstart, chef or puppet build a full-blown host. Then you have the actual binary representation of the container on disk, in /var/lib/docker. For a binary, it’s the .text section. For a host it’s its filesystem. Finally when it runs a container has a unique ID, much like a process has a PID and a host has a hostname. So a container is this intermediary between a single binary and a full-blown host. It’s lik a static binary with a fully-functioning IP stack. To put it simply if you look at it from a dev point of view, a container looks like a binary. If you are think about it from an operations point of view, a container is closer to a host.
- #12 Let’s recap for a minute. We know that a container is a lightweight VM We know roughly what current deployments look like in number of containers per instance. We know how to measure the performance of a single container. How do we monitor the whole thing. Here I want to make the case that Docker introduces operational complexity
- #13 This is how the stack has evolved over the past 15 years. On the left, without virtualization. Off-the-shelf could be your J2EE runtime, or your database. Then when virtualization and services like EC2 were introduced, in the middle. It’s allowed better utilization and quasi-instant provisioning but for an engineer, few things have changed. And now running Docker containers inside EC2 instances on top of real hardware. There is a clear trend here toward a lot more moving parts than before. It also puts engineering much closer to operations.
- #14 Specifically by an order of magnitude or so given the 5 containers per instance on average.. This affects a lot of different things at run-time. provisioning: docker configuration: etcd, confd, consul, etc. orchestration: kubernetes, mesos monitoring: where I can contribute the most
- #16 Let’s look at monitoring an EC2 instance. I counted 10 CloudWatch metrics, about 100 metrics coming from the OS, 50 metrics coming from a container, 10-15 of which are critical to monitor, and let’s say 50 metrics for an off-the-shelf component, for instance a database. This is a conservative estimate as we see our customers use many more metrics per instance.
- #17 Now let’s plug in some numbers. Assuming you have 100 instances, and 5 containers per instance, you have 500 containers to manage and monitor. And remember, from a management standpoint, containers behave like hosts. Single-purpose hosts, but hosts none the less.
- #18 So for a given instance, you have moved from 160 metrics per instance, to about 410. Again assuming, 5 containers per host and being conservative on the number of metrics you need to keep an eye on.
- #19 If I recap, 100 instances, 41,000 metrics generated. That’s already 3x what you had before.
- #20 And it gets worse. Much worse Let’s talk about velocity. If you compare the “half-life” of an EC2 instance, and by half-life I mean the median uptime of your instances. You’re likely having a mix of hourly instances and long-lived instances that will go on for months. Compare this to containers. A container’s half-life can be in minutes, days at the most.
- #21 On top of that, you’ll have to layer in much faster provisioning, where new versions of containers are created on a daily basis, so you rotate your container fleet on a daily basis between versions. Much faster and much more often than doing an OS upgrade. And you add autonomic orchestration that go from imperative to declarative. So you can say, I need 1 container of this kind per instance per zone, at all times. And the scheduler makes sure it’s always the case. If you use mesos or kubernetes, this is your new reality
- #22 In summary, from a management and monitoring standpoint, it means a lot more and a lot faster. More moving parts that change pretty much all the time with limited predictability.
- #23 If your monitoring is still centered around hosts, this is what your world view looks like: complicated. When we talk to customers, they feel that the move to EC2 was a key factor to rethink their monitoring. Because instances come and go, different groups within their organization would spin up new stacks with little advance notice. Imagine if you throw containers in the mix. The old, host-centric monitoring practice simply stops working altogether. The host-centric monitoring practice that has you track individual hosts. It’s a bit like ptolemaic astronomy. Put the earth at the center of the universe and account for the movement of the planets. It gets pretty complicated.
- #24 In other words host-centric monitoring does not really understand containers, so either you treat them as hosts, and you have a lot of hosts that come and go every few minutes, which makes your life miserable because the host-centric monitoring system thinks half of your infrastructure is on fire. Or you don’t track containers, and you essentially have a gap. You see the OS, you see the app, and what happens in the middle, well…
- #25 So in short, if you think about monitoring containers like you’ve monitored hosts before, you’re in for a painful ride very very quickly.
- #26 So how do we do it properly?
- #27 We need a new approach, that does not treat everything like a host. The picture here, as you’ve guessed, comes from Copernicus. He suggested a radical approach to simplifying the universe. Don’t put the earth at the center of it… Compared to putting the earth at the center of the universe, this one is striking in clarity and simplicity.
- #28 So what’s the secret sauce? It’s simple: forget about hosts, think in layers and tags. What do you I mean by that…
- #29 Using a layered monitoring approach is pretty simple. This is where you want to be: have coverage from the bottom of the stack all the way to the top.
- #30 Which means using monitoring tools that don’t leave any gap. At the bottom, CloudWatch to know about the VMs. In the middle, an infrastructure monitoring system that understands containers. Ad at the top, an application performance monitoring tool.
- #31 So in terms of what you can see through these tools: At the bottom, raw resources like cpu, network, io of the VM. In the middle, anything from the OS to docker metrics. At the top, application throughput.
- #32 The key here is to have 1 shared timeline for everything. You want to get CloudWatch metrics, OS metrics, Docker metrics and app metrics, ideally in 1 place, all on the same timeline so that you can see when things break, how changes ripples through the different layers.
- #33 That’s the first part of the equation. Layers.
- #34 Tags is the second half of the equation. The good news is that you use them already. How are they relevant to monitoring in general and monitoring containers in particular?
- #35 Think of monitoring like ASG. Think of monitoring like container orchestration. Don’t think “imperative”, think “declarative”. Don’t monitor host X, Y and Z. Instead, monitor everything that share a common property, for instance being located in the same AZ. Think in terms of queries and you will see that tags work beautifully because queries operate on tags.
- #36 Here’s an example: Monitor… to make sure a container does not blow up in memory.
- #37 You can see the tags: Name of container image: web AWS Region: us-west-2 Instance type: c3.xlarge Do you see how powerful this is?
- #38 Once you have queries in place, you can express even more interesting things such as: Monitor …
- #39 Ok, demo time.