Containers come and go rapidly, which is great for scalable or fast-evolving infrastructure. However, the short life of containers make it more challenging to monitor, leaving many with questions such as: How many containers can you run on a given Amazon EC2 instance type? Which metric should you look at to measure contention? How do you manage fleets of containers at scale? In this session, we'll present the challenges and benefits of running containers at scale, how to use quantitative performance patterns to monitor your infrastructure at this magnitude and complexity, and we'll discuss proven strategies for monitoring your containerized infrastructure on AWS and ECS. Learning Objectives: - Set up the infrastructure to monitor your containers running on AWS - Understand the metrics available and what they mean - Define a strategy to monitor your containers

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brandon Chavis, AWS Solutions Architect
Ilan Rabinovitch, Director of Technical Community, Datadog
20th September 2016
Monitoring Containers at Scale

2. Agenda
Introduction to ECS
AWS Logging options: Cloudwatch Logs, Cloudwatch,
Cloudtrail
Logging containers with Datadog

3. Amazon EC2 Container Service (ECS)
Container Management
at Any Scale
Flexible Container
Placement
Integration
with the AWS Platform

4. Components of Amazon ECS
Task
One or more containers
running together on an
Instance
Task Definition
Definition of containers and
environment configuration
Cluster
Fleet of EC2 instances on
which tasks run
Cluster Manager
Manage cluster resource
and state of tasks
Scheduler
Places tasks onto cluster
Agent
Coordinate EC2 instances
and Manager

5. Cluster, Scheduler, Task Scheduler
Manager
Cluster
Task Definition
Task
Agent

6. Monitoring & Logging

7. AWS logging tools:
Cloudwatch
Cloudwatch Logs
Cloudtrail

8. CloudWatch Logs with awslogs driver
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon S3
Amazon Kinesis
AWS Lambda
Amazon Elasticsearch Service
Amazon ECS Store
Stream
Process
Search

9. CloudWatch Logs driver

10. Configuring Logging in Task Definition
logConfiguration task definition parameter
Requires version 1.18 or greater of the Docker Remote API
Maps to docker run --log-driver option
Log drivers: json-file, syslog, journald, gelf, fluentd, awslogs

11. Configuring Logging in Task Definition
"containerDefinitions": [ {
"memory": 300,
"portMappings": [ {
"hostPort": 80,
"containerPort": 80 } ],
"entryPoint": [ "sh", "-c" ],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-test",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "nginx" }
},
"name": "simple-app",
"image": "httpd:2.4",
"command": [ "/bin/sh -c "echo 'Congratulations! Your application is now running on a container in
Amazon ECS.' > /usr/local/apache2/htdocs/index.html && httpd-foreground"" ], "cpu": 10 } ],
"family": "cw-logs-example"
}

12. Monitoring with Amazon CloudWatch
Metric data sent to CloudWatch in 1-minute periods and
recorded for a period of two weeks
Available metrics: CPUReservation, MemoryReservation,
CPUUtilization, MemoryUtilization
Available dimensions: ClusterName, ServiceName

13. Monitoring with Amazon CloudWatch

14. Monitoring with Amazon CloudWatch

15. Monitoring with Amazon CloudWatch
Use the Amazon CloudWatch Monitoring Scripts to monitor
additional metrics, e.g. disk space:
# Edit crontab
> crontab -e
# Add command to report disk space utilization to CloudWatch every five minutes
*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail
--disk-path=/ --from-cron

16. Logging Amazon ECS API with AWS CloudTrail
{
"eventVersion": "1.03",
"userIdentity": {…},
"eventTime": "2015-10-12T13:57:33Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "CreateCluster",
"awsRegion": "eu-west-1",
"sourceIPAddress": "54.240.197.227",
"userAgent": "console.amazonaws.com",
"requestParameters": {
"clusterName": "ecs-cli"
},

17. Logging Amazon ECS API with AWS CloudTrail
"responseElements": {
"cluster": {
"clusterArn": "arn:aws:ecs:eu-west-
1:560846014933:cluster/ecs-cli",
"pendingTasksCount": 0,
"registeredContainerInstancesCount": 0,
"status": "ACTIVE",
"runningTasksCount": 0,
"clusterName": "ecs-cli",
"activeServicesCount": 0
}
},
[…]

18. Monitoring Amazon ECS with Datadog

19. • SaaS based infrastructure and application monitoring
• Focus on modern environments
• Cloud, Containers, Micro Services
• Processing nearly a trillion data points per day
• Intelligent Alerting and Insightful Dashboards
Datadog Overview

20. Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores,
Caches, Queues and more...
Monitor Everything

22. CloudWatch and ECS
Resources
CPUReservation
MemoryReservation
CPUUtilization
MemoryUtilization

24. How do we get at the upper layers?

25. Pseudo-files
• Provide visibility into container metrics via the file system.
• Generally under:
/cgroup/<resource>/docker/$CONTAINER_ID/
or
/sys/fs/cgroup/<resource>/docker/$CONTAINER_ID/

26. Pseudo-files: CPU Metrics
$ cat /sys/fs/cgroup/cpuacct/docker/$CONTAINER_ID/cpuacct.stat
> user 2451 # time spent running processes since boot
> system 966 # time spent executing system calls since boot
$ cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.stat
> nr_periods 565 # Number of enforcement intervals that have elapsed
> nr_throttled 559 # Number of times the group has been throttled
> throttled_time 12119585961 # Total time that members of the group were throttled (12.12 seconds)
Pseudo-files: CPU Throttling

27. Docker API
• Detailed streaming metrics as JSON HTTP socket
$ curl -v --unix-socket /var/run/docker.sock
http://localhost/containers/28d7a95f468e/stats

28. STATS Command
# Usage: docker stats CONTAINER [CONTAINER...]
$ docker stats $CONTAINER_ID
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O BLOCK I/O
ecb37227ac84 0.12% 71.53 MiB/490 MiB 14.60% 900.2 MB/275.5 MB 266.8 MB/872.7 MB

29. Side Car Containers

30. Agents and Daemons
• Ideally we’d want to schedule an agent or daemon on
each node via ECS Tasks.
• Current Solutions:
1. Bake it into your image.
2. Install on each host at provision time.
3. Automate with User Scripts and Launch
Configs

31. Grant Privileges via IAM
$ aws iam create-role
--role-name ecs-monitoring
--assume-role-policy-document file://trust.policy
$ aws iam put-role-policy
--role-name ecs-monitoring
--policy-name ecs-monitoring-policy
--policy-document file://ecs.policy
$ aws iam create-instance-profile
--instance-profile-name ECSNode
$ aws iam add-role-to-instance-profile
--instance-profile-name ECSNode
--role-name ecs-monitoring

32. Create A User Script

33. Auto-Scale!
$ aws autoscaling create-launch-configuration
--launch-configuration MyECSCluster --key-name my-key
--image-id AMI_ID --instance-type INSTANCE_TYPE
--user-data file://launch-script.txt
--iam-instance-profile IAM_ROLE

34. Full Stack Monitoring
Docker API ECS & CloudWatch
Monitoring Agent
Container
Containers List &
Metadata
Additional Metadata
(Tags, events, etc)
Host Level
Metrics

35. Monitoring Amazon ECS with Datadog

36. Aren’t we still missing a layer?

37. Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores,
Caches, Queues and more...
Monitor Everything

38. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service Discovery

39. Service Discovery
Docker API ECS & CloudWatch
Monitoring Agent
Container
A O A O
Containers List &
Metadata
Additional Metadata
(Tags, etc)
Config Backend
Integration Configurations
Host Level
Metrics

40. Custom Metrics
• Instrument custom applications
• You know your key transactions best.
• Use async protocols like STATSD

41. Demo

42. Monday, October 24, 2016
JW Marriot Austin
https://aws.amazon.com/events/devday-austin
Free, one-day developer event featuring tracks,
labs, and workshops around Serverless,
Containers, IoT, and Mobile
Q&A If you want to learn more, register for our upcoming DevDay Austin: