Running and managing large scale applications with microservices architectures can be difficult and often requires operating complex container management infrastructure. Amazon EC2 Container Service (ECS) is a highly scalable, high performance service for running and managing Docker applications. In this session, we will walk through a number of patterns and tools used by our customers to run their applications on Amazon ECS. We will show you how to setup, manage, and scale your Amazon ECS resources, and keep them secure. We will also discuss how to monitor your containerized application running on Amazon ECS and we'll provide best practices for logging and service discovery.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Prahlad Rao, Solutions Architect
October 24th, 2016
Amazon EC2 Container
Service Deep Dive

3. Agenda
Infrastructure Setup
Infrastructure Management
PaaS on ECS

4. Amazon ECS Infrastructure
Setup

5. Amazon ECS Cluster Setup

6. Cluster Setup with AWS CloudFormation
CloudFormation supports ECS cluster, service and task
definition resources
Use AWS::IAM::Role to create ECS service role and
container instances role
Launch container instances using
AWS:AutoScaling::LaunchConfiguation and
AWS:AutoScaling::AutoScalingGroup

7. Cluster Setup with AWS CloudFormation
"Resources" : {
"ECSCluster": {
"Type": "AWS::ECS::Cluster"
},
"ECSAutoScalingGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"VPCZoneIdentifier" : { "Ref" : "SubnetID" },
"LaunchConfigurationName" : { "Ref" : "ContainerInstances" },
"MinSize" : "1",
"MaxSize" : { "Ref" : "MaxSize" },
"DesiredCapacity" : { "Ref" : "DesiredCapacity" }
},
[…]
},

8. Cluster Setup with AWS CloudFormation
"ContainerInstances": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"commands" : {
"01_add_instance_to_cluster" : {
"command" : { "Fn::Join": [ "", [ "#!/bin/bashn", "echo
ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] }
}
},
[…]
}
}
}

9. Cluster Setup with AWS OpsWorks
One ECS Cluster layer per stack
One cluster can only be associated with one stack

10. Cluster Setup with AWS OpsWorks
Update OpsWorks IAM role to allow ecs:* actions
Add instances to layer (24/7, time-based, load-based)
Manage security updates, user permission and access

11. Amazon ECR Setup

12. Amazon ECR Setup
You have read and write access to the repositories you
create in your default registry, i.e.
<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com
Repository names can support namespaces, e.g. team-
a/web-app.
Repositories can be controlled with both IAM user access
policies and repository policies.

13. Amazon ECR Setup
# Authenticate Docker to your Amazon ECR registry
> aws ecr get-login
docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
# Create a repository called ecr-demo
> aws ecr create-repository --repository-name ecr-demo
# Push an image to your repository
> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

14. Amazon ECR Docker Credential Helper
Available today - http://bit.ly/25yXdC3
Place the docker-credential-ecr-login binary on your PATH
Set the contents of ~/.docker/config.json file to be:
{ "credsStore": "ecr-login" }
Push and pull images from ECR without docker login

15. Amazon ECS Infrastructure
Management

16. Amazon EC2 Simple Systems
Manager (SSM)

17. Amazon EC2 Simple Systems Manager (SSM)
Use Amazon EC2 SSM to execute commands on container
instances, e.g. yum update
• Add AmazonEC2RoleForSSM
to instances IAM role to
process Run Commands
• Install SSM Agent
• Create SSM document –
similar to CloudInit userdata
• Lock down AWS-* documents

18. Monitoring & Logging

19. Monitoring with Amazon CloudWatch
Metric data sent to CloudWatch in 1-minute periods and
recorded for a period of two weeks
Available metrics: CPUReservation, MemoryReservation,
CPUUtilization, MemoryUtilization
Available dimensions: ClusterName, ServiceName

20. Monitoring with Amazon CloudWatch

21. Monitoring with Amazon CloudWatch

22. Monitoring with Amazon CloudWatch
Use the Amazon CloudWatch Monitoring Scripts to monitor
additional metrics, e.g. disk space:
# Edit crontab
> crontab -e
# Add command to report disk space utilization to CloudWatch every five minutes
*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-
space-avail --disk-path=/ --from-cron

23. CloudWatch Logs with awslogs driver
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon S3
Amazon Kinesis
AWS Lambda
Amazon Elasticsearch Service
Amazon ECS Store
Stream
Process
Search

24. CloudWatch Logs driver

25. Configuring Logging in Task Definition
logConfiguration task definition
parameter
Requires version 1.18 or greater of the Docker Remote API
Maps to docker run --log-driver option
Log drivers: json-file, syslog, journald, gelf, fluentd,
awslogs

26. Configuring Logging in Task Definition
"containerDefinitions": [ {
"memory": 300,
"portMappings": [ {
"hostPort": 80,
"containerPort": 80 } ],
"entryPoint": [ "sh", "-c" ],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-test",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "nginx" }
},
"name": "simple-app",
"image": "httpd:2.4",
"command": [ "/bin/sh -c "echo 'Congratulations! Your application is now running on a
container in Amazon ECS.' > /usr/local/apache2/htdocs/index.html && httpd-foreground"" ],
"cpu": 10 } ],
"family": "cw-logs-example"
}

27. Logging Amazon ECS API with AWS CloudTrail
{
"eventVersion": "1.03",
"userIdentity": {…},
"eventTime": "2015-10-12T13:57:33Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "CreateCluster",
"awsRegion": "eu-west-1",
"sourceIPAddress": "54.240.197.227",
"userAgent": "console.amazonaws.com",
"requestParameters": {
"clusterName": "ecs-cli"
},

28. Logging Amazon ECS API with AWS CloudTrail
"responseElements": {
"cluster": {
"clusterArn": "arn:aws:ecs:eu-west-
1:560846014933:cluster/ecs-cli",
"pendingTasksCount": 0,
"registeredContainerInstancesCount": 0,
"status": "ACTIVE",
"runningTasksCount": 0,
"clusterName": "ecs-cli",
"activeServicesCount": 0
}
},
[…]

29. Monitoring Amazon ECS with Datadog

30. Monitoring Amazon ECS with Sysdig Cloud

31. Scaling Amazon ECS

32. Setup ECS Cluster with AutoScaling
Create LaunchConfiguration
• Pick instance type
depending on resource
requirements, e.g. memory
or CPU
• Use latest Amazon Linux
ECS-optimized AMI, other
distros available
Create AutoScaling group and
set to cluster initial size

33. Auto Scaling your Amazon ECS Cluster
Create CloudWatch alarm
on a metric, e.g.
MemoryReservation
Configure scaling policies to
increase and decrease the
size of your cluster

34. Auto Scaling your Amazon ECS services

35. Auto Scaling your Amazon ECS services

36. Service Discovery &
Configuration Management

37. Service Discovery with ECS Services & Route 53
Route 53 private hosted zone
Set search path on hosts with DHCP option sets
Define ECS services with ELB
Create CNAMEs for each ELB

38. Service Discovery with ECS Services & Route 53
Task
Task TaskTask
ECS
Service
Application
router, e.g.
nginx
Internal ELB with
CNAME, e.g.
api.example.com
Route 53 private
zone, e.g.
example.com

39. Service Discovery with Weaveworks
DNS interface for cross-host
container communication
Gossip protocol to share
grouped updates
Overlay network between hosts

40. Service Discovery and Configuration
Management with Consul
Three main components:
• Consul agent - Runs on each node, responsible for
checking the health of the services and of the node
itself.
• One or more Consul servers - Store and replicate
data, leader elected using the Raft consensus
algorithm
• Registrator agent - Automatically
register/deregisters services based on published
ports and metadata from the container environment
variables defined in the ECS task definition

41. Service Discovery and Configuration
Management with Consul
ECSCluster
consul-server
ECS Instance
consul-agent
registrator
ECS Instance
Back end 1
Back end 2
consul-agent
registrator
ECS Instance
Front end
ECSCluster

42. Service Discovery and Configuration
Management with etcd
etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd

43. Security

44. ECS IAM Policies and Roles
The ECS agent calls the ECS APIs on your behalf, so
container instances require an IAM policy and role that
allows these calls.
The ECS service scheduler calls the EC2 and ELB APIs on
your behalf to register and deregister container instances
with your load balancers.
Use AmazonEC2ContainerServiceforEC2Role and
AmazonEC2ContainerServiceRole managed policies
(respectively)

45. ECR IAM Policies and Roles
ECR uses resource-based permissions to control access.
By default, only the repository owner has access to a
repository.
You can apply a policy document that allows others to
access your repository.
Use managed policies for IAM users or roles that allow
differing levels of control:
AmazonEC2ContainerRegistryFullAccess,
AmazonEC2ContainerRegistryPowerUser or
AmazonEC2ContainerRegistryReadOnly

46. IAM Roles for ECS Tasks
{
"family": “signup-app",
"taskRoleArn":
"arn:aws:iam::123456789012:role/DynamoDBRoleForTask",
"volumes": [],
"containerDefinitions": [{
"environment": [ ... ],
"name": “signup-web",
"mountPoints": [],
"image": “amazon/signup-web",
"cpu": 25,
"portMappings": [ ... ],
"entryPoint": [ ... ],
"memory": 100,
"essential": true,
"volumesFrom": []
}
]}

47. Image Vulnerability Scanning with Twistlock

48. Secrets Management
• Option 1: Task Definition Environment Variables
• Easy to get Started
• Configuration stored Directly into Task Definition
• Version in Immutable Definition; Easy Rollback
• Not Great for Secrets
• Option 2: Encrypted DynamoDB or S3
• Use Environment Variables to Provide Pointer
• Use AWS Encryption Clients to Securely Store
• Use VPC-Endpoints, IAM Policies, and IAM Roles to Restrict
Access

49. Secrets Management
Task
ECS Cluster
Container instance

50. PaaS on ECS

51. AWS Elastic Beanstalk
Uses Amazon ECS to coordinate deployments to
multicontainer Docker environments
Takes care of tasks including cluster creation, task
definition and execution

52. AWS Elastic Beanstalk
Elastic Beanstalk uses a Dockerrun.aws.json file that
describes how to deploy containers.
The Dockerrun.aws.json file includes three sections:
• AWSEBDockerrunVersion: Set to "2" for multicontainer
Docker environments.
• containerDefinitions: An array of container definitions.
• volumes: Creates mount points in the container instance
that a container can use.

53. Convox

54. Convox
# Initialize your app and create default manifest
> convox init
# Locally build and run your app as declared in the manifest
> convox start
# Create app
> convox apps create my_app
# Deploy app, output ELB DNS name
> convox deploy
[...]
web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

55. Remind Empire
Control layer on top of Amazon ECS that provides a
Heroku like workflow
Any tagged Docker image can be deployed to Empire as
an app
• When you deploy a Docker image to Empire, it will
extract a Procfile from the WORKDIR
• Each process type in the Procfile maps directly to an
ECS Service

56. Remind Empire
Routing layer backed by internal ELBs
• An application that specifies a web process will get an
internal ELB attached to its ECS Service
• When a new internal ELB is created, an associated
CNAME record is created in Route53 under the internal
TLD, enabling service discovery via DNS

57. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
All attendees will receive a special giveaway gift!
Please join us for the
AWS DevDay Networking Reception
5:00 - 6:30 PM
JW Grand Foyer

58. Thank You!

59. Don’t Forget Evaluations!