This document summarizes a presentation about deploying and managing containerized microservices applications on Amazon ECS. It discusses key components like ECS, ECR, and container instances. It then covers topics like deployment with CloudFormation, cost optimization with reserved instances and spot instances, scaling, security practices like patching and IAM roles, and monitoring with CloudWatch.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brent Langston - @brentContained
Developer Advocate, AWS
March 2018 (Dev Days)
Amazon ECS Deep Dive
From zero to production

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
About me
Brent Langston
Sr Developer Advocate
Amazon Web Services
• 16 years of dev, SRE, and systems architecture background
• Developer: Python/Ruby/Crystal/Node
• Helped build: Tumblr.com, Spotify.com, HiOscar.com and
CloudPassage.com
Twitter: @brentContained
Email: blangs@amazon.com

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from this talk
• Build and deploy a containerized microservices application
• Twitter analyzer
• Go, RPC, Amazon Kinesis Firehose, AWS SSM Parameter Store
• Amazon ECS
• Deployment
• Availability
• Cost optimization
• Scaling
• Security
• Monitoring & logging

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Components
Development cluster
Container instance Container instance
Container instance
Production cluster
Container instance Container instance
Container instance
Amazon Elastic Container Service
(Amazon ECS)
Container
Container
Volume
Task definition
Amazon Elastic Container Registry
(Amazon ECR)

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Components
Development cluster
Container instance Container instance
Container instance
Production cluster
Container instance Container instance
Container instance
Amazon Elastic Container Service
(Amazon ECS)
Container
Container
Volume
Task definition
Amazon EC2 Container Registry
(Amazon ECR)
NEW
NEW

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Component: ECS
AWS is responsible for
operations of the cloud
You are responsible for operations in the cloud
using the building blocks provided.
Deployment
Security
Patching
Monitoring
Scaling
Availability
Cost Control
$ aws ecs create-cluster --cluster-name dev
AWS
Customer

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Component: ECR
Deployment
Security
Cost Control
AWS
Customer
Monitoring
Scaling
Availability
Patching
AWS is responsible for
operations of the cloud
You are responsible for operations in the cloud
using the building blocks provided.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Component: Container Instances
Development cluster
Cluster instance Cluster instance
Cluster instance
Deployment Cost Control
Patching Monitoring
Scaling Availability
Security
AWS
Customer
AWS is responsible for
operations of the cloud
You are responsible for operations in the cloud
using the building blocks provided.

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Instances: Building Blocks Provided
Deployment
Security
Patching
Monitoring
Scaling
Availability
Cost Control
CloudFormation
Update your AMI, replace instances
CloudWatch
Auto Scaling group
Reserved Instances
CLI SDKs etc...
IAM Inspector VPC Flow Logs etc...
Spot Fleet

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Component: Tasks & Containers
Container
Container
Volume
Deployment
Security
Patching
Monitoring
Scaling
Availability
Logging
AWS
Customer
AWS is responsible for
operations of the cloud
You are responsible for operations in the cloud
using the building blocks provided.

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Should I Set This Up?
Use the AWS
Management
Console?

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Should I Set This Up?
Flex your scripting skills?
What happens if
my script fails
halfway through?
How long
should I
pause?
How do I upgrade /
roll back?

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployments should be:
- A self-contained, deployable unit
- Repeatable
- Auditable
- Self-documenting

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CloudFormation: Infrastructure-as-Code

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Time to deploy!
…or…

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Time to update…
…or…

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When a new environment is required…
…or…

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
ECR
CloudFormation (YAML)
Resources:
MyRepository:
Type: AWS::ECR::Repository
Properties:
Name: myapp

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Using ECR
Use AWS CLI to perform ‘docker login’
Tip: Use the Amazon ECR Credential Helper for automatic logins
https://github.com/awslabs/amazon-ecr-credential-helper

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
ECS Cluster
CloudFormation (YAML)
Resources:
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: preprod

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS Container Instances
• Highly available architecture,
distributed across multiple Availability
Zones
• VPC with public and private subnets
• Application Load Balancer with path
based routing for inbound traffic
• NAT gateways for outbound traffic
• Auto Scaling group of container
instances
• CloudWatch Logs for centralized
container logging
Private Subnet
Availability Zone Availability Zone
Internet
Gateway
Public Subnet Public Subnet
Private Subnet
Nat GatewayNat Gateway
AutoScaling GroupContainer InstanceContainer Instance Container InstanceContainer Instance
Application
Load Balancer
CloudWatch Logs
(container logs)

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inbound Traffic
• Incoming HTTP/HTTPS traffic comes in
via the Application Load Balancer (ALB)
in public subnets
• The ALB uses path based routing to
route /products/* to the container
instances in private subnets running our
product’s service
• Supports dynamic host port mapping,
allowing multiple containers of the
same type on each host
Internet
Gateway
AutoScaling Group
Container Instance Container Instance
Application
Load Balancer

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Outbound Traffic
• Our container instances are in private
subnets, with no direct internet access
• At some point, they might need access
to external services
• NAT gateways provide a highly scalable
and available solution
Private Subnet
Internet
Gateway
Public Subnet Public Subnet
Private Subnet
Nat GatewayNat Gateway
Container Instance Container Instance

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logging
Container Instance Container Instance
CloudWatch Logs
(container logs)
• ECS integrates directly with
CloudWatch Logs (as well as others)
• Centralized collection of container logs
• Centralized collection of instance logs
• Search, filter, and alert on log
conditions
• (more to come later…)

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
tl;dr - ECS Reference Architecture on GitHub
https://github.com/awslabs/ecs-refarch-cloudformation

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let's build an application

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Parameter Store
(for Twitter Credentials)
Overview
Tweet
Collector
Twitter API
Tweet
Archiver
Kinesis Firehose
Amazon S3 (archive)
AWS Lambda (realtime)
Elasticsearch (analyze)

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Microservices and RPC at Twitch
• Used for inter-service communication
• Structured RPCs are much easier to
design and maintain compared to
REST
• Focus on data models, not
transports/routing
• Works with protobufs or JSON
• HTTP/1 compatible (unlike gRPC)
• Simplicity
https://blog.twitch.tv/twirp-a-
sweet-new-rpc-framework-for-
go-5f2febbf35f

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RPC with Twitch Twirp
• Write a spec describing your API
(using protobufs)
• Generate a client and server
from the specification
• Limited to Go today, but more
language support in progress.
• (hands-on demo in a few slides)

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tweet Collector
• Written in Go
• Uses Twitter API to subscribe to
search terms
• Environment variables:
• SEARCH_TERMS
• ARCHIVE_ENDPOINT
• IAM role:
• AWS SSM Parameter Store
(for Twitter API credentials)
• Sends tweets to archiving service via
client SDK generated by Twitch
Twirp.
AWS Parameter Store
(for Twitter Credentials)
Tweet
Collector

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tweet Archiver
• Written in Go
• Hosts RPC server that receives tweets
• Sends tweets to Amazon Kinesis via aws-sdk-go
• Environment variables
• KINESIS_STREAM_NAME
• IAM role
• Write access to Kinesis stream
• Responds with Kinesis sequence number or error
Tweet
Archiver
Kinesis Firehose

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Local
• Run locally with docker-
compose
• Logs to stdout/stderr
• Local AWS credentials
• Build/push containers
Development Workflow
AWS
• Deploy to ECS with
CloudFormation
• Logs in AWS CloudTrail Logs
• IAM Task Role
• Metrics in CloudWatch

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://github.com/paulmaddox/rpc-demo

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Taking it further
• Sentiment analysis with
Amazon Comprehend
• Dashboards with Amazon
Quicksight
https://aws.amazon.com/blo
gs/machine-learning/build-a-
social-media-dashboard-
using-machine-learning-and-
bi-services

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What else do we need for
production?

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cost Optimization

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reserved Instances
Up to 75%
Savings*
• Use Auto Scaling groups
• Reserve ECS container
instances when you have
known baseline capacity
requirements.
• Use On-Demand pricing for
capacity peaks.
* Dependent on specific AWS service, size/type, and region

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Spot Instances
Up to 90%
Savings*
• Use Spot Fleet to maintain
instance availability and
define cluster based on
required CPU/memory.
* Compared to On-Demand price based on specific EC2 instance type, region, and Availability Zone

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple ECS Clusters
Creating multiple ECS clusters is easy, and often more cost efficient.
Consider availability and compute requirements.
Example: Development Cluster
Spot Fleet
Example: Production Cluster
Auto Scaling group with Reserved Instances for baseline and
On-Demand for capacity peaks
Example: Batch Processing Cluster
Spot Fleet of GPU Instances

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling ECS Container Instances Automatically
Min
Desired
Scale out as needed
Max
• Use Auto Scaling groups
• Set Auto Scaling group
min, max, desired
• Scale in and out based
on CloudWatch alarms

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling ECS Container Instances Automatically
Tip
Use the ECS cluster
MemoryReservation
CloudWatch metric
Tutorial: Scaling Container Instances with CloudWatch Alarms

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application Auto Scaling for ECS Services

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application Auto Scaling for ECS Services

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Patching ECS Container Instances
ECSLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: ami-1924770e
ECSAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
MinSize: 2
MaxSize: 8
DesiredCapacity: 2
AutoScalingRollingUpdate:
MinInstancesInService: 2
MaxBatchSize: 2
PauseTime: PT15M
WaitOnResourceSignals: true
1. Ensure you have an
AutoScalingRollingUpdate
policy on your Auto Scaling group
2. Update the AMI in your
CloudFormation template
3. aws cloudformation update-stack
4. Let CloudFormation perform a rolling
update to your ECS container
instances

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Patching Containers

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minimal Containers
• Use the smallest FROM base
container to minimize
surface attack
• FROM scratch is ideal for
Go and other languages
that compile a (near) static
binary

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM Roles
IAM roles for container instances:
• Bound to the ECS container instance
• Applies to all containers running on the host
• Pulling images from ECR
• CloudWatch Logs
IAM roles for tasks:
• Bound to specific ECS tasks
• Task-specific access to AWS services
Tip Use principle of least privilege – prefer IAM roles for tasks where applicable

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring & Logging

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with CloudWatch

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with CloudWatch

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Prometheus
https://github.com/slok/ecs-exporter

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Centralized Logging with CloudWatch Logs
• Defined within the task definition
• Available log drivers
• awslogs
• fluentd
• gelf
• journald
• json-file
• splunk
• Syslog

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Centralized Logging with CloudWatch Logs

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tip: Use Metric Filters with CloudWatch Logs
5

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Everything about everything ECS.
https://github.com/nathanpeck/awesome-ecs

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Twitter: @brentContained
Email: blangs@amazon.com