Amazon Amazon Elastic Container Service (Amazon ECS)

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Maddox - @paulmaddox
Developer Technologies, AWS
January 2018 (Nordic Dev Days)
Amazon ECS Deep Dive
From zero to production

About me
Paul Maddox
Specialist Solutions Architect
Amazon Web Services
• 16 years of dev, SRE, and systems architecture background
• 7 of 7 8 AWS certifications
• Developer: Go/Java/C/Node
Twitter: @paulmaddox
Email: pmaddox@amazon.com
@paulmaddox
Paul

What to expect from this talk
• Build and deploy a containerized microservices application
• Twitter analyzer
• Go, RPC, Amazon Kinesis Firehose, AWS SSM Parameter Store
• Amazon ECS
• Deployment
• Availability
• Cost optimization
• Scaling
• Security
• Monitoring & logging

Key Components
Development cluster
Container instance Container instance
Container instance
Production cluster
Container instance
Amazon Elastic Container Service
(Amazon ECS)
Container
Container
Volume
Task definition
Amazon Elastic Container Registry
(Amazon ECR)

Key Components
Development cluster
Container instance
Production cluster
Container instance
Amazon Elastic Container Service
(Amazon ECS)
Container
Container
Volume
Task definition
Amazon EC2 Container Registry
(Amazon ECR)
NEW
NEW

Component: ECS
AWS is responsible for
operations of the cloud
You are responsible for operations in the cloud
using the building blocks provided.
Deployment
Security
Patching
Monitoring
Scaling
Availability
Cost Control
$ aws ecs create-cluster --cluster-name dev
AWS
Customer

Component: ECR
Deployment
Security
Cost Control
AWS
Customer
Monitoring
Scaling
Availability
Patching

Component: Container Instances
Development cluster
Cluster instance Cluster instance
Cluster instance
Deployment Cost Control
Patching Monitoring
Scaling Availability
Security
AWS
Customer

Container Instances: Building Blocks Provided
Deployment
Security
Patching
Monitoring
Scaling
Availability
Cost Control
CloudFormation
Update your AMI, replace instances
CloudWatch
Auto Scaling group
Reserved Instances
CLI SDKs etc...
IAM Inspector VPC Flow Logs etc...
Spot Fleet

Component: Tasks & Containers
Container
Container
Volume
Deployment
Security
Patching
Monitoring
Scaling
Availability
Logging
AWS
Customer

Deployment

How Should I Set This Up?
Use the AWS
Management
Console?

How Should I Set This Up?
Flex your scripting skills?
What happens if
my script fails
halfway through?
How long
should I
pause?
How do I upgrade /
roll back?

Deployments should be:
- A self-contained, deployable unit
- Repeatable
- Auditable
- Self-documenting

AWS CloudFormation: Infrastructure-as-Code

Time to deploy!
…or…

Time to update…
…or…

When a new environment is required…
…or…

AWS CLI
ECR
CloudFormation (YAML)
Resources:
MyRepository:
Type: AWS::ECR::Repository
Properties:
Name: myapp

Using ECR
Use AWS CLI to perform ‘docker login’
Tip: Use the Amazon ECR Credential Helper for automatic logins
https://github.com/awslabs/amazon-ecr-credential-helper

AWS CLI
ECS Cluster
CloudFormation (YAML)
Resources:
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: preprod

ECS Container Instances
• Highly available architecture,
distributed across multiple Availability
Zones
• VPC with public and private subnets
• Application Load Balancer with path
based routing for inbound traffic
• NAT gateways for outbound traffic
• Auto Scaling group of container
instances
• CloudWatch Logs for centralized
container logging
Private Subnet
Availability Zone Availability Zone
Internet
Gateway
Public Subnet Public Subnet
Private Subnet
Nat GatewayNat Gateway
AutoScaling GroupContainer InstanceContainer Instance Container InstanceContainer Instance
Application
Load Balancer
CloudWatch Logs
(container logs)

Inbound Traffic
• Incoming HTTP/HTTPS traffic comes in
via the Application Load Balancer (ALB)
in public subnets
• The ALB uses path based routing to
route /products/* to the container
instances in private subnets running our
product’s service
• Supports dynamic host port mapping,
allowing multiple containers of the
same type on each host
Internet
Gateway
AutoScaling Group
Container Instance Container Instance
Application
Load Balancer

Outbound Traffic
• Our container instances are in private
subnets, with no direct internet access
• At some point, they might need access
to external services
• NAT gateways provide a highly scalable
and available solution
Private Subnet
Internet
Gateway
Public Subnet Public Subnet
Private Subnet
Nat GatewayNat Gateway

Logging
CloudWatch Logs
(container logs)
• ECS integrates directly with
CloudWatch Logs (as well as others)
• Centralized collection of container logs
• Centralized collection of instance logs
• Search, filter, and alert on log
conditions
• (more to come later…)

tl;dr - ECS Reference Architecture on GitHub
https://github.com/awslabs/ecs-refarch-cloudformation

hands-on-demo
(because slides are boring)

Let's build an application

AWS Parameter Store
(for Twitter Credentials)
Overview
Tweet
Collector
Twitter API
Tweet
Archiver
Kinesis Firehose
Amazon S3 (archive)
AWS Lambda (realtime)
Elasticsearch (analyze)

Microservices and RPC at Twitch
• Used for inter-service communication
• Structured RPCs are much easier to
design and maintain compared to
REST
• Focus on data models, not
transports/routing
• Works with protobufs or JSON
• HTTP/1 compatible (unlike gRPC)
• Simplicity
https://blog.twitch.tv/twirp-a-
sweet-new-rpc-framework-for-
go-5f2febbf35f

RPC with Twitch Twirp
• Write a spec describing your API
(using protobufs)
• Generate a client and server
from the specification
• Limited to Go today, but more
language support in progress.
• (hands-on demo in a few slides)

Tweet Collector
• Written in Go
• Uses Twitter API to subscribe to
search terms
• Environment variables:
• SEARCH_TERMS
• ARCHIVE_ENDPOINT
• IAM role:
• AWS SSM Parameter Store
(for Twitter API credentials)
• Sends tweets to archiving service via
client SDK generated by Twitch
Twirp.
AWS Parameter Store
(for Twitter Credentials)
Tweet
Collector

Tweet Archiver
• Written in Go
• Hosts RPC server that receives tweets
• Sends tweets to Amazon Kinesis via aws-sdk-go
• Environment variables
• KINESIS_STREAM_NAME
• IAM role
• Write access to Kinesis stream
• Responds with Kinesis sequence number or error
Tweet
Archiver
Kinesis Firehose

Local
• Run locally with docker-
compose
• Logs to stdout/stderr
• Local AWS credentials
• Build/push containers
Development Workflow
AWS
• Deploy to ECS with
CloudFormation
• Logs in AWS CloudTrail Logs
• IAM Task Role
• Metrics in CloudWatch

https://github.com/paulmaddox/rpc-demo

Taking it further
• Sentiment analysis with
Amazon Comprehend
• Dashboards with Amazon
Quicksight
https://aws.amazon.com/blo
gs/machine-learning/build-a-
social-media-dashboard-
using-machine-learning-and-
bi-services

What else do we need for
production?

Cost Optimization

Reserved Instances
Up to 75%
Savings*
• Use Auto Scaling groups
• Reserve ECS container
instances when you have
known baseline capacity
requirements.
• Use On-Demand pricing for
capacity peaks.
* Dependent on specific AWS service, size/type, and region

Spot Instances
Up to 90%
Savings*
• Use Spot Fleet to maintain
instance availability and
define cluster based on
required CPU/memory.
* Compared to On-Demand price based on specific EC2 instance type, region, and Availability Zone

Multiple ECS Clusters
Creating multiple ECS clusters is easy, and often more cost efficient.
Consider availability and compute requirements.
Example: Development Cluster
Spot Fleet
Example: Production Cluster
Auto Scaling group with Reserved Instances for baseline and
On-Demand for capacity peaks
Example: Batch Processing Cluster
Spot Fleet of GPU Instances

Scaling

Scaling ECS Container Instances Automatically
Min
Desired
Scale out as needed
Max
• Use Auto Scaling groups
• Set Auto Scaling group
min, max, desired
• Scale in and out based
on CloudWatch alarms

Scaling ECS Container Instances Automatically
Tip
Use the ECS cluster
MemoryReservation
CloudWatch metric
Tutorial: Scaling Container Instances with CloudWatch Alarms

Application Auto Scaling for ECS Services

Security

Patching ECS Container Instances
ECSLaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: ami-1924770e
ECSAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
MinSize: 2
MaxSize: 8
DesiredCapacity: 2
AutoScalingRollingUpdate:
MinInstancesInService: 2
MaxBatchSize: 2
PauseTime: PT15M
WaitOnResourceSignals: true
1. Ensure you have an
AutoScalingRollingUpdate
policy on your Auto Scaling group
2. Update the AMI in your
CloudFormation template
3. aws cloudformation update-stack
4. Let CloudFormation perform a rolling
update to your ECS container
instances

Patching Containers

Minimal Containers
• Use the smallest FROM base
container to minimize
surface attack
• FROM scratch is ideal for
Go and other languages
that compile a (near) static
binary

IAM Roles
IAM roles for container instances:
• Bound to the ECS container instance
• Applies to all containers running on the host
• Pulling images from ECR
• CloudWatch Logs
IAM roles for tasks:
• Bound to specific ECS tasks
• Task-specific access to AWS services
Tip Use principle of least privilege – prefer IAM roles for tasks where applicable

Monitoring & Logging

Monitoring with CloudWatch

Prometheus
https://github.com/slok/ecs-exporter

Centralized Logging with CloudWatch Logs
• Defined within the task definition
• Available log drivers
• awslogs
• fluentd
• gelf
• journald
• json-file
• splunk
• Syslog

Centralized Logging with CloudWatch Logs

Tip: Use Metric Filters with CloudWatch Logs
5

Everything about everything ECS.
https://github.com/nathanpeck/awesome-ecs

Amazon Amazon Elastic Container Service (Amazon ECS)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Amazon Amazon Elastic Container Service (Amazon ECS)

Similar to Amazon Amazon Elastic Container Service (Amazon ECS) (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Amazon Amazon Elastic Container Service (Amazon ECS)