Container Networking Deep Dive with ECS and AWSVPC Mode

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Networking Deep
Dive with Amazon ECS
D e e p a k D a y a m a , S r . P r o d u c t M a n a g e r — A m a z o n E C S
S h a k e e l S o r a t h i a , V P E n g i n e e r i n g , F o x D C G
N o v e m b e r 3 0 , 2 0 1 7
C O N 4 0 1

Agenda
Container Networking with Amazon ECS
• Bridge Mode
• AWS VPC Mode
Networking for Microservices @Fox DCG

Terminology

Elastic Network Interface (ENI)
“An ENI is a virtual network interface that
you can attach to an instance* in a VPC”
An ENI can have following attributes:
• A primary private IPv4 address
• One or more secondary private IPv4 addresses
• One public IPv4 address
• One or more IPv6 addresses
• One or more security groups
• A MAC address
• A source/destination check flag
• A description
* We will revisit this later in the presentation

Define application containers: Image
URL, CPU & Memory requirements,
etc.
register
Task Definition
create
Cluster
• Infrastructure Isolation
boundary
• IAM Permissions boundary
run
Task
• A running instantiation of
a task definition
create
Service
Application
Load Balancer
• Maintain n running copies
• Integrated with ALB
• Unhealthy tasks
automatically replaced
ECS CONSTRUCTS

Task Definition
Example
• Embodiment of the application
requirements
• Defines containers, their image source,
logging preferences, and its
networking configuration

Container Networking Modes with ECS

Container Networking Modes with ECS
Host mode
Bridge mode
Task Networking (awsvpc) mode (New!)
None

Bridge Mode

Bridge Mode—Quick Primer
Containers share the same network interface as the instance
Each container gets a private IP and uses the Docker bridge for any communication.
Multiple Tasks use the same ENI
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0

Overview of Bridge Networking Mode
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0

Inter-Container Communication 1
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0

Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0

Working with Load Balancers
ECS Instance
172.31.2.165:8080
ENI
Private IP
172.31.2.165
172.31.2.165:80
Internet
ECS Instance
172.31.2.164:8080
ENI
Private IP
172.31.2.164
172.31.2.164:80
Containers
register with the
load balancer on
different ports
Security group
rules set to allow
ports 80 and 8080

Working with Load Balancers
Q. What if multiple applications
were listening on the same port?
Use dynamic port mapping

ECS Instance
80
ENI
Private IP
172.31.2.165
80
Internet
ECS Instance
80
ENI
Private IP
172.31.2.164
80172.31.2.164:6000
172.31.2.164:5000
172.31.2.165:1234
172.31.2.165:2345

Challenges
Performance
Lack of finer grained access control policies
No routable IP addresses for containers

AWSVPC Mode
New!!

Task Networking for ECS
https://github.com/aws/amazon-ecs-agent/blob/master/proposals/eni.md

AWSVPC Mode for Fargate

Configuration
- We create an ENI on your behalf
- ENI gets a private IP from the subnet
- Security group allows local traffic only
aws register-task-definition
{
“family": "helloFargate",
"networkMode":"awsvpc"
"cpu": 1024,
"memory": 512,
"containerDefinitions": [ ... ],
...
}
aws run-task
-- launch-type FARGATE
-- network-configuration
“awsVpcConfiguration = {
subnets=[subnet-id],
securityGroups=[sg-id]
}”
...

Task Networking – Task Definition

Task Networking – RunTask

Task Networking – ENI attachment
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
1. Pre ENI Attachment: The
Primary ENI (eth0) is in the
default namespace
2. ENI Attachment: The new
ENI (eth1) is in the default
namespace
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0eth1
ECS Instance ECS Instance

Task Networking – ENI provisioning
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
1. Pre ENI Attachment: The
Primary ENI (eth0) is in the
default namespace
2. ENI Attachment: The new
ENI (eth1) is in the default
namespace
3. ENI Provisioned: The ECS Agent
invokes CNI plugins to move the
new ENI into a new namespace and
configure it with addresses and routes
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0eth1
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Task Namespace
ECS Instance ECS Instance ECS Instance

ENI Provisioning Workflow
AWS CLI Run Task Find Resources
(i.e., find the instance
to place it on)
Create ENI Attach ENI Task=RUNNING

ENI De-Provisioning Workflow
AWS CLI Stop Task Detach ENI Destroy ENI Terminate Task

Task Networking – Attachment Details

ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
10.0.0.28/24
10.0.0.29/24
10.0.0.26/24
10.0.0.27/24
Task 2Task 1
AWS VPC

Demo

Choosing Between Bridge
and Task Networking Modes
Consider using AWS VPC mode for
new services that are attached to ALB/NLB
• Migration from bridge to task networking mode (coming soon!)
Amazon EC2 ENI limits
• Fargate is an alternative

Running ECS
with a Service Mesh
Shakeel Sorathia, VP Engineering
F o x D C G

21st Century Fox is home to a global portfolio
of cable and broadcasting networks and properties,
including FOX, FX, FXX, FXM, FS1, Fox News Channel,
Fox Business Network, FOX Sports, Fox Sports
Network, National Geographic, STAR India, 28 local
television stations in the U.S., and more than 350
international channels; film studio Twentieth
Century Fox Film; and television production studios
Twentieth Century Fox Television and a 50%
ownership interest in Endemol Shine Group.
The Digital Consumer Group is an organization inside
the company tasked with the digital distribution of
the companies’ content to consumers.
About Fox and the
Digital Consumer Group

How We Deliver
All APIs powering our experiences are
running on AWS
API’s delivered through Amazon API Gateway
Microservice architecture with
services written in Node.js and Go
Different teams deploy services differently
All services are delivered through
Docker containers running on ECS
Amazon API
Gateway
Amazon ECS

Problems That
Needed to Be Solved
Microservice architecture with lots
of services, utilizing bridge mode
networking, across 100s of instances
Teams needed to be able to determine
how to route an individual service
between blue/green deployments
Teams needed to be able to route a single
request to a specific service for testing
Maintain multiple versions of the same
service
Be very ephemeral and deal with all kinds
of failures

Investigated Solutions
Utilizing ALBs in front of all microservices
• Target groups have a limit of 1000 targets
• This led to 100s of ALBs, plus API Gateway
integrations were very complex
Direct service-to-service
communication through DNS
• Very complex DNS structure, especially
if trying to do blue/green deployments
Utilizing software to create a service mesh
• This is ultimately where we ended up

What is a Service Mesh?
A dedicated infrastructure layer for
making service-to-service communication
safe, fast, and reliable
Utilizing some form of service discovery allows
the service mesh software to route requests
to a healthy instance of a specific service
The service mesh software that
we elected to utilize is linkerd

What Does the Implementation Look Like?
Amazon API Gateway
NLB
Internet
Communication secured through client
certification such that only API Gateway
can talk to linker thru NGINX
NGINX layer used for
Client Cert Validation
While they look
different here,
they could be
the same set of
physical servers
Amazon ECS
Amazon ECS FOX DCG Microservices
Auto Scaling

Service-to-Service Communication
Service containers are
deployed through ECS and
use Bridge networking
All service requests go
to the local linkerd
linkerd is deployed to every ECS Container Instance linkerd finds a service instance and proxies request
ECS Container Instance ECS Container Instance
Service B Service A
linkerd linkerd
Private Subnet
AZ
VPC
Region
VPC

Implementation Details
All ECS Instances reside in private subnet
Service containers are deployed through ECS
Container instances utilize the ECS-optimized AMI
Linkerd is added to all ECS Container
Instances at startup through userdata
Services send all service-service
requests to their local linkerd instance
All services have a name that
other services can use to talk to it
Linkerd uses this name to find
that service’s routing in its delegation
/serviceA_v1 => .95 * /blue/
serviceA_v1 & .05 * /green/serviceA_v1;
Linkerd decides how to route it and then
finds an available instance and proxies the request
ECS Instances are horizontally scalable and use EC2 Auto Scaling
groups to scale out when available memory or CPU is running low

What We Achieved
Gave teams the ability to instantly
change routing for a microservice.
• By modifying the delegation tab for a service,
a team can decide where the traffic should go.
Linkerd has some resiliency functions built in.
For example, it can retry idempotent requests.
Because it ties in with a service discovery
backend, it can immediately find healthy
containers to route to.
Another nice feature of linkerd is to be able
to determine slower instances and route fewer
requests to these instances. Helps to deal
with the noisy neighbor issue.

Challenges We Plan to Tackle
Security around specific
services is still a challenge
• Task IAM roles effective for resource access
• Using Task ENIs would give us the ability to use
Security Groups on containers. (Big win for PII or
PCI data)
Linkerd doesn’t natively understand
how to find the closest container
(though latency routing helps with this)

Thank You!
dayamad@amazon.com
@saysdd

Container Networking Deep Dive with ECS and AWSVPC Mode

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Container Networking Deep Dive with ECS and AWSVPC Mode

Similar to Container Networking Deep Dive with ECS and AWSVPC Mode (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Container Networking Deep Dive with ECS and AWSVPC Mode