Amazon EKS - Elastic Container Service for Kubernetes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
E L A S T I C C O N TA I N E R S E R V I C E F O R K U B E R N E T E S
(EKS)
Re Alvarez Parmar
@realz reparmar@amazon.com
Nikita Patil
nikipat@amazon.com

Agenda
10:00am EKS Overview and Roadmap
12:00pm Lunch
12:30pm EKS Workshop Hands on Lab

Why Containers?
• Speed
• Efficiency
• Easier packaging
• Less risky deployments
• Better Development
experience
• Microservices Photo & Licence

Customer use cases
Microservices
PaaS
Platform-as-a-Service Enterprise App
Migration
Machine Learning

Challenges of Containers at Scale
• More transient
• More distributed and complex
• Networking
• Scheduling / Resource Management
• Less isolated (containers can share a kernel)

AWS container services landscape
Management
Deployment, Scheduling,
Scaling & Management of
containerized applications
Hosting
Where the containers run
Amazon Elastic
Container Service
Amazon Elastic
Container Service
for Kubernetes
Amazon EC2 AWS Fargate
Image Registry
Container Image Repository
Amazon Elastic
Container Registry

Open source container
management platform
Helps you run
containers at scale
Gives you primitives
for building modern
applications
What is Kubernetes?

Cloud-native applications
M I C R O S E R V I C E
T O O L I N G
C L O U D N AT I V E
A P P L I C AT I O N S

But where you run Kubernetes matters
Quality of the
cloud platform
Quality of the
applications
Your users

51%of Kubernetes workloads
run on AWS today
— Cloud Native Computing Foundation

“Run Kubernetes for me.”

Tenet 1
EKS is a platform for enterprises
to run production-grade
workloads
Tenet 2
EKS provides a native and
upstream Kubernetes experience
Tenet 3
Provide seamless integration with
AWS services and eliminate
undifferentiated heavy lifting
Tenet 4
EKS team actively contributes
to the Kubernetes project

EKS is Kubernetes Certified
Kubernetes Conformance
• Guaranteed Portability and Interoperability
• Timely Updates
• Confirmability

0
20
40
60
80
100
120
140
160
jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
AWS Contributions to Kubernetes in 2018
AWS contributions to Kubernetes

EKS Architecture Overview

Kubernetes on AWS
Highly available
Scalable
Secure
3x Kubernetes API Servers for HA
3x Kubernetes etcd Servers for HA

Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
API Servers
etcd
Worker nodes
EKS Overview
AWS
Managed
Customer
Managed

[mycluster].eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl
AWS Cloud
VPC

EKS – Creating your cluster
C r e a t e c l u s t e r
C r e a t e H A c o n t r o l
p l a n e
I A M i n t e g r a t i o n
C e r t i f i c a t e
m a n a g e m e n t
S e t u p L B
A d d W o r k e r
N o d e s
aws eks create-cluster --name devel --role-arn arn:aws:iam::111122223333:role/eks-service-
role-AWSServiceRoleForAmazonEKS-EXAMPLEBKZRQR --resources-vpc-config
subnetIds=subnet-a9189fe2,subnet-50432629,securityGroupIds=sg-f5c54184

Describe Cluster
$ aws eks describe-cluster --name devel
HTTP/1.1 200 Content-type: application/json
{ "cluster":
{
"clusterName": "string",
"createdAt": number,
"currentMasterVersion": "string",
"desiredMasterVersion": "string",
"masterEndpoint": "string",
"roleArn": "string",
"status": "string",
"statusMessage": "string"
}
}

EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet

EKS Logging

EKS Logging
EKS ManagedCustomer Account
Internet
Amazon
CloudWatch
AWS
CloudTrail

WorkerWorkerMaster
WorkerWorkerMaster
ASG
AZ1
Region
AZ2
ASG
CloudWatch
Logs
Elasticsearch
Kibana
Fluentd
DaemonSet
Kubectl logs
Elasticsearch (index),
Fluentd (store), and
Kibana (visualize)
EKS Logging

Helm: Package Manager for Kubernetes

• Helm not only is a package manager, but also a Kubernetes
application deployment management tool
• It helps you to:
• achieve a simple (one command) and repeatable deployment
• manage application dependency, using specific versions of other
application and services
• manage multiple deployment configurations: test, staging, production and
others
• execute post/pre deployment jobs during application deployment
• update/rollback and test application deployments
Helm: package manager for Kubernetes

EKS Identity & Access Management

AWS Identity & Access Management (IAM)
IAM enables secure access to AWS services and resources
Granular controls for Kubectl and pods
Two IAM roles created
for K8s cluster

K8s action allowed/denied
IAM roles for Kubectl
Authorizes AWS Identity with RBAC
K8s API
Passes AWS Identity
Verifies AWS Identity
AWS Auth
1
2
3
4
Kubectl

Worker provisioning
k u b e c t l
A W S A u t h
c o n f i g m a p & R B A C
W o r k e r s
R o l e
R o l e
config map

IAM roles for Pods

IAM Roles for Pods
AWS proposed solution
Makes use of Kubernetes TokenRequestProjection
Tokens to assume an IAM role
Needs Kubernetes v1.11 & latest AWS SDKs
AWS CloudTrail support at launch
Coming
Soon

SAML 2.0 – Integrate with AD and SSO
M A
AWS Directory
Service
On-premises users
and groups
Manage Permissions to
AWS accounts
sts:AssumeRole
On-premises
Microsoft Active
Directory
AD Connector / AD
Trust
IAM role

EKS Networking

Native VPC networking
with CNI plugin
Pods have the same VPC
address inside the pod
as on the VPC
Simple, secure networking
Open source and
on Github
…{ }
https://github.com/aws/amazon-vpc-cni-k8s

VPC CNI plugin
• Bridge between the K8s and – AWS VPC
• A W S R o u t a b l e I P s
• Thin layer – no performance impact
• Pod IP ENI secondary IP

R u n t i m e
N e t w o r k
p l u g i n
N e t w o r k
c o n f i g u r a t i o n
CNI Infrastructure

K u b e l e t
V P C C N I
p l u g i n
1 . C N I A d d / D e l e t e
E C 2
E N I E N I E N I
P o d P o d P o d P o d
V P C
N e t w o r k
.........
0 . C r e a t e E N I
2 . S e t u p v e t h
VPC Networking Internals

VPC CNI plugin architecture
K u b e l e t
V P C C N I
p l u g i n
N e t w o r k l o c a l
c o n t r o l p l a n e
E N I s /
S e c o n d a r y I P s
C N I A d d / D e l e t e
g R P C
E C 2

AWS VPC CNI plugin
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet –
10.0.0.0/24
Instance 1 Instance 2
VPC

AWS VPC CNI plugin – Understanding IP allocation
• Primary CIDR range è RFC 1918 addresses è 10/8, 172.16/12, 192.168/16
Used in EKS for:
• Pods
• X-account ENIs for (masters à workers) communication (exec, logs, proxy
etc.)
• Internal Kubernetes services network (10.100/16 or 172.20/16 – chosen based on
your VPC range)
Setup: EKS cluster creation è provide list of subnets (in at least 2 AZs!) è tagging

AWS VPC CNI plugin – Understanding IP Allocation
• Secondary CIDR ranges (new!) è non-RFC 1918 address blocks (100.64.0.0/10
and 198.19.0.0/16)
Used in EKS for:
• Pods only
How?
• EKS custom network config è enable è create ENIConfig CRD è annotate
nodes
CNI
1.2.1+

AWS VPC CNI plugin - configurability
• Custom Network Configs
• SNAT / External SNAT
• Configurable warm pool

Packet flow : pod - to - pod
E C 2
Default namespace
Pod namespace
veth veth
Main RT
E C 2
Default namespace
Pod namespace
veth
Route
Table
Main RT
ENI RT
veth
VPC
fabric
ENI RT
Route
Table

Packet flow : pod - to external
E C 2
Default namespace
Pod namespace
veth
Route
Table
Main RT
ENI RT
veth
External
Network
IPTables

Frontend
Cats Dogs
kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: default-deny
spec:
podSelector:
matchLabels: {}
dev-namespace

Frontend
Cats Dogs
dev-namespace
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:
podSelector:
matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP
port: 80

Frontend
Cats Dogs
dev-namespace
kind: NetworkPolicy
metadata:
name: frontend-to-cats
spec:
podSelector:
matchLabels:
role: cats
ingress:
- from:
- podSelector:
matchLabels:
role: “frontend”
ports:
- protocol: TCP
port: 80

That looks good. But, what about isolation?

kubectl create namespace prod-namespace

Frontend
Cats Dogs
dev-namespace
Frontend
Cats Dogs
prod-namespace

Frontend
Cats Dogs
prod-namespace
kind: NetworkPolicy
metadata:
name: frontend-to-cats-and-dogs
namespace: prod-namespace
spec:
podSelector:
matchLabels:
role: cats-and-dogs
ingress:
- from:
- podSelector:
matchLabels:
role: “frontend”
ports:
- protocol: TCP
port: 80

Networking: Pod to service

• Exposes the service on a cluster-internal IP
• Only reachable from within the cluster
• Access possible via kube-proxy
• Useful for debugging services, connecting from
your laptop or displaying internal dashboards
Kubernetes ServiceType: ClusterIP

Kubernetes ServiceType: NodePort
• Exposes the service on each Node’s IP at a
static port.
• Routes to a ClusterIP service, which is
automatically created.
• from outside the cluster:
<NodeIP>:<NodePort>
• 1 service per port
• Uses ports 30000-32767

• Exposes the service externally using a cloud
provider’s load balancer.
• NodePort and ClusterIP services (to which LB
will route) automatically created.
• Each service exposed with a LoadBalancer (ELB
or NLB) will get its own IP address
• Exposes L4 (TCP) or L7 (HTTP) services
Kubernetes ServiceType: LoadBalancer

Service load balancer: NLB
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer

Service load balancer: NLB
• NLB supports forwarding the client’s IP through to the node
• .spec.externalTrafficPolicy = Local è client ip passed to pod
• Nodes with no matching pods will be removed by specified NLB’s health check
.spec.healthCheckNodePort
• Use DaemonSet or pod anti-affinity to verify even traffic split

• exposes HTTP/HTTPS routes
to services within the cluster
• Many implementations: ALB,
Nginx, F5, HAProxy etc
• Default Service Type:
ClusterIP
Kubernetes Ingress Object

ALB Ingress Controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort

Kubernetes Autoscaling with Amazon EKS

Pod Autoscaling Options
• HPA: de-facto method
• Cluster-proportional-autoscaler
• k8s-rabid-pod-autoscaler
• Kube-sqs-autoscaler

EKS – Pod Scaling by HPA
• Horizontal pod scaling
• Automatically scales number of pods in replication
controller
• Implemented as a Kubernetes API resource and a
controller.
• The controller periodically adjusts # of replicas in a
replication controller or deployment to match the observed
average CPU utilization to the target specified by user.
• HPA can scale on custom and external metrics (any metric
collected by metrics server api) instead of simply CPU and
memory..

Cluster Autoscaling Options
• Cluster Auto-scaler (CA): de-facto method
• Kubernetes-ec2-autoscaler
• kube-aws-autoscaler
• Atlassian Escalator: batch or job optimized horizontal
autoscaler

Cluster auto-scaler
• Deployed As a daemon set
• Scale down cluster capacity if nodes are under-utilized
• Scale up in case of unschedulable pods
• Solely responsible for managing scaling!!
• Manages multiple ASG’s
• Only for EC2 Instance scaling

Using Spot instances with EKS

EKS worker node provisioning with EC2 Spot
• Recommend using the node labels to identify EC2 Spot
instances
• Launch EC2 Spot instances as part of AutoScaling group
• Use EC2 Spot instances best practice of mixed instance
types

Handling interruptions
Instance gets terminated / maintenance / interrupted,
workflow should be:
• Stop directing new-traffic to the pods on EC2 Spot
Instance which will be terminated.
• Gracefully drain connections of traffic already being
served
• Re-instantiate pods elsewhere in EKS cluster

Interrupt handler
• Catch EC2 Spot interruption event
• Leverage K8’s functionality to set node state to DRAIN
• Leverage K8’s schedulers to re-instantiate pods
https://github.com/kube-aws/kube-spot-termination-notice-handler

Put it all together
[mycluster].eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl
VPC
Instance
Auto Scaling group for m4.large Spot instances
Auto Scaling group for t2.medium Spot instances
Auto Scaling group for On-Demand instances
Cluster Autoscaler Daemonset
Spot Interruption handler
Daemonset

Storage with EKS

Storage
• Persistent Volume
• Persistent Volume Claims
• StatefulSets
• Storage classes

Lifecycle of a storage volume
Provisioning Binding Using Reclaiming
• Static
• Dynamic*
• Control loop watches
for PVC requests and
satisfies if PV is
available.
• For Dynamic, PVC
will provision PV
• PVC to PV binding is
one-to-one mapping
• Cluster mounts
volume based on
PVC
• Retain (default)
• Recycle
• Delete

What if I need specific volume type?
StorageClass
gp2 io1 sc1 encrypted
io1
st1
1) Admin pre-provisions
StorageClass based
on workload needs
2) End user requests for
specific volume types
(For ex, encrypted
io1 volume)
3) Control loop watches
PVC request and
allocates volume if
PV exists
MySQL Pods
4) End user creates
stateful workload

Statefulset Properties
• Network identifiers
• Persistent Storage
• Ordered graceful deployment and scaling
• Ordered graceful termination
• Ordered rolling updates
• If none of these fit your portfolio, use
Deployment or Replicaset

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
StorageClass
gp2 io1 sc1 encrypted
io1
st1
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Network
Identifiers
Ordered
Deployment

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request 3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Ordered
Scaling
mysql-4

Scheduling with Kubernetes

Controlling scheduling
Resource requirements
Constraints
• Taints Node-level
• Tolerations Pod-level
Affinity/Anti-Affinity
Volume filters
Resource filters
Topology filters
Prioritization

Limit resource usage
Container A Container B
limit
request
600m
600m
limit
request
800m
400m
⎲
⎳ Pod CPU and memory resources

Resource Quotas
apiVersion: v1
kind: Pod
metadata:
name: production
spec:
containers:
- name: nginx-pod
image: nginx
resources:
limits:
memory: "800Mi"
cpu: "800m" # 0.8 vCPU
requests:
memory: "600Mi"
cpu: "400m“ # 0.4 vCPU
Applied per Namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: production
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
ResourceQuota
defined both, so
Pod must define
both
Pod Resource Request

Taints and Tolerations
# Taint node
$ kubectl taint nodes ip-10-0-32-12.us-west-2.compute.internal
skynet=false:NoSchedule
# Tolerations
kind: Pod
spec:
tolerations:
- key: skynet
operator: Equal
value: “false”
effect: NoSchedule
[...]
Match taint to
schedule onto
tainted node

Affinity / Anti-Affinity
● Control scheduling onto nodes
○ Combine with Taints & Tolerations
● Distribute Pods across cluster
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "beta.kubernetes.io/instance-type"
operator: In
values: [“r4.large",“r4.xlarge"]

CICD for applications deployed on EKS

Jenkins

Kubernetes continuous deployment
AWS CodePipeline
AWS CodeCommit AWS CodeBuild AWS Lambda
Amazon ECR
1 2 4
3 5
6
1
Developers continuously integrate
changes into a main branch hosted
within a repo
2
Triggers an execution of the pipeline
when a new version is found, builds
a new image with build id
3
Pushes the newly built image
tagged with build id to ECR repo
4
Invokes a Lambda function to
trigger application deployment
5
Leverages Kubernetes Python SDK
to update a deployment
6
Fetches new container image
and performs a rolling update
of deployment
Developer

dashboard.eventengine.run/login
Eksworkshop.com

Amazon EKS - Elastic Container Service for Kubernetes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Amazon EKS - Elastic Container Service for Kubernetes

Similar to Amazon EKS - Elastic Container Service for Kubernetes (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Amazon EKS - Elastic Container Service for Kubernetes