Kubernetes Networking in Amazon EKS (CON412) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes Networking in Amazon EKS
C O N 4 1 2
Liwen Wu
Software Engineer
Amazon Web Services
Sri Saran Balaji Vellore Rajakumar
Software Engineer
Amazon Web Services

Agenda
• Overview of Amazon EKS Architecture
• How Kubernetes networking abstractions are implemented in
Amazon Elastic Container Service for Kubernetes (Amazon EKS)
• Pod-to-pod communication (CNI)
• Pod-to-service communication (Elastic Load Balancing, Amazon Network Load
Balancer [NLB], Kube-Proxy/IPTables)
• External-to service-communication (ingress ALB)
• How pods communicate with Kubernetes masters in Amazon EKS
• Cross-account ENIs
• Kubectl logs/execs
• Pods read/write/watch Kubernetes API servers objects

Overview of Amazon EKS Architecture
W o r k e r
V P C
Availability Zone 2
Kubelet
Kubelet
Availability Zone 1
M a s t e r
V P C
APIServer
Availability Zone 1
Availability Zone 2
Etcd
Etcd
APIServer
EKS-Owned
ENI
EKS-Owned
ENI

amazon-vpc-cni-k8s Container Networking
Interface(CNI) Tenets
• Integrates Amazon Virtual Private Cloud networking into Kubernetes
• Should use Amazon VPC networking natively to forward pod-to-pod traffic
• Use AWS routable IP addresses for Pods
• Pods is 1st class citizen in Amazon VPC networking
• There is NO on-ramp/off-ramp for
• Pod to AWS services (e.g. Amazon S3, Amazon DynamoDB) communication
• Pod to on-premises communication (e.g. VPN/direct-connect)
• Should make sure Pods have fast startup time such that: Pods/Containers MUST be able to send
and receive traffic in the matter of seconds (compare to minutes for VM)

VPC network
Amazon EC2
PodPod Pod Pod Pod
CNI networking internals - DataPlane

Life of a packet: pod1-to-pod2, inside node
EC2
Pod1
eth0
Pod2
eth0
ENI
root
veth-pod1 veth-pod2

Life of a packet: pod1-to-pod3, across nodes
ENI
EC2
node1
Pod1
eth0
Pod2
eth0
root
veth-pod1 veth-pod2
EC2
node2
Pod3
eth0
Pod4
eth0
root
veth-pod3 veth-pod4

CNI networking internals - Control plane
• Kubelet invokes CNI add or delete commands for pods
• CNI request secondary IPs from ipamD and setups networking stack for
pod
• For fast pods startup time, ipamD creates a secondary IP warm pool
with one more ENI and its IP address

CNI networking internals - Control plane
Amazon EC2

Inside pod – IP address
# ip addr show
1; lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
3: eth0@if231: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
state UP
link/ether 56:41:95:26:17:41 brd ff:ff:ff:ff:ff:ff
inet 10.0.97.30/32 brd 10.0.97.226 scope global eth0 <<<<<<< ENI's secondary IP
address
inet6 fe80::5441:95ff:fe26:1741/64 scope link

Inside pod, routes, static ARP
# ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0
# arp -a
? (169.254.1.1) at 2a:09:74:cd:c4:62 [ether] PERM on eth0

On host side
# ip route show
10.0.96.0/19 dev eth0 proto kernel scope link src 10.0.104.183
10.0.97.30 dev aws8db0408c9a8 scope link <------------------------Pod's IP
10.0.97.159 dev awsbcd978401eb scope link
10.0.97.226 dev awsc2f87dc4cdd scope link
10.0.102.98 dev aws4914061689b scope link
...
# ip route show table eni-1
10.0.96.1 dev eth1 scope link
# ip rule list
0: from all lookup local
512: from all to 10.0.97.30 lookup main <---------- to Pod's traffic
1025: not from all to 10.0.0.0/16 lookup main
1536: from 10.0.97.30 lookup eni-1 <-------------- from Pod's traffic

Kubernetes service
• A logical set of pods
• A policy by which to access them

Kubernetes service
Service
PodPodPod

Services in Kubernetes
[ec2-user@ip-172-31-9-36 ~]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2d
[ec2-user@ip-172-31-9-36 ~]$ kubectl describe svc
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.100.0.1
Port: https 443/TCP
TargetPort: 443/TCP
Endpoints: 192.168.119.102:443,192.168.154.135:443
Session Affinity: ClientIP
Events: <none>

IPtables (pod to services)
kubectl describe svc
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.100.0.1
Port: https 443/TCP
TargetPort: 443/TCP
Endpoints: 192.168.131.23:443,192.168.85.200:443  Endpoint IPs
Session Affinity: ClientIP
Events: <none>

Implement Kubernetes service by
Kube-proxy, Linux IPtables
• Kube-proxy
• Watch services, end-points
• Program Linux IPtables
• Pod to service
• IPtable, pre-routing phase
• Pod  service IP  IPtables  one of end-point IPs

Kube-proxy, Linux IPtables
Amazon EC2
Pod
Pod
Pod
Kube-proxy
API Server

IPtables (pod to services)
*nat
:PREROUTING ACCEPT [1:60]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES // 1
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -
m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-SIZTKALRBUTOHR3N -s 192.168.131.23/32 -m comment --comment
"default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-SIZTKALRBUTOHR3N -p tcp -m comment --comment "default/kubernetes:https" -m
recent --set --name KUBE-SEP-SIZTKALRBUTOHR3N --mask 255.255.255.255 --rsource -m tcp
-j DNAT --to-destination 192.168.131.23:443 //4
-A KUBE-SEP-U7WSV5R4I437O3C7 -s 192.168.85.200/32 -m comment --comment
"default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-U7WSV5R4I437O3C7 -p tcp -m comment --comment "default/kubernetes:https" -m
recent --set --name KUBE-SEP-U7WSV5R4I437O3C7 --mask 255.255.255.255 --rsource -m tcp
-j DNAT --to-destination 192.168.85.200:443 //6

IPtables (Pod to Services)
-A KUBE-SERVICES -d 10.100.0.10/32 -p udp -m comment --comment "kube-system/kube-
dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.100.0.10/32 -p tcp -m comment --comment "kube-system/kube-
dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.100.0.1/32 -p tcp -m comment --comment
"default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-
NPX46M4PTMTKRN6Y // 2
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m
recent --rcheck --seconds 10800 --reap --name KUBE-SEP-SIZTKALRBUTOHR3N --mask
255.255.255.255 --rsource -j KUBE-SEP-SIZTKALRBUTOHR3N //3
recent --rcheck --seconds 10800 --reap --name KUBE-SEP-U7WSV5R4I437O3C7 --mask
255.255.255.255 --rsource -j KUBE-SEP-U7WSV5R4I437O3C7 // 5
statistic --mode random --probability 0.50000000000 -j KUBE-SEP-SIZTKALRBUTOHR3N
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j
KUBE-SEP-U7WSV5R4I437O3C7

Kubernetes DNS Pod (kube-dns)
• Kubernetes schedules a DNS Pod (kube-dns) and Service on
the cluster
• Configures the Kubelets to tell individual containers to use
the DNS Service’s IP to resolve DNS names
• DNS Pod communicate with Kubernetes Service IP (e.g
10.100.0.1) and build map of Service Name and Service IP
[ec2-user@ip-172-31-9-36 ~]$ kubectl get pod kube-dns-fcd468cb-cbz8z -n kube-system
NAME READY STATUS RESTARTS AGE
kube-dns-fcd468cb-cbz8z 3/3 Running 0 5d

Life of a packet: pod (kube-dns)-to-service
(Kubernetes service)
ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
veth-dns-pod veth-pod2
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135I
Customer VPC EKS VPC
10.100.0.1
APIServer
Availability Zone 1
APIServer
Availability Zone 2

ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135
APIServer
Availability Zone 1
APIServer
Availability Zone 2

ENI
EC2
node1
Kube-dns
eth0
Pod2
eth0
root
EKS-Owned ENI
192.168.119.102
EKS-Owned EN
192.168.154.135I
APIServer
Availability Zone 1
APIServer
Availability Zone 2

Amazon Classic Load Balancer
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
annotations: {}
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer

Classic Load Balancer
Pod
Pod

Amazon Network Load Balancer (NLB) (Linux IPtables-as-a-service for VPCS)
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer

Amazon Network Load Balancer (NLB)
Pod
Pod

Kubernetes ingress
• A collection of rules
• Allow inbound connections to reach the cluster services

Kubernetes ingress
Service (foo)
Pod Pod Pod
Service (bar)
Pod Pod Pod
Foo.example.com example.com/bar
ingress

Ingress in Kubernetes
#apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echoserver
namespace: echoserver
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/subnets: subnet-0061ab916d8e0f34f
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: echoserver
servicePort: 80

Ingress in Kubernetes
# kubectl describe ingress -n echoserver
Name: echoserver
Namespace: echoserver
Address: 23604d3e-echoserver-echose-2ad7-1066162608.us-west-
2.elb.amazonaws.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/ echoserver:80 (<none>)
Annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/subnets: subnet-0061ab916d8e0f34f,subnet-
08dc85488dba37eda,

Service reached through ingress in Kubernetes
# kubectl get service -n echoserver
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver ClusterIP 10.100.193.12 <none> 80/TCP 1d
# kubectl describe service echoserver -n echoserver
Name: echoserver
Namespace: echoserver
Selector: app=echoserver
Type: ClusterIP
IP: 10.100.193.12
Port: <unset> 80/TCP
TargetPort: 8080/TCP
Endpoints: 192.168.164.97:8080
Session Affinity: None
Events: <none>

Implement Kubernetes ingress by
Amazon Application Load Balancer (ALB)

Kubernetes cluster
Implement Kubernetes ingress by
L7 load balancers (for example, Envoy, Ngnix)
Service (bar)
Pod
Pod
Pod
Service (foo)
Pod
Pod
Pod
L7 Envoy, Ngnix
L7 Envoy, Ngnix

Kubernetes Exec and Logs implementation
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Communication
across VPCs
Internet

Troubleshooting Networking issues
• Misconfigured control plane
security group
• Control plane security group is assigned to
ENIs created in the worker node subnets.
• When launching worker nodes, control
plane security group is configured to receive
packets from worker nodes.
• if different control plane security group is
specified while creating worker nodes, pods
will not be able to communicate with
master
W o r k e r
V P C
Kubelet
ENI
Pods
Worker
Node
Worker Node
Security Group
Control Plane
Security Group

Troubleshooting Networking issues
• Amazon VPC related issues
• Deleting subnets in your Amazon VPC
• Removing Ingress and Egress required for Master and Worker node
communication.
• Reaching ENI limits for an AWS Account.
• Exhausting IPs available in the control plane subnets.
• Incorrect permissions on the role could stop Amazon EKS from
managing Kubernetes clusters.
• Use Managed policy provided by Amazon EKS.
• Avoid attaching deny permissions on APIs required by Amazon EKS for
managing ENIs in your Amazon VPC.

Thank you!

Kubernetes Networking in Amazon EKS (CON412) - AWS re:Invent 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kubernetes Networking in Amazon EKS (CON412) - AWS re:Invent 2018

Similar to Kubernetes Networking in Amazon EKS (CON412) - AWS re:Invent 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Kubernetes Networking in Amazon EKS (CON412) - AWS re:Invent 2018