Kubernetes networking

- Kubernetes Networking
- Paul Sim
APAC Technical Lead
paul.sim@canonical.com

● Basic concept
● Expose Kubernetes Service
○ hostNetwork
○ hostPort
○ nodePort
○ LoadBalancer
○ Ingress
● Kubernetes network plugins(CNI)
○ Flannel
○ Calico
Index

● Kubernetes network model[1]
○ Pods on a node can communicate with all pods on all nodes without NAT
○ Agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that
node
○ Pods in the host network of a node can communicate with all pods on all nodes without
NAT
● Each pod has its own namespace and shares it among containers running on the pod
Basic concept
[1] https://kubernetes.io/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model

Basic concept
Worker
- Pod to Pod on a node(CNI : flannel)
cni0
Pod1 namespace Pod2 namespace
veth** veth**
eth* eth*
Container Container Container
VETH pair

Basic concept
- To determine a namespace of a pod and run a command in the namespace
ubuntu@worker-1:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9ccf18aa9c03 64f5d945efcc "sleep 86400" 32 minutes ago Up 32 minutes
k8s_busybox_busybox2_default_65195239-7cac-11e9-a0ad-525400acdead_0
a8a7b94d56de k8s.gcr.io/pause:3.1 "/pause" 32 minutes ago Up 32 minutes
k8s_POD_busybox2_default_65195239-7cac-11e9-a0ad-525400acdead_0
41a80dbe0356 64f5d945efcc "sleep 86400" About an hour ago Up About an hour
k8s_busybox-2_busybox-multi_default_5a044af0-7ca9-11e9-a0ad-525400acdead_0
…
ubuntu@worker-1:~$ sudo nsenter -t $(sudo docker inspect --format '{{ .State.Pid }}' 41a80dbe0356) -n ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 5e:69:ad:2e:69:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.1.12/24 scope global eth0
ubuntu@worker-1:~$ip addr
…
9: veth24718b0b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 72:41:50:c6:67:73 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::7041:50ff:fec6:6773/64 scope link

Basic concept
Worker-1
- Pod to Pod between remote nodes
Worker-2
flannel.1
10.244.0.0/32
flannel.1
10.244.1.0/32
Pod1
eth*
10.244.0.11/24
Container
eth0
Pod2
Container
eth0
Depends on CNI
ubuntu@worker-1:~$ route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.30.1.254 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
eth*
10.244.1.13/24
cni0 cni0

Basic concept : ClusterIP
Worker-1 Worker-2
Pod1
influxdb
Service “my-influxdb”
Pod2
influxdb
Pod3
influxdb
Pod4
nginx
ClusterIP
Client Client
Service B
ClusterIP

Basic concept : ClusterIP
ubuntu@master-1:~$kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-influxdb-7d7c56fd6c-hkxcm 1/1 Running 0 21m 10.1.71.7 juju-2d6bac-2 <none> <none>
my-influxdb-7d7c56fd6c-nrw96 1/1 Running 0 21m 10.1.52.16 juju-2d6bac-1 <none> <none>
my-influxdb-7d7c56fd6c-tlcv2 1/1 Running 0 21m 10.1.71.6 juju-2d6bac-2 <none> <none>
ubuntu@master-1:~$kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 57m <none>
my-influxdb ClusterIP 10.152.183.62 <none> 8086/TCP 21m run=my-influxdb
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.62 /* default/my-influxdb: cluster
IP */ tcp dpt:8086
0 0 KUBE-SVC-HFIBEWMHTJTJLTVI tcp -- * * 0.0.0.0/0 10.152.183.62 /* default/my-influxdb: cluster
IP */ tcp dpt:8086
Chain KUBE-SVC-HFIBEWMHTJTJLTVI (1 references)
0 0 KUBE-SEP-ECEWNW4CQDEBZ5IC all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability
0.33332999982
0 0 KUBE-SEP-DTNINEUP5BSSI6C5 all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability
0.50000000000
0 0 KUBE-SEP-ULXQYDTSZVM5BZGA all -- * * 0.0.0.0/0 0.0.0.0/0

Basic concept : kube-proxy mode
● Kube-proxy mode :
○ User space mode
■ Since v1.0
○ Iptables mode
■ Default mode since v1.2
○ IPVS mode
■ Since v1.8
○ kube-proxy --proxy-mode ProxyMode
○ $curl localhost:10249/proxyMode
○ https://kubernetes.io/docs/concepts/services-networking/service/

Expose Kubernetes Service : hostNetwork
Worker-1
Pod1
Influxd
eth0
172.30.1.187
ubuntu@worker-1:~$cat influxdb-hostnetwork.yaml
apiVersion: v1
kind: Pod
metadata:
name: influxdb
spec:
hostNetwork: true
containers:
- name: influxdb
image: influxdb
Client
http://172.30.1.187:8086/ping
ubuntu@worker-1:~$sudo netstat -anp | grep LISTEN
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 4855/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 10134/kube-proxy
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1748/sshd
tcp 0 0 127.0.0.1:8088 0.0.0.0:* LISTEN 15123/influxd
tcp6 0 0 :::10250 :::* LISTEN 4855/kubelet
tcp6 0 0 :::10256 :::* LISTEN 10134/kube-proxy
tcp6 0 0 :::8086 ::::* LISTEN 15123/influxd

Expose Kubernetes Service : hostPort
Worker-1
Pod1
Container
cni0
eth0
172.30.1.187
eth*
10.244.1.18/24
Client
Iptables rules applied
ubuntu@worker-1:~$cat influxdb-hostPort.yaml
apiVersion: v1
kind: Pod
metadata:
name: influxdb
spec:
containers:
- name: influxdb
image: influxdb
ports:
- containerPort: 8086
hostPort: 8086
http://172.30.1.187:8086/ping

ubuntu@worker-1:~$sudo nsenter -t $(sudo docker inspect --format '{{ .State.Pid }}' 006ed5ed86b8) -n netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8088 0.0.0.0:* LISTEN 19411/influxd
tcp6 0 0 :::8086 :::* LISTEN 19411/influxd
ubuntu@worker-1:~$sudo iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 2 packets, 530 bytes)
1127 346K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
22 3053 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
19 2873 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain CNI-HOSTPORT-DNAT (2 references)
8 480 CNI-DN-01f966da5c91ab03b1317 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "cbr0" id:
"b91cd157f55a8cd9ecd3744c56375e78f6752033fd50ec8f3762a34dd3f64bb8" */ multiport dports 8086
Chain CNI-DN-01f966da5c91ab03b1317 (1 references)
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.244.1.18 0.0.0.0/0 tcp dpt:8086
0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:8086
8 480 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8086 to:10.244.1.18:8086
Expose Kubernetes Service : hostPort

Worker-1
Pod1
nginx
Client
eth0
172.30.1.185:30180
10.244.0.16:80
Iptables rules applied
ubuntu@worker-1:~$ sudo netstat -anp | grep LISTEN
...
tcp6 0 0 :::6443 :::* LISTEN 2531/kube-apiserver
tcp6 0 0 :::10251 :::* LISTEN 2477/kube-scheduler
tcp6 0 0 :::10252 :::* LISTEN 4577/kube-controlle
Expose Kubernetes Service : nodePort

externalTrafficPolicy: Cluster
ubuntu@worker-1:~$cat my-nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
ubuntu@worker-1:~$cat my-nginx-service-nodePort.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
type: NodePort
ports:
- port: 80
nodePort: 30180
name: http
selector:
run: my-nginx

Worker-1 Worker-2
eth0
172.30.1.185:30180
Pod2
nginx
eth0
172.30.1.187:30180
eth*
10.244.1.21:80
Client
http://172.30.1.187:30180
Pod1
nginx
eth*
10.244.0.15:80
http://172.30.1.185:30180
flannel.1
flannel.1

janghoon@janghoon-gram:~$ curl http://172.30.1.185:30180/host.html
Local IP : 10.244.0.15
Local IP : 10.244.1.21
Local IP : 10.244.0.15
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.1.21

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
10.244.0.0/32
SRC: x.x.x.x
DST :172.30.1.185

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
10.244.0.0/32
SRC: x.x.x.x
DST :10.244.0.15
cni0
10.244.0.1

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
10.244.0.0/32
SRC: 10.244.0.1
DST :10.244.0.15
cni0
10.244.0.1

externalTrafficPolicy: Cluster - to Pod2
Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
10.244.0.0/32
SRC: x.x.x.x
DST :10.244.1.21

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
10.244.0.0/32
SRC: 10.244.0.0
DST :10.244.1.21
Kernel IP routing table
0.0.0.0 10.132.0.1 0.0.0.0 UG 100 0 0 ens4
10.244.1.0 10.244.1.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.0.0 0.0.0.0 255.255.255.0 UG 0 0 0 cni0
flannel.1
10.244.1.0/32

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
x.x.x.x
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
10.244.0.0/32
SRC: 10.244.0.0
DST :10.244.1.21
flannel.1
10.244.1.0/32

…
6 360 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE
match dst-type LOCAL
Chain KUBE-NODEPORTS (1 references)
8 480 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http */ tcp dpt:30180
8 480 KUBE-SVC-SV7AMNAGZFKZEMQ4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http */ tcp dpt:30180
Chain KUBE-MARK-MASQ (16 references)
8 480 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-SVC-SV7AMNAGZFKZEMQ4 (2 references)
2 120 KUBE-SEP-UFBDZJYLV6AP2WNO all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.50000000000
6 360 KUBE-SEP-J5DVFQH7UKWOKPLS all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SEP-UFBDZJYLV6AP2WNO (1 references)
0 0 KUBE-MARK-MASQ all -- * * 10.244.0.16 0.0.0.0/0
2 120 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.0.15:80
Chain KUBE-SEP-J5DVFQH7UKWOKPLS (1 references)
6 360 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.1.21:80

Chain POSTROUTING (policy ACCEPT 10 packets, 801 bytes)
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
4342 371K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
Chain KUBE-POSTROUTING (1 references)
10 750 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

externalTrafficPolicy: Local
Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
http://172.30.1.187:30180
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1

Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.0.15
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.1.21
Local IP : 10.244.1.21

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
SRC: x.x.x.x
DST :172.30.1.185

Worker-1 Worker-2
eth0
172.30.1.185
Pod2
nginx
eth0
172.30.1.187
eth*
10.244.1.21
Client
Pod1
nginx
eth*
10.244.0.15
http://172.30.1.185:30180
flannel.1
flannel.1
SRC: x.x.x.x
DST :10.244.0.15

...
3 180 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE
match dst-type LOCAL
Chain KUBE-NODEPORTS (1 references)
0 0 KUBE-MARK-MASQ tcp -- * * 127.0.0.0/8 0.0.0.0/0 /* default/my-nginx:https */ tcp dpt:31443
0 0 KUBE-XLB-WJQIRF6WKN5OOCTI tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:https */ tcp dpt:31443
0 0 KUBE-MARK-MASQ tcp -- * * 127.0.0.0/8 0.0.0.0/0 /* default/my-nginx:http */ tcp dpt:30180
5 300 KUBE-XLB-SV7AMNAGZFKZEMQ4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http */ tcp dpt:30180
Chain KUBE-XLB-SV7AMNAGZFKZEMQ4 (1 references)
0 0 KUBE-SVC-SV7AMNAGZFKZEMQ4 all -- * * 10.244.0.0/16 0.0.0.0/0 /* Redirect pods trying to reach external loadbalancer VIP to clusterIP */
5 300 KUBE-SEP-UFBDZJYLV6AP2WNO all -- * * 0.0.0.0/0 0.0.0.0/0 /* Balancing rule 0 for default/my-nginx:http */
Chain KUBE-SEP-UFBDZJYLV6AP2WNO (2 references)
5 300 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.244.0.15:80

ubuntu@worker-1:~$cat my-nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
ubuntu@worker-1:~$cat my-nginx-service-lb.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
type: LoadBalancer
ports:
- port: 80
name: http
selector:
run: my-nginx
Expose Kubernetes Service : LoadBalancer

ubuntu@worker-1:~$kubectl describe deployment my-nginx
Name: my-nginx
Namespace: default
CreationTimestamp: Tue, 28 May 2019 08:57:33 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision:
1
Selector: run=my-nginx
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0
unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 5% max unavailable, 25% max surge
Pod Template:
Labels: run=my-nginx
Containers:
my-nginx:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
...
ubuntu@worker-1:~$kubectl describe svc my-nginx
Name: my-nginx
Namespace: default
Labels: run=my-nginx
Annotations: <none>
Selector: run=my-nginx
Type: LoadBalancer
IP: 10.152.183.45
LoadBalancer Ingress: 35.201.130.67
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 31213/TCP
Endpoints: 10.1.41.3:80,10.1.69.2:80
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 32602
Events: <none>

ubuntu@worker-1~$ sudo netstat -anp | grep LISTEN
...
tcp 0 0 252.252.128.1:53 0.0.0.0:* LISTEN 10142/dnsmasq
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 744/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1404/sshd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 22129/nginx: master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 22129/nginx: master
tcp6 0 0 :::18080 :::* LISTEN 22129/nginx: master
tcp6 0 0 :::10254 :::* LISTEN 22101/nginx-ingress
tcp6 0 0 :::111 :::* LISTEN 17866/rpcbind
tcp6 0 0 :::22 :::* LISTEN 1404/sshd
tcp6 0 0 :::443 ::* LISTEN 22129/nginx: master
...

Worker-1 Worker-2
cni0
eth0
::: 31213
Pod2
nginx
eth0
::: 31213
eth*
10.1.69.2
Pod1
nginx
eth*
10.1.41.3
Worker-3
eth0
::: 31213
NodePort, 31213 opened but not used in GCP.
LoadBalancer from
Cloud provider
Client
x.x.x.x
cni0
SRC: x.x.x.x
DST :35.201.130.67:80
SRC: x.x.x.x
DST :10.1.41.3:80
http://35.201.130.67

LoadBalancer on GCP
Worker-1 Worker-2
cni0
eth0
10.140.15.198: 3xxxx
Pod2
nginx
eth0
10.140.15.199:3xxxx
eth*
10.1.69.2
Pod1
nginx
eth*
10.1.41.3
Worker-3
eth0
10.140.15.202:3xxxx
LoadBalancer from
Cloud provider
Client
http://35.201.130.67
LoadBalancer Ingress
cni0

ubuntu@juju-70b5fa-5:~$ sudo iptables -vnL -t nat
6529 433K DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
...
0 0 KUBE-MARK-MASQ tcp -- * * !10.1.0.0/16 10.152.183.45 /* default/my-nginx:http cluster IP */ tcp dpt:80
0 0 KUBE-SVC-SV7AMNAGZFKZEMQ4 tcp -- * * 0.0.0.0/0 10.152.183.45 /* default/my-nginx:http cluster IP */ tcp dpt:80
0 0 KUBE-FW-SV7AMNAGZFKZEMQ4 tcp -- * * 0.0.0.0/0 35.201.130.67 /* default/my-nginx:http loadbalancer IP */ tcp dpt:80
...
Chain KUBE-FW-SV7AMNAGZFKZEMQ4 (1 references)
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http loadbalancer IP */
0 0 KUBE-SVC-SV7AMNAGZFKZEMQ4 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http loadbalancer IP */
0 0 KUBE-MARK-DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-nginx:http loadbalancer IP */
Chain KUBE-SVC-SV7AMNAGZFKZEMQ4 (3 references)
0 0 KUBE-SEP-GVMZRJY5UIB545AM all -- * * 0.0.0.0/0 0.0.0.0/0 statistic mode random probability 0.50000000000
0 0 KUBE-SEP-XP4T4NJ7CIG4MQ7Q all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SEP-GVMZRJY5UIB545AM (1 references)
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.1.41.3:80
Chain KUBE-SEP-XP4T4NJ7CIG4MQ7Q (1 references)
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp to:10.1.69.2:80

LoadBalancer on GCP
Worker-1 Worker-2
cni0
eth0
10.140.15.198: 3xxxx
Pod2
nginx
eth0
10.140.15.199:3xxxx
eth*
10.1.69.2
Pod1
nginx
eth*
10.1.41.3
Worker-3
eth0
10.140.15.202:3xxxx
LoadBalancer from
Cloud provider
Client
http://35.201.130.67
LoadBalancer Ingress
Worker-3 health check failed.
cni0

Worker-1
Service A
Ingress
nginx
Pod1
Container
Pod2
Container
Service B
Client
Expose Kubernetes Service : Ingress

Ingress
ubuntu@worker-1:~$ cat coffee.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: coffee
spec:
replicas: 2
selector:
matchLabels:
app: coffee
template:
metadata:
labels:
app: coffee
spec:
containers:
- name: coffee
image: nginxdemos/hello:plain-text
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: coffee-svc
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: coffee
ubuntu@worker-1:~$ cat tee.yaml
kind: Deployment
metadata:
name: tea
spec:
replicas: 3
selector:
matchLabels:
app: tea
template:
metadata:
labels:
app: tea
spec:
containers:
- name: tea
image: nginxdemos/hello:plain-text
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: tea-svc
labels:
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:

ubuntu@worker-1:~$ cat cafe-ingress.yaml
kind: Ingress
metadata:
name: cafe-ingress
spec:
tls:
- hosts:
- cafe.example.com
secretName: cafe-secret
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
ubuntu@worker-1:~$ cat cafe-secret.yaml
[ omitted ]
https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/complete-example

Worker -1 Worker -2
Ingress Controller
eth0
Pod1
Worker -3
Pod2
eth0 eth0
Pod3 Pod4
Ingress Controller Ingress Controller
Default backend
Client
hostNetwork: true

ubuntu@master-1:~$ kubectl get pods --all-namespaces -o wide
...
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-5b8b477c-zxqgl 1/1 Running 1 13h 10.1.45.16 worker-3
<none> <none>
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-lb8lv 1/1 Running 1 13h 172.30.1.173 worker-3
<none> <none>
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-mhl4z 1/1 Running 1 13h 172.30.1.172 worker-2
<none> <none>
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-wghdf 1/1 Running 0 52m 172.30.1.176 worker-1
<none>
...
ubuntu@master-1:~$ kubectl get daemonset --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
NODE SELECTOR AGE
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker 3 3 3 3 3 juju-
application=kubernetes-worker 13h

Worker -1 Worker -2
Ingress Controller
eth0
Pod1
Worker -3
Pod2
eth0 eth0
Pod1 Pod2
Ingress Controller Ingress Controller
Default backend
Client
Pod1 : app: tea
http://cafe.example.com/tea
Pod2 : app: coffee
http://cafe.example.com/coffee

ubuntu@master-1:~$ kubectl exec -it nginx-ingress-controller-kubernetes-worker-lb8lv -n ingress-nginx-kubernetes-
worker -- cat /etc/nginx/nginx.conf
...
location /coffee {
set $namespace "default";
set $ingress_name "cafe-ingress";
set $service_name "coffee-svc";
set $service_port "80";
set $location_path "/coffee";
...
location /tea {
set $namespace "default";
set $ingress_name "cafe-ingress";
set $service_name "tea-svc";
set $service_port "80";
set $location_path "/tea";
...

Kubernetes network plugins(CNI)
● Kubernetes Networking Model
Kubernetes imposes the following fundamental requirements on any networking
implementation (barring any intentional network segmentation policies):
● pods on a node can communicate with all pods on all nodes without NAT
● agents on a node (e.g. system daemons, kubelet) can communicate with all pods on
that node
● pods in the host network of a node can communicate with all pods on all nodes
without NAT
- https://kubernetes.io/docs/concepts/cluster-administration/networking/
● Container Network Interface(CNI) Network Providers for Kubernetes
- Comparison table
- https://docs.google.com/spreadsheets/d/1qCOlor16Wp5mHd6MQxB5gUEQILnijyDLIExEpq
mee2k/edit?usp=sharing
- Benchmark
- https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-10gbit-s-netwo
rk-36475925a560
- CDK supports
- Flannel
- Calico
- Weave??

Worker -1
Kubernetes network plugins(CNI) - Flannel
Pod1
flanneld
Pod2
flanneld
ubuntu@u-VM-worker-1:~$ kubectl describe configmap kube-flannel-cfg -
n kube-system
...
net-conf.json:
----
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
...
ubuntu@etc-0:~$ etcdctl get /coreos.com/network/config
{"Network": "10.1.0.0/16", "Backend": {"Type": "vxlan"}}
1. Flanneld
a. Responsible for flannel.x port and routing table for cluster networking
b. running on all nodes as a service(CDK) or as a daemonset(manual deployment)
c. Backend : UDP, vxlan
d. https://github.com/coreos/flannel
2. flannel.x : a vxlan network port
Etcd cluster
etcd
ubuntu@worker-1~$ sudo ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 3a:b4:eb:44:3a:4a brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 1 local 172.30.1.172 dev eth0 srcport 0 0 dstport 8472 nolearning ttl inherit ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64
numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
eth0
cni0
flannel.1

● Flannel Packer flow
○ E.g. ping 10.7.1.12 -> 10.1.45.21
Worker-1 Worker-2
cni0
10.1.7.1
eth0
172.30.1.172
eth0
172.30.1.173:8472
Pod1
eth*
10.1.7.12
Pod2
eth*
10.1.45.21
flannel.1
...
10.1.7.0 0.0.0.0 255.255.255.0 UG 0 0 0 cni0
10.1.32.0 10.1.32.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.45.0 10.1.45.0 255.255.255.0 UG 0 0 0 flannel.1
...
Vxlan encapsulated
SRC: 10.1.7.12
DST :10.1.45.21
cni0
10.1.45.1
flannel.1

● Flannel Packer flow
○ E.g. ping 10.7.1.12 -> 10.1.45.21
Worker-1 Worker-2
cni0
10.1.7.1
Pod1
eth*
10.1.7.12
Pod2
eth*
10.1.45.21
flannel.1
Vxlan
decapsulated
SRC: 10.1.7.12
DST :10.1.45.21
cni0
10.1.45.1
...
10.1.7.0 10.1.7.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.32.0 10.1.32.0 255.255.255.0 UG 0 0 0 flannel.1
10.1.45.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
...
flannel.1
eth0
172.30.1.172
eth0
172.30.1.173:8472

Worker -1
Kubernetes network plugins(CNI) - Calico
Pod1
calixxxxxxxxxxx
Calico Pod
BIRD
1. Calico
a. L3 routing based.
i. https://www.projectcalico.org/
b. BIRD : BGP routing daemon. BIRD runs on every host as a Daemonset.
c. Confd: monitors the etcd datastore for any changes to BGP configuration and write
it to a file. BIRD reads this configuration file and updates changes.
d. Felix : writes the routing tables and iptables for network policy.
Etcd cluster
etcd
eth0
Confd
Felix
eth0
VETH pair

● Calico Packer flow
○ E.g. ping 10.244.1.4 -> 10.244.2.3
Worker-1 Worker-2
eth0
172.30.1.171
eth0
172.30.1.172
Pod1
eth*
10.244.1.4
Pod2
eth*
10.244.2.3
...
0.0.0.0 172.30.1.254 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 172.30.1.192 255.255.255.0 UG 0 0 0 eth0
10.244.1.0 0.0.0.0 255.255.255.0 U 0 0 0 *
10.244.1.4 0.0.0.0 255.255.255.255 UH 0 0 0 calibcca51a88a1
10.244.2.0 172.30.1.172 255.255.255.0 UG 0 0 0 eth0
...
calibcca51a88a1
SRC :10.244.1.4
DST :10.244.2.3
cali5285e8d3fd4

● Calico Packer flow
○ E.g. ping 10.244.1.4 -> 10.244.2.3
Worker-1 Worker-2
eth0
172.30.1.171
eth0
172.30.1.172
Pod1
eth*
10.244.1.4
Pod2
eth*
10.244.2.3
...
10.244.1.0 172.30.1.171 255.255.255.0 UG 0 0 0 eth0
10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 *
10.244.2.3 0.0.0.0 255.255.255.255 UH 0 0 0 cali5285e8d3fd4
10.244.3.0 172.30.1.173 255.255.255.0 UG 0 0 0 eth0
172.30.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
...
calibcca51a88a1 cali5285e8d3fd4
SRC :10.244.1.4
DST :10.244.2.3

Kubernetes networking

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kubernetes networking

Similar to Kubernetes networking (20)

More from Sim Janghoon

More from Sim Janghoon (6)

Recently uploaded

Recently uploaded (20)

Kubernetes networking