COSCUP2018
x
Outline
What network functions about k8s
How does above function implement
What is the challenge about k8s

COSCUP2018
x
What network function kubernetes
providers?

COSCUP2018
x
What
Container Network
○ Connectivity
○ DNS
Kubernetes services

COSCUP2018
x
Do You Know How Container Works?

COSCUP2018
x
Containersvs.VMs
Containers are isolated, but share OS
and where appropriate bins/libraries

COSCUP2018
x
HowDockerWorks
We know docker is isolated, but how
does it works?
Linux kernel support the Namespaces
mechanisms to partition kernel
resources to different processes

COSCUP2018
x
HowDockerWorks
Mount namespaces
IPC namespaces
PID namespaces
Network namespaces
User namespaces
UTS namespaces
○ Unix Time System

COSCUP2018
x
NetworkNamespace
Isolate the network functions.
Including the
○ Network interfaces
○ Routing rules
○ Netfilter (iptables)

COSCUP2018
x
Linux Host

COSCUP2018
x
docker0
Linux Host Linux Host

COSCUP2018
x
docker0 docker0
ns1
Linux Host Linux Host Linux Host

COSCUP2018
x
docker0 docker0
docker0
ns1
ns1
veth1
veth0
Linux Host

COSCUP2018
x
docker0 docker0
docker0docker0
ns1
ns1ns1
vth1
vth0
veth
eth0
Linux HostLinux Host

COSCUP2018
x
Before we talk about service, we must
know why service exist.

COSCUP2018
x
Pods/Deployments
We can deploy our applications as a
containers in the kubernetes.
There’re many kind of the container
we can deploy
○ Pod
○ Deployment
○ Statefulset
○ DaemonSet

COSCUP2018
x
Deployment
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
Deployment:
○ Ngnix
○ Replica: 3
10.123.234.56 10.123.234.57 10.123.234.58

COSCUP2018
x
Access
How we application access those
Nginx servers?
By IP address
○ 10.123.234.56:80
○ 10.123.234.57:80
○ 10.123.234.58:80
What’s the problem

COSCUP2018
x
Deployment
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
Deployment:
○ Ngnix
○ Replica: 3
10.123.234.56 10.123.234.57 10.123.234.75

COSCUP2018
x
Access
How we application access those
Nginx servers?
By IP address
○ 10.123.234.56:80
○ 10.123.234.57:80
○ 10.123.234.58:80
○ 10.123.234.75:80
It’s not easy for our application to
handle those ip-changed situation.

COSCUP2018
x
The Service is used to solve this problem.

COSCUP2018
x
Service
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
10.123.234.56 10.123.234.57 10.123.234.58
App
Service Nginx

COSCUP2018
x
Service
Application to Service
○ We use the DNS to access the service.
○ $(service).$(namespace).cluster.local
Service to Pods
○ Service maintains all IP addresses of all
Pods.
○ We call it endpoints

COSCUP2018
x
Service
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
10.123.234.56 10.123.234.57 10.123.234.58
App
Service Nginx
nginx.default
endpoints

COSCUP2018
x
How does kubernetes implements those
functions?

COSCUP2018
x
What
Container Network Connectivity
○ Container Network Interface(CNI)
Kubernetes Services
○ There’re many implementation we can
choose

COSCUP2018
x
ContainerNetworkInterface

COSCUP2018
x
Cloud Native Computing Foundation
Project
Consists of a specification and
libraries.
Configure network interfaces in Linux
containers
Concerns itself only with network
connectivity of containers
○ Create/Remove

COSCUP2018
x
Removing allocated resources when
the container is deleted

COSCUP2018
x
Who is using CNI?

COSCUP2018
x
FromtheGITHUB
l rkt - container engine
l Kubernetes - a system to simplify container operations
l OpenShift - Kubernetes with additional enterprise
features
l Cloud Foundry - a platform for cloud applications
l Apache Mesos - a distributed systems kernel
l Amazon ECS - a highly scalable, high performance
container management service

COSCUP2018
x
So, How to use the CNI?

COSCUP2018
x
StepbyStep
Create a kubernetes cluster
Setup your CNI plugin
Deploy your first Pod

COSCUP2018
x
Just follow the installation to install the
kubernetes

COSCUP2018
x
How do we install the CNI?

COSCUP2018
x

COSCUP2018
x
Handbyhand
In the kubelet, we have the following
parameters for CNI.
--cni-bin-dir
○ /opt/cni/bin
--cni-conf-dir
○ /etc/cni/net.d/
We should config the CNI for every
k8s nodes.

COSCUP2018
x
Let Deploy a Pod

COSCUP2018
x
Beforewestart
Pod
○ A collection of containers

COSCUP2018
x
Steps
Load the Pod config
○ Multiple containers
Find a node to deploy the pod
Create a Pause container

COSCUP2018
x
Linux Host
Pause
Container

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config

COSCUP2018
x
Linux Host
Pause
Container
Load the CNI config from /etc/cni/net.d/…

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config
Execute the CNI

COSCUP2018
x
Callthebinary
Load the binary from the config
Find the binary from the /opt/cni/bin/

COSCUP2018
x
Linux Host
Pause
Container
Call the /opt/cni/bin/flannel
Network
Connectivity

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config
Execute the CNI
Create target containers and attach to
Pause container

COSCUP2018
x
Linux Host
Pause
Container
Network
Connectivity
Busybox

COSCUP2018
x
Attachtoothercontainer
docker run –net=…
○ bridge
○ host
○ containerID

COSCUP2018
x
Linux Host
Pause
Container
Network
Connectivity
Busybox
Pod

COSCUP2018
x
Kubernetesservice
There’re three implementation now.
○ User-space Kube-Proxy
○ Kernel-space iptables (default)
○ Kernel-space ipvs
We use the iptables to explain how
service(ClusterIP) works

COSCUP2018
x
LAB
Get the Service
○ kubectl get service

COSCUP2018
x
LAB
Get the endpoints
○ kubectl get endpoints

COSCUP2018
x
LAB
Get the pod ip address
○ kubectl get pods –o wide

COSCUP2018
x
Now.Trytofetchthenginx
Service name is k8s-nginx-cluster
Use the nslookup to lookup the IP
○ nslookup k8s-nginx-cluster
○ nslookup k8s-nginx-cluster.default
■ default is the namespace of the service

COSCUP2018
x
Trytofetchthenginx
In the pod: curl k8s-nginx-cluster

COSCUP2018
x
Howitworks?
First, we can get the VIP from the
hostname.
○ It’s just a DNS request.
Second, we can access the nginx from
that VIP address.
○ iptables!!

COSCUP2018
x
Lte’swatchtheiptables
First, we can use the service name to
filter the iptables rules.
sudo iptables-save | grep ”k8s-
nginx-cluster”

COSCUP2018
x
Lte’swatchtheiptables
Remember? There’re three endpoints
for the service now.

COSCUP2018
x
workflowsPackets
Packets
Packets
Match Services’
ClusterIP
Find the
endpoints
DNAT
KUBE-SERVICES
KUBE-SVC-XXXX
KUBE-SEP-XXXX
Enter iptables PREROUTING
Jump to other chain Jump to other chain
Jump to other chain

COSCUP2018
x
Howdowechoosewhichonetouse?
When we match the clusterIP:Port,
goto another custom chain.
○ 10.105.100.214:80

COSCUP2018
x
Use the random to choose what
endpoint we use.

COSCUP2018
x
EP1 EP2 EP3
P < 0.33
P < 0.5
EP1
EP2 EP3
P= 1/3
P= 2/3 * 1/2
= 1/3
P= 2/3 * 1/2
= 1/3

COSCUP2018
x
EP1 EP2 EP3
P < 0.2
P < 0.25
EP1
EP2
EP3
EP4 EP5
P < 0.33
P < 0.5
EP4 EP5
P = 0.2
P = 4/5 * 1/4
= 1/5
P = 4/5 * 3/4 * 1/3
= 1/5
P = 4/5 * 3/4 * 2/3 * 1/2
= 1/5
P = 4/5 * 3/4 * 2/3 * 1/2
= 1/5

COSCUP2018
x
K8S create a custom chain for each
endpoints.
First rule is a SNAT
○ The Ngnix want to access outside.
Second is DNAT
○ Change the IP to one of the endpoints

COSCUP2018
x
Now, We Know The Basic Function Of
Kubernetes Network.

COSCUP2018
x
What is the next step of kubernetes
network

COSCUP2018
x
challenges
For different use cases
○ 5G/NFV/IoT
Network features
○ High performance
○ Low latency
Network infrastructure
○ Multiple network
■ Separate the data/control network.

COSCUP2018
x
Networkfeaturesc
We use the hardware/smart NIC for
those requirements before.
We also have some mechanism in the
software.
○ DPDK
○ SR-IOV
○ …
○ etc

COSCUP2018
x
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Kernel Space
User Space
DPDK

COSCUP2018
x
How integrate those with kubernetes?

COSCUP2018
x
CNI
We use some CNI for those functions.
Intel had developed the CNI for those
functions.
○ Call sriov-cni
○ https://github.com/intel/sriov-cni

COSCUP2018
x
Node1 Node2 Node3
PodA PodA PodA PodB PodA PodB
Flannel
(Control Network)
br0 br0 br0
Data Network
(192.168.0.0/16)

COSCUP2018
x
Problem
For those container using the
DPDK/SR-IOV, it can’t use any
kubernetes service now.
Since the network function is handled
by DPDK/SR-IOV now.
How to solve this?

COSCUP2018
x
Multus
There’s a discussion in the github
about that requirement.
Intel develop a CNI plugin to support
multiple network for a Pod.
○ It’s called Multus CNI
Multus call CNIs one by one.

COSCUP2018
x
Pod
Container
eth0
Pod
Container eth0
eth1
eth2
Previous What Multus Provide

COSCUP2018
x
,
,
You need to create first

COSCUP2018
x
Node1 Node2 Node3
PodA PodA PodA PodB PodA PodB
Flannel
(Control Network)
br0 br0 br0
Data Network
(192.168.0.0/16)
Data Network
(10.56.10/24)

COSCUP2018
x
Q&A

Overview of kubernetes network functions

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Overview of kubernetes network functions

Similar to Overview of kubernetes network functions (20)

More from HungWei Chiu

More from HungWei Chiu (20)

Recently uploaded

Recently uploaded (20)

Overview of kubernetes network functions