All Things Open, profile picture
Uploaded byAll Things Open
612 views

How to build a Kubernetes networking solution from scratch

The document details the process of building a Kubernetes networking solution from scratch, primarily using Open vSwitch (OVS) as a Container Network Interface (CNI) plugin. It provides insights into Kubernetes components, networking concepts, and the role of various networking policies and methods for pod communication within clusters. Additionally, it highlights how to configure pod networking and manage inter-node traffic with OVS, along with integration techniques and practical examples.

Technology◦
Related topics:
All Things Open•

Embed presentation

Downloaded 37 times
How to Build a Kubernetes Networking Solution from Scratch Antonin Bas, Jianjun Shen Project Antrea maintainers @VMware ATO, October 2020
Agenda 2 Container and K8s networking Building a K8s network plugin with Open vSwitch Introducing Project Antrea More visibility into K8s networks with Project Antrea Q&A
3 Basics of Container Networking Network Namespace • Isolated network environment provided by Linux kernel Interconnect • A simple way: veth devices & Linux bridge Communication across hosts • Network address translation and port mapping Docker bridge network on Linux docker0 (Linux bridge) 10.10.0.1/24 container1 – netns ns1 eth0 lo container2 – netns ns2 eth0 lo ens0 veth1 veth2 10.10.0.11/24 10.10.0.12/24 root netns Docker host SNAT 172.1.1.11/16
4 Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. What is Kubernetes?
5 Kubernetes Components K8s Cluster consists of Master(s) and Nodes K8s Master Components • API Server • Scheduler • Controller Manager • etcd K8s Node Components • kubelet • kube-proxy • Container Runtime K8s master K8s master K8s Master Controller Manager K8s API Server Key-Value Store dashboard Scheduler K8s node K8s node K8s node K8s node K8s Nodes kubelet c runtime kube-proxy > _ Kubectl CLI K8s Master(s)
6 Kubernetes Pod "Pods are the smallest deployable units of computing that you can create and manage in Kubernetes" A Pod comprises a group of one or more containers that shares an IP address and a network namespace. Pod pause container (‘owns’ the IP stack) 10.24.0.0/16 10.24.0.2 nginx tcp/80 mgmt tcp/22 logging udp/514 IPC External IP Traffic
7 Kubernetes Namespace “Namespaces are a way to divide cluster resources between multiple users” “Namespaces provide a scope for names” Namespace level access control is supported. Namespace: foo Base URI: /api/v1/namespaces/foo 'redis-master' Pod: /api/v1/namespaces/foo/pods/redis-master 'redis' Service: /api/v1/namespaces/foo/services/redis Namespace: bar Base URI: /api/v1/namespaces/bar 'redis-master' Pod: /api/v1/namespaces/bar/pods/redis-master 'redis' Service: /api/v1/namespaces/bar/services/redis
8 Kubernetes Service "An abstract way to expose an application running on a set of Pods as a network service" Serves multiple functions: • Service Discovery / DNS • East/West load balancing in the Cluster (Type: ClusterIP) • External load balancing for L4 TCP/UDP (Type: LoadBalancer) • External access to the Service through the Nodes IPs (Type: NodePort) Redis Pods Redis Service 10.24.0.5 ClusterIP 172.30.0.24 Web Front-End Pods 10.24.2.7 ▶ kubectl describe svc redis Name: redis Namespace: default Selector: app=redis Type: LoadBalancer IP: 172.30.0.24 LoadBalancer Ingress: 134.247.200.20 Port: <unnamed> 6379/TCP Endpoints: 10.24.0.5:6379, 10.24.2.7:6379 DNS: redis.<ns>.cluster.local è 172.30.0.24 ExternalIP 134.247.200.20 DNS: redis.external.com è 134.247.200.20
9 Kubernetes NetworkPolicy “A specification of how groups of Pods are allowed to communicate with each other and other network endpoints“ Selects Pods to apply the NetworkPolicy with matching labels Redis Pods Redis Service 10.24.0.5 ClusterIP 172.30.0.24 Web Front-End Pods 10.24.2.7 ▶ kubectl describe netpol web-front-redis Name: web-front-redis Namespace: default Spec: PodSelector: app=redis Allowing ingress traffic: To Port: 6379/TCP From: PodSelector: app=web-front-end Policy Types: Ingress
10 Kubernetes Cluster Networking Three communication patterns must be enabled Pod -to- Pod Pod -to- Service External -to- Service POD POD POD P P P P P P
12 What is a Kubernetes CNI Network Plugin responsible for? Pod Network Connectivity Plumbing eth0 (network interface) into Pod network IP Address Management (IPAM) E-W Service Load Balancing (optional) Make traffic available to upstream kube-proxy, or Implement native service load balancing – VIP DNAT NetworkPolicy Enforcement (optional) Enforcing Kubernetes Network Policy Traffic Shaping Support (experimental)
13 kubenet Relies on cloud network to route traffic between Nodes • Typically works with a Cloud Provider implementation that adds routes to the cloud router. • Supported on AWS, Azure, GCP. No NetworkPolicy support Out-of-box Kubernetes network plugin cbr0 (Linux bridge) 10.10.1.1/24 Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 cbr0 (Linux bridge) 10.10.2.1/24 Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud Network Fabric 172.1.1.11 172.1.2.22 Destination Target 10.10.1.0/24 172.1.1.11 10.10.2.0/24 172.1.2.22
14 kube-proxy Implements distributed load- balancing for Services of ClusterIP and NodePort types Supports: IPTables, IPVS, and user space proxy modes E-W Service Load-Balancing Picture from: https://kubernetes.io/docs/concepts/services-networking/service
15 Container Network Interface (CNI) Where does the CNI fit in the Pod’s lifecycle? K8s control plane kubelet Container Runtime (e.g. containerd) Network Plugin Pod K8s Node Pod Network 1. User creates Pod spec 2. Pod is scheduled on Node 3.CRI call 5.CNI call 4. Run Pod 6. Add to Pod network
18 And why use it for K8s networking? What is Open vSwitch (OVS)? A high-performance programmable virtual switch • Connects to VMs (tap) and containers (veth) Linux foundation project, very active Portable: Works out of the box on all Linux distributions and supports Windows Programmability: Supports many protocols, build your own forwarding pipeline High-performance • DPDK, AF_XDP • Hardware offload available across multiple vendors Rich feature set: • Multi-layers – L2 to L4 • Advanced CLI tools • Statistics, QoS • Packet tracing
19 Configuring Pod networking with OVS step-by-step CNI_COMMAND=ADD CNI_CONTAINERID=79ba130ac32e1c621e0e10ea10e3e8b7c0b101932f309ead54ee93fdf1795768 CNI_NETNS=/proc/1125/ns/net CNI_IFNAME=eth0 CNI_ARGS="K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx-66b6c48dd5- skx7z;K8S_POD_INFRA_CONTAINER_ID=79ba130ac32e1c621e0e10ea10e3e8b7c0b101932f309ead54ee93fdf1795768" CNI_PATH=/opt/cni/path # from stdin { "cniVersion": "0.3.0", "name": "antrea", "type": "antrea", “dns":{}, "ipam":{ "type": "host-local", "subnet": "10.10.1.0/24", "gateway": "10.10.1.1” } } From environment variables From stdin
20 Connecting the Pod to the OVS bridge OVS bridge (br-int) Container nginx lo ens0root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns ovs-vsctl add-br br-int
21 Connecting the Pod to the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1
22 Connecting the Pod to the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1
23 Connecting the Pod to the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1 Ø ip link set eth0 mtu <MTU> Ø ip addr add 10.10.1.2/24 dev eth0 Ø ip route add default via 10.10.1.1 dev eth0 Ø ip link set dev eth0 up Ø exit 10.10.1.2/24
24 Connecting the Pod to the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1 Ø ip link set eth0 mtu <MTU> Ø ip addr add 10.10.1.2/24 dev eth0 Ø ip route add default via 10.10.1.1 dev eth0 Ø ip link set dev eth0 up Ø exit ovs-vsctl add-port br-int veth1 ovs-vsctl show Bridge br-int … Port veth1 Interface veth1 … ovs_version: "2.14.0" 10.10.1.2/24
25 Intra-Node Pod-to-Pod traffic By default OVS behaves like a regular L2 Linux bridge A network plugin using OVS can provide additional security by preventing IP / ARP spoofing OVS bridge (br-int) PodA eth0 PodB eth0 ens0 veth1 veth2 10.10.1.2/24 10.10.1.3/24 root netns K8s Node ovs-ofctl add-flow br-int table=0,priority=200,arp,in_port=nginx,arp_spa=10.10.1.2,a rp_sha=<MAC>,actions=goto_table=10 ovs-ofctl add-flow br-int table=0,priority=200,ip,in_port=nginx,nw_src=10.10.1.2,dl_ src=<MAC>,actions=goto_table=10 ovs-ofctl add-flow br-int table=0,priority=0,actions=drop ovs-ofctl add-flow br-int table=10,priority=0,actions=NORMAL
26 Inter-Node Pod-to-Pod traffic The default gateway for Pod1A is 10.10.1.1, which is assigned to the OVS bridge (internal port) All traffic that’s not destined to a local Pod will be forwarded to gw0. Then what? è Build an overlay network OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 ? gw0 10.10.1.1/24 gw0 10.10.2.1/24 Destination Target 10.10.1.0/24 - * 10.10.1.1
27 Inter-Node Pod-to-Pod traffic Supported protocols: • Geneve / VXLAN / GRE / STT Building an overlay network with OVS OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 # on Node 1 ovs-vsctl add-port br-int tun0 -- set interface tun0 type=geneve options:remote_ip=flow options:key=flow ovs-vsctl show Bridge br-int … Port tun0 Interface tun0 type: geneve options: {key=flow, remote_ip=flow} Port gw0 Interface gw0 type: internal ovs_version: "2.14.0” gw0 10.10.1.1/24 gw0 10.10.2.1/24 tun0 tun0
28 Inter-Node Pod-to-Pod traffic Each Node has its own Pod subnet Broadcast domain is limited to a single Node New flows for inter-Node traffic Each Node’s Pod subnet is read from K8s API Building an overlay network with OVS OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 gw0 10.10.1.1/24 gw0 10.10.2.1/24 tun0 tun0 # on Node 1 ovs-ofctl add-flow br-int table=10,priority=200,ip, nw_dst=10.10.2.0/24,actions=dec_ttl,load:172.1.1.11- >NXM_NX_TUN_IPV4_DST[],output:tun0 ovs-ofctl add-flow br-int table=10,priority=200,ip, in_port=tun0,nw_dst=10.10.1.11,actions=mod_dl_dst:<MAC_PO D1A>,mod_dl_src:<MAC_GW0>,output:veth1 ovs-ofctl add-flow br-int table=10,priority=200,ip, in_port=tun0,nw_dst=10.10.1.12,actions=mod_dl_dst:<MAC_PO D1B>,mod_dl_src:<MAC_GW0>,output:veth1
30 K8s Networking with Open vSwitch L2 switching for local Pod-to- Pod traffic Overlay network for Inter- Node traffic SNAT for Pod-to-external traffic OVS programmability supports implementing the entire K8s network model Recap Node 1 (VM) Node 2 (VM) Pod A Pod B OvS bridge eth0 eth0 NIC Cloud Network Fabric vethA gw0 tun0 vethB Pod C Pod D OvS bridge eth0 eth0 NIC vethA gw0 tun0 vethB SNAT pod-to-external pod-to-pod (inter-node) pod-to-pod (intra-node)
31 Kubernetes CNI Plugins Dataplane technologies Open vSwitch BIRD (BGP), IPTables, eBPF (since v3.16.0) eBPF Linux bridge Network modes Overlay (Geneve, VXLAN, GRE, STT) or no-encapsulation Overlay (IPIP, VXLAN) or BGP routing Overlay (Geneve, VXLAN) or no-encapsulation Overlay (VXLAN) or no-encapsulation NetworkPolicy Open vSwitch Centralized policy computation IPTables or eBPF eBPF N/A Windows Support Open vSwitch Windows BGP, Virtual Filtering Platform N/A win-bridge or win-overlay 26 “third party” plugins listed at: https://github.com/containernetworking/cni, besides the “core plugins” maintained by the CNI project. CNI plugins for specific cloud / IaaS platform:
32 Project Antrea is an open source CNI network plugin for Kubernetes based on Open vSwitch, providing: • Pod network connectivity • NetworkPolicy enforcement • Service load balancing = ++ https://antrea.io @ProjectAntrea https://github.com/vmware-tanzu/antrea Kubernetes Slack – #antrea
33 Antrea is a community driven project focusing on • simplifying usability & diagnostics, • adapting any cloud and network topology, • providing comprehensive security policies, and • improving scaling & performance for container networking in Kubernetes. https://antrea.io @ProjectAntrea https://github.com/vmware-tanzu/antrea Kubernetes Slack – #antrea 782 GitHub Stars 136 GitHub Forks 42 ContributorsPrivate Cloud Public Cloud Edge Linux Windows runs on
34 Open vSwitch provides a flexible and performant data plane. Project Antrea Architecture Worker Node Worker Node Master Node kubelet antrea agent kube- proxy kubectlpod A pod B kube- api control-plane data-plane CRDsNetwork Policy Gateway Gateway Tunnel CNI CNI antrea agent IPtables kube- proxy IPtables veth pair veth pair Antrea Agent • Manages Pod network interfaces and OVS bridge. • Implements overlay network, NetworkPolicies, and Service load balancing with OVS. Antrea Controller • Computes NetworkPolicies and publishes the results to Antrea Agents. • High performance channel to Agents based on the K8s apiserver lib. Built with K8s technologies • Leverages K8s and K8s solutions for API, control plane, deployment, UI and CLI. • Antrea Controller and Agent are based on K8s controller and apiserver libs. kubectl apply -f https://github.com/vmware- tanzu/antrea/releases/download/v0.10.1/antrea.yml antrea controller
35 Demo Video 1 Antrea Setup and OVS Networking https://youtu.be/KGjGimuLXSI
36 NetworkPolicy Implementation Node 1 OVS bridge Antrea Agent Pod1A app=client 10.10.1.2 Openflow Node 2 OVS bridge Antrea Agent Pod2A app=server 10.10.2.2 Openflow Pod2A app=server 10.10.2.3 Node 3 OVS bridge Antrea Agent Pod3A app=other 10.10.3.2 Openflow Pod3B app=server 10.10.3.3 Antrea Controller K8s apiserver lib K8s API NetworkPolicy = “Pods with label ‘app=server’ can only receive traffic from Pods with label ‘app=client’ and only on port ‘TCP 80’. AppliedToGroup = Name: “foo” Pods: {Pod3B} AddressGroup = Name: “bar” Pods: {10.10.1.2} NetworkPolicy = Rule: Direction: Ingress From: {“bar”} Ports: {TCP/80} AppliedTo: {“foo”} Span = {Node2, Node3} … … NetworkPolicies Pods (Namespace, labels, IP addr, Node) Namespaces (labels) Table 90 (IngressRule table): priority=200,ip,nw_src=10.10.1.2 actions=conjunction(1,1/3) priority=200,ip,reg1=0x4 actions=conjunction(1,2/3) priority=200,tcp,tp_dst=80 actions=conjunction(1,3/3) priority=190,conj_id=1,ip actions=goto_table:105 Table 100 (IngressRuleDefault table): priority=200,ip,reg1=0x4 actions=drop # reg1 saves the input OVS port’s ofport number. # Pod3B’s ofport = 4. Centralized controller for NetworkPolicy computation • Each Node’s Agent receives only the relevant data • Compute once. Agent just performs simple conversion to OVS flows. • Controller = single source of truth. High performance communication channel built with K8s apiserver library. Use OVS flow conjunction • Reduce number of flows
37 Demo Video 2 Network Policies and Traceflow https://youtu.be/Sv_7qI7A2YY
38 The Open vSwitch pipeline
39 Antrea in the cloud-native ecosystem Providing visibility into the network Prometheus metrics exported from Agents & Controller Octant plugin to monitor components and trace packets ELK stack to visualize flow maps for the cluster network
40 Demo Video 3 K8s Network Visibility with Antrea https://youtu.be/qzTeUaePJRo
43 Network Plugins implement the CNI and provide L2/L3 connectivity in K8s clusters Open vSwitch can implement the full K8s network model with a unified data plane Project Antrea: a production-grade Network Plugin built in < 1 year OVS as the data plane K8s libraries for a highly-scalable control plane Integrations with cloud-native ecosystem tools to provide visibility into the network Suggest new integrations to us on Github! Conclusion
44 Come help us continually improve Kubernetes Networking! Kubernetes Slack #antrea Community Meeting, Mondays @ 9PM PT Zoom Link https://github.com/vmware-tanzu/antrea • Good first issues • Help us improve our documentation • Propose new features • File Bugs projectantrea-announce projectantrea projectantrea-dev (Google Groups) @ProjectAntrea @ https://antrea.io • Documentation • Blogs

More Related Content

PPTX
Kubernetes Networking 101
byWeaveworks
 
PDF
Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...
byEdureka!
 
PDF
NGINX Ingress Controller for Kubernetes
byNGINX, Inc.
 
PDF
Kubernetes extensibility: CRDs & Operators
bySIGHUP
 
PDF
Kubernetes a comprehensive overview
byGabriel Carro
 
PDF
Kubernetes Architecture - beyond a black box - Part 1
byHao H. Zhang
 
PDF
Introduction to kubernetes
byRaffaele Di Fazio
 
PPTX
Docker Networking Overview
bySreenivas Makam
 
Kubernetes Networking 101
byWeaveworks
 
Kubernetes Networking | Kubernetes Services, Pods & Ingress Networks | Kubern...
byEdureka!
 
NGINX Ingress Controller for Kubernetes
byNGINX, Inc.
 
Kubernetes extensibility: CRDs & Operators
bySIGHUP
 
Kubernetes a comprehensive overview
byGabriel Carro
 
Kubernetes Architecture - beyond a black box - Part 1
byHao H. Zhang
 
Introduction to kubernetes
byRaffaele Di Fazio
 
Docker Networking Overview
bySreenivas Makam
 

What's hot

PPTX
Ceph Introduction 2017
byKaran Singh
 
PDF
A crash course in CRUSH
bySage Weil
 
PDF
Introduction to Red Hat OpenShift 4
byHngNguyn748044
 
PDF
Openshift
byKnoldus Inc.
 
PPTX
CloudStack Networking
byCloudStack - Open Source Cloud Computing Project
 
PDF
Getting Started with Kubernetes
byVMware Tanzu
 
PPTX
Kubernetes Introduction
byMartin Danielsson
 
PDF
Kubernetes Basics
byEueung Mulyana
 
PDF
Hands-On Introduction to Kubernetes at LISA17
byRyan Jarvinen
 
PDF
Evolution of containers to kubernetes
byKrishna-Kumar
 
PDF
Kubernetes security
byThomas Fricke
 
PDF
OpenShift Overview
byroundman
 
PPTX
Introduction to the Container Network Interface (CNI)
byWeaveworks
 
PDF
Linking Metrics to Logs using Loki
byKnoldus Inc.
 
PDF
[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기
byIan Choi
 
PDF
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
byJi-Woong Choi
 
PDF
Deep dive into Kubernetes Networking
bySreenivas Makam
 
ODP
VPC Implementation In OpenStack Heat
bySaju Madhavan
 
PPTX
ceph optimization on ssd ilsoo byun-short
byNAVER D2
 
PDF
Introduction to Kubernetes Workshop
byBob Killen
 
Ceph Introduction 2017
byKaran Singh
 
A crash course in CRUSH
bySage Weil
 
Introduction to Red Hat OpenShift 4
byHngNguyn748044
 
Openshift
byKnoldus Inc.
 
CloudStack Networking
byCloudStack - Open Source Cloud Computing Project
 
Getting Started with Kubernetes
byVMware Tanzu
 
Kubernetes Introduction
byMartin Danielsson
 
Kubernetes Basics
byEueung Mulyana
 
Hands-On Introduction to Kubernetes at LISA17
byRyan Jarvinen
 
Evolution of containers to kubernetes
byKrishna-Kumar
 
Kubernetes security
byThomas Fricke
 
OpenShift Overview
byroundman
 
Introduction to the Container Network Interface (CNI)
byWeaveworks
 
Linking Metrics to Logs using Loki
byKnoldus Inc.
 
[OpenStack] 공개 소프트웨어 오픈스택 입문 & 파헤치기
byIan Choi
 
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
byJi-Woong Choi
 
Deep dive into Kubernetes Networking
bySreenivas Makam
 
VPC Implementation In OpenStack Heat
bySaju Madhavan
 
ceph optimization on ssd ilsoo byun-short
byNAVER D2
 
Introduction to Kubernetes Workshop
byBob Killen
 

Similar to How to build a Kubernetes networking solution from scratch

PDF
Kubernetes networking-made-easy-with-open-v switch
byInfraEngineer
 
PDF
Container network security
byDaisuke Nakajima
 
PDF
Kubernetes Networking 101 kubecon EU 2022
byssuser1490e8
 
PPTX
KuberNETes - meetup
byNathan Ness
 
PPTX
Open stackaustinmeetupsept21
byBrent Doncaster
 
PDF
Meetup 2023 - Gateway API.pdf
byRed Hat
 
PPTX
Networking in Docker Containers
byAttila Kanto
 
PDF
LF_OVS_17_OVN and Containers - An update.
byLF_OpenvSwitch
 
PDF
F5 OpenShift Workshop
byTyler Hatton
 
PPTX
Kubernetes on open stack
byNaveen Joy
 
PDF
Networking in Kubernetes
byMinhan Xia
 
PDF
Collabnix Online Webinar - Demystifying Docker & Kubernetes Networking by Bal...
byAjeet Singh Raina
 
PPTX
Container world hybridnetworking_rev2
byPrem Sankar Gopannan
 
PPTX
Nynog-K8s-networking-101.pptx
byDanielHertzberg4
 
PPTX
Docker networking Tutorial 101
byLorisPack Project
 
PDF
Packet walks in_kubernetes-v4
byInfraEngineer
 
PDF
Demystfying container-networking
byBalasundaram Natarajan
 
PDF
Kubernetes Networking - Giragadurai Vallirajan
byNeependra Khare
 
PDF
Kubernetes Networking
byCJ Cullen
 
PPTX
Introduction to CNI (Container Network Interface)
byHungWei Chiu
 
Kubernetes networking-made-easy-with-open-v switch
byInfraEngineer
 
Container network security
byDaisuke Nakajima
 
Kubernetes Networking 101 kubecon EU 2022
byssuser1490e8
 
KuberNETes - meetup
byNathan Ness
 
Open stackaustinmeetupsept21
byBrent Doncaster
 
Meetup 2023 - Gateway API.pdf
byRed Hat
 
Networking in Docker Containers
byAttila Kanto
 
LF_OVS_17_OVN and Containers - An update.
byLF_OpenvSwitch
 
F5 OpenShift Workshop
byTyler Hatton
 
Kubernetes on open stack
byNaveen Joy
 
Networking in Kubernetes
byMinhan Xia
 
Collabnix Online Webinar - Demystifying Docker & Kubernetes Networking by Bal...
byAjeet Singh Raina
 
Container world hybridnetworking_rev2
byPrem Sankar Gopannan
 
Nynog-K8s-networking-101.pptx
byDanielHertzberg4
 
Docker networking Tutorial 101
byLorisPack Project
 
Packet walks in_kubernetes-v4
byInfraEngineer
 
Demystfying container-networking
byBalasundaram Natarajan
 
Kubernetes Networking - Giragadurai Vallirajan
byNeependra Khare
 
Kubernetes Networking
byCJ Cullen
 
Introduction to CNI (Container Network Interface)
byHungWei Chiu
 

More from All Things Open

PDF
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PDF
Open Source Post-Quantum Cryptography - Matt Caswell
byAll Things Open
 
PDF
Rolling out Enterprise AI: Tools, Insights, and Team Empowerment
byAll Things Open
 
PPTX
AI and Linux in 2025 - Jay LaCroix, Learn Linux TV
byAll Things Open
 
PDF
Unconventional Backgrounds: The Secret Sauce of Diversity
byAll Things Open
 
PDF
With AIs Wide Open… - All Things Open 2025
byAll Things Open
 
PDF
Incorporating AI Into Your SDLC: Leveraging AI tooling across the phases of y...
byAll Things Open
 
PDF
Keynote: X-rAI Specs, Submarines and Juice
byAll Things Open
 
PDF
Authoring a GNOME Shell Extension: A Developer's Journey as an Open-Source Pr...
byAll Things Open
 
PDF
AI-Driven DevOps: How LLMs and Agents Are Changing Software Delivery
byAll Things Open
 
PDF
Developing Kubernetes Integrations for the On-Premises Cloud
byAll Things Open
 
PPTX
What If the Quietest Person is the Smartest in the Room?
byAll Things Open
 
PDF
ERG as a Bridge: Driving Inclusion in Open Source
byAll Things Open
 
PDF
The Missing Link to Inclusion and Performance
byAll Things Open
 
PDF
Linux Command-line Tips and Tricks - ATO 2025
byAll Things Open
 
PDF
Supply Chain Robots, Electric Sheep, and SLSA
byAll Things Open
 
PDF
Everything You Need to Know About Running LLMs Locally
byAll Things Open
 
PDF
Agentic AI, Interoperability, and Automation: Lessons from the Past, Visions ...
byAll Things Open
 
PDF
Tech Hiring Is Not Dead - You Just Actually Have To Try
byAll Things Open
 
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
Open Source Post-Quantum Cryptography - Matt Caswell
byAll Things Open
 
Rolling out Enterprise AI: Tools, Insights, and Team Empowerment
byAll Things Open
 
AI and Linux in 2025 - Jay LaCroix, Learn Linux TV
byAll Things Open
 
Unconventional Backgrounds: The Secret Sauce of Diversity
byAll Things Open
 
With AIs Wide Open… - All Things Open 2025
byAll Things Open
 
Incorporating AI Into Your SDLC: Leveraging AI tooling across the phases of y...
byAll Things Open
 
Keynote: X-rAI Specs, Submarines and Juice
byAll Things Open
 
Authoring a GNOME Shell Extension: A Developer's Journey as an Open-Source Pr...
byAll Things Open
 
AI-Driven DevOps: How LLMs and Agents Are Changing Software Delivery
byAll Things Open
 
Developing Kubernetes Integrations for the On-Premises Cloud
byAll Things Open
 
What If the Quietest Person is the Smartest in the Room?
byAll Things Open
 
ERG as a Bridge: Driving Inclusion in Open Source
byAll Things Open
 
The Missing Link to Inclusion and Performance
byAll Things Open
 
Linux Command-line Tips and Tricks - ATO 2025
byAll Things Open
 
Supply Chain Robots, Electric Sheep, and SLSA
byAll Things Open
 
Everything You Need to Know About Running LLMs Locally
byAll Things Open
 
Agentic AI, Interoperability, and Automation: Lessons from the Past, Visions ...
byAll Things Open
 
Tech Hiring Is Not Dead - You Just Actually Have To Try
byAll Things Open
 

Recently uploaded

PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
PDF
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
PDF
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PDF
Advanced SELinux Management - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PPTX
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
PPTX
TravelTech Paris 2025 | Beyond the pipes: Why Channel Management isn’t really...
byapidays
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PDF
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
GenerationAI Paris 2025 | City of Light (COL)
byapidays
 
PPTX
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
PDF
Python Automation in Action: Real-World Use Cases and Practical Implementation
byDenys Smirnov
 
PPTX
Celonis Optimizing your future with process
bydevjeremyappleyard
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
Post Quantum Cryptography for Dummies.pdf
byAde Ismail Isnan
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
Advanced SELinux Management - RHCSA (RH134).pdf
byLinuxCert Guru
 
GenerationAI Paris 2025 | How Agentic AI is Reinventing Organizations
byapidays
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
TravelTech Paris 2025 | Beyond the pipes: Why Channel Management isn’t really...
byapidays
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
BEP-20 Token on BNB Chain: From Concept to Deployment.pdf
byimoliviabennett
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
GenerationAI Paris 2025 | City of Light (COL)
byapidays
 
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
Python Automation in Action: Real-World Use Cases and Practical Implementation
byDenys Smirnov
 
Celonis Optimizing your future with process
bydevjeremyappleyard
 

How to build a Kubernetes networking solution from scratch

  • 1.
    How to Builda Kubernetes Networking Solution from Scratch Antonin Bas, Jianjun Shen Project Antrea maintainers @VMware ATO, October 2020
  • 2.
    Agenda 2 Container and K8snetworking Building a K8s network plugin with Open vSwitch Introducing Project Antrea More visibility into K8s networks with Project Antrea Q&A
  • 3.
    3 Basics of ContainerNetworking Network Namespace • Isolated network environment provided by Linux kernel Interconnect • A simple way: veth devices & Linux bridge Communication across hosts • Network address translation and port mapping Docker bridge network on Linux docker0 (Linux bridge) 10.10.0.1/24 container1 – netns ns1 eth0 lo container2 – netns ns2 eth0 lo ens0 veth1 veth2 10.10.0.11/24 10.10.0.12/24 root netns Docker host SNAT 172.1.1.11/16
  • 4.
    4 Kubernetes is anopen-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. What is Kubernetes?
  • 5.
    5 Kubernetes Components K8s Clusterconsists of Master(s) and Nodes K8s Master Components • API Server • Scheduler • Controller Manager • etcd K8s Node Components • kubelet • kube-proxy • Container Runtime K8s master K8s master K8s Master Controller Manager K8s API Server Key-Value Store dashboard Scheduler K8s node K8s node K8s node K8s node K8s Nodes kubelet c runtime kube-proxy > _ Kubectl CLI K8s Master(s)
  • 6.
    6 Kubernetes Pod "Pods arethe smallest deployable units of computing that you can create and manage in Kubernetes" A Pod comprises a group of one or more containers that shares an IP address and a network namespace. Pod pause container (‘owns’ the IP stack) 10.24.0.0/16 10.24.0.2 nginx tcp/80 mgmt tcp/22 logging udp/514 IPC External IP Traffic
  • 7.
    7 Kubernetes Namespace “Namespaces area way to divide cluster resources between multiple users” “Namespaces provide a scope for names” Namespace level access control is supported. Namespace: foo Base URI: /api/v1/namespaces/foo 'redis-master' Pod: /api/v1/namespaces/foo/pods/redis-master 'redis' Service: /api/v1/namespaces/foo/services/redis Namespace: bar Base URI: /api/v1/namespaces/bar 'redis-master' Pod: /api/v1/namespaces/bar/pods/redis-master 'redis' Service: /api/v1/namespaces/bar/services/redis
  • 8.
    8 Kubernetes Service "An abstractway to expose an application running on a set of Pods as a network service" Serves multiple functions: • Service Discovery / DNS • East/West load balancing in the Cluster (Type: ClusterIP) • External load balancing for L4 TCP/UDP (Type: LoadBalancer) • External access to the Service through the Nodes IPs (Type: NodePort) Redis Pods Redis Service 10.24.0.5 ClusterIP 172.30.0.24 Web Front-End Pods 10.24.2.7 ▶ kubectl describe svc redis Name: redis Namespace: default Selector: app=redis Type: LoadBalancer IP: 172.30.0.24 LoadBalancer Ingress: 134.247.200.20 Port: <unnamed> 6379/TCP Endpoints: 10.24.0.5:6379, 10.24.2.7:6379 DNS: redis.<ns>.cluster.local è 172.30.0.24 ExternalIP 134.247.200.20 DNS: redis.external.com è 134.247.200.20
  • 9.
    9 Kubernetes NetworkPolicy “A specificationof how groups of Pods are allowed to communicate with each other and other network endpoints“ Selects Pods to apply the NetworkPolicy with matching labels Redis Pods Redis Service 10.24.0.5 ClusterIP 172.30.0.24 Web Front-End Pods 10.24.2.7 ▶ kubectl describe netpol web-front-redis Name: web-front-redis Namespace: default Spec: PodSelector: app=redis Allowing ingress traffic: To Port: 6379/TCP From: PodSelector: app=web-front-end Policy Types: Ingress
  • 10.
    10 Kubernetes Cluster Networking Threecommunication patterns must be enabled Pod -to- Pod Pod -to- Service External -to- Service POD POD POD P P P P P P
  • 11.
    12 What is a Kubernetes CNINetwork Plugin responsible for? Pod Network Connectivity Plumbing eth0 (network interface) into Pod network IP Address Management (IPAM) E-W Service Load Balancing (optional) Make traffic available to upstream kube-proxy, or Implement native service load balancing – VIP DNAT NetworkPolicy Enforcement (optional) Enforcing Kubernetes Network Policy Traffic Shaping Support (experimental)
  • 12.
    13 kubenet Relies on cloudnetwork to route traffic between Nodes • Typically works with a Cloud Provider implementation that adds routes to the cloud router. • Supported on AWS, Azure, GCP. No NetworkPolicy support Out-of-box Kubernetes network plugin cbr0 (Linux bridge) 10.10.1.1/24 Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 cbr0 (Linux bridge) 10.10.2.1/24 Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud Network Fabric 172.1.1.11 172.1.2.22 Destination Target 10.10.1.0/24 172.1.1.11 10.10.2.0/24 172.1.2.22
  • 13.
    14 kube-proxy Implements distributed load- balancingfor Services of ClusterIP and NodePort types Supports: IPTables, IPVS, and user space proxy modes E-W Service Load-Balancing Picture from: https://kubernetes.io/docs/concepts/services-networking/service
  • 14.
    15 Container Network Interface(CNI) Where does the CNI fit in the Pod’s lifecycle? K8s control plane kubelet Container Runtime (e.g. containerd) Network Plugin Pod K8s Node Pod Network 1. User creates Pod spec 2. Pod is scheduled on Node 3.CRI call 5.CNI call 4. Run Pod 6. Add to Pod network
  • 15.
    18 And why useit for K8s networking? What is Open vSwitch (OVS)? A high-performance programmable virtual switch • Connects to VMs (tap) and containers (veth) Linux foundation project, very active Portable: Works out of the box on all Linux distributions and supports Windows Programmability: Supports many protocols, build your own forwarding pipeline High-performance • DPDK, AF_XDP • Hardware offload available across multiple vendors Rich feature set: • Multi-layers – L2 to L4 • Advanced CLI tools • Statistics, QoS • Packet tracing
  • 16.
    19 Configuring Pod networkingwith OVS step-by-step CNI_COMMAND=ADD CNI_CONTAINERID=79ba130ac32e1c621e0e10ea10e3e8b7c0b101932f309ead54ee93fdf1795768 CNI_NETNS=/proc/1125/ns/net CNI_IFNAME=eth0 CNI_ARGS="K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx-66b6c48dd5- skx7z;K8S_POD_INFRA_CONTAINER_ID=79ba130ac32e1c621e0e10ea10e3e8b7c0b101932f309ead54ee93fdf1795768" CNI_PATH=/opt/cni/path # from stdin { "cniVersion": "0.3.0", "name": "antrea", "type": "antrea", “dns":{}, "ipam":{ "type": "host-local", "subnet": "10.10.1.0/24", "gateway": "10.10.1.1” } } From environment variables From stdin
  • 17.
    20 Connecting the Podto the OVS bridge OVS bridge (br-int) Container nginx lo ens0root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns ovs-vsctl add-br br-int
  • 18.
    21 Connecting the Podto the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1
  • 19.
    22 Connecting the Podto the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1
  • 20.
    23 Connecting the Podto the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1 Ø ip link set eth0 mtu <MTU> Ø ip addr add 10.10.1.2/24 dev eth0 Ø ip route add default via 10.10.1.1 dev eth0 Ø ip link set dev eth0 up Ø exit 10.10.1.2/24
  • 21.
    24 Connecting the Podto the OVS bridge OVS bridge (br-int) Container nginx lo ens0 eth0 veth1 root netnsK8s Node K8s Pod nginx-66b6c48dd5-skx7z /proc/1125/ns/net netns nsenter -t 1125 -n bash Ø ip link add eth0 type veth peer name veth1 Ø ip link set veth1 netns 1 Ø ip link set eth0 mtu <MTU> Ø ip addr add 10.10.1.2/24 dev eth0 Ø ip route add default via 10.10.1.1 dev eth0 Ø ip link set dev eth0 up Ø exit ovs-vsctl add-port br-int veth1 ovs-vsctl show Bridge br-int … Port veth1 Interface veth1 … ovs_version: "2.14.0" 10.10.1.2/24
  • 22.
    25 Intra-Node Pod-to-Pod traffic Bydefault OVS behaves like a regular L2 Linux bridge A network plugin using OVS can provide additional security by preventing IP / ARP spoofing OVS bridge (br-int) PodA eth0 PodB eth0 ens0 veth1 veth2 10.10.1.2/24 10.10.1.3/24 root netns K8s Node ovs-ofctl add-flow br-int table=0,priority=200,arp,in_port=nginx,arp_spa=10.10.1.2,a rp_sha=<MAC>,actions=goto_table=10 ovs-ofctl add-flow br-int table=0,priority=200,ip,in_port=nginx,nw_src=10.10.1.2,dl_ src=<MAC>,actions=goto_table=10 ovs-ofctl add-flow br-int table=0,priority=0,actions=drop ovs-ofctl add-flow br-int table=10,priority=0,actions=NORMAL
  • 23.
    26 Inter-Node Pod-to-Pod traffic Thedefault gateway for Pod1A is 10.10.1.1, which is assigned to the OVS bridge (internal port) All traffic that’s not destined to a local Pod will be forwarded to gw0. Then what? è Build an overlay network OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 ? gw0 10.10.1.1/24 gw0 10.10.2.1/24 Destination Target 10.10.1.0/24 - * 10.10.1.1
  • 24.
    27 Inter-Node Pod-to-Pod traffic Supportedprotocols: • Geneve / VXLAN / GRE / STT Building an overlay network with OVS OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 # on Node 1 ovs-vsctl add-port br-int tun0 -- set interface tun0 type=geneve options:remote_ip=flow options:key=flow ovs-vsctl show Bridge br-int … Port tun0 Interface tun0 type: geneve options: {key=flow, remote_ip=flow} Port gw0 Interface gw0 type: internal ovs_version: "2.14.0” gw0 10.10.1.1/24 gw0 10.10.2.1/24 tun0 tun0
  • 25.
    28 Inter-Node Pod-to-Pod traffic EachNode has its own Pod subnet Broadcast domain is limited to a single Node New flows for inter-Node traffic Each Node’s Pod subnet is read from K8s API Building an overlay network with OVS OVS bridge (br-int) Pod1A eth0 Pod1B eth0 ens0 veth1 veth2 10.10.1.11/24 10.10.1.12/24 Node 1 OVS bridge (br-int) Pod2A eth0 Pod2B eth0 ens0 veth1 veth2 10.10.2.11/24 10.10.2.12/24 Node 2 Cloud / Physical Network Fabric 172.1.1.11 172.1.2.22 gw0 10.10.1.1/24 gw0 10.10.2.1/24 tun0 tun0 # on Node 1 ovs-ofctl add-flow br-int table=10,priority=200,ip, nw_dst=10.10.2.0/24,actions=dec_ttl,load:172.1.1.11- >NXM_NX_TUN_IPV4_DST[],output:tun0 ovs-ofctl add-flow br-int table=10,priority=200,ip, in_port=tun0,nw_dst=10.10.1.11,actions=mod_dl_dst:<MAC_PO D1A>,mod_dl_src:<MAC_GW0>,output:veth1 ovs-ofctl add-flow br-int table=10,priority=200,ip, in_port=tun0,nw_dst=10.10.1.12,actions=mod_dl_dst:<MAC_PO D1B>,mod_dl_src:<MAC_GW0>,output:veth1
  • 26.
    30 K8s Networking withOpen vSwitch L2 switching for local Pod-to- Pod traffic Overlay network for Inter- Node traffic SNAT for Pod-to-external traffic OVS programmability supports implementing the entire K8s network model Recap Node 1 (VM) Node 2 (VM) Pod A Pod B OvS bridge eth0 eth0 NIC Cloud Network Fabric vethA gw0 tun0 vethB Pod C Pod D OvS bridge eth0 eth0 NIC vethA gw0 tun0 vethB SNAT pod-to-external pod-to-pod (inter-node) pod-to-pod (intra-node)
  • 27.
    31 Kubernetes CNI Plugins Dataplane technologies OpenvSwitch BIRD (BGP), IPTables, eBPF (since v3.16.0) eBPF Linux bridge Network modes Overlay (Geneve, VXLAN, GRE, STT) or no-encapsulation Overlay (IPIP, VXLAN) or BGP routing Overlay (Geneve, VXLAN) or no-encapsulation Overlay (VXLAN) or no-encapsulation NetworkPolicy Open vSwitch Centralized policy computation IPTables or eBPF eBPF N/A Windows Support Open vSwitch Windows BGP, Virtual Filtering Platform N/A win-bridge or win-overlay 26 “third party” plugins listed at: https://github.com/containernetworking/cni, besides the “core plugins” maintained by the CNI project. CNI plugins for specific cloud / IaaS platform:
  • 28.
    32 Project Antrea isan open source CNI network plugin for Kubernetes based on Open vSwitch, providing: • Pod network connectivity • NetworkPolicy enforcement • Service load balancing = ++ https://antrea.io @ProjectAntrea https://github.com/vmware-tanzu/antrea Kubernetes Slack – #antrea
  • 29.
    33 Antrea is acommunity driven project focusing on • simplifying usability & diagnostics, • adapting any cloud and network topology, • providing comprehensive security policies, and • improving scaling & performance for container networking in Kubernetes. https://antrea.io @ProjectAntrea https://github.com/vmware-tanzu/antrea Kubernetes Slack – #antrea 782 GitHub Stars 136 GitHub Forks 42 ContributorsPrivate Cloud Public Cloud Edge Linux Windows runs on
  • 30.
    34 Open vSwitch providesa flexible and performant data plane. Project Antrea Architecture Worker Node Worker Node Master Node kubelet antrea agent kube- proxy kubectlpod A pod B kube- api control-plane data-plane CRDsNetwork Policy Gateway Gateway Tunnel CNI CNI antrea agent IPtables kube- proxy IPtables veth pair veth pair Antrea Agent • Manages Pod network interfaces and OVS bridge. • Implements overlay network, NetworkPolicies, and Service load balancing with OVS. Antrea Controller • Computes NetworkPolicies and publishes the results to Antrea Agents. • High performance channel to Agents based on the K8s apiserver lib. Built with K8s technologies • Leverages K8s and K8s solutions for API, control plane, deployment, UI and CLI. • Antrea Controller and Agent are based on K8s controller and apiserver libs. kubectl apply -f https://github.com/vmware- tanzu/antrea/releases/download/v0.10.1/antrea.yml antrea controller
  • 31.
    35 Demo Video 1 AntreaSetup and OVS Networking https://youtu.be/KGjGimuLXSI
  • 32.
    36 NetworkPolicy Implementation Node 1 OVSbridge Antrea Agent Pod1A app=client 10.10.1.2 Openflow Node 2 OVS bridge Antrea Agent Pod2A app=server 10.10.2.2 Openflow Pod2A app=server 10.10.2.3 Node 3 OVS bridge Antrea Agent Pod3A app=other 10.10.3.2 Openflow Pod3B app=server 10.10.3.3 Antrea Controller K8s apiserver lib K8s API NetworkPolicy = “Pods with label ‘app=server’ can only receive traffic from Pods with label ‘app=client’ and only on port ‘TCP 80’. AppliedToGroup = Name: “foo” Pods: {Pod3B} AddressGroup = Name: “bar” Pods: {10.10.1.2} NetworkPolicy = Rule: Direction: Ingress From: {“bar”} Ports: {TCP/80} AppliedTo: {“foo”} Span = {Node2, Node3} … … NetworkPolicies Pods (Namespace, labels, IP addr, Node) Namespaces (labels) Table 90 (IngressRule table): priority=200,ip,nw_src=10.10.1.2 actions=conjunction(1,1/3) priority=200,ip,reg1=0x4 actions=conjunction(1,2/3) priority=200,tcp,tp_dst=80 actions=conjunction(1,3/3) priority=190,conj_id=1,ip actions=goto_table:105 Table 100 (IngressRuleDefault table): priority=200,ip,reg1=0x4 actions=drop # reg1 saves the input OVS port’s ofport number. # Pod3B’s ofport = 4. Centralized controller for NetworkPolicy computation • Each Node’s Agent receives only the relevant data • Compute once. Agent just performs simple conversion to OVS flows. • Controller = single source of truth. High performance communication channel built with K8s apiserver library. Use OVS flow conjunction • Reduce number of flows
  • 33.
    37 Demo Video 2 NetworkPolicies and Traceflow https://youtu.be/Sv_7qI7A2YY
  • 34.
  • 35.
    39 Antrea in thecloud-native ecosystem Providing visibility into the network Prometheus metrics exported from Agents & Controller Octant plugin to monitor components and trace packets ELK stack to visualize flow maps for the cluster network
  • 36.
    40 Demo Video 3 K8sNetwork Visibility with Antrea https://youtu.be/qzTeUaePJRo
  • 37.
    43 Network Plugins implementthe CNI and provide L2/L3 connectivity in K8s clusters Open vSwitch can implement the full K8s network model with a unified data plane Project Antrea: a production-grade Network Plugin built in < 1 year OVS as the data plane K8s libraries for a highly-scalable control plane Integrations with cloud-native ecosystem tools to provide visibility into the network Suggest new integrations to us on Github! Conclusion
  • 38.
    44 Come help uscontinually improve Kubernetes Networking! Kubernetes Slack #antrea Community Meeting, Mondays @ 9PM PT Zoom Link https://github.com/vmware-tanzu/antrea • Good first issues • Help us improve our documentation • Propose new features • File Bugs projectantrea-announce projectantrea projectantrea-dev (Google Groups) @ProjectAntrea @ https://antrea.io • Documentation • Blogs