SlideShare a Scribd company logo
1 of 23
Download to read offline
Cilium:
Fast IPv6 Container Networking with
BPF and XDP
LinuxCon 2016, Toronto
Thomas Graf (@tgraf__)
Kernel, Cilium & Open vSwitch Team
Noiro Networks (Cisco)
The Cilium Experiment
Scale
– Addressing: IPv6?
– Policy: Linear lists don’t scale. Alternative?
Extensibility
– Can we be as extensible as userspace networking
in the kernel?
Simplicity
– What is an appropriate abstraction away from
traditional networking?
Performance
– Do we sacrifice performance in the process?
Scaling Addressing
Solution:
– IPv6 addresses with host scope allocator
Pros:
– Everything is globally addressable
– No NAT
– Path to ILA for mobility of tasks
Cons:
– Legacy IPv4 only endpoints/applications
→ Optional IPv4 addressing (+ NAT)
→ NAT46: Provide IPv6 only applications to IPv4
only clients
IPv6 Status in Kubernetes/Docker
● Kubernetes (CNI): Almost there
– Pods are IPv6-only capable as of k8s 1.3.6
(PR23317, PR26438, PR26439, PR26441)
– Kubeproxy (services) not done yet
● Docker (libnetwork): Working on it
– PR826 - “Make IPv6 Great Again”
Not merged yet
Scaling Policy
LB Frontend Backend
Scaling Policy
LB BEFE
LB FE
FE BE
LB
LB Frontend Backend
Policy:
NetworkPolicy Kubernetes policy spec
as discussed and standardized in the
Networking SIG
https://github.com/kubernetes/kubernetes/blo
b/master/docs/proposals/network-policy.md
Scaling Policy
LB QA BE QAFE QA
LB Prod BE ProdFE Prod
LB FE
FE BE
LB
LB Frontend Backend
QA
Prod
Policy:
Scaling Policy
LB QA BE QAFE QA
LB Prod BE ProdFE Prod
LB FE
FE
QA
Prod
BE
LB QA
Prod
requires
requires
LB Frontend Backend
QA
Prod
Policy:
Cilium extension
Not yet part of
Kubernetes spec
QA
Scaling Policy Enforcement
LB FE
FE
QA
Prod
BE
LB QA
Prod
requires
requires
LB QA
FE QA
LB Prod10
11
12
13
Policy enforcement cost becomes a single hashtable
lookup regardless of number of containers or policy
complexity.
BE QA
FE Prod 14
BE Prod 15
Distributed Label ID Table:Policy:
QA
This ID is carried in packet as
metadata to provide security
context at destination host
Extensibility
Kernel
Userspace
Source
Code
Byte
Code
LLVM/clang
Sockets
netdevice
Network
StackTC
Ingress
TC
Egress
netdevice
Verifier
+ JIT
add eax,edx
shl eax,2
add eax,edx
shl eax,2
BPF – Berkley Packet Filter
Kernel
Userspace
BPF
Program
Userspace
Process
BPF Maps & Perf Ring Buffer
BPF Map
Hashtable
BPF Map
Array
Userspace
Process
BPF
Program
Per Ring
Buffer
Data DataTail Call
BPF Features
(As of Aug 2016)
● Efficient data sharing via maps
– Per-CPU/global arrays & hashtables
● Rewrite packet content
● Extend/trim packet size
● Redirect to other net_device
● Attachment of tunnel metadata
● Cgroups integration
● Access to high performance perf ring buffer
● …
Kernel
Userspace
XDP – Express Data Path
Source
Code
Byte
Code
LLVM/clang
Sockets
Netdevice
Network
Stack
Verifier
+ JIT
add eax,edx
shl eax,2
Driver
Access to
DMA buffer
Kernel
Cilium Layer
Orchestration
systems
eth0
BPF
Program
Cilium
Daemon
Cilium
Monitor
Cilium
CLI
BPF Program
Conntrack Policy
Bytecode injection
Events
BPF Program
Conntrack Policy
Code
Generation
Plugins
Policy
Repository
Cilium Architecture
Why is this awesome?
On the fly BPF program generation means:
● Extensibility of userspace networking in the kernel
● MAC, IP, port number, … all become constants
→ compiler can optimize heavily!
● BPF programs can be recompiled and replaced without
interrupting the container and its connections
– Features can be compiled in/out at runtime with
container granularity
● Access to fast BPF maps and perf ring buffer to interact
with userspace.
– Drop monitor in n*Mpps context
– Use notifications for policy learning, IDS, logging, ...
Available Building Blocks
● L3 forwarding (IPv6 & IPV4)
● Host connectivity
● Encapsulation
(VXLAN/Geneve/GRE)
● ICMPv6 generation
● NDisc & ARP responder
● Access Control
Currently working on:
● Fragmentation handling
● Mobility
● Port Mapping (TCP/UDP)
● Connection tracking
● L3/L4 Load Balancer
● Statistics
● Events (perf ring buffer)
● Debugging framework
● NAT46
● End to end encryption
Networking should be invisible,
it is not.
Simplicity
Simplicity
● L3 only (Calico gets this right)
– No L2 scaling issues, no broadcast domains, no L2
vulnerabilities
● No “Networks”
– No need for containers to join multiple networks
to access multiple isolation domains. No need for
multiple addresses.
● Policy definition independent of addressing
– As specified in Kubernetes Networking SIG
– All policies based on container labels
Performance
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
0
100
200
300
400
500
600
Container to container on local node
# Cores
Gbit
netperf -t TCP_SENDFILE -H beef::aa0:18:ee5e
1 TCP flow per core, 10’000 policies
Intel Xeon 3.5Ghz Sandy Bridge, 24 cores
Performance
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Container to container over 10GiB NICs
64
128
256
512
1024
64000
# Cores
MBit
netperf -t TCP_SENDFILE -H beef::aa0:18:ee5e
1 TCP flow per core, 10’000 policies
Intel Xeon 3.5Ghz Sandy Bridge, 24 cores
<Insert Cool Demo Here>
Q&A
Image Sources:
● Cover (Toronto)
Rick Harris (https://www.flickr.com/photos/rickharris/)
● The Invisible Man
Dr. Azzacov (https://www.flickr.com/photos/drazzacov/)
Start hacking with BPF for containers:
http://github.com/cilium/cilium
Contact:
Slack: cilium.slack.com
Twitter: @tgraf__ Mail: tgraf@tgraf.ch
Team:
● André Martins
● Daniel Borkmann
● Madhu Challa
● Thomas Graf

More Related Content

What's hot

Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelThomas Graf
 
netfilter and iptables
netfilter and iptablesnetfilter and iptables
netfilter and iptablesKernel TLV
 
Cilium - Bringing the BPF Revolution to Kubernetes Networking and Security
Cilium - Bringing the BPF Revolution to Kubernetes Networking and SecurityCilium - Bringing the BPF Revolution to Kubernetes Networking and Security
Cilium - Bringing the BPF Revolution to Kubernetes Networking and SecurityThomas Graf
 
Cilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPCilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPThomas Graf
 
Introduction to eBPF and XDP
Introduction to eBPF and XDPIntroduction to eBPF and XDP
Introduction to eBPF and XDPlcplcp1
 
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...Maximilan Wilhelm
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking WalkthroughThomas Graf
 
Tutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerTutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
 
Using GTP on Linux with libgtpnl
Using GTP on Linux with libgtpnlUsing GTP on Linux with libgtpnl
Using GTP on Linux with libgtpnlKentaro Ebisawa
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveMichal Rostecki
 
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U Translation
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U TranslationIETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U Translation
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U TranslationKentaro Ebisawa
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecturehugo lu
 
Replacing iptables with eBPF in Kubernetes with Cilium
Replacing iptables with eBPF in Kubernetes with CiliumReplacing iptables with eBPF in Kubernetes with Cilium
Replacing iptables with eBPF in Kubernetes with CiliumMichal Rostecki
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KernelThomas Graf
 
Cilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshCilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshChristian Posta
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)Brendan Gregg
 

What's hot (20)

Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux Kernel
 
netfilter and iptables
netfilter and iptablesnetfilter and iptables
netfilter and iptables
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
Cilium - Bringing the BPF Revolution to Kubernetes Networking and Security
Cilium - Bringing the BPF Revolution to Kubernetes Networking and SecurityCilium - Bringing the BPF Revolution to Kubernetes Networking and Security
Cilium - Bringing the BPF Revolution to Kubernetes Networking and Security
 
Cilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPCilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDP
 
GTPing, How To
GTPing, How ToGTPing, How To
GTPing, How To
 
Introduction to eBPF and XDP
Introduction to eBPF and XDPIntroduction to eBPF and XDP
Introduction to eBPF and XDP
 
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...
Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what ca...
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking Walkthrough
 
Tutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerTutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting router
 
Using GTP on Linux with libgtpnl
Using GTP on Linux with libgtpnlUsing GTP on Linux with libgtpnl
Using GTP on Linux with libgtpnl
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep Dive
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
 
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U Translation
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U TranslationIETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U Translation
IETF 104 Hackathon VPP Prototyping Stateless SRv6/GTP-U Translation
 
eBPF Workshop
eBPF WorkshopeBPF Workshop
eBPF Workshop
 
The linux networking architecture
The linux networking architectureThe linux networking architecture
The linux networking architecture
 
Replacing iptables with eBPF in Kubernetes with Cilium
Replacing iptables with eBPF in Kubernetes with CiliumReplacing iptables with eBPF in Kubernetes with Cilium
Replacing iptables with eBPF in Kubernetes with Cilium
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux Kernel
 
Cilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshCilium + Istio with Gloo Mesh
Cilium + Istio with Gloo Mesh
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 

Similar to Cilium - Fast IPv6 Container Networking with BPF and XDP

FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)Kirill Tsym
 
FD.IO Vector Packet Processing
FD.IO Vector Packet ProcessingFD.IO Vector Packet Processing
FD.IO Vector Packet ProcessingKernel TLV
 
VNIX-NOG 2021: IPv6 Deployment Update
VNIX-NOG 2021: IPv6 Deployment UpdateVNIX-NOG 2021: IPv6 Deployment Update
VNIX-NOG 2021: IPv6 Deployment UpdateAPNIC
 
Ocpeu14
Ocpeu14Ocpeu14
Ocpeu14KALRAY
 
Introduction to DPDK
Introduction to DPDKIntroduction to DPDK
Introduction to DPDKKernel TLV
 
Making our networking stack truly extensible
Making our networking stack truly extensible Making our networking stack truly extensible
Making our networking stack truly extensible Olivier Bonaventure
 
Osnug meetup-tungsten fabric - overview.pptx
Osnug meetup-tungsten fabric - overview.pptxOsnug meetup-tungsten fabric - overview.pptx
Osnug meetup-tungsten fabric - overview.pptxM.Qasim Arham
 
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...PROIDEA
 
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun  to do the impossible with DPDKDPDK summit 2015: It's kind of fun  to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDKLagopus SDN/OpenFlow switch
 
DPDK Summit 2015 - NTT - Yoshihiro Nakajima
DPDK Summit 2015 - NTT - Yoshihiro NakajimaDPDK Summit 2015 - NTT - Yoshihiro Nakajima
DPDK Summit 2015 - NTT - Yoshihiro NakajimaJim St. Leger
 
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
 
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...Jim St. Leger
 
Data Plane Evolution: Towards Openness and Flexibility
Data Plane Evolution: Towards Openness and FlexibilityData Plane Evolution: Towards Openness and Flexibility
Data Plane Evolution: Towards Openness and FlexibilityAPNIC
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane Michelle Holley
 
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceAdding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceSamsung Open Source Group
 
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...Haidee McMahon
 
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia
 

Similar to Cilium - Fast IPv6 Container Networking with BPF and XDP (20)

FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)
 
FD.IO Vector Packet Processing
FD.IO Vector Packet ProcessingFD.IO Vector Packet Processing
FD.IO Vector Packet Processing
 
FD.io - The Universal Dataplane
FD.io - The Universal DataplaneFD.io - The Universal Dataplane
FD.io - The Universal Dataplane
 
VNIX-NOG 2021: IPv6 Deployment Update
VNIX-NOG 2021: IPv6 Deployment UpdateVNIX-NOG 2021: IPv6 Deployment Update
VNIX-NOG 2021: IPv6 Deployment Update
 
Ocpeu14
Ocpeu14Ocpeu14
Ocpeu14
 
Introduction to DPDK
Introduction to DPDKIntroduction to DPDK
Introduction to DPDK
 
Making our networking stack truly extensible
Making our networking stack truly extensible Making our networking stack truly extensible
Making our networking stack truly extensible
 
Osnug meetup-tungsten fabric - overview.pptx
Osnug meetup-tungsten fabric - overview.pptxOsnug meetup-tungsten fabric - overview.pptx
Osnug meetup-tungsten fabric - overview.pptx
 
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...
PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifie...
 
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun  to do the impossible with DPDKDPDK summit 2015: It's kind of fun  to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
 
DPDK Summit 2015 - NTT - Yoshihiro Nakajima
DPDK Summit 2015 - NTT - Yoshihiro NakajimaDPDK Summit 2015 - NTT - Yoshihiro Nakajima
DPDK Summit 2015 - NTT - Yoshihiro Nakajima
 
100 M pps on PC.
100 M pps on PC.100 M pps on PC.
100 M pps on PC.
 
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
 
Lustre, RoCE, and MAN
Lustre, RoCE, and MANLustre, RoCE, and MAN
Lustre, RoCE, and MAN
 
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
 
Data Plane Evolution: Towards Openness and Flexibility
Data Plane Evolution: Towards Openness and FlexibilityData Plane Evolution: Towards Openness and Flexibility
Data Plane Evolution: Towards Openness and Flexibility
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane
 
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceAdding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
 
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...
Software Network Data Plane - Satisfying the need for speed - FD.io - VPP and...
 
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...
 

More from Thomas Graf

BPF & Cilium - Turning Linux into a Microservices-aware Operating System
BPF  & Cilium - Turning Linux into a Microservices-aware Operating SystemBPF  & Cilium - Turning Linux into a Microservices-aware Operating System
BPF & Cilium - Turning Linux into a Microservices-aware Operating SystemThomas Graf
 
Cilium - API-aware Networking and Security for Containers based on BPF
Cilium - API-aware Networking and Security for Containers based on BPFCilium - API-aware Networking and Security for Containers based on BPF
Cilium - API-aware Networking and Security for Containers based on BPFThomas Graf
 
Linux Native, HTTP Aware Network Security
Linux Native, HTTP Aware Network SecurityLinux Native, HTTP Aware Network Security
Linux Native, HTTP Aware Network SecurityThomas Graf
 
BPF: Next Generation of Programmable Datapath
BPF: Next Generation of Programmable DatapathBPF: Next Generation of Programmable Datapath
BPF: Next Generation of Programmable DatapathThomas Graf
 
Cilium - BPF & XDP for containers
Cilium - BPF & XDP for containersCilium - BPF & XDP for containers
Cilium - BPF & XDP for containersThomas Graf
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking ExplainedThomas Graf
 
LinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSLinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSThomas Graf
 
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)Thomas Graf
 
2015 FOSDEM - OVS Stateful Services
2015 FOSDEM - OVS Stateful Services2015 FOSDEM - OVS Stateful Services
2015 FOSDEM - OVS Stateful ServicesThomas Graf
 
Open vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATOpen vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATThomas Graf
 
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThe Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThomas Graf
 
SDN & NFV Introduction - Open Source Data Center Networking
SDN & NFV Introduction - Open Source Data Center NetworkingSDN & NFV Introduction - Open Source Data Center Networking
SDN & NFV Introduction - Open Source Data Center NetworkingThomas Graf
 

More from Thomas Graf (12)

BPF & Cilium - Turning Linux into a Microservices-aware Operating System
BPF  & Cilium - Turning Linux into a Microservices-aware Operating SystemBPF  & Cilium - Turning Linux into a Microservices-aware Operating System
BPF & Cilium - Turning Linux into a Microservices-aware Operating System
 
Cilium - API-aware Networking and Security for Containers based on BPF
Cilium - API-aware Networking and Security for Containers based on BPFCilium - API-aware Networking and Security for Containers based on BPF
Cilium - API-aware Networking and Security for Containers based on BPF
 
Linux Native, HTTP Aware Network Security
Linux Native, HTTP Aware Network SecurityLinux Native, HTTP Aware Network Security
Linux Native, HTTP Aware Network Security
 
BPF: Next Generation of Programmable Datapath
BPF: Next Generation of Programmable DatapathBPF: Next Generation of Programmable Datapath
BPF: Next Generation of Programmable Datapath
 
Cilium - BPF & XDP for containers
Cilium - BPF & XDP for containersCilium - BPF & XDP for containers
Cilium - BPF & XDP for containers
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
 
LinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSLinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVS
 
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
 
2015 FOSDEM - OVS Stateful Services
2015 FOSDEM - OVS Stateful Services2015 FOSDEM - OVS Stateful Services
2015 FOSDEM - OVS Stateful Services
 
Open vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATOpen vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NAT
 
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThe Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
 
SDN & NFV Introduction - Open Source Data Center Networking
SDN & NFV Introduction - Open Source Data Center NetworkingSDN & NFV Introduction - Open Source Data Center Networking
SDN & NFV Introduction - Open Source Data Center Networking
 

Recently uploaded

英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdf
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdfInnovate and Collaborate- Harnessing the Power of Open Source Software.pdf
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdfYashikaSharma391629
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsAhmed Mohamed
 
Lecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptLecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptesrabilgic2
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Matt Ray
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfAlina Yurenko
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfkalichargn70th171
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf31events.com
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identityteam-WIBU
 

Recently uploaded (20)

英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdf
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdfInnovate and Collaborate- Harnessing the Power of Open Source Software.pdf
Innovate and Collaborate- Harnessing the Power of Open Source Software.pdf
 
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML DiagramsUnveiling Design Patterns: A Visual Guide with UML Diagrams
Unveiling Design Patterns: A Visual Guide with UML Diagrams
 
Lecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptLecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).ppt
 
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Advantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your BusinessAdvantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your Business
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
 

Cilium - Fast IPv6 Container Networking with BPF and XDP

  • 1. Cilium: Fast IPv6 Container Networking with BPF and XDP LinuxCon 2016, Toronto Thomas Graf (@tgraf__) Kernel, Cilium & Open vSwitch Team Noiro Networks (Cisco)
  • 2. The Cilium Experiment Scale – Addressing: IPv6? – Policy: Linear lists don’t scale. Alternative? Extensibility – Can we be as extensible as userspace networking in the kernel? Simplicity – What is an appropriate abstraction away from traditional networking? Performance – Do we sacrifice performance in the process?
  • 3. Scaling Addressing Solution: – IPv6 addresses with host scope allocator Pros: – Everything is globally addressable – No NAT – Path to ILA for mobility of tasks Cons: – Legacy IPv4 only endpoints/applications → Optional IPv4 addressing (+ NAT) → NAT46: Provide IPv6 only applications to IPv4 only clients
  • 4. IPv6 Status in Kubernetes/Docker ● Kubernetes (CNI): Almost there – Pods are IPv6-only capable as of k8s 1.3.6 (PR23317, PR26438, PR26439, PR26441) – Kubeproxy (services) not done yet ● Docker (libnetwork): Working on it – PR826 - “Make IPv6 Great Again” Not merged yet
  • 6. Scaling Policy LB BEFE LB FE FE BE LB LB Frontend Backend Policy: NetworkPolicy Kubernetes policy spec as discussed and standardized in the Networking SIG https://github.com/kubernetes/kubernetes/blo b/master/docs/proposals/network-policy.md
  • 7. Scaling Policy LB QA BE QAFE QA LB Prod BE ProdFE Prod LB FE FE BE LB LB Frontend Backend QA Prod Policy:
  • 8. Scaling Policy LB QA BE QAFE QA LB Prod BE ProdFE Prod LB FE FE QA Prod BE LB QA Prod requires requires LB Frontend Backend QA Prod Policy: Cilium extension Not yet part of Kubernetes spec QA
  • 9. Scaling Policy Enforcement LB FE FE QA Prod BE LB QA Prod requires requires LB QA FE QA LB Prod10 11 12 13 Policy enforcement cost becomes a single hashtable lookup regardless of number of containers or policy complexity. BE QA FE Prod 14 BE Prod 15 Distributed Label ID Table:Policy: QA This ID is carried in packet as metadata to provide security context at destination host
  • 12. Kernel Userspace BPF Program Userspace Process BPF Maps & Perf Ring Buffer BPF Map Hashtable BPF Map Array Userspace Process BPF Program Per Ring Buffer Data DataTail Call
  • 13. BPF Features (As of Aug 2016) ● Efficient data sharing via maps – Per-CPU/global arrays & hashtables ● Rewrite packet content ● Extend/trim packet size ● Redirect to other net_device ● Attachment of tunnel metadata ● Cgroups integration ● Access to high performance perf ring buffer ● …
  • 14. Kernel Userspace XDP – Express Data Path Source Code Byte Code LLVM/clang Sockets Netdevice Network Stack Verifier + JIT add eax,edx shl eax,2 Driver Access to DMA buffer
  • 15. Kernel Cilium Layer Orchestration systems eth0 BPF Program Cilium Daemon Cilium Monitor Cilium CLI BPF Program Conntrack Policy Bytecode injection Events BPF Program Conntrack Policy Code Generation Plugins Policy Repository Cilium Architecture
  • 16. Why is this awesome? On the fly BPF program generation means: ● Extensibility of userspace networking in the kernel ● MAC, IP, port number, … all become constants → compiler can optimize heavily! ● BPF programs can be recompiled and replaced without interrupting the container and its connections – Features can be compiled in/out at runtime with container granularity ● Access to fast BPF maps and perf ring buffer to interact with userspace. – Drop monitor in n*Mpps context – Use notifications for policy learning, IDS, logging, ...
  • 17. Available Building Blocks ● L3 forwarding (IPv6 & IPV4) ● Host connectivity ● Encapsulation (VXLAN/Geneve/GRE) ● ICMPv6 generation ● NDisc & ARP responder ● Access Control Currently working on: ● Fragmentation handling ● Mobility ● Port Mapping (TCP/UDP) ● Connection tracking ● L3/L4 Load Balancer ● Statistics ● Events (perf ring buffer) ● Debugging framework ● NAT46 ● End to end encryption
  • 18. Networking should be invisible, it is not. Simplicity
  • 19. Simplicity ● L3 only (Calico gets this right) – No L2 scaling issues, no broadcast domains, no L2 vulnerabilities ● No “Networks” – No need for containers to join multiple networks to access multiple isolation domains. No need for multiple addresses. ● Policy definition independent of addressing – As specified in Kubernetes Networking SIG – All policies based on container labels
  • 20. Performance 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 0 100 200 300 400 500 600 Container to container on local node # Cores Gbit netperf -t TCP_SENDFILE -H beef::aa0:18:ee5e 1 TCP flow per core, 10’000 policies Intel Xeon 3.5Ghz Sandy Bridge, 24 cores
  • 21. Performance 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Container to container over 10GiB NICs 64 128 256 512 1024 64000 # Cores MBit netperf -t TCP_SENDFILE -H beef::aa0:18:ee5e 1 TCP flow per core, 10’000 policies Intel Xeon 3.5Ghz Sandy Bridge, 24 cores
  • 23. Q&A Image Sources: ● Cover (Toronto) Rick Harris (https://www.flickr.com/photos/rickharris/) ● The Invisible Man Dr. Azzacov (https://www.flickr.com/photos/drazzacov/) Start hacking with BPF for containers: http://github.com/cilium/cilium Contact: Slack: cilium.slack.com Twitter: @tgraf__ Mail: tgraf@tgraf.ch Team: ● André Martins ● Daniel Borkmann ● Madhu Challa ● Thomas Graf