Simon Horman, Netronome, "Layer 3 Tunnel Support for Open vSwitch" from July 14, 2016 at LinuxCon in Tokyo, Japan.

1. © 2016 NETRONOME SYSTEMS, INC. 1© 2016 NETRONOME SYSTEMS, INC.
Simon Horman
Layer 3 Tunnel Support
for Open vSwitch (OVS)

2. © 2016 NETRONOME SYSTEMS, INC. 2
Motivation
Would like to:
▶ Allow rx and tx of packets over tunnels whose payload packet does not have an
Ethernet header
▶ Add these features to upstream OVS then offload them

3. © 2016 NETRONOME SYSTEMS, INC. 3
Importance of Offloading
5
10
15
20
25
30
OVS Kernel
Datapath with
Netdev to VMs
OVS User-Space
Datapath with
Netdev to VMs
100 Wildcard Rules
1000 Wildcard Rules
10000 Wildcard Rules
64000 Wildcard Rules
Packetspersecond(millions)
12 CPU Cores
8 CPU Cores
OVS Offload to
iNIC with
PMD to VMs
1 CPU CoreOVS L2/L3 Forwarding to 8 VMs with 64K Flows
OVS Offload to
iNIC with
Netdev to VMs
1 CPU Core

4. © 2016 NETRONOME SYSTEMS, INC. 4
Scope
Datapaths:
▶ Linux Kernel
▶ User-Space with and without DPDK
Encapsulation Protocol:
▶ GRE (non-TEB) (rfc2794):
• IP protocols over GRE
• MPLS in GRE (rfc4023)

5. © 2016 NETRONOME SYSTEMS, INC. 5
Background: Tunnel vPorts
Encapsulation and decapsulation is handled by output to/input from
tunnel vports
Not currently exposed in Open-Flow

6. © 2016 NETRONOME SYSTEMS, INC. 6
Kernel Datapath Tunnel vPorts
Kernel Datapath:
▶ On rx tunnel vport decapsulates packet passing the result and metadata to the
datapath
▶ On tx tunnel vport encapsulates packet based on metadata

7. © 2016 NETRONOME SYSTEMS, INC. 7
User-Space Tunnel vPorts
Native Tunneling:
▶ Tunnel ingress and egress on separate OvS bridge
▶ Internal rules match ingress and egress packets for tunnel vPorts and apply push
and pop tunnel actions accordingly
▶ Like the Kernel Datapath tunnel metadata is:
• Available in flow key after decapsulation
• Used as parameters for encapsulation

8. © 2016 NETRONOME SYSTEMS, INC. 8
Layer 3 Tunneling: Basic Concepts
Layer 2 and 3 vPorts
push_eth and pop_eth datapath actions
Datapath Attributes and packet type

9. © 2016 NETRONOME SYSTEMS, INC. 9
Layer 2 and 3 vPorts
Layer 2 or 3 is a mode of vports
Default is layer 2: behavior of all vports until now

10. © 2016 NETRONOME SYSTEMS, INC. 10
pop_eth and push_eth Actions
Add or remove an Ethernet header to/from start of packet
Packets with a VLAN not currently permitted
MPLS is treated as L2.5 and left alone
Not currently exposed to OpenFlow:
▶ Automatically included in actions of datapath flow

11. © 2016 NETRONOME SYSTEMS, INC. 11
Datapath Attributes and Packet Type
Presence of ETHERTYPE and ETHERNET attributes indicates L2 packet
Presence of ETHERTYPE but not ETHERNET attribute indicates L3 packet
ETHERTYPE corresponds to Protocol Type in GRE header

12. © 2016 NETRONOME SYSTEMS, INC. 12
GRE Header
Checksum (optional) Reserved (optional)
Protocol TypeVerReserved0C
C: Checksum Present
Delivery Header
Payload Packet

13. © 2016 NETRONOME SYSTEMS, INC. 13
Operation
OvS User-Space (ovs-vswitchd) is aware of which vports are Layer 2 and which
are Layer 3
It is aware of the input port for each flow
And thus when translating from OpenFlow to datapath flows it can add push_eth
and pop_eth actions before output actions as necessary

14. © 2016 NETRONOME SYSTEMS, INC. 14
Packet Flow
Key: eth_type, ...
Actions: push_eth, output
Layer 3
vPort
Layer 2
vPort
Key: eth, eth_type, ...
Actions: pop_eth, output
Layer 2
vPort
Layer 3
vPort

15. © 2016 NETRONOME SYSTEMS, INC. 15
vPort Implementations
User-Space (non-Datapath)
Kernel Datapath
User-Space Datapath

16. © 2016 NETRONOME SYSTEMS, INC. 16
User-Space (non-Datapath) vPorts
vPorts have new layer3 flag to distinguish layer mode
vPorts of the same type (e.g. GRE) but different layer mode share the same
datapath vport

17. © 2016 NETRONOME SYSTEMS, INC. 17
Kernel Datapath vPorts
Switch to using ipgre rather than gretap netdev in kernel
ipgre (and ipvxlan) vports have recently been enhanced to allow rx/tx of TEB as
well as non-TEB packets
Thus facilitating a single datapath vport for use with both layer 2 and 3 user-
space vports
This design was motivated by a desire to avoid vport type explosion

18. © 2016 NETRONOME SYSTEMS, INC. 18
User-Space Datapath vPorts
New user-space datapath only NEXT_BASE_LAYER flow key attribute
Used to distinguish flows with layer 2 and 3 payload packets

19. © 2016 NETRONOME SYSTEMS, INC. 19
GRE Header
Checksum (optional) Reserved (optional)
Protocol TypeVerReserved0C
C: Checksum Present
Delivery Header
Payload Packet

20. © 2016 NETRONOME SYSTEMS, INC. 20
Configuration Example
ovs-vsctl add-port br0 tun1 --
set Interface tun1 type=gre
options:remote_ip=10.0.0.2
options:key=flow
options:layer3=true

21. © 2016 NETRONOME SYSTEMS, INC. 21
Future Work
Encapsulation Protocols:
▶ MPLS in IP (rfc4023)
▶ MPLS in UDP (rfc7510)
▶ NSH (draft-ietf-sfc-nsh-05)
▶ VXLAN-GPE (draft-ietf-nvo3-vxlan-gpe-02)
▶ LISP (rfc6830)

22. © 2016 NETRONOME SYSTEMS, INC. 22
Credits
Many, including:
▶ Lorand Jakub, Thomas Morin: Original implementation
▶ Jiri Benc: Kernel Tunnel Enhancements

23. © 2016 NETRONOME SYSTEMS, INC. 23
Availability
Open vSwitch (User-Space):
▶ https://github.com/horms/openvswitch l3-vpn
Kernel (Datapath):
▶ https://github.com/horms/linux l3-vpn
Working towards upstream merge!

24. © 2016 NETRONOME SYSTEMS, INC. 24© 2016 NETRONOME SYSTEMS, INC.
Simon Horman
Thank You