Tim Rozet, Red Hat, Feng Pan, Red Hat This session will explore the challenges and lessons learned with integrating accelerated dataplanes into OPNFV deployments. More specifically the talk will focus on FD.IO (VPP) and OVS DPDK integration into Apex, including different types of configuration options, platform requirements, performance tuning, and deployment challenges. This talk will also provide context to how OpenStack functions differently with these types of dataplanes, and how integration with the OpenDaylight controller works.

2. Accelerated Dataplanes Integration
Feng Pan (Red Hat)
Tim Rozet (Red Hat)

3. To be covered
•OVS DPDK and FD.IO (VPP)
•Common Dependencies of OVS DPDK and FD.IO (VPP)
•Configuration/Integration Differences between OVS DPDK and FD.IO
•Integration Challenges
•Performance Tuning
•Automated Configuration options
•OpenDaylight Integration - design and operation

4. High Level OVS DPDK
•Runs as an application or system service in Linux userspace
•Packets incoming to the host are not processed by the OVS kernel
driver which handles packets in the non-DPDK scenario
•Ports on the host are bound to DPDK, then assigned to bridges in OVS
•VM ports on the host are created as vhostuser
•Poll Mode Driver (PMD) threads continuously poll for new packets
incoming to the DPDK dataplane

5. High Level FD.IO (VPP)
•Runs as an application or system service in Linux userspace
•Ports on the host are bound to DPDK, then assigned to interfaces in
VPP
•VM ports on the host are created as vhostuser
•Support L2 and L3 features (NAT/ACL/NSH/IPSEC)

6. Common Dependencies
•Both rely on PMD drivers, of which there are a few (uio_pci_generic,
vfio-pci)
•Both require hugepage configuration (required for DPDK)
•IOMMU is required for vfio-pci
•nova scheduler filters must be set to include NUMATopologyFilter (for
KVM)
•Host NIC must support DPDK
•libvirt must be able to read vhostuser sockets
•Though config is different, both require similar performance tuning

7. Integration Differences
•OVS requires being put into DPDK mode, and then restarted:
−ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-init=true
•DPDK ports are added live to a “netdev” type bridge in OVS DPDK
•In FD.IO, DPDK ports are specified in startup config before FD.IO is run
•With OVS DPDK, Host NIC is bound manually, while it is automatic in
FD.IO

8. Integration Challenges
•After binding to DPDK, NIC essentially does not exist in Linux kernel
anymore
•Packets flowing in userspace makes debugging more difficult
•NUMA Topology and hardware properties about each Host should be
considered before deploying accelerated dataplanes
•Configuration changes may require host reboot and additional steps
•Using existing Neutron agents (DHCP/L3) requires patches, uses kernel
data path (allows VPP veth pair to Agent namespace)

9. Common Performance Tuning
•PMD threads should be isolated and pinned to their own cores using
isolcpus kernel argument
•How many hugepages/which socket to assign to the dataplane?
•Which NUMA node does the PCIe bus with DPDK NIC belong to?
•Nova instances should be pinned to cores on the same socket as PMD
•Use 1GB huge page if possible
•BIOS settings for maximum performance:
- Disable C state
- Enable Turboboost/Speedstep

10. OVS DPDK Performance Tuning
•PMD threads are pinned to cores via cpu mask:
−ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6
−If hyper-threading is enabled, care should be given to core sibling
relationships
−Cores should be allocated to the NUMA node that PCIe bus of the
DPDK NIC belongs to
−Allocating more than 1 core will result in multiple PMD threads
spawned

11. OVS DPDK Performance Tuning (cont.)
•Dataplane hugepages should be configured to same socket:
−ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-
mem="1024,0"
−format is “<MB of socket 0>, <MB of socket 1>, ...”
•Pin IRQs away from isolated cores using tuna or tuned

12. VPP Performance Tuning
• Run in multithreading mode
cpu/workers <n>: Create n worker threads
cpu/coremask-workers <mask> and cpu/corelist-workers <list>: Place worker threads
according to mask or list
cpu/main-core <n>: Assign main thread to specific core. Defaults to first available
core.
cpu/skip-cores <n>: Leave low n bits of process affinity mask clear
dpdk/coremask <mask>: process level core mask
dpdk/num-mbufs <n>: Number of I/O buffers. Defaults to 16384
dpdk/socket-mem <list>: Buffer memory allocation. Defaults to 256MB on each
NUMA node
• Tickless Kernel: nohz_full and rcu_nocbs kernel parameters, use with isolcpus

13. Automated Configuration Options
•puppet-vswitch
−manifests/dpdk.pp
−Capable of configuring OVS with DPDK performance options
−Part of OpenStack Puppet Modules
•puppet-fdio
−Capable of configuring fd.io with performance options
−Can also configure honeycomb agent for OpenDaylight
−Own project in FD.IO repository

14. OpenDaylight + OVS DPDK
Computenode-0 Computenode-1
Tenantnetworki/f
Tenantnetworki/f
Tenant network i/f
Bridge
br-phy
VM 2
vhost-
user
Controlnode-0
VXLAN
OVS DPDK
External network i/f
Internet
DHCP
tap
VXLAN
VXLAN
OVS
OpenStack Services Network Control
VM 1
Bridge
br-int
Bridge
br-int
Bridge
br-phy
Bridge
br-int
vhost-
user
OVS DPDK
Bridge
br-ex

15. OpenDaylight + FD.IO
Computenode-0 Computenode-1
Tenantnetworki/f
Tenantnetworki/f
Tenant network i/f
Bridge
Domain
VM 2
vhost-
user
Controlnode-0
VXLAN
VPP
External network i/f
Internet
DHCP
tap
VXLAN
VXLAN
VPP
VPP
OpenStack Services Network Control
VM 1
Bridge
Domain
Bridge
Domain
vhost-
user
HoneyComb HoneyComb
HoneyComb

16. OpenDaylight + FD.IO current work
● Add support for Nirvana stack (converged GBP+Netvirt)
● Add more complete performance tuning options to deployment
● Add interface bonding support
● Add DVR support
● Distributed DHCP to compute nodes (no VPP needed on control
node anymore)

17. OPNFV Apex support
•Apex is the only installer project that supports both ovs-dpdk and VPP
data path
•All deployment and performance tuning steps can be specified in
config files and automated
•Apex supports OpenDaylight with both dataplanes over multiple
scenarios

18. Questions?