This document discusses troubleshooting Network Functions Virtualization (NFV) architecture and OpenStack with DPDK. It describes potential failures in configuring OVS-DPDK, including instances becoming unreachable over the DPDK network. Methodologies are provided for investigating and solving VNF issues, OVS-DPDK failures and performance problems. Solutions involve adjusting MTUs, enabling port security, and changing virtual network device drivers.

2. AGENDA
●
●
●

4. NFV ARCHITECTURE
●
●
●
●
●
●

5. OSS/B
SS
NFVI
VNF
MANO
network
network compute
Other
NFVI

6. OSS/B
SS
NFVI
VNF
MANO

8. VNF PANICS
VNF CRASHES WHEN IT STARTS SENDING NETWORK PACKETS
●
●
●
COMPUTE-1 COMPUTE-2
VNF-1 VNF-1VNF-1

9. VNF PANICS: WHO SHOULD INVESTIGATE
INVESTIGATION THROUGH COLLABORATION
●
○
●
○
●
COMPUTE-1 COMPUTE-2
VNF-1 VNF-1
●
○

10. INVESTIGATION: HINT
INVESTIGATION THROUGH COLLABORATION
●
●
●
●
●
●
COMPUTE-1 COMPUTE-2
VNF-1 VNF-1
Packets
defragmented
here
1500 1500eth0 eth0

11. PACKET FLOW AND MTU
tapx
brx
qvb
qvo br-int
int-br
ph-br
enox enox
ph-br
int-br
br-int qvo
qvb
brx
tapx
VNF
eth0
VNF
eth0
9000 1500
●
●
●
○
enox
ph-br
int-br
br-int qvo
qvb
brx
tapx tapx

12. POSSIBLE SOLUTIONS
tapx
brx
qvb
qvo br-int
int-br
ph-br
enox enox
ph-br
int-br
br-int qvo
qvb
brx
tapx
VNF
eth0
VNF
eth0
9000 1500
enox
ph-br
int-br
br-int qvo
qvb
brx
eth0
tapx tapx
eth0eth0 eth0eth0 eth0

13. SOLUTION
Change the mtu of vnf to 9000
Change the mtu of tapx to 1500
Enable port security
Set bridge-nf-call-iptables to 0
Inspect what can be changed on vnf. Change e1000 to
virtio
Add support for per port mtu in Neutron

15. OVS-DPDK TROUBLESHOOTING METHODOLOGY
●
○
○
○
●
○
○
○
○

16. OVS-DPDK FAILURE TROUBLESHOOTING METHODOLOGY
●
○
●
○
○
○
●
●
○ driverctl -v list-devices | grep vfio
○ for i in `ps -eLo tid,comm | grep pmd | cut -f2 -d ' '` ; do grep huge
/proc/$i/numa_maps ;done
○
●

17. OVS-DPDK PERFORMANCE TROUBLESHOOTING
METHODOLOGY
●
●
●
○
○
○
○
●
# tuned-adm profile cpu-partitioning
# grep tuned_params /boot/grub2/grub.cfg
set tuned_params="nohz=on nohz_full=2,4,8-11 rcu_nocbs=2,4,8-11
intel_pstate=disable nosoftlockup isolcpus=2,4,8-11"
linux16 /boot/vmlinuz-3.10.0-514.2.2.el7.x86_64
root=UUID=a69bf0c7-8d41-42c5-b1f0-e64719aa7ffb ro console=tty0
console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB
hugepagesz=1G hugepages=12 iommu=pt intel_iommu=on selinux=0 $tuned_params

18. OVS-DPDK PERFORMANCE TROUBLESHOOTING
METHODOLOGY
●
●
●
●
●

20. OVS-DPDK FAILURES
● After configuring ovs-dpdk , the
instances cannot be reached via
dpdk based network.
DPDK NIC
core core
core core
core core
core core
core core
core core
Numa Node 0 Numa Node 1
core core core core

22. OVS-DPDK FAILURES
●
●
After configuring ovs-dpdk , the instances cannot be reached via dpdk based network.
Symptom - I

23. ●
○
●
●
●
OVS-DPDK Failures

24. ●
●
OVS-DPDK Failures
DPDK NIC
core core
core core
core core
core core
core core
core core
Numa Node 0 Numa Node 1
PMD THREAD PMD THREAD
core core core core

26. OVS-DPDK Failures
●
After configuring ovs-dpdk , the instances cannot be reached via dpdk based network.
Symptom - II

27. OVS-DPDK Failures
●
●
●

28. OVS-DPDK Failures
●
●
●

29. OVS-DPDK Failures
●
●
DPDK NIC
core core
core core
core core
core core
core core
core core
Numa Node 0 Numa Node 1
core core core core
echo 4 >
/sys/devices/system/node/node1/hugepage
s/hugepages-1048576kB/nr_hugepages

30. References
●
●
●
●

31. OSS/B
SS
NFVI
VNF
MANO