It's challenging to have low-latency VNFs in virtualization and cloud environment. OPNFV KVM4NFV project, together with other OPNFV projects like OVSNFV, helps to achieve low latency network functionality. This session will firstly introduce KVM4NFV project. Then some DPDK workload will be used to show how the KVM4NFV project helps reducing the packet latency and compare the result with and without OPNFV environment. In the end, experience will be shared on how to setup the OPNFV environment correctly, and how to tune the OPNFV environment to meet the latency and performance requirement.

1. June 20–23, 2016 | Berlin, Germany

2. Achieving Low Latency Network Function
with OPNFV
Jiming Sun (Intel)
Tapio Tallgren (Nokia)

3. Agenda
• Network Latency on NFV
• KVM4NFV and Network Latency
• Low Latency Testing Enhancement on OPNFV

4. Low network latency is important
• Voice quality for VOIP service - 150ms (tolerable by ears)
• LTE Cloud RAN (Radio Access Network): 1 ms
• - With 5G, the latencies need to be 1/10th of that
• A VNF that receives 10 Gbit/s network traffic of 256 byte packets
will receive 3900 packets in 1 ms
• Profit or loss in algorithmic trading requires microsecond latencies

5. HW
Network Latency and NFV
Open vSwitch VNF1
DPDK Apps
virtio NIC
Linux
HW
NIC NIC
vHost
PMD
QEMU
KVMNIC
DPDK Apps

6. Hypervisor and Network Latency
• Virtualization overhead
• vCPU pre-emption/scheduling
• Virtual switch scheduling/service
• Resource allocation

7. KVM4NFV Project Introduction
• Join OPNFV on B release
• 3 focus area
• Low Latency
• Fast live migration
• Inter-VM communication
• Upstream based effort
Today’s focus

8. Network Latency with KVM4NFV – B release
• Timer latency (Average: 5 us, Maximum: 15 us)
• Interrupt latency (Maximum: 20 us)
• Maximum guest vCPU pre-emption period
(Maximum: 10 us)
• Pre-emption periods in total within 1 ms window
(Maximum: 20 us)

9. Network Latency with KVM4NFV – C release
• Real network application
• SR-IOV Packet forwarding
• DPDK Packet forwarding
• Add complexity to the test environment
• Stress workload/Noicy neighbor
• Kernel update
• From 4.1 LTS to 4.4 LTS

10. VNF1
DPDK L2 FWD
Linux
HW
NIC1 NIC2
Traffic
Gen
NIC1 NIC2
Packet Forwarding Latency Enhancement
Kernel Average
Latency
Maximum
Latency
General
Kernel
289.6 us 373.9 us
KVM4NFV 19.6 us 101.8 us
• Guest
• NIC devices assigned to guest directly
• DPDK BasicFwd workload
• Host: KVM4NFV
• TrafficGen: 64 byte packet with 10G line rates throughput for 24
hours

11. VNF1
DPDK Apps
virtio NIC
Hypervisor
Open vSwitch
User Level User Level
virtio NIC
HW
NIC NIC
vHost
PMD PMD
vHost
Packet Latency Enhancement (Phy-VM-Phy)
Test Environment:
• VM with 4 vCPU
• 2 vCPU for data path
• PMD and vCPU threads are pinned to isolated CPUs
• 2 rx/tx queues per NIC port
• 2 rx/tx queues per vHost user port
Testpmd on VNF
• 10 minutes testing
• 30% line rates and 100% line rates

12. Packet Latency Enhancement (Phy-VM-Phy)
64 128 256 512 1024 1280 1518
Average (30% ) 14.2 11.2 10.8 10.6 11.4 10 10
Maxium (30%) 31.3 29.9 33.7 29.2 36.1 33 35.8
Average(100%) 25.9 28.8 39.5 46.1 37.7
Maxium(100%) 38.5 103.4 109.5 115 115.2
0
20
40
60
80
100
120
140
Average (30% ) Maxium (30%) Average(100%) Maxium(100%)
Based on 2 hours of tests

13. HW
CAT and Isolation
Linux KVM
core core
LLC
CPU
Memory
VNFStress
• LLC (Last Level Cache) is shared
by all Cores on the CPU
• LLC noise impact VNF
• Local LLC hit: 40~75 cycles *
• Local DRAM access: 60ns
*http://software.intel.com/sites/products/collate
ral/hpc/vtune/performance_analysis_guide.pdf
LLC LLC

14. CAT and Isolation
0
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
0 3 6 9 1215182124273033363942454851545760636669727578818487
Occurances
Latency
Cyclictest on Idle System
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
0 3 6 9 1215182124273033363942454851545760636669727578818487
Occurances
Latency
Cyclictest + stress load with CAT disabled
0
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
0 3 6 9 1215182124273033363942454851545760636669727578818487
Occurances
Latency
Cyclictest + stress load with CAT

15. Low Latency Testing on OPNFV
• A automatic latency test system is important
• Cover different scenarios
• Catch regression
• But challenges
• Complex environment setup
• Openstack or no openstack
• Not ready for environment setup
• Analysis not easy with openstack
• Resource requirement
• 24 hours testing for each scenario

16. Target of latency test
• Easy to support different scenarios
• Cyclictest->packet forwarding->…..
• Flexible testing scenarios
• 24 hours release testing
• Short-term patch testing
• Automatic data publish

17. Low Latency Test Status
Patch
Submission
Artifact Build
Gerrit/Jekins Octopus
Verify or
Testing
Job
Dispatch
Octopus
Pharos
Yardstick
vsperf
See discussion on https://etherpad.opnfv.org/p/nfv-kvm-test
Data
publish
?

18. Call to Actions
• Participate in KVM4NFV project
• Requirements
• Try and provide feedback
• Development
• Collaboration on test workloads
• vsperf
• Yardstick