Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The analysis of Microburs (Burstiness) on Virtual Switch

173 views

Published on

OPNFV OVSNFV meeting (2016.09.19)
URL:
https://wiki.opnfv.org/display/ovsnfv/Open+vSwitch+For+NFV+Home

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

The analysis of Microburs (Burstiness) on Virtual Switch

  1. 1. Copyright 2016 FUJITSU LABORATORIES LIMITED The Analysis of Microburst (Burstiness) on Virtual Switch Chunghan Lee Fujitsu Laboratories 09.19.2016
  2. 2. Background  What is Network Function Virtualization (NFV) ?  NFV virtualizes network functions (e.g. Firewall, Load balancer) on IA servers •The function is called as virtualized network function (VNF) •Virtual switch is one of major components for virtual network on NFV Infrastructure (NFVI) •QoS property is also important for NFV 1 Copyright 2014 FUJITSU LABORATORIES LIMITED An image is from http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/01.01.01_60/gs_NFV002v010101p.pdf Virtual network* *Virtual switch is included in virtual network
  3. 3. Microburst – (1)  What is microburst ?  Spikes in shot time period, but causing decreased performance  Impact of microburst on network Although traffic with QoS property is generated, the spikes can be occurred by massive traffic in very short time period •This phenomenon cannot catch the ordinary monitoring (SNMP) •When packet drop is occurred by the spike, it is changed as the microburst 2 Copyright 2016 FUJITSU LABORATORIES LIMITED Sudden spikes in throughput Packet drop !! SNMP (Coarse-grained) Packet (Find-grained) Spike
  4. 4. Microburst – (2)  Impact of microburst on NFVI  Although traffic with QoS property is generated on VNF, the sudden spikes would be occurred due to resource state  Packet loss can be occurred by the spikes (microbursts) 3 Copyright 2016 FUJITSU LABORATORIES LIMITED Traffic with QoS Sudden spikes !! NFVI (tx) NFVI (rx)
  5. 5. Problem  Microburst on NFVI  Ordinary application measurement cannot catch the microbursts on NFVI  There are queues and buffers between NFVIs  A deep understanding of packet processing issued by virtual switch and Linux kernel is required to clarify the cause of microburst 4 Copyright 2016 FUJITSU LABORATORIES LIMITED Traffic with QoS Sudden spikes !! NFVI (tx) NFVI (rx)
  6. 6. GOAL Investigate the occurrence of microburst on NFVI Find the cause of microbursts on NFVI 5 Copyright 2014 FUJITSU LABORATORIES LIMITED
  7. 7. Approach  Prepare two types of UDP traffic to observe the microbursts on an OvS bridge  Foreground : Target with QoS property  Background : Occur a lack of CPU resource at kernel  Generate fore/background traffic and capture their packets with tcpdump  Analyze them in packet-level to observe the microburst  Profile kernel functions to clarify the cause of microbursts 6 Copyright 2016 FUJITSU LABORATORIES LIMITED
  8. 8. Overview of testbed  Foreground traffic (UDP)  Sending rate : 2Gbps [Datagram size : 1400 bytes (No fragmentation)]  vport tx queue : default value (0), UDP buffer : default value (200 KBytes)  Background traffic (UDP)  8 vports are used on the same OVS bridge  iperf using UDP mode with 10 parallel flows per vport (iperf option –p)  Sending rate of UDP flow : 1Gbps (configured bandwidth by iperf) 7 Copyright 2016 FUJITSU LABORATORIES LIMITED
  9. 9. Server and switch spec.  Overview of spec.  All servers have the same spec.  Switch spec. •Fujitsu SR-X 526 (10G switch) 8 Server type : Fujitsu RX100S7 CPU (4 cores) Intel(R) Xeon(R) CPU E31220 @ 3.10GHz Memory 16GB (Speed: 1333 MHz) OS CentOS 7.2 Kernel version 3.18.25 NIC 10G:Intel X710 iperf 2.0.8 Open vSwitch 2.4.0 (release version) vport queue Default value (txqueuelen : 0) UDP buffer Default value (200 Kbytes) Copyright 2016 FUJITSU LABORATORIES LIMITED
  10. 10. Investigation of microburst 9 Copyright 2016 FUJITSU LABORATORIES LIMITED
  11. 11. Measurement points  6 throughput measurement points  Sender : iperf, Entering OVS, Leaving OVS  Receiver : iperf, Entering OVS, Leaving OVS 10 Copyright 2016 FUJITSU LABORATORIES LIMITED iperf User Kernel NIC driver < Sender > < Receiver > iperf OVS (bridge) NIC driver OVS (bridge) Network iperf (RX)iperf (TX) Entering OVS (tcpdump) Leaving OVS (tcpdump) Throughput measurement points Entering OVS (tcpdump) Leaving OVS (tcpdump)
  12. 12. Throughput at 6 measurement points  Throughput of foreground per second  Leaving OVS (sender), the throughput is decreased  At receiver, the throughput is also decreased at iperf (RX) 11 Copyright 2016 FUJITSU LABORATORIES LIMITED < Sender > < Receiver > Decreased throughput Decreased throughput
  13. 13. Throughput of foreground at sender  Throughput of foreground per 1 millisecond  Entering OVS, the sending rate is fluctuated due to a lack of CPU  Leaving OVS, sudden spikes in throughput are found •Throughput with QoS property is changed 12 Copyright 2016 FUJITSU LABORATORIES LIMITED < Entering OVS > < Leaving OVS > Unit time : 1 millisecond NIC driver iperf OVS (bridge) < Sender > Entering OVS Leaving OVS Network
  14. 14. Packet drop and throughput at receiver  A relation between throughput and packet drop  The timing of packet drop is similar to the timing of sudden spikes in throughput at receiver  The overflow of socket buffer is frequently occurred by the sudden spikes (spikes → microbursts) 13 Copyright 2016 FUJITSU LABORATORIES LIMITED < Packet drop at socket > < Leaving OVS > NIC driver iperf OVS (bridge) < Receiver > Leaving OVS Network Socket
  15. 15. Packet drop and spacing at receiver  A relation between packet drop and packet spacing  The packet spacing with moving average (MA)* is decreased while the number of drop packets on socket buffer is increased 14 Copyright 2016 FUJITSU LABORATORIES LIMITED *MA leg : 1000 Packet spacing Burstiness Packet loss ↓ ↑ ↑ ↑ ↓ ↓ Microburst NIC driver iperf OVS (bridge) < Receiver > Leaving OVS Network Socket
  16. 16. Cause of microburst 15 Copyright 2016 FUJITSU LABORATORIES LIMITED
  17. 17. Find the cause of microburst  Profiling kernel functions using perf  Profiling rate (sampling rate) is 1 millisecond  Focus on process (iperf with 2Gbps) only for the profiling 16 Copyright 2016 FUJITSU LABORATORIES LIMITED Profiling Kernel by perf
  18. 18. Common function call graph  Packet processing at Linux kernel 17 Copyright 2016 FUJITSU LABORATORIES LIMITED Net I/F Enqueue (qdisc, tx-ring) TCP/UDP layer (TX) iperf OVS (bridge) IP layer (TX) Kernel User From vport to OVS Copy data From OVS to qdisc
  19. 19. Measurement points with TCP/IP stack  Leaving OVS, the packet capture is occurred after qdisc 18 Copyright 2016 FUJITSU LABORATORIES LIMITED Entering OVS Leaving OVSEntering OVS Leaving OVS Throughput measurement points
  20. 20. Summary  Throughput of foreground per second  Leaving OVS (sender), the throughput is decreased  At receiver, the throughput is also decreased at iperf (RX) 19 Copyright 2016 FUJITSU LABORATORIES LIMITED < Sender > < Receiver > Packet drop at qdisc Packet drop at socket buffer by microbursts
  21. 21. Conclusion We investigated the occurrence of microbursts on NFVI  A major cause of microburst is packet queuing on qdisc, and the packet loss at socket buffer on receiver is occurred by the microbursts  At qdisc, the throughput is decreased to 63% and the queue size is not enough to absorb the packets  At socket buffer (rx), the throughput is decreased to 41% and the buffer capacity is also not enough We found the cause of microbursts using kernel profiling  Although the total sending rate is 10 Gbps, qdisc at the sender is frequently full 20 Copyright 2016 FUJITSU LABORATORIES LIMITED
  22. 22. Future work  Clarify the cause why qdisc is frequently full although the total sending rate is 10 Gbps  Analyze the profiling results with kernel trace  Modify Linux kernel to change the packet capture point  Extend the experiments  Without OVS (Clarify the overhead of OVS)  With TCP, DPDK OVS, VMs based on vhost 21 Copyright 2016 FUJITSU LABORATORIES LIMITED
  23. 23. Copyright 2015 FUJITSU LABORATORIES LIMITED22

×