Xensummit2009 Io Virtualization Performance

926 views
851 views

Published on

Xen.org community presentation from Xen Summit Asia 2009

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
926
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Xensummit2009 Io Virtualization Performance

  1. 1. IO Virtualization Performance HUANG Zhiteng (zhiteng.huang@intel.com)
  2. 2. Agenda • IO Virtualization Overview • Software solution • Hardware solution • IO performance Trend • How IO Virtualization Performed in Micro Benchmark • Network • Disk • Performance in Enterprise Workloads • Web Server: PV, VT-d and Native performance • Database: VT-d vs. Native performance • Consolidated Workload: SR-IOV benefit • Direct IO (VT-d) Overhead Analysis Software 2 & Services group
  3. 3. IO Virtualization Overview IO Virtualization enables VMs to utilize Input/Output resources of hardware VMN platform. In this session, we cover network VM0 and storage. Software Solutions: Two solution we are familiar with on Xen: •Emulated Devices – QEMU Good compatibility, very poor performance •Para-Virtualized Devices Need driver support in guest, provides optimized performance compared to QEMU. Both require participation of Dom0 (driver domain) to serve an VM IO request. Software 3 & Services group
  4. 4. IO Virtualization Overview – Hardware Solution • VMDq (Virtual Machine Device Queue) • Separate Rx & Tx queue pairs of NIC for each VM, Network Only Offer 3 kinds of Software “switch”. Requires specific OS and VMM support. H/W assists to accelerate IO. • Direct IO (VT-d) VM Exclusively Single or •Improved IO performance through direct assignment of a Owns Device combination of I/O device to an unmodified or paravirtualized VM. technology can be used to • SR-IOV (Single Root I/O Virtualization) address various One Device, •Changes to I/O device silicon to support multiple PCI Multiple usages. device ID’s, thus one I/O device can support multiple Virtual direct assigned guests. Requires VT-d. Function Software 4 & Services group
  5. 5. IO Virtualization Overview - Trend Much Higher throughput, denser IO capacity. 40Gb/s and 100Gb/s Ethernet Scheduled to release Draft 3.0 in Nov. 2009, Standard Approval in 2010**. Fibre Channel over Ethernet (FCoE) Unified IO consolidates network (IP) and storage (SAN) to single connection. Solid State Drive (SSD) Provides hundreds MB/s bandwidth and >10,000 IOPS for single devices*. PCIe 2.0 Doubles the bit rate from 2.5GT/s to 5.0GT/s. Software 5 & Services * http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3403 group ** See http://en.wikipedia.org/wiki/100_Gigabit_Ethernet
  6. 6. How IO Virtualization Performed in Micro Benchmark – Network iperf: Transmiting Performance SR-IOV Dual-Port Iperf with 10Gb/s Ethernet NIC were 10GbE NIC 20 100% 18 used to benchmark TCP bandwidth of 16 19 80% Bandwidth (Gb/s) 14 12 60% different device models(*). 10 8 1.13x 9.54 40% 6 8.47 1.81x Thanks to VT-d, VM can easily 4 4.68 20% 2 0 0% achieved 10GbE line-rate in both cases HVM + PV driver PV Guest HVM + VT-d 30VMs+SR-IOV perf cpu% with relatively much lower resource iperf: Receiving Performance consumption. 10 100% 8 9.43 80% Bandwidth (Gb/s) Also with SR-IOV, we were able to get 6 3.04x 60% 19Gb/s transmitting performance with 4 40% 2 2.12x 3.10 20% 30VFs assigned to 30 VMs. 0 1.46 0% HVM + PV driver PV Guest HVM + VT-d perf cpu% Software 6 * HVM+ VT-d uses 2.6.27 kernel while PV guest and HVM+PV driver uses 2.6.18. * We turned off multiqueue support in NIC driver of HVM+VT-d due 2.6.18 kernel doesn’t have multi TX queue support. So for & Services iperf test, there was only one TX/RX queue in the NIC and all interrupts are sent to one physical core only. group * ITR (Interrupt Throttle Rate) was set to 8000 for all cases.
  7. 7. How IO Virtualization Performed in Micro Benchmark – Network (cont.) Packet Transmitting performance is another Packet Transmitting Performance essential aspect of high throughput network. 9.00 8.15 Using Linux kernel packet generator(pktgen) 8.00 Million Packet Per Second (mpp/s) 7.00 2.1x with small UDP packets (128 Byte), HVM+VT-d 6.00 5.00 3.90 can send over 4 million packets/s with 1 TX 4.00 3.00 queue and >8 million packets/s with 4 TX 11.7x 2.00 1.00 0.33 queue. 0.00 PV Guest 1 queue HVM+VT-d 1 queue HVM+VT-d 4 queue PV performance was far behind due to its long packet processing path. Software 7 & Services group
  8. 8. How IO Virtualization Performed in Micro Benchmark – Disk IO IOmeter: Disk Bandwidth 6,000 100.00% 4,911 5,000 80.00% We measured disk bandwidth with sequential Bandwidth (MB/s) 4,000 60.00% 3,000 2.9x 1,711 40.00% read and IOPS with random read to check block 2,000 1,000 20.00% device performance. 0 0.00% PV Guest HVM + VT-d HVM+VT-d out-perform PV guest to ~3.0x in IOmeter: Disk IOPS 20,000 18,725 100.00% both tests. 16,000 80.00% IO per Second 12,000 60.00% 2.7x 8,000 7,056 40.00% 4,000 20.00% 0 0.00% PV Guest HVM + VT-d Software 8 & Services group
  9. 9. Performance in Enterprise Workloads – Web Server Web Server simulates a support website where connected users browse and Web Server Performance download files. 30,000 100.00% 24,500 We measures maximum simultaneous user 25,000 80.00% sessions that web server can support while 20,000 60.00% satisfying the QoS criteria. Sessions 15,000 2.7x 40.00% 10,000 9,000 5,000 1.8x Only HVM+VT-d was able to push server’s 20.00% 5,000 utilization to ~100%. PV solution hit some - 0.00% HVM + PV driver PV guest HVM+ VT-d bottleneck that they failed to pass QoS while performance CPU utilization utilization is still <70%. Software 9 & Services group
  10. 10. Performance in Enterprise Workloads – Database Decision Support DB Performnce OLTP DB Performance 11,443 199.71 12000 10,762 184.08 200.00 10000 94.06% 150.00 127.56 8000 92.2% QphH 6000 100.00 4000 50.00 63.9% 2000 0 0.00 Native HVM + VT-d Native HVM + VT-d Storage & NIC HVM + VT-d Storage Decision Support DB requires high disk bandwidth while OLTP DB is IOPS bound and requires certain network bandwidth to connect to clients. HVM+VT-d combination achieved > 90% of native performance in these two DB workloads. Software 10 & Services group
  11. 11. Performance in Enterprise Workloads – Consolidation with SR-IOV Workload consolidates multiple tiles of servers to run on same physical machine. One tile SR-IOV Benefit: Consolidated Workload consists of 1 instance of Web Server, 1 J2EE 1.60 1.49 100 AppServer and 1 Mail Server, altogether 6 VMs. 1.40 90 1.20 80 It’s a complex workload, which consumes CPU, Ratio of Performance 70 1.00 1.00 memory, disk and network. 60 0.80 50 0.60 1.49x 40 PV solution could only supports 4 tiles on two 30 0.40 20 socket server and fails to pass QoS of Web 0.20 10 Server criteria before it saturates CPU. 0.00 0 PV Guest HVM+SR-IOV As a pilot, we enabled SR-IOV NIC for Web Ratio System Utilization Server. This brought >49% performance increase also allowed system to support two more tiles (12 VMs). Software 11 & Services group
  12. 12. Direct IO (VT-d) Overhead VT-d cases: Utilization Breakdown VT-d: XEN Cycles Breakdown 100 14.00% 90 11.878 12.00% 80 70 10.00% Utilization (%) 60 Dom0 4.73% 8.00% Interrupt Windows 50 Xen INTR 40 Guest Kernel 6.00% 30 APIC ACCESS Guest User 4.00% 20 IO INSTRUCTION 6.37% 10 5.94 2.00% 0 Disk Bandwidth Web Server 0.00% SPECWeb Web Server APIC access and interrupt delivery consumed the most cycles. (Note that some amount of interrupts arrive when CPU is HLT thus they are not counted.) In various workloads, we’ve seen Xen brings in about 5~12% overhead, which is being mainly spent on serving interrupts. Intel OTC team has developed patch set to eliminate part of Xen software overhead. Check out Xiaowei’s session for details. Software 12 & Services group
  13. 13. CREDIT Great thanks to DUAN, Ronghui and XIANG, Kai for providing data of VT-d network and SR-IOV. Software 13 & Services group
  14. 14. QUESTIONS? Software 14 SSG/SSD/SPA/PRC Scalability Lab & Services group
  15. 15. BACKUP Software 15 SSG/SSD/SPA/PRC Scalability Lab & Services group
  16. 16. Configuration Hardware Configuration: Intel® Nehalem-EP Server System Test Case VM Configuration CPU Info: 2 socket Nehalem 2.66 GHZ with 8MB LLC Cache, C0 stepping. Network Micro 4 vCPU, 64GB memory Hardware Prefetches OFF benchmark Turbo mode OFF, EIST OFF. NIC device Intel 10Gb XF SR NIC (82598EB)—2 single port NIC installed on Storage Micro 2 vCPU, 12GB memory machine and one dual port NIC installed on server. Benchmark RAID bus controller: LSI Logic MegaRAID SAS 8888elp x3 4 vCPU, 64GB memory DISK array x6 (each with 70GB X 12 SAS HDD). Web Server Memory Info 64GB memory (16x 4GB DDR3 1066MHz) , 32GB on each node. Database 4 vCPU, 12GB memory Software Configuration: Xen C/S:18771 for network/disk micro benchmark, 19591 for SR-IOV test Software 16 & Services group
  17. 17. 17 SSG/SSD/SPA/PRC Scalability Lab

×