SR-IOV, KVM and Intel X520 10Gbps cards on Debian/Stable

  • 1,140 views
Uploaded on

Playing with SR-IOV and KVM virtual machines under GNU/Linux Debian Operating Systems with Intel X520 (82599EB) 10Gbps cards

Playing with SR-IOV and KVM virtual machines under GNU/Linux Debian Operating Systems with Intel X520 (82599EB) 10Gbps cards

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,140
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
24
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. SR-IOV and KVM virtual machines under GNU/Linux Debian Intel X520 (82599EB) 10Gbps cards Yoann Juet @ University of Nantes, France Information Technology Services Version 1.1 (19 Jun 2014)
  • 2. 2/19 Our goal • Virtualize high-performance servers, firewalls requiring: - Low network latency and jitter - Low processor impact (I/O) - High throughput (10Gbps) • Solution: Single Root – IO Virtualization (SR-IOV) - A single PCI card is showed up as multiple virtual PCI cards - Exposes n virtual interfaces from a single physical interface > Shared bandwidth
  • 3. 3/19 Prerequisites • Virtualization Technology for Directed I/O: Intel VT-d or AMD-Vi - Must be supported by both the CPU and the chipset - Guest machines gain direct memory access (DMA) to PCI(e) devices, such as Ethernet cards • PCI-SIG Single Root I/O Virtualization: SR-IOV - Must be supported by both the Ethernet cards and the BIOS - Guest machines are able to achieve ~ bare metal performance
  • 4. 4/19 Technical environment • Dell PowerEdge R620 - Intel Xeon CPU E5-2609 - Dual Intel I350 (82599) 1000Base-T interfaces > Logical names eth0, eth1 - Dual Intel X520 (82599EB) SFP+ 10Gbps interfaces > SR-IOV compatible card > Logical names eth2, eth3 - Operating System Debian 7 (code name "Wheezy") > Installed on both hosts and guests machines
  • 5. 5/19 BIOS Host machine • Ensure SR-IOV BIOS option is enabled - System BIOS > Integrated Devices > SR-IOV Global Enable
  • 6. 6/19 Debian: Starting with SR-IOV Host machine • Some Kernel Requirements: CONFIG_PCI_IOV=y CONFIG_PCI_STUB=y CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y ONFIG_INTEL_IOMMU_DEFAULT_ON=y → Default Debian 7 kernel is not recommended for use with SR-IOV feature. Rather, prefer a recent kernel (we are using at this time 3.12.22) that fixes important bugs related to SR-IOV and provides performance improvements.
  • 7. 7/19 Debian: Starting with SR-IOV Host machine • Check for SR-IOV hardware support: # lspci -v … 01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Dell Ethernet 10G 4P X520/I350 rNDC ... Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: ixgbe 01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Dell Ethernet 10G 4P X520/I350 rNDC ... Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: ixgbe eth2 eth3
  • 8. 8/19 Debian: Starting with SR-IOV Host machine • Check for Intel's VT-d IOMMU support: # dmesg | egrep -i “DMA|IOMMU” ... dmar: IOMMU 0: reg_base_addr d3000000 ver 1:0 cap d2078c106f0462 ecap f020fe dmar: IOMMU 1: reg_base_addr df100000 ver 1:0 cap d2078c106f0462 ecap f020fe ... IOMMU: Setting identity map for device xxxxxxxx ... PCI-DMA: Intel(R) Virtualization Technology for Directed I/O … https://www.kernel.org/doc/Documentation/vfio.txt
  • 9. 9/19 Debian: Starting with SR-IOV Host machine • Activate SR-IOV on both 10Gbps interfaces with 8 VFs (64 max. allowed) per PF - First method (deprecated): ixgbe module option # echo “options ixgbe max_vfs=8” >> /etc/modprobe.d/ixgbe.conf Then reboot or reload the ixgbe driver as shown below: # rmmod ixgbe && modprobe ixgbe max_vfs=8 ixgbe 0000:01:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface. ... ixgbe 0000:01:00.1: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface. - Second method (preferred): Kernel 3.8+ brings sysfs interface support for getting the maximal number of VF for a given PF, as well as for getting and setting the current number of VF: # echo 8 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs # echo 8 > /sys/bus/pci/devices/0000:01:00.1/sriov_numvfs device ID for eth2, eth3
  • 10. 10/19 Debian: Starting with SR-IOV Host machine • Ensure that new virtual PCIe devices (Virtual Functions) are visible: # lspci ... 01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 8 VFs on the second PF (eth3) 8 VFs on the first PF (eth2)
  • 11. 11/19 Debian: Starting with SR-IOV Host machine • Each VF behaves like a traditional network interface - below, logical names eth4 eth19→ # ip link show | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 6: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 7: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 8: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 9: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 ... 17: eth13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 18: eth14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 19: eth11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 20: eth15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 21: eth16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 22: eth17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 23: eth19: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 16 unused VFs are visible 4 NICs on-board
  • 12. 12/19 Debian: PCI passthrough with libvirt Host machine • First method: Assignment with <hostdev> block <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='<dom_id>' bus='<bus_id>' slot='<slot_id>' function='<func_id>'/> </source> </hostdev> Where <dom_id>, <bus_id>, <slot_id> and <func_id> are given by: # lspci -D 0000:01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ... 0000:01:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) <func_id> <slot_id> <bus_id> <dom_id> - First virtual PCIe device (VF0): <address domain='0x0000' bus='0x01' slot='0x10' function='0x0'/> - Last virtual PCIe device (VF7): <address domain='0x0000' bus='0x01' slot='0x11' function='0x6'/> Excerpt from guest XML file
  • 13. 13/19 Debian: PCI passthrough with libvirt Host machine • Second method: Assignment with <interface type='hostdev'> block <interface type='hostdev' managed='yes'> <mac address='<virtual_mac_address>'/> <source> <address domain='<dom_id>' bus='<bus_id>' slot='<slot_id>' function='<func_id>'/> </source> </interface> Where <virtual_mac_address>' is the guest interface virtual mac address. <dom_id>, <bus_id>, <slot_id>, <func_id> are defined in the previous slide. Unfortunately, such an assignment method doesn't work on a standard Debian 7 distro (qemu-kvm 1.1.2, libvirt 0.9.12) need to upgrade qemu-kvm to version 1.3 or later→ # virsh define 01-test.xml Domain 01-test defined from 01-test.xml # virsh start 01-test error: Failed to start domain 01-test error: An error occurred, but the cause is unknown Excerpt from guest XML file
  • 14. 14/19 Debian: PCI passthrough with libvirt Host machine • Third method: Assignment from a pool of VFs <network> <name>sriov</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='<iface>'/> </forward> </network> <interface type='network'> <source network='sriov'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> Again, such an assignment method is currently unsupported on Debian 7 need to upgrade libvirt to version 0.10.0→ or later as well as qemu-kvm for VFIO PCI device assignment Network XML file Directory /etc/libvirt/qemu/networks/ Excerpt from guest XML file
  • 15. 15/19 Debian: PCI passthrough with libvirt Host machine • Third method is preferred for its simplicity, but, it requires newer versions of libvirt and qemu-kvm Debian/Backports→ # echo "deb ftp://<mirror-debian>/debian wheezy-backports main" >> /etc/apt/sources.list # apt-get update # apt-get -t wheezy-backports install libvirt-bin # apt-get -t wheezy-backports install qemu-kvm • Check for correct installation: # virsh version Compiled against library: libvirt 1.2.4 Using library: libvirt 1.2.4 Using API: QEMU 1.2.4 Running hypervisor: QEMU 2.0.0
  • 16. 16/19 Debian: PCI passthrough with libvirt Host machine • Assign two pools of PCIe devices to passthrough ; no need to worry about VF PCI IDs... # vi /etc/libvirt/qemu/networks/pf-eth2.xml <network> <name>pf-eth2</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='eth2'/> </forward> </network> # virsh net-define /etc/libvirt/qemu/networks/pf-eth2.xml # virsh net-start pf-eth2 # virsh net-autostart pf-eth2 # modprobe vfio # vi /etc/libvirt/qemu/networks/pf-eth3.xml <network> <name>pf-eth3</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='eth3'/> </forward> </network> # virsh net-define /etc/libvirt/qemu/networks/pf-eth3.xml # virsh net-start pf-eth3 # virsh net-autostart pf-eth3 # virsh net-list
  • 17. 17/19 Debian: PCI passthrough with libvirt Host machine • In each guest XML file, specify the source pool, vlan id as well as (if required) the interface mac address # vi /etc/libvirt/qemu/myguest.xml ... <interface type='network'> <source network='pf-eth<2|3>'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> ... # virsh define myguest.xml # virsh autostart myguest # virsh start myguest # vi /etc/libvirt/qemu/myguest.xml ... <interface type='network'> <mac address='<mac-address>'/> <source network='pf-eth<2|3>'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> ... # virsh define myguest.xml # virsh autostart myguest # virsh start myguest OR
  • 18. 18/19 Debian: Starting Guest machine • “a pure” Debian 7 (kernel 3.2.x) works perfectly on guest machines • Virtual interfaces are using the driver ixgbevf
  • 19. 19/19 University of Nantes – IT Services Questions Yoann (dot) Juet (at) univ–nantes.fr