SR-IOV, KVM and Intel X520 10Gbps cards on Debian/Stable
Upcoming SlideShare
Loading in...5
×
 

SR-IOV, KVM and Intel X520 10Gbps cards on Debian/Stable

on

  • 282 views

Playing with SR-IOV and KVM virtual machines under GNU/Linux Debian Operating Systems with Intel X520 (82599EB) 10Gbps cards

Playing with SR-IOV and KVM virtual machines under GNU/Linux Debian Operating Systems with Intel X520 (82599EB) 10Gbps cards

Statistics

Views

Total Views
282
Slideshare-icon Views on SlideShare
227
Embed Views
55

Actions

Likes
0
Downloads
9
Comments
0

2 Embeds 55

http://centoros.wordpress.com 45
https://centoros.wordpress.com 10

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    SR-IOV, KVM and Intel X520 10Gbps cards on Debian/Stable SR-IOV, KVM and Intel X520 10Gbps cards on Debian/Stable Presentation Transcript

    • SR-IOV and KVM virtual machines under GNU/Linux Debian Intel X520 (82599EB) 10Gbps cards Yoann Juet @ University of Nantes, France Information Technology Services Version 1.1 (19 Jun 2014)
    • 2/19 Our goal • Virtualize high-performance servers, firewalls requiring: - Low network latency and jitter - Low processor impact (I/O) - High throughput (10Gbps) • Solution: Single Root – IO Virtualization (SR-IOV) - A single PCI card is showed up as multiple virtual PCI cards - Exposes n virtual interfaces from a single physical interface > Shared bandwidth
    • 3/19 Prerequisites • Virtualization Technology for Directed I/O: Intel VT-d or AMD-Vi - Must be supported by both the CPU and the chipset - Guest machines gain direct memory access (DMA) to PCI(e) devices, such as Ethernet cards • PCI-SIG Single Root I/O Virtualization: SR-IOV - Must be supported by both the Ethernet cards and the BIOS - Guest machines are able to achieve ~ bare metal performance
    • 4/19 Technical environment • Dell PowerEdge R620 - Intel Xeon CPU E5-2609 - Dual Intel I350 (82599) 1000Base-T interfaces > Logical names eth0, eth1 - Dual Intel X520 (82599EB) SFP+ 10Gbps interfaces > SR-IOV compatible card > Logical names eth2, eth3 - Operating System Debian 7 (code name "Wheezy") > Installed on both hosts and guests machines
    • 5/19 BIOS Host machine • Ensure SR-IOV BIOS option is enabled - System BIOS > Integrated Devices > SR-IOV Global Enable
    • 6/19 Debian: Starting with SR-IOV Host machine • Some Kernel Requirements: CONFIG_PCI_IOV=y CONFIG_PCI_STUB=y CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y ONFIG_INTEL_IOMMU_DEFAULT_ON=y → Default Debian 7 kernel is not recommended for use with SR-IOV feature. Rather, prefer a recent kernel (we are using at this time 3.12.22) that fixes important bugs related to SR-IOV and provides performance improvements.
    • 7/19 Debian: Starting with SR-IOV Host machine • Check for SR-IOV hardware support: # lspci -v … 01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Dell Ethernet 10G 4P X520/I350 rNDC ... Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: ixgbe 01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Dell Ethernet 10G 4P X520/I350 rNDC ... Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: ixgbe eth2 eth3
    • 8/19 Debian: Starting with SR-IOV Host machine • Check for Intel's VT-d IOMMU support: # dmesg | egrep -i “DMA|IOMMU” ... dmar: IOMMU 0: reg_base_addr d3000000 ver 1:0 cap d2078c106f0462 ecap f020fe dmar: IOMMU 1: reg_base_addr df100000 ver 1:0 cap d2078c106f0462 ecap f020fe ... IOMMU: Setting identity map for device xxxxxxxx ... PCI-DMA: Intel(R) Virtualization Technology for Directed I/O … https://www.kernel.org/doc/Documentation/vfio.txt
    • 9/19 Debian: Starting with SR-IOV Host machine • Activate SR-IOV on both 10Gbps interfaces with 8 VFs (64 max. allowed) per PF - First method (deprecated): ixgbe module option # echo “options ixgbe max_vfs=8” >> /etc/modprobe.d/ixgbe.conf Then reboot or reload the ixgbe driver as shown below: # rmmod ixgbe && modprobe ixgbe max_vfs=8 ixgbe 0000:01:00.0: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface. ... ixgbe 0000:01:00.1: Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface. - Second method (preferred): Kernel 3.8+ brings sysfs interface support for getting the maximal number of VF for a given PF, as well as for getting and setting the current number of VF: # echo 8 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs # echo 8 > /sys/bus/pci/devices/0000:01:00.1/sriov_numvfs device ID for eth2, eth3
    • 10/19 Debian: Starting with SR-IOV Host machine • Ensure that new virtual PCIe devices (Virtual Functions) are visible: # lspci ... 01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 01:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 8 VFs on the second PF (eth3) 8 VFs on the first PF (eth2)
    • 11/19 Debian: Starting with SR-IOV Host machine • Each VF behaves like a traditional network interface - below, logical names eth4 eth19→ # ip link show | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 6: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 7: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 8: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 9: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 ... 17: eth13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 18: eth14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 19: eth11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 20: eth15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 21: eth16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 22: eth17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 23: eth19: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 16 unused VFs are visible 4 NICs on-board
    • 12/19 Debian: PCI passthrough with libvirt Host machine • First method: Assignment with <hostdev> block <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='<dom_id>' bus='<bus_id>' slot='<slot_id>' function='<func_id>'/> </source> </hostdev> Where <dom_id>, <bus_id>, <slot_id> and <func_id> are given by: # lspci -D 0000:01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ... 0000:01:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) <func_id> <slot_id> <bus_id> <dom_id> - First virtual PCIe device (VF0): <address domain='0x0000' bus='0x01' slot='0x10' function='0x0'/> - Last virtual PCIe device (VF7): <address domain='0x0000' bus='0x01' slot='0x11' function='0x6'/> Excerpt from guest XML file
    • 13/19 Debian: PCI passthrough with libvirt Host machine • Second method: Assignment with <interface type='hostdev'> block <interface type='hostdev' managed='yes'> <mac address='<virtual_mac_address>'/> <source> <address domain='<dom_id>' bus='<bus_id>' slot='<slot_id>' function='<func_id>'/> </source> </interface> Where <virtual_mac_address>' is the guest interface virtual mac address. <dom_id>, <bus_id>, <slot_id>, <func_id> are defined in the previous slide. Unfortunately, such an assignment method doesn't work on a standard Debian 7 distro (qemu-kvm 1.1.2, libvirt 0.9.12) need to upgrade qemu-kvm to version 1.3 or later→ # virsh define 01-test.xml Domain 01-test defined from 01-test.xml # virsh start 01-test error: Failed to start domain 01-test error: An error occurred, but the cause is unknown Excerpt from guest XML file
    • 14/19 Debian: PCI passthrough with libvirt Host machine • Third method: Assignment from a pool of VFs <network> <name>sriov</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='<iface>'/> </forward> </network> <interface type='network'> <source network='sriov'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> Again, such an assignment method is currently unsupported on Debian 7 need to upgrade libvirt to version 0.10.0→ or later as well as qemu-kvm for VFIO PCI device assignment Network XML file Directory /etc/libvirt/qemu/networks/ Excerpt from guest XML file
    • 15/19 Debian: PCI passthrough with libvirt Host machine • Third method is preferred for its simplicity, but, it requires newer versions of libvirt and qemu-kvm Debian/Backports→ # echo "deb ftp://<mirror-debian>/debian wheezy-backports main" >> /etc/apt/sources.list # apt-get update # apt-get -t wheezy-backports install libvirt-bin # apt-get -t wheezy-backports install qemu-kvm • Check for correct installation: # virsh version Compiled against library: libvirt 1.2.4 Using library: libvirt 1.2.4 Using API: QEMU 1.2.4 Running hypervisor: QEMU 2.0.0
    • 16/19 Debian: PCI passthrough with libvirt Host machine • Assign two pools of PCIe devices to passthrough ; no need to worry about VF PCI IDs... # vi /etc/libvirt/qemu/networks/pf-eth2.xml <network> <name>pf-eth2</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='eth2'/> </forward> </network> # virsh net-define /etc/libvirt/qemu/networks/pf-eth2.xml # virsh net-start pf-eth2 # virsh net-autostart pf-eth2 # modprobe vfio # vi /etc/libvirt/qemu/networks/pf-eth3.xml <network> <name>pf-eth3</name> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <pf dev='eth3'/> </forward> </network> # virsh net-define /etc/libvirt/qemu/networks/pf-eth3.xml # virsh net-start pf-eth3 # virsh net-autostart pf-eth3 # virsh net-list
    • 17/19 Debian: PCI passthrough with libvirt Host machine • In each guest XML file, specify the source pool, vlan id as well as (if required) the interface mac address # vi /etc/libvirt/qemu/myguest.xml ... <interface type='network'> <source network='pf-eth<2|3>'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> ... # virsh define myguest.xml # virsh autostart myguest # virsh start myguest # vi /etc/libvirt/qemu/myguest.xml ... <interface type='network'> <mac address='<mac-address>'/> <source network='pf-eth<2|3>'/> <vlan> <tag id='<vlan_id>'/> </vlan> </interface> ... # virsh define myguest.xml # virsh autostart myguest # virsh start myguest OR
    • 18/19 Debian: Starting Guest machine • “a pure” Debian 7 (kernel 3.2.x) works perfectly on guest machines • Virtual interfaces are using the driver ixgbevf
    • 19/19 University of Nantes – IT Services Questions Yoann (dot) Juet (at) univ–nantes.fr