Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OpenNebulaConf2019 - Nested Virtualization with OpenNebula: A Practical Approach - Venko Moyankov - StorPool

29 views

Published on

In this talk, StorPool will present the concept of nested virtualization and PCI pass-through. Nested virtualization is when you have virtual machines running inside other virtual machines. If done right, nested virtualization is extremely suitable for testing large virtualized environments, without the need for having hundreds of physical servers.

This approach empowers system administrators by reducing the time and effort needed to run/test multiple dynamic environments. This is done by having easier automation/provisioning, API control and much more. In a nested virtualization solution, the physical hardware is sliced and diced fast and easy, which gives you a large set of opportunities to experiment without the need for huge investments in physical hardware or time and effort needed to reinstall/set-up software.

Join this talk to learn more about why and how to do nested virtualization with OpenNebula and achieve an inception moment of having virtual machines, inside virtual machines, inside virtual machines…

Published in: Software
  • Be the first to comment

  • Be the first to like this

OpenNebulaConf2019 - Nested Virtualization with OpenNebula: A Practical Approach - Venko Moyankov - StorPool

  1. 1. Nested Virtualization with OpenNebula (and PCI Passthrough) A Practical Approach Venko Moyankov OpenNebulaConf 2019 Barcelona, October 21-22
  2. 2. The Problem KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM H-V H-V H-V XEN XEN XEN
  3. 3. KVM KVM StorPool Lab KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM also and more ... H-V H-V H-V XEN XEN XEN
  4. 4. The Technologies Behind ● Hardware virtualization (VT-x) ● Nested Virtualization ● VMCS Shadowing ● IOMMU (PCI Passthrough) ● SR-IOV ● ACS (IOMMU groups) ● OpenNebula PCI Passthrough ● libvirt Domain XML tweaks (VF Net only) Host OpenNebula
  5. 5. Hardware Virtualization VT-x AMD-V or SVM All CPUs, but may need to enable it in BIOS
  6. 6. Nested Virtualization Enables hardware virtualization in the guest KVM feature
  7. 7. VMCS Shadowing ● Hardware feature ● Accelerates nested virtualization ● Available in most CPUs since 2013 (Haswell) Source: https://software.intel.com/en-us/blogs/2014/12/12/enabling-virtual-machine-control-structure-shadowing-on-a-nested-virtual-machine
  8. 8. PCI Passthrough Allows guests to have direct exclusive access to PCI devices ● I/O MMU virtualization (IOMMU) VT-d AMD-Vi Mostly used for GPU and NIC Host GPU1 Hypervisor / VMM VM Host driver driver VM driver VM driver GPU2 GPU3GPU0 PCIe IOMMU / VT-d
  9. 9. SR-IOV ● Single Root I/O Virtualization ● One physical device appears as multiple virtual functions (VF) ● Allows different VMs to share a single PCIe hardware ● Mostly used for NIC Host NIC VF1 Hypervisor / VMM VM Host driver driver VM driver VM driver VF2 VF3PF PCIe IOMMU / VT-d
  10. 10. ACS and IOMMU Groups Host GPU1 Hypervisor / VMM VM Host driver driver VMVM driver VM driver GPU2 GPU3GPU0 IOMMU / VT-dRAM ACS PCIe
  11. 11. Let's Do It
  12. 12. Host Hardware Virtualization Check it is available with: lscpu | grep vmx Nested Virtualization kvm-intel.conf: options kvm-intel nested=1 Check it in the guest with: lscpu | grep vmx VMCS Shadowing kvm-intel.conf: options kvm-intel enable_shadow_vmcs=1
  13. 13. PCI Passthrough & SR-IOV PCI Passthrough Enabled via kernel options: Intel: intel_iommu=on iommu=pt AMD: amd_iommu=pt SR-IOV ● Ensure SR-IOV and VT-d are enabled in BIOS. ● Setup VFs: # echo '8' > /sys/class/net/eth3/device/sriov_numvfs
  14. 14. ACS and IOMMU Groups ● Check ACS is supported lspci -vv | egrep “Access Control Services” https://heiko-sieger.info/iommu-groups -what-you-need-to-consider/ ● Check IOMMU groups for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
  15. 15. Congratulations! You are ready to run Nested Virtualization ! Almost
  16. 16. Enable PCI passthrough in OpenNebula /var/lib/one/remotes/im/kvm-probes.d/pci.rb /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf :filter: '15b3:1018' :short_address: [] :device_name: []
  17. 17. Tweak domain.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xd8' slot='0x00' function='0x5'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0'/> </hostdev> <interface managed="yes" type="hostdev"> <driver name="vfio" /> <mac address="02:00:11:ab:cd:01" /> <source> <address bus="0xd8" domain="0x0000" function="0x5" slot="0x00" type="pci" /> </source> <address bus="0x01" domain="0x0000" function="0" slot="0x01" type="pci" /> </interface> https://github.com/OpenNebula/addon-storpool/blob/master/docs/ advanced_configuration.md#vms-domain-xml-tweaking
  18. 18. OpenNebula
  19. 19. Summary CPU BIOS Kernel KVM OpenNebula VT-x 🗸 🗸 Nested Virt. 🗸 VMCS Shadowing 🗸 🗸 🗸 IOMMU (PCI Passthrough) 🗸 🗸 🗸 🗸 SR-IOV 🗸 🗸 ACS (IOMMU groups) 🗸 libvirt Domain XML (VF Net only) 🗸
  20. 20. Q&A
  21. 21. Venko Moyankov venko@storpool.com StorPool Storage www.storpool.com @storpool Thank you!

×