Kvm optimizations

•

5 likes•3,022 views

OpenNebula Project

CentOS Dojo 2013

KVM Optimizations and Best Practices
From the Desktop to the Datacenter

Jaime Melis (opennebula.org)

Summary
● Tools
● Drivers
● Image Backends
● CPU
● Memory
● Networking
● Desktop Sharing
● Cloud point-of-view Challenges

Tools
● Libvirtd !!!!!!
● virt-manager
● virt-install
● qemu-img
● virt-alignment-scan (libguestfs)
● lstopo
● OpenNebula

Drivers
● Easy... use virtio!!!
● Huge performance gain
● No emulation
● Cooperation with the hypervisor
● Integrated into the linux kernel

Image Backends
● Regular file
● Qcow2
– snapshots (beware of the sync!)
– additonal layer, less performance
– compressed
● LVM → block device (san storage)
● Virt-alignment-scan
● Cache
– Writethrough (host page on, guest disk write cache off)
– Writeback (host page on, guest disk write cache on)
● Good overall I/O Performance
– None (host page off, guest disk write cache on)
● Good write performance

CPU - pinning
● NUMA (Non-Uniform Memory Access)
● Shared cache ~ 15% improvement
● Bad for different tasks
● Processor availability
● virsh capabilities
● lstopo

CPU - pinning

<cputune>
<vcpupin vcpu="0" cpuset="1-4,^2"/>
<vcpupin vcpu="1" cpuset="0,1"/>
<vcpupin vcpu="2" cpuset="2,3"/>
<vcpupin vcpu="3" cpuset="0,4"/>
...
</cputune>

CPU - cgroups
● Limit, account and
● Bad for different tasks
● Processor availability
● virsh capabilities
● lstopo

CPU - cgroups
● Limit, account and isolate resource usage

<cputune>
<shares>2048</shares>
...
</cputune>

CPU - model
● Subset of features
● Increase performance

● Nested virtualization

● /usr/share/libvirt/cpu_map.xml

<cpu match='exact'>
<model fallback='allow'>core2duo</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='2' threads='1'/>
<feature policy='disable' name='lahf_lm'/>
</cpu>

Memory - KSM
● Kernel Samepage Merging
● Combines memory private pages
● Very useful for VMs !!!!
● Increases VM density
● Enable by default

Networking7
● MacVTap (direct)... NO!
● Bridged networking... way to go!
● Disable STP
● Optimal configuration
● NAT (masquerading)
● iptables

Desktop Sharing
● VNC
● SPICE
– qlx driver
– Redirect printers
– Usb (mass-storage)
– Audio

Cloud Challenges
● Virt-manager + LVM + DHCP => 10 hosts
● Live-migration to any host
● Image layout – don't waste space
● Contextualization: install once and deploy
many
● Network Isolation
● Mac/IP spoofing
● Multi-tenancy

Cloud Challenges - ebtables
● Isolate
● Prevent mac spoofing

# Drop packets that don't match the network's MAC Address
-s ! <mac_address>/ff:ff:ff:ff:ff:0 -o <tap_device> -j DROP

# Prevent MAC spoofing
-s ! <mac_address> -i <tap_device> -j DROP

Cloud Challenges – What storage?
● No Storage Holy Grail

LVM2 iSCSI Qcow2 + DFS

Provisioning time + ++ +++

I/O ++ ++ +

Network ++ +++ +

Snapshotting - - +

Livemigration + - +

Thanks for attending!

jmelis - #opennebula @Freenode

What's hot

Achieving the ultimate performance with KVM

ShapeBlue

Disk Performance Comparison Xen v.s. KVM

nknytk

Achieving the ultimate performance with KVM

StorPool Storage

SiteGround Tech TeamBuilding

Marian Marinov

Cinder Live Migration and Replication - OpenStack Summit Austin

Ed Balduf

Gluster volume snapshot

Rajesh Joseph

Libvirt and bhyve under FreeBSD

Craig Rodrigues

Control your service resources with systemd

Marian Marinov

Dave Gilbert - KVM and QEMU

Danny Abukalam

Virtual machines in the cloud typically run existing general-purpose operating systems such as Linux. We notice that the cloud’s hypervisor already provides some features, such as isolation and hardware abstraction, which are duplicated by traditional operating systems, and that this duplication comes at a cost. We present the design and implementation of OSv, a new guest operating system designed specifically for running a single application on a virtual machine in the cloud. It addresses the duplication issues by using a low-overhead library-OS-like design. It runs existing applications written for Linux, as well as new applications written for OSv. We demonstrate that OSv is able to efficiently run a variety of existing applications. We demonstrate its sub-second boot time, small OS image and how it makes more memory available to the application. For unmodified network-intensive applications, we demonstrate up to 25% increase in throughput and 47% decrease in latency. By using non-POSIX network APIs, we can further improve performance and demonstrate a 290% increase in Memcached throughput.

OSv at Usenix ATC 2014

Don Marti

libvirt is an important piece of the overall open source virtualization management puzzle. Many of the open source virtualization management applications that users enjoy are based on libvirt, since it provides a normalized API for managing heterogeneous hypervisors. For Xen to enjoy this greater ecosystem of open source virtualization tools, it must be well supported and maintained in libvirt. This presentation will give a basic overview of libvirt, discuss the latest status of the libvirt libxenlight driver (also known as the libxl driver), and discuss future improvements planned for the driver.

XPDS14: libvirt support for libxenlight - James Fehlig, SUSE

The Linux Foundation

Storage systems continue to deliver better performance year after year. High performance solutions are now available off-the-shelf, allowing users to boost their servers with drives capable of achieving several GB/s worth of throughput per host. To fully utilise such devices, workloads with large queue depths are often necessary. In virtual environments, this translates into aggregate workloads coming from multiple virtual machines. Having previously addressed the impact of low latency devices in virtualised platforms, we are now aiming at optimising aggregate workloads. We will discuss the existing memory grant technologies available in Xen and compare trade-offs and performance implications of each: grant mapping, persistent grants and grant copy. For the first time, we will present grant copy as an alternative and show measurements over 7 GB/s, maxing out a set of local SSDs.

XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix

The Linux Foundation

KVM tools and enterprise usage

vincentvdk

During last year, we have seen the rise of LibraryOSes in the hypervisor world. Fast, scalable and light, they bring to hypervisors the resource efficiency of Containers while maintaining many of its well known isolation and manageability advantages. While most LibraryOSes are niche in what they do, OSv is designed to run almost any POSIX compliant application and just have a slight focus in the Java application. Which applications can we run? And why? What's the story with Java and what can users gain from it? And more importantly: How can Xen users and the Xen community at large benefit from OSv? Those are some of the questions that I intend to cover in this presentation.

XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems

The Linux Foundation

Improve your storage with bcachefs

Marian Marinov

OpenNebulaConf2015 2.02 Backing up your VM’s with Bacula - Alberto García

OpenNebula Project

Vagrant

Denys Haryachyy

s6196-chris-huybregts-microsoft-new-gpu-virtualization-technologies

Chris Huybregts

OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...

OpenNebula Project

OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT

OpenNebula Project

What's hot (20)

Achieving the ultimate performance with KVM

Disk Performance Comparison Xen v.s. KVM

Achieving the ultimate performance with KVM

SiteGround Tech TeamBuilding

Cinder Live Migration and Replication - OpenStack Summit Austin

Gluster volume snapshot

Libvirt and bhyve under FreeBSD

Control your service resources with systemd

Dave Gilbert - KVM and QEMU

OSv at Usenix ATC 2014

XPDS14: libvirt support for libxenlight - James Fehlig, SUSE

XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix

KVM tools and enterprise usage

XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems

Improve your storage with bcachefs

OpenNebulaConf2015 2.02 Backing up your VM’s with Bacula - Alberto García

Vagrant

s6196-chris-huybregts-microsoft-new-gpu-virtualization-technologies

OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...

OpenNebulaConf 2016 - The DRBD SDS for OpenNebula by Philipp Reisner, LINBIT

Similar to Kvm optimizations

Lightweight Virtualization with Linux Containers and Docker | YaC 2013

dotCloud

Lightweight Virtualization with Linux Containers and Docker I YaC 2013

Docker, Inc.

Lightweight virtualization", also called "OS-level virtualization", is not new. On Linux it evolved from VServer to OpenVZ, and, more recently, to Linux Containers (LXC). It is not Linux-specific; on FreeBSD it's called "Jails", while on Solaris it’s "Zones". Some of those have been available for a decade and are widely used to provide VPS (Virtual Private Servers), cheaper alternatives to virtual machines or physical servers. But containers have other purposes and are increasingly popular as the core components of public and private Platform-as-a-Service (PAAS), among others. Just like a virtual machine, a Linux Container can run (almost) anywhere. But containers have many advantages over VMs: they are lightweight and easier to manage. After operating a large-scale PAAS for a few years, dotCloud realized that with those advantages, containers could become the perfect format for software delivery, since that is how dotCloud delivers from their build system to their hosts. To make it happen everywhere, dotCloud open-sourced Docker, the next generation of the containers engine powering its PAAS. Docker has been extremely successful so far, being adopted by many projects in various fields: PAAS, of course, but also continuous integration, testing, and more.

"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...

Yandex

KVM (Kernel-based Virtual Machine) is a full virtualization solution built into the Linux kernel. OpenStack Foundation user surveys consistently indicate that KVM is the most commonly used Hypervisor for OpenStack deployments, managed using the Libvirt driver for OpenStack Compute (Nova). Despite this sustained popularity development of the driver, and indeed the underlying Hypervisor itself, continues at a frantic pace. This presentation will help you make sense of it all starting with an overview of the way Nova, Libvirt, and KVM interact before analysing progress made in Kilo on utilizing key Libvirt/KVM features in Nova including: Instance vCPU pinning Huge page backed instances Enhanced NUMA topology awareness ...and more! The session will close with a discussion of how in addition to exposing existing Libvirt/KVM features emerging OpenStack use cases - such as Network Function Virtualization (NFV) and High Performance Computing (HPC) - are driving open innovation in the Libvirt, QEMU, and KVM projects themselves.

Libvirt/KVM Driver Update (Kilo)

Stephen Gordon

Malware analysis

xabean

Ganeti is a robust cluster virtualization management software tool. It’s built on top of existing virtualization technologies such as Xen and KVM and other Open Source software. Its integration with various technologies such as DRBD and LVM results in a cheaper High Availability infrastructure and linear scaling. This hands-on tutorial will cover a basic overview of Ganeti, the step-by-step install & setup of a single-node and multi-node Ganeti cluster, operating the cluster, and some best practices of Ganeti.

Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012

Lance Albertson

Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ

Jérôme Petazzoni

Docker Introduction + what is new in 0.9

Jérôme Petazzoni

Data deduplication is a hot topic in storage and saves significant disk space for many environments, with some trade offs. We’ll discuss what deduplication is and where the Open Source solutions are versus commercial offerings. Presentation will lean towards the practical – where attendees can use it in their real world projects (what works, what doesn’t, should you use in production, etcetera).

Open Source Data Deduplication

RedWireServices

Storage-Performance-Tuning-for-FAST-Virtual-Machines_Fam-Zheng.pdf

aaajjj4

UKOUG 2011: Practical MySQL Tuning

FromDual GmbH

Deploying containers and managing them on multiple Docker hosts, Docker Meetu...

dotCloud

Many companies build new-age KVM clouds, only to find out that their applications & workloads do not perform well. In this talk we’ll show you how to get the most out of your KVM cloud and how to optimize it for performance: You’ll understand why performance matters and how to measure it properly. We’ll teach you how to optimize CPU and memory for ultimate performance and how to tune the storage layer for performance. You’ll find out what are the main components of an efficient new-age cloud and which network components work best. In addition, you’ll learn how to select the right hardware to achieve unmatched performance for your new-age cloud and applications. Venko Moyankov is an experienced system administrator and solutions architect at StorPool storage. He has experience with managing large virtualizations, working in telcos, designing and supporting the infrastructure of large enterprises. In the last year, his focus has been in helping companies globally to build the best storage solution according to their needs and projects.

Achieving the Ultimate Performance with KVM

data://disrupted®

OpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebula

OpenNebula Project

Qemu - Raspberry | while42 Singapore #2

While42

Docker and-containers-for-development-and-deployment-scale12x

rkr10

LXC on Ganeti

kawamuray

OpenNebulaConf 2013 - Hands-on Tutorial: 2. Installing and Basic Usage

OpenNebula Project

Kvm performance optimization for ubuntu

Sim Janghoon

Deploying Containers and Managing Them

Docker, Inc.

Similar to Kvm optimizations (20)

Lightweight Virtualization with Linux Containers and Docker | YaC 2013

Lightweight Virtualization with Linux Containers and Docker I YaC 2013

"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...

Libvirt/KVM Driver Update (Kilo)

Malware analysis

Hands on Virtualization with Ganeti (part 1) - LinuxCon 2012

Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ

Docker Introduction + what is new in 0.9

Open Source Data Deduplication

Storage-Performance-Tuning-for-FAST-Virtual-Machines_Fam-Zheng.pdf

UKOUG 2011: Practical MySQL Tuning

Deploying containers and managing them on multiple Docker hosts, Docker Meetu...

Achieving the Ultimate Performance with KVM

OpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebula

Qemu - Raspberry | while42 Singapore #2

Docker and-containers-for-development-and-deployment-scale12x

LXC on Ganeti

OpenNebulaConf 2013 - Hands-on Tutorial: 2. Installing and Basic Usage

Kvm performance optimization for ubuntu

Deploying Containers and Managing Them

More from OpenNebula Project

We've made our way into the world of open cloud — where each organization can find the right cloud for its unique needs. A single cloud management platform cannot be all things to all people. There will be a cloud space with several offerings focused on different environments and/or industries. The OpenNebula commitment to the open cloud is at the very base of its mission — to become the simplest cloud enabling platform — and its purpose — to bring simplicity to the private and hybrid enterprise cloud. OpenNebula exists to help companies build simple, cost-effective, reliable, open enterprise clouds on existing IT infrastructure. The OpenNebula Conference will be a great opportunity to communicate and share our vision and commitment, to look back at how the project has grown in the last 9 years, and to shed some insight into what to expect from the project in the near future.

OpenNebulaConf2019 - Welcome and Project Update - Ignacio M. Llorente, Rubén ...

OpenNebula Project

Computer networks are undergoing a phenomenal growth, driven by the rapidly increasing number of nodes constituting the networks. At the same time, the number of security threats on Internet and intranet networks is constantly increasing, and the testing and experimentation of cyber defense solutions require the availability of separate, test environments that best reflect the complexity of a real system. Such environments support the deployment and monitoring of complex mission-driven network scenarios, and cyber security training activities, thus enabling enterprises to study cyber defense strategies and allowing security researchers to evaluate their algorithms at scale. The main objective is delivering to researchers and practitioners an overview of the technological means and the practical steps to setup a private cloud platform based on OpenNebula for the creation and management of virtual environments that support cyber-security activities of training and testing, as well as an overview of its possible applications in the cyber security domain. In particular: 1. We describe our infrastructure based on OpenNebula 2. We overview our application, sitting on top of OpenNebula, as well as the technological tools involved in the management of its lifecycle (e.g., Ansible) . 3. We show how the platform can support various examples of security research activities  [References] Building an emulation environment for cyber security analyses of complex networked systems, Tanasache, Florin Dragos and Sorella, Mara and Bonomi, Silvia and Rapone, Raniero and Meacci, Davide, ICDCN '19, ACM, 2019

OpenNebulaConf2019 - Building Virtual Environments for Security Analyses of C...

OpenNebula Project

OpenNebulaConf2019 - CORD and Edge computing with OpenNebula - Alfonso Aureli...

OpenNebula Project

OpenNebulaConf2019 - 6 years (+) OpenNebula - Lessons learned - Sebastian Man...

OpenNebula Project

OpenNebula users have a range of storage options available to them, including proprietary appliances, proprietary software and Open Source software projects. This session will present a fully Open Source approach, that tightly integrates with Linux, and makes full use of the mature building blocks within the Linux kernel (LVM, Software RAID, DM-crypt, NVMe-oF Target, DRBD, etc...), and delivers one of the highest performance open source storage stacks currently available. The core goal is to expose the improved performance of NVMe storage devices to VMs and containers. The solution covers both local NVMe drives and NVMe-oF. For interacting with NVMe-oF targets it supports the Swordfish-API and LVM & Linux’s software NVMe-oF target. The solution contains a storage addon for OpenNebula.

OpenNebulaConf2019 - Performant and Resilient Storage the Open Source & Linux...

OpenNebula Project

OpenNebulaConf2019 - Image Backups in OpenNebula - Momčilo Medić - ITAF

OpenNebula Project

At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud. I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.

OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Ph...

OpenNebula Project

OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...

OpenNebula Project

Replacing vCloud with OpenNebula

OpenNebula Project

NTS: What We Do With OpenNebula - and Why We Do It

OpenNebula Project

OpenNebula from the Perspective of an ISP

OpenNebula Project

NTS CAPTAIN / OpenNebula at Julius Blum GmbH

OpenNebula Project

Performant and Resilient Storage: The Open Source & Linux Way

OpenNebula Project

NetApp Hybrid Cloud with OpenNebula

OpenNebula Project

NSX with OpenNebula - upcoming 5.10

OpenNebula Project

Security for Private Cloud Environments

OpenNebula Project

CheckPoint R80.30 Installation on OpenNebula

OpenNebula Project

DE-CIX: CloudConnectivity

OpenNebula Project

DDC Demo

OpenNebula Project

Cloud Disaggregation with OpenNebula

OpenNebula Project

More from OpenNebula Project (20)