2010 xen-lisa

Xen Hypervisor Deployment, Management, and Cloud Computing Tools Todd Deshane and Patrick F. Wilbur Clarkson University

Copyright NoticeCopyright 2010, Todd Deshane and Patrick F. Wilbur.Last modified: September 30, 2010 11:02 PM EST.The Xen panda logo is property of Xen.org .==LICENSE:Todd Deshane and Patrick F. WilburDepartment of Mathematics and Computer ScienceClarkson UniversityPotsdam, NY USACurrent slides available at: http://cosi.clarkson.edu/docs/installingxen/These slides and content are released under the Creative Commons Attribution-Share Alike 3.0 Unportedlicense, available online at http://creativecommons.org/licenses/by-sa/3.0/You may share (copy, distribute, and transmit) this work, and remix (adapt) this work, as long as youattribute this work to the author and share adapted works under the same or similar license by leaving thisentire notice in place (including the original authors name/contact information/URL and this licensenotice).

About UsTodd Deshane is a Ph.D. Patrick F. Wilbur is a Ph.Dgraduate of Clarkson student at ClarksonUniversity. University.His research interests include His research interests includeinformation technology, security, usability, policy, andsecurity, virtualization, and systems architecture.usability.http://todddeshane.net http://pdub.net

AcknowledgmentsThis 2010 Xen Training / Tutorial, by Todd Deshane andPatrick F. Wilbur, is derived from the 2009 Xen Training /Tutorial as updated by Zach Shepherd and Jeanna Matthewsfrom the original version written by Zach Shepherd and WenjinHu, originally derived from materials written by Todd Deshaneand Patrick F. Wilbur.Portions of this work are inspired by JeremyFitzhardinges Pieces of Xen slides.Variations of this work have been presented numerous times atthe USENIX Annual Technical Conference, USENIX LISA, andat other various locations across the United States.

Overview● Session 1: Xen Introduction and Installing Xen● Session 2: Guest Creation and Management● Session 3: Xen in the Datacenter● Session 4: Xen in the Cloud

Session 1Xen Introduction and Installing Xen

Xen Overview

Xen and the Art of Virtualization ● Xen is a virtualization system supporting both paravirtualization and hardware-assisted full virtualization ● Name from neXt gENeration virtualization ● Initially created by University of Cambridge Computer Laboratory ● Open source (licensed under GPLv2)

Xen Virtualization Basics● A physical machine runs a program to manage virtual machines (Virtual Machine Monitor or hypervisor)● On the physical machine, there are one or more virtual machines (domains) running● A virtual machine is an encapsulated operating system which can run applications as a physical machine● The management virtual machine (Domain0) is responsible for interacting with the hypervisor● Other virtual machines are called guests

Ways to Use Virtualization● Fully utilize hardware resources: consolidation of workloads on single machine, exploitation of multiple cores● Running heterogeneous environments on one machine: different operating systems, different libraries● Isolation: separate workloads that have different requirement and/or to avoid attacks on one from compromising another● Manageability: rapid deployment and provisioning, backup/disaster recovery, portability

Types of VirtualizationEmulation:Fully-emulate the underlying hardware architectureFull virtualization:Simulate the base hardware architectureParavirtualization:Abstract the base architectureOS-level virtualization:Shared kernel (and architecture), separate user spaces

Virtualization in XenParavirtualization: ● Uses a modified Linux kernel ● Guest loads Dom0s pygrub or Dom0s kernel ● Front-end and back-end virtual device model ● Cannot run Windows ● Guest "knows" its a VM and talks with the hypervisorHardware-assisted full virtualization: ● Uses the same, normal, OS kernel ● Guest contains grub and kernel ● Normal device drivers ● Can run Windows ● Guest doesnt "know" its a VM, so hardware manages it

Reasons to Use XenParavirtualization (PV): ● High performance (claim to fame) ● High scalability ● Uses a modified operating systemHardware-assisted full virtualization (HVM): ● Co-evolution of hardware and software on x86 architecture ● Uses an unmodified operating system

Reasons to Use Xen● Xen is powered by a growing and active community and a diverse range of products and services● Xen offers high performance and an isolating architecture

Xen Architecture

Xen: Hypervisor Role● Thin, privileged abstraction layer between the hardware and operating systems● Defines the virtual machine that guest domains see instead of physical hardware: ○ Grants portions of physical resources to each guest ○ Exports simplified devices to guests ○ Enforces isolation among guests

Xen: Domain0 (Dom0) Role ● Creates and manages guest VMsxm (Xen management tool) A client application to send commands to xend ● Interacts with the Xen hypervisorxend (Xen daemon) Daemon to communicate with the hypervisor ● Supplies device and I/O services: ○ Runs (backend) device drivers ○ Provides domain storage

Normal Linux Boot ProcessBIOS Master Boot Record (MBR)GRUB Kernel ModuleLinux

The Xen Boot ProcessGRUB starts KernelHypervisor starts ModuleDomain0 starts DaemonXend starts xmGuest domain starts

Installing the Open-source Xen Hypervisor

Installing Xen from a Packageroot:~>yum install xen

Installing Xen from a Package ● OpenSUSE: Install with YaSThttp://www.susegeek.com/general/how-to-install-configure-xen-virtualization-in-opensuse-110/ ● Gentoo: Install with portagehttp://www.gentoo.org/doc/en/xen-guide.xml ● NetBSD: Xen package support as of BSD 4.0http://www.netbsd.org/ports/xen/howto.html

Installing Xen from SourceReasons to use the latest Xen version: ● Performance optimization, cutting-edge features ● Security and bug fixes ● Support for additional Dom0 OSes (Linux, Solaris, BSD) ● Ability to patch/customizeXen4 installation instructions, including from source:http://wiki.xensource.com/xenwiki/Xen4.0

Installing Xen from SourceNew in Xen4: ● blktap2 for VHD image support, snapshots and cloning ● Primary graphics card GPU passthru for high-performance 3D graphics and hardware-accelerated video ● TMEM allows improved utilization of unused (for example page cache) PV guest memory ● Memory page sharing and page-to-disc for HVM guests ● Copy-on-Write sharing of memory pages between VMs

Installing Xen from SourceAlso new in Xen4: ● Netchannel2 for improved networking acceleration, smart NICs, multi-queue support, SR-IOV functionality ● On-line resize of guest disks without reboot/shutdown ● Remus Fault Tolerance: live transactional synchronization of VM state between physical servers ● RAS features: physical cpu/memory hotplug

GRUB ConfigurationSample Xen GRUB Configuration:title Xen 3.4root (hd0,0)kernel /boot/xen-3.4.0.gzmodule /boot/vmlinuz-2.6.18.8-xen root=/dev/sda1module /boot/initrd.img-2.6.18.8-xenSample Normal Linux GRUB Configuration:title Ubuntu 2.6.24-23root (hd0,0)kernel /boot/vmlinuz-2.6.24-23-generic root=/dev/sda1initrd /boot/initrd.img-2.6.24-23-generic

Xend ConfigurationXen daemons configuration in /etc/xen/xend-config.sxp : ● Configure Xen to listen for remote connections ● Set max/min Dom0 CPU and memory resources ● Set up the virtual network: ○ Bridging ○ Routing ○ NAT ● Configure live migration (enable and set relocation port)

Sample Xend Configuration# (logfile /var/log/xen/xend.log)# (loglevel DEBUG)# (xend-http-server no)(xend-unix-server yes)(xend-unix-path /var/lib/xend/xend-socket)(network-script network-bridge)(vif-script vif-bridge)(dom0-min-mem 256)(dom0-cpus 0)# (xend-relocation-server no)# (xend-relocation-port 8002)# (vnc-listen 127.0.0.1)(vncpasswd )

Network ModesBridging mode:Guest domains are transparently on the samenetwork as Dom0Routing mode:Guest domains sit behind Dom0 and packets arerelayed to the network by Dom0NAT mode:Guest domains hide behind Dom0 using Dom0sIP for external traffic

Bridge Mode

Routing Mode

NAT Mode

Network Configuration1. Set network-script and vif-script in xend config.2. Restart Xen daemon:/etc/init.d/xend restartservice xend restart

Bridge Mode Configuration● Default network mode for Xen● Xen uses bridge-utils in Dom0 to provide virtual network bridging

Routing Mode Configuration ● Modify xend configuration:(network-script network-route)(vif-script vif-route) ● Xen uses iptables in Dom0 to perform software routing

NAT Mode Configuration ● Modify xend configuration:(network-script network-nat)(vif-script vif-nat) ● Xen uses iptables in Dom0 to perform NAT

/etc/init.d/xend startecho $?0 - everythings fine1 - everythings not so fine4 - you did not run as root xend

xm list (No Guests Running)

Session 2Guest Creation and Management

Guest Configuration

Local StorageRaw File: ● Use a filesystem within a single file ● Takes advantage of loopback devicesPartition: ● Use a partition on a logical partition ● Can be physical partition or on an LVM volumePartitioned File: ● Less common ● Treats a raw file as a disk (instead of single partition)

Local Storage: Raw File for PV1. Allocate storage: dd if=/dev/zero of=/path/to/image.img bs=1024k count=10242. Format: mkfs.ext3 -F /path/to/image.img3. Mount the storage: mkdir /mnt/tmp; mount -o loop /path/to/new/image.img /mnt/tmp4. Install the operating system (needs PV drivers): debootstrap hardy /mnt/tmp or cp -a /* /mnt/tmp

Local Storage: Raw File for PV5. Modify various files in guest filesystem and unmount: e.g. /etc/fstab , /etc/hostname , /etc/ifconfig6. Create the guest configuration file for Xen to use

Local Storage: Raw File for HVM1. Allocate storage: dd if=/dev/zero of=/path/to/image.img bs=1024k count=10242. Create the guest configuration file3. Install the operating system

Guest Storage Configuration OptionsArray of disk specifications:real dev in dom0, virtual dev in domU, Access (r, w)SCSI (sd) and IDE (hd) examples:disk = [ phy:sda, sda, w, phy:/dev/cdrom, cdrom:hdc, r ]disk = [ tap:aio:hdb1, hdb1, w, phy:/dev/LV/disk1, sda1, w ]Xen virtual device example:disk = [ tap:aio:hdb1, xvdb1, w, phy:/dev/LV/disk1,xvda1, w ]

General Guest Configuration Options(For both PV and HVM guests)name ● The name of the guest ● (defaults to configuration filename)vcpus ● The number of virtual CPUs ● (defaults to 1)memory ● The amount of memory (in MB) ● (defaults to 128)

Guest Network Configuration ● Array of virtual interface network parameters specify MAC Address, IP Address, for each interface ● Examples:vif = [ ] # Default bridge, random MAC addressvif = [ mac=00:16:3e:36:a1:e9,ip=192.168.1.25, bridge=xenbr0 ]

Guest Network ConfigurationBridge mode networking (default in xend config):Set vif statement in the DomUs configuration fileRouting mode networking (if chosen in xend config):Set DomUs gateway (in guest OSs network configuration) toDom0s external IP (e.g. 192.0.32.10)NAT mode networking (if chosen in xend config):Set DomUs gateway (in guest OSs network configuration) toDom0s internal IP (e.g. 10.0.0.1)

Guest Creation

HVM-specific Configuration Optionskernel The location of the HVM loaderbuilder Domain build function ("hvm" for an unmodified kernel)device_model Location of the device emulation tool (e.g. "qemu_dm")boot The boot order (CD-ROM, hard drive)vnc Enable VNC utility for the guest to display

Sample HVM Guest Configurationvcpus = 1memory = 512kernel = "/usr/lib64/xen/boot/hvmloader"builder = "hvm"device_model = "/usr/lib64/xen/bin/qemu-dm"boot = "cd"disk = [ tap:aio:/xen/images/hvm.disk, ioemu:hda,w,phy:/dev/cdrom, ioemu:hdc:cdrom,r ]vif = [ type=ioemu, bridge=eth0 ]vnc = 1 # (or sdl = 1)

Installing HVM Guest OSes (CD/.iso)1. Allocate disk image for the VM2. Create HVM config. with CD/.iso as first boot device3. Boot the guest: xm create /path/to/guest.cfg4. Follow normal installation process of guest OS5. Change boot order in guest configuration file, reboot

PV-specific Configuration Optionskernel Location of the Xen-modified kernel in Dom0s filespaceramdisk Location of the initial RAM disk image in Dom0s filespace or:bootloader The location of the bootloader (e.g. pygrub)

PV-specific Configuration Optionsroot The partition to use as root inside the guestextra The parameters appended to the kernel command line (as would be normally set at the end of a kernel line)vfb Virtual framebuffer for PV guest to use in addition to console

Sample PV Guest Configurationvcpus = 1memory = 64kernel = "/boot/vmlinuz-2.6.18.8-xen"extra = xencons=ttyramdisk = "/boot/initrd.img-2.6.18.8-xen"disk = [ phy:hda1,xvda1,w ]root = "/dev/xvda1"vif = [ ]vfb = [type=vnc,vncunused=1]

Installing PV Guest OSes1. Allocate disk image for the guest VM2. Mount and populate disk image with distro tools: ○ Stacklet Bundler ○ virt-install ○ virt-manager (discussed further later) ○ vmbuilder ○ debootstrap ○ The tool that comes with your favorite distro3. Unmount image and create PV guest configuration4. Boot the guest: xm create /path/to/guest.cfg

Pre-built Guest ImagesSources: ● http://stacklet.com ● http://rpath.com ● http://jumpbox.comAdvantages: ● Simple to download and extract the images ● Available with different distribution OSes and pre-installed applications

P2V : Physical Machine to a VMConversion of a physical machine into a virtual machineScenarios: ● Virtualizing existing infrastructure ● Supporting legacy applications ● System administration benefits of virtualizationAvailable Tools: ● Use existing backup tools to create a file backup ● P2V LiveCD ● XenServer conversion tool ● Various third-party tools

Guest Access Methods ● The simplest way: consolexm console domU_name ● A better way: SSH directly to DomUssh user@xxx.xxx.xxx.xxx ● Simple graphics: SSH with X11 forwarding to DomUssh -X user@xxx.xxx.xxx.xxx ● Better graphics: SDL or VNC ○ Install vncviewer packageEnable the vnc or sdl option in guest config file

xm create

xm console

xm list

xentop

Session 3Xen in the Datacenter

Network Storage

Network Storage OptionsATA over Ethernet (AoE): ● Export block devices over the network ● Lightweight Ethernet layer protocol ● No built-in securityInternet Small Computer System Interface (iSCSI): ● Exports block devices over the network ● Network layer protocol ● Scales with network bandwidth ● Client and user-level securityNetwork File System (NFS): ● Exports file system over the network ● Network layer protocol ● Known performance issues as root file system

Network Storage OptionsNetwork Block Device (NBD): ● Exports block devices over the network ● Network layer protocol ● Scales with network bandwidth ● Not recommended as root file systemDistributed Replicated Block Device (DRBD): ● Exports and shares block devices over the network ● Integration with Heartbeat ● No additional storage server necessary

Using AoE1. Install required packages: ○ Install vblade on the storage server ○ Install aoe-tools and the aoe module in the Domain02. Export a guest image from the storage server: vbladed 1 1 eth0 /dev/ (for partitions) ... vbladed 1 1 eth0 /path/to/image.img (for files)3. Point the guest configuration to the image: disk = [phy:etherd/e1.1,xvda1,w]Notes: ● Remember that AoE provides no security ● Never use the same shelf/slot for two images

Using DRBD1. Install required packages: ○ Ubuntu/Debian: drbd8-utils and drbd8-moduleRed Hat/CentOS: drbd and drbd-km2. Configure DRBD: ○ Mostly beyond the scope of this presentation ○ Disable sendpage in /etc/modprobe.d/drbd.conf :options drbd disable_sendpage=13. Point the guest configuration to the image:disk = [ drbd:resource,xvda,w ]Documentation:http://www.drbd.org/users-guide/ch-xen.html

Management Tools and Integration

Guest Management ToolsSimplify: ● Creation of guest images ● Manipulation of guest domains ● Generation of guest configuration files ● Monitoring resource usage by guestsPopular tools: ● Convirt ○ Open-source ○ Third-party product and support ● Zentific ○ Open-source ○ Web-based tool ● Virtual Machine Manager ○ Open-source ○ Desktop tool

Convirt● Designed for full datacenter management● Allows for managing the complete lifecycle of Xen (and KVM) guests and hosts● Open-source with commercial support

Convirt: Manage VMs

Convirt: Manage Domain0s

Convirt: Provision VMs Based on Templates

Convirt: Manage VM Configuration

Zentific ● Open source web-based management tool ● Allows for managing and provisioning Xen guests

Zentific: Main Screen

Zentific: VM Status Panel

Zentific: VM Configuration

Zentific: Web-based Guest Console

Virtual Machine Manager● Graphical user interface for managing virtual machines● Allows for Xen guest performance monitoring, resource allocation, and domain creation.● Open source with Red Hat support

Virt-manager: Virtual Machine Manager

Virt-manager: Creating a PV CentOS Guest by URL

Virt-manager: Configuring a PV Guest

Virt-manager: Installing PV guest

Virt-manager: Accessing a PV CentOS Guest

Virt-manager: Configuring an HVM Guest

Installing a Windows HVM Guest from CD-ROM

Virt-manager: Windows HVM Guest Running in Xen

Xen Integration and Compatibilitylibvirt:Provides a uniform interface with different virtualization technologiesMainline Virtualization API (pv_ops):Provides a common paravirtualization interface in mainstream Linux kernel forincreased performance and capabilitiesOpen Virtual Machine Format (OVF):Defines a set of metadata tags that can be used to deploy virtual environmentacross multiple virtualization platformsXen API (XAPI)

Advanced Networking

Multiple Dom0 Network InterfacesMotivation:Segregate DomUs over different networksProcedure: 1. Run network bridge script for each physical interface: /etc/xen/scripts/network-bridge start vifnum=0 netdev=eth1 bridge=xenbr1 2. Configure the DomUs vif option for each bridge: vif = [bridge=xenbr1, ...]

Multiple DomU Network InterfacesMotivation:Allow a DomU to connect to different virtual bridgesProcedure:Modify DomU configuration file:vif = [bridge=xenbr0, bridge=xenbr1, ...]

DomU Network IsolationMotivation:Isolate DomUs from external network, but allow them tocommunicate with one anotherProcedure: 1. Create a dummy bridge in Dom0 in network configuration or with brctl 2. Configure DomUs to connect to that dummy bridge:vif = [bridge = dummy0]

DomU Network Rate LimitingMotivation:Rate limiting for DomU network usage for better performanceisolationProcedure: Configure DomUs vif option with rate parameter : vif = [..., rate=50Kb/s]

Performance and Scalability

Measuring PerformanceCPU:xm top / xentopBuffer:xentraceDisk I/O:xenmonHardware Events:xenoprof

Memory and Scalability● Using memory overcommitment, more memory can be allocated than is on the system● Memory allocated to, but unused by, a VM is made available for use by other VMs● Reduces wasted resources, allowing greater scalability● Risk poor performance due to swapping

Session 4Xen in the Cloud

Guest Relocation

Guest Relocation● Cold Relocation● Warm Migration● Live Migration

Cold RelocationMotivation:Moving guest between hosts without shared storage or withdifferent architectures or hypervisor versionsProcess: 1. Shut down a guest on the source host 2. Move the guest from one Domain0s file system to anothers by manually copying the guests disk image and configuration files 3. Start the guest on the destination host

Cold RelocationBenefits: ● Hardware maintenance with less downtime ● Shared storage not required ● Domain0s can be different ● Multiple copies and duplicationsLimitation: ● More manual process ● Service should be down during copy

Warm MigrationMotivation:Move a guest between hosts when uptime is not criticalCommand:xm migrateResult: 1. Pauses a guests execution 2. Transfers guests state across network to a new host 3. Resumes guests execution on destination host

Warm MigrationBenefits: ● Guest and processes remains running ● Less data transfer than live migrationLimitations: ● For a short time, the guest is not externally accessible ● Requires shared storage ● Network connections to and from guest are interrupted and will probably timeout

Live MigrationMotivation:Load balancing, hardware maintenance, andpower managementCommand:xm migrate --liveResult: 1. Begins transferring guests state to new host 2. Repeatedly copies dirtied guest memory (due to continued execution) until complete 3. Re-routes network connections, and guest continues executing with execution and network uninterrupted

Live MigrationBenefits: ● No downtime ● Network connections to and from guest often remain active and uninterrupted ● Guest and its services remain availableLimitations: ● Requires shared storage ● Hosts must be on the same layer 2 network ● Sufficient spare resources needed on target machine ● Hosts must be similar

Xen Cloud Platform (XCP)

Xen Cloud Platform (XCP)● Xen Cloud Platform (XCP) is turnkey virtualization solution that provides out-of-the-box virtualization/cloud computing● XCP includes: ○ Open-source Xen hypervisor ○ Enterprise-level XenAPI (XAPI) management tool stack ○ Support for Open vSwitch (open-source, standards- compliant virtual switch)● XCP was originally derived from Citrix XenServer (a free enterprise product), is open-source, and is free● XCP promises to contain cutting-edge features that will drive future developments of Citrix XenServer

XCP Features● Fully-signed Windows PV drivers● Single Root I/O Virtualization (SR-IOV) support● Heterogeneous machine resource pool support● Installation by templates for many different guest OSes

XCP XenAPI Management Tool Stack● VM lifecycle: live snapshots, checkpoint, migration● Resource pools: live relocation, auto configuration, disaster recovery● Flexible storage, networking, and power management● Event tracking: progress, notification● Upgrade and patching capabilities● Real-time performance monitoring and alerting

XCPs xsconsole (SSH or Local)

XCPs XAPI Viewer

XCP Management Tools

OpenXenManager ● Open-source clone of Citrix XenCenter ● Manages both Citrix XenServer and Xen Cloud Platform ● Freely available:http://www.openxenmanager.com

OpenXenManager: Multi-host, Multi-guest Status

OpenXenManager: Host and Guest Consoles

XenWebManager ● Web-based utility that shares OpenXenManagers codebase ● Open-source and freely available:http://sourceforge.net/projects/xenwebmanager/

XenWebManager: Guest Creation with Templates

Xen VNC Proxy (XVP) ● Web-based, open-source management for both Citrix XenServer and Xen Cloud Platform ● VNC guest console via web browser ● Freely available as software or a virtual appliance:http://www.xvpsource.org

XVP: Host Pool and Guest Status

XVP: Manage Guests

XVP: Two-server Pool Running Virtual Appliances

XVP: Select Server to Boot VM

XVP: Web-based Guest Consoles

Xen Cloud Control System (XCCS) ● "XCCS is a lightweight front end package for the excellent Xen Cloud Platform cloud computing system. XCCS is totally web based so any computer or smart phone with a web browser can be used with it!" ● Open-source and freely available as software/appliance:http://www.xencloudcontrol.com

Xen Cloud Control System (XCCS)

Closing Remarks

Cloud Computing BoFTuesday, 8:00pm:Open Source and Open Standards-based Cloud Computing(Room: Willow Glen)Todd Deshane and Patrick F. Wilbur, Clarkson UniversityBen Pfaff, Nicira NetworksJason Faulkner, RackspaceIn this session, we will describe some of the open source components availableto support hybrid (public/private) cloud computing. We have some interest andexpertise with various open source components, such as the hypervisor (Xen),the infrastructure platform (the Xen Cloud Platform (XCP)), the virtualnetworking switch layer (Open vSwitch), and the cloud computing software(OpenStack). We invite others that are interested in learning about, describingexperiences with, and discussing the role open source and open standards-based solutions play in the cloud.

Cloud Computing Sessions● Wednesday, 4:00pm: Experiences with Eucalyptus: Deploying an Open Source Cloud● Thursday, 2:00pm: Flying Instruments-Only: Navigating Legal and Security Issues from the Cloud● Thursday, 4:00pm: RC2 -- A Living Lab for Cloud Computing● Thursday, 4:00pm: Panel: Legal and Privacy Issues in Cloud Computing

Useful Resources and ReferencesCommunity: ● Xen Mailing List: http://www.xen.org/community/ ● Xen Wiki: http://wiki.xensource.com/xenwiki/ ● Xen Blog: http://blog.xen.org ● http://wiki.xensource.com/xenwiki/XenCommonProblemsBooks: ● The Definitive Guide to the Xen Hypervisor ● Running Xen: A Hands-On Guide to the Art of VirtualizationDiscussion: ● http://www.xen.org/community/xenpapers.html ● Abstracts, slides, and videos from Xen Summits

2010 xen-lisa

by Todd Deshane

Accessibility

Categories

Upload Details

Usage Rights

Statistics

2010 xen-lisa Presentation Transcript