Your SlideShare is downloading. ×
  • Like
The Ongoing Evolution of Xen
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

The Ongoing Evolution of Xen



  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. The Ongoing Evolution of Xen Ian Pratt Dan Magenheimer XenSource HP Hollis Blanchard Jimi Xenidis IBM IBM Jun Nakajima Anthony Liguori Intel IBM Abstract single physical system with close-to-native per- formance. Xen also enables advanced features Xen 3 was released in December 2005, bring- such as dynamic virtual memory- and CPU- ing new features such as support for SMP guest hotplug, and the live relocation of virtual ma- operating systems, PAE and x86_64, initial chines between physical hosts. The most recent support for IA64, and support for CPU hard- major release of Xen, Xen 3.0.0, took place on ware virtualization extensions (VT/SVM). In 5 December 2005. this paper we provide a status update on Xen, reflecting on the evolution of Xen so far, and In this paper we discuss some of the highlights look towards the future. We will show how of the work involved in the ongoing evolution Xen’s VT/SVM support has been unified and of Xen 3. In particular we cover: describe plans to optimize our support for un- modified operating systems. We discuss how a • HVM: the unified abstraction layer which single ‘xenified’ kernel can run on bare metal as allows Xen to seamlessly support both well as over Xen. We report on improvements Intel and AMD processor virtualization made to the Itanium support and on the status technologies; of the ongoing PowerPC port. Finally we con- clude with a discussion of the Xen roadmap. • the work to allow a single Linux kernel bi- nary image to run both on Xen and on the bare metal with minimal performance and 1 Introduction complexity costs; Xen is an open-source para-virtualizing virtual • the progress made in the IA64 port of Xen, machine monitor or hypervisor. Xen can se- which has been much improved over the curely execute multiple virtual machines on a last year; and
  • 2. 256 • The Ongoing Evolution of Xen • the ongoing port of Xen to the Pow- (user registers, control registers, msrs, etc.); erPC architecture both with and without and to interrogate guest operating modes (e.g. firmware enhancements. to determine if the guest is running in real mode or protected mode). By using this indirection mechansism, most of the Xen code is isolated Finally we look towards the future development from the details of which underlying hardware of key technologies for Xen. virtualization implementation is in use. Underneath this interface, vendor-specific code 2 Hardware Virtual Machines invokes the appropriate HVM functions to deal with intercepts (e.g. when I/O or MMIO op- erations occur), and can share a considerable Although Xen has excellent performance, the amount of common code (e.g. the implementa- paravirtualization techniques it applies require tion of shadow page tables, and interfacing with the modification of the guest operating system I/O emulation). kernel code. While this is of course possible for open source systems such as Linux, it is In the following we look first at a detailed case an issue when attempting to host unmodifiable study—the implementation of HVM guests on proprietary operating systems such as MS Win- 64-bit x86 platforms—and then look toward fu- dows. ture work. Fortunately, recent processors from Intel and AMD sport extensions to enable the safe and 2.1 x86-64 efficient virtualization of unmodified operating systems. In Xen 3.0.0, initial support for In- tel’s VT-x extensions was introduced; later in One of the notable things in x86-64 Xen 3.0 Xen 3.0.2, further support was added to enable is that we now support three types of HVM AMD’s SVM extensions. guests, including: Although the precise details of VT-x and SVM differ, in many aspects they are quite simi- • x86-32 (2-level page tables), lar. Noticing this, we decided that we could best support both technologies by introducing • x86-32 PAE (3-level page tables), and a layer of abstraction: the hardware virtual ma- chine (HVM) interfaces. The design of the • x86-64 (4-level page tables). HVM architecture involved people from Intel, AMD, XenSource, and IBM, but the primary author of the code was IBM’s Leendert van In this section, we discuss the challenges, and Doorn. we present the approaches we took for the x86- 64 Xen. At the core of HVM is an indirection table of function pointers ( struct hvm_function_ The HVM architecture allows 64-bit VMMs table). This provides mechanisms to create (Virtual Machine Monitor) to run 32-bit guests and destroy the additional resources required securely by setting up the HVM control struc- for a HVM domain (e.g. a vmcs on VT-x or a ture as such. Given such hardware support, we vmcb on SVM); to load and store guest state needed to work on the two major areas:
  • 3. 2006 Linux Symposium, Volume Two • 257 1. shadow page tables: x86-32 Xen sup- If one can efficiently tell which guest page ta- ported only guests with 2-level page ta- ble entries have been modified since the last bles, and we needed to significantly extend TLB flush operations, we can reuse the previ- the code to support various paging models ous shadow page tables by updating the only in the single code base. the page table entries that have been modified. 2. SMP support: SMP is the default configu- The key technique that we used in Xen is sum- ration on many x86-64 and PAE systems. marized as follows: Supporting SMP HVM guests turns out to be far from trivial. 1. When allocating a shadow page upon #PF from the guest, write protect the corre- 2.1.1 Overview of Shadow Page Tables in sponding guest page table page. By write- Xen 3.0 protecting the guest page tables, we can detect attempts to modify page tables. A shadow page table is the active or native page 2. Upon #PF against a guest page table page, table (i.e., with entries containing machine we save a ‘snapshot’ of the page; give physical page frame numbers) for a guest, and write permission to the page; and add the it is constructed by Xen to reflect the guest’s page is added to an ‘out of sync list’ along operations on the guest page tables while en- with information relating to the access at- suring that the guest address space is isolated tempt (e.g. which address, etc.). securely. 3. Next we give write-permission to the page, A guest page table is managed and updated by thus allowing the guest to directly update the guest as it were in the native system, and it the page table page. is inactive or virtual in terms of address trans- lations (i.e., with entries containing guest phys- 4. When the guest executes an operation that ical page frame numbers). This is done by results in the flush TLB operation, reflect intercepting mov from/to cr3 instructions by all the entries on the “out of sync list” to hardware virtualization. In other words, the the shadow page table. By comparing the value read from or written to cr3is virtual in snapshot and the current page in the guest HVM guests. This implies that the paging lev- page table, we can update the shadow page els could even be different in the shadow and table efficiently by checking if the page guest page tables. frame numbers in the guest page tables (and thus the corresponding shadow en- From the performance point of view, shadow tries) are valid. page table handling is critical because page faults are very frequent in memory intensive workloads. A rudimentary (and very slow) im- plementation is to construct shadow page ta- 2.1.2 2-Level Guest Page Table and 3-Level bles from scratch every time the guest updates Shadow Page Table cr3 or flushes the TLB(s). This is not efficient because the shadow page tables are lost when The issue with running a guest with 2-level the guest kernel schedules the next processes page tables is that such page tables can specify to run. Frequent context switches would cause only page frames below 4GB. If we simply use significant performance regression. a 2-level page table for the shadow page table,
  • 4. 258 • The Ongoing Evolution of Xen the page frames that we can use are restricted we compile the same basic code three times— on any machine with more than 4GB memory. once each for x86, x86 PAE, and x86-64—but with different implementations of key macros The technique we use is to run such guests in each time. The appropriate shadow_ops is PAE mode that utilizes 3-level (shadow) page set at runtime according to the virtual CPU tables, but retaining the the illusion that they state. are handling 2-level page tables. This compli- cates the shadow page table handling code in a number of ways: 2.1.4 SMP Support • The size of a PTE (Page Table Entry) is SMP support requires various additional func- different: 4-bytes in the guest (2-level) tionality: page tables but 8-bytes in the shadow (3- level) page tables. • Local APIC. To handle IPIs (interproces- sor interrupts), SMP guests require the lo- • As a consequence, whenever the guest al- cal APIC. The local APIC virtualization locates an L1 (lowest level) page table has been incorporated in the Xen hyper- page, the shadow page table code needs to visor to optimize performance. allocate two L1 pages for it. • I/O APIC. SMP guests also typically re- • Furthermore, the shadow code also needs quire the use of one or more I/O APIC(s). to manage an additional page table table I/O APIC virtualization has been incorpo- (L3) which has no direct correspondence rated in the Xen hypervisor for the same in the guest page table. reason above. • ACPI. The ACPI MADT table is dynami- 2.1.3 PSE (Page Size Extensions) Support cally set up to specify the number of vir- tual CPUs. Once ACPI is present, guests The PSE flag enables large page sizes: either 4- expect that the standard or full ACPI fea- MByte pages or 2-MByte pages when the PAE tures be available. During development flag is set. For 32-bit guests, we simply disable this caused a succession of problems, in- PSE by cpuid virtualization. For x86-64 or cluding support for the ACPI timer, event x86 PAE guests PSE is often a prerequisite: the handling, etc. system may not even boot if the CPU does not • SMP-safe shadow page tables. At the time have the PSE feature. To address this, we em- of writing the shadow page table code uses ulate the behavior of PSE in the shadow page a single ‘big lock’ per-domain so as to tables. The current implementation of 2MB simplify the implementation. To improve page support is to fragment it into a set of 4KB the scalability we need fine-grained locks. pages in the shadow page table, since there is no generic 2MB page allocator in Xen. Although HVM SMP guests are stable, we are A set of primitives against the guest and still working on performance optimizations and shadow page tables are defined as shadow scalability. For example, I/O device models operations—shadow_ops. To avoid code du- need to be multi-threaded to handle simultane- plication, we currently use a technique whereby ous requests from guests.
  • 5. 2006 Linux Symposium, Volume Two • 259 2.2 Ongoing Work Hence a simple IDE controller, for example, can be emulated entirely within the super- emulator but ultimately end up issuing block Ongoing work is looking at optimizing the per- read/write requests across a standard Xen de- formance of emulated I/O devices. Unlike par- vice channel. As well as providing excellent avirtualized guest operating systems, HVM do- performance, this also means that HVM do- mains are not aware that they are running on top mains appear the same as paravirtual domains of Xen. Hence they attempt to detect H/W de- from the point of view of the tools, thus allow- vices and load the appropriate standard device ing us to unify and simplify the code. drivers. Xen must then emulate the behaviour of the relevant devices, which has a consequent performance impact. 3 MiniXen: A Single Xen Kernel The general model introduced before Xen 3.0.0 shipped was to use a device model assistant pro- The purpose of the miniXen project is to take cess running in Domain 0 which could emulate a guest Linux kernel which has been ported a set of useful devices. I/O accesses from the to the Xen interfaces and combine this with unmodified guest would trap, and cause a small a thin version of Xen that interacts directly I/O packet message to be sent to the emulator. with the native hardware allowing a single do- The latency involved here can be quite large main to run with near native performance. This and so reduces the overall performance of the miniXen performs the bare minimum of opera- guest operating system. tions needed to support a guest OS and where possible passes events, such as interrupts and A first step to improving performance was to exceptions, directly to the guest OS omitting move the emulation of certain simple platform the normal Xen protection mechanisms. devices into Xen itself. For example, the APIC and PIT are not particularly complicated and, An important first step in this was to allow the crucially, do not require any interaction with guest kernel to run in x86 privilege ring zero the outside world. These devices tend to be ac- rather than ring one as a normal Xen guest does. cessed frequently within operating systems and This is easily achieved by using the features hence emulating these within Xen itself reduces flags which Xen exports to all guests: a guest latency and improves performance. checks for the relevant flag and runs in either ring zero or one as appropriate. A more substantial set of changes will address both performance and isolation aspects: the An unfortunate side-effect of running the guest super-emulator. This approach involves asso- kernel in ring zero is that a privilege level ciating a paravirtualized stub-domain with ev- change no longer occurs when making a hyper- ery HVM domain which runs with the same se- call or when an interrupt or exception interrupts curity credentials and within the same resource the guest kernel. This means that the hardware container. Then when a HVM domain attempts will no longer automatically switch the guest to perform I/O, the trap is instead reflected to kernel stack for Xen’s own stack. Xen relies the stub-domain which performs the relevant on its own stack in order to store certain state emulation, translating the device requests into information. Therefore miniXen must check the equivalent operations on a paravirtual de- at each entry point whether the stack pointer vice interface. points to memory owned by the guest kernel
  • 6. 260 • The Ongoing Evolution of Xen or Xen and if necessary to fix up the stack by guest kernel’s stack allowing the kernel to im- locating the Xen stack via the TSS and moving mediately switch to its own stack at the sysenter the current stack frame to the Xen stack before entry point. continuing. Normally Xen prevents a guest OS from cre- These checks and the movement of the stack ating a writable mapping of pages which are frame necessarily incur a performance penalty part of a page table or descriptor table (GDT which it is desirable to avoid. As all hypercalls or LDT) in order to trap any writes and audit were to be either reimplemented or stripped them for safety. For this reason guest kernels down in order to achieve the goal of perform- only create read-only mappings to such pages ing the bare-minimum of work in miniXen it and a write therefore involves creating an addi- was possible to also ensure that the hypercalls tional writable mapping of a page in the hyper- did not require any state from the Xen stack. visor’s address space. However miniXen does Once this was achieved it was possible to turn not need to audit the page or descriptor tables each hypercall into a simple call instruction by and by making use of feature flags exported rewriting the miniXen hypercall transfer page, from Xen to the guest kernel can cause the ker- thus avoiding an int 0x82 hypercall and the nel to create writable mappings of these pages. expensive stack fixup logic. A direct call into This allows miniXen to write directly to these the hypervisor is possible because unlike Xen, pages and therefore allows page table updates miniXen does not need to truncate the guest with native performance. kernel’s segment descriptors in order to isolate the kernel from the hypervisor. Work is currently on-going to audit miniXen’s 4 IA64 interrupt and exception handling code to re- move any need for state to be stored on Xen’s stack and so allow those routines to run on the In the year since the last symposium, Xen/ia64 guest kernel’s stack. Once this is complete then has made excellent progress and is slowly miniXen should have no need for a stack of its catching up to Xen on x86-based plat- own and can simply piggy-back on the guest forms. The Xen/ia64 community has grown kernels stack except for under very specialized substantially, with major contributions from circumstances such as during boot. organizations around the world; and the xen-ia64-devel mailing list has grown to Running the guest kernel in ring 0 allows us to over 160 subscribers. once again take advantage of the sysenter/ sysexit instructions to optimize system calls Since Xen’s first release on x86, paravirtual- from user space to the guest kernel compared ization has delivered near-native performance with the normal int 0x80 mechanism. Nor- and it is important to demonstrate that these mally the sysenter instruction is not avail- techniques apply to other architectures. Much able to guests running under Xen because the effort was put into a paravirtualized version target code segment must be in ring 0. How- of Linux/ia64 and using innovative techniques ever with the guest kernel running in ring 0 it is (e.g. “hyper-registers” and “hyper-privops”), simple to arrange for the sysenter instruc- performance was driven to within 2% of na- tion to jump directly into the guest kernel. The tive (as measured by a software develop- sysenter stack is configured such that it points ment benchmark). With guidance from the to the location of the TSS entry containing the Linux/ia64 maintainers, an elegant patch was
  • 7. 2006 Linux Symposium, Volume Two • 261 developed that adds a clean abstraction layer tunings, Xen/ia64 can rapidly incorporate these to certain privileged instruction macros and re- changes. places only a handful of assembly routines in the Linux/ia64 source. Interestingly, after this The value of this direct leverage was demon- patch is applied the resultant Linux/ia64 binary strated last fall when SMP host support was can be run both under Xen and on bare metal— added to Xen/ia64. Addition of SMP support a concept dubbed transparent paravirtualiza- to an operating system is often a long painful tion. process, requiring extensive debugging to, for example, isolate and repair overlooked locks. Block driver support using Xen’s fron- For Xen/ia64, SMP support was added by one tend/backend driver model was implemented developer in a few weeks because so much last summer and integrated into the Xen tree. working SMP code was already present or eas- Soon thereafter Xen/ia64 was supporting ily leveraged from Linux/ia64. Indeed, SMP multiple Linux domains and work was recently guest support was also recently added in-tree completed to cleanly shutdown domains and and testing for both SMP host and SMP guest reclaim resources. All architectural differences support is showing surprising stability. were carefully designed and implemented to fully leverage the Xen user-space tools so that While Xen/ia64 has made great progress, much administrators use identical commands on more work lies ahead. Driver domain sup- both Xen/x86 and Xen/ia64 for deploying and port, recently added back into Xen/x86, is es- controlling guest domains. pecially important on the large machines com- Preliminary support for Intel Virtualization monly found in most vendors’ Itanium prod- Technology for Itanium processors (VT-i) was uct lines. Migration support may prove simi- completed last fall and has become quite ro- larly important. Some functionality has been bust. It is now possible to run unmodified (fully serialized behind a community decision to fun- virtualized) domains in parallel with paravir- damentally redesign the Xen/ia64 physical-to- tualized domains. Here also, existing device machine mapping mechanisms, which also was models and administrative control panel inter- a prerequisite for maximal leverage and enable- faces were leveraged from the VT-x implemen- ment of the Xen networking split driver code. tation for Xen/x86 to minimize maintenance With the completion of this redesign in late and maximize compatibility. Spring, networking performs well and it is be- lieved that driver domains, as well as domain In many ways, Xen is an operating system and, save/restore and migration will all be easier to as an open-source operating system, there is no implement and will come up quickly. need to re-create the wheel. Xen/x86 leveraged a fair amount of code from an earlier version of Linux and periodically integrates code from newer versions. Xen/ia64 goes one step further 5 PowerPC and utilizes over 100 code modules and header files from Linux/ia64 directly. About half of these files are completely unchanged and the Xen is being ported to the PowerPC architec- remainder require only minor changes, which, ture, specifically the PowerPC 970. The 970 for maintenance purposes, are clearly marked processor contains processor-level extensions with ifdefs. Since Linux/ia64 is relatively to the PowerPC architecture designed to sup- immature and subject to frequent bug fixes and port paravirtualized operating systems. These
  • 8. 262 • The Ongoing Evolution of Xen hardware modifications, made for the hypervi- Linux kernels in unprivileged domains. At the sor running on IBM’s pSeries servers, allow for time of publication, the DomUs have no sup- minimal kernel changes and very little perfor- port for virtual IO drivers, so interaction with mance degradation for the guest operating sys- the domain isn’t currently possible. However, tems. The challenge in a Xen port to PowerPC one can see from their boot output that they is fitting the Xen model, developed on desktop- make it to user-space. class x86 systems, into this PowerPC architec- ture. The PowerPC development tree is currently be- ing merged with the main Xen development One of the challenges for Xen on PowerPC tree. PowerPC Xen is not yet integrated with isn’t related to Xen itself, but rather the avail- Xen’s management tools, and the unprivileged ability of hardware platforms capable of run- domains, lacking device drivers, do not yet per- ning Xen. Although IBM’s PowerPC-based form meaningful work. Once they do, it will servers have the processor hypervisor exten- also become important to integrate with Xen’s sions we are exploiting in Xen, they also con- testing infrastructure and also to package builds tain firmware that doesn’t allow user-supplied in a more user-friendly manner. code to exploit those extensions. Current Pow- erPC Xen development efforts have been on the Maple 970 evaluation board and the IBM Full 5.2 Design Decisions System Simulator. The existing firmware on Apple’s Power Macintosh G5 systems, which are based on the PowerPC 970, disables the hy- The PowerPC architecture differs from the x86 pervisor extensions completely. in a number of significant ways. In the follow- ing we comment on some of the key design de- As a secondary task, work is underway to run cisions we made during the port. Xen on PowerPC 970 with hypervisor mode disabled, specifically Apple G5 systems. Al- though possible, this model requires signifi- Hypervisor in Machine Mode PowerPC, cant modifications to the guest operating sys- like most RISC-like processors, is able to ad- tem (pushing it to user privilege mode) and will dress all memory while translation is off (i.e., also incur substantial performance impact. This MMU disabled). This allows the hypervisor port will be a stepping stone for supporting all to execute completely in machine (or “real”) “legacy” PowerPC architectures such as Apple mode which removes any impact to the MMU G4 and G3 systems, and embedded processors. when transitioning from domain to hypervisor and back. Bypassing the MMU allows us to 5.1 Current Status avoid TLB flushes, a significant factor in per- formance. This decision does have two nega- tive impacts though: it complicates access to On PowerPC, the Xen hypervisor boots from MMIO registers and inhibits the use of domain firmware and then boots a Linux kernel in virtual addresses. Dom0. The Dom0 kernel has direct access to the hardware of the system, and so device The processor must access MMIO registers drivers initialize the hardware as in a normal without using the data cache. This is usually Linux boot. Once Dom0 has booted, a small implemented via attributes in MMU translation set of user-land tools are used to load and start entries, but since we run without translation this
  • 9. 2006 Linux Symposium, Volume Two • 263 method is unavailable to the hypervisor. Fortu- the same memory management interfaces as nately, there is an architected mode that allows the POWER Hypervisor. However, the sup- the processor to perform the cache-inhibited port for interrupts and Virtual IO come from load/store operations while translation is off. the Xen model. The strategy has resulted in The problem of accessing memory through a a small patch (< 200 LOC) for existing Linux domain virtual address requires a more com- code, and an additional Xen “platform” for plex software solution. arch/powerpc, with the rest of the Xen- specific code in the common drivers/ direc- Many Xen hypercalls pass the virtual addresses tory. Since PowerPC Linux supports multiple of data structures into the hypervisor. Xen platforms in the same binary, the same kernel could attempt to translate the virtual addresses can run on Xen, hardware, or other hypervisors, to machine addresses without the aid of the with no recompile needed. MMU, but the MMU translation mechanism of PowerPC is complex enough to make soft- ware MMU translation infeasible. An addi- Interrupt Model On most PowerPC proces- tional complication is that these data structures sors, interrupts are handled by delivering ex- could span a page boundary (and some span ceptions to fixed vectors in real-mode, and many pages), and although those pages are vir- are differentiated into two classes, synchronous tually contiguous, they will likely be discon- and asynchronous. tiguous in the machine address space. Synchronous interrupts are “instruction- After much agony, the solution developed to caused,” which include page faults, system overcome this problem is essentially to create a calls, and instruction traps. Under Xen, the physically addressed scatter/gather list to map processor is run in a mode so that all syn- the data structures being passed to the hypervi- chronous interrupts are delivered directly to the sor. Since user-space is unaware of the phys- domain. The hypervisor extensions provide a ical addresses, the kernel must intercept hy- special form of system call that is delivered to percalls originating from user-space and create the hypervisor, which is used for hypercalls. these scatter/gather structures with physical ad- dresses. The kernel then passes the physical Asynchronous interrupts are caused by timers address of the scatter/gather structure to Xen and devices. Timer interrupts are also delivered (which is able to trivially convert physical ad- directly to the domain. The hypervisor exten- dresses to machine addresses). The end result sions provide us with an additional timer that is is Xen’s copy_from_guest() is passed the delivered to the hypervisor in order to preempt address of this special data structure, and copy the active domain. data from frames scattered throughout the ma- chine address space. Device interrupts, called “external exceptions,” are delivered directly to the hypervisor, which then creates an event for the appropriate do- Hypercalls PowerPC Linux currently runs main. In PowerPC operating systems, the exter- on the hypervisor that currently ships with IBM nal exception handler probes an interrupt con- high end products. Like Xen, the POWER Hy- troller to identify the source of the interrupt. In pervisor virtualizes the processor, memory, in- order to deliver the interrupt to the domain, we terrupts and presents a Virtual IO interface. In supplied an interrupt controller driver for Linux order to capitalize on the existing Linux im- that consults Xen’s event channel mechanism to plementation, Xen on PowerPC has adopted determine the pending interrupt.
  • 10. 264 • The Ongoing Evolution of Xen Memory Management The PowerPC hyper- 6 Xen Roadmap visor extensions define a Real Mode Area (RMA), which isolates memory accessed in real mode (without the MMU). This allows for Xen continues to develop apace. In this final the interrupts that are delivered directly to the section we discuss four interesting ongoing or domain to be delivered in real mode—the same future pieces of work. way they would work without a hypervisor. A side effect of this is that domain “physical” address space must be zero based. However, 6.1 NUMA Optimization since the physical address space is now differ- ent from the machine address space, supporting DMA becomes problematic. Rather than ex- pose the machine address space to the domain NUMA machines, once rarities except on big- and write a DMA allocator in Linux to exploit iron systems, are becoming more and more the it, on 970-based systems we use the I/O Mem- norm with the introduction of multi-core and ory Management Unit (IOMMU) that Linux al- many-core processors. Hence understanding ready exploits. memory locality and node topology has be- come even more important. For Xen, this involves work in at least three Atomic Operations The primary target of areas. Firstly, we need to build a NUMA- Xen are the 32- and 64-bit variants of the aware physical memory allocator for Xen itself. x86 architectures. That architecture contains a This will enable the allocation of memory from plethora of atomic memory operations that are the correct zone or zones and avoid the per- normally not present in RISC processors. In formance overheads associated with non-local particular, PowerPC will only perform atomic memory accesses. operations on 4-byte words (64-bit processors can also perform 8-byte operations), and they Secondly, we need to make Xen’s CPU sched- must be naturally aligned. This presents a uler NUMA-aware: in particular it is important portability issue that must be resolved to sup- to schedule a guest on nodes with local access port non-x86 architectures. to the domain’s memory as far as is possible. This is complicated on Xen since each guest may have a large number of virtual CPUs (vC- PUs) and an opposing tension will be to dis- 5.3 Conclusion perse these to maximize benefit from the un- derlying hardware. Xen has created an exceptional virtualization Finally, the NUMA information really should model for an architecture that many consider be propagated all the way to the guest operat- overly complex and has trailed the industry ing system itself, so that a NUMA-aware guest in virtualization support. Xen’s virtualization OS can make sensible memory allocation and model developed in the absense of a hardware scheduling decisions for itself. All of this be- framework, so the overall challenge of the Pow- comes even more challenging as vCPUs may erPC port has been to adapt the Xen model to migrate between physical CPUs from time to exploit the capabilities of our hardware. time.
  • 11. 2006 Linux Symposium, Volume Two • 265 6.2 Supporting IOMMUs leveraging such hardware to enable direct ac- cess from guests (initially kernel-mode but ul- timately perhaps even from guest user-mode). An IOMMU provides an address translation mechanism for I/O memory requests. Popular As with the above-mentioned work on IOM- on big iron machines, they are becoming more MUs, this will build on the current grant-table and more prevalent on regular x86 boxes— architecture. However additional thought is re- indeed, a primitive IOMMU in the form of the quired to correctly support temporal aspects AGP GART has been found on x86 boxes for such as the controlled scheduling of user re- a number of years. IOMMUs can help avoid quests. Initial work here is focusing on Infini- problems with legacy devices (e.g., 32-bit-only band controllers where we hope to be able to PCI devices) and can enhance security and re- provide extreme low-latency while maintaining liability by preventing buggy device drivers or safety. devices from performing out-of-bounds mem- ory accesses. 6.4 Virtual Framebuffer This latter ability is incredibly promising. One reason for the catastrophic effect of driver fail- Past versions of Xen have focused mostly on ure on system stability is the total lack of isola- server oriented environments. In these environ- tion that pervades device interactions on com- ments, a virtual serial console that is accessible modity systems. By wisely using IOMMU- remotely through a TCP socket or SSH is usu- technology in Xen, we hope we shall be able ally enough for most use cases. As Xen ex- to build fundamentally more robust systems. pands its user base and begins to be used in other types of environments, a more advanced Ideally this will not require too much work guest interface is required. The Xen 3.0.x se- since Xen’s grant-table interfaces were explic- ries will introduce the first of a series of fea- itly designed with IOMMUs in mind. In tures designed to target these new environments essence, each device driver (virtual or other- starting with a paravirtual framebuffer. wise) can register a page of memory as inbound or outbound for I/O—after Xen has checked the A paravirtual framebuffer provides a virtual permissions and ownership, an IOMMU entry graphics adapter that can be used to run graph- can be installed allowing the access. After the ical distribution installers, console mode with I/O has completed, the entry can be removed. virtual terminal switching and scrollback, and windowing environments such as the X Win- dow System. The current implementation al- 6.3 Interfacing with ‘Smart’ Devices lows these applications to run with no modifi- cations and no special userspace drivers. Fu- ture versions will additionally provide special A number of newer hardware devices incor- interfaces to userspace so that custom drivers porate additional logic to allow direct access can be written for additional performance (for from user-mode processes. Most such devices instance, a custom driver). are targeted toward low-latency zero-copy net- working, although storage and graphic devices Traditional virtualization systems such as are moving in the same direction. One inter- QEmu, Bochs, or VMware provide graphics esting piece of future work in Xen will involve support by emulating an actual VGA device.
  • 12. 266 • The Ongoing Evolution of Xen This requires a rather large amount of emula- framebuffer is updated during a given time pe- tion code since VGA devices tend to be rather riod. We are able to mitigate this by using a complex. VGA emulation is difficult to op- timer to periodically invalidate all guest map- timize as it often requires MMIO emulation pings of the framebuffer’s memory. We can for many performance critical operations such then keep track of which pages were mapped in as blitting. Many full virtualization systems during this interval. Based on the dirtied page use hybrid drivers featuring additional hard- information, we can calculate the framebuffer ware features that provide virtualization spe- region that was dirtied. cific optimized graphics modes to avoid MMIO emulation. In practice, this optimization provides updates at a scanline granularity. In the future, we In contrast, a fully paravirtual graphics driver plan on enhancing the guest’s userspace ren- offers all of the performance advantages of a dering applications to provide greater granular- hybrid driver with only a small fraction of the ity in updates. This is particularly important amount of code. Emulation for the Cirrus Logic for bandwidth sensitive clients (such as a VNC chipset provided by QEmu requires over 6,000 client). lines of code (not including the VGA Bios code). The current Xen paravirtual framebuffer We also plan on exploring other graphics re- is implemented in less than 500 lines of code lated features such as 2D acceleration (region and supports arbitrary graphic resolutions. The copy, cursor offloading, etc) and 3D support. QEmu graphics driver is limited to resolutions There are also some interesting security related up to 1024x768. features to explore although that work is just beginning to take shape. Future versions of Xen The paravirtual framebuffer is currently imple- may also provide support for other desktop- mented for Linux guests although the expecta- related virtual hardware such as remote sound. tion is that it will be relatively easy to port to other guest OSes. The driver reserves a por- tion of the guests memory for use as a frame- buffer. The location of this memory is commu- 7 Conclusion nicated to the framebuffer client. The frame- buffer client is an application, usually running in the administrative domain, that is responsible Xen continues to evolve. Although it already for either rendering the guest framebuffer to the provides high performance paravirtualization, host graphics system or over the network using we are working on optimizing full virtualiza- a protocol such as VNC. The client can directly tion to better serve those who cannot modify map the paravirtual framebuffer’s memory us- the source code of their operating system. To ing the Xen memory sharing APIs. simplify system administration, we are work- ing on supporting a single linux kernel binary An initial optimization we have implemented which can run either directly on the hardware uses the guest’s MMU to reduce the amount of or on top of Xen. To allow a broader applica- client updates. Normally, applications within a bility, we are enhancing or developing support guest expect to be able to write directly to the for non x86 architectures. And we are look- framebuffer memory without needing to pro- ing beyond these to develop new features and vide any sort of flushing or update information. hence to ensure that Xen remains the world’s This presents a problem for the client since it best open source hypervisor. has no way of knowing which portions of the
  • 13. Proceedings of the Linux Symposium Volume Two July 19th–22nd, 2006 Ottawa, Ontario Canada
  • 14. Conference Organizers Andrew J. Hutton, Steamballoon, Inc. C. Craig Ross, Linux Symposium Review Committee Jeff Garzik, Red Hat Software Gerrit Huizenga, IBM Dave Jones, Red Hat Software Ben LaHaise, Intel Corporation Matt Mackall, Selenic Consulting Patrick Mochel, Intel Corporation C. Craig Ross, Linux Symposium Andrew Hutton, Steamballoon, Inc. Proceedings Formatting Team John W. Lockhart, Red Hat, Inc. David M. Fellows, Fellows and Carr, Inc. Kyle McMartin Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights to all as a condition of submission.