This document provides an overview of virtualization technologies for high performance computing environments. It discusses what virtualization and hypervisors are, as well as common hypervisors like Xen, KVM, and VirtualBox. The document analyzes these hypervisors based on features, usability, and performance benchmarks. Benchmark results are presented from running SPEC CPU, memory, and I/O benchmarks, as well as HPCC benchmarks, on virtual machines created with each hypervisor compared to native hardware. The analysis finds that all hypervisors provide acceptable virtualization features for HPC and that Xen supports the most resources but also has more complex installation.

1. Analysis of Virtualization
Technologies for High Performance
Computing Environments
https://portal.futuregrid.org 1

2. Introduction
• What is Virtualization?
– A method of partitioning a physical computer into multiple
“virtual” computers, each acting independently as if they were
running directly on hardware.
• What is a Hypervisor?
– A technique used to run multiple operating systems
simultaneously on a single resource.
– Also called a Virtual Machine Monitor (VMM).
• What is a Virtual Machine?
– A software implementation of a machine that executes as if it
was running on a physical resource directly.
2
https://portal.futuregrid.org

3. https://portal.futuregrid.org 3
• Fundamental idea – abstract hardware of a single computer into several
different execution environments
– Similar to layered approach
– But layer creates virtual system (virtual machine, or VM) on which
operation systems or applications can run
• Several components
– Host – underlying hardware system
– Virtual machine manager (VMM) or hypervisor – creates and runs
virtual machines by providing interface that is identical to the host
• (Except in the case of paravirtualization)
– Guest – process provided with virtual copy of the host
• Usually an operating system
• Single physical machine can run multiple operating systems concurrently,
each in its own virtual machine
Cont…

4. Motivation
• Most “Cloud” deployments rely on virtualization.
– Amazon EC2, GoGrid, Azure, Rackspace Cloud …
– Nimbus, Eucalyptus, OpenNebula, OpenStack …
• Number of Virtualization tools or Hypervisors available today.
– Xen, KVM, VMWare, Virtualbox, Hyper-V …
• Need to compare these hypervisors for use within the
scientific computing community.
4
https://portal.futuregrid.org

5. Current Hypervisors
5
https://portal.futuregrid.org

6. System Models
(a) Nonvirtual machine (b) Virtual machine

7. Hypervisors
• Evaluate Xen, KVM, and VirtualBox hypervisors against native
hardware
– Common, well documented
– Open source, open architecture
– Relatively mature & stable
• Cannot benchmark VMWare hypervisors due to proprietary
licensing issues.
7
https://portal.futuregrid.org

8. Hypervisors And Hosts
• A hypervisor is a piece of computer software, firmware or
hardware that creates and runs virtual machines.
• A computer on which a hypervisor is running one or more virtual
machines is defined as a host machine.
• Each virtual machine has a guest operating systems, which is
managed by the hypervisor.
• Multiple instances of a variety of operating systems may share the
virtualized hardware resources.

9. Implementation of VMMs
• Vary greatly, with options including:
– Type 0 hypervisors - Hardware-based solutions that provide support for
virtual machine creation and management via firmware
• IBM LPARs and Oracle LDOMs are examples
– Type 1 hypervisors - Operating-system-like software built to provide
virtualization
• Including VMware ESX, Joyent SmartOS, and Citrix XenServer
– Type 1 hypervisors – Also includes general-purpose operating systems that
provide standard functions as well as VMM functions
• Including Microsoft Windows Server with HyperV and RedHat Linux
with KVM
– Type 2 hypervisors - Applications that run on standard operating systems
but provide VMM features to guest operating systems
• Including VMware Workstation and Fusion, Parallels Desktop, and
Oracle VirtualBox

10. Implementation of VMMs (cont.)
• Other variations include:
– Paravirtualization - Technique in which the guest operating system is modified
to work in cooperation with the VMM to optimize performance
– Programming-environment virtualization - VMMs do not virtualize real
hardware but instead create an optimized virtual system
• Used by Oracle Java and Microsoft. Net
– Emulators – Allow applications written for one hardware environment to run
on a very different hardware environment, such as a different type of CPU
– Application containment - Not virtualization at all but rather provides
virtualization-like features by segregating applications from the operating
system, making them more secure, manageable
• Including Oracle Solaris Zones, BSD Jails, and IBM AIX WPARs
• Much variation due to breadth, depth and importance of virtualization in modern
computing

11. Related Research
• Some work has already been done to evaluate performance…
• Karger, P. & Safford, D. I/O for virtual machine monitors: Security and performance issues.
Security & Privacy, IEEE, IEEE, 2008, 6, 16-23
• Koh, Y.; Knauerhase, R.; Brett, P.; Bowman, M.; Wen, Z. & Pu, C. An analysis of performance
interference effects in virtual environments. Performance Analysis of Systems & Software,
2007. ISPASS 2007. IEEE International Symposium on, 2007, 200209.
• K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. Wasserman, and N.
Wright, “Performance Analysis of High Performance Computing Applications on the
Amazon Web Services Cloud,” in 2nd IEEE International Conference on Cloud Computing
Technology and Science. IEEE, 2010, pp. 159–168.
• P. Barham, B. Dragovic, K. Fraser, S. Hand, T. L. Harris, A. Ho, R. Neugebauer, I. Pratt, and A.
Warfield, “Xen and the art of virtualization,” in Proceedings of the 19th ACM Symposium
on Operating Systems Principles, New York, U. S. A., Oct. 2003, pp. 164–177.
• Adams, K. & Agesen, O. A comparison of software and hardware techniques for x86
virtualization. Proceedings of the 12th international conference on Architectural support
for programming languages and operating systems, 2006, 2-13.
11
https://portal.futuregrid.org

12. Features
Xen KVM VirtualBox VMware
Paravirtualization Yes No No No
Full Virtualization Yes Yes Yes Yes
Host CPU X86, X86_64, IA64 X86, X86_64, IA64,
PPC
X86, X86_64 X86, X86_64
Guest CPU X86, X86_64, IA64 X86, X86_64, IA64,
PPC
X86, X86_64 X86, X86_64
Host OS Linux, Unix Linux Windows, Linux, Unix Proprietary Unix
Guest OS Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix
VT-x / AMD-v Opt Req Opt Opt
Supported Cores 128 16* 32 8
Supported Memory 4TB 4TB 16GB 64GB
3D Acceleration Xen-GL VMGL Open-GL Open-GL, DirectX
Licensing GPL GPL GPL/Proprietary Proprietary
12
https://portal.futuregrid.org

13. Usability
• KVM and VirtualBox trivial to install & deploy.
– Xen requires special kernel, leading to more
complications.
– VMWare ESX runs as a standalone hypervisor.
• All are supported under Libvirt API.
– Used by many IaaS frameworks.
• Xen & Virtualbox have nice CLI, VMWare has an
advanced web based GUI.
13
https://portal.futuregrid.org

14. Performance Analysis
• In order to assess various performance metrics,
benchmarks are needed.
– Provide a fair, apples-to-apples comparison between
each hypervisor.
– Comparisons can be made across other benchmark
submissions on different machines.
– Reproducible and verifiable results.
– Open standards, no special optimizations or tricks
available (hopefully).
14
https://portal.futuregrid.org

15. Testing Environment
• All tests conducted on the
IU India cluster as part of
FutureGrid.
• Identical nodes used for
each hypervisor, as well
as a bare-metal (native)
machine for the control
group.
• Each host OS runs RedHat
Enterprise Linux 5.5.
• India: 256 1U compute nodes
– 2 Intel Xeon 5570 Quad core
CPUs at 2.93Ghz
– 24GB DDR2 Ram
– 500GB 10k HDDs
– InfiniBand DDR 20Gbs
15

16. Testing Environment (cont)
• Guest Operating System:
– RedHat Enterprise Linux 5.5 x86_64
– Kernel 2.6.18-194.8.1.el5
– Minimal server installation (Base, Core)
– Default settings and services
– Full virtualization using Intel VT-x
– Each VM gets 16GB of memory
– Virtualization Add- ons:
• Xen: version 3.1.0 with 2.6.18-194.8.1.el5xen kernel
• KVM: version 83
• VirtualBox: version 3.2.10_66523_rhel5
• Bare-Metal OS = Host OS
16
https://portal.futuregrid.org

17. Benchmark Setup
HPCC
• Industry standard HPC
benchmark suite from
University of Tennessee.
• Sponsored by NSF, DOE,
DARPA.
• Includes Linpack, FFT
benchmarks, and more.
• Targeted for CPU and
Memory analysis.
SPEC OMP
• From the Standard
Performance Evaluation
Corporation.
• Suite of applications
aggrigated together
• Focuses on well-rounded
OpenMP benchmarks.
• Full system performance
analysis for OMP.
17
https://portal.futuregrid.org

18. Benchmarks
• SPEC Benchmarks
– specCPU 2006 – CPU bound benchmark
– specMPI 2007 – MPI: cpu, memory, network
– specOMP 2001 – Open MP: cpu, memory, IPC
– specVIRT 2010 – CPU, memory, disk I/O, network
• HPCC Benchmarks
– HPL (Linpack), DGEMM, STREAM, PTRANS, RandomAccess,
FFT, etc
http://futuregrid.org 18

19. Feature Roundup
• All hypervisors evaluated have acceptable level of features for
x86 virtualization.
• Xen provides best expandability, supporting up to 128 CPUs
and 4TB of RAM.
– Can remove CPU limit for KVM.
– VirtualBox needs to add support for >16GB RAM.
• All have API plugins to allow for simplified IaaS usage.
19
https://portal.futuregrid.org

20. Motivation Example: Forbes.com
• You offer on-line real
time stock market
data
• Why pay for capacity
weekends, overnight?
9 AM - 5 PM,
M-F
ALL OTHER
TIMES
Rate of
Server
Accesses

21. Forbes' Solution
• Host the web site in Amazon's EC2 Elastic Compute Cloud
• Provision new servers every day, and deprovision them every
night
• Pay just $0.10* per server per hour
– * more for higher capacity servers
• Let Amazon worry about the hardware!

22. Cloud computing takes virtualization to the next step
• You don’t have to own the hardware
• You “rent” it as needed from a cloud
• There are public clouds
– e.g. Amazon EC2, and now many others (Microsoft, IBM,
Sun, and others ...)
• A company can create a private one
– With more control over security, etc.

23. Goal 1 – Cost Control
• Cost
– Many systems have variable demands
• Batch processing (e.g. New York Times)
• Web sites with peaks (e.g. Forbes)
• Startups with unknown demand (e.g. the Cash for
Clunkers program)
– Reduce risk
• Don't need to buy hardware until you need it

24. Goal 2 - Business Agility
• More than scalability - elasticity
– Ely Lilly in rapidly changing health care business
• Used to take 3 - 4 months to give a department a
server cluster, then they would hoard it
– Using EC2, about 5 minutes
• And they give it back when they are done
• Scaling back is as important as scaling up

25. Goal 3 - Stick to Our Business
• Most companies don't WANT to do system administration
– Forbes says:
• We are is a publishing company, not a software
company
• But beware:
– Do you really save much on sys admin?
– You don't have the hardware, but you still need to
manage the OS!

26. Some Commercial Cloud Offerings
26

27. Cloud Taxonomy
27

28. Where is all of this?

29. Data Centers
• 10 billion spent on electricity per year for data centers
• 3% of global energy use
• Clouds are the future of the way companies do business
on the Internet

30. Data Centers By Size and Region
Forecast: Data Centers, Worldwide, 2010-2016, 2Q12 Update
Table 1-1 (Top) (Front Page)
Data Centers by Size and Region, 2010-2016
Sum of Sites Year
Region Site Class 2010 2011 2012 2013 2014 2015 2016
Asia/Pacific Single 769,012 769,455 780,252 792,999 819,342 851,068 900,039
Rack/Computer Room 58,702 60,311 63,183 66,916 70,173 72,564 74,249
Midsize DC 4,478 4,656 4,989 5,455 5,892 6,260 6,559
Enterprise DC 984 1,026 1,110 1,239 1,379 1,523 1,666
Large DC 106 110 120 136 156 179 204
Canada Single 66,519 66,012 64,384 62,311 61,575 62,246 63,956
Rack/Computer Room 14,390 14,194 13,828 13,241 12,686 12,068 11,328
Midsize DC 650 643 631 613 599 584 564
Enterprise DC 210 209 208 210 217 228 241
Large DC 22 22 23 25 28 32 37
Eastern Europe Single 147,274 151,468 161,916 176,398 194,389 213,602 238,792
Rack/Computer Room 28,750 28,892 28,829 28,761 28,799 29,077 29,380
Midsize DC 1,102 1,112 1,121 1,134 1,149 1,172 1,197
Enterprise DC 196 198 202 208 216 228 243
Large DC 44 44 46 49 53 58 65
Japan Single 286,416 274,109 251,600 225,292 212,947 213,494 222,012
Rack/Computer Room 27,532 27,050 25,885 23,673 21,100 18,334 15,706
Midsize DC 380 372 354 324 294 264 236
Enterprise DC 346 341 330 313 301 294 292
Large DC 85 84 83 83 86 92 101
Latin America Single 195,547 196,703 199,196 204,643 216,169 233,087 253,961
Rack/Computer Room 13,325 13,541 13,786 13,957 13,939 13,846 13,774
Midsize DC 821 832 845 858 868 881 899
Enterprise DC 213 216 222 230 241 258 278
Large DC 18 18 19 20 22 25 28
Middle East and Africa Single 108,868 114,214 126,323 143,733 162,834 182,687 207,031
Rack/Computer Room 21,549 21,793 22,133 22,412 22,650 22,793 22,963
Midsize DC 871 881 894 908 921 934 952
Enterprise DC 120 122 126 131 139 148 159
Large DC 21 21 22 23 24 26 27
United States Single 770,925 769,095 749,290 716,352 685,760 664,601 660,355
Rack/Computer Room 184,457 182,963 179,818 174,492 168,593 162,051 154,496
Midsize DC 2,506 2,483 2,435 2,372 2,319 2,276 2,223
Enterprise DC 2,404 2,392 2,377 2,382 2,438 2,539 2,660
Large DC 571 571 574 589 621 669 724
Western Europe Single 536,090 531,772 528,022 525,520 545,062 583,768 647,273
Rack/Computer Room 139,790 138,022 133,181 125,030 116,790 109,967 105,045
Midsize DC 4,860 4,788 4,608 4,337 4,093 3,921 3,822
Enterprise DC 1,196 1,181 1,148 1,106 1,089 1,105 1,153
Large DC 244 243 242 244 256 280 313
Grand Total 3,391,592 3,382,159 3,364,353 3,338,716 3,376,208 3,469,231 3,645,002
Source: Gartner (August 2012)

31. • Most Challenging Technology is Waiting
for all….
Big Data Analytics The Revolution Has Just Begun
31
https://portal.futuregrid.org

32. Conclusion
• Features: All hypervisors are similar.
• Performance: KVM and VirtualBox is fastest across most
benchmarks.
• Overall, I have found KVM to be the best hypervisor choice for
HPC.
– Currently moving to KVM for all of FutureGrid.
32
https://portal.futuregrid.org

33. THANK YOU!
33
https://portal.futuregrid.org