Building an infrastructure as a service cloud involves a number of technology decisions, many of which could have unforeseen impact. Hypervisors form the core of an IaaS cloud, and whether you are a fan of Microsoft Hyper-V, VMware vSphere, KVM in any Linux variant or XenServer from Citrix, each of these hypervisors provide unique capabilities within an Apache CloudStack 4.4 based cloud.

1. Hypervisor Selection in Apache CloudStack 4.4
Understanding the choices available
BACD-RICON
Tim Mackey – XenServer Community Manager and Evangelist

2. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
whoami
Name: Tim Mackey
Current roles: XenServer Community Manager and Evangelist; occasional coder
Cool things I’ve done
• Designed laser communication systems
• Early designer of retail self-checkout machines
• Embedded special relativity algorithms into industrial control system
Find me
• Twitter: @XenServerArmy
• SlideShare: slideshare.net/TimMackey

3. What are we trying to accomplish?
Building a successful cloud

4. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Service Offerings
Clearly define what you want to offer
• What types of applications
• Who has access, and who owns them
• What type of access
Define how templates need to be managed
• Operating system support
• Patching requirements
Define expectations around compliance and availability
• Who owns backup and monitoring

5. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Define Tenancy Requirements
Department data local to department
• Where is the application data stored
Data and service isolation
• VM migration and host HA
• Network services
Encryption of PII/PCI
• Where do keys live when data location unknown
• Need encryption designed for the cloud
Showback to stakeholders
• More than just usage, compliance and audits

6. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Virtualization Infrastructure
Hypervisor defined by service offerings
• Don’t select hypervisor based on “standards”
• Understand true costs of virtualization
• Multiple hypervisors are “OK”
• Bare metal can be a hypervisor
To “Pool” resources or not
• Is there a real requirement for pooled resources
• Can the cloud management solution do better?
• Real cost of shared storage
Primary storage defined by hypervisor

7. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Apache CloudStack
Current release: 4.4 (4.4.1 imminent)
Highly scalable
Monolithic architecture
Mostly written in Java
Multi-hypervisor support
• XenServer, KVM, OracleVM, vSphere, Linux
Containers, Hyper-V, Bare metal
Back Up
DB
Management
Server
MySQL
DB
Replication
Management
Server
Management
Server
Load
Balancer
Infrastructure
Resources

8. Hypervisor choices ….

9. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Networking StorageCompute
Simplified XenServer Architecture Diagram
Xen Project Hypervisor
Standard Linux
Distribution (dom0)
q
e
m
u
drivers
xapi
Guest
Driver front
Driver back
Guest
Driver front

10. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
KVM (Linux + KVM only)
Standard Linux Distribution
qemudrivers
Guest
Virtual driver
virtio
Guest
Virtual driver
KVM Module
libvirt
agent
Networking StorageCompute

11. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
vSphere 5.1 – Managed by vCenter
vmkernel
Guest
Virtual driver
vSCSI
Guest
Virtual driver
Task
Scheduler
Service
Console
vmklinux
vCenter
drivers
vNIC
Networking StorageCompute

12. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Linux Containers
Standard Linux Distribution
Namespac
e
Container
Namespac
e
Container
KVM Module
libvirt
agentCgroups
Cgroup Cgroup
Namesspaces
Networking StorageCompute

13. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Microsoft Hyper-V
Networking StorageCompute
Hyper-V Hypervisor
Standard Windows Server (parent partition)
drivers
WMI
Guest
VMBus
Guest
Virtual SP
VMBus VMBus
Devices Devices

14. Defining the network

15. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Flat Network – Basic Layer 3 Network
65.11.1.2
65.11.1.3
65.11.1.4
65.11.1.5
Public Network
65.11.0.0/16
Guest VM 1
Guest VM 2
Guest VM 3
Guest VM 4
DHCP,
DNS
CloudStack
Virtual Router
Security Group 1
Security Group 2
Option XenServer vSphere KVM Hyper-V
Security Groups Yes- bridge No Yes Yes
IPv6 Yes No Yes No
Multiple IPs per NIC Yes Yes Yes Yes
Nicira NVP Yes No Yes No
BigSwitch VNS Yes No Yes No

16. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
VLANs for Private Cloud
10.1.1.1
10.1.1.3
10.1.1.4
10.1.1.5
Public
Network/Internet
Guest Virtual Network 10.0.0.0/8
VLAN 100
DHCP, DNS
NAT
Load
Balancing
VPN
Public IP
65.37.14.1
Gateway
10.1.1.1
Guest VM
1
Guest VM
2
Guest VM
3
Guest VM
4
CloudStack
Virtual
Router
Option XenServer vSphere KVM Hyper-V
Max VLANs 800 254 1024 4094
IPv6 Yes No Yes No
Multiple IPs per
NIC
Yes Yes Yes Yes
Nicira NVP Yes No Yes No
BigSwitch VNS Yes No Yes No
MidoKura No No Yes No
VPC Yes Yes Yes Yes
NetScaler Yes Yes Yes Yes
F5 BigIP Yes Yes Yes Yes
Juniper SRX No Yes Yes Yes
Juniper EX/QFX No Yes Yes No
Cisco VNMC No Yes No No

17. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Beyond the VLAN – Network Virtualization
Option XenServer vSphere KVM Hyper-V
OVS GRE tunnels Yes No No No
Nicira STT tunnel Yes Yes Yes No
MidoNet No No Yes No
VXLAN No Yes Yes No
NVGRE No No No No
Nexus 1000v No Yes No No
Juniper Contrail Yes No No No
Palo Alto Yes Yes Yes No

18. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Virtual Private Cloud and nTier Applications
Web
App
DB
Router
DC1
DC2
DC3
DC4
DC5
DC6
VLAN 1
VLAN 2
VLAN 3
S2S VPN
Private
GW
Feature XenServer vSphere KVM Hyper-V
PVLAN Yes - ovs Yes ovs Yes –
Hyper-V
VR
required
IPv6 Yes No Yes No
Distributed routing Yes - ovs No ovs No

19. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Delivering specific network services
KVM
• IPv6
• Security groups
• Large quantity of VLANs
vSphere
• VXLAN requires vSphere Enterprise Plus
• Cisco Nexus 1000v and ASA 1000v require vSphere Enterprise Plus
XenServer
• Security groups
• Large quantity of VLANs
• IPv6
• Juniper Contrail

20. Instances need a home …
Storage, Storage and more Storage

21. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Primary Storage Options
Cluster
Host
Host
Primary Storage
Feature XenServer vSphere KVM Hyper-V
Local storage Yes Yes Yes Yes
NFS Yes Yes Yes No
SMB No No No SMB3
Single path iSCSI Yes Yes Yes No
Multipath iSCSI PreSetup No No No
Direct array No VAAI No No
Shared Mount No No Yes No
Template format VHD OVA QCOW2, VHD,
VMDK, RAW,
IMG
VHD, VHDX
SolidFire Plugin Yes Yes Yes No
NetApp Plugin Yes Yes Yes No
CloudBytes Elastistor Yes No No No
Zone wide No Yes Yes No
Ceph RBD No No Yes No
Clustered LVM No No Yes No

22. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Secondary Storage Options
Option XenServer vSphere KVM LXC Hyper-V
NFS Yes Yes Yes Yes No
Swift(1) Yes Yes Yes Yes No
S3 compatible
(2)
Yes Yes Yes Yes No
SMB No No No No Yes
(1) Requires NFS staging area
(2) Can be region wide, but must not have NFS secondary storage in zone
Zone
Secondary Storage
Pod
Cluster
Host
Host
Primary Storage

23. Core virtualization capabilities
The limits and features which matter

24. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
CloudStack Features
Feature XenServer vSphere KVM Hyper-V
Disk IO Statistics Yes No Yes Yes
Memory Overcommit Yes (4x) Yes No No
Dedicated resources Yes Not with HA/DRS Yes Yes
Disk IO throttling No No Yes No
Disk snapshot (running) Yes Yes No No
Disk snapshot (pluggable) Partial Partial No No
Disk snapshot (Stopped) Yes Yes Yes Yes
Memory snapshot Yes Yes Yes No
Zone wide primary storage No Yes Yes SMB 3.0 only
Resize disk Offline Online Grow Online No
High availability CloudStack Native CloudStack CloudStack
CPU sockets 6.2 and higher Yes Yes Yes
Affinity groups Yes Yes Yes Yes
GPU passthrough/vGPU 6.2 SP1 and higher No No No
AutoScaling VM Instances Native, NetScaler NetScaler NetScaler NetScaler

25. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
XenServer 6.2
Feature
Source code model Open Source (GPLv2)
Maximum VM Density 650 (Linux)
CloudStack VM Density 500
CloudStack integration Direct XAPI calls
OpenStack driver OpenStack nova-compute domU
Maximum native cluster size 16
Maximum pRAM 1 TB
Largest supported VM 16 vCPU/128GB
Windows Operating System All Windows supported by Microsoft
Linux Operating Systems RHEL, CentOS, Debian, Ubuntu, SLES, OEL
Advanced features supported (CloudStack) ovs, Storage XenMotion, DMC
Advanced features supported (OpenStack) ovs, Storage XenMotion

26. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
vSphere 5.5
Feature
Source code model Proprietary
Maximum VM Density 512
CloudStack VM Density 128
CloudStack integration vCenter
OpenStack driver vCenter – nova-compute node per cluster
Maximum native cluster size 32
Maximum pRAM 4 TB
Largest VM 64 vCPU/1TB
Windows Operating Systems DOS, All Windows Server/Client
Linux Operating Systems Most
Advanced features supported (CloudStack) HA, DRS, vDS, Storage vMotion
Advanced features supported (OpenStack) HA, DRS, vMotion

27. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
KVM
Feature
Source code model Open Source (GPLv2)
Maximum VM Density 10 times the number of pCores
CloudStack VM Density 50
CloudStack integration CloudStack Agent (libvirt)
OpenStack driver libvirt driver
Maximum native cluster size No native cluster support
Maximum pRAM 2 TB
Largest VM 160 vCPU/2TB
Windows Operating Systems Windows XP and higher
Linux Operating Systems Varies
Advanced features supported (CloudStack) None
Advanced features supported (OpenStack) None

28. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Linux Containers
Feature
Source code model Open Source (GPLv2)
Maximum container Density 6000 (theoretical)
CloudStack container Density 50
CloudStack integration CloudStack Agent (libvirt), requires KVM for
SVMs
Maximum native cluster size N/A
Maximum pRAM 2 TB
Largest container 2TB
Windows Operating Systems N/A
Linux Operating Systems Kernel compatible distros

29. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Microsoft Hyper-V
Feature
Source code model Proprietary
Maximum VM Density 1024
CloudStack VM Density 1024
CloudStack integration CloudStack Agent (C# calling WMI)
OpenStack driver Use Cloudbase driver
Maximum native cluster Size 64
Maximum pRAM 4 TB
Largest VM 64 vCPU/1TB
Windows Operating Systems All Windows supported by Microsoft
Linux Operating Systems RHEL, CentOS, Debian, Ubuntu, SLES, OEL
Advanced features supported (CloudStack) VHDX, Storage Motion (shared only)
Advanced features supported (OpenStack) None

30. Picking the “best one”
When to use which hypervisor…

31. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
KVM
Primary value proposition:
• Low cost with available vendor support and familiar administration model
• Broad feature set with active development in both CloudStack and OpenStack
Cloud use cases:
• Linux centric workloads
• Dev/test clouds
• Web hosting
• Tenant density which dictates SDN options
Weaknesses:
• CloudStack: Requires use of an installed libvirt agent
• Limited native storage options
• No use of advanced native features

32. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Microsoft Hyper-V
Primary value proposition:
• Unlimited Windows Server VM licenses
• Familiar Windows management paradigm
Cloud use cases:
• Windows and Linux workloads
• Dev/test clouds
• .Net application web hosting
• Desktop as a Service clouds
Weaknesses:
• Minimal use of advanced native features
• CloudStack: First introduced with CloudStack 4.3
• CloudStack: Can not mix Hyper-V in a Zone

33. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
vSphere
Primary value proposition:
• Broad application and operating system support with large eco-system of vendor partners
• Readily available pool of vSphere administration talent
• CloudStack: Many features are native implementations
• Direct feature integration via vCenter
Cloud use cases:
• Private enterprise clouds
• Dev/test clouds
Weaknesses:
• vSphere up-front license and ongoing support costs
• vCenter integration requires redundant designs
• CloudStack: Single data center per zone model

34. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
XenServer
Primary value proposition:
• Low cost with available vendor support
• Broad feature set with active development in both CloudStack and OpenStack
• CloudStack: Large install base
• Direct integration via XAPI toolstack
Cloud use cases:
• Linux centric workloads
• Dev/test clouds and web hosting providers
• Desktop as a Service clouds
• Large VM density and secure tenant isolation
Weaknesses:
• Minimal use of advanced native features

35. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Multiple Hypervisor Support in CloudStack
Networking
• Ensure network labels match
• Topology is intersect of chosen hypervisors
• Hyper-V requires Hyper-V system VMs
Storage
• For system VMs to specific hypervisor type
• Zone with primary storage limited
Operations
• vSphere Datacenter can not span zones
• Hyper-V may not be mixed with other hypervisors in a zone
• HA won’t migrate between hypervisors
• Capacity planning at the cluster/pod level more difficult

36. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
Tying it all Together
1. Define success criteria
2. Select a topology which works
3. Decide on storage options
4. Define supported configurations
5. Select preferred hypervisor(s)
6. Validate matrix
7. Build your Cloud

37. © 2014 Citrix. Hypervisor Selection in Apache CloudStack 4.4 – Build a Cloud Day at RICON 2014 Tim Mackey @XenServerArmy
WORK BETTER. LIVE BETTER.