Nested CloudStack with VMware - Presented by Paul Angus at CloudStack Collaboration Conference 2014 in Budapest

1. Nested CloudStack with VMware
Paul Angus
Cloud Architect ShapeBlue
paul.angus@shapeblue.com
Twitter: @CloudyAngus

2.  Why – not just a science project
@shapeblue #ccceu14
 How
 What (you can do with it)
 Next steps
Nested CloudStack with VMware

3.  Cloud Architect with ShapeBlue
@shapeblue #ccceu14
 Specialise in….
 Designing and deploying enterprise and
public clouds
 Involved with CloudStack before donation to
Apache
 Designed Clouds Orange, TomTom,
PaddyPower, Ascenty, BSkyB
 CloudStack Committer
About Me

4. About ShapeBlue
“ShapeBlue are expert builders of public &
private clouds. They are the leading global
Apache CloudStack
integrator & consultancy”
@shapeblue #ccceu14

6. Nested CloudStack with VMware
Why?
@shapeblue #ccceu14

7. @shapeblue #ccceu14
CEO
Why
“Building CloudStack
environments using CloudStack?
Are you just having fun?”

8. Why
Talented Cloud Architect
“If we can automate the
building of environments
including the networking, and
generate any topology we
want quickly, we can achieve
the tasks that I’ve highlighted
in these slides that I have with
me, just in case you asked.”
@shapeblue #ccceu14
“Building CloudStack
environments using CloudStack?
Are you just having fun?”

9. Why
@shapeblue #ccceu14
CEO
“That would be excellent.
Go ahead.
Oh, and here’s a pay rise”
building of environments
including the networking, and
generate any topology we
want quickly, we can achieve
the tasks that I’ve highlighted
in these slides that I have with
me, just in case you asked.”

10. Why?
@shapeblue #ccceu14
Client:
“We have a large global multi-zone public deployment, which we need to
upgrade. Upgrade requires all virtual routers to be restarted. Virtual router
restarts need to be done in phases (due to time zones). We need to know
what the users will experience while CloudStack has been upgraded but
the virtual routers have not be restarted. We also want to know if the
latency between some zones will be an issue during the upgrade.
No, we don’t have a couple of dozen hosts that you can use”

11. @shapeblue #ccceu14
Test/Dev
Need to be able to create full environments to test:
 CloudStack release candidates
 CloudStack features
 ShapeBlue patches
Why

12. Why?
@shapeblue #ccceu14
Training
 Bootcamp runs in Virtualbox – limited by amount of RAM and cores in laptops
 We can’t use KVM or ESXi hosts because no HVM support for guests.

13. Why?
@shapeblue #ccceu14
Client Demos
 Need to create demo environments to show features

14. Nested CloudStack with VMware
How
@shapeblue #ccceu14

15. Nested Hypervisors
 Nested virtualisation has been possible in ESXi since 4.1 with some hacking
(editing /etc/vmware/config and changing the CPU IDs) if the CPU supported
VT-x or AMD-V
 Since 5.1 nested virtualisation more powerful with the introduction of support
for 2nd generation hardware virtualisation within processors. Extended Page
Tables (EPT) from Intel (Nehalem onwards) or Rapid Virtualization (RV) from
AMD (0x10 Barcelona onwards)
 Still not supported for production loads
@shapeblue #ccceu14

16. Nested Hypervisors
@shapeblue #ccceu14
Enabling nested virtualisation in
CloudStack enables the hardware
virtualisation feature.

17. Networking
 Trunking VLANs no problem with vSwitches/bridges on any hypervisor
 Portgroup/vSwitch need to be set to promiscuous mode to nest network
@shapeblue #ccceu14
interfaces.
 To create a hypervisor VM purely in CloudStack we need to add a network
interface which can use a range of VLANs for the guest VLANs
 But within CloudStack you can only map one VLAN to any network

18. Networking
On an ESXi host, using VLANID 4095 tells the host to trunk all VLANs through the
port group.
@shapeblue #ccceu14

19. Guest Networks
@shapeblue #ccceu14
 Trunked Guest port group can
share vSwitch with other traffic
or be on a dedicated vSwitch

20. Shared Guest Networks
Create a shared network on VLAN 4095
@shapeblue #ccceu14

21. Isolation of Shared Networks
ESXi host Switch
VLAN
150
ESXi host
vSwitch0
@shapeblue #ccceu14
 Each CloudStack zone has a
range of guest VLANs.
 The range will be created within
the nested CloudStack.
May need to avoid overlapping
VLANs.
vSwitch0
VLANs 100-150
Top Secret Data
VLANs 100-150
VLANs 120-200

22. Isolation of Shared Networks
vSwitch0
VLAN 150
@shapeblue #ccceu14
 Either physically
separate the vSwitch
 Or do not connect an
uplink to the vSwitch
vSwitch1
VLANs 100-150
Top Secret Data
ESXi host Switch
VLANs 151-200

23. Isolation
@shapeblue #ccceu14
If isolation required:
 Create additional vSwitch
 Create additional physical guest network mapped to this vSwitch
 Multiple guest networks means network offerings must be tagged.
If no isolation:
 ‘Usual’ guest networking vSwitch OK.

24. @shapeblue #ccceu14
Hypervisor Instance
 Guest Traffic
- Shared (trunked) network
 Public Traffic
- Shared network
 Management Traffic
- Isolated network
Building Environments

25. Isolated Network Dynamic VLAN per tenant
@shapeblue #ccceu14
 Simple CloudStack
architecture
Building Environments
Host1
Host2
ACSMan
With NFS MySQL
Public Network (Shared) Fixed VLAN (no DHCP)
Guest Network (Shared) Trunked

26. Nested CloudStack with VMware
What
(you can do with it)
@shapeblue #ccceu14

27. Other Infrastructure Components
@shapeblue #ccceu14
 Any hypervisor
 Multiple management servers
 MySQL master/slave, Galera clusters
etc
 Separate NFS inc.
NetApp ONTAP Edge
 Software load balancers
 Virtual appliances
 Object Storage for sec storage
 User portals

28. @shapeblue #ccceu14
Create bare VM
/ hosts from
templates
Install Ansible
on guest
instance
Git clone mega-playbook
Update hosts
and group_vars
Push
application
stack to VMs
Building Environments

29. Building Environments
@shapeblue #ccceu14
Hypervisor Builds
 ESXi – multiple templates as cold standby
 XenServer – deployed from template:
 Reset installation_uuid, state.db and control_domain_uuid
 Recreate network interfaces and storage devices
 KVM – Base OS then push cloudstack-agent
 Windows vCenter - multiple templates as cold standby

30. MySQL
Slave
@shapeblue #ccceu14
 Easy to build up
complex
environments
Building Environments
Host1
Host2
ACSMan1
MySQL
Master
Isolated Network Dynamic VLAN per tenant
Public Network (Shared) Fixed VLAN (no DHCP)
Guest Network (Shared) Trunked
NFS
ACSMan1
NetScaler
NetScaler
CSForge
Server

31. MySQL
MySQL
@shapeblue #ccceu14
 3 zones
 2 geographic
locations
 Upgrade done,
then tests run
for a week. Then
VRs restarted
Client Test Environment
CCP3.0.7B
CPBM 2.2
ESXi 1a
ESXi 1b
ESXi 1c
vCenter
Appliance
ESXi 2a
ESXi 2b
ESXi 2c
vCenter
Appliance
Zone 1 (local)
Zone 2 (local)
Zone 3 (remote)
NFS
NFS
VPN VPN
ESXi 2a
ESXi 2b
ESXi 2c
NFS
vCenter
Appliance

32. What you can do with it
@shapeblue #ccceu14
Test Dev
 Automate the building of entire environments to test releases, features and
patches against.
Client Demos
 Need to create demo environments to show features

33. @shapeblue #ccceu14
Training
 Hosted training in environments containing all hypervisors with a full
infrastructure.
What you can do with it

34. Nested CloudStack with VMware
Next Steps
@shapeblue #ccceu14

35. Nested CloudStack with VMware
@shapeblue #ccceu14
Next steps:
 Automated build of Windows VMs
 Bootstrapped build of ESXi hosts.
 Using AWS module to provision VMs
 CloudStack module
 Dynamic inventories

36. Nested CloudStack with VMware
Paul Angus
Cloud Architect ShapeBlue
paul.angus@shapeblue.com
Twitter: @CloudyAngus