This document summarizes CloudStack networking features and architecture. CloudStack provides orchestration of network services like IPAM, DNS, firewalls, load balancing, and VPN. It supports various network isolation techniques including VLANs, L3 isolation, and overlay networks. The CloudStack virtual router provides default network services, and external devices can also be integrated. CloudStack networking supports advanced configurations including multi-tier networks, bring your own services, and software defined networking.

1. CloudStack Networking
Chiradeep Vittal
May 2 2012

2. Outline
• CloudStack Networking Features
• CloudStack Networking Configuration
• CloudStack Networking APIs
• CloudStack Network Architecture
• Virtual Router deep dive

3. Feature overview
• Orchestration of L2 – L7 network services
– IPAM, DNS, Gateway, Firewall, NAT, LB, VPN, etc
• Mix-and-match services and providers
• Out-of-the-box integration with automated deployment of virtual routers
– Highly available network services using CloudStack HA and VRRP
• Orchestrate external providers such as hardware firewalls and load
balancers
– Devices can provide multiple services
– Admin API to configure external devices
– Plugin-based extensions for network behavior and admin API extensions
• Multiple multi-tenancy [network isolation] options
• Integrated traffic accounting
• Access control
• Software Defined Networking too

4. Basic vs Advanced Networking
• Segmentation based on feature set and ease-of-
deployment
• Both are feature-rich
• Basic implements true AWS-style L3-isolation
– Tenants do not get contiguous IP addresses or subnets
– Network segmentation based on Security Groups
– Tremendous scale (tens of thousands)
• Advanced Zone offers full L3 subnets
– VLANs are default implementation (4K limit)
– More features (source NAT, PF, VPN)

5. CloudStack Terminology
• Guest network
– The tenant network to which instances are attached
• Storage network
– The physical network which connects the hypervisor to primary storage
• Management network
– Control Plane traffic between CloudStack management server and hypervisor clusters
• Public network
– “Outside” the cloud *usually Internet+
– Shared public VLANs trunked down to all hypervisors
• All traffic can be multiplexed on to the same underlying physical network using
VLANs
– Usually Management network is untagged
– Storage network usually on separate nic (or bond)
• Admin informs CloudStack how to map these network types to the underlying
physical network
– Configure traffic labels on the hypervisor
– Configure traffic labels on Admin UI

6. PHYSICAL NETWORK IN A ZONE
Core (L3) Network
Pod 1 Pod 2 Pod N
Cloudstack Access Switch(es)
Server
Cloudstack
Servers
…
CLUSTER 1
Hypervisor 1
VM Traffic …
Hypervisor 8
Control Plane Traffic
Storage Traffic Storage 2
Storage 1
Public Traffic
…
CLUSTER 4
Hypervisor N
Hypervisor N+1
Storage k

7. L2 Features
• Choice of network isolation
– Physical, VLAN, L3 (anti-spoof), Overlay[GRE]
– Physical isolation through network labels [limited to # of nics or bonds]
• Multi-nic
– Deploy instance in multiple networks
– Control default route
• Access control
– Shared networks, project networks
– Dedicated VLANs offer MPLS integration
• Anti-spoofing for L3-isolated networks
• QoS [max rate]
• Traffic monitoring
• Broadcast & multicast suppression in L3-isolated networks
• Hot-plug / detach of nics [upcoming]

8. L3 Features
• IPAM [DHCP], Public IP address management
– VR acts as DHCP server
– Can request multiple public IPs per tenant
• Gateway (default gateway)
– Redundant VR (using VRRP)
– Inter-subnet routing [upcoming]
– Static routing control [upcoming]
• Remote Access VPN
– L2TP over IPSec using PSK
– Virtual Router only
• Firewall based on source cidr
• Static NAT [1:1]
– Including “Elastic IP” in Basic Zone
• Source NAT
– Per-network, or interface NAT
• Public Traffic usage
– Monitoring on the Virtual Router / External network device
– Integration with sFlow collectors
• Site-to-Site VPN [upcoming]
– IPSec VPN based on VR
• L3 ACLs [upcoming]

9. L4 Features
• Security groups for L3-isolation
– “Basic Zone” in docs
– Default AWS-style networking
– Scales much better than VLANs
• Stateful firewall for TCP, UDP and ICMP
• Port forwarding *“Advanced Zone”+
– Conserve public Ips

10. L7 features
• Loadbalancer
– VR has HAProxy built in
– External Loadbalancer support
• Netscaler (MPX/SDX/VPX)
• F5 BigIP
• Can dedicate an LB appliance to an account or share it among tenants
– Loadbalancer supported with L3-isolation as well
– Stickiness support
– SSL support [future]
– Health Checks [future]
• User-data & meta-data
– Fetched from virtual router
• Password change server

11. Physical Network
Operations
Users
Admin and
Cloud API
CloudStack
Mgmt Server
Cluster Router
MySQL
Load Balancer
Availability Zone
L3 Core Switch
Access
Layer
Switches
Secondary
Servers
… … … … … Storage
Pod 1 Pod 2 Pod 3 Pod N

12. Layer 3 cloud networking
Web DB Web
VM VM VM
Web DB
Security Security
Group Group
Web Web DB
VM VM VM
… … …
Web Web
VM VM

13. Guest Networks with L3 isolation
Public Public IP Guest Guest
Internet address 1 VM address
65.37.141.11 1 10.1.0.2
10.1.0.1 Guest
65.37.141.24 Pod 1 L2 Guest
65.37.141.36 Switch 2 VM address
65.37.141.80 1 10.1.0.3
Guest Guest
1 VM address
L3 Core
Pod 2 L2
Switch
10.1.8.1
… 2 10.1.0.4
Switch
Guest Guest
Load 10.1.16. 2 VM address
Pod 3 L2
Balancer 1 2 10.1.16.12
Switch
Guest
Guest
2 VM
address
3
10.1.16.21
… Guest
1 VM
Guest
address
3
10.1.16.47
Guest
Guest
1 VM
address
4
10.1.16.85

14. Virtual Networks (L2 isolation)
Core (L3) Network
Pod K Pod M Pod N
Access Switch(es) V
Hypervisor
V
V
Hypervisor
R
…
CLUSTER 1
Hypervisor 1
R
VM Traffic …
Hypervisor 8
Public Traffic
…
CLUSTER 4
V V
Hypervisor N
V Tenant VM
Hypervisor N+1
V
R Tenant Virtual Router

15. Guest virtual layer-2 network
Guest Virtual Network
10.1.1.0/24
Public Public IP Guest
Gateway Guest
Network address 1 VM
address address
65.37.141.11 1
10.1.1.1 10.1.1.2
65.37.141.36
Guest 1 Guest Guest
Public Virtual 1 VM address
Internet Router 2 10.1.1.3
NAT
Guest Guest
DHCP
1 VM address
Load
3 10.1.1.4
Balancing
VPN Guest Guest
1 VM address
4 10.1.1.5
Guest Virtual Network
Public IP 10.1.1.0/24
address Gateway Guest Guest
65.37.141.24 address 2 VM address
65.37.141.80 10.1.1.1 1 10.1.1.2
Guest 2 Guest Guest
Virtual 2 VM address
Router 2 10.1.1.3
NAT
Guest Guest
DHCP
2 VM address
Load
3 10.1.1.4
Balancing
VPN

16. Layer-2 Guest Virtual Network
CS Virtual Router provides Network Services External Devices provide Network Services
Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8
VLAN 100 VLAN 100
Public Public
Network/Intern Network/Intern
et Guest et Guest
Public IP Private IP 10.1.1.1
10.1.1.1 VM 1 10.1.1.111 VM 1
Gateway 65.37.141.11 Juniper
Public IP 1 SRX
address
65.37.141.11 CS Firewall
10.1.1.1 Guest Guest
Virtual
10.1.1.3 VM 2 10.1.1.3 VM 2
Router
Public IP Private IP
DHCP, DNS 65.37.141. NetScaler 10.1.1.112
NAT Guest 112 Load Guest
Load Balancing 10.1.1.4 VM 3 Blancer VM 3
10.1.1.4
VPN
Guest Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
DHCP, Virtual
Router
DNS

17. Other Topologies
No services [Static Ips] Dedicated VLAN with DHCP and DNS
User can request specific IP[s] for NIC
Guest Virtual Network 10.1.1.0/24 Guest Virtual Network 10.1.1.0/24
VLAN 100 VLAN 100
Guest Guest
VM 1 10.1.1.1 VM 1
10.1.1.1
Gateway address
10.1.1.1
Guest Guest
10.1.1.3 VM 2 Gateway 10.1.1.3 VM 2
address
10.1.1.1
Guest Guest
Core switch 10.1.1.4 VM 3 VM 3
10.1.1.4
Guest
Core switch Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
DHCP, Virtual
Router
DNS
User-data

18. Other topologies
MPLS Shared VLAN with DHCP and DNS
Guest Virtual Network 10.1.1.0/24 Guest Virtual Network 10.1.1.0/24
VLAN 100 VLAN 100
MPLS VLAN 100 Guest Guest
VM 1 10.1.1.1 VM 1
10.1.1.100
Gateway address
10.1.1.1
Guest Guest
10.1.1.200 VM 2 Gateway 10.1.1.3 VM 2
address
10.1.1.1
Guest Guest
Core switch 10.1.1.101 VM 3 VM 3
10.1.1.4
Guest
Core switch Guest
10.1.1.11 VM 4 10.1.1.5 VM 4
5
CS CS
DHCP, Virtual DHCP, Virtual
Router Router
DNS DNS
User-data User-data

19. Multi-tier network
Multi-tier network
Virtual Network Virtual Network
Virtual Network 10.1.2.0/24 10.1.3.0/24
10.1.1.0/24 VLAN 1001 VLAN 141
Public VLAN 100
Network/Intern App VM
10.1.2.31 1 10.1.3.21
et Web VM
Public IP Private IP 10.1.1.1 1 10.1.2.21
65.37.141.11 Juniper 10.1.1.111
1 SRX App VM
Firewall 10.1.2.24 10.1.3.45
Web VM 2
10.1.1.3 2 10.1.2.18
Public IP Private IP
65.37.141. Netscaler 10.1.1.112
112 Load Web VM
Balancer 10.1.1.4 3 10.1.2.38 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4 10.1.2.39
CS DHCP, CS DHCP,
Virtual Virtual DNS CS
DHCP, DNS, Router Virtual
Router
User- User-
DNS Router
data data,
User-
Source
data Public IP
-NAT,
65.37.141.115

20. Bring-your-own Service
Public VLAN(s)
VR
Guest VLAN
Customer
installs static
route to point
to his routing Your
vm VM VM VM Routing
VM
Monitoring VLAN
(shared)

21. Bring-your-own Service[site-to-site-vpn]
Public VLAN(s)
VR
Guest VLAN
Customer
installs static
route
(manually/au Your
tomated VM VM VM Routing
config) to VM
point to his
routing vm.
Routing VM
provides Site- Shared Public VLAN
to-site VPN
(configured
directly on
routing VM,
not by
CloudStack)

22. Multi-tier unified [vision]
Internet
IPSec or SSL site-to-site VPN
CS
Virtual Router / Customer
Loadbalancer Other Premises
Monitoring VLAN
Virtual Router Services
App VM
• IPAM 10.1.2.31 1
• DNS 10.1.1.1
Web VM
1
• LB [intra]
• S-2-S VPN App VM
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 2
• NAT, PF
• FW [ingress & egress]
Web VM
• BGP 10.1.1.4 3 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4
Virtual Network Virtual Network Virtual Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24
VLAN 100 VLAN 1001 VLAN 141

23. Multi-tier unified with SDN[vision]
Internet
IPSec or SSL site-to-site VPN
CS
Loadbalancer Virtual Router / Customer
Other Premises
Virtual Appliance
Monitoring VLAN
Virtual Router Services
App VM
• IPAM 10.1.2.31 1
• DNS 10.1.1.1
Web VM
1
• LB [intra]
• S-2-S VPN App VM
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 2
• NAT, PF
• FW [ingress & egress]
Web VM
• BGP 10.1.1.4 3 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4
Overlay Overlay Overlay
Network Network Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24

24. Network Offerings
• Cloud provider defines the
feature set for guest networks
• Toggle features or service levels
– Security groups on/off
– Load balancer on/off
– Load balancer software/hardware
– VPN, firewall, port forwarding
• User chooses network offering
when creating network
• Enables upgrade between
network offerings
• Default offerings built-in
– For classic CloudStack networking

25. Service Offerings
Specify Resource Levels Configure Properties Define Scope
Compute Disk Network
Name Name Name
CPU Cores Custom Disk Size Network Rate
CPU (MHz) Disk Size (GB) Redundant VR
Storage Tag Firewall
Memory (MB)
Load balancer
Host Tag
Enable HA Public Public
CPU Cap
Public

26. CloudStack Network Service Providers
• A Network Service Provider is hardware or virtual
appliance that makes a network service possible in
CloudStack ; for example, a Citrix NetScaler
appliance can be installed in the cloud to provide
Load-Balancing services.
• Administrators can have multiple instances of the
same service provider in a network; for example,
more than one Citrix NetScaler or Juniper SRX
device can be added to CloudStack
• CloudStack supports the following Network
Providers:
– CloudStack Virtual Router (default)
– Citrix NetScaler SDX, VPX and MPX models
– Juniper SRX
– F5 BigIP

27. Adding an Additional Network Offerings
Network Network
Offering Offering Order
Status control

28. Network Service Providers Matrix
• Network offerings is basically a definition of what Network Services are
available when this offering is used. The available Network Services are: VPN,
DHCP, DNS, Firewall, Load Balancer, User Data, Source NAT, Static NAT, Port
Forwarding and Security Groups*
Feature Virtual Router Citrix Juniper SRX F5 BigIP
NetScaler
Remote Access VPN YES N/A N/A N/A
Firewall YES N/A YES N/A
Source NAT YES N/A YES N/A
Static NAT YES YES YES N/A
Load Balancing YES YES N/A YES
Port Forwarding YES N/A YES N/A
Elastic IP N/A YES N/A N/A
Elastic LB N/A YES N/A N/A
DHCP/DNS/User Data YES N/A N/A N/A

29. CloudStack User APIs [sample]
• Networks (L2)
– createNetwork [requires network offering id],
– deleteNetwork (A), listNetworks,
– restartNetwork (A): restarts all devices (if allowed)
supporting the network and re-applies
configuration
– updateNetwork: update network offering and
restart network

30. Adding a Shared Guest Network
• Only Administrators can add a Shared Guest Network for an Advanced zone

31. Adding a Shared Guest Network
VLAN required!

32. Editing Guest Networks
When editing a guest network
users can change the network
offering. They can either upgrade
to a “premium” network offering
(for example offering that uses
hardware Load-balancer) or
downgrade to a “cheaper”
network.

33. Restarting and Cleaning Up a Guest Network
• Restarting the network will
simply resend all the LB,
Firewall and Port-Forwarding
rules to the network provider
• Restarting the Network with
“Clean up”:
• restarting network elements - virtual
routers, DHCP servers
• If virtual router is used, it will be destroyed
and recreated
• Reapplying all public IPs to the network
provider
• Reapplying load-Balancing/Port-
Forwarding/Firewall rules

34. Deleting a Guest Network
• An Isolated Guest Network can only be deleted if no VMs are
using these network (e.g. Completely destroyed and expunged)
• Deleting a Network will Destroy the Virtual Router (if used) and
will release the Public IPs back to the IP Pool

35. Extending CloudStack Networking
2. prepare (Network, Nic, DeployDestination, VmInfo)
1. prepare (part of start vm)
Network Network Element PluggableService
Manager
Needs to be added as of 5/2/2012 Device Configuration
MyDnsDeviceSer Admin API (CRUD)
DnsService
vice
3. addDnsRecord(ip, fqdn)
Demonstrates one way to MyDnsDeviceMa MySQL
MyDnsElement
inform an external DNS nager
server when an instance
starts. AgentManag
4.Enqueue AddDnsRecord er Queue
Classes shaded blue form a
plugin / service bundle to
integrate an external DNS MyDnsDeviceRes
server. Clients of the ource
instance can then use DNS
names to access the 5.API call to Dns Device
instance.

36. CloudStack Virtual Router (Virtual
Router)
• The Virtual Router will be deployed once (when the first
instance is deployed in a Zone) when a Shared Network is used
providing DHCP and DNS services for the Zone’s Instances (IPs
will be allocated from the Public IP Range entered in
CloudStack)
• When Advanced is used the Router will be deployed Per-
Account (and Per Unique Isolated Guest Network)
• Virtual Router can serve and isolate VMs even if deployed on a
different Hypervisor

37. CloudStack Virtual Router
• The Virtual Router will have 3 NICs:
– Eth0 will be connected to the Isolated Guest Network (for Advanced VLAN). It will have the first IP in
the CIDR (for example10.1.1.1) and it will be the DNS, DHCP and Gateway for the Instances in the
Private Guest Network.
– Eth1 resides on local-link network (only for KVM and XenServer) or the Management Network (on
VMware) and is used by CloudStack to configure the virtual router. On VMware it will use an IPs from
the Management Network IP Range (e.g. Pod Private Range)
– Eth2 resides on the Public Network and assigned with a Public IP from the range entered in CloudStack
(users can ‘Acquire New IPs’ if needed)
• In the default Isolated Mode - Source NAT is automatically configured on
the virtual router to forward outbound traffic for all guest VMs and block all
incoming traffic (users can manage incoming rules from UI)

38. Virtual Router Information (applies to
all Sys. VMs)
• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security
APT repository. No extraneous accounts
• 32-bit for enhanced performance on Xen/VMWare
• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,
dns, sendmail are not installed.
• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-
standard port. SSH logins only using keys (keys are generated at install time and are unique for
every customer)
• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum
performance on all hypervisors. Xen tools inclusion allows performance monitoring
• Template is built from scratch and is not polluted with any old logs or history
• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved
security and speed
• Latest version of jre from Oracle ensures improved security and speed