CloudStack provides an orchestration platform that abstracts physical network resources and allows third party plugins to integrate their networking services. It separates orchestration from actual provisioning, with CloudStack only handling orchestration events and notifications, while provisioning is handled by plugins. This allows services to scale independently of CloudStack. CloudStack defines common concepts like Networks, but plugins determine how these map to physical networks through interfaces like NetworkGuru. This architecture enables innovation from partners through well-defined plugin APIs and abstraction layers.

1. CloudStack Orchestration
by Alex Huang

2. A little bit about me
• Cloud.Com Founding Engineer
• Software architect for CloudPlatform
• Responsible for overall
architecture, performance, and scalability
• Committer and PPMC member
• BS from UC Berkeley and MS from Stanford

3. Outline
• Difference between
automation, orchestration, and provisioning
• Networking Example
• Anatomy of a plugin
• Myth busting

4. Automation, Orchestration, &
Provisioning

5. What is the difference?
• The key is the data center abstraction layer
– Virtual Machine, Template, Nic, IP
Address, Volume, Network, Rules, Snapshot
• Orchestration orchestrates within this abstraction
layer
• Provisioning manifests concepts in abstraction
layer on physical resources
• Automation orchestrates above the data center
abstraction layer to bring greater functionality

6. Examples
• Orchestration
– VM deploy, Volume creation, Network creation,
Network rules propagation
• Provisioning
– Starting a VM on a hypervisor
– The actual movement of a volume from one storage
pool to another
• Automation
– HA
– Putting a physical resource into maintenance mode

7. Enabling Innovation
• CloudStack Orchestration must not define the
innovation.
– Partners define their own APIs.
– Partners and CloudStack can work together on unified APIs
through design process on Apache.
• Differentiate between orchestration and provisioning.
– CloudStack only orchestrates.
– Provisioning is always pushed to the partner.
• Clearly defined data center abstraction layer.
– Changes in this layer are broadcasted to partners.
• Utilize CloudStack’s orchestration to deploy and auto-
scale partners’ technologies.

8. Networking Example

9. CloudStack Terminology (End User)
• Network
– A single concept to encapsulate multiple network technologies to
simplify representation to the end user.
– Each Network always carries its Network Traffic Type.
– CloudStack DOESN’T understand how to provision this conceptual
network on to the physical network.
• NetworkService
– L2-L7 network services that partners have written to operate within a
Network.
– Currently defined: Load Balancing, Port
Forwarding, Firewall, Gateway, DNS, DHCP, Static NAT, VPN, Source
NAT, User Data.
• NetworkOffering
– A packaging of the NetworkServices provided to the end user on a
particular Network.
– NetworkOfferings are put together by cloud service provider.

10. CloudStack Terminology (Provider)
• Network Traffic Type
– Traffic types are mapped to the underlying physical
network by the cloud service provider.
– Traffic type is not the same as network (Guest traffic type
can actually be carried on multiple networks)
– Currently defined: Public, Guest, Storage (Backup really),
Management
• NetworkServiceProvider
– Plugin that understands how to provide one or more
NetworkServices by using VPX or physical resource.
• PhysicalNetwork
– Actual wiring of the data center.

11. CloudStack Terminology (Partner)
• NetworkGuru
– Plugin that understands the network isolation technology, mac addressing scheme, and IP
addressing scheme deployed and how to map Network Traffic Types to the underlying physical
network.
– CloudStack passes Network to NetworkGuru to “implement” before the network is needed by
a virtual machine.
– CloudStack asks the NetworkGuru to issue ip, mac, and isolation to a virtual machine before it
starts.
– CloudStack informs the NetworkGuru when a virtual machine stops so it can collect resources.
– When all virtual machines in a Network are stopped, CloudStack garbage-collects the Network
by asking the NetworkGuru to shutdown the network.
– CloudStack provides a default implementation for VLAN based isolation technology.
• NetworkElement
– Interface that specifies the events CloudStack signals to the NetworkServiceProviders when a
Network needs to be “implemented” and shutdown and when a virtual machine joins and
leaves a Network.

12. “Architect” Model
• The builder offers multiple blueprints for the owner to build the house.
• Owner chooses on a blueprint and then adds on with additional enhancements
such as hardwood floors, granite counter tops, etc.
• General contractor builds to the blueprint by orchestrating between different sub-
contractors to build different parts of the blueprint.
• There are two general category of contractors.
– Rough-in sub-contractors who take care of plumbing, electricity, framing, foundation.
– Finish sub-contractors who put in flooring, kitchen cabinets etc.
• Each sub-contractor is responsible for only their work but looks over the entire
blueprint to make sure their work can actually be done.
– E.g. A lighting plan may conflict or needs to change depending on the framing plan.
• General contractor is responsible for sequencing the sub-contractors to make sure
everything the sub-contractor is dependent on is ready when the sub-contractor
arrives to do his work.
• Every change requires a the blueprint to be republished so every sub-contractor
can make their appropriate changes.

13. Comparison
Building a house Building a network
• Owner • End user
• Builder • Cloud Service Provider
• General Contractor • CloudStack
• Rough-in Sub-Contractors • NetworkGurus
• Finish Sub-Contractors • NetworkServiceProviders
• Blueprint • Abstraction Layer
• Cabinets, Flooring, Counter • NetworkServices
Tops, etc

14. Architectural Principles
• CloudStack clearly defines the difference between orchestration
and provisioning.
– Orchestration the ordering of what needs to happen in CloudStack’s
abstraction layer.
– Provisioning is the actual work performed at the resource.
• CloudStack clearly defines the difference between network
definition and network services.
– Network definition is handled by NetworkGuru.
– Network services is handled by NetworkServiceProvider.
• CloudStack broadcasts changes in the network every time
NetworkServices and virtual machines changes in the Network.
• CloudStack allows the Cloud Service Provider to setup the
appropriate mappings between virtual concepts such as Network
and Network Traffic Type to the underlying physical network.

15. Sequence Flow for VM Creation
Engine
End User Security User VM VirtualMac Network Storage Network Job
Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate IP
Allocate Volume
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status

16. Sequence Flow for VM Deployment
Deploymen Server
User VM VirtualMac Network Storage Network Network Template t
Job Threads Services API Resources
Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner
Start VM
Start User VM
Start VM
Get a Deployment Plan (Host and StoragePool)
Prepare Nics
Reserve resources for Nic
Notify that Nic is about to be started in network
Agent Calls
Prepare Volumes
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result

17. Anatomy of a plugin

18. CloudStack WebServices API
End User
OAM&P API AWS API Pluggable Service API Engine
API
Virtual Router Netwrok Service Provider
Accounts Business Logic
User Dispersing Deployment Planner
CloudStack Plugins
NetScaler Netwrok Service Provider
Security
First Fit Deployment Planner
Resource
Manager
Manager
Manager
Manager
Manager
Capacity
Update
Manager
Rules
HA
Events
Manager
Usage
Manager
CloudStack Orchestration Adapters
Network Guru
Domain
Network Manager
Storage Manager
Virtual Machine
Manager Network
Snapshot
Template
Manager
Manager
Manager Element
Account Deployment
Manager Planner
Hypervisor
Limits Guru
Manager
Framework
Agent Manager Cluster Manager Data Access Layer

19. Anatomy of a Plugin
• Server Component:
– Deployed on management server
Rest API – Can implement multiple Plugin
APIs to affect its feature
Implementation – Can expose its own API through
Plugin API
Pluggable Service so
administrators can configure the
Data Access Layer plugin
– Can access the database
• ServerResource:
ServerResource – Deployed co-located with the
- Optional. Required if Plugin needs to be co-
located with the resource physical resource
- Implements translation layer to talk to resource
- Communicates with server component via JSON – Cannot access the database

20. Anatomy of a Plugin
• Can be two jars: server component to be
deployed on management server and an optional
ServerResource component to be deployed co-
located with the resource
• Server component can implement multiple Plugin
APIs to affect its feature
• Can expose its own API through Pluggable Service
so administrators can configure the plugin
• As an example, OVS plugin actually implements
both NetworkGuru and NetworkElement

21. Plugin Interfaces Available
• NetworkGuru – Implements various physical network technologies
and ip address
• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and
volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes

22. Adding a Plugin to CloudStack
• Components are configured though
components.xml
• Supports DAO, Manager, and Adapter patterns

23. Components.xml Example
<components.xml>
<system-integrity-checker class="com.cloud.upgrade.DatabaseUpgradeChecker">
<checker name="ManagementServerNode" class="com.cloud.cluster.ManagementServerNode"/>
<checker name="EncryptionSecretKeyChecker"
class="com.cloud.utils.crypt.EncryptionSecretKeyChecker"/>
<checker name="DatabaseIntegrityChecker" class="com.cloud.upgrade.DatabaseIntegrityChecker"/>
<checker name="DatabaseUpgradeChecker" class="com.cloud.upgrade.PremiumDatabaseUpgradeChecker"/>
</system-integrity-checker>
<interceptor library="com.cloud.configuration.DefaultInterceptorLibrary"/>
<management-server class="com.cloud.server.ManagementServerExtImpl"
library="com.cloud.configuration.PremiumComponentLibrary">
<adapters key="com.cloud.storage.allocator.StoragePoolAllocator">
<adapter name="LocalStorage" class="com.cloud.storage.allocator.LocalStoragePoolAllocator"/>
<adapter name="Storage" class="com.cloud.storage.allocator.FirstFitStoragePoolAllocator"/>
</adapters>
<pluggableservice name="VirtualRouterElementService"
key="com.cloud.network.element.VirtualRouterElementService"
class="com.cloud.network.element.VirtualRouterElement"/>
</management-server>
</components.xml>

24. Extending CloudStack Networking
2. prepare (Network, Nic, DeployDestination, VmInfo)
1. prepare (part of start vm)
Network Network Element PluggableService
Manager
Device Configuration
MyDnsDeviceSer Admin API (CRUD)
vice
DnsService
3. addDnsRecord(ip, fqdn)
Demonstrates one way to MyDnsDeviceMa MySQL
MyDnsElement
inform an external DNS nager
server when an instance
starts. AgentManag
4.Enqueue AddDnsRecord er Queue
Classes shaded blue form a
plugin / service bundle to
integrate an external DNS MyDnsDeviceRes
server. Clients of the ServerResource ource
instance can then use DNS
names to access the 5.API call to Dns Device
instance.

25. Let’s do some myth busting

26. Myth 1
• Network is Layer 2
• Network is Layer 3
• Network confines how I implement my
physical network.

27. Busted
• No. Network is just a concept to represent
where virtual machines can access each other.
– One Network to rule them all, One Network to
define them, One Network to bring them all and
in the cloud bind them.
• What defines a network to be layer 2 or layer
3 is the NetworkGuru.

28. Myth 2
• I must convert my service to run inside
CloudStack’s plugin.
• CloudStack controls the deployment of my
service.

29. Busted
• CloudStack’s plugin interfaces specify the
provisioning events and data associated with
the events.
• Plugin implementations can be client stubs
that call out to your actual service upon
receiving these events.
• If you so desired, plugin implementations can
also expose APIs to configure the plugins.

30. Myth 3
• My service cannot scale independently from
CloudStack.
• My service’s availability is tied together with
CloudStack’s availability.

31. Busted
• CloudStack does not need to know how a
service scales.
– CloudStack does not manage instances of a
service.
– CloudStack does not manage instances of the
same plugin. Plugin code is always a singleton.

32. The future needs you!
Project web site: http://incubator.apache.org/projects/cloudstack.html
Mailing lists:
cloudstack-dev-subscribe@incubator.apache.org
cloudstack-users-subscribe@incubator.apache.org
IRC: #CloudStack on irc.freenode.net

33. Thank You!
Alex Huang
Email: alex.huang@gmail.com
Blog: http://xueyuan.github.com/