The core components of a CloudStack implementation are:Hosts – Hosts are servers from at least one of the supported virtualization providers. CloudStack fully supports hosts from multiple providers, but does not convert VM images from one hypervisor type to another. Depending on the hypervisor, a “host” may be a higher level concept. For example, in XenServer a CloudStack “host” is equivalent to a XenServer resource pool and the “host” entry is the pool master.Primary Storage – Primary storage is the hypervisor level storage containing the deployed VM storage. Primary storage options will vary by hypervisor, and depending upon the hypervisor selected, CloudStack may impose requirements upon it.Cluster – Host groups are combined into Clusters which contain the primary storage options for the Cluster. Primary storage isn’t shared outside of a Cluster. In the case of CloudStack, a Cluster in of itself does not imply modification of any clustering concept within the hypervisor. For example, in XenServer a resource pool is a host to CloudStack, and CloudStack does not create a super set of Cluster functionality for XenServer. Pod -- Host groups are combined first into Clusters and then into Pods. For many customers, a pod represents a high level physical concept like a server rackNetwork – Network is the logical and physical network associated with service offerings. Multiple concurrent network service offerings and topologies can be supported within CloudStackSecondary Storage – Secondary storage is the storage system used for template and ISO management. It also is where snapshot events occur.Zone – A zone is a collection pods to form some level of service availability. While Amazon EC2 defines an availability zone as a data center, CloudStack keeps the concept more abstract allowing cloud operators to have multiple availability zones within a given data center.Management Server Farm – The CloudStack management server farm is a grouping of CentOS/RHEL CloudStack servers forming a web farm, with an underlying MySQL cluster database. The management server farm can manage multiple Zones, and can be virtualized.
Slide 2 (control), Slide 3 network topology (AWS), Slide 4 network topology (multi-tier), Slide 5 Network topology (management network), Slide 5 network topology (MPLS)However, this problem gets a little more complicated when we consider what a cloud operator wants: Control. On the hardware resource side, the cloud operator wants to control which hardware resource is used to provide the cloud infrastructure. They want to say the hypervisor is XenServer, KVM, or VmWare and the load balancer is a Netscaler or F5 and so on. On the end user side, they want to control the who can do what to which virtual entities, place governace rules, and limit the amount of resources a user can consume. This is still fairly straight-forward. We place two sets of APIs: Services API and Resource API. Above the Service API, CloudStack performs Auth and ACL checks, single sign-ons, integration with accounts databases such as ADC and LDAP, and provides checks against limits and governnes rules. Under the Service API is the CloudStack Orchestration Engine. When Orchestration Engine needs to contact a certain hardware, it speaks the Resource API with the abstraction layer implemented by ServerResource. And to ensure that the management server observes this abstraction layer, it can only communicate with ServerResource over a JSON based message bus.
Apache CloudStack Architecture by Alex Huang
Apache CloudStackArchitecture Alex Huang Software Architect, Citrix Systems
Components• Hosts • Servers onto which services will be provisioned VM• Primary Storage Host • VM disk storage Network VM• Cluster Host • A grouping of hosts and their associated storage Primary• Pod Storage • Collection of clusters in the same failure boundary• Network Cluster • Logical network associated with service offerings Secondary Cluster• Secondary Storage Storage • Template, snapshot and ISO storage CloudStack Pod• Zone • Collection of pods, network offerings and secondary storage CloudStack Pod• Management Server Farm • Management and provisioning tasks Zone
Two Types of StoragePrimary Storage• Stores disk volumes for VMs in a cluster• Configured at Cluster-level.• Close to hosts for better performance L3 switch• Cluster have at least one primary storage• Requires high IOPs (can be expensive) Pod 1 L2 switch Secondary Cluster 1 Storage Host 1 PrimarySecondary Storage Host 2 Storage• Stores all Templates, ISOs and Snapshots• Configured at Zone-level• Zone can have one or more secondary storages• High capacity, low cost commodity storage
Deployment Architecture Internet Hypervisor is the basic unitManagementServer Cluster of scale. Zone 1 Cluster consists of one ore more hosts of same L3 hypervisor Pod 1 Pod N All hosts in cluster have L2 Secondary access to shared (primary) …. Storage storage Cluster N Pod is one or more clusters, usually with L2 switches. …. Availability Zone has one or Cluster 1 more pods, has access to secondary storage. Host 1 Primary One or more zones Host 2 Storage represent cloud
Management Server Cluster MS is stateless. MS can be deployed as physical server or VM MySQLUser API Management Single MS node can Server manage up to 10K hosts. Load Balancer Replication Multiple nodes can beAdmin API Management deployed for scale or Server redundancy Replica RHEL 5.4+, Ubuntu 10.0.4, Fedora 16 Infrastructure Resources
The Three C’s of Complexity• Control• Choice• Compliance
Giving Control Brings Complexity Org A Org B • ACL Users • Limits Admin Admin • Governance End User Users UsersAdmin Compute Network Storage VM Net Local Hyper-V Cisco ASA NFS Ware Scaler Disk Xen Oracle F5 iSCSI Swift Server VM Bare Jun. SRX FC HDFS KVM Metal
Guest Virtual Layer-2 Network Guest 1 Virtual Network 10.1.1.0/24 Guest 1 Guest VM 1 10.1.1.2 Public IP Guest 1 Gateway Guest 1 Guest 126.96.36.199 Virtual 10.1.1.1 VM 2 10.1.1.3 188.8.131.52 Router Guest 1 Guest VM 3 10.1.1.4Internet Guest 2 Guest VM 1 10.1.1.2 Public IP Guest 2 Gateway Guest 2 Guest 184.108.40.206 Virtual 10.1.1.1 VM 2 10.1.1.3 220.127.116.11 Router Guest 2 Guest VM 3 10.1.1.4 Guest 2 Virtual Network 10.1.1.0/24
Multi-tier Network Virtual Network Virtual Network Virtual Network 10.1.2.0/24 10.1.3.0/24 10.1.1.0/24 VLAN 1001 VLAN 141 VLAN 100 App VM 10.1.2.31 1 10.1.3.21 Private IP Web VMPublic IP 10.1.1.1 1 10.1.2.21 Juniper 10.1.1.11165.37.141. SRX App VM111 Firewall 10.1.2.24 2 10.1.3.45 Web VM 10.1.1.3 2 10.1.2.18 Public IP Private IP Netscaler 10.1.1.112 65.37.1 Load Web VM 41.112 Balancer 10.1.1.4 3 10.1.2.38 10.1.3.24 DB VM 1 Web VM 10.1.1.5 4 10.1.2.39 DHCP, Virtual DHCP, Virtual DNS Virtual DHCP, DNS, DNS Router User- Router User- Router data data, User- Source data -NAT, VPN Public IP 18.104.22.168
Unified Multi-tier Network Internet IPSec or SSL site-to-site VPN Virtual Router Customer Load Balancer Premises Monitoring VLANVirtual Router Services App• IPAM 10.1.2.31 Web VM 1• DNS 10.1.1.1• LB [intra] VM 1• S-2-S VPN App 10.1.2.24• Static Routes Web VM 2• ACLs 10.1.1.3 VM 2• NAT, PF• FW [ingress & egress] Web DB VM• BGP 10.1.3.24 10.1.1.4 VM 3 1 Web 10.1.1.5 VM 4 Virtual Network Virtual Network Virtual Network 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24 VLAN 100 VLAN 1001 VLAN 141
Other Topologies Dedicated VLAN with DHCP and DNSNo services [Static IPs] User can request specific IP[s] for NIC Guest Virtual Network Guest Virtual Network 10.1.1.0/24 10.1.1.0/24 VLAN 100 VLAN 100 Guest Guest VM 1 10.1.1.1 VM 1 10.1.1.1 Gateway address 10.1.1.1 Guest Guest 10.1.1.3 VM 2 Gateway 10.1.1.3 VM 2 address 10.1.1.1 Guest Guest Core switch 10.1.1.4 VM 3 VM 3 10.1.1.4 Guest Core switch Guest 10.1.1.5 VM 4 10.1.1.5 VM 4 Virtual DHCP, DNS Router User-data
Other TopologiesMPLS Use Case Shared VLAN with DHCP and DNS Guest Virtual Network 10.1.1.0/24 Guest Virtual Network 10.1.1.0/24 VLAN 100 VLAN 100MPLS VLAN 100 Guest Guest VM 1 10.1.1.1 VM 1 10.1.1.100 Gateway address 10.1.1.1 Guest Guest 10.1.1.200 VM 2 Gateway 10.1.1.3 VM 2 address 10.1.1.1 Guest Guest Core switch 10.1.1.101 VM 3 VM 3 10.1.1.4 Guest Core switch Guest 10.1.1.11 VM 4 10.1.1.5 VM 4 5 CS CS DHCP, Virtual DHCP, Virtual Router Router DNS DNS User-data User-data
Layer 3 Networking (Amazon Style) Web DB Web VM VM VM Web DB Security Security Group Group Web Web DB VM VM VM … … … Web Web VM VM
Cloud Other UI CLI Clients Portal Management Server REST API OAM&P API End User API EC2 API Other APIs Pluggable Service API EngineConsole Proxy ACL & Authentication Security AdaptersManagement - Accounts, Domains, and Projects - ACL, limits checking Account Management Connectors Template Access Services API DB Plugin API Deployment Planning HA Orchestration Engine - Drives long running VM Services API Network Gurus Usage operations Calculations - Syncs between resources managed and DB Network Elements Additional - Generates events Services Hypervisor Gurus Cluster Resource Job Alert & Event Database Management Management Management Management Access Message Bus Event Bus Usage Server Resource API Hypervisor Network Storage Image Snapshot Resources Resources Resources Resources Resources
Orchestration Engine• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)• Well defined process steps• Calls Plugin API to execute functionalities that it needs
Plugins• Various ways to add more capability to CloudStack• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module
Anatomy of a Plugin • Can be two jars: server component to be deployed on management server and an optional ServerResource Rest API component to be deployed co- - Optional. Required only if needs to expose located with the resource configuration API to admin. • Server component can implement multiple Plugin APIs to add its featurePlugin API Implementation • Can expose its own API through Pluggable Service so administrators Data Access Layer can configure the plugin • As an example, OVS plugin actually implements both NetworkGuru and ServerResource NetworkElement - Optional. Required if Plugin needs to be co- located with the resource - Implements translation layer to talk to resource - Communicates with server component via JSON
Plugin Interfaces Available• NetworkGuru – Implements various network isolation and ip address technologies• NetworkElement – Facilitate network services on network elements to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes
Separating Data and Control Cloud Management Servers control allData Center 1 Data Center 3 resources, both virtual Managem and physical ent VR VR Server SSVMs deployed to CPVM SSVM SSVM CPVM transfer data between Transfer of zones Templates, ISOs, Snapshots CPVMs deployed to Internet transfer VNC console Data Center 2 traffic VR SSVM VR deployed for traffic into public internet CPVM Management Server is never in the data path
Sequence Flow for VM Creation Kernel End User Security User VM VirtualMac Network Storage Network Job Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling Deploy VM ACL Checks Allocate Entity in CS Allocate VM Allocate NIC Allocate IP Allocate Volume Schedules Deploy Job Returns with job id, VM id Query Job ResultReturns with job status
Sequence Flow for VM Creation Deploymen Server User VM VirtualMac Network Storage Network Network Template tJob Threads Services API Resources Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner Start VM Start User VM Start VM Get a Deployment Plan (Host and StoragePool) Prepare Nics Reserve resources for Nic Notify that Nic is about to be started in network Agent Calls Prepare Volumes Prepare template on Primary Storage Agent Calls Agent Start VM Call Stores job result
Design Goals for CloudStack• Design for complexity – Clear interfaces• Design for scalability – Separate out data path and control paths – Design to maximize the use of database connections• Design against failure – Provide clear boundaries (process and compilation) – Utilize cloud administrator to give guidance
More Information• http://cloudstack.org• Apache mailing lists – firstname.lastname@example.org – email@example.com• Thank you 27
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.