CloudStack Architecture


Published on

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • CloudStack works within multiple enterprise strategies and mandates, as well as supporting multiple cloud strategies from a provider perspective. As an initial step beyond traditional server virtualization, many organizations are looking to private cloud implementations as a means to satisfy flexibility while still retaining control over service delivery. The private cloud may be hosted by the IT organization itself, or sourced from a managed service provider, but the net goals of total control and security without compromising SLAs is achieved.For some organizations, the managed service model is stepped up one level with all resources sourced from a hosted solution. SLA guarantees and security concerns often dictate the types of providers an enterprise will look towards. At the far end of the spectrum are public cloud providers with pay as you go pricing structures and elastic scaling. Since public clouds often abstract details such as network topology, a hybrid cloud strategy allows IT to retain control over key aspects of their operations such as data, while leveraging the benefits of elastic public cloud capacity.
  • The core components of a CloudStack implementation are:Hosts – Hosts are servers from at least one of the supported virtualization providers. CloudStack fully supports hosts from multiple providers, but does not convert VM images from one hypervisor type to another. Depending on the hypervisor, a “host” may be a higher level concept. For example, in XenServer a CloudStack “host” is equivalent to a XenServer resource pool and the “host” entry is the pool master.Primary Storage – Primary storage is the hypervisor level storage containing the deployed VM storage. Primary storage options will vary by hypervisor, and depending upon the hypervisor selected, CloudStack may impose requirements upon it.Cluster – Host groups are combined into Clusters which contain the primary storage options for the Cluster. Primary storage isn’t shared outside of a Cluster. In the case of CloudStack, a Cluster in of itself does not imply modification of any clustering concept within the hypervisor. For example, in XenServer a resource pool is a host to CloudStack, and CloudStack does not create a super set of Cluster functionality for XenServer. Pod -- Host groups are combined first into Clusters and then into Pods. For many customers, a pod represents a high level physical concept like a server rackNetwork – Network is the logical and physical network associated with service offerings. Multiple concurrent network service offerings and topologies can be supported within CloudStackSecondary Storage – Secondary storage is the storage system used for template and ISO management. It also is where snapshot events occur.Zone – A zone is a collection pods to form some level of service availability. While Amazon EC2 defines an availability zone as a data center, CloudStack keeps the concept more abstract allowing cloud operators to have multiple availability zones within a given data center.Management Server Farm – The CloudStack management server farm is a grouping of CentOS/RHEL CloudStack servers forming a web farm, with an underlying MySQL cluster database. The management server farm can manage multiple Zones, and can be virtualized.
  • Primary StoragePrimary storage is used for all active VM storage of both root and data disks. This storage is local to the CloudStack Pod and is directly available to the hypervisors hosts in the pod. The two universally supported connection methods are NFS and iSCSI, and CloudStack manages these connections. Additionally, options exist for FC and local storage, but these options do vary by hypervisor type. New for CloudStack 3.0 is OpenStack Swift integration.Secondary StorageSecondary storage is used for all template, ISO and volume snapshot activities. This storage is local to each CloudStack availability zone and is accessed through the CloudStack secondary storage server. This system VM connects to the underlying secondary storage device using NFS.Templates and ISOsTemplates and ISOs are imported into CloudStack secondary storage through the use of the storage system VM. The import process is through HTTP. ISOs can be defined as being bootable, and templates must be of a file type which matches hypervisors within the zone. CloudStack won’t convert a template from one hypervisor disk format to another.
  • When a user requests a VM instance, there are several steps performed.The user logs in and selects the desired availability zone for their instance, and then selects the desired template from the list of templates available to them. This is the trigger for the provisioning process.Depending on the instance and zone requirements, optional network services such as routing, dhcp and load balancing are provisioned for the zone. If these services are already provisioned, and can be shared by the user, then shared instances are used; otherwise isolated instances of the network services are used.The template representing the root disk of the VM is copied from the secondary storage for the zone to the primary storage for the cluster. CloudStack attempts to localize services for accounts to as few clusters as possible. This is done partly for security reasons, and partly to ensure optimal performance for provisioned services.If the instance requires any data volumes, the data volumes are created on primary storage for the cluster. Note that the storage preferences for the root volume and data volumes may be different resulting in the volumes occupying different primary storage devices within a given cluster. For example, data disks may have attributes which place them on a primary storage device which is continuously backed up while the root volume might be located on local storage.CloudStack then instructs the host to create and start the instance VM
  • When using XenServer, you will first add the XenServer pool master to CloudStack as a host, and CloudStack will transparently add all slave hosts to CloudStack.
  • Limitations: No snapshot because OVM is using raw format for volumeNo system VM because OVM won’t support Debian guestNeed a helper cluster(xenserver/kvm/vmware)Advantage:Oracle provides lots of templates which have Oracle DB frameworks, applications built in, customer can quickly deploy Oracle serviceCreate templateCreate template from root volume of VMStart system VMAdd a helper cluster(XenServer/KVM/Vmware) before creating any OVM VmThe domain router will automatically be created in helper cluster when creating first OVM instanceNo OVM manager and CloudStack mixedOvs-agent will store data in local database on hostSupported OS typeAll Linux/Solaris templates must be from Oracle siteWindows can be installed from ISOOracle Cluster File SystemOracle recommendation solution for using ISCSIUser responsibilitySetup ISCSI device on every hostCreate OCFS2 file system on every deviceCloudStack responsibilityConfigure every ocfs2 nodeAdd/Remove node on demand
  • For KVM, Support is only for RHEL 6 based KVM and Ubuntu 10.04. No other flavors of KVM are supported, including RHEV.
  • vCenter cluster/hostA vCenter cluster is mapped directly to a CloudStack cluster under PodA vCenter cluster for CloudStack can only belong to one vCenter datacenterWhy?vCenter Datastore used by vCenter cluster is at scope of vCenter datacentervCenter vSwitch used by vCenter cluster is at scope of vCenter datacenterSharing vCenter datacenter resource outside of CloudStack will be problematicSystem VM bootstrapFirst generation is done by CloudStack management serverSecond/beyond generations is done through a running SSVMSSVM (Secondary Storage VM)SSVM for template processingSSVM for VMware volume/snapshot/template operationCommand delegationSystem VM, extension of CloudStack management serverResource manager can be running in context of a system VMCommand delegation in CloudStack management serverSnapshotsCloudStack snapshot is taken at volume basisSnapshot in vCenter is take at VM basisFill the gapTake a VM snapshot, if it is for a detached volume in CloudStack, create a worker VMParse VM snapshot meta data, build up disk chain information at volume basisCreate intermediate VM on top of a selected disk chainExport VM (full backup) to secondary storageCleanupsvCenter vSwitchvSwitch setup is done through vCenterNIC-bonding is done through vCenterCloudStack creates networks (portgroups) dynamicallyCloudStack propagates networks across clusterWhy? To support independent VM live migration both in CloudStack and vCenterDefault vSwitch portsNot enough, usually needs to extend
  • Network OfferingsThe administrator starts off with deciding the network offerings they want to provide throughout their entire cloud offering. Network Offerings group together a set of network services such as firewall, dhcp, dns, etc.Network Offerings allow specific network service providers to be specified.Network Offerings can be tagged to specifically choose the underlying network.Network Offerings have the following states: Disabled, Enabled, Inactive.  All Network Offerings are created in the Disabled state.  Once a network offering has been configured to the correct stateCertain Network Offerings are for used by the system only.  This means end users cannot see them.Network Offerings can be updated to enable/disable services and providers.  Once that is done, it is up to the administrator to reprogram all of the networks that are based on that network offering.Network Offerings tags cannot be updated.  However, the tags on the physical networks can be updated and deleted.CloudStack is deployed with three default network offerings for the end users, virtual network offering and shared network offering without security group and a shared network offering with security group.
  • For latest information:
  • CloudStack Architecture

    1. 1. CloudStack Architecture 4/29/2012 Chiradeep Vittal Alex Huang
    2. 2. Outline• Overview of CloudStack• Problem Definition• Feature set overview• System VMs• System Architecture & Context• Component View
    3. 3. What is CloudStack? • Secure, multi-tenant cloud orchestration platform – Turnkey platform for delivering IaaS clouds Build your cloud the way the – Hypervisor agnosticworld’s most successful clouds are built – Scalable, secure and open – Open source, open standards – Deploys on premise or as a hosted solution • Deliver cloud services faster and cheaper
    4. 4. CloudStack Supports Multiple Cloud Strategies Private Clouds Public Clouds On-premise Hosted Multi-tenant Enterprise Cloud Enterprise Cloud Public Cloud • Dedicated • Dedicated • Mix of shared and resources resources dedicated • Security & total • Security resources control • SLA bound • Elastic scaling • Internal network • 3rd party owned • Pay as you go • Managed by and operated • Public internet, Enterprise or 3rd VPN access party
    5. 5. CloudStack Provides On-demand Access to Infrastructure Through a Self-Service Portal Org A Org B Users Admin AdminEnd User Users Users Compute Network StorageAdmin
    6. 6. Open Flexible PlatformCompute Hypervisor XenServer VMware Oracle VM KVM Bare metalStorage Block & Object Fiber Local Disk iSCSI NFS Swift Channel Primary Storage Secondary StorageNetwork Network & Network Services Network Load Isolation Firewall VPN Type balancer
    7. 7. Problem Definition• Offer a scalable, flexible, manageable IAAS platform that follows established cloud computing paradigms• IAAS – Orchestrate physical and virtual resources to offer self-service infrastructure provisioning and monitoring• Scalable – 1 -> N hypervisors / VMs / virtual resources – 1 -> N end users• Flexible – Handle new physical resource types • Hypervisors, storage, networking – Add new APIs – Add new services – Add new network models
    8. 8. Problem Definition (contd)• Manageable – Hide complexity of underlying resources – Rich functional end-user and admin UI – Admin API to automate operations – Easy install, upgrade for small -> large clouds – Simple scaling, automated resilience• Established Paradigms – EC2 –inspired • Semantic variations based on cloud provider needs, hypervisor capabilities
    9. 9. End-User ExperienceSelf-service Portal to Create &Manage VMs
    10. 10. Create Custom Virtual Machines via Service Offerings Select Operating System • Windows, Linux Select Compute Offering • CPU & RAM Select Disk Offering • Volume Size Select Network Offering • Network & Services Create VM
    11. 11. Dashboard Provides Overview of Consumed Resources• Running, Stopped & Total VMs• Public IPs• Private networks• Latest Events
    12. 12. Virtual Machine Management Users ChangeVM Operations Console Access VM Status Service Offering Start • CPU Utilized 2 CPUs 4 CPUs Stop 1 GB 4 GB • Network Read RAM RAM Restart • Network Writes 20 GB 200 GB Destroy 20 Mbps 100 Mbps
    13. 13. Volume & Snapshot Management VM 1 Add / Delete Volumes Volume Create Templates Volume Template from Volumes Hourly Weekly Schedule Now Snapshots Daily Monthly …. View Snapshot History
    14. 14. Network & Network Services• Create Networks and attach VMs• Acquire public IP address for NAT & load balancing• Control traffic to VM using ingress and egress firewall rules• Set up rules to load balance traffic between VMs
    15. 15. CloudStack Deployment Architecture CloudStac Inter k net  Hypervisor is the basic unit of Manage scale. mentZone 1 Server  Cluster consists of one ore more hosts of same hypervisor L3 core  All hosts in cluster have access to shared (primary) storagePod 1 Access Layer Pod N Secondary  Pod is one or more clusters, …. Storage usually with L2 switches. Cluster N  Availability Zone has one or more pods, has access to …. secondary storage.  One or more zones represent Cluster 1 cloud Host 1 Primary Storage Host 2
    16. 16. CloudStack Cloud Architecture CloudData Center 1 Data Center 2 Data Center 2 Data Center 3 Zone 2 Zone 2 Zone1 Zone 3 Zone 4 3 Zone CloudStack Cloud can have one or more Availability Zones (AZ). Data Center 2 Data Center 2 Data Center 2 Zone 2 Zone 2 ZoneZone 3 2 Zone 3 Zone 3 - Do Not Distribute
    17. 17. Management Server Managing Multiple Zones CloudData Center 1 Data Center 2  Single Management Server can Data Center 2 Managem Data Center 3 manage multiple zones ent Server Zone 2  Zones can be geographically Zone 2 distributed but low latency links are Zone 3 expected for better performance Zone1 Zone 4 3 Zone  Single MS node can manage up to 5K hosts.  Multiple MS nodes can be deployed Data Center 2 as cluster for scale or redundancy Data Center 2 Data Center 2 Zone 2 Zone 2 Zone Zone 3 2 Zone 3 Zone 3 - Do Not Distribute
    18. 18. Management Server Deployment Architecture Single-node Multi-node Deployment Deployment Managem ent ServerUser API User API Managem Managem ent MySQL Load ent Server DB Balancer ServerAdmin API Admin API Managem MySQL ent DB Server Back Up Replication DB  MS is stateless. MS can be deployed as physical server or VM  Single MS node can manage up to Infrastructure Infrastructure 10K hosts. Multiple nodes can be Resources Resources deployed for scale or redundancy  Commercial: RHEL 5.4+; FOSS: Ubuntu 10.0.4, Fedora 16 - Do Not Distribute
    19. 19. CloudStack Storage Primary Storage• Configured at Cluster-level. Close to hosts for better performance L3 switch• Stores all disk volumes for VMs in a cluster• Cluster can have one or more primary storages Pod 1 L2 switch• Local disk, iSCSI, FC or NFS Secondary Cluster 1 Storage Host 1 Primary Secondary Storage Storage Host 2• Configured at Zone-level• Stores all Templates, ISOs and Snapshots• Zone can have one or more secondary storages• NFS, OpenStack Swift - Do Not Distribute
    20. 20. Core CloudStack Components VM• Hosts • Servers onto which services will be provisioned Host VM• Primary Storage Network • VM storage Host• Cluster Primary • A grouping of hosts and their associated storage Storage• Pod • Collection of clusters Cluster• Network Secondary • Logical network associated with service offerings Storage Cluster• Secondary Storage • Template, snapshot and ISO storage CloudStack Pod• Zone • Collection of pods, network offerings and secondary storage CloudStack Pod• Management Server Farm • Responsible for all management and provisioning tasks Zone
    21. 21. Understanding the Role of Storage and Templates• Primary Storage • Cluster level storage for VMs Host • Connected directly to hosts • NFS, iSCSI, FC and Local Host• Secondary Storage Primary Storage • Zone level storage for template, ISOs and Cluster snapshots • NFS or OpenStack Swift via CloudStack Pod System VM• Templates and ISOs • Imported into CloudStack • Can be private or public Secondary Storage Zone Template
    22. 22. Provisioning Process1. User Requests Instance VM2. Provision Optional Network Services Host3. Copy instance template from Host secondary storage to primary storage Primary Storage on appropriate cluster Cluster4. Create any requested data volumes on primary storage for the cluster Pod5. Create instance Template6. Start instance Secondary Storage Zone
    23. 23. Citrix XenServer CloudStack• Integrates directly with XenServer Manager Pool Master• Snapshots at host level XenServer Pool Master Host• System VM control channel at host level XenServer Host• Network management is host level XenServer Host XenServer Host XenServer Host XenServer Resource Pool
    24. 24. Oracle VM CloudStack• Integrates with ovs-agent Manager• Snapshots at host level OVS Agent• System VM control channel at OVM Host host level OVS Agent• Network management is host OVM Host level OVS Agent• Does not use OVM Manager OVM Host• All templates must be from Oracle OVS Agent• CloudStack configures ocfs2 nodes OVM Host• Requires “helper” cluster
    25. 25. RedHat Enterprise Linux (KVM)• Integrates with libvirt using Cloud CloudStack Agent Manager• Snapshots at host level Cloud Agent• System VM control channel at host level Libvirt KVM Host• Network management is host level Cloud Agent• Only RHEL 6, not RHEV Libvirt • Also supports Ubuntu 10.04 KVM Host
    26. 26. VMware vSphere CloudStack• Integration through vCenter Manager vSphere Host• System VM control channel via vCenter CloudStack private network vSphere Host• Snapshot and volume vSphere Cluster management via Secondary Storage VM vSphere Host• Networking via vSphere vSwitch vSphere Host vSphere Host vSphere Cluster Data Center
    27. 27. Management Server Interaction with Hypervisors Managem ent Server XAPI HTTP vCenter Agent Agent XenServer KVM OVM ESX• XS 5.6, 5.6FP1, 5.6 SP2, 6.0 • ESX 4.1, 5.0 (coming) • RHEL 6.0, 6.1, 6.2 (coming) • OVM 2.2• Incremental Snapshots • Full Snapshots • Full Snapshots (not live) • No Snapshots• VHD • VMDK • QCOW2 • RAW• NFS, iSCSI, FC & Local disk • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS & iSCSi• Storage over-provisioning: • Storage over-provisioning: • Storage over-provisioning: • No storage over- NFS NFS, iSCSI NFS provisioning
    28. 28. Multi-tenancy & Account Management Cloud Resources Domain VMs, IPs, Snapshots… • Domain is a unit of Org A isolation that represents Admin a customer org, business unit or a reseller Domain Reseller A • Domain can have Admin Resources arbitrary levels of sub- Sub-Domain VMs, IPs, Org C Snapshots… domains Admin • A Domain can have one Account or more accounts Group A • An Account represents Account one or more users and is Group B the basic unit of isolation User 1 • Admin can limit resources at the Account User 2 or Domain levels
    29. 29. Physical Network Operations Users Admin and Cloud API CloudStack Mgmt Server Cluster Router MySQL Load Balancer Availability Zone L3 Core Switch Access LayerSwitches Secondary Servers … … … … … Storage Pod 1 Pod 2 Pod 3 Pod N
    30. 30. Layer 3 cloud networking Web DB Web VM VM VM Web DB Security Security Group Group Web Web DB VM VM VM… … … Web Web VM VM
    31. 31. Guest Networks with L3 isolationPublic Public IP Guest GuestInternet address 1 VM address 1 Guest Pod 1 L2 Guest Switch 2 VM address 1 Guest Guest 1 VM address L3 Core Pod 2 L2 Switch … 2 Switch Guest Guest Load 10.1.16. 2 VM address Pod 3 L2 Balancer 1 2 Switch Guest Guest 2 VM address 3 … Guest 1 VM Guest address 3 Guest Guest 1 VM address 4
    32. 32. Virtual Networks (L2 isolation) Core (L3) Network Pod K Pod M Pod N Access Switch(es) V Hypervisor V V Hypervisor R … CLUSTER 1 Hypervisor 1 RVM Traffic … Hypervisor 8Public Traffic … CLUSTER 4 V V Hypervisor N V Tenant VM Hypervisor N+1 V R Tenant Virtual Router
    33. 33. Guest virtual layer-2 network Guest Virtual Network Public Public IP Guest Gateway Guest Network address 1 VM address address 1 Guest 1 Guest Guest Public Virtual 1 VM address Internet Router 2 NAT Guest Guest DHCP 1 VM address Load 3 Balancing VPN Guest Guest 1 VM address 4 Guest Virtual Network Public IP address Gateway Guest Guest address 2 VM address 1 Guest 2 Guest Guest Virtual 2 VM address Router 2 NAT Guest Guest DHCP 2 VM address Load 3 Balancing VPN
    34. 34. Layer-2 Guest Virtual Network CS Virtual Router provides Network Services External Devices provide Network Services Guest Virtual Network Guest Virtual Network VLAN 100 VLAN 100Public PublicNetwork/Intern Network/Internet Guest et Guest Public IP Private IP VM 1 VM 1 Gateway JuniperPublic IP 1 SRX address65.37.141.11 CS Firewall Guest Guest Virtual VM 2 VM 2 Router Public IP Private IP DHCP, DNS 65.37.141. NetScaler NAT Guest 112 Load Guest Load Balancing VM 3 Blancer VM 3 VPN Guest Guest VM 4 VM 4 CS DHCP, Virtual Router DNS
    35. 35. Layer-3 Guest NetworkNetwork Services Managed Externally Network Services Managed by CS Public Network Security Group Security Group Public 1 1 Network/Intern et Guest VM Guest VM 1 1 NetScal L3 Guest VM Guest VM er switc 2 2 Load h Blancer EIP, ELB Guest VM Guest VM 3 3 Guest VM Guest VM 4 4 CS CS Virtual Virtual DHCP, Security Group DHCP, Security Group Router Router DNS 2 DNS 2
    36. 36. Network Offerings• Cloud provider defines the feature set for guest networks• Toggle features or service levels – Security groups on/off – Load balancer on/off – Load balancer software/hardware – VPN, firewall, port forwarding• User chooses network offering when creating network• Enables upgrade between network offerings• Default offerings built-in – For classic CloudStack networking
    37. 37. CloudStack System VMs• System VMs optimize and scale the datapath on behalf of CloudStack – Stateless, can be destroyed and recreated from database state – Highly Available – Communicates with Management Server over management network – Usually have 3 interfaces: control, guest and public• Console Proxy VM – Provides AJAX-style HTTP-only console viewer – Grabs VNC output from hypervisor – Scales out (more spawned) as load increases – Java-based server Communicates with MS over message bus• Secondary Storage VM – Provides image (template) management services – Download from HTTP file share or Swift – Copy between zones – Scale out to handle multiple NFS mounts – Java-based server communicates with MS over message bus
    38. 38. CloudStack System VMs• Virtual Router VM – Provides multiple network services – IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN – User-data, Meta-data, SSH keys and password change server – Redundancy via VRRP – MS configures VR over SSH • Proxied via the hypervisor on XS and KVM
    39. 39. Virtual Router Information (applies to all Sys. VMs)• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu, dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non- standard port (3922). SSH logins only using keys (keys are generated at install time and are unique for every customer)• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved security and speed• Latest version of jre from Sun/Oracle ensures improved security and speed
    40. 40. System VM contd• SSH keys and password are unique to cloud installation• Code can be patched by restarting system vm – Mounts a special ISO file with latest code at boot – If ISO contents differ, patch and reboot• Same system vm works on XS, KVM, VMWare – Bootstrap step for the cloud is to install the template for this system vm• Ready to be re-purposed for other specialized tasks
    41. 41. Architecture Components Service Management (Billing, Metering, Accounts, etc.) User Interface Image LibrariesMaintenance and Provisioning Administrato Operation, Administration, End User Console Application Catalog r Integration API Custom Templates Developer API Operating System ISOs Amazon OpenStack Custom Availability and Security Backup Load Balancing High Availability Monitoring Resource Management Dynamic Workload Management Virtualization Layer Compute Network Storage
    42. 42. Interactions OVM Cluster Primary Storage vcenter Monitoring Primary CS API vSphere Cluster Storage End User UI Primary XS Cluster Storage Admin UI Clustered CloudStack XAPI Domain CS Admin & CloudStack CloudStack Admin End-user API Primary UI Management JSON KVM Cluster Storage Server NetConf Juniper SRXCloud user Nitro API{API client (Fog/etc)} VNC JSON ec2 API JSON Netscaler Cloud user Console Console {ec2 API client } Proxy VM Proxy VM NFS MySQL Server {Proxied} SSH Sec. Storage NFS NFS Sec. Storage VM Ajax HTTPS VM Console Router VM HTTP (Template Download) Router VM HTTP (Template Copy) Router VM Cloud user HTTP (Swift)
    43. 43. Inside a Management Server Plugins cmd.execute() PluginsCloudStack Commands Async Plugins API API Job Ser Queu Serv vlet e ices Kernel Responses API Mgr Agent API Messa Resources (Cmds) ge Bus Local Or Remote Agent Manager Hypervisor Network Native Device APIs API MySQL