CloudStack Architecture

42,509 views

Published on

Published in: Technology

CloudStack Architecture

  1. 1. CloudStack Architecture 4/29/2012 Chiradeep Vittal Alex Huang
  2. 2. Outline• Overview of CloudStack• Problem Definition• Feature set overview• System VMs• System Architecture & Context• Component View
  3. 3. What is CloudStack? • Secure, multi-tenant cloud orchestration platform – Turnkey platform for delivering IaaS clouds Build your cloud the way the – Hypervisor agnosticworld’s most successful clouds are built – Scalable, secure and open – Open source, open standards – Deploys on premise or as a hosted solution • Deliver cloud services faster and cheaper
  4. 4. CloudStack Supports Multiple Cloud Strategies Private Clouds Public Clouds On-premise Hosted Multi-tenant Enterprise Cloud Enterprise Cloud Public Cloud • Dedicated • Dedicated • Mix of shared and resources resources dedicated • Security & total • Security resources control • SLA bound • Elastic scaling • Internal network • 3rd party owned • Pay as you go • Managed by and operated • Public internet, Enterprise or 3rd VPN access party
  5. 5. CloudStack Provides On-demand Access to Infrastructure Through a Self-Service Portal Org A Org B Users Admin AdminEnd User Users Users Compute Network StorageAdmin
  6. 6. Open Flexible PlatformCompute Hypervisor XenServer VMware Oracle VM KVM Bare metalStorage Block & Object Fiber Local Disk iSCSI NFS Swift Channel Primary Storage Secondary StorageNetwork Network & Network Services Network Load Isolation Firewall VPN Type balancer
  7. 7. Problem Definition• Offer a scalable, flexible, manageable IAAS platform that follows established cloud computing paradigms• IAAS – Orchestrate physical and virtual resources to offer self-service infrastructure provisioning and monitoring• Scalable – 1 -> N hypervisors / VMs / virtual resources – 1 -> N end users• Flexible – Handle new physical resource types • Hypervisors, storage, networking – Add new APIs – Add new services – Add new network models
  8. 8. Problem Definition (contd)• Manageable – Hide complexity of underlying resources – Rich functional end-user and admin UI – Admin API to automate operations – Easy install, upgrade for small -> large clouds – Simple scaling, automated resilience• Established Paradigms – EC2 –inspired • Semantic variations based on cloud provider needs, hypervisor capabilities
  9. 9. End-User ExperienceSelf-service Portal to Create &Manage VMs
  10. 10. Create Custom Virtual Machines via Service Offerings Select Operating System • Windows, Linux Select Compute Offering • CPU & RAM Select Disk Offering • Volume Size Select Network Offering • Network & Services Create VM
  11. 11. Dashboard Provides Overview of Consumed Resources• Running, Stopped & Total VMs• Public IPs• Private networks• Latest Events
  12. 12. Virtual Machine Management Users ChangeVM Operations Console Access VM Status Service Offering Start • CPU Utilized 2 CPUs 4 CPUs Stop 1 GB 4 GB • Network Read RAM RAM Restart • Network Writes 20 GB 200 GB Destroy 20 Mbps 100 Mbps
  13. 13. Volume & Snapshot Management VM 1 Add / Delete Volumes Volume Create Templates Volume Template from Volumes Hourly Weekly Schedule Now Snapshots Daily Monthly …. View Snapshot History
  14. 14. Network & Network Services• Create Networks and attach VMs• Acquire public IP address for NAT & load balancing• Control traffic to VM using ingress and egress firewall rules• Set up rules to load balance traffic between VMs
  15. 15. CloudStack Deployment Architecture CloudStac Inter k net  Hypervisor is the basic unit of Manage scale. mentZone 1 Server  Cluster consists of one ore more hosts of same hypervisor L3 core  All hosts in cluster have access to shared (primary) storagePod 1 Access Layer Pod N Secondary  Pod is one or more clusters, …. Storage usually with L2 switches. Cluster N  Availability Zone has one or more pods, has access to …. secondary storage.  One or more zones represent Cluster 1 cloud Host 1 Primary Storage Host 2
  16. 16. CloudStack Cloud Architecture CloudData Center 1 Data Center 2 Data Center 2 Data Center 3 Zone 2 Zone 2 Zone1 Zone 3 Zone 4 3 Zone CloudStack Cloud can have one or more Availability Zones (AZ). Data Center 2 Data Center 2 Data Center 2 Zone 2 Zone 2 ZoneZone 3 2 Zone 3 Zone 3 - Do Not Distribute
  17. 17. Management Server Managing Multiple Zones CloudData Center 1 Data Center 2  Single Management Server can Data Center 2 Managem Data Center 3 manage multiple zones ent Server Zone 2  Zones can be geographically Zone 2 distributed but low latency links are Zone 3 expected for better performance Zone1 Zone 4 3 Zone  Single MS node can manage up to 5K hosts.  Multiple MS nodes can be deployed Data Center 2 as cluster for scale or redundancy Data Center 2 Data Center 2 Zone 2 Zone 2 Zone Zone 3 2 Zone 3 Zone 3 - Do Not Distribute
  18. 18. Management Server Deployment Architecture Single-node Multi-node Deployment Deployment Managem ent ServerUser API User API Managem Managem ent MySQL Load ent Server DB Balancer ServerAdmin API Admin API Managem MySQL ent DB Server Back Up Replication DB  MS is stateless. MS can be deployed as physical server or VM  Single MS node can manage up to Infrastructure Infrastructure 10K hosts. Multiple nodes can be Resources Resources deployed for scale or redundancy  Commercial: RHEL 5.4+; FOSS: Ubuntu 10.0.4, Fedora 16 - Do Not Distribute
  19. 19. CloudStack Storage Primary Storage• Configured at Cluster-level. Close to hosts for better performance L3 switch• Stores all disk volumes for VMs in a cluster• Cluster can have one or more primary storages Pod 1 L2 switch• Local disk, iSCSI, FC or NFS Secondary Cluster 1 Storage Host 1 Primary Secondary Storage Storage Host 2• Configured at Zone-level• Stores all Templates, ISOs and Snapshots• Zone can have one or more secondary storages• NFS, OpenStack Swift - Do Not Distribute
  20. 20. Core CloudStack Components VM• Hosts • Servers onto which services will be provisioned Host VM• Primary Storage Network • VM storage Host• Cluster Primary • A grouping of hosts and their associated storage Storage• Pod • Collection of clusters Cluster• Network Secondary • Logical network associated with service offerings Storage Cluster• Secondary Storage • Template, snapshot and ISO storage CloudStack Pod• Zone • Collection of pods, network offerings and secondary storage CloudStack Pod• Management Server Farm • Responsible for all management and provisioning tasks Zone
  21. 21. Understanding the Role of Storage and Templates• Primary Storage • Cluster level storage for VMs Host • Connected directly to hosts • NFS, iSCSI, FC and Local Host• Secondary Storage Primary Storage • Zone level storage for template, ISOs and Cluster snapshots • NFS or OpenStack Swift via CloudStack Pod System VM• Templates and ISOs • Imported into CloudStack • Can be private or public Secondary Storage Zone Template
  22. 22. Provisioning Process1. User Requests Instance VM2. Provision Optional Network Services Host3. Copy instance template from Host secondary storage to primary storage Primary Storage on appropriate cluster Cluster4. Create any requested data volumes on primary storage for the cluster Pod5. Create instance Template6. Start instance Secondary Storage Zone
  23. 23. Citrix XenServer CloudStack• Integrates directly with XenServer Manager Pool Master• Snapshots at host level XenServer Pool Master Host• System VM control channel at host level XenServer Host• Network management is host level XenServer Host XenServer Host XenServer Host XenServer Resource Pool
  24. 24. Oracle VM CloudStack• Integrates with ovs-agent Manager• Snapshots at host level OVS Agent• System VM control channel at OVM Host host level OVS Agent• Network management is host OVM Host level OVS Agent• Does not use OVM Manager OVM Host• All templates must be from Oracle OVS Agent• CloudStack configures ocfs2 nodes OVM Host• Requires “helper” cluster
  25. 25. RedHat Enterprise Linux (KVM)• Integrates with libvirt using Cloud CloudStack Agent Manager• Snapshots at host level Cloud Agent• System VM control channel at host level Libvirt KVM Host• Network management is host level Cloud Agent• Only RHEL 6, not RHEV Libvirt • Also supports Ubuntu 10.04 KVM Host
  26. 26. VMware vSphere CloudStack• Integration through vCenter Manager vSphere Host• System VM control channel via vCenter CloudStack private network vSphere Host• Snapshot and volume vSphere Cluster management via Secondary Storage VM vSphere Host• Networking via vSphere vSwitch vSphere Host vSphere Host vSphere Cluster Data Center
  27. 27. Management Server Interaction with Hypervisors Managem ent Server XAPI HTTP vCenter Agent Agent XenServer KVM OVM ESX• XS 5.6, 5.6FP1, 5.6 SP2, 6.0 • ESX 4.1, 5.0 (coming) • RHEL 6.0, 6.1, 6.2 (coming) • OVM 2.2• Incremental Snapshots • Full Snapshots • Full Snapshots (not live) • No Snapshots• VHD • VMDK • QCOW2 • RAW• NFS, iSCSI, FC & Local disk • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS & iSCSi• Storage over-provisioning: • Storage over-provisioning: • Storage over-provisioning: • No storage over- NFS NFS, iSCSI NFS provisioning
  28. 28. Multi-tenancy & Account Management Cloud Resources Domain VMs, IPs, Snapshots… • Domain is a unit of Org A isolation that represents Admin a customer org, business unit or a reseller Domain Reseller A • Domain can have Admin Resources arbitrary levels of sub- Sub-Domain VMs, IPs, Org C Snapshots… domains Admin • A Domain can have one Account or more accounts Group A • An Account represents Account one or more users and is Group B the basic unit of isolation User 1 • Admin can limit resources at the Account User 2 or Domain levels
  29. 29. Physical Network Operations Users Admin and Cloud API CloudStack Mgmt Server Cluster Router MySQL Load Balancer Availability Zone L3 Core Switch Access LayerSwitches Secondary Servers … … … … … Storage Pod 1 Pod 2 Pod 3 Pod N
  30. 30. Layer 3 cloud networking Web DB Web VM VM VM Web DB Security Security Group Group Web Web DB VM VM VM… … … Web Web VM VM
  31. 31. Guest Networks with L3 isolationPublic Public IP Guest GuestInternet address 1 VM address 65.37.141.11 1 10.1.0.2 10.1.0.1 Guest 65.37.141.24 Pod 1 L2 Guest 65.37.141.36 Switch 2 VM address 65.37.141.80 1 10.1.0.3 Guest Guest 1 VM address L3 Core Pod 2 L2 Switch 10.1.8.1 … 2 10.1.0.4 Switch Guest Guest Load 10.1.16. 2 VM address Pod 3 L2 Balancer 1 2 10.1.16.12 Switch Guest Guest 2 VM address 3 10.1.16.21 … Guest 1 VM Guest address 3 10.1.16.47 Guest Guest 1 VM address 4 10.1.16.85
  32. 32. Virtual Networks (L2 isolation) Core (L3) Network Pod K Pod M Pod N Access Switch(es) V Hypervisor V V Hypervisor R … CLUSTER 1 Hypervisor 1 RVM Traffic … Hypervisor 8Public Traffic … CLUSTER 4 V V Hypervisor N V Tenant VM Hypervisor N+1 V R Tenant Virtual Router
  33. 33. Guest virtual layer-2 network Guest Virtual Network 10.1.1.0/24 Public Public IP Guest Gateway Guest Network address 1 VM address address 65.37.141.11 1 10.1.1.1 10.1.1.2 65.37.141.36 Guest 1 Guest Guest Public Virtual 1 VM address Internet Router 2 10.1.1.3 NAT Guest Guest DHCP 1 VM address Load 3 10.1.1.4 Balancing VPN Guest Guest 1 VM address 4 10.1.1.5 Guest Virtual Network Public IP 10.1.1.0/24 address Gateway Guest Guest 65.37.141.24 address 2 VM address 65.37.141.80 10.1.1.1 1 10.1.1.2 Guest 2 Guest Guest Virtual 2 VM address Router 2 10.1.1.3 NAT Guest Guest DHCP 2 VM address Load 3 10.1.1.4 Balancing VPN
  34. 34. Layer-2 Guest Virtual Network CS Virtual Router provides Network Services External Devices provide Network Services Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8 VLAN 100 VLAN 100Public PublicNetwork/Intern Network/Internet Guest et Guest Public IP Private IP 10.1.1.1 10.1.1.1 VM 1 10.1.1.111 VM 1 Gateway 65.37.141.11 JuniperPublic IP 1 SRX address65.37.141.11 CS Firewall 10.1.1.1 Guest Guest Virtual 10.1.1.3 VM 2 10.1.1.3 VM 2 Router Public IP Private IP DHCP, DNS 65.37.141. NetScaler 10.1.1.112 NAT Guest 112 Load Guest Load Balancing 10.1.1.4 VM 3 Blancer VM 3 10.1.1.4 VPN Guest Guest 10.1.1.5 VM 4 10.1.1.5 VM 4 CS DHCP, Virtual Router DNS
  35. 35. Layer-3 Guest NetworkNetwork Services Managed Externally Network Services Managed by CS Public Network 65.11.0.0/16 Security Group Security Group Public 1 1 Network/Intern 10.1.2.3 65.11.1.2 et Guest VM Guest VM 1 1 65.11.1.2 NetScal L3 10.2.12.4 65.11.1.3 Guest VM Guest VM er switc 65.11.1.3 2 2 Load h 65.11.1.4 Blancer EIP, ELB 10.5.2.99 65.11.1.4 Guest VM Guest VM 3 3 10.1.2.18 65.11.1.5 Guest VM Guest VM 4 4 CS CS Virtual Virtual DHCP, Security Group DHCP, Security Group Router Router DNS 2 DNS 2
  36. 36. Network Offerings• Cloud provider defines the feature set for guest networks• Toggle features or service levels – Security groups on/off – Load balancer on/off – Load balancer software/hardware – VPN, firewall, port forwarding• User chooses network offering when creating network• Enables upgrade between network offerings• Default offerings built-in – For classic CloudStack networking
  37. 37. CloudStack System VMs• System VMs optimize and scale the datapath on behalf of CloudStack – Stateless, can be destroyed and recreated from database state – Highly Available – Communicates with Management Server over management network – Usually have 3 interfaces: control, guest and public• Console Proxy VM – Provides AJAX-style HTTP-only console viewer – Grabs VNC output from hypervisor – Scales out (more spawned) as load increases – Java-based server Communicates with MS over message bus• Secondary Storage VM – Provides image (template) management services – Download from HTTP file share or Swift – Copy between zones – Scale out to handle multiple NFS mounts – Java-based server communicates with MS over message bus
  38. 38. CloudStack System VMs• Virtual Router VM – Provides multiple network services – IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN – User-data, Meta-data, SSH keys and password change server – Redundancy via VRRP – MS configures VR over SSH • Proxied via the hypervisor on XS and KVM
  39. 39. Virtual Router Information (applies to all Sys. VMs)• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu, dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non- standard port (3922). SSH logins only using keys (keys are generated at install time and are unique for every customer)• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved security and speed• Latest version of jre from Sun/Oracle ensures improved security and speed
  40. 40. System VM contd• SSH keys and password are unique to cloud installation• Code can be patched by restarting system vm – Mounts a special ISO file with latest code at boot – If ISO contents differ, patch and reboot• Same system vm works on XS, KVM, VMWare – Bootstrap step for the cloud is to install the template for this system vm• Ready to be re-purposed for other specialized tasks
  41. 41. Architecture Components Service Management (Billing, Metering, Accounts, etc.) User Interface Image LibrariesMaintenance and Provisioning Administrato Operation, Administration, End User Console Application Catalog r Integration API Custom Templates Developer API Operating System ISOs Amazon OpenStack Custom Availability and Security Backup Load Balancing High Availability Monitoring Resource Management Dynamic Workload Management Virtualization Layer Compute Network Storage
  42. 42. Interactions OVM Cluster Primary Storage vcenter Monitoring Primary CS API vSphere Cluster Storage End User UI Primary XS Cluster Storage Admin UI Clustered CloudStack XAPI Domain CS Admin & CloudStack CloudStack Admin End-user API Primary UI Management JSON KVM Cluster Storage Server NetConf Juniper SRXCloud user Nitro API{API client (Fog/etc)} VNC JSON ec2 API JSON Netscaler Cloud user Console Console {ec2 API client } Proxy VM Proxy VM NFS MySQL Server {Proxied} SSH Sec. Storage NFS NFS Sec. Storage VM Ajax HTTPS VM Console Router VM HTTP (Template Download) Router VM HTTP (Template Copy) Router VM Cloud user HTTP (Swift)
  43. 43. Inside a Management Server Plugins cmd.execute() PluginsCloudStack Commands Async Plugins API API Job Ser Queu Serv vlet e ices Kernel Responses API Mgr Agent API Messa Resources (Cmds) ge Bus Local Or Remote Agent Manager Hypervisor Network Native Device APIs API MySQL

×