Your SlideShare is downloading. ×
CloudStack Overview    Alan Chu@ TCloud Computing
Outline•   Overview of CloudStack•   Problem Definition•   Feature set overview•   Network•   Storage
What is CloudStack                                 • Multi-tenant cloud                                   orchestration pl...
History of Apache CloudStack• 2009: Cloud.com, 100% proprietary• 2010: Cloud.com, open core, GPL v3• 2011: Citrix Systems,...
CloudStack Supports Multiple Cloud Strategies        Private Clouds                          Public Clouds           On-pr...
CloudStack Provides On-demand Access             Org A           Org B                                             Users  ...
Problem Definition• Offer a scalable, flexible, manageable IaaS platform that  follows established cloud computing paradig...
Problem Definition (cntd)• Manageable  –   Hide complexity of underlying resources  –   Rich functional end-user and admin...
Feature Set Overview
Open Flexible PlatformCompute   Hypervisor             XenServer              VMware              KVM               Bare m...
Service Offering
Create Custom Virtual Machines via Service Offerings                                     Select Operating System          ...
Dashboard Provides Overview of Consumed Resources•    Running, Stopped &     Total VMs•    Public IPs•    Private networks...
Virtual Machine Management                            Users                                                              C...
Volume & Snaphost Management                         VM 1        Add / Delete          Volumes                            ...
Network & Network Services•   Create Networks•   Acquire public IP address for NAT &    load balancing•   Control traffic ...
CloudStack Deployment Architecture                                     Internet            Hypervisor is the basic unit o...
Management Server Managing Multiple Zones            CloudData Center 1                Data Center 2                      ...
Management Server Managing Multiple Zones            CloudData Center 1                Data Center 2           Single Man...
Management Server Deployment Architecture                       Single-node                                               ...
Core CloudStack Components•    Hosts    • Servers onto which services will be provisioned                                 ...
CloudStack Storage    Primary Storage•     Configured at Cluster-level. Close to hosts for better      performance        ...
Provisioning Process1.   User Requests Instance                                   VM2.   Provision Optional Network       ...
Citrix XenServer                                               CloudStack•    Integrates directly with XenServer Pool    M...
RedHat Enterprise Linux (KVM)•   Integrates with libvirt using Cloud    Agent                                 CloudStack  ...
VMware vSphere                                     CloudStack•   Integration through vCenter       Manager•   System VM co...
Management Server Interaction with Hypervisors                                             Management                     ...
Multi-tenancy & Account Management     Cloud                      Resources    Domain                    VMs, IPs, Snapsho...
CloudStack Network
CloudStack Network Features The Service ofVirtual NetworkThe Management of Physical Network
Physical Network          Operations                                                Users          Admin and          Clou...
CloudStack Network Traffic TypeNetwork Traffic type:   Public Network:       Public traffic is generated when VMs in the c...
CloudStack Network Mode  Basic Network                       Advanced Network• AWS-style networking             • Account’...
CloudStack Basic Network• VR provides service: DHCP, DNS• VMs Isolation by Security Group
Security Group Isolation
CloudStack Advanced Network               Public Network                        Guest Network 1                           ...
VLAN Isolation
CloudStack Advanced Network Service•   Firewall•   Source NAT•   Static NAT•   Load Balancing•   Port Forwarding•   VPN
Advanced Network - NAT • Default Deny all the
Advanced Network - Load Balancing
Advanced Network - Port Forwarding
Guest Virtual Network With Physical Device  CS Virtual Router provides Network Services                        External De...
Network Offerings• Cloud provider defines the  feature set for guest networks• Toggle features or service levels   –   Sec...
Physical Network – Guest Network Mapping                                     VM Instance                         • Choose ...
Advanced Network – Multi-tier NetworkPublic Network                    Guest Network 1                   Guest Network 2  ...
Advanced Network – Virtual Private Network                                 Internet                                       ...
CloudStack Storage
Storage                                                                                    • Primary Storage              ...
Primary Storage Support Matrix        Type       XenServer       VMWare         KVMLocal Disk      Supported      Supporte...
Storage Tagging•   Supported via storage tags for primary storage•   Specify a tag when adding a storage pool•   Specify a...
WORM Storage• Write Once Read Many storage pattern is  supported by two different storage types  – Secondary Storage (NFS ...
CloudStack System VM
CloudStack System VMs•   System VMs optimize and scale the data path on behalf of CloudStack     –   Stateless, can be des...
CloudStack System VMs• Virtual Router VM   – Provides multiple network services   – IPAM (DHCP), DNS, NAT, Source NAT, Fir...
System VM spec•   Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security    APT r...
MS Internals• Architecture• Workflow• High Availability• Scalability
Inside a Management Server                                                                                   Plugins      ...
Old Architecture                                                                                                         A...
New Deployment Architecture            • Scales horizontally to              different pressure points            • Automa...
New Architecture – API Server    UI                Cloud                                CLI                               ...
New Architecture – Execution                   Server                  Execution Server                      • Execution S...
New Architecture – Resources          Agent                              • Resources are carried in    Hypervisor Resource...
Cloud                                                        Other  UI                                              CLI   ...
Kernel Module• Understands how to orchestrate long running  processes (i.e. VM starts, Snapshot copies,  Template propagat...
Plugins• Various ways to add more capability to  CloudStack• Implements clearly defined interfaces• All operations must be...
Anatomy of a Plugin                                      Rest API-       Optional. Required only if needs to expose config...
Anatomy of a Plugin• Can be two jars: server component to be  deployed on management server and an optional  ServerResourc...
Plugin Interfaces Available• NetworkGuru – Implements various network isolation technologies  and ip address technologies•...
Adding a Plugin to CloudStack• Components are configured through  components.xml• Supports DAO, Manager, and Adapter patte...
High Availability
High Availability• Service Offering contains a flag for whether  HA should be supported for the VM• Does not use the nativ...
Triggering High AvailabilityVM HA are triggered via the following methods:• VM Sync detects out of band VM changes• Resour...
Scalability
Current Status• 10k resources managed per management server  node• Scales out horizontally (must disable stats  collector)...
Balancing Incoming Requests• Each management server has two worker thread pools for incoming  requests: effectively two se...
Comparison of two Approaches• Stats Collector – collects capacity statistics   – Fires every five minutes to collect stats...
Resource Load Balancing•   As management server is added into the cluster, resources are rebalanced    seamlessly.     –  ...
Interactions                                                                                                              ...
CloudStack vs. OpenStack vs.Eucalyptus
CloudStack•   Mainly written in Java•   ASL2.0 license•   Has more than 100 production clouds (Around May, 2012)•   Suppor...
OpenStack•   Mainly written in Python•   ASL2.0 license•   Support private/hybrid/public cloud•   Immature for commercial ...
Eucalyptus (Open Source edition)•   Mainly written in Java•   GPLv3 license•   Focus on private cloud•   Support KVM/Xen a...
2012 CloudStack Design Camp in Taiwan--- CloudStack Overview-1
Upcoming SlideShare
Loading in...5
×

2012 CloudStack Design Camp in Taiwan--- CloudStack Overview-1

1,695

Published on

CloudStack Design Camp in Taiwan Overview Slide
by TCloud Computing

Published in: Technology

Transcript of "2012 CloudStack Design Camp in Taiwan--- CloudStack Overview-1"

  1. 1. CloudStack Overview Alan Chu@ TCloud Computing
  2. 2. Outline• Overview of CloudStack• Problem Definition• Feature set overview• Network• Storage
  3. 3. What is CloudStack • Multi-tenant cloud orchestration platform – Turnkey solution for delivering IaaS clouds – Hypervisor agnostic Build your cloud the way the – Scalable and flexibleworld’s most successful clouds – Open source, open standards are built • Deliver cloud services faster and cheaper
  4. 4. History of Apache CloudStack• 2009: Cloud.com, 100% proprietary• 2010: Cloud.com, open core, GPL v3• 2011: Citrix Systems, 100% open, GPL v3• 2012: ASF, 100% open, Apache License 2
  5. 5. CloudStack Supports Multiple Cloud Strategies Private Clouds Public Clouds On-premise Hosted Multi-tenant Enterprise Cloud Enterprise Cloud Public Cloud • Dedicated • Dedicated • Mix of shared and resources resources dedicated • Security & total • Security resources control • SLA bound • Elastic scaling • Internal network • 3rd party owned • Pay as you go • Managed by and operated • Public internet, Enterprise or 3rd VPN access party
  6. 6. CloudStack Provides On-demand Access Org A Org B Users Admin AdminEnd User Users Users Compute Network StorageAdmin
  7. 7. Problem Definition• Offer a scalable, flexible, manageable IaaS platform that follows established cloud computing paradigms• IaaS – Orchestrate physical and virtual resources to offer self-service infrastructure provisioning and monitoring• Scalable – 1 -> N hypervisors / VMs / virtual resources – 1 -> N end users• Flexible – Handle new physical resource types • Hypervisors, storage, networking – Add new APIs – Add new services – Add new network models
  8. 8. Problem Definition (cntd)• Manageable – Hide complexity of underlying resources – Rich functional end-user and admin UI – Admin API to automate operations – Easy install, upgrade for small -> large clouds – Simple scaling, automated resilience
  9. 9. Feature Set Overview
  10. 10. Open Flexible PlatformCompute Hypervisor XenServer VMware KVM Bare metalStorage Block & Object Fiber Local Disk iSCSI NFS Swift Channel Primary Storage Secondary StorageNetwork Network & Network Services Connection Load Isolation Firewall VPN Type balancer
  11. 11. Service Offering
  12. 12. Create Custom Virtual Machines via Service Offerings Select Operating System • Windows, Linux Select Compute Offering • CPU & RAM Select Disk Offering • Volume Size Select Network Offering • Network & Services Create VM
  13. 13. Dashboard Provides Overview of Consumed Resources• Running, Stopped & Total VMs• Public IPs• Private networks• Latest Events
  14. 14. Virtual Machine Management Users ChangeVM Operations VM Access VM Status Service Offering Start • CPU Utilized 2 CPUs 4 CPUs Stop 1 GB RAM 4 GB RAM • Network Read 20 GB 200 GB Restart • Network Writes 20 Mbps 100 Mbps Destroy
  15. 15. Volume & Snaphost Management VM 1 Add / Delete Volumes Volume Create Templates Volume Template from Volumes Hourly Weekly Schedule Now Snapshots Daily Monthly …. View Snapshot History
  16. 16. Network & Network Services• Create Networks• Acquire public IP address for NAT & load balancing• Control traffic to VM using ingress and egress firewall rules• Set up rules to load balance traffic between VMs
  17. 17. CloudStack Deployment Architecture Internet  Hypervisor is the basic unit of CloudStack Management scale. ServerZone 1  Cluster consists of one ore more hosts of same hypervisor L3 core  All hosts in cluster have access to shared (primary) storagePod 1 Access Layer Pod N Secondary  Pod is one or more clusters, …. Storage usually with L2 switches. Cluster N  Availability Zone has one or more pods, has access to …. secondary storage.  One or more zones represent Cluster 1 cloud Host 1 Primary Storage Host 2
  18. 18. Management Server Managing Multiple Zones CloudData Center 1 Data Center 2 Data Center 2 Data Center 3 Zone 2 Zone 2 Zone1 Zone 3 Zone 4 3 Zone CloudStack Cloud can have one or more Availability Zones (AZ). Data Center 2 Data Center 2 Data Center 2 Zone 2 Zone 2 Zone 2 3 Zone Zone 3 Zone 3
  19. 19. Management Server Managing Multiple Zones CloudData Center 1 Data Center 2  Single Management Server can Data Center 2 Mgmt Data Center 3 manage multiple zones Server Zone 2  Zones can be geographically Zone 2 distributed but low latency links are Zone 3 expected for better performance Zone1 Zone 4 3 Zone  Single MS node can manage up to 10K hosts.  Multiple MS nodes can be deployed Data Center 2 as cluster for scale or redundancy Data Center 2 Data Center 2 Zone 2 Zone 2 Zone Zone 3 2 Zone 3 Zone 3
  20. 20. Management Server Deployment Architecture Single-node Multi-node Deployment Deployment MS User API User API MS MySQL Load MS DB BalancerAdmin API Admin API MySQL MS DB Back Up Replication DB  MS is stateless. MS can be deployed as physical server or VM Infrastructure Infrastructure Resources  Single MS node can manage up to Resources 10K hosts. Multiple nodes can be deployed for scale or redundancy
  21. 21. Core CloudStack Components• Hosts • Servers onto which services will be provisioned VM• Primary Storage Host • VM storage VM• Cluster Host • A grouping of hosts and their associated storage• Pod Primary Storage • Collection of clusters• Network Cluster • Within the switch Secondary Storage Secondary• Storage Network Cluster • Template, snapshot and ISO storage• Zone CloudStack Pod • Collection of pods, network offerings and secondary storage• Management Server Farm CloudStack Pod • Responsible for all management and provisioning tasks Zone
  22. 22. CloudStack Storage Primary Storage• Configured at Cluster-level. Close to hosts for better performance L3 switch• Stores all disk volumes for VMs in a cluster• Cluster can have one or more primary storages Pod 1 L2 switch• Local disk, iSCSI, FC or NFS Secondary Cluster 1 Storage Host 1 Primary Secondary Storage Storage Host 2• Configured at Zone-level• Stores all Templates, ISOs and Snapshots• Zone can have one or more secondary storages• NFS, OpenStack Swift
  23. 23. Provisioning Process1. User Requests Instance VM2. Provision Optional Network Host Services Host3. Copy instance template from Primary Storage secondary storage to primary Cluster storage on appropriate cluster4. Create any requested data Pod volumes on primary storage for the cluster Template5. Create instance Secondary Storage6. Start instance Zone
  24. 24. Citrix XenServer CloudStack• Integrates directly with XenServer Pool Manager Master• Snapshots at host level XenServer Pool Master Host• System VM control channel at host level• Network management is host level XenServer Host XenServer Host XenServer Host XenServer Host XenServer Resource Pool
  25. 25. RedHat Enterprise Linux (KVM)• Integrates with libvirt using Cloud Agent CloudStack Manager• Snapshots at host level• System VM control channel at host Cloud Agent level Libvirt• Network management is host level KVM Host Cloud Agent Libvirt KVM Host
  26. 26. VMware vSphere CloudStack• Integration through vCenter Manager• System VM control channel via vSphere Host CloudStack private network vCenter vSphere Host• Snapshot and volume management via Secondary Storage VM vSphere Cluster• Networking via vSphere vSwitch vSphere Host vSphere Host vSphere Host vSphere Cluster Data Center
  27. 27. Management Server Interaction with Hypervisors Management Server XAPI HTTPS vCenter Agent XenServer KVM ESX • XS 5.6, 5.6FP1, 5.6 SP2, • ESX 4.1, 5.0 • RHEL 6.0, 6.1, 6.2 6.0.2 • Full Snapshots • Full Snapshots (not live) • Incremental Snapshots • VMDK • QCOW2 • VHD • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS, iSCSI, FC & Local disk • Storage over-provisioning: • Storage over-provisioning: • Storage over-provisioning: NFS, iSCSI NFS NFS
  28. 28. Multi-tenancy & Account Management Cloud Resources Domain VMs, IPs, Snapshots… • Domain is a unit of Org A isolation that represents Admin a customer org, business unit or a reseller Domain Reseller A • Domain can have Admin Resources arbitrary levels of sub- Sub-Domain Org C VMs, IPs, Snapshots… domains Admin • A Domain can have one Account or more accounts Group A • An Account represents Account one or more users and is Group B the basic unit of isolation User 1 • Admin can limit resources at the Account User 2 or Domain levels
  29. 29. CloudStack Network
  30. 30. CloudStack Network Features The Service ofVirtual NetworkThe Management of Physical Network
  31. 31. Physical Network Operations Users Admin and Cloud API CloudStack MS Cluster Router MySQL Load Balancer Availability Zone L3 Core Switch Access Layer Switches Secondary Servers … … … … … Storage Pod 1 Pod 2 Pod 3 Pod N
  32. 32. CloudStack Network Traffic TypeNetwork Traffic type: Public Network: Public traffic is generated when VMs in the cloud access the internet, e.g Virtual Router Guest Network: The tenant network to which instances are attached. Storage Network: The physical network which connects the hypervisor to the storages. Management Network: Control Plane traffic between CloudStack management server and hypervisor clusters
  33. 33. CloudStack Network Mode Basic Network Advanced Network• AWS-style networking • Account’s VM Isolation by VLAN• Account’s VM Isolation by • VR can provide more services : Security Group NAT, Firewall, PF, LB, VPN• VR provides service: DHCP, DNS • Guest Network supports Isolated• Each VM has only one NICs and Shared Network types (Network) • Each VM can have more NICs (Network)
  34. 34. CloudStack Basic Network• VR provides service: DHCP, DNS• VMs Isolation by Security Group
  35. 35. Security Group Isolation
  36. 36. CloudStack Advanced Network Public Network Guest Network 1 VLAN 101 Public IP Guest 1 10.1.1.2 Gateway address VM 1 address 65.37.141.11 10.1.1.1 Guest 1 Virtual Guest 1 10.1.1.3 Public Router VM 2 Internet NAT Guest 1 10.1.1.4 DHCP VM 3 Load Balancing Port Forwarding Firewall VPN Guest Network 2 Public IP VLAN 102 address Gateway Guest 2 10.1.1.2 65.37.141.24 address VM 1 10.1.1.1 Guest 2 Virtual Guest 2 10.1.1.3 Router VM 2 Guest 2 10.1.1.4 VM 3
  37. 37. VLAN Isolation
  38. 38. CloudStack Advanced Network Service• Firewall• Source NAT• Static NAT• Load Balancing• Port Forwarding• VPN
  39. 39. Advanced Network - NAT • Default Deny all the
  40. 40. Advanced Network - Load Balancing
  41. 41. Advanced Network - Port Forwarding
  42. 42. Guest Virtual Network With Physical Device CS Virtual Router provides Network Services External Devices provide Network Services Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8 VLAN 100 VLAN 100Public PublicNetwork/Internet Network/Internet Guest Private IP Guest 10.1.1.2 VM 1 Public IP 10.1.1.1 VM 1 65.37.141.111 Juniper 10.1.1.111 GatewayPublic IP SRX address CS Firewall65.37.141.11 10.1.1.1 Guest Guest Virtual 10.1.1.3 VM 2 10.1.1.3 VM 2 Router Public IP Private IP DHCP, DNS NetScaler 10.1.1.112 65.37.141.112 NAT Guest Load Guest Load Balancing 10.1.1.4 VM 3 Blancer VM 3 10.1.1.4 VPN Guest Guest 10.1.1.5 VM 4 10.1.1.5 VM 4 CS Virtual DHCP, DNS Router
  43. 43. Network Offerings• Cloud provider defines the feature set for guest networks• Toggle features or service levels – Security groups on/off – Load balancer on/off – Load balancer software/hardware – VPN, firewall, port forwarding• User chooses network offering when creating network• Enables upgrade between network offerings• Default offerings built-in – For classic CloudStack networking
  44. 44. Physical Network – Guest Network Mapping VM Instance • Choose the instantiated guest network Guest Network • Instance of Network Offering • Shared: created by Admin • Isolated: Created and owned by user • One virtual router for one network • Cross pod, within Zone • VLAN id picked from the pool Physical Network Network Offering • Zone level • Only for Guest traffic • Defined by NIC • Guest network type: Shared or Isolated • Assigned with traffic type (P, G, M, S) • Defined a set of network services, • Associated by label/vswitch name such as DHCP, Firewall, VPN, NAT… • Attached with device as service • Bandwidth provider Tag
  45. 45. Advanced Network – Multi-tier NetworkPublic Network Guest Network 1 Guest Network 2 Guest Network 3Internet 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24 VLAN 100 VLAN 101 VLAN 102 App VM 10.1.2.31 1 10.1.3.21 Web VM 10.1.1.2 1 10.1.2.21 Public IP 65.37.141.111 App VM 10.1.2.24 2 10.1.3.45 Private IP Web VM CS 10.1.1.1 10.1.1.3 2 10.1.2.18 Virtual Router Web VM DHCP, DNS 3 10.1.2.38 10.1.3.24 DB VM 1 10.1.1.4 NAT, Firewall LB, VPN, Port Forwarding Web VM 10.1.1.5 4 10.1.2.39 CS Virtual CS DHCP, DNS Router DHCP, DNS Virtual Router
  46. 46. Advanced Network – Virtual Private Network Internet Internal VLAN CS Loadbalancer Virtual Router / IPSec site-to-site VPN Other Data CenterVirtual Router Services App VM• DNS 1 10.1.2.31• LB Web VM 1• Site-to-Site VPN 10.1.1.1• Static Routes App VM• Network ACLs Web VM 10.1.2.24 2• NAT, PF 10.1.1.3 2• FW [ingress & egress] Web VM 3 DB VM 1 10.1.1.4 10.1.3.24 Web VM 10.1.1.5 4 Guest Network Guest Network Guest Network 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24 VLAN 100 VLAN 101 VLAN 102
  47. 47. CloudStack Storage
  48. 48. Storage • Primary Storage Zone-Level Layer 3 Switch Private Network – Block device to the VM – IOPs intensive – Accessible from host orPod 1 Pod Pod cluster wide • WORM Storage Pod-Level Layer-2 2 N Switch Scale-Out NFS – Secondary Storage or Object Computing Server 1 Primary Store for templates, ISO, and Storage snapshot archiving Cluster 2 Computing Primary – High capacity Server 2 Storage • CloudStack manages the Computing Scale-Out storage between the two to Server 3 NFS achieve maximum benefit and Cluster 1 Primary Storage resiliency Computing Server 4
  49. 49. Primary Storage Support Matrix Type XenServer VMWare KVMLocal Disk Supported Supported SupportediSCSI Supported Supported Not SupportedFiber Channel Supported Supported Not SupportedNFS Supported Supported Supported
  50. 50. Storage Tagging• Supported via storage tags for primary storage• Specify a tag when adding a storage pool• Specify a tag when adding a disk offering• Only storage pools with the tag will be allocated for the volume
  51. 51. WORM Storage• Write Once Read Many storage pattern is supported by two different storage types – Secondary Storage (NFS Server within an availability zone) – Object Store (Swift implementation for cross-zone)• Objective for WORM storage – High capacity, cheap storage – Easy to increase capacity• Used to store templates, ISOs, and snapshots
  52. 52. CloudStack System VM
  53. 53. CloudStack System VMs• System VMs optimize and scale the data path on behalf of CloudStack – Stateless, can be destroyed and recreated from database state – Highly Available – Communicates with Management Server over management network – Usually have 3 interfaces: control(linked-local), mgmt and public• Console Proxy VM – Provides AJAX-style HTTP-only console viewer – Grabs VNC output from hypervisor – Scales out (more spawned) as load increases – Java-based server Communicates with MS• Secondary Storage VM – Provides image (template) management services – Download from HTTP file share or Swift – Copy between zones – Scale out to handle multiple NFS mounts – Java-based server communicates with MS
  54. 54. CloudStack System VMs• Virtual Router VM – Provides multiple network services – IPAM (DHCP), DNS, NAT, Source NAT, Firewall, Port Forwarding, VPN – User-data, Meta-data, guest SSH keys and password change server – Redundancy via VRRP – MS configures VR over SSH • Proxied via the hypervisor on XS and KVM
  55. 55. System VM spec• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu, dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non- standard port (3922). SSH logins only using keys (keys are generated at install time and are unique for every customer)• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved security and speed• Latest version of jre from Sun/Oracle ensures improved security and speed
  56. 56. MS Internals• Architecture• Workflow• High Availability• Scalability
  57. 57. Inside a Management Server Plugins cmd.execute() Plugins Cmds Plugins AsyncCS API API Job Services Servlet Queue API Mgr Kernel Responses Agent API (Commands) Agent Resources Manager Local Or Remote Hypervisor Network Native Device APIs API MySQL
  58. 58. Old Architecture API Layer Pros EC2 CloudStack Access Control • Agile development for existing developersVirtual Machine Manager Console Proxy Manager • Scales well horizontally Async Job Manager Snapshot Manager Template Manager Network Manager Storage Manager Cons … • Monolithic • Difficult to educate new and third-party Agent Manager XenServ KVM SRX F5 NetScal Other developers er er • Easy to introduce bugs Resour Resour Resour Resourc Resourc ce ce Resour es ce e ce 59
  59. 59. New Deployment Architecture • Scales horizontally to different pressure points • Automatically scales service VMs in zones to facilitate most efficient data path transfers • Fault isolation between API servers and Execution Servers and resources within zones
  60. 60. New Architecture – API Server UI Cloud CLI Other Clients • API Server isolates Portal integration code from REST Execution Server API Server • API Server can OAM&P API Pluggable API Engine End User EC2 Other horizontally scale to Management Services API API ACL & APIs Integration handle traffic- Resource management- Configuration Authentication - Accounts, Domains, and • Easily adds other API- Additional operations added Projects - ACL, limits compatibility • Easily exposes API by third party checking Framework- Job Queue-- Database Access Layer OSGi needed by third party vendors
  61. 61. New Architecture – Execution Server Execution Server • Execution Server protected by job queue • Kernel kept small for stability. It Services API Kernel Plugins• Drives long running VM operations • Storage only drives processes.• Syncs between resources managed and DB Handling • Network • Plugins provide mappings of• Generates events Handling • Deployment virtual entities to physical planning • Hypervisor resources Handling • Third party plugins to provide vendor differentiation in CloudStack• Framework Cluster Management • Component Framework • Communicates with resources•• Job Management • Alert & Event Management (OSGi) Transaction Management within data center over message•• Database Access Layer Messaging Layer bus
  62. 62. New Architecture – Resources Agent • Resources are carried in Hypervisor Resources service VMs to be in close network proximity to the Network Resources physical resources it Storage Resources manages Image & Template Resources • Easily scales to utilize the most abundant resource in Snapshot Resources data center (CPU & RAM) • Communicates with Execution Server over message bus (JSON) • Can be replicated for fault tolerance
  63. 63. Cloud Other UI CLI Clients Portal Management Server REST API OAM&P API End User API EC2 API Other APIs Pluggable Service API EngineConsole Proxy ACL & Authentication Security AdaptersManagement - Accounts, Domains, and Projects - ACL, limits checking Account Management Connectors Template Services API Access Deployment Planning Plugin API HA Kernel - Drives long running VM Services API Network Configurations Usage operations Calculations - Syncs between resources managed and DB Network Elements Additional - Generates events Services Hypervisor Gurus Cluster Resource Job Alert & Event Database Management Management Management Management Access Event Bus Message Bus Hypervisor Network Storage Image Snapshot Resources Resources Resources Resources Resources
  64. 64. Kernel Module• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)• Well defined process steps• Calls Plugin API to execute functionalities that it needs
  65. 65. Plugins• Various ways to add more capability to CloudStack• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module
  66. 66. Anatomy of a Plugin Rest API- Optional. Required only if needs to expose configuration API to admin. ServerResource - Optional. Required if Plugin needs to be co- located with the resource - Implements translation layer to talk to resource - Communicates withPlugin API Implmentation server component via JSON Data Access Layer
  67. 67. Anatomy of a Plugin• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co- located with the resource• Server component can implement multiple Plugin APIs to affect its feature• Can expose its own API through Pluggable Service so administrators can configure the plugin• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement
  68. 68. Plugin Interfaces Available• NetworkGuru – Implements various network isolation technologies and ip address technologies• NetworkElement – Facilitate network services on network elements to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes
  69. 69. Adding a Plugin to CloudStack• Components are configured through components.xml• Supports DAO, Manager, and Adapter patterns• Open to other component frameworks (OSGi a possibility)
  70. 70. High Availability
  71. 71. High Availability• Service Offering contains a flag for whether HA should be supported for the VM• Does not use the native HA capability of hypervisors for XenServer and KVM• Uses adapters to fine tune HA process
  72. 72. Triggering High AvailabilityVM HA are triggered via the following methods:• VM Sync detects out of band VM changes• Resource Management detects that a resource is unreachable and its state can not be determined.• VM start/stop has been sent to the resource but resource does not return• Details of how high availability is done is at http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/CloudStack_High_Availability_- _Developers_Guide
  73. 73. Scalability
  74. 74. Current Status• 10k resources managed per management server node• Scales out horizontally (must disable stats collector)• Real production deployment of tens of thousands of resources• Internal testing with software simulators up to 30k physical resources with 300k VMs managed by 4 management server nodes• We believe we can at least double that scale per management server node
  75. 75. Balancing Incoming Requests• Each management server has two worker thread pools for incoming requests: effectively two servers in one. – Executor threads provided by tomcat – Job threads waiting on job queue• All incoming requests that requires mostly DB operations are short in duration and are executed by executor threads because incoming requests are already load balanced by the load balancer• All incoming requests needing resources, which often have long running durations, are checked against ACL by the executor threads and then queued and picked up by job threads.• # of job threads are scaled to the # of DB connections available to the management server• Requests may take a long time depending on the constraint of the resources but they don’t fail.
  76. 76. Comparison of two Approaches• Stats Collector – collects capacity statistics – Fires every five minutes to collect stats about host CPU and memory capacity – Smart server and dumb client model: Resource only collects info and management server processes – Runs the same way on every management server• VM Sync – Fires every minute – Peer to peer model: Resource does a full sync on connection and delta syncs thereafter. Management server trusts on resource for correct information. – Only runs against resources connected to the management server node
  77. 77. Resource Load Balancing• As management server is added into the cluster, resources are rebalanced seamlessly. – MS2 signals to MS1 to hand over a resource – MS1 wait for the commands on the resources to finish – MS1 holds further commands in a queue – MS1 signals to MS2 to take over – MS2 connects – MS2 signals to MS1 to complete transfer – MS1 discards its resource and flows the commands being held to MS2• Listeners are provided to business logic to listen on connection status and adjusts work based on who’s connected.• By only working on resources that are connected to the management server the process is on, work is auto-balanced between management servers.• Also reduces the message routing between the management servers.
  78. 78. Interactions OVM Cluster Primary Storage vcenter Monitoring Primary CS API vSphere Cluster Storage End User UI Primary XS Cluster Storage Admin UI Clustered CloudStack XAPI Domain CS Admin & CloudStack CloudStack Admin End-user API Primary UI Management JSON KVM Cluster Storage Server NetConf Juniper SRXCloud user Nitro API{API client (Fog/etc)} VNC JSON ec2 API JSON Netscaler Cloud user Console Console {ec2 API client } Proxy VM Proxy VM NFS MySQL Server {Proxied} SSH Sec. Storage NFS NFS Sec. Storage VM Ajax HTTPS VM Console Router VM HTTP (Template Download) Router VM HTTP (Template Copy) Router VM Cloud user HTTP (Swift)
  79. 79. CloudStack vs. OpenStack vs.Eucalyptus
  80. 80. CloudStack• Mainly written in Java• ASL2.0 license• Has more than 100 production clouds (Around May, 2012)• Support private/hybrid/public cloud• Scale to 30K physical host in commercial environment• Support XenServer/Vsphere/KVM/OVM/Baremetal as hypervisor• Multiple geographically distributed datacenters management• Flexible and rich network functionality• Easy installation and management• Amazon EC2 API compatible• Well documented• Active community
  81. 81. OpenStack• Mainly written in Python• ASL2.0 license• Support private/hybrid/public cloud• Immature for commercial usage• Support XenServer/Vsphere/KVM/Xen/Hyper-V as hypervisor• Network is single point of failure• Weak VPN support for enterprise hybrid cloud• All inter-module communication are based on MQ• Not well documented• A bit hard to install• Amazon EC2 API partially compatible
  82. 82. Eucalyptus (Open Source edition)• Mainly written in Java• GPLv3 license• Focus on private cloud• Support KVM/Xen as hypervisor• Fully compatible with Amazon EC2• Fully compatible with Amazon S3 via Walrus• EBS support via AoE and iSCSI• Both web UI and command line tools for cloud administration• Well documented• Difficult to getting started

×