Management server internals


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Management server internals

  1. 1. EC2 API Layer CloudStack Access ControlVirtual Machine Manager Console Proxy Manager Async Job Manager Network Manager Template Manager Snapshot Manager Storage Manager … Agent ManagerXenServer KVM vSphere SRX F5 NetScaler OVMResource Resource Resource Resource Resource Resource Resource
  2. 2. Cloud Other UI CLI Clients Portal Management Server REST API OAM&P API End User API EC2 API Other APIs Pluggable Service API EngineConsole Proxy ACL & Authentication Security AdaptersManagement - Accounts, Domains, and Projects - ACL, limits checking Account Management Connectors Template Services API Access Deployment Planning Plugin API HA Kernel Job - Drives long running VM Queue Services API Network Configurations Usage operations Calculations - Syncs between resources managed and DB Network Elements Additional - Generates events Services Hypervisor Gurus Cluster Resource Job Alert & Event Database Management Management Management Management Access DB Event Bus Message Bus Hypervisor Network Storage Image Snapshot Resources Resources Resources Resources Resources
  3. 3. Kernel Module• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)• Well defined process steps• Calls Plugin API to execute functionalities that it needs
  4. 4. Plugins• Various ways to add more capability to CloudStack• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module
  5. 5. Anatomy of a Plugin Rest API- Optional. Required only if needs to expose configuration API to admin. ServerResource - Optional. Required if Plugin needs to be co- located with the resource - Implements translation layer to talk to resource - Communicates withPlugin API Implmentation server component via JSON Data Access Layer
  6. 6. Anatomy of a Plugin• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co- located with the resource• Server component can implement multiple Plugin APIs to affect its feature• Can expose its own API through Pluggable Service so administrators can configure the plugin• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement
  7. 7. Plugin Interfaces Available• NetworkGuru – Implements various network isolation technologies and ip address technologies• NetworkElement – Facilitate network services on network elements to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes
  8. 8. Adding a Plugin to CloudStack• Components are configured though components.xml• Supports DAO, Manager, and Adapter patterns• Open to other component frameworks (OSGi a possibility)
  9. 9. Components.xml Example<components.xml> <system-integrity-checker class=""> <checker name="ManagementServerNode" class=""/> <checker name="EncryptionSecretKeyChecker"class=""/> <checker name="DatabaseIntegrityChecker" class=""/> <checker name="DatabaseUpgradeChecker" class=""/> </system-integrity-checker> <interceptor library=""/> <management-server class=""library=""> <adapters key=""> <adapter name="LocalStorage" class=""/> <adapter name="Storage" class=""/> </adapters> <pluggableservice name="VirtualRouterElementService"key=""class=""/> </management-server></components.xml>
  10. 10. Sequence Flow for deploy VM Kernel End User Security User VM VirtualMac Network Storage Network Job Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling Deploy VM ACL Checks Allocate Entity in CS Allocate VM Allocate NIC Allocate IP Allocate Volume Schedules Deploy Job Returns with job id, VM id Query Job ResultReturns with job status
  11. 11. Sequence Flow for deploy VM Deploymen Server User VM VirtualMac Network Storage Network Network Template tJob Threads Services API Resources Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner Start VM Start User VM Start VM Get a Deployment Plan (Host and StoragePool) Prepare Nics Reserve resources for Nic Notify that Nic is about to be started in network Agent Calls Prepare Volumes Prepare template on Primary Storage Agent Calls Agent Start VM Call Stores job result
  12. 12. ServerResource• Translation layer between CloudStack commands and resource API• May be Co-located with resource• Have no access to DB• API defined in JSON messages
  13. 13. DAO• SQL generation done mostly in GenericDaoBase• Uses JPA annotations• Very little code to write for each individual DAO• Database Access Layer for Kernel• No support for more complicated features such as fetch strategy• Welcome to use other types of ORM in other modules but like to hear about preferred library. (Hibernate is out due to licensing issues)
  14. 14. Example DAO// // public interface ExampleDao@Table(name=“example”) extends GenericDao<ExampleVO, Long> {public class ExampleVO { } @Id @GeneratedValue(strategy= // ExampleDaoImpl.javaGenerationType.IDENTITY) @Local(value=ExampleDao.class) @Column(name=“id”) public class ExampleDaoImpl long id; extends GenericDaoBase<ExampleVO, Long> implements ExampleDao { @Column(name=“name”) String name; protected ExampleDaoImpl() { } @Column(name=“value”) } String value;}
  15. 15. Triggering High AvailabilityVM HA are triggered via the following methods:• VM Sync detects out of band VM death• Resource Management detects that a resource is unreachable and its state can not be determined.• VM start/stop has been sent to the resource but resource does not return• Details of how high availability is done is at _Developers_Guide
  16. 16. High Availability Future• Moving toward using the native HA capability of the hypervisor.• Looking to do more in the DRS area to coordinate recovery of wide spread outage.
  17. 17. VM Sync• Currently a sync of VM state, not entire VM• VM Sync happens between management server and hypervisor resources• Peer-to-peer sync• Hypervisor DB is considered to be the DB of truth• Two steps: – Full Sync – Intermittent delta sync• Establishes full sync when first connecting to the hypervisor resource• After full sync, hypervisor resource keeps track of the last sync results and only report out of band changes on delta sync• Utilizes the most abundant resources in data center: CPU and memory• Conserve the most scarce resource: DB connections• Virtually no DB connections utilized during delta sync unless there are out of band changes.
  18. 18. Storage Zone-Level Layer 3 Switch Private NetworkPod 1 Pod 2 Pod N Pod-Level Layer-2 Switch … Scale-Out NFS Computing Server 1 Primary Storage Cluster 2 Computing Server 2 Primary Storage Computing Server 3 Cluster 1 Primary Storage Computing Server 4
  19. 19. Storage• CloudStack supports two types of storage – Primary Storage: block device to the VM – WORM Storage: Secondary or Object Store for templates, ISO, and snapshot archiving• Primary storage is high on IOPs (expensive)• Secondary storage is high on capacity (cheap)• CloudStack manages the storage between the two to achieve maximum benefit and resiliency
  20. 20. Disk Offering• Disk Offering is how disks are offered to the end user• Disk Offering has storage tags which can be used to implementing storage tiering• Service Offering actually contains a disk offering for the root disk
  21. 21. Snapshots• Snapshots are used as backups• Taken on the primary storage and moved to secondary storage• Full snapshots on VmWare and KVM. Need help.• Incremental snapshots on XenServer
  22. 22. XenServer Snapshot Base Copy Full1 Full2D1 Template S11 S21D2 S1 S12 S22VM S2 S13 S23
  23. 23. Storage Future• Secondary Storage multi-homed in Pod• Object Store for between zones• Direct Access to Storage for enterprise• Storage Plugins