OpenStack Architecture

4. PAGE 4Copyright © 2014 Mirantis, Inc. All rights reserved Initial State Assumes Project is created, provisioning quota is available, user has an access to Horizon/CLI Cloud Operator, DevOp, etc. Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent

5. PAGE 5Copyright © 2014 Mirantis, Inc. All rights reserved Step 1: Request Provisioning – From UI • Login to Horizon • Specify parameters of VM • VM Name • Image (OS type) • Flavor (specifies CPU, Memory, Disk) • Network (required for Neutron) • Optional (SSH Keys, Persistent volumes, comments, etc.) • Select "Create" button

7. PAGE 7Copyright © 2014 Mirantis, Inc. All rights reserved Horizon • Is “stateless” — doesn’t require a database • Delegates error handling to the back-end • Doesn’t support all the API functions • Can use memcached or database to store sessions • Gets updated via API polling

8. PAGE 8Copyright © 2014 Mirantis, Inc. All rights reserved Step 1: Request VM Provisioning via UI/CLI Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Cloud Operator, DevOp, etc. User logs in to UI Specifies VM params: name, flavor, keys, etc. and hits "Create" button

9. PAGE 9Copyright © 2014 Mirantis, Inc. All rights reserved Step 1: Request Provisioning – Under the Hood • Form parameters are converted to POST data • "Create" request initiates HTTP POST request to back-end • To Keystone if auth token is not cached – step 2

10. PAGE 10Copyright © 2014 Mirantis, Inc. All rights reserved The OpenStack Identity Service (Keystone) Keystone provides Identity, Token, Catalog and Policy services for use specifically by projects in the OpenStack family.

11. PAGE 11Copyright © 2014 Mirantis, Inc. All rights reserved Keystone: Identity Management • User • Credentials • Token • Associated with a user, an arbitrary bit of text that is used to access resources • Group of users • Project • Synonym to tenant • Role • Assigned to sers or groups for projects • Domain • Higher level of hierarchy – users and projects belong to domains

12. PAGE 12Copyright © 2014 Mirantis, Inc. All rights reserved Keystone: Service Catalog • Service • An OpenStack service, such as Compute (Nova), Object Storage (Swift), or Image Service (Glance). • Endpoint • A network-accessible address, usually described by URL, from where you access an OpenStack service • Rule • A set of requirements for performing an action over the endpoint.

13. PAGE 13Copyright © 2014 Mirantis, Inc. All rights reserved Keystone Architecture OpenStack Services Catalog Backend Token Backend Policy Backend Assignments Backend Identity Backend Credentials Backend Rule management interface and rule- based authorization Contains temporary tokens Contains endpoint registry Contains users and groups Contains credentials, e.g. EC2 tokens Contains domains, projects, roles and role assignments Deploys with its own DB but can also be substituted with LDAP or other EASKeystone API

14. PAGE 14Copyright © 2014 Mirantis, Inc. All rights reserved Keystone:Role Based Access Control (RBAC) 1. User gets Token from Keystone. 2. Token includes the list of user Projects and Roles in them. 3. User calls the Service specifying the Token. 4. Service interprets the Roles: • Service consults its policy.json file. • Policy.json specifies the list of available rules. • "admin_required": [["role:admin"], ["is_admin:1"]], • “owner" : [["project_id:%(project_id)s"]], • "admin_or_owner": [["rule:admin_required"], ["rule:owner"]], • Policy.json specifies which rules are enforced for operations and resources. • "volume:create": [["rule:admin_or_owner"]],

15. PAGE 15Copyright © 2014 Mirantis, Inc. All rights reserved Step 2: Validate Auth Data Horizon sends HTTP request to Keystone. Auth info is specified in HTTP headers. Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent

16. PAGE 16Copyright © 2014 Mirantis, Inc. All rights reserved Step 2: Validate Auth Data - Success Keystone sends temporary token back to Horizon via HTTP. Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent

17. PAGE 17Copyright © 2014 Mirantis, Inc. All rights reserved Step 3: Send API Request to Nova API Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Horizon sends POST request to Nova API (signed with given token).

19. PAGE 19Copyright © 2014 Mirantis, Inc. All rights reserved Nova API • Exposes REST API via HTTP • Provides system for managing multiple APIs on different sub-domains: • EC2-compatible—starting to be deprecated • Compute API—all innovation happens here • Is the only "allowed" way to interact with Nova • Is “stateless”

20. PAGE 20Copyright © 2014 Mirantis, Inc. All rights reserved Step 4: Validate API Token Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova API sends HTTP request to validate API token to Keystone.

22. PAGE 22Copyright © 2014 Mirantis, Inc. All rights reserved Keystone /w PKI - Token Validation • User gets one-time-password on creation • User uses it to establish a key-pair • Public key is signed and stored on Keystone • From this point user uses client certificate to login • Nova API performs offline check of the validity of token using CA&Cert it has from Keystone Keystone Key generate_cms_token (meta, keystone_key)user/pass/tenant signed_cms_token signed_cms_token Nova CA&Cert from Keystone verify(signed_cms_token, Cafile,certfile)

23. PAGE 23Copyright © 2014 Mirantis, Inc. All rights reserved Step 4: Validate API Token - Sucess Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Keystone validates API token and sends HTTP response with token acceptance/rejection info.

24. PAGE 24Copyright © 2014 Mirantis, Inc. All rights reserved Step 5: Process API Request • Validate request parameters: • Typographical errors are verified on code level • Cloud-related parameters are validated via database requests • If the request cannot be processed: • Throw an exception • If the request can be processed: • Save initial state to the database

26. PAGE 26Copyright © 2014 Mirantis, Inc. All rights reserved Nova Database • In theory, can be any relational database • Most of the deployments are done with MySQL or PostgreSQL • Nova API talks to database via SQLAlchemy (python ORM (Object Relational Mapper)) • Database HA should be done via external tools: • Galera • Multi-Master replication Model for MySQL (MMM)

27. PAGE 27Copyright © 2014 Mirantis, Inc. All rights reserved Step 5: Process API Request Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova API parses request to python object model and validates it by fetching data from Nova DB. If request is valid, it saves initial db entry about VM to the database.

28. PAGE 28Copyright © 2014 Mirantis, Inc. All rights reserved Step 6: Process API Request • Validate request parameters • Typographical errors are verified on code level. • Cloud-related parameters are validated via database requests. • If request cannot be processed: • Throw an exception. • If request can be processed: • Save initial state to the database. • Send message with next actions to Message Queue (MQ)

30. PAGE 30Copyright © 2014 Mirantis, Inc. All rights reserved Messaging Process Example • OpenStack uses two modes: • rpc.cast—don't wait for result (fire and forget) • rpc.call—wait for result (when there is something to return) Ex. Nova API Ex. Nova Scheduler

31. PAGE 31Copyright © 2014 Mirantis, Inc. All rights reserved OpenStack Messaging Notes • Uses multiple queues within single RabbitMQ instance. • Used by services to build machine state • Each compute node has a queue • Message traffic is not intensive. • Doesn't send broadcast messages. • For example: monitoring uses API polling • HA should be configured separately. • For example: mirrored queues not handled by OpenStack

32. PAGE 32Copyright © 2014 Mirantis, Inc. All rights reserved Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Step 6: Publish Provisioning Request Nova API makes rpc.cast to Scheduler. It publishes a short message to scheduler queue with VM info. Request has been validated, but no action has been taken yet, i.e. which host, IP address, etc.

33. PAGE 33Copyright © 2014 Mirantis, Inc. All rights reserved Step 7: Pick up Provisioning Request Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Scheduler picks up the message from MQ.

34. PAGE 34Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler Nova Scheduler is a daemon, which determines, on which compute host the request should run. • Only provisioning time component—not like VMware’s Distributed Resource Scheduler (DRS) • Typically co-located with the Cloud Controller

35. PAGE 35Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Available Schedulers Scheduler Description Behavior Chance Picks a host that is up Random Filter Picks the best-suited host which satisfies selected filter(s) Custom Prioritisation

37. PAGE 37Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • All Hosts • AllHostsFilter • All the available hosts, even non-active • ComputeFilter • All hosts that are operational and enabled • Based on Host statically configured properties • SimpleCIDRAffinityFilter • Host within the specified IP range • --hint build_near_host_ip=<1st IP in range> --hint cidr=<subnet size> • AvailabilityZoneFilter • Hosts that are in the desired Availability Zone • --availability-zone <availability-zone>

38. PAGE 38Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • Based on already running individual VMs • SameHostFilter • Same host as another instance in a set of instances • --hint same_host=<VM ID> • DifferentHostFilter • Different host from a set of instances • --hint different_host=<VM ID> • Collocate/Distribute group of VMs • GroupAffinityFilter • Each instance in a group is on the same host • --hint group=<group name> • GroupAntiAffinityFilter • Each instance in a group is on a different host • --hint group=<group name>

39. PAGE 39Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • Based on flavor used (“nova-manage instance_type list”) • ComputeCapabilitiesFilter • Hosts with properties that match flavor extra specs (“nova hypervisor-show <host name>”) • AggregateInstanceExtraSpecsFilter • Hosts from host aggregates with properties that match flavor extra specs ( “nova aggregate-details <aggregate ID>”) • TrustedFilter • Hosts with integrity obtained from the Attestation service (Intel TXT) that matches flavor extra specs (“nova hypervisor-show <host name>”) • Based on Project (Tenant) • AggregateMultiTenancyIsolation • Hosts from host aggregates with filter_tenant_id property value containing the Project (tenant) ID

40. PAGE 40Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • Based on Host resources left • CoreFilter • Hosts with sufficient CPU cores left, uses global cpu_allocation_ratio setting to overcommit (16.0 by default) • RamFilter • Hosts with sufficient RAM left, uses global ram_allocation_ratio setting to overcommit (1.5 by default) • DiskFilter • Hosts with sufficient disk space left for root and ephemeral storage, uses global disk_allocation_ratio setting to overcommit (1.0 by default) • AggregateCoreFilter • Same as CoreFilter but uses cpu_allocation_ratio set for Host Aggregate • AggregateRAMFilter • Same as RAMFilter but uses ram_allocation_ratio set for Host Aggregate

41. PAGE 41Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • Based on Host load • IoOpsFilter • Hosts with less than max_io_ops_per_host concurrent I/O operations (8 by default) • NumInstancesFilter • Hosts with less than max_instances_per_host instances (50 by default) • Based on image used • ImagePropertiesFilter • Hosts that can support the specified image properties: architecture, hypervisor type and virtual machine mode (hypervisor ABI) • IsolatedHostsFilter • Specific set of images and a specific set of hosts (from nova.conf)

42. PAGE 42Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Filters • Scheduling history • RetryFilter • Hosts that haven’t been attempted for scheduling purposes • Custom • JsonFilter • Hosts that pass a custom filter constructed by passing a scheduler hint in JSON format • Supported operators: =, <, >, in, <=, >=, not, or, and • Supported variables: $free_ram_mb, $free_disk_mb, $total_usable_ram_mb, $vcpus_total, $vcpus_used • --hint query='[">=","$free_ram_mb",1024]‘ • Inherit from nova.scheduler.driver.Scheduler • Override schedule_prep_resize and schedule_run_instance methods

44. PAGE 44Copyright © 2014 Mirantis, Inc. All rights reserved Nova Scheduler: Weights • Weigher - host property of integer type • Every weigher can have multiplier for it • weight = sum(weigher_i * multiplier_i) • The only currently available RamWeigher: • Spread VMs across hosts evenly • ram_weight_multiplier=1.0 • Stack VMs on the hosts (fill-up) • ram_weight_multiplier=-1.0

45. PAGE 45Copyright © 2014 Mirantis, Inc. All rights reserved Step 8a: Schedule Provisioning Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Scheduler fetches information about the whole cluster from database, filters, selects compute node and updates DB with its ID

46. PAGE 46Copyright © 2014 Mirantis, Inc. All rights reserved Step 8b: Provision Scheduled Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Scheduler publishes message to the compute queue (based on host ID) to trigger VM provisioning

47. PAGE 47Copyright © 2014 Mirantis, Inc. All rights reserved Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Step 9a: Start VM Provisioning Nova Compute gets message from MQ

49. PAGE 49Copyright © 2014 Mirantis, Inc. All rights reserved Nova Compute Drivers Nova Compute XCP VM VM VMWare Power VM VM VM libvirt KVM VM VM Xen VM VM Qemu VM VM LXC Cont ainer Cont ainer Allows multiple hypervisor types per cloud. Libvirt / KVM is most commonly used in deployment Maintained by Microsoft Maintained by IBM Bare Metal Docker Contain er Contain er XenAPI Naitive support comes in IcehousePXETilera Somewhat experimental Maintained by Citrix HyperV VM VM ESXi VM VM vSphere VM VM

50. PAGE 50Copyright © 2014 Mirantis, Inc. All rights reserved Nova Compute Notes • Functionality provided by drivers is not 100% similar. • Exact "run_instance" flow depends on the driver implementation. • Most features are developed and tested on KVM.

51. PAGE 51Copyright © 2014 Mirantis, Inc. All rights reserved Nova Compute • Asks Nova Conductor for VM info from database – step 9 • Queries Neutron (previously Nova-Network in Essex) to allocate networking information – step 10 • Queries Cinder to allocate volume information (optional step for persistent data) – step 11 • Fetches VM image (base OS) from Glance – steps 12- 14 • Passes all information about VM (in a single message) to Hypervisor and Hypervisor (KVM / Xen) creates an instance – step 15

52. PAGE 52Copyright © 2014 Mirantis, Inc. All rights reserved Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Step 9b: Start VM Provisioning Nova Compute makes rpc.call to Nova Conductor for information on VM from DB

54. PAGE 54Copyright © 2014 Mirantis, Inc. All rights reserved Nova Conductor Notes • Eliminites remote DB access (security) • Horizontal scalability: spawn multiple worker threads operating in parallel (performance) • Hides DB implementation/schema from the Nova Compute (upgrades) • Possible offloading of long-running operations from other services, not just Nova Compute • Beneficial for operations that cross multiple compute nodes (migration, resizes) controller node DB nova-conductor compute node nova-compute rpc.call()

57. PAGE 57Copyright © 2014 Mirantis, Inc. All rights reserved Neutron Notes • Provides a flexible API (POST / GET) for service providers or their tenants to manage OpenStack network topologies. • Create networks, associate VMs, set routers, etc. • Presents a logical API and a corresponding plug-in architecture that separates the description of network connectivity from its implementation. • In Icehouse, one can still choose to stay with nova- network (Essex approach) or to go with Neutron.

58. PAGE 58Copyright © 2014 Mirantis, Inc. All rights reserved Step 10: Configure Network Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute makes a call to Neutron API to provision network for the instance

59. PAGE 59Copyright © 2014 Mirantis, Inc. All rights reserved Step 10: Configure Network (Continued) Neutron configures IP, gateway, DNS name, L2 connectivity, etc. Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent

61. PAGE 61Copyright © 2014 Mirantis, Inc. All rights reserved Open Stack Storage Concepts • Ephemeral storage: • Persists until VM is terminated • Accessible from within VM as local file system • Used to run operating system and/or scratch space • Managed by Nova • Block storage: • Persists until specifically deleted by user • Accessible from within VM as a block device (e.g. /dev/vdc) • Used to add additional persistent storage to VM and/or run operating system • Managed by Cinder • Object storage: • Persists until specifically deleted by user • Accessible from anywhere • Used to add store files, including VM images • Managed by Swift

63. PAGE 63Copyright © 2014 Mirantis, Inc. All rights reserved Cinder Resources • Volume: • Is a persistent R/W block storage device • Can be attached to VMs as secondary storage • Can be root store to boot VMs • Can be attached only to one instance at a time • Keeps its state independent of an instance • Snapshot: • Is a read-only point in time copy of a volume • Can then be used to create a new volume • Backup: • An archived copy of a volume

64. PAGE 64Copyright © 2014 Mirantis, Inc. All rights reserved Cinder Cinder Architecture Cinder DB Queue Cinder Volume Backend Storage Devices Scheduler Cinder API Cinder CLI NovaHorizon HTTP AMQP SQL 3rd-party Cinder Backup Object Storage

65. PAGE 65Copyright © 2014 Mirantis, Inc. All rights reserved Cinder Volume Driver • iSCSI: • Dell EqualLogic • EMC VMAX/VNX • Hitach HDS • HP 3PAR (StoreServ) • HP / Lefthand SAN (StoreVirtual) • Huawei T/Dorado/HVS • IBM Storwize family/SVC/XIV • LVM (Reference Implementation) • Nexenta • NetApp • SolidFire • VMware VMDK • Windows Server 2012 • Zadara • GlusterFS NFS (volumes as sparse files) • IBM General Parallel File System (GPFS) (volumes as sparse files): • GPFS NSD • ATA over Ethernet (AoE): • Coraid • Fibre Channel: • NetApp • HP 3PAR (StoreServ) • Huawei T/Dorad/HVS • IBM Storwize family/SVC/XIV • VMware VMDK • NFS (volumes as sparse files): • NFS • Nexenta • NetApp • VMware VMDK • Zadara • XenAPI Storage Manager • RADOS Block Devices (RBD): • Ceph • Shared SAS: • VMware VMDK • Scale Out File System (SOFS) (volumes as sparse files): • Scality • VirtIO (Local raw storage) (volumes as sparse files)

67. PAGE 67Copyright © 2014 Mirantis, Inc. All rights reserved Cinder Notes • Functionality provided by drivers is not 100% similar. • Exact volume provisioning and attachment flow depends on driver implementation and Hypervisor used. • Most features are developed and tested with LVM over ISCSI and KVM.

68. PAGE 68Copyright © 2014 Mirantis, Inc. All rights reserved Step 11: Request Volume Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent It is assumed a volume is already created. Nova Compute contacts Cinder to get volume data. Can also attach volumes after VM is built.

69. PAGE 69Copyright © 2014 Mirantis, Inc. All rights reserved Step 11: Request volume (Continued) Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute sets up the host mount if needed & instructs the Hypervisor to use vol. as a new block device

72. PAGE 72Copyright © 2014 Mirantis, Inc. All rights reserved Definition of Image • Copy of the entire contents of a storage device, for example: • hard drive • DVD/CD • other storage media • Includes all the partition information, boot sectors, the file allocation table, operating system installation and application software.

73. PAGE 73Copyright © 2014 Mirantis, Inc. All rights reserved Custom Image Creation Steps • Get Operating System installation ISO • Create VM (qemu-img create) • Install Operating System • Start VM with ISO connected as CD-ROM • Connect to VM via VNC console or SPICE • Install Operating System as you normally do • Install and configure cloud-init (Linux) or cloudbase-init (Windows) • Install virtIO drivers (Windows only) • Prepare and cleanup the image • Remove the MAC address details, etc, with virt-sysprep (Linux) or sysprep.exe (Windows) • Install any additional applications (OpenSSH, Apache, MySQL, etc.) • Convert image to a different format if desired (qemu-img convert)

74. PAGE 74Copyright © 2014 Mirantis, Inc. All rights reserved Glance • Provides Images-as-a-Service. • Can use multiple back-ends for image storage. • Can store the same image in multiple locations. • Supports multiple image formats.

75. PAGE 75Copyright © 2014 Mirantis, Inc. All rights reserved Glance API Glance CLI NovaHorizon Glance Glance Architecture HTTP AMQP SQL 3rd-party Glance Registry Glance DB Store Adapter Swift Cinder File System Amazon S3 HTTP GridFS Sheepdog Ceph (RBD)

76. PAGE 76Copyright © 2014 Mirantis, Inc. All rights reserved Glance Notes • Functionality provided by backends is not 100% similar. • Exact Glance capabilities available depend on the backend and Hypervisor used. • Most of the features are developed and tested with the file system or Swift.

77. PAGE 77Copyright © 2014 Mirantis, Inc. All rights reserved Glance Disk Formats Disk Format Description raw an unstructured (unrestricted) disk image format vhd VHD disk format, a common disk format used by virtual machine monitors from VMWare, Xen, Microsoft, VirtualBox, and others vmdk Another common disk format supported by many common virtual machine monitors vdi disk format supported by VirtualBox virtual machine monitor and the QEMU emulator iso archive format for the data contents of an optical disc (e.g. CDROM) qcow2 disk format supported by the QEMU emulator that can expand dynamically and supports Copy on Write aki indicates what is stored in Glance is an Amazon kernel image ari indicates what is stored in Glance is an Amazon ramdisk image ami indicates what is stored in Glance is an Amazon machine image

78. PAGE 78Copyright © 2014 Mirantis, Inc. All rights reserved Step 12: Request VM Image from Glance Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute requests VM image from Glance via Image ID

79. PAGE 79Copyright © 2014 Mirantis, Inc. All rights reserved Step 13: Get Image URI from Glance Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent If image with given image ID can be found - return URI – HTTP Get URI

80. PAGE 80Copyright © 2014 Mirantis, Inc. All rights reserved Step 14: Direct Image File Copy Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute can download image using URI, given by Glance, directly from Swif

81. PAGE 81Copyright © 2014 Mirantis, Inc. All rights reserved Step 14 alternative: Image Copy through Glance Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent To leaverage Glance Server caching mechanism and additional access restriction, the Image copy can go through Glance

83. PAGE 83Copyright © 2014 Mirantis, Inc. All rights reserved Step 15: Start VM Rendering via Hypervisor In case of KVM / libvirtd this is a single XML VM config file Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute creates a command to Hypervisor and delegates VM rendering to Hypervisor.

84. PAGE 84Copyright © 2014 Mirantis, Inc. All rights reserved Step 16: VM is UP Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Nova Compute sends a message to Nova Conductor to update DB with VM state

85. PAGE 85Copyright © 2014 Mirantis, Inc. All rights reserved Step 17: User is Happy Nova Nova DB Queue Nova API Scheduler Conductor UI: Horizon or CLI Swift Object Store Proxy Server Keystone KeystoneAPI Keystone DB Glance Glance API Glance Registry Glance DB Neutron Neutron DB Queue Neutron API Scheduler Plugin/Agent Compute NodeCompute Node Network VM Hypervisor nova- compute Network Node DHCP/IPAM Router/GW Block Storage NodeStorage Cinder Cinder API Scheduler Cinder Backup Cinder DB Queue Cinder Vol Ceilometer Ceilometer API Collector Agent Horizon polls Nova API for VM status and power state, which is taken from Database.

86. PAGE 86Copyright © 2014 Mirantis, Inc. All rights reserved OpenStack Architecture Recap • Users log into Horizon and initiate a VM create • Keystone authorizes • Nova initiates provisioning and saves state to DB • Nova Scheduler finds appropriate host • Neutron configures networking • Cinder provides block device • Image URI is looked up through Glance • Image is retrieved via Swift • VM is rendered by Hypervisor