OpenStack Architecture

Copyright © 2014 Mirantis, Inc. All rights reservedCopyright © 2014 Mirantis, Inc. All rights reserved Release H2.0.3
OpenStack
Architecture
Polina Petriuk
Sr. Technical Trainer
Mirantis Inc.

Copyright © 2014 Mirantis, Inc. All rights reserved
OpenStack Architecture
Provision a VM Request Flow

VM Provisioning
• Is the most common and complex process in
OpenStack
• Involves interaction of most of OpenStack
components

Initial State
Assumes Project is created,
provisioning quota is available, user
has an access to Horizon/CLI
Cloud Operator, DevOp, etc.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Compute NodeCompute Node
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent

Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify parameters of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Neutron)
• Optional (SSH Keys, Persistent volumes, comments, etc.)
• Select "Create" button

The OpenStack Dashboard
(Horizon)
Horizon provides a
baseline user interface
for managing OpenStack services.

Horizon
• Is “stateless” — doesn’t require a database
• Delegates error handling to the back-end
• Doesn’t support all the API functions
• Can use memcached or database to store
sessions
• Gets updated via API polling

Step 1: Request VM
Provisioning via UI/CLI
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Cloud Operator, DevOp, etc.
User logs in to UI
Specifies VM params: name,
flavor, keys, etc. and hits
"Create" button

Step 1: Request Provisioning
– Under the Hood
• Form parameters are converted to POST data
• "Create" request initiates HTTP POST request
to back-end
• To Keystone if auth token is not cached – step 2

The OpenStack Identity
Service (Keystone)
Keystone provides
Identity, Token, Catalog and
Policy services
for use specifically by projects
in the OpenStack family.

Keystone:
Identity Management
• User
• Credentials
• Token
• Associated with a user, an arbitrary bit of text that is used to access
resources
• Group of users
• Project
• Synonym to tenant
• Role
• Assigned to sers or groups for projects
• Domain
• Higher level of hierarchy – users and projects belong to domains

Keystone:
Service Catalog
• Service
• An OpenStack service, such as Compute (Nova), Object Storage
(Swift), or Image Service (Glance).
• Endpoint
• A network-accessible address, usually described by URL, from where
you access an OpenStack service
• Rule
• A set of requirements for performing an action over the endpoint.

Keystone Architecture
OpenStack
Services
Catalog
Backend
Token
Backend
Policy
Backend
Assignments
Backend
Identity
Backend
Credentials
Backend
Rule management
interface and rule-
based authorization
Contains
temporary
tokens
Contains
endpoint registry
Contains users
and groups
Contains
credentials, e.g.
EC2 tokens
Contains domains,
projects, roles and
role assignments
Deploys with its own DB but
can also be substituted with
LDAP or other EASKeystone API

Keystone:Role Based Access
Control (RBAC)
1. User gets Token from Keystone.
2. Token includes the list of user Projects and Roles in
them.
3. User calls the Service specifying the Token.
4. Service interprets the Roles:
• Service consults its policy.json file.
• Policy.json specifies the list of available rules.
• "admin_required": [["role:admin"], ["is_admin:1"]],
• “owner" : [["project_id:%(project_id)s"]],
• "admin_or_owner": [["rule:admin_required"], ["rule:owner"]],
• Policy.json specifies which rules are enforced for operations and
resources.
• "volume:create": [["rule:admin_or_owner"]],

Step 2: Validate Auth Data
Horizon sends HTTP request to
Keystone. Auth info is specified
in HTTP headers.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent

Step 2: Validate Auth Data - Success
Keystone sends temporary token
back to Horizon via HTTP.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent

Step 3: Send API Request to Nova API
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Horizon sends POST request to
Nova API (signed with given token).

The OpenStack Compute API
(Nova API)
Nova API is a
RESTful API web service
which is used to interact with Nova.

Nova API
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains:
• EC2-compatible—starting to be deprecated
• Compute API—all innovation happens here
• Is the only "allowed" way to interact with Nova
• Is “stateless”

Step 4: Validate API Token
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova API sends HTTP request to
validate API token to Keystone.

Auth Token Usage
Neutron

Keystone /w PKI - Token
Validation
• User gets one-time-password on creation
• User uses it to establish a key-pair
• Public key is signed and stored on Keystone
• From this point user uses client certificate to login
• Nova API performs offline check of the validity of token using CA&Cert it has from Keystone
Keystone
Key
generate_cms_token
(meta, keystone_key)user/pass/tenant
signed_cms_token
signed_cms_token
Nova
CA&Cert from
Keystone
verify(signed_cms_token,
Cafile,certfile)

Step 4: Validate API Token - Sucess
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Keystone validates API token and
sends HTTP response with token
acceptance/rejection info.

Step 5:
Process API Request
• Validate request parameters:
• Typographical errors are verified on code level
• Cloud-related parameters are validated via database
requests
• If the request cannot be processed:
• Throw an exception
• If the request can be processed:
• Save initial state to the database

Nova Database
Nova Database stores current
state of all objects in compute
cluster.

Nova Database
• In theory, can be any relational database
• Most of the deployments are done with MySQL or
PostgreSQL
• Nova API talks to database via SQLAlchemy
(python ORM (Object Relational Mapper))
• Database HA should be done via external tools:
• Galera
• Multi-Master replication Model for MySQL (MMM)

Step 5: Process API Request
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova API parses request to
python object model and
validates it by fetching data from
Nova DB. If request is valid, it
saves initial db entry about VM to
the database.

Step 6:
Process API Request
• Validate request parameters
• Typographical errors are verified on code level.
• Cloud-related parameters are validated via database
requests.
• If request cannot be processed:
• Throw an exception.
• If request can be processed:
• Save initial state to the database.
• Send message with next actions to Message Queue
(MQ)

Message Queue
Message Queue is a unified way for
collaboration between sub-
components.

Messaging Process Example
• OpenStack uses two modes:
• rpc.cast—don't wait for result (fire and forget)
• rpc.call—wait for result (when there is something to return)
Ex. Nova API Ex. Nova Scheduler

OpenStack Messaging Notes
• Uses multiple queues within single RabbitMQ
instance.
• Used by services to build machine state
• Each compute node has a queue
• Message traffic is not intensive.
• Doesn't send broadcast messages.
• For example: monitoring uses API polling
• HA should be configured separately.
• For example: mirrored queues not handled by OpenStack

Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 6: Publish Provisioning Request
Nova API makes rpc.cast to
Scheduler. It publishes a
short message to scheduler
queue with VM info.
Request has been validated, but no
action has been taken yet, i.e. which
host, IP address, etc.

Step 7: Pick up Provisioning Request
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler picks up the
message from MQ.

Nova Scheduler
Nova Scheduler is a daemon, which
determines, on which compute
host the request should run.
• Only provisioning time component—not like VMware’s Distributed Resource
Scheduler (DRS)
• Typically co-located with the Cloud Controller

Nova Scheduler: Available
Schedulers
Scheduler Description Behavior
Chance Picks a host that is up Random
Filter
Picks the best-suited
host which satisfies
selected filter(s)
Custom Prioritisation

Nova Scheduler: Filtering
Affinity, Anti-affinity,
etc.
Eliminate
inapplicable hosts

Nova Scheduler: Filters
• All Hosts
• AllHostsFilter
• All the available hosts, even non-active
• ComputeFilter
• All hosts that are operational and enabled
• Based on Host statically configured properties
• SimpleCIDRAffinityFilter
• Host within the specified IP range
• --hint build_near_host_ip=<1st IP in range> --hint cidr=<subnet
size>
• AvailabilityZoneFilter
• Hosts that are in the desired Availability Zone
• --availability-zone <availability-zone>

• Based on already running individual VMs
• SameHostFilter
• Same host as another instance in a set of instances
• --hint same_host=<VM ID>
• DifferentHostFilter
• Different host from a set of instances
• --hint different_host=<VM ID>
• Collocate/Distribute group of VMs
• GroupAffinityFilter
• Each instance in a group is on the same host
• --hint group=<group name>
• GroupAntiAffinityFilter
• Each instance in a group is on a different host
• --hint group=<group name>

• Based on flavor used (“nova-manage instance_type
list”)
• ComputeCapabilitiesFilter
• Hosts with properties that match flavor extra specs (“nova
hypervisor-show <host name>”)
• AggregateInstanceExtraSpecsFilter
• Hosts from host aggregates with properties that match flavor extra
specs ( “nova aggregate-details <aggregate ID>”)
• TrustedFilter
• Hosts with integrity obtained from the Attestation service (Intel TXT)
that matches flavor extra specs (“nova hypervisor-show <host
name>”)
• Based on Project (Tenant)
• AggregateMultiTenancyIsolation
• Hosts from host aggregates with filter_tenant_id property value
containing the Project (tenant) ID

• Based on Host resources left
• CoreFilter
• Hosts with sufficient CPU cores left, uses global
cpu_allocation_ratio setting to overcommit (16.0 by default)
• RamFilter
• Hosts with sufficient RAM left, uses global ram_allocation_ratio
setting to overcommit (1.5 by default)
• DiskFilter
• Hosts with sufficient disk space left for root and ephemeral storage,
uses global disk_allocation_ratio setting to overcommit (1.0 by
default)
• AggregateCoreFilter
• Same as CoreFilter but uses cpu_allocation_ratio set for Host
Aggregate
• AggregateRAMFilter
• Same as RAMFilter but uses ram_allocation_ratio set for Host
Aggregate

• Based on Host load
• IoOpsFilter
• Hosts with less than max_io_ops_per_host concurrent I/O operations
(8 by default)
• NumInstancesFilter
• Hosts with less than max_instances_per_host instances (50 by
default)
• Based on image used
• ImagePropertiesFilter
• Hosts that can support the specified image properties: architecture,
hypervisor type and virtual machine mode (hypervisor ABI)
• IsolatedHostsFilter
• Specific set of images and a specific set of hosts (from nova.conf)

• Scheduling history
• RetryFilter
• Hosts that haven’t been attempted for scheduling purposes
• Custom
• JsonFilter
• Hosts that pass a custom filter constructed by passing a scheduler
hint in JSON format
• Supported operators: =, <, >, in, <=, >=, not, or, and
• Supported variables: $free_ram_mb, $free_disk_mb,
$total_usable_ram_mb, $vcpus_total, $vcpus_used
• --hint query='[">=","$free_ram_mb",1024]‘
• Inherit from nova.scheduler.driver.Scheduler
• Override schedule_prep_resize and schedule_run_instance
methods

Nova Scheduler: Weighting
RAM, CPU, etc.
Integer values

Nova Scheduler: Weights
• Weigher - host property of integer type
• Every weigher can have multiplier for it
• weight = sum(weigher_i * multiplier_i)
• The only currently available RamWeigher:
• Spread VMs across hosts evenly
• ram_weight_multiplier=1.0
• Stack VMs on the hosts (fill-up)
• ram_weight_multiplier=-1.0

Step 8a: Schedule Provisioning
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler fetches
information about the
whole cluster from
database, filters, selects
compute node and
updates DB with its ID

Step 8b: Provision Scheduled
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Scheduler publishes
message to the compute
queue (based on host ID) to
trigger VM provisioning

Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 9a: Start VM Provisioning
Nova Compute gets
message from MQ

Nova Compute
Nova Compute is a worker
daemon, which primarily creates
and terminates VMs via
Hypervisor API.

Nova Compute Drivers
Nova
Compute
XCP
VM
VM
VMWare
Power
VM
VM
VM
libvirt
KVM
VM
VM
Xen
VM
VM
Qemu
VM
VM
LXC
Cont
ainer
Cont
ainer
Allows multiple hypervisor types per
cloud. Libvirt / KVM is most
commonly used in deployment
Maintained
by Microsoft
Maintained
by IBM
Bare
Metal
Docker
Contain
er
Contain
er
XenAPI
Naitive support
comes in IcehousePXETilera
Somewhat
experimental
Maintained
by Citrix
HyperV
VM
VM
ESXi
VM
VM
vSphere
VM
VM

Nova Compute Notes
• Functionality provided by drivers is not 100%
similar.
• Exact "run_instance" flow depends on the
driver implementation.
• Most features are developed and tested on
KVM.

Nova Compute
• Asks Nova Conductor for VM info from database –
step 9
• Queries Neutron (previously Nova-Network in Essex)
to allocate networking information – step 10
• Queries Cinder to allocate volume information
(optional step for persistent data) – step 11
• Fetches VM image (base OS) from Glance – steps 12-
14
• Passes all information about VM (in a single message)
to Hypervisor and Hypervisor (KVM / Xen) creates an
instance – step 15

Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Step 9b: Start VM Provisioning
Nova Compute makes rpc.call to
Nova Conductor for information on
VM from DB

Nova Conductor
The Nova Conductor service is key
to completing no-db-compute

Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability: spawn multiple worker threads operating in parallel (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

The OpenStack Networking
(Neutron)
network-as-a-service

Neutron Notes
• Provides a flexible API (POST / GET) for service
providers or their tenants to manage OpenStack
network topologies.
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of
network connectivity from its implementation.
• In Icehouse, one can still choose to stay with nova-
network (Essex approach) or to go with Neutron.

Step 10: Configure Network
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute makes a call to
Neutron API to provision
network for the instance

Step 10: Configure Network (Continued)
Neutron configures IP, gateway,
DNS name, L2 connectivity, etc.
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent

Open Stack Storage Concepts
• Ephemeral storage:
• Persists until VM is terminated
• Accessible from within VM as local file system
• Used to run operating system and/or scratch space
• Managed by Nova
• Block storage:
• Persists until specifically deleted by user
• Accessible from within VM as a block device (e.g. /dev/vdc)
• Used to add additional persistent storage to VM and/or run operating system
• Managed by Cinder
• Object storage:
• Persists until specifically deleted by user
• Accessible from anywhere
• Used to add store files, including VM images
• Managed by Swift

The OpenStack Block Storage
(Cinder)
block storage as-a-service

Cinder Resources
• Volume:
• Is a persistent R/W block storage device
• Can be attached to VMs as secondary storage
• Can be root store to boot VMs
• Can be attached only to one instance at a time
• Keeps its state independent of an instance
• Snapshot:
• Is a read-only point in time copy of a volume
• Can then be used to create a new volume
• Backup:
• An archived copy of a volume

Cinder
Cinder Architecture
Cinder DB
Queue
Cinder Volume
Backend
Storage
Devices
Scheduler
Cinder API
Cinder CLI NovaHorizon
HTTP
AMQP
SQL
3rd-party
Cinder Backup
Object Storage

Cinder Volume Driver
• iSCSI:
• Dell EqualLogic
• EMC VMAX/VNX
• Hitach HDS
• HP 3PAR (StoreServ)
• HP / Lefthand SAN (StoreVirtual)
• Huawei T/Dorado/HVS
• IBM Storwize family/SVC/XIV
• LVM (Reference Implementation)
• Nexenta
• NetApp
• SolidFire
• VMware VMDK
• Windows Server 2012
• Zadara
• GlusterFS NFS (volumes as sparse files)
• IBM General Parallel File System (GPFS) (volumes as sparse
files):
• GPFS NSD
• ATA over Ethernet (AoE):
• Coraid
• Fibre Channel:
• NetApp
• HP 3PAR (StoreServ)
• Huawei T/Dorad/HVS
• IBM Storwize family/SVC/XIV
• VMware VMDK
• NFS (volumes as sparse files):
• NFS
• Nexenta
• NetApp
• VMware VMDK
• Zadara
• XenAPI Storage Manager
• RADOS Block Devices (RBD):
• Ceph
• Shared SAS:
• VMware VMDK
• Scale Out File System (SOFS) (volumes as sparse files):
• Scality
• VirtIO (Local raw storage) (volumes as sparse files)

Cinder Backup Drivers
• Swift
• Ceph
• IBM Tivoli Storage Manager (TSM)

Cinder Notes
• Functionality provided by drivers is not 100%
similar.
• Exact volume provisioning and attachment flow
depends on driver implementation and
Hypervisor used.
• Most features are developed and tested with
LVM over ISCSI and KVM.

Step 11: Request Volume
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
It is assumed a volume is
already created. Nova
Compute contacts Cinder to
get volume data. Can also
attach volumes after VM is
built.

Step 11: Request volume (Continued)
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute sets up the
host mount if needed &
instructs the Hypervisor to
use vol. as a new block
device

The OpenStack Image Service
(Glance)
Glance provides services for
discovering, registering, and
retrieving virtual machine images.

Definition of Image
• Copy of the entire contents of a storage device,
for example:
• hard drive
• DVD/CD
• other storage media
• Includes all the partition information, boot
sectors, the file allocation table, operating
system installation and application software.

Custom Image Creation Steps
• Get Operating System installation ISO
• Create VM (qemu-img create)
• Install Operating System
• Start VM with ISO connected as CD-ROM
• Connect to VM via VNC console or SPICE
• Install Operating System as you normally do
• Install and configure cloud-init (Linux) or cloudbase-init (Windows)
• Install virtIO drivers (Windows only)
• Prepare and cleanup the image
• Remove the MAC address details, etc, with virt-sysprep (Linux) or
sysprep.exe (Windows)
• Install any additional applications (OpenSSH, Apache, MySQL, etc.)
• Convert image to a different format if desired (qemu-img convert)

Glance
• Provides Images-as-a-Service.
• Can use multiple back-ends for image storage.
• Can store the same image in multiple locations.
• Supports multiple image formats.

Glance API
Glance CLI NovaHorizon
Glance
Glance Architecture
HTTP
AMQP
SQL
3rd-party
Glance Registry
Glance DB
Store Adapter
Swift
Cinder
File System
Amazon S3
HTTP
GridFS
Sheepdog
Ceph (RBD)

Glance Notes
• Functionality provided by backends is not 100%
similar.
• Exact Glance capabilities available depend on
the backend and Hypervisor used.
• Most of the features are developed and tested
with the file system or Swift.

Glance Disk Formats
Disk Format Description
raw an unstructured (unrestricted) disk image format
vhd VHD disk format, a common disk format used by virtual machine monitors from VMWare, Xen,
Microsoft, VirtualBox, and others
vmdk Another common disk format supported by many common virtual machine monitors
vdi disk format supported by VirtualBox virtual machine monitor and the QEMU emulator
iso archive format for the data contents of an optical disc (e.g. CDROM)
qcow2 disk format supported by the QEMU emulator that can expand dynamically and supports Copy
on Write
aki indicates what is stored in Glance is an Amazon kernel image
ari indicates what is stored in Glance is an Amazon ramdisk image
ami indicates what is stored in Glance is an Amazon machine image

Step 12: Request VM Image from Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute requests VM
image from Glance via Image ID

Step 13: Get Image URI from Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
If image with given image ID
can be found - return URI –
HTTP Get URI

Step 14: Direct Image File Copy
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute can download
image using URI, given by
Glance, directly from Swif

Step 14 alternative: Image Copy through Glance
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
To leaverage Glance Server caching
mechanism and additional access restriction,
the Image copy can go through Glance

Step 15: Start VM Rendering via Hypervisor
In case of KVM / libvirtd this is
a single XML VM config file
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute creates a
command to Hypervisor and
delegates VM rendering to
Hypervisor.

Step 16: VM is UP
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Nova Compute sends a
message to Nova
Conductor to update
DB with VM state

Step 17: User is Happy
Nova
Nova DB
Queue
Nova API
Scheduler
Conductor
UI: Horizon or CLI
Swift
Object Store
Proxy Server
Keystone
KeystoneAPI
Keystone DB
Glance
Glance API
Glance
Registry
Glance DB
Neutron
Neutron DB
Queue
Neutron API
Scheduler
Plugin/Agent
Network
VM
Hypervisor
nova-
compute
Network Node
DHCP/IPAM
Router/GW
Block Storage
NodeStorage
Cinder
Cinder API
Scheduler
Cinder
Backup
Cinder DB
Queue
Cinder Vol
Ceilometer
Ceilometer
API
Collector
Agent
Horizon polls Nova API
for VM status and power
state, which is taken
from Database.

OpenStack Architecture
Recap
• Users log into Horizon and initiate a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Neutron configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift
• VM is rendered by Hypervisor

Questions?

Thank You!

OpenStack Architecture

More Related Content

What's hot

Viewers also liked

Similar to OpenStack Architecture

More from Mirantis

Recently uploaded

OpenStack Architecture