Some Advanced OpenStack Overview Document

1
ADVANCED OPENSTACK
ADMINISTRATION

2
AGENDA
 Getting to know OpenStack
 Horizon Dashboard & OpenStack Client CLI
 Keystone Identity Management
 Glance – Image Management
 Nova – Compute Service
 Cinder – Block Storage
 Neutron – Networking
 Troubleshooting

3
I. Getting to know OpenStack

4
1. OpenStack Overview & History
2. OpenStack Infrastructure as a Service – Cloud User
Perspective
3. High Level Overview of OpenStack Architecture
4. Architecting & Implementing OpenStack Deployment

5
Objective
• What is OpenStack?
• OpenStack Community & Foundation
• Who is using OpenStack and how?
• Getting the Software
OpenStack Overview &
History

6
• Cloud Operating System
• Abstraction Layer for Infrastructure as a Service
• Collection of Projects implementing IaaS Services
• Well-defined set of Common APIs
• Self-service & Real-time Automation of Cloud Resources
• Programmable Infrastructure for Infrastructure as Code
What is OpenStack?

7
• Open Source
• No, Enterprise Editions
• Apache License 2.0
• Open Design
• Community controlled Design Process
• Project Teams Gatherings open to Everyone
• Open Development
• Everyone can participate at every stage
• Open Community
• Everyone has a voice
OpenStack – Open Sources Project

8
• Board of Directors
• Technical Committee
• Active Technical Contributors (ATCs)
• User Committee
• Active User Contributors (AUCs)
• Project Teams:
• Project Team Leaders (PTLs)
• Active Project Contributors (APCs)
• Core Team Members
OpenStack Foundation

12
OpenStack Community
• Individual Members
• Corporate Members: Platinum, Gold, Sponsors
• Community Open Communication:
• IRC Channels
• Wiki Pages
• Etherpads
• Mailing Lists
• Community Events:
• Summit
• Project Team Gathering
• OpenStack Day & MeetUp
• User Groups

13
OpenStack Publications
• Docs
• Guides
• Superuser
• OpenStack Foundation YouTube Channel
• Ask OpenStack
• The OpenStack Blog & News
• Analyst Reports

14
Contributing to OpenStack
• Code and Documentation
• Events – Help to organize
• User Groups – Join & make them Alive
• Users – Share your Experience & Advocate New Features

15
Typical OpenStack Deployments
• Private Cloud:
• Big Data, DBaaS, High Performance Computing, Enterprise Apps, Container Optimized, DevOps
• Public Cloud:
• Web Applications & Hosting, eCommerce, HPC
• Telco Cloud:
• NFV & SDN, Video Processing & Content Delivery

16
OpenStack Foundation User Survey

17
OpenStack Foundation User Survey

18
OpenStack Users
• Academic/Research & Government – CERN, Postal Savings Bank of China
• Cloud Providers – Rackspace, T-Systems Open Telecom Cloud
• Telecom Carriers – AT&T, Verizon, China Mobile
• Finance – Insurance Australia Group, BBVA, China UnionPay
• Information Technology – Tencent, City Network, Liveperson
• Manufacturing – Volkswagen AG, JFE Steel Corporation, SGCC
• Retail/eCommerce – Snapdeal, Nike, eBay, GAP Inc, Bloomberg

19
Getting OpenStack Software
• Community Version:
• Installation Tutorials for Ubuntu, Red Hat an SUSE Linux
• 3-5 Standard Servers at minimum
• Start with Core Projects, extend with Optional Projects
• Community Support
• Requires Skills & Experience
• Commercial Releases:
• Red Hat, Cannonical, SUSE, Huawei, Mirantis, IBM, Oracle, VMWare Integrated OpenStack
• Bug fixing and Security Hardening for Enterprise Grade
• Only a handful of Supported Modules
• Vendor Support, Architecture & Deployment Services
• Vendor System Management Services

20
Objective
• High Level Overview of OpenStack Services
• Consumption Models
• Detailed Review of Services
OpenStack Infrastructure as a
Service

21
Using OpenStack Services
• Manual, On-demand:
• Horizon Dashboard – GUI available via Web Browser
• OpenStack Client – Command Line Interface available on Linux, Windows and MacOS
• Cloud Management Platforms – ManageIQ/CloudForms, InContinuum Cloud Controller, Scalr
• Automated, Infrastructure-as-Code:
• Heat Orchestration Templates
• Ansible Playbooks
• Puppet Modules
• Chef Cookbooks
• Shell Scripts with OpenStack Client

22
Compute Services
• Major Compute Abstractions available in OpenStack:
• Virtual Machine
• Bare Metal Server
• Container Orchestration Engine
• Container (Docker)
• Other Abstractions in Compute Services:
• Image, Flavor, Key Pair, Server Group, Availability Zone, Host Aggregate, Cluster, Cluster Policy
• Common User Interface regardless of:
• Hypervisor type (KVM, VMware or Xen) managing virtual machine
• Power and Management (IPMI or Redfish), Boot Interface (PXE or Virtual Media)
• Particular COE (Kubernetes, Docker Swarm or Apache Mesos)

23
Compute Projects
• Nova – COA Focus Project, delivery and management of Compute Instances
• Glance - COA Focus Project, management of Instance Images
• Ironic – delivery and management of Bare Metal Compute Instances
• Magnum – provisioning and management of Compute Orchestration Engines
• Zun – delivery and management of Container Instance
• Senlin – standard OpenStack Clustering Service
• Storlets – implements Functions as a Service in context of Object Storage

24
Security, Identity & Compliance Services
• Major abstractions available in this Security Group:
• Region, Domain, Project, Group, User, Role
• Service, Endpoint, Catalog of Services
• Secret, Secret Store
• Policy
• Workload, Task, Action, Cron-trigger, Workflow Workbook
• Identity Database (LDAP, AD, SQL, … )
• Secret Store implementation: Database, HSM, …

25
Security, Identity & Compliance Projects
• Keystone – core Identity Project of OpenStack
• Barbican – secure provisioning, management and storage of secrets (password,
X.509 certificates, encryptions keys)
• Congress – Governance Service Project
• Mistral – Workflow Service Project

26
Storage, Backup & Recovery Services
• Volume, Volume Snapshot, Consistency Group, Backup
• Object, Object Container
• Share, Filesystem Snapshot
• Storage Backend implementation: Disk Array, Storage Appliance, Software Define Storage
• Backup Technology

27
Storage, Backup & Recovery Projects
• Cinder – core Block Storage Service Project
• Swift – core Object Storage Service Project
• Manila – File Share Service Project
• Karbor – deployed Application Data and Meta-Data backup & restore
• Freezer – file and filesystem Backup & Recovery Service

28
Networking & Content Delivery Services
• Network, Subnet, Port
• Floating IP, Router
• Security Group, Firewall, VPN
• Load Balancer
• Port Chain, Port Group, Port Pair, Flow Classifier
• Zones, PTR Records
• Virtual Network Function
• Underlying Physical Network technology
• Usage of SDN

29
Networking & Content Delivery Projects
• Neutron – core OpenStack Networking Project
• Designate – multi-tenant DNSaaS
• Dragonflow – distributed SDN controller for Neutron
• Kuryr – Docker network plugin
• Octavia – LBaaS Project
• Tacker – Virtual Network Function Manager Project
• Tricircle – Extending Network abstractions across multiple OpenStack instances

30
Data & Analytics Services
• Datastore Type, Database Instance, Database Configuration Group, Database Backup, Database
Replication
• Data Analytics Cluster, Cluster Template
• Database Engine (MySQL, Cassandra, CouchBase, CouchDB, DB2, MariaDB, MongoDB, Percona,
PostgresSQL, Redis, Vertica)
• Data Analytics Engine (Hadoop, Spark, Storm)

31
Data & Analytics Projects
• Trove – DBaaS Project
• Sahara – Data Analytics Project
• Searchlight – provides indexing and search capabilities across OpenStack resources

32
Application Services
• Orchestration Template, Orchestration Stack
• Application Package, Application Environment, Application Catalogue
• Message, Message Queue, Data Processing Node Group, Data Processing Cluster

33
Application Projects
• Heat – core Orchestration Project
• Zaqar – Message Queue as a Service
• Murano – Application Catalog Service
• Solum – aPaaS (Appliction Platform as a Service) Project

34
Monitoring & Metering Services
• Alarm
• Meter
• Event
• Publisher
• Transformer
• Pipeline

35
Management Projects
• Horizon – Dashboard (GUI) Service
• OpenStack Client – primary Command Line utility
• Rally – OpenStack deployment Benchmarking & Profiling Service
• Vitrage – Root Cause Analysis Service
• Watcher – Infrastructure Optimization for Projects

36
Deployment Tools
• Kolla – Container-based Deployment for OpenStack Cloud
• TripleO – OpenStack-Over-OpenStack
• OpenStack-Ansible – OpenStack Cloud installation with Ansible
• Chef OpenStack – Chef Cookbooks for OpenStack deployment
• Puppet OpenStack – Puppet Modules for OpenStack deployment
• OpenStack Charms – Juju Charms for OpenStack deployment

37
Monitoring & Metering Projects
• Ceilometer – primary Telemetry Service
• Aodh – Alarming Service
• Gnocchi – Time Based Series Database Service
• Panko – Event Storage Service
• Monasca – Highly-scalable Monitoring as a Service

38
Objective
• General OpenStack Architecture
• Anatomy of OpenStack Service
• Mapping Services to Physical Servers
• Control Plane and User Plane Components
High Level Overview of OpenStack
Architecture

39
General OpenStack Architecture

40
Anatomy of OpenStack Service

41
Launching a New Instance
#source user_x-openrc
#openstack server create --image a --flavor z --nic net-id=b new_vm

42
CLI Client
Keystone
1. Authenticate user “x”

43
CLI Client
Keystone
2. Token for user “x”

44
CLI Client
Keystone
3. Give me endpoint for Compute Service
Nova

45
CLI Client
Keystone
4. Here is the endpoint
Nova

46
CLI Client
Keystone
5. Launch new instance with image
“a”, connect to network “b”, here is
the token for user “x”
Nova

47
CLI Client
Keystone
6. Validate the token and access
permissions for user “x”
Nova

48
CLI Client
Keystone
7. Validate, OK
Nova

49
CLI Client
Keystone
8. Give me image “a”
for user “x”
Nova
Glance

50
CLI Client
Keystone
9. Validate token for
user “x”
Nova
Glance

51
CLI Client
Keystone
10. Validate, OK
Nova
Glance

52
CLI Client
Keystone
11. Here is the
image “a”
Nova
Glance

53
CLI Client
Keystone
12. Give me a port on
network “b” for user “x”
Nova
Glance
Neutron

54
CLI Client
Keystone
user “x”
Nova
Glance
Neutron

55
CLI Client
Keystone
14. Validate OK
Nova
Glance
Neutron

56
CLI Client
Keystone
15. Here is the port
Nova
Glance
Neutron

57
CLI Client
Keystone
16. Give me a volume for
user “x”
Nova
Glance
Neutron
Cinder

58
CLI Client
Keystone
user “x”
Nova
Glance
Neutron
Cinder

59
CLI Client
Keystone
18. Validate OK
Nova
Glance
Neutron
Cinder

60
CLI Client
Keystone
19. Here is the
volume
Nova
Glance
Neutron
Cinder

61
CLI Client
Keystone
20. Instance spawned
Nova
Glance
Neutron
Cinder

62
Mapping Service Components to Hosts
• Control Nodes:
• 1 – 3+: Linux Hosts
• Infrastructure Components (Database, Message Queue, Memory Object Caching, HTTP Server)
• OpenStack Services app
• Compute Nodes:
• 2+: OS plus Hypervisor, Compute Service Agents, Network Agents, Telemetry Agents
• Storage Nodes:
• 2+: Linux, Block Storage Agents, Object Storage Agents, Fileshare Storage Agents, Telemetry Agents
• Network Nodes:
• 1+: Linux, L3 Agent, DHCP Agent, LBaaS, VPNaaS, FWaaS, Telemetry Agents

63
Objective
• OpenStack Infrastructure Components
• Implementing Control Plane
• Architecting Compute Resource Pool
• Implementing OpenStack Underlay Network
• Architecting and Implementing Storage Backend
• Considerations for other Services
Architecting & Implementing OpenStack
Deployment

64
OpenStack Infrastructure Components
• HTTP Server – Apache2 + WSGI, Nginx
• SQL Database – MySQL, MariaDB, PostgreSQL
• Advanced Message Queue Protocol (AMQP) Server – RabbitMQ, Qpid, Zeromq
• Memory Object Caching Server - Memcached

65
Control Plane Implementation
• Infrastructure Components
• API Endpoints
• Conductor Services
• Scheduling Services
• User Dashboard
• Authentication and Authorization for Identity Management
• Image Management Service

66
Control Plane High Availability
• Galera Cluster for Database
• HAProxy as Load Balancer for API Endpoints HA
• Systemd for local Host HA
• Pacemaker – Linux HA Technology
• Collapsed – every Component run on every node
• Segregated – every Component runs in own 3+ node cluster
• Mixed – select best strategy for each Component

67
Compute Resource Pool
• Virtual Machine Instances – Hypervisors:
• Linux KVM
• Xen
• VMware ESXi/vCenter
• Hyper-V
• Bare Metal Server Pools
• Container Instances – Docker Hosts
• Availability Zones
• Host Aggregates

68
Compute Hardware Considerations
• Instance Density & Workload Patterns:
• Popular Instance Flavors
• Server Configuration vs Cost
• COTS Rack Server vs Blade/HCI
• Power & Cooling Density
• OpenStack Overcommit Ratios:
• CPU – 16:1 (default)
• RAM – 1.5:1 (default)
• Hardware Acceleration:
• GPUs
• SSD or Flash PCI cards

69
Physical Network of OpenStack Instance
• Traffic Segregation: management, external (provider network), storage, internal user
• Hardware-based Network Security
• Redundancy – multiple NICs in Compute Nodes, switching layer redundancy
• Network Performance:
• Throughput
• Latency
• QoS
• IPv6 Support
• Network Acceleration
• Intelligent NICs
• SRIOV
• Hardware VTEPs – VXLAN Termination in Switches

70
SDN in OpenStack Implementation
• User Workload Networking:
• Direct mapping to underlying VLANs
• Virtual L2 Subnets and Tunneling: VXLAN, GRE
• Linux Networking – LinuxBridge
• Open Virtual Switch OVS
• Open Source or Commercial SDN Controller:
• Scalability
• Orchestration of existing DC Networking
• OpenContrail, OpenDaylight, VMware NSX, Cisco

71
Storage Backend Implementation
• Hardware Storage Backends:
• Disk Arrays
• Performance
• Redundancy & Availability
• Dell EMC, NetApp, HPE
• Software Defined Storage Backends
• Scalability & Flexibility
• Reliability & Availability
• LVM, Ceph, GlusterFS
• Data Path between Compute Nodes and Storage Nodes:
• Local Storage on Compute Nodes
• Fiber Channel
• Ethernet – iSCSI, FCoE

72
Other Services
• Telemetry Services (ceilometer, aodh, gnocchi, panco, monasca)
• Database Service
• Data Analytics Service
• Container Orchestration Engine Service

73
II. OpenStack Dashboard and Command Line
Clients

74
1. OpenStack Dashboard & Command Line Client
2. Dashboard & CLI Summary and Review

75
Objective
• Introduction to OpenStack Dashboard
• OpenStack Command Line Client
• Command Line Clients Configuration Options
• OpenStack command Structure and Arguments
OpenStack Dashboard & Command Line
Clients

76
Horizon Dashboard Introduction
• Available through Web Browser
• Typical URL:
• http://management_vip/horizon
• Login with Username and Password, optional Domain name
• User Menu:
• Settings
• Themes
• Project Menu:
• List of User’s projects to set Project Scope
• Main Menu:
• Project Menu
• Admin Menu
• Identity Menu
• Horizon Plugins

77
OpenStack Command Line Client
• Primary CLI command: openstack
• Project specific command:
• cinder
• glance
• keystone
• neutron
• nova
• swift

78
CLI Clients Configuration Options
• Authentication Parameters:
• Domain
• Project
• Username
• Password
• Authentication URL
• Identity API Version
• Environment Variables
• Command execution options
• Clouds: yaml File
• Token/URL Authentication Type

79
CLI Clients Environment Variables
$ export OS_USERNAME=demo
$ export OS_PASSWORD=openstack
$ export OS_PROJECT_NAME=demo
$ export OS_USER_DOMAIN=Default
$ export OS_PROJECT_DOMAIN_NAME= Default
$ export OS_AUTH_URL=http://10.0.0.11:35357/v3
$ export OS_IDENTITY_API_VERSION=3
$ openstack [command] [options]
• openrc scripts are used set Environment Variables before command execution

80
CLI Clients Execution Options
$ openstack --os-username demo
--os-password openstack
--os-project-name demo
--os-user-domain-name Default
--os-project-domain-name Default
--os-auth-url http://10.0.0.11:35357/v3
--os-identity-api-version 3
[command] [options]

81
clouds.yaml File
~/.config/openstack/clouds.yaml
clouds:
demo:
auth:
auth_url: http://10.0.0.11:35357/v3
project_name: demo
username: demo
password: openstack
region_name: RegionOne
$ openstack --os-cloud demo [command] [options]
Or
$ export OS_CLOUD=demo
$ openstack [command] [options]

82
openstack Command structure and arguments
• General structure:
$ openstack [global-options] [object] [action] [command arguments]
• where:
• [global-options] are the --os-…options, for example setting authorization parameters
• [object] – specifies what type of Service is in scope of the command, examples on next page
• [action] – specifies what we want to do with [object], examples on next page
• [command arguments] – additional parameters, necessary to perform the command

83
Command [object] and [action]
Example [object]
endpoint
flavor
keypair
image
network
router
server
user
volume
Example [action]
add
create
delete
issue
list
migrate
remove
show
start
• Example commands:
$ openstack token issue
$ openstack keypair delete my-keypair
$ openstack server start web-vm

84
openstack Command Interactive Mode
$ export OS_CLOUD=admin
$ openstack
(openstack) image list
(openstack) token issue
(openstack) exit

85
III. Keystone Identity Management

86
1. Understanding OpenStack Identity Management

87
Objective
• Keystone Overview and Architecture
• Keystone Services
• Resource Management Hierarchy
• Detailed Overview of Identity Objects
• Service Catalogue & Endpoints
Understanding OpenStack Identity
Management

88
Keystone Overview
• What it is?
• OpenStack Users – who is required to authenticate?
• What services it delivers?
• How this services are delivered?

89
Keystone Architecture
• Serving REST API calls on ports 5000 and 35357
• Universal Unique Identifiers - UUIDs
• Keystone backends
• Identity – SQL, LDAP, SAML v2, Oauth v1.0a, OpenID Connect
• Other – SQL, Memcache, Files
• Regions – each may have own Keystone Instance
• Federated Keystone – multiple Regions sharing a common Keystone Database

90
Keystone Services
• Identity – Users and Groups
• Resource – Projects (Tenants) and Domains
• Assignment - Roles
• Token
• Catalog and Endpoint
• Policy

91
Resource Management Hierarchy
Domain A
Domain B
Domain C
Group 1
Group N Role: user
Role: admin
Role: admin
X
Y
Z
V
M
1
V
M
2
V
M
3
Project Y
Project X

Domain
92
• Contains Users, Groups & Projects
Properties Actions
list
description create
show
name
set
delete
enabled

Project
93
• Contains Resources (Networks, Instances, etc)
Properties Actions
list
description create
show
name
set
delete
enabled
is_domain
domain_id

User
94
Properties Actions
list
description
create
show
name
set
delete
enabled
domain_id
project_id
password
email

Group
95
Properties Actions
list
description create
show
name
set
delete
domain_id
add user
remove user
contain user
Contains Users

96
Roles
• Implement Role Based Access Control (RBAC)
• User is assigned a Role in Project within Domain
• Role name is arbitrary, most often: admin, user/_member_
• Role names are lookup by a Service in policy.json file to determine user’s
authorization for Service actions:
“create_subnet”: “rule:admin_or_network_owner”

97
Service Catalog & Endpoints
• Service Catalog enables standardized discovery of Service API Endpoint –URLs
targets for REST calls.
• Service Properties:
• name
• type
• description
• enabled
• Service Name & Type examples:
• Keystone – identity
• Nova – compute
• Cinder – volume
• Neutron - network

98
• Endpoint Types:
• Public – for Cloud Users, URL on Public network
• Internal – for Cloud Users, URL on Internal network
• Admin – for Cloud Admins & Operators, URL on secure Internal or Admin network
• Endpoint Properties:
• Region
• Service_name
• Service_type
• Interface
• url
• Enabled
• Endpoint Examples:
http://10.0.0.11:9292
http://controller:35357/v3
http://controller:8080/v1/AUTH_%(tenant_id)s

99
• Service Operations
• create
• delete
• set
• list
• show
• Endpoint Operations
• create
• delete
• set
• list
• show

100
Operations available CLI only
• Domain Operations: create, delete, list, set, show
• Service Operations: create, delete, set
• Endpoint Operations: create, delete, set
• All operations on Other Domains (non default)

101
IV. Glance – Image Management

103
Objective
• Introduction to OpenStack Images
• Glance Overview and Architecture
• Image Types, Properties and Actions
• Downloading ready-made Cloud Images
• Creating Images
• Modifying and Converting Images
Inside of Glance

104
OpenStack Images
• Server Image is a file containing ready to run, bootable virtual disk
• Virtual Server in OpenStack is a Running Instance of an Image
• Image must include some sort of an Operating System
• OpenStack supports Linux and Windows images by default
• OpenStack Images include other mandatory software components, like cloud-init
• Image may contain middleware or application software, like Apache, MySQL and
PHP – full LAMP Stack
• Glance can store any Binary Blob as an Image

105
Glance Overview & Architecture
• Image Service manages and stores Server Images for Virtual Server Instances, Bare
Metal Server Instances and Zun Containers
Image Service API @ http://controller:9292
Glance Registry
File
system
Swift HTTP
SQL DB
Glance Backend

106
From Image Store to running Instance
Nova Compute Agent KVM + Libvirt
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Compute Node Volume Service
Image Service

107
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
boot disk
local disk
Persistent disk
/dev/sda
/dev/sdb
/dev/sdc
• Step 1: Create new Server Instance

108
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
/dev/sda
/dev/sdb
/dev/sdc
• Step 2: Get the Image

109
Nova Compute Agent
KVM + Libvirt
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
/dev/sda
/dev/sdb
/dev/sdc
base
Instanc
e disk

110
Nova Compute Agent
KVM + Libvirt
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
/dev/sda
/dev/sdb
/dev/sdc
base
Instanc
e disk

111
Nova Compute Agent
KVM + Libvirt
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
/dev/sda
/dev/sdb
/dev/sdc
• Step 3: Create local disk
base
Instanc
e disk

112
Nova Compute Agent
KVM + Libvirt
Volume API
Volume Backend
Image Service API
Image Storage
Backend
Image Service
server 1
/dev/sda
/dev/sdb
/dev/sdc
• Step 4: Attack persistent disk
base
Instanc
e disk

113
Image Type & Formats
• OpenStack Image Service supports multiple Compute Abstractions: Virtual
Machines, Bare Metal Servers and Containers
• OpenStack Image Service also stores Images for:
• Magnum – Container Orchestration Engine Service
• Trove – Database Service
• Sahara – Data Processing Service
• Tacker – NFV Orchestration Service

114
Image Type & Formats
• OpenStack Image Service supports following Disk Formats
• aki, ami, ari
• iso
• qcow2
• raw
• vdi
• vhd, vhdx
• vmdk, ova

115
Image Actions
• Create
• Delete
• List
• Show
• Set
• Unset
• Save
• Add project – make Image available in given Project
• Remove project – remove the Project from image

116
Image requirements- Linux
• Disk partitions and filesystems allow resizing on boot
• MAC addresses removed
• SSH server installed and running
• Firewall disabled
• Cloud-init installed and running
• Accept Public Key for SSH and User Data for startup script
• Paravirtualized Xen support in Linux kernel (if you plan to use Xen Hypervisor)

117
Downloading ready-made Cloud Images
• Preconfigured Cloud Images available from Major Linux Distributions: CentOS,
CirrOS, Debian, Fedora, Ubuntu, SUSE, RHEL
• Windows Cloud Images for OpenStack maintained by Cloudbase Solutions

118
Creating custom Images
Tools to create Cloud Images:
• Diskimage-builder
• Oz
• VeeWee
• Packer
• Image-bootstrap
• Imagefactory
• KIWI
• SUSE Studio
• virt-builder

119
Creating custom Images
Creation process:
1. Launch an Instance from a standard Image
2. SSH to the Instance
3. Install additional software
4. Customize System Configuration
5. Shutdown the Instance
6. Snapshot the Instance
7. Create a Volume from the Snapshot
8. Create an Image from the Volume

120
Modifying and Converting Images
Guestfish and virt-builder support extensive Image customization.
Qemu-img converts Images to formats:
• Qcow2 (KVM, Xen, QEMU)
• Qed (KVM)
• Raw
• Vdi (VirtualBox)
• Vpc (VHD – HyperV)
• Vmdk

121
Operation available in CLI only
• Download Images
• Share Images with other Projects

122
V. Nova – Compute Service

123
1. Compute Service Introduction
2. Virtual Server Instances – Deep Dive

124
Objectives
• Compute Service Architecture
• Implementation of Compute Hosts in User Plane
• Managing Flavors
• Managing Keypairs
• Security Groups
• Resource Quotas
• Manage Compute Hosts
• Compute Service Monitoring
Compute Service Introduction

125
Compute Service Architecture
Keystone
Neutron
Glance
Cinder
NovaAPI
Nova
Conductor
Nova
Scheduler
Compute
Host
Compute
Host
SQL DB

126
Compute Hosts in User Plane
Keystone
Neutron
Glance
Cinder
NovaAPI
Nova
Conductor
Nova
Scheduler
nova-compute
KVM Host
Compute
Host
nova-compute
Nova ESX Host
Bare Metal Host
zun-compute
Docker Host
ESX Host1 ESX Host1 ESX Host1
vCenter
Ironic API
Ironic
Conductor
Zun API
Zun
wsproxy
$openstack server … $openstack appcontainer …

127
Managing Keypair
• Keypair – Public and Private Keys
• Public Key injected to an Instance at boot time
• Private Key stored in secured file on local system
• Private Key used by ssh command to access an Instance
• Public Key owned by User, not Project
• Key actions:
• Create
• Delete
• List
• Show

128
Managing Flavors
• Flavor is a Virtual Server Architecture Template
• Flavor Properties:
• Name
• vCPU
• RAM
• Disk
• Ephemeral
• Swap
• Rxtx
• Is_public
• Extra specs

129
Managing Flavors
• Flavor actions:
• Create
• Delete
• List
• Show
• Set
• Unset
Flavor Name vCPUs RAM Disk(GB)
m1.tiny 1 512 1
m1.small 1 2048 20
m1.medium 2 4096 40
m1.large 4 8192 80
m1.xlarge 8 16384 160

130
Security Groups
• Security Groups control incoming and outgoing network traffic of the Instance
• Each Instance has at least one Security Group assigned
• Security Group has 1 or more Rules, controlling particular traffic pattern
• Each Project has got a default Security Group, which blocks all incoming traffic and
allows all outgoing traffic
• Instances on the same Subnet communicate without limits, by default
• Set allow_same_net_traffic to false in /etc/nova/nova.conf to enforce Security
Groups for all traffic.

131
Security Groups
• Properties:
• Name
• Description
• Project_id
• Revision_number
• Created_at
• Updated_at
• Rules
• Actions:
• Create, detelet, list, show, set

132
Security Group Rules
• Rule define allowed traffic pattern
• Rules are either ingress or egress
• Rule Definition specifies Protocol – TCP/UDP, ICMP or any other IP based Protocol,
defined as Interger 0:255
• For ICMP protocol ICMP Code and ICMP Type can be specified, otherwise all ICMP
traffic is regulated.
• For TCP and UDP destination port or port range (123:125) can specified
• Remote IP defines address range in from of CIDR (10.0.1.0/24)
• Remote Group – if present, rule allows traffic with all instances having specified
Security Group assigned.

133
Security Group Rules
• Rules Properties:
• Created_at
• Updated_at
• Description
• Project_id
• Security_group_id
• Revision_number
• Direction – ingress or egress
• Ethertype – IPv4 or IPv6
• Port_range_min
• Port_range_max
• Protocol – TCP, UDP, ICMP
• Remote_group_id
• Remote_ip_prefix – for example 0.0.0.0/0 for all IP addresses

134
Resource Quotas
• OpenStack maintains basic control mechanism of workload size – Resource Quotas
• Quotas are set by Project, can also be set by User
• There are default values assigned at Project creation
• Cloud Admin can alter specific values at any time

135
Resource Quotas
• List of Available Quotas
Compute Block Storage Network
Cores Volumes Networks
Instances Snapshots Subnets
Key-pairs Backup Routers
Ram Gigabytes Subnetpools
… … …

136
Resource Quotas
• Quota Operations:
• openstack quota set
• openstack quota show
• nova quota-show --user <projectUser> --tenant <project_id>
• Show current Quota usage for the Project
• nova limits --tenant <project_id>

137
Manage Compute Hosts
• Compute Host must be disabled for Maintenance
openstack compute service set --disable --disable-reason “maintenance” coa-lab nova-compute
• Enable Compute Host after Maintenance
openstack compute service set --enable coa-lab nova-compute
• Instances running on Compute Host can be:
• Live Migrated: opentack server migrate <instance> --live TargetHost
• Evacuated: nova evacuate <instance_id> TargetHost

138
Monitor Compute Service
• List all Hosts and their Services
$ openstack host list
• Show Host statistics
$ openstack host show coa-lib
$ openstack hypervisor show coa-lab
• Show Instance statistics
$ openstack server list
$ openstack diagnostics myInstance
• Summary Usage Statistics for Projects
$ openstack usage list

139
Objectives
• Instance Overview
• Instance Properties
• Launching a New Instance
• Star, Stop, Delete and other Actions
• Manage NICs & IP Addresses of an Instance
• Accessing an Instance with Nova host consoles
• Manage Instance Snapshots
Virtual Server Instance

140
Instance Overview
• OpenStack Instance is based on Linux Virtual Machine Model, extended to cover
Bare Metal Servers.
• Virtual Server Instance – “hardware” configuration:
• Virtual CPUs (Cores)
• Virtual RAM
• Virtual I/O Devices: Disks (Block Volumes), Network Cards (vNICs)
• Nova Scheduler selects Compute Node to run an Instance, based on:
• Flavor – “hardware” configuration
• Properties (architecture,…) and Metadata (GPU required, …)

141
Instance Properties
• Name– OpenStack Instance Name
• OS-EXT-SRV-ATTR:instance_name – Instance name in hosting Hypervior
• Image – name of the image used to create the Instance
• ….

142
• Minimum parameters:
• Name
• Flavor
• Image
• Network
• Optional parameters:
• Security Groups
• Keypair
• User Data
• Files to inject at boot time
• Server Groups
• Availability Zone
• Scheduler Hints
• Metadata

143
Cloud-init & User Data
• Check Cloud Init Community Page
• User Data can be a shell script, include file or list of HTTP URLs to download and
interpret
• To run a shell script at rc.local boot sequence, put a first line:
#! /bin/sh
• Place an arbitrary file in the new instance with:
$ openstack server create … --file destination-file=source-file

144
Instance Actions
• Basic Instance Management Actions:
• Create
• Delete
• List
• Show
• Set
• Unset
• Block Instance Access to Admin Users:
• Lock
• Unlock

145
Instance Actions
• Virtual Machine State Management:
• Start, stop, reboot (hard, soft)
• Pause, unpause
• Suspend, resume
• Shelve, unshelve

146
Instance Actions
• Virtual Machine Configuration Management Actions:
• Resize
• Rescue, unrescue
• Rebuild
• Migrate

147
Instance Actions
• Instance Data Management Actions:
• Add volume, remove volume
• Dump create
• Image create

148
Manage Instance NICs & IP Addresses
• Instance must have at least one Virtual Network Interface Card (vNIC), can have
many
• Additional vNIC can be attached during Launch or later
• Each vNIC must be connected to a Network or Port
• Attached vNICs must be configured in Instance’s Operating System
• Each vNIC has got unique MAC and IP addresses
• IP address types:
• DHCP
• Fixed IP
• Floating IP

149
Instance Actions
• Instance Network Management Actions:
• Add fixed ip, remove fixed ip
• Add floating ip, remove floating ip
• Add security group, remove security group

150
Accessing Instances
• Use Nova Console:
• Horizon Dashboard
• CLI:
$ openstack console url show myFirstInstance --novnc
• View console log of instance
$ openstack console log show myFirstInstance
• Security Groups (allow SSH), Floating IP

151
Manage Instance Snapshots
• Instance Snapshot preserves its boot disk state only
• Snapshot stopped Instance, or make sure to flush buffers and synchronies disk
• Create Snapshot in Horizon Dashboard or CLI:
$ openstack server image create --name <snap_name> <instance>
• Instance Snapshot creates actual Boot Volume Snapshot and a New Image based
on this Snapshot
• Cloud User can immediately launch a new Instance with Instance Snapshot

152
VI. Cinder – Block Storage

153
1. Understanding Block Storage

154
Objectives
• Block Storage Service Architecture
• Block Storage Backends
• Managing Volume Quotas
• Creating and Managing Volumes
• Attaching Volume to Instance
• Volume Snapshots
• Volume Backups
• Encrypted Volumes
• Block Storage Monitoring
Understanding Block Storage

155
Architecture of Block Storage Service
CLI Client Cinder-API
Cinder
Scheduler
Cinder
Volume
Cinder
Backup
VM Instance
Driver Driver
Swift
Container
SQL Database
Message Queue
Data Paths
Storage
Backend (SAN)

156
Block Storage Backends
• Cinder support multiple Backends Drivers:
• Ceph RADOS
• LVM
• Dell EqualLogic
• EMC VNX, VMAX, ScaleIO and XtremIO
• Fujitsu
• Hitachi
• HPE
• IBM
• NetApp
• Pure Storage
…

157
Block Storage Backends
• Cinder-Volume exposes Storage Pools to enable Volume Scheduling to specific
Storage Devices and/or Locations
• Storage Pools are defined by Cloud Operator in cinder.conf file:
enabled_backends = lvm, lvm-2
[lvm]
volume_group=cinder-volumes
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name=LVM
[lvm-2]
volume_group=cinder-volumes-2
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name=LVM-2

158
Managing Volume Quotas
• To show Block Storage related Quota:
$ cinder quota-show <project>
$ cinder quota-usage <project>
• To set Quota for a Project
$ openstack quota set --snapshots 50 <project>

159
Creating and Managing Volumes
• Volume Properties
• Name
• Description
• Availability_zone
• Os-vol-host-attr:host
• Bootable
• Encrypted
• Properties
• Size
• Status
• Type
• Volume Actions
• Create
• Delete
• List
• Show
• Set
• Unset

160
• Volume can be created as Empty Volume:
$ openstack volume create … <name>
• Volume can be created by cloning
• Another Volume: - - source <volume>
• A Volume Snapshot: - - snapshot <snapshot>
• An Image: - - image <image>
• Volume can be migrated
$ openstack volume migrate - - host host@backend#pool <volume>
• Volume resize (only increase):
$ openstack volume set - - size 15 <volume>

161
• Attach a Volume to an Instance:
$ openstack server add volume - - device /dev/vdb <instance> <volume>
• Login to an Instance, check if device available:
# fdisk -l
• Format the device
# mkfs.ext3 /dev/vdb
• Mount the device
# mount /dev/vdb /mnt
• Unmount the device
# umount /dev/vdb
• Detach the volume from an Instance
$ openstack server remove volume <instance> <volume>

162
• Volume can be transferred from one owner to another with Transfer Request
• Initiate the transfer:
$ cinder transfer-create - - name <name> <volume>
• Accept the transfer:
$ cinder transfer-accept <transfer> <auth_key>

163
Volume Snapshots
• Volume Snapshot Properties:
• Description
• Name
• Properties
• Size
• Status
• Volume_id
• Volume Snapshot Actions:
• Create
• Delete
• List
• Show
• Set, unset

164
Volume Snapshots
• To create a Volume Snapshot:
$ openstack snapshot create - - force - - volume <volume> <snap_name>
• There is NO Snapshot Restore “in place”
• Snapshot can be cloned into a New Volume:
$ openstack volume create … - - snapshot <snapshot> … <name>

165
Volume Backups
• Volume Backup is a copy of Volume Data, stored in other location
• Cinder Volume Backup drivers:
• Ceph
• NFS
• Filesystem
• Swift
• Google Cloud Storage
• IBM Tivoli

166
Volume Backups
• Backup can be created from Inactive/Detached Volume, or - - force option is
required
• Backup can be created from specific Snapshot of The Volume, - - snapshot
option
• Restore the Backup:
$ openstack volume backup restore <backup> <volume>
• <volume> can be any volume with matching size

167
Encrypted Volumes
• Block Volume Service supports Volume encryption on per-tenant (project) basis
• Volume Encryption encodes data stored in Backend, but also encodes data
transmitted between Backend and an Instance
• Volume Encryption can be integrated Key Management Service (Barbican)

168
Encrypted Volumes
• To encrypt Volumes, Cloud Admin has to create Encrypted Volume Type:
$ openstack volume type create LUKS
• And add encryption details to the Volume Type:
$ cinder encryption-type-create - - cipher aes-xts-plain64 - - key_size 512 - - control_location
front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
• Create encrypted Volume
$ openstack volume create - - size 1 - - type LUKS <volume>

169
Block Storage Monitoring
• Status of Block Volume Service:
$ openstack volume service list
• Display Backend Host Capabilities:
$ cinder get-capabilities host@backend
• Display Pools details:
$ cinder get-pools - - detail

170
VII. Neutron – Networking

171
1. Network Service Architecture
2. Virtual Network Resources

172
Objectives
• Network Service Architecture
• Neutron Implementation with LinuxBridge
• Network flows
Network Service Architecture

173
OpenStack Virtual Network Overview
Subnet 1
192.168.1.0/24
Subnet 3
192.168.3.0/24
Network 1
Subnet 2
192.168.2.0/24
Network 2
Subnet 4
192.168.4.0/24
Provider Network
VM1 VM2
SG1
SG2
VM3 SG3
Internet
vRouter 1 vRouter 2
Layer 2 Connectivity

174
OpenStack Virtual Network Overview
Subnet 1
192.168.1.0/24
Subnet 3
192.168.3.0/24
Network 1
Subnet 2
192.168.2.0/24
Network 2
Subnet 4
192.168.4.0/24
Provider Network
VM1 VM2
SG1
SG2
VM3 SG3
Internet
vRouter 1 vRouter 2
Layer 3 Addressing
Floating IP

175
Network Service Architecture
Internet
Network Node
Control Node
Compute Node
Neutron-server
Cinder
Keystone
Nova
Glance
Neutron-plugin-agent
L3 Agent
Neutron-metadata-
agent
DHCP Agent
LBaaS, FWaaS
Nova-compute
Neutron-plugin-agent
Management Network
API Network
Tenant Network
External Network

176
Implementation with LinuxBridge
Internet
Network Node Compute Node 2
DHCP
Namespace
qdhcp1
External
Bridge
Router
Namespace
qrouter2
Router
Namespace
qrouter1
Management Network
Tenant Network
External Network
Compute Node 1
eth1 eth1 eth1
br-int br-int br-int
eth2
DHCP
Namespace
qdhcp2
qg qr qg qr
Tunnel
Bridge
Tunnel
Bridge
VXLAN Iface VXLAN Iface
Port tap
VM1 VM2
iptables2
Tunnel
Bridge
VXLAN Iface
eth0 eth0
tap0
iptables1
Tunnel
Bridge
VXLAN Iface
tap0
VM4
iptables3 iptables4
Tunnel Bridge
VXLAN Iface
eth0
tap0
VM3
eth0
tap1
eth0 eth0 eth0

177
Objectives
• Network & Subnets
• Routers
• Floating Ips
• Network Service Quotas
• Network Interfaces in Instance
• Monitoring Network Service
Virtual Network Resource

178
Network & Subnets
• Self-service Networks – Project Networks – enable regular Cloud Users to
provision basic network connectivity on demand, in real time, without Network
Admin invention
• Provider Networks – External Networks – map to existing physical networks in
the Data Center, provide L2 and L3 connectivity between Self-service Network
and Internet
• Neutron (Virtual) Network provides L2 connectivity for Instance, is a L2
Broadcast Domain, can be configured with DHCP
• Neutron Subnet is IP Address Segment, providing IPv4 and/or IPv6 address
space to the Network
• Neutron Port has a MAC & IP addresses, plus Security Groups, can be attached
to an Instance, detached to another.

179
Network & Subnets
• Network Properties:
• Name
• Description
• Project_id
• Status
• Mtu
• …
• Network Actions:
• Create
• Delete
• List
• Show
• Set
• …

180
Network & Subnets
• Subnet Properties:
• Name
• Description
• Project_id
• Network_id
• CIDR
• …
• Subnet Actions:
• Create
• Delete
• List
• Show
• Set
• Unset
• …

181
Network & Subnets
• Create a Project/Tenant Network
$ openstack network create - - dns-namespace 8.8.8.8 project-private-net1
• Create an External Network
$ openstack network create - - provider-physical-network provider – provider-network-type flat –
external - - share public-net
• Create a Subnet
$ openstack subnet create - - network project-private-net1 – subnet-range 192.168.100.0/24
private-net1-subnet

182
Routers
• Router provides L3 connectivity between Networks
• Two major types:
• East-West Router – connects 2 or more Project Networks
• Nort-South Router – connects 1 or more Project Networks to an External Network
• Nort-South Router has an External Gateway Interface, connected to External
Network
• Nort-South Router perform SNAT on internal addresses
• Instance can become an IP destination with Floating IP only, which is One-to-
One NAT

183
Routers
• Router Properties:
• Name
• Description
• Project_id
• Routes
• …
CLI command to show Router Interface
$ openstack port list - - router <router>
• Router Actions:
• Create
• Delete
• List
• Show
• Set
• Unset
• Add port
• Add subnet
• …

184
Floating IPs
• Floating IPs are required, if we want an Instance to be accessible from External
Network
• Floating IP is simply a Public IP address
• Cloud Operator, creating External Provider Networks defines Public Address
pools in their Subnets.
• Floating IP must be first allocated to a Project:
$ openstack floating ip create <external_net>
• Cloud User can attach allocated Floating IP to an Instance
$ openstack server add floating ip <instance> <floating_ip>

185
Network Service Quotas
• Network related Quotas
• FloatingIP
• Network
• RBAC_policy
• Router
• Security_group
• Security_group_rule
• Subnet
• Subnetpool

186
Network Interfaces in Instances
• New instance must have at least one
• Default Operating System configuration of first NIC depends on the Image
• Assumes DHCP configuration for Network Interface

187
Network Interfaces in Instances
• Add a new vNIC to the Instance in Horizon Dashboard
• Add a new vNIC to the Instance on Command Line:
$ nova interface-attach --net-id …
• Login to the Instance, edit /etc/network/interfaces to add eth1:
auto eth1
iface eth1 inet dhcp

188
Monitoring Network Service
• Basic Sanity Check of Network Service:
# openstack network agent list
• Networks and IP Namespaces:
# openstack network list
# ip netns | grep dhcp
• Routers and IP Namespaces:
# openstack router list
# ip netns | grep router
• IP Availability:
# openstack ip availability list

189
Monitoring Network Service
• Neutron Log files located in /var/log/neutron
• neutron-server.log
• neutron-metadata-agent.log
• neutron-linuxbridge-agent.log
• neutron-linuxbridge-cleanup.log
• neutron-dhcp-agent.log
• neutron-l3-agent.log

190
IX. OpenStack Troubleshooting

191
1. Troubleshooting overview

192
Objectives
• Configuration Files
• OpenStack Services
• Log Files
• Central Database
• OpenStack Message Queues
• Quotas Verification
• Troubleshooting Server Instance
• Troubleshooting Network Connections
Troubleshooting Overview

193
OpenStack Configuration Files
• OpenStack Infrastructure Configuration Files:
• NTP Time Server: /etc/chrony/chrony.conf
• MariaDB: /etc/mysql/mariadb.conf.d/99-openstack.cnf
• Memcached: /ect/memcached.conf
• Apache: /etc/apache2/apache2.conf

194
OpenStack Configuration Files
• OpenStack Services Configuration Files:
• Keystone: /etc/keystone/keystone.conf
• Glance: /etc/glance/glance-api.conf; /etc/glance/glance-registry.conf
• Nova: /ect/nova/nova.conf; /ect/nova/nova-compute.conf
• Neutron: /etc/neutron/neutron.conf
• Cinder: /ect/cinder/cinder.conf
• Swift: /ect/swift/swift.conf
• Horizon: /etc/openstack-dashboard/local-settings.py

195
OpenStack System Service
• Most of OpenStack Services Server Software run as System Services
• To check Service Status:
# systemctl status <service_name>
• To start System Service:
# systemctl start <service_name>

196
• OpenStack Infrastructure
• NTP Time Server: chrony
• MariaDB: mysql
• RabbitMQ: rabbitmq-server
• Memcached: memcached
• Apache2: apache2

197
• OpenStack Services:
• Keystone: run as mod_wsgi in Apache2, no System Service
• Glance: glance-api, glance-registry
• Nova: nova-api, nova-consoleauth, nova-scheduler, nova-conductor, nova-novncproxy, nova-
compute
• Neutron: neutron-server, neutron-linuxbridge-agent, neutron-dhcp-agent, neutron-metadata-
agent, neutron-l3-agent
• Horizon: run as mod_wsgi in Apache2, no System Service
• Cinder: cinder-scheduler, cinder-api, cinder-volume
• Swift: rsync, swift-proxy, swift-account, swift-container

198
• Quick check of OpenStack Services:
# openstack service list –long
# openstack compute service list
# neutron agent-list
# cinder service-list

199
OpenStack Log Files
• Most of OpenStack Infrastructure Services write Log files to /var/log:
/var/log/apache2/access.log
/var/log/apache2/error.log
/var/log/libvirt/qemu/instance-*.log
/var/log/mysql/error.log
/var/log/rabbitmq/rabbit.log

200
OpenStack Log Files
• OpenStack Services write Log files to:
• Keystone: /var/log/apache2/keystone*.log
• Cinder: /var/log/cinder/cinder*.log
• Glance: /var/log/glance/glance*.log
• Neutron: /var/log/neutron/neutron*.log
• Nova: /var/log/nova/nova*.log
• Swift: /var/log/syslog

Some Advanced OpenStack Overview Document

More Related Content

Similar to Some Advanced OpenStack Overview Document

Recently uploaded

Some Advanced OpenStack Overview Document

Editor's Notes