Openstack HA

OpenStack HA、性能调优及排错技巧

 OpenStack HA
◦ HA基本概念
◦ 计算节点 HA
◦ 控制节点 HA
 OpenStack 性能调优
 OpenStack 日志分析与排错技巧

4
最小的数据丢失
最小的停机时间

 消除单点（SPOFs）
◦ 网络冗余，比如：交换机、路由器
◦ 应用服务冗余活自动服务迁移
◦ 存储冗余
◦ 基础设施冗余，比如：电源

 无状态服务（Stateless services）
◦ 请求之间没有依赖
◦ 比如: Nova API, Nova Scheduler, etc.
 有状态服务（Stateful services）
◦ 一次操作需要多个关联请求完成
◦ 比如: MySQL, Qpid, etc.
8

 Active/Passive
◦ Redundant instances of stateless services are load
balanced
◦ For Stateful services a replacement resource can be
brought online.
 Active/Active
◦ Redundant instances of stateless services are load balanced
◦ Stateful services are managed in such a way that services are redundant,
and that all instances have an identical state.
◦ Updates to one instance of a database would also update all
other instances.
9

 Failover
◦ Migration of a service from the “primary” to the
“secondary”
 Failback
◦ Migration of service back to the “primary”

 Compute HA
◦ Instance HA
 Controller HA
◦ MySQL
◦ Qpid
◦ OpenStack APIs (keystone, nova-api etc.)
◦ Nova, Neutron, Cinder, Swift, and so on
 Overview HA solution

12
Evacuation
Evacuation
 Without Shared Storage
◦ The instance will be booted
from a new disk, but will
preserve the configuration,
e.g. id, name, uid, ip...etc.
 With Shared Storage
◦ The instance will be booted
from same disk and data
will be preserved

 非常重要的一个组成
 每个OpenStack的组件都用到了数据库
 需要使用数据库记录相当多的信息

20
 Pacemaker
◦ high availability and load
balancing stack for the Linux
platform.
◦ Interacts with applications
through Resource Agents (RA)
 Corosync
◦ Totem single-ring ordering and
membership protocol
◦ UDP and InfiniBand based
messaging, quorum, and
cluster membership to
Pacemaker.
 DRBD (Distributed
Replication Block
Device)
◦ Synchronizes Data at the
block device
◦ Uses a journaling system
(such as ext3 or ext4)
http://dev.mysql.com/doc/mysql-ha-
scalability/en/ha-drbd.html

Synchronous multi-master
cluster technology for
MySQL/InnoDB
◦ MySQL patched for wsrep
(Write Set REPlication)
◦ Active/active multi-master
topology
◦ Read and write to any
cluster node
◦ True parallel replication, in
row level
◦ No slave lag or integrity
issues
22

23
Database Replication
method
Strengths Weakness/Limit
ations
Keepalived/HAPr
oxy/VRRP
Works on MySQL
master-master
replication
Simple to
implement and
understand.
Works for any
storage system.
Master-master
replication does
not work beyond
2 nodes.
Pacemaker/Coro
sync/DRBD
Mirroring on
Block Devices
Well tested More complex
to setup. Split
Brain possibility
Galera Based on write-
set Replication
(wsrep)
No Slave lag Needs at least 3
nodes. Relatively
new.
Others MySQL Cluster,
RHCS with
DAS/SAN
storage
Well tested More complex
setup.

 Pacemaker managed without clustering
 Clustered without pacemaker
 Pacemaker managed with clustering

 keystone
 glance-api
 nova-api
 cinder-api
 neutron-api
 Swift-proxy

 HAProxy
◦ Load Balancing and Proxying for HTTP and TCP Applications
◦ Works over multiple connections
◦ Used to load balance API services
 VRRP (Virtual Router Redundancy Protocol)
◦ Eliminates SPOF in a static default routed environment
 Keepalived
◦ Based on Linux Virtual Server (IPVS) kernel module to
provide layer 4 Load Balancing
◦ Implements a set of checkers to check service status and to
maintain health
◦ Leverage the VRRP Protocol to remap VIPS in event of failure
30

 Nova-schedule
 Nova-compute
 Cinder-schedule

Keystone的HA
 Keystone是整个OpenStack的关键
 负责用户管理和权限控制
 OpenStack每个组件都和Keystone有交互
 单个Keystone压力比较大
 单点

DB
Keystone
client
DB cluster
Keystone
client
Keystone
load balancer
不同的部署模式

Glance的HA
 Glance负责整个OpenStack的image管理，上传下载操作多，
对后端存储压力大
 Image文件丢失与损坏
 Glance API接收所有的外部请求，压力比较大

DB Cluster
Glance Registry
Swfit
API
API
load balancer
Glance的HA部署

Swift，Cinder以及Neutron的HA
Swift本身就自带了HA
只需要对多个Proxy-Server做负载均衡即可。
Neutron HA功能暂时还不是特别完善

Account Server
Container
Server
Object Server
Proxy Server
Load Balance
Account Server
Container
Server
Object Server
Proxy Server
Account Server
Container
Server
Object Server
Proxy Server
Swift的HA部署

Nova的HA
 Nova支持MuiltHost模式部署
 可以做HA的包括nova-api和nova-scheduler，其中，
nova-scheduler本身支持HA
 MuiltHost模式部署
 对nova-api进行负载

scheduler
compute
network
load balancer
API
scheduler
compute
network
API
scheduler
compute
network
API
scheduler
compute
network
API
DB Cluster
Nova的HA部署

 Active/active
◦ dhcp-agent / openvswitch-agent/neutron-server
 support active/ passive
◦ L3-agent and metadata-agent

42
.…
Availability
Zone 1
Dedicated Firewalls
BOND0
BOND1
BOND0
BOND1
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef Server
Recipes
Load Balancers
Redundant Network Switches
Storage
EMC, NetApp, or Solidfire Vols
BOND2
Redundant Network Switches
Inside LB VLAN
Storage Network (private)
Fixed Network (private)
Compute 1
KVM
G2
G1
G4
G3
Compute N
KVM
G6
G5
G7
BOND0
BOND1
BOND2
.…
Availability
Zone 2
BOND0
BOND1
BOND2
Compute 1
KVM
Compute N
KVM
G16
G15
G17
BOND0
BOND1
BOND2
BOND0
BOND1
BOND2
G12
G11
G14
G13
BOND2
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef Server
Recipes

 Nova configuration
 kvm

 I/O :
◦ All instances use local file system to host file systems.
Scarce resources as more instances are run per server by
increasing cores. First to be hit.
 Memory/RAM:
◦ Second factor to be hit after disks. Most VMs use RAM
more extensively than CPU. With increasing cores and
larger VMs RAM contention becomes a problem
 CPU:
◦ Usually the last to be hit. Not as much of a problem any
more because of Hyper threading and multiple cores.

 Flavors:
◦ Only allow sensible Flavors for the users. Example on a
compute node with 8 CPU cores and 96 GB ram avoid
creating a Flavors that uses 1 vCPU and 64 GB RAM
 Quotas:
◦ Used to limit the number of resources used by a particular
tenant: number of instances, block volume number and
space, or number of snapshots and images kept in Glance.
Consider the potential number of tenants and available
hardware.
 Over provisioning:
◦ Use technologies like thin provisioning, hyper threading to
over provision resources but have to be careful about
performance hits.

• KVM
• Xen
• Hyper-V
• VMWare ESXi

 Kernel I/O scheduler to “Deadline”.
◦ Default is cfq, good enough for most work loads
but for over provisioning use “deadline”
 Huge pages enabled
 Kernel same-page merging enabled (KSM)
 Hyper threading turned on
 Place guest file systems directly on hypervisor
block devices instead of in files.

 There are couple of important benefits of
Huge Pages:
• Page size is set 2MB instead of 4KB
• Memory used by Huge Pages is locked and cannot
be swapped.

 The ksm tuned process work in the following
way:
◦ scans through the memory finding duplicate pages
◦ Merges duplicate page to single page
◦ Map to all virtual machine locations
◦ Set copy on write
◦ Separate page when individual guests write to it.

 Nova的日志：/var/log/nova
 Glance的日志：/var/log/glance
 Keystone的日志：/var/log/keystone
 Cinder的日志：/var/log/cinder
 Neutron的日志：/var/log/neutron

 Email：yongluo2013@gmail.com
 Blog ： http://yongluo2013.github.com
 WeiXin：@嗨皮
 WeiBo ：@warrior_罗勇

Openstack HA

More Related Content

What's hot

Viewers also liked

Similar to Openstack HA

Recently uploaded

Openstack HA