OpenStack HA

1
HA in OpenStack
September 24th 2013

RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Agenda
• What is HA
• Compute HA
• Controller HA
• Corosync, Pacemaker and DRBD
• Galera
• HAProxy, keepalived, VRRP
• Resources and Summary
2

What is HA?
3

High Availability
4
Minimize data loss
Minimize system downtime

High Availability Concepts
• Stateless services
– There is no dependency between requests
– For example: Nova API, Nova Scheduler, etc.
• Stateful services
– An action typically comprises multiple requests
– For example: MySQL, RabbitMQ, etc.
5

High Availability Concepts
• Active/Passive
– Redundant instances of stateless services are load balanced
– For Stateful services a replacement resource can be brought online.
• Active/Active
– Redundant instances of stateless services are load balanced
– Stateful services are managed in such a way that services are redundant, and that all
instances have an identical state.
– Updates to one instance of a database would also update all other instances.
6

Compute HA
7

No vSphere Style HA with KVM
8

Server Evacuation
9
Evacuation
Evacuation
• Without Shared Storage
– The instance will be booted from a
new disk, but will preserve the
configuration, e.g. id, name, uid,
ip...etc.
• With Shared Storage
– The instance will be booted from
same disk and data will be preserved

Instance Migration
10

11
Virtualization vs. Cloud
• Virtualization needs care and feeding
– Name the VM
– Tune and groom regularly
– Feed it with good food and supplements
– Take to the vet when sick
• Cloud servers are disposable
– VMs are not unique
– Tune and groom apps not the cows
– Keep the cow upright
– Shoot the cow when it is sick

Scale Up vs. Scale Out
12
Traditional Cloud

Application HA
13

Controller HA Options
14

Pacemaker, Corosync and DRBD
15
• Pacemaker
– high availability and load balancing stack for
the Linux platform.
– Interacts with applications through Resource
Agents (RA)
• Corosync
– Totem single-ring ordering and membership
protocol
– UDP and InfiniBand based messaging,
quorum, and cluster membership to
Pacemaker.
• DRBD (Distributed Replication Block
Device)
– Synchronizes Data at the block device
– Uses a journaling system (such as ext3 or
ext4)

Galera
• Synchronous multi-master cluster
technology for MySQL/InnoDB
– MySQL patched for wsrep (Write Set
REPlication)
– Active/active multi-master topology
– Read and write to any cluster node
– True parallel replication, in row level
– No slave lag or integrity issues
16

HA with Keepalived and HAProxy
17

Keepalived, HAProxy and VRRP
• HAProxy
– Load Balancing and Proxying for HTTP and TCP Applications
– Works over multiple connections
– Used to load balance API services
• VRRP (Virtual Router Redundancy Protocol)
– Eliminates SPOF in a static default routed environment
• Keepalived
– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing
– Implements a set of checkers to check service status and to maintain health
– Leverage the VRRP Protocol to remap VIPS in event of failure
18

Sample OpenStack HA Architecture
19
.…
Availability
Zone 1
Dedicated Firewalls
BOND
0
BOND
1
BOND
0
BOND
1
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef
Server
Recipes
Load Balancers
Redundant Network Switches
Storage
EMC, NetApp, or
Solidfire
Vols
BOND
2
Redundant Network Switches
Inside LB VLAN
Storage Network (private)
Fixed Network (private)
Compute 1
KVM
G2
G1
G4
G3
Compute N
KVM
G6
G5
G7
BOND
0
BOND
1
BOND
2
.…
Availability
Zone 2
BOND
0
BOND
1
BOND
2
Compute 1
KVM
Compute N
KVM
G1
6
G15
G17
BOND
0
BOND
1
BOND
2
BOND
0
BOND
1
BOND
2
G12
G11
G14
G13
BOND
2
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef
Server
Recipes

HA on OpenStack
20

HA on OpenStack
21

22
Comparison
Database Replication
method
Strengths Weakness/Limita
tions
Keepalived/HAPro
xy/VRRP
Works on MySQL
master-master
replication
Simple to
implement and
understand.
Works for any
storage system.
Master-master
replication does
not work beyond 2
nodes.
Pacemaker/Coros
ync/DRBD
Mirroring on Block
Devices
Well tested More complex to
setup. Split Brain
possibility
Galera Based on write-
set Replication
(wsrep)
No Slave lag Needs at least 3
nodes. Relatively
new.
Others MySQL Cluster,
RHCS with
DAS/SAN storage
Well tested More complex
setup.

Resources and Contact
23

Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #openstack on webchat.freenode.net
• OpenStack HA
– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf
– https://github.com/rcbops-cookbooks/
• MySQL HA
– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-
deployment-guide/
– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html
– https://www.hastexo.com/
– http://www.drbd.org/

2
5
For More Information
You can reach me at:
Kenneth Hui
Open Cloud Architect
Rackspace
E-mail: ken.hui@rackspace.com
Twitter: @hui_kenneth
Blog: http://cloudarchitectmusings.com

OpenStack HA

More Related Content

What's hot

Similar to OpenStack HA

More from Kenneth Hui

Recently uploaded

OpenStack HA

Editor's Notes