0
HA for OpenStack:
Connecting the dots
Raghavan “Rags” Srinivas
Rackspace

OpenStack Meetup,
Boston on Feb. 19th 2014
Rags
• 
• 
• 

Solutions Architect at Rackspace for OpenStack-based Rackspace Private Cloud
Speaker at JavaOne, RSA confer...
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
OpenStack Design Tenets
•  Scalability and elasticity are our main goals
•  Any feature that limits our main goals must be...
What is HA?
• 
• 
• 

Minimization of system downtime
Minimization of data/transaction loss
In case of multiple (or interr...
Implementing HA
•  Elimination of Single Point of Failure (SPOFs)
•  Redundancy of network components such as switchers an...
Components (High Level)
Client
VIP
NODE 1

NODE 2

Replication Services

Replication Services

Health Check

Health Check
...
Concepts
State

Description

• There is no dependency between requests
Stateless

• No need for data replication/synchroni...
More Concepts
Terminology

Description

Failover

Migration of a service from the “primary” to the
“secondary”

Failback

...
Much more concepts
Active/Passive

Active/Active

o  There is a single master

o  Multiple masters

o  Load balance statel...
HA for OpenStack
•  OpenStack APIs (nova, cinder, etc.)
•  RabbitMQ
•  MySQL
•  Cinder, Swift, and so on
•  Heat (still Wo...
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
HA on OpenStack
• 

Overall Philosophy (Don’t reinvent the wheel)

• 
• 
• 

Leverage time-tested Linux utilities such as ...
Keepalived
• 
• 
• 

Based on Linux Virtual Server (IPVS) kernel module providing layer 4 Load
Balancing
Implements a set ...
HAProxy
• Load Balancing and Proxying for HTTP and TCP
Applications

• Works over multiple connections
HA with Keepalived, VRRP &
HAProxy
Application
VRRP

Network Layer

Host1
HAProxy

Application Layer

Realserver1

Host2
K...
HA on Rackspace Private
Cloud
INTERNET

Controller 1

VIP(Keepalived, VRRP)
HAProxy

Active-Passive Infrastructure service...
HA on Rackspace Private
Cloud (switchover)
INTERNET
VIP(HAProxy)
Controller 2

Controller 1
Active-Passive Infrastructure ...
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
RabbitMQ HA Options
•  Health Check without Clustering
•  Clustering without Health Check
•  Health Check and Clustering
RabbitMQ HA

Ethernet
VRID 13
192.168.236.199
Master (Active)
Controller 1
VRID 13
IP address:
192.168.236.11

Backup (Pas...
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
MYSQL HA: MASTER/MASTER REPLICATION
MySQL – Master/Master
Replication

Ethernet
VRID 12
192.168.236.198
Master (Active)

Backup (Passive)

MySQL
Controller 1
...
MySQL – Master/Master
Replication simplified
MYSQL HA: COROSYNC, PACEMAKER AND DRBD
Pacemaker, Corosync and DRBD

Image from: http://dev.mysql.com/doc/refman/5.0/en/ha-drbd.html"

RACKSPACE® HOSTING

|

WWW...
Pacemaker, Corosync,
DRBD
Pacemaker

Corosync

DRBD

High availability and load
balancing stack for the Linux
platform

To...
DRBD
Service

Service

FILE SYSTEM

FILE SYSTEM

BUFFER CACHE

BUFFER CACHE

DRBD

RAW DEVICE

RAW DEVICE

TCP/IP

TCP/IP
...
MYSQL HA: GALERA
Galera
CLIENTS

•  Synchronous multi-master cluster
technology for MySQL/InnoDB
•  MySQL patched for wsrep (Write Set
REPl...
Multi-master replication
•  Based on Optimistic Concurrency Control
•  In case of two transactions modifying the same row ...
Multi-master Replication

read & write

read & write

MySQL

read & write

Multi-master cluster looks
like one big databas...
Multi-master conflicts

write

write

MySQL

MySQL

GALERA REPLICATION

MySQL
Multi-master conflicts

write

write

MySQL

MySQL

GALERA REPLICATION

MySQL

Conflict detected
Multi-master conflicts

write

OK

MySQL

MySQL

GALERA REPLICATION

MySQL

Deadlock
error
OpenStack and Galera

Image from http://www.severalnines.com/blog/clustering-mysql-backendopenstack"
Galera on Rackspace Private
Cloud/OpenStack
A How To: OFFICIALLY UNSUPPORTED
1.  Install Rackspace Private Cloud on 2 cont...
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
A PEEK INTO HA METHODS
HA methods
Vendor

Clustering/Replication
Technique

Rackspace

Keepalived, HAProxy,
VRRP, native clustering

Red Hat

Pac...
HA on the Public Cloud
Agenda	

What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
A Peek into HA Methods
Resources and Summary
HA methods
Infrastructure

Clustering/Replication
Technique

Characteristics

None required (Stateless)

•  HA also serves...
HA methods for MySQL
Clustering Method

Replication Technique

Pacemaker/Corosync/DRBD Mirroring on Block Devices

Keepali...
Resources
• 

OpenStack HA guide

• 
• 
• 

http://docs.openstack.org/high-availability-guide/content/ch-intro.html
https:...
Book
Summary
• 

In general leverage existing methods of HA

• 

There are several time-tested and more recent methods for impl...
Thank you!
Raghavan “Rags” Srinivas
Solutions Architect
Rackspace
High Availability for OpenStack
Upcoming SlideShare
Loading in...5
×

High Availability for OpenStack

5,210

Published on

The primary requirements for OpenStack based clouds (public, private or hybrid) is that they must be massively scalable and highly available. There are a number of interrelated concepts which make the understanding and implementation of HA complex. The potential for not implementing HA correctly would be disastrous.

This session was presented at the OpenStack Meetup in Boston Feb 2014. We discussed interrelated concepts as a basis for implementing HA and examples of HA for MySQL, Rabbit MQ and the OpenStack APIs primarily using Keepalived, VRRP and HAProxy which will reinforce the concepts and show how to connect the dots.

Published in: Technology

Transcript of "High Availability for OpenStack"

  1. 1. HA for OpenStack: Connecting the dots Raghavan “Rags” Srinivas Rackspace OpenStack Meetup, Boston on Feb. 19th 2014
  2. 2. Rags •  •  •  Solutions Architect at Rackspace for OpenStack-based Rackspace Private Cloud Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences Trying to help make OpenStack more App Developer friendly
  3. 3. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  4. 4. OpenStack Design Tenets •  Scalability and elasticity are our main goals •  Any feature that limits our main goals must be optional •  Everything should be asynchronous –  a) If you can't do something asynchronously, see #2 •  All required components must be horizontally scalable •  Always use shared nothing architecture (SN) or sharding –  a) If you can't Share nothing/shard, see #2 •  Distribute everything –  a) Especially logic. Move logic to where state naturally exists. •  Accept eventual consistency and use it where it is appropriate. •  Test everything RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
  5. 5. What is HA? •  •  •  Minimization of system downtime Minimization of data/transaction loss In case of multiple (or interrelated) failures, minimization of data loss is preferred over minimization of system downtime HA as Nines Downtime/Year 99% (two nines) 3.65 days 99.9% 8.76 hours 99.99% 52.56 minutes 99.999% 5.26 minutes 99.9999% (six nines) 31.5 seconds
  6. 6. Implementing HA •  Elimination of Single Point of Failure (SPOFs) •  Redundancy of network components such as switchers and routers •  Redundancy of applications and automatic service migrations •  Redundancy of storage components •  Redundancy of facilities services such as power, AC, etc.
  7. 7. Components (High Level) Client VIP NODE 1 NODE 2 Replication Services Replication Services Health Check Health Check Cluster Communication Cluster Communication
  8. 8. Concepts State Description • There is no dependency between requests Stateless • No need for data replication/synchronization. Failed request may need to be restarted on a different node. Example Apache web server, Nova API, Nova Scheduler, etc. • An action typically comprises multiple requests Stateful • Data needs to be replicated and synchronized between redundant services (to preserve state and consistency) MySQL, RabbitMQ, etc.
  9. 9. More Concepts Terminology Description Failover Migration of a service from the “primary” to the “secondary” Failback Migration of service back to the “primary” Switchover Migration is initiated manually
  10. 10. Much more concepts Active/Passive Active/Active o  There is a single master o  Multiple masters o  Load balance stateless services using a VIP and a load balancer such as HAProxy o  Load balance stateless services using a VIP and a load balancer such as HAProxy o  For Stateful services a replacement resource can be brought online. A separate application monitors these services, bringing the backup online as necessary o  Stateful Services are managed in such a way that services are redundant, and that all instances have an identical state o  After a failover the system will encounter a speed bump since the passive node has to notice the fault in the active node and become active o  Updates to one instance of database would propagate to all other instances o  After a failover the system will function in a degraded state
  11. 11. HA for OpenStack •  OpenStack APIs (nova, cinder, etc.) •  RabbitMQ •  MySQL •  Cinder, Swift, and so on •  Heat (still Work in Progress) •  Application running on OpenStack (Application dependent)
  12. 12. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  13. 13. HA on OpenStack •  Overall Philosophy (Don’t reinvent the wheel) •  •  •  Leverage time-tested Linux utilities such as Keepalived, HAProxy and Virtual IP (using VRRP) Leverage Hardware Load Balancers Leverage replication services for RabbitMQ/MySQL such as RabbitMQ Clustering, MySQL master-master replication, Corosync, Pacemaker, DRBD, Galera and so on
  14. 14. Keepalived •  •  •  Based on Linux Virtual Server (IPVS) kernel module providing layer 4 Load Balancing Implements a set of checkers to maintain health and Load Balancing HA is implemented using VRRP Protocol 1 vrrp_script rabbitmq {! script “usr/sbin/service 2 interval 5 3 weight -2 4 rise 2 5 fall -2 6 }! 7 rabbitmq-server status" # Check the service status! # check every 5 seconds! # adjust priority by -2 if OK! # required number of failures for KO switch! # required number of successes for OK switch!
  15. 15. HAProxy • Load Balancing and Proxying for HTTP and TCP Applications • Works over multiple connections
  16. 16. HA with Keepalived, VRRP & HAProxy Application VRRP Network Layer Host1 HAProxy Application Layer Realserver1 Host2 Keepalived Backup Realserver2
  17. 17. HA on Rackspace Private Cloud INTERNET Controller 1 VIP(Keepalived, VRRP) HAProxy Active-Passive Infrastructure services (MySQL, Rabbit) Active-Active Infrastructure services (API services) Heartbeat Compute Node 1 Compute Node 2 VMs Instantiated Controller 2 Redundant Active-Passive Infrastructure services Redundant Active-Active Infrastructure services Compute Node N
  18. 18. HA on Rackspace Private Cloud (switchover) INTERNET VIP(HAProxy) Controller 2 Controller 1 Active-Passive Infrastructure services (MySQL, Rabbit) Heartbeat Compute Node 1 Compute Node 2 VMs Instantiated Infrastructure services Compute Node N
  19. 19. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  20. 20. RabbitMQ HA Options •  Health Check without Clustering •  Clustering without Health Check •  Health Check and Clustering
  21. 21. RabbitMQ HA Ethernet VRID 13 192.168.236.199 Master (Active) Controller 1 VRID 13 IP address: 192.168.236.11 Backup (Passive) RabbitMQ RabbitMQ RabbitMQ Clustering Controller 2 VRID 13 IP address: 192.168.236.12
  22. 22. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  23. 23. MYSQL HA: MASTER/MASTER REPLICATION
  24. 24. MySQL – Master/Master Replication Ethernet VRID 12 192.168.236.198 Master (Active) Backup (Passive) MySQL Controller 1 VRID 12 IP address: 192.168.236.11 MySQL Master/Master Controller 2 VRID 12 IP address: 192.168.236.12
  25. 25. MySQL – Master/Master Replication simplified
  26. 26. MYSQL HA: COROSYNC, PACEMAKER AND DRBD
  27. 27. Pacemaker, Corosync and DRBD Image from: http://dev.mysql.com/doc/refman/5.0/en/ha-drbd.html" RACKSPACE® HOSTING | WWW.RACKSPACE.COM 27
  28. 28. Pacemaker, Corosync, DRBD Pacemaker Corosync DRBD High availability and load balancing stack for the Linux platform Totem single-ring ordering and membership protocol Synchronizes data at the block device Interacts with applications through Resource Agents (RA) UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker Uses a journaling system (such as ext3 or ext4)
  29. 29. DRBD Service Service FILE SYSTEM FILE SYSTEM BUFFER CACHE BUFFER CACHE DRBD RAW DEVICE RAW DEVICE TCP/IP TCP/IP DISK SCHED DRBD DISK SCHED DISK DRIVER NIC DRIVER NIC DRIVER DISK DRIVER DISK NIC NIC DISK
  30. 30. MYSQL HA: GALERA
  31. 31. Galera CLIENTS •  Synchronous multi-master cluster technology for MySQL/InnoDB •  MySQL patched for wsrep (Write Set REPlication) Transparent Connections •  Active/active multi-master topology •  Read and write to any cluster node DBMS DBMS DBMS •  True parallel replication, in row level wsrep API wsrep API wsrep API •  No slave lag or integrity issues Galera Replication
  32. 32. Multi-master replication •  Based on Optimistic Concurrency Control •  In case of two transactions modifying the same row on different nodes, one of the transactions will abort •  Victim transaction will get Deadlock Error •  Application needs to handle this error
  33. 33. Multi-master Replication read & write read & write MySQL read & write Multi-master cluster looks like one big database with multiple entry points
  34. 34. Multi-master conflicts write write MySQL MySQL GALERA REPLICATION MySQL
  35. 35. Multi-master conflicts write write MySQL MySQL GALERA REPLICATION MySQL Conflict detected
  36. 36. Multi-master conflicts write OK MySQL MySQL GALERA REPLICATION MySQL Deadlock error
  37. 37. OpenStack and Galera Image from http://www.severalnines.com/blog/clustering-mysql-backendopenstack"
  38. 38. Galera on Rackspace Private Cloud/OpenStack A How To: OFFICIALLY UNSUPPORTED 1.  Install Rackspace Private Cloud on 2 controllers with HA mode (Haproxy, Keepalived and VRRP is already installed) 2.  Install Galera (with ws-rep) on 3 separate nodes 3.  Mysqldump from controller nodes to Galera node 4.  Grant privileges to OpenStack (nova, glance, etc.) and haproxy users 5.  Update keepalived and haproxy and OpenStack configuration files on controller/compute 6.  Stop/Uninstall MySQL services on controller nodes and restart controller nodes
  39. 39. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  40. 40. A PEEK INTO HA METHODS
  41. 41. HA methods Vendor Clustering/Replication Technique Rackspace Keepalived, HAProxy, VRRP, native clustering Red Hat Pacemaker, Corosync, DRBD Cisco Keepalived, HAProxy, Galera for MySQL HP Microsoft Windows based installation with Hyper-V Characteristics •  Automatic install on 2 controller nodes via Chef recipes •  Manual installation. Fewer components to install •  Manual install, at least 3 controller nodes •  MS SQL server and other Windowsbased methods
  42. 42. HA on the Public Cloud
  43. 43. Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA Methods Resources and Summary
  44. 44. HA methods Infrastructure Clustering/Replication Technique Characteristics None required (Stateless) •  HA also serves as scale out using RabbitMQ Clustering •  RabbitMQ Clustering is setup for single/ Heat TBD •  Application Dependent (No standard MySQL Many •  Discussed later slide OpenStack APIs RabbitMQ HAProxy multiple nodes methods yet).
  45. 45. HA methods for MySQL Clustering Method Replication Technique Pacemaker/Corosync/DRBD Mirroring on Block Devices Keepalived/HAProxy/VRRP Works on MySQL master-master replication Characteristics •  Well tested, more complex to setup. •  Split brain possibility •  Simple to implement and understand. •  Works for any storage system. •  Master-master replication does not work beyond 2 nodes. Galera Based on write-set Replication (wsrep) Others MySQL Cluster, RHCS with DAS/ SAN Storage •  No Slave lag •  Needs at least 3 nodes •  Deadlock erros on hotspot rows. •  Relatively new •  Some relatively new (GTID) •  Some well test •  More complex setup
  46. 46. Resources •  OpenStack HA guide •  •  •  http://docs.openstack.org/high-availability-guide/content/ch-intro.html https://wiki.ubuntu.com/ServerTeam/OpenStackHA Other Resources •  http://www.rackspace.com/blog/implementing-high-availability-ha-for-rackspace-private-cloud/ •  http://www.rackspace.com/blog/high-availability-ha-with-galera-for-rackspace-private-cloud/https://www.hastexo.com/ •  http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/ •  http://docwiki.cisco.com/wiki/OpenStack_Havana_Release:_High-Availability_Manual_Deployment_Guide •  http://www.drbd.org/ •  http://www.codership.com/ •  http://www.severalnines.com/blog/clustering-mysql-backend-openstack •  https://wiki.openstack.org/wiki/BasicDesignTenets •  http://db.cs.berkeley.edu/papers/hpts85-nothing.pdf
  47. 47. Book
  48. 48. Summary •  In general leverage existing methods of HA •  There are several time-tested and more recent methods for implementing MySQL HA. •  Rackspace Private Cloud provides Chef cookbooks and recipes for implementing HA via Keepalived, HAProxy and VRRP. •  Galera is gaining more popularity. Since it’s Active/Active it does scale out and is HA. •  Few steps to get from Rackspace Private Cloud to MySQL with Galera (officially unsupported). •  Corosync/Pacemaker/DRBD is recommended by Oracle/MySQL. •  OpenStack HA guide goes through all these options in more detail.
  49. 49. Thank you! Raghavan “Rags” Srinivas Solutions Architect Rackspace
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×