Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
HA in OpenStack
September 24th 2013
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Agenda
• What is HA
• Compute HA
• Controller HA
• Corosync, Pacemaker and DRBD
• G...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is HA?
3
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability
4
Minimize data loss
Minimize system downtime
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Stateless services
– There is no dependency between re...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Active/Passive
– Redundant instances of stateless serv...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Compute HA
7
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
No vSphere Style HA with KVM
8
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Server Evacuation
9
Evacuation
Evacuation
• Without Shared Storage
– The instance w...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Instance Migration
10
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11
Virtualization vs. Cloud
• Virtualization needs care and feeding
– Name the VM
–...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Scale Up vs. Scale Out
12
Traditional Cloud
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Application HA
13
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Controller HA Options
14
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Pacemaker, Corosync and DRBD
15
• Pacemaker
– high availability and load balancing ...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Galera
• Synchronous multi-master cluster
technology for MySQL/InnoDB
– MySQL patch...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA with Keepalived and HAProxy
17
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Keepalived, HAProxy and VRRP
• HAProxy
– Load Balancing and Proxying for HTTP and T...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Sample OpenStack HA Architecture
19
.…
Availability
Zone 1
Dedicated Firewalls
BOND...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
20
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
21
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
22
Comparison
Database Replication
method
Strengths Weakness/Limita
tions
Keepalive...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources and Contact
23
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #ope...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
2
5
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
For More Information
You can reach me at...
26
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | W...
Upcoming SlideShare
Loading in …5
×

OpenStack HA

22,414 views

Published on

Walks through how HA is handled at the compute and controller layer within OpenStack. Presented at vBrownbag on 2013-09-24.

Published in: Technology, Business
  • Be the first to comment

OpenStack HA

  1. 1. 1 HA in OpenStack September 24th 2013
  2. 2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Agenda • What is HA • Compute HA • Controller HA • Corosync, Pacemaker and DRBD • Galera • HAProxy, keepalived, VRRP • Resources and Summary 2
  3. 3. RACKSPACE® HOSTING | WWW.RACKSPACE.COM What is HA? 3
  4. 4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability 4 Minimize data loss Minimize system downtime
  5. 5. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability Concepts • Stateless services – There is no dependency between requests – For example: Nova API, Nova Scheduler, etc. • Stateful services – An action typically comprises multiple requests – For example: MySQL, RabbitMQ, etc. 5
  6. 6. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability Concepts • Active/Passive – Redundant instances of stateless services are load balanced – For Stateful services a replacement resource can be brought online. • Active/Active – Redundant instances of stateless services are load balanced – Stateful services are managed in such a way that services are redundant, and that all instances have an identical state. – Updates to one instance of a database would also update all other instances. 6
  7. 7. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Compute HA 7
  8. 8. RACKSPACE® HOSTING | WWW.RACKSPACE.COM No vSphere Style HA with KVM 8
  9. 9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Server Evacuation 9 Evacuation Evacuation • Without Shared Storage – The instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc. • With Shared Storage – The instance will be booted from same disk and data will be preserved
  10. 10. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Instance Migration 10
  11. 11. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11 Virtualization vs. Cloud • Virtualization needs care and feeding – Name the VM – Tune and groom regularly – Feed it with good food and supplements – Take to the vet when sick • Cloud servers are disposable – VMs are not unique – Tune and groom apps not the cows – Keep the cow upright – Shoot the cow when it is sick
  12. 12. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Scale Up vs. Scale Out 12 Traditional Cloud
  13. 13. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Application HA 13
  14. 14. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Controller HA Options 14
  15. 15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Pacemaker, Corosync and DRBD 15 • Pacemaker – high availability and load balancing stack for the Linux platform. – Interacts with applications through Resource Agents (RA) • Corosync – Totem single-ring ordering and membership protocol – UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker. • DRBD (Distributed Replication Block Device) – Synchronizes Data at the block device – Uses a journaling system (such as ext3 or ext4)
  16. 16. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Galera • Synchronous multi-master cluster technology for MySQL/InnoDB – MySQL patched for wsrep (Write Set REPlication) – Active/active multi-master topology – Read and write to any cluster node – True parallel replication, in row level – No slave lag or integrity issues 16
  17. 17. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA with Keepalived and HAProxy 17
  18. 18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Keepalived, HAProxy and VRRP • HAProxy – Load Balancing and Proxying for HTTP and TCP Applications – Works over multiple connections – Used to load balance API services • VRRP (Virtual Router Redundancy Protocol) – Eliminates SPOF in a static default routed environment • Keepalived – Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing – Implements a set of checkers to check service status and to maintain health – Leverage the VRRP Protocol to remap VIPS in event of failure 18
  19. 19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Sample OpenStack HA Architecture 19 .… Availability Zone 1 Dedicated Firewalls BOND 0 BOND 1 BOND 0 BOND 1 Controller API Services API & Horizon Cinder API Nova Scheduler Keystone Glance RabbitMQ MYSQL Chef Server Recipes Load Balancers Redundant Network Switches Storage EMC, NetApp, or Solidfire Vols BOND 2 Redundant Network Switches Inside LB VLAN Storage Network (private) Fixed Network (private) Compute 1 KVM G2 G1 G4 G3 Compute N KVM G6 G5 G7 BOND 0 BOND 1 BOND 2 .… Availability Zone 2 BOND 0 BOND 1 BOND 2 Compute 1 KVM Compute N KVM G1 6 G15 G17 BOND 0 BOND 1 BOND 2 BOND 0 BOND 1 BOND 2 G12 G11 G14 G13 BOND 2 Controller API Services API & Horizon Cinder API Nova Scheduler Keystone Glance RabbitMQ MYSQL Chef Server Recipes
  20. 20. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA on OpenStack 20
  21. 21. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA on OpenStack 21
  22. 22. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22 Comparison Database Replication method Strengths Weakness/Limita tions Keepalived/HAPro xy/VRRP Works on MySQL master-master replication Simple to implement and understand. Works for any storage system. Master-master replication does not work beyond 2 nodes. Pacemaker/Coros ync/DRBD Mirroring on Block Devices Well tested More complex to setup. Split Brain possibility Galera Based on write- set Replication (wsrep) No Slave lag Needs at least 3 nodes. Relatively new. Others MySQL Cluster, RHCS with DAS/SAN storage Well tested More complex setup.
  23. 23. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Resources and Contact 23
  24. 24. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Resources • OpenStack – openstack.org – launchpad.net/openstack – #openstack – #openstack on webchat.freenode.net • OpenStack HA – http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf – https://github.com/rcbops-cookbooks/ • MySQL HA – http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration- deployment-guide/ – http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html – https://www.hastexo.com/ – http://www.drbd.org/
  25. 25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2 5 RACKSPACE® HOSTING | WWW.RACKSPACE.COM For More Information You can reach me at: Kenneth Hui Open Cloud Architect Rackspace E-mail: ken.hui@rackspace.com Twitter: @hui_kenneth Blog: http://cloudarchitectmusings.com
  26. 26. 26 RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

×