Your SlideShare is downloading. ×
0
1
HA in OpenStack
September 24th 2013
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Agenda
• What is HA
• Compute HA
• Controller HA
• Corosync, Pacemaker and DRBD
• G...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is HA?
3
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability
4
Minimize data loss
Minimize system downtime
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Stateless services
– There is no dependency between re...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Active/Passive
– Redundant instances of stateless serv...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Compute HA
7
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
No vSphere Style HA with KVM
8
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Server Evacuation
9
Evacuation
Evacuation
• Without Shared Storage
– The instance w...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Instance Migration
10
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11
Virtualization vs. Cloud
• Virtualization needs care and feeding
– Name the VM
–...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Scale Up vs. Scale Out
12
Traditional Cloud
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Application HA
13
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Controller HA Options
14
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Pacemaker, Corosync and DRBD
15
• Pacemaker
– high availability and load balancing ...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Galera
• Synchronous multi-master cluster
technology for MySQL/InnoDB
– MySQL patch...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA with Keepalived and HAProxy
17
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Keepalived, HAProxy and VRRP
• HAProxy
– Load Balancing and Proxying for HTTP and T...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Sample OpenStack HA Architecture
19
.…
Availability
Zone 1
Dedicated Firewalls
BOND...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
20
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
21
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
22
Comparison
Database Replication
method
Strengths Weakness/Limita
tions
Keepalive...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources and Contact
23
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #ope...
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
2
5
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
For More Information
You can reach me at...
26
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | W...
Upcoming SlideShare
Loading in...5
×

OpenStack HA

15,872

Published on

Walks through how HA is handled at the compute and controller layer within OpenStack. Presented at vBrownbag on 2013-09-24.

Published in: Technology, Business
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,872
On Slideshare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
471
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide
  • I work as an Open Cloud Architect with Rackspace. Rackspace has the distinction of being one of the founders of OpenStack and the proprietor of the largest OpenStack-based Public Clouds in the world. As such, we have a vested desire and interest in driving the adoption of OpenStack and the growth of it’s community.However, Rackspace does not own OpenStack and has in fact given it to the community via the Linux Foundation and the the Apache 2 license. In that same spirit, I am happy to help anyone who wishes to learn more about OpenStack, regardless of rather you are now or ever will be a Rackspace customer. So please feel free to contact me and to ask questions or to have me talk about OpenStack with your company or user group. I tweet frequently about OpenStack and maintain a blog that often has OpenStack and/or Cloud Computing related posts.
  • Transcript of "OpenStack HA"

    1. 1. 1 HA in OpenStack September 24th 2013
    2. 2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Agenda • What is HA • Compute HA • Controller HA • Corosync, Pacemaker and DRBD • Galera • HAProxy, keepalived, VRRP • Resources and Summary 2
    3. 3. RACKSPACE® HOSTING | WWW.RACKSPACE.COM What is HA? 3
    4. 4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability 4 Minimize data loss Minimize system downtime
    5. 5. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability Concepts • Stateless services – There is no dependency between requests – For example: Nova API, Nova Scheduler, etc. • Stateful services – An action typically comprises multiple requests – For example: MySQL, RabbitMQ, etc. 5
    6. 6. RACKSPACE® HOSTING | WWW.RACKSPACE.COM High Availability Concepts • Active/Passive – Redundant instances of stateless services are load balanced – For Stateful services a replacement resource can be brought online. • Active/Active – Redundant instances of stateless services are load balanced – Stateful services are managed in such a way that services are redundant, and that all instances have an identical state. – Updates to one instance of a database would also update all other instances. 6
    7. 7. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Compute HA 7
    8. 8. RACKSPACE® HOSTING | WWW.RACKSPACE.COM No vSphere Style HA with KVM 8
    9. 9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Server Evacuation 9 Evacuation Evacuation • Without Shared Storage – The instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc. • With Shared Storage – The instance will be booted from same disk and data will be preserved
    10. 10. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Instance Migration 10
    11. 11. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11 Virtualization vs. Cloud • Virtualization needs care and feeding – Name the VM – Tune and groom regularly – Feed it with good food and supplements – Take to the vet when sick • Cloud servers are disposable – VMs are not unique – Tune and groom apps not the cows – Keep the cow upright – Shoot the cow when it is sick
    12. 12. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Scale Up vs. Scale Out 12 Traditional Cloud
    13. 13. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Application HA 13
    14. 14. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Controller HA Options 14
    15. 15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Pacemaker, Corosync and DRBD 15 • Pacemaker – high availability and load balancing stack for the Linux platform. – Interacts with applications through Resource Agents (RA) • Corosync – Totem single-ring ordering and membership protocol – UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker. • DRBD (Distributed Replication Block Device) – Synchronizes Data at the block device – Uses a journaling system (such as ext3 or ext4)
    16. 16. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Galera • Synchronous multi-master cluster technology for MySQL/InnoDB – MySQL patched for wsrep (Write Set REPlication) – Active/active multi-master topology – Read and write to any cluster node – True parallel replication, in row level – No slave lag or integrity issues 16
    17. 17. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA with Keepalived and HAProxy 17
    18. 18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Keepalived, HAProxy and VRRP • HAProxy – Load Balancing and Proxying for HTTP and TCP Applications – Works over multiple connections – Used to load balance API services • VRRP (Virtual Router Redundancy Protocol) – Eliminates SPOF in a static default routed environment • Keepalived – Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing – Implements a set of checkers to check service status and to maintain health – Leverage the VRRP Protocol to remap VIPS in event of failure 18
    19. 19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Sample OpenStack HA Architecture 19 .… Availability Zone 1 Dedicated Firewalls BOND 0 BOND 1 BOND 0 BOND 1 Controller API Services API & Horizon Cinder API Nova Scheduler Keystone Glance RabbitMQ MYSQL Chef Server Recipes Load Balancers Redundant Network Switches Storage EMC, NetApp, or Solidfire Vols BOND 2 Redundant Network Switches Inside LB VLAN Storage Network (private) Fixed Network (private) Compute 1 KVM G2 G1 G4 G3 Compute N KVM G6 G5 G7 BOND 0 BOND 1 BOND 2 .… Availability Zone 2 BOND 0 BOND 1 BOND 2 Compute 1 KVM Compute N KVM G1 6 G15 G17 BOND 0 BOND 1 BOND 2 BOND 0 BOND 1 BOND 2 G12 G11 G14 G13 BOND 2 Controller API Services API & Horizon Cinder API Nova Scheduler Keystone Glance RabbitMQ MYSQL Chef Server Recipes
    20. 20. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA on OpenStack 20
    21. 21. RACKSPACE® HOSTING | WWW.RACKSPACE.COM HA on OpenStack 21
    22. 22. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22 Comparison Database Replication method Strengths Weakness/Limita tions Keepalived/HAPro xy/VRRP Works on MySQL master-master replication Simple to implement and understand. Works for any storage system. Master-master replication does not work beyond 2 nodes. Pacemaker/Coros ync/DRBD Mirroring on Block Devices Well tested More complex to setup. Split Brain possibility Galera Based on write- set Replication (wsrep) No Slave lag Needs at least 3 nodes. Relatively new. Others MySQL Cluster, RHCS with DAS/SAN storage Well tested More complex setup.
    23. 23. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Resources and Contact 23
    24. 24. RACKSPACE® HOSTING | WWW.RACKSPACE.COM Resources • OpenStack – openstack.org – launchpad.net/openstack – #openstack – #openstack on webchat.freenode.net • OpenStack HA – http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf – https://github.com/rcbops-cookbooks/ • MySQL HA – http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration- deployment-guide/ – http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html – https://www.hastexo.com/ – http://www.drbd.org/
    25. 25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2 5 RACKSPACE® HOSTING | WWW.RACKSPACE.COM For More Information You can reach me at: Kenneth Hui Open Cloud Architect Rackspace E-mail: ken.hui@rackspace.com Twitter: @hui_kenneth Blog: http://cloudarchitectmusings.com
    26. 26. 26 RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×