MySQL High Availability Deep Dive


Published on

A tutorial on MySQL High Availability on the Pacemaker stack. Covers both MySQL on DRBD, and MySQL with MySQL replication.

Presented by Florian Haas and Yves Trudeau at the Percona Live MySQL Conference & Expo, 2012

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MySQL High Availability Deep Dive

  1. 1. MySQL High Availability Deep Dive Florian Haas (hastexo) Yves Trudeau (Percona)
  2. 2. HA Sprint: Get Prepared!● Tutorial instructions at:● Follow along -- or just watch! (up to you)If you need help: were the guys with the blueSPEAKER tabs, just holler!
  3. 3. About Us
  4. 4. Our virtual environment
  5. 5. An Overviewof the Linux HA stack
  6. 6. Application Interface MySQL RACluster Resource Mgmt PacemakerCluster Messaging CorosyncStorage Replication(optional) DRBD
  7. 7. Physical Physical Host Host MySQL MasterMySQL MySQL MySQL App AppSlave Slave Slave Server Server
  8. 8. Physical Physical Host Host MySQL MasterMySQL MySQL MySQL App AppSlave Slave Slave Server Server
  9. 9. DRBD http://www.drbd.orgDRBD & the DRBD logo are registered trademarks of Linbit Information Technologies
  10. 10. Creating a DRBD resource● Name: mysql● Device name: /dev/drbd0● Backing disk: /dev/pacemaker-1/lvMySQL● Meta data: internal● Replication: ↔ 2● Filesystem: ext3
  11. 11. Corosync
  12. 12. Creating a Corosync Cluster● Enable secauth● 1 ring● No logging to files, syslog only● Start Corosync● Check connectivity● Check membership
  13. 13. Pacemaker
  14. 14. Create a Pacemaker Cluster● Add the “pacemaker” service to /etc/corosync.d● Use ver: 1● Restart corosync● Start pacemakerd● Verify both nodes have joined the cluster (crm_mon)
  15. 15. STONITH/Fencing
  16. 16. Setting up the cluster● Open the shell with crm configure● Set no-quorum-policy to ignore● Disable STONITH ○ Kids, dont do this at home!● Commit the configuration
  17. 17. Pacemaker Resources
  18. 18. Startedstart stop Stopped
  19. 19. Master/Slave Sets
  20. 20. Started Startedstart stop start stop Stopped Stopped
  21. 21. Master promote demote Slave Slavestart stop start stop Stopped Stopped
  22. 22. Adding a DRBD Master/Slave Set● Create the DRBD primitive p_drbd_mysql ○ Use the ocf:linbit:drbd RA ○ Add 2 monitor operations (role=Master, role=Slave) with non-equal intervals● Wrap it in a master/slave set ○ notify=true, clone-max=2, master-max=1● Commit the configuration● Watch the cluster promote
  23. 23. Constraints
  24. 24. Creating a Filesystem● Add the Filesystem resource ○ Name: p_fs_mysql ○ Device: /dev/drbd0 ○ Directory: /var/lib/mysql ○ File system type: ext3● Add required order & colocation constraints
  25. 25. Groups
  26. 26. Add a Group● Create the g_mysql resource group● Add p_fs_mysql and p_ip_mysql
  27. 27. Requirements for MySQL/DRBDNo MyISAMlog-binsync_binlogBind or No bindinnodb_log_file_sizeinnodb_auto_lru_dump /innodb_buffer_pool_restore_at_startup
  28. 28. Add the MySQL resource● Run mysql_install_db● Create an ocf:heartbeat:mysql primitive p_mysql● Add resource level monitoring● Add it to the g_mysql group● Watch the constraints update automagically● Commit, Rejoice!
  29. 29. One more thing...
  30. 30. Physical Physical Host Host MySQL MasterMySQL MySQL MySQL App AppSlave Slave Slave Server Server
  31. 31. Physical Physical Host Host MySQL MasterMySQL MySQL MySQL App AppSlave Slave Slave Server Server
  32. 32. Physical Physical Host Host MySQL Master InnoDB Recovery Umm.. WTF? Hurry up!MySQL MySQL MySQL App AppSlave Slave Slave Server Server Booooring
  33. 33. Physical Physical Host Host MySQL MasterMySQL MySQL MySQL App AppSlave Slave Slave Server Server
  34. 34. Handling MySQL replicationRead-onlyPromotion of the masterAttaching slavesMonitoring slave stateDemotion
  35. 35. Promote MySQL1 Notify Slaves Final State Start resources Initial state RVIP1 RVIP1 Write master info RVIP2 WVIP MySQL1 set readerOK RVIP3 enable writer VIP WVIP Turn Off Read-only Start MySQL --skip-slave-start master_host = MySQL1 master_log_file=file1 master_log_pos=1RVIP2 RVIP3 MySQL2 MySQL3Read CIBStart MySQLMonitor Start MySQL Read CIB MonitorStart Slave--skip-slave-startUpdate master score --skipe-slave-start Start slave Update master scoreUpdate readerOKUpdate readerOK Update readerOK
  36. 36. Promotion failure Failover finalMySQL2 Master of state RVIP1 WVIP MySQL1 Fencing master_host = MySQL1 MySQL2RVIP2 master_log_file=file12 master_log_file=file1 master_log_pos=123 master_log_pos=1 RVIP1RVIP2WVIP RVIP3 RVIP3 MySQL2 MySQL3Write master infoMonitor Stop slave Monitorset readerOKUpdate master score read CIB Master info Update master scoreenable readerOKUpdate writer VIP Start salve Update readerOKTurn Off Read-only
  37. 37. SetupMySQL GrantsNode IP attributeMySQL primitiveVIP primitiveMaster-slave ms resourceLocation rulesColocation rulesOrder rule
  38. 38. Variation on the themeSlave only nodesHow to add slave nodesSetup with no VIPAvoid cluster collapse
  39. 39. Further Information http://www.drbd.org
  40. 40. Liked this talk?● Go to● Log in (you can just use your OpenID from Wordpress, Blogger, Yahoo, Google Apps, Google Profile)● Leave us a message!