Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MHA: Getting started & moving past quirks percona live santa clara 2013


Published on

Published in: Technology
  • Be the first to comment

MHA: Getting started & moving past quirks percona live santa clara 2013

  1. 1. MHA: Getting Started& Moving Past QuirksColin Charles, Monty Program | byte@bytebot.net | | @bytebot on TwitterPercona Live 2013, Santa Clara, CA, USA25 April 2013
  2. 2. whoami• Chief Evangelist, MariaDB at MontyProgram Ab• Formerly MySQL AB/Sun Microsystems• Past lives included Fedora Project(FESCO),
  3. 3. Aims• Why MHA?• What does MHA do?• How does MHA do it?• Running MHA,VIP failover, integration, etc• Who uses MHA?
  4. 4. My experience withMHA• Started November 2011 (MHA 0.52, 0.53)• NRE work to make it run on Solaris 10environment... with no Internet access :-)• Continued deployment advice+work fordata centre use• Much thanks to SkySQL for the experience
  5. 5. Why this talk• High Performance MySQL,3rd Edition• Published: March 16 2012
  6. 6. Where did MHA comefrom?• 2011 MySQLCommunity AwardCorporate Contributor:DeNA• MHA came shortlyafter the award• Written byYoshinoriMatsunobu, Oracle ACEDirector
  7. 7. What is MHA?• MHA for MySQL: Master High AvailabilityManager tools for MySQL• Goal: automating master failover & slavepromotion with minimal downtime• Set of Perl scripts•
  8. 8. Why MHA?1. Automating monitoring of your replicationtopology for master failover2. Scheduled online master switching to a differenthost for online maintenance• switch back after OPTIMIZE/ALTER table,software or hardware upgrade3. Interactive/non-interactive master failover (justfor failover, with detection of master failure +VIPtakeover to Pacemaker)
  9. 9. Why is master failoverhard?• Whenmaster fails,no morewrites tillfailovercomplete• MySQLreplicationisasynchronous (MHAworks withasync +semi-syncreplication)• slave2 islatest,slave1+3have missingevents,MHA does:• copyid=10frommaster ifpossible• apply allmissingevents
  10. 10. Semi-sync replication• if master crashes (or is shutdown :P), MHAcannot save binlogs, latest data is lost• minimises risk of binlog event existing only oncrashed master• guarantees at least one slave receives binlogevents at commit•
  11. 11. MHA:Typical scenario• Monitor replication topology• If failure detected on master, immediatelyswitch a candidate master or the mostcurrent slave to become new master• CHANGE MASTER for all slaves to newmaster• Print (stderr)/email report, stop monitoring
  12. 12. What does MHA do?
  13. 13. Typical timeline• Usually no more than 10-30 seconds• 0-10s: Master failover detected in around 10seconds• (optional) 10-20s: 10 seconds to power offmaster• 10-20s: apply differential relay logs to newmaster• Practice: 4s @ DeNA, usually less than 10s
  14. 14. How does MHA work?• Save binlog events from crashed master• Identify latest slave• Apply differential relay log to other slaves• Apply saved binlog events from master• Promote a slave to new master• Make other slaves replicate from new master
  15. 15. GettingStarted• MHArequires nochanges toyourapplication• You are ofcourse towrite to avirtual IP(VIP) foryour master• MHA doesnot buildreplicationenvironments for you -that’s DIY
  16. 16. MHA Node• Download mha4mysql-node & installthis on all machines: master, slaves, monitor• Packages (DEB, RPM) available• Manually, make sure you haveDBD::mysql & ensure it knows the pathof your MySQL
  17. 17. What’s in MHA node• save_binary_logs - save & copy master’sbinlogs• apply_diff_relay_logs - find differentialrelay log events & apply missing events• purge_relay_logs - purge relay log files• filter_mysqlbinlog - OBSOLETE + notused by MHA (trims ROLLBACK statements &equivalent binlog events added by mysqlbinlog)
  18. 18. MHA Manager server• Monitor server doesn’t have to be powerfulat all, just remain up• This is a single-point-of-failure so monitorthe manager server where MHA Managergets installed• If MHA Manager isn’t running, your app stillruns, but automated failover is nowdisabled
  19. 19. MHA Manager• You must install mha4mysql-node thenmha4mysql-manager• Manager server has many Perl dependencies:DBD::mysql, Config::Tiny,Log::Dispatch,Parallel::ForkManager, Time::HiRes• Package management fixes dependencies, else useCPAN
  20. 20. Configuring MHA• Application configuration file: seesamples/conf/app1.cnf• Place this in /etc/MHA/app1.cnf• Global configuration file: see /etc/MHA/masterha_default.cnf (seesamples/conf/masterha_default.cnf)
  21. 21. app1.cnf[server default]manager_workdir=/var/log/masterha/app1manager_log=/var/log/masterha/app1/manager.log[server1]hostname=host1[server2]hostname=host2candidate_master=1[server3]hostname=host3[server4]hostname=host4no_master=1no need to specify master asMHA auto-detects thissets priority, but doesn’t necessarily mean it gets promotedas a default (say its too far behind replication).But maybe this is a more powerful box, or has a better setupwill never be the master. RAID0 instead of RAID1+0?Slave is in another data centre?
  22. 22. masterha_default.cnf[server default]user=rootpassword=rootpassssh_user=rootmaster_binlog_dir= /var/lib/mysql,/var/log/mysqlremote_workdir=/data/log/masterhaping_interval=3# secondary_check_script=masterha_secondary_check -s remote_host1 -sremote_host2# master_ip_failover_script= /script/masterha/master_ip_failover# shutdown_script= /script/masterha/power_manager# report_script= /script/masterha/send_report# master_ip_online_change_script= /script/masterha/master_ip_online_changecheck master activity frommanager->remote_hostN->master (multiple hosts toensure its not a network issue)
  23. 23. MHA uses SSH• MHA uses SSH actively; passphraseless login• In theory, only require Manager SSH to allnodes• however, remembermasterha_secondary_check•masterha_check_ssh --conf=/etc/MHA/app1.cnf
  24. 24. Check replication• masterha_check_repl --conf=/etc/MHA/app1.cnf• If you don’t see MySQL ReplicationHealth is OK, MHA will fail• Common errors? Master binlog in differentposition, read privileges on binary/relay log notgranted, using multi-master replication w/o read-only=1 set (only 1 writable master allowed)
  25. 25. MHA Manager• masterha_manager --conf=/etc/MHA/app1.cnf• Logs are printed to stderr by default, setmanager_log• Recommended running with nohup, oreven daemontools•
  26. 26. So, the MHA playbook• Install MHA node, MHA manager• masterha_check_ssh --conf=/etc/app1.cnf• masterha_check_repl --conf=/etc/app1.cnf• masterha_manager --conf=/etc/app1.cnf• That’s it!
  27. 27. master_ip_failover_script• Pacemaker can monitor & takeoverVIP ifrequired or you can have a catalog database• SharedVIP is easy to implement withminimal changes tomaster_ip_failover itself (however,use shutdown_script to power offmachine)
  28. 28. master_ip_online_change• similar to master_ip_failover script, butused for online maintenance•masterha_master_switch --master_state=alive• MHA executes FLUSH TABLES WITHREAD LOCK after the writing freeze
  29. 29. Test the failover!•masterha_check_status --conf=/etc/MHA/app1.cnf• Kill MySQL (kill -9, shutdown server, kernelpanic)• MHA should go thru failover (stderr)• parse the log as well• Upon completion, it stops running
  30. 30. masterha_master_switch• manual failover• --master_state=dead• scheduled online master switchover• great for upgrades to server, etc.• masterha_master_switch --master_state=alive --conf=/etc/MHA/app1.cnf --new_master_host=host2
  31. 31. HandlingVIPsmy $vip = ‘”;my $interface = “0”;my $ssh_start_vip = “sudo /sbin/ifconfig eth0:$key $vip”;my $ssh_stop_vip = “sudo /sbin/ifconfig eth0:$key down”;...sub start_vip() {`ssh $ssh_user@$new_master_host ” $ssh_start_vip ”`; }sub stop_vip() {`ssh $ssh_user@$orig_master_host ” $ssh_stop_vip ”`; }master_ip_online_changemaster_ip_failover
  32. 32. Integration with otherHA solution• Pacemaker• on RHEL6, you need some HA add-on, justuse the CentOS packages• /etc/ha.d/haresources to configureVIP• A mostly 80% guide on how to get it going(using corosync+pacemaker):•
  33. 33. Solaris 10• MHA tested to work on Linux and Solaris 10 & greater• Use a .pkg MySQL• Solaris 10 needs a compiler (SolarisStudio - register w/Oracle to download)• CPAN: manual builds of dependencies• Solaris doesn’t have md5sum (call md5)• Solaris ssh isn’t OpenSSH, missing some features likeConnectionTimeout
  34. 34. What about replicationdelay?• by default, MHA checks to see if slave is behindmaster. By more than 100MB, it is never a candidateslave• If you have candidate_master=1 set, considersetting check_repl_delay=0• You can integrate it with pt-heartbeat fromPercona Toolkit•
  35. 35. MHA deployment tips• You really should install this asroot• SSH needs to work across allhosts• If you don’t want plaintextpasswords in config files, useinit_conf_load_script• Each monitor can monitormultiple MHA pairs (henceapp1, app2, etc.)• You can have a standby master,make sure its read-only• By default, master1->master2->slave3 doesn’t work• MHA manages master1->master2 w/o issue• use multi_tier_slave=1option• Make sure replication userexists on candidate master too!
  36. 36. Alternative solutions• Heartbeat + DRBD• cost $$$ -> passive master• innodb_flush_log_at_trx_commit=1,sync_binlog=1• use MariaDB/Percona Server: (MySQL 5.6 has too)
  37. 37. Alternative solutions II• MySQL NDB Cluster• its not InnoDB...• Galera Cluster• Percona Replication Manager (PRM)• Tungsten Replicator• mysqlfailover from mysql-utilities usingGTIDs in 5.6+
  38. 38. Where is MHA used• DeNA• Premaccess (Swiss HA hosting company)• Ireland’s national TV & radio service• Jetair Belgium (MHA + MariaDB!)• Samsung• SK Group• DAPA• Facebook
  39. 39. MHA 0.55• MHA is being worked on & developedwhile Matsunobu-san is at Facebook• MHA 0.55 is released December 18 2012• It is OK for node to be 0.54 in thissituation, due to bug in manager only•
  40. 40. Roadmap• Improve examples in package (0.56 isalready in works)• Make more integrations: pt-heartbeat,commit Solaris code upstream, easierpackages with Pacemaker/corosync• More automated master recovery if possible• Front-end that isn’t console
  41. 41. Support• SkySQL:• Training from AccelerationDB•
  42. 42. Automated tools• 4-hostVagrant setup for MySQL MHA:• Palomino Cluster Tool:• Ansible playbooks for MHA
  43. 43. Video• Yoshinori Matsunobu talking about HighAvailability & MHA at Oracle MySQL day:• Alex Alexander (AccelerationDB) talksabout MHA, with an example of failover, andhow it compares to Tungsten:
  44. 44. References• Design document:• Configuration parameters:• JetAir MHA use case:• MySQL binary log:
  45. 45. SkySQL Solutions Day• TOMORROW at Hyatt• FREE + FREE BEER• MHA creator talks about MHA and othertalks•
  46. 46. Q& | byte@bytebot.net | @bytebot | url: