MHA: Getting started & moving past quirks percona live santa clara 2013
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


MHA: Getting started & moving past quirks percona live santa clara 2013






Total Views
Views on SlideShare
Embed Views



7 Embeds 801 424 212 152
http://localhost 7 4 1 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

MHA: Getting started & moving past quirks percona live santa clara 2013 Presentation Transcript

  • 1. MHA: Getting Started& Moving Past QuirksColin Charles, Monty Program | byte@bytebot.net | | @bytebot on TwitterPercona Live 2013, Santa Clara, CA, USA25 April 2013
  • 2. whoami• Chief Evangelist, MariaDB at MontyProgram Ab• Formerly MySQL AB/Sun Microsystems• Past lives included Fedora Project(FESCO),
  • 3. Aims• Why MHA?• What does MHA do?• How does MHA do it?• Running MHA,VIP failover, integration, etc• Who uses MHA?
  • 4. My experience withMHA• Started November 2011 (MHA 0.52, 0.53)• NRE work to make it run on Solaris 10environment... with no Internet access :-)• Continued deployment advice+work fordata centre use• Much thanks to SkySQL for the experience
  • 5. Why this talk• High Performance MySQL,3rd Edition• Published: March 16 2012
  • 6. Where did MHA comefrom?• 2011 MySQLCommunity AwardCorporate Contributor:DeNA• MHA came shortlyafter the award• Written byYoshinoriMatsunobu, Oracle ACEDirector
  • 7. What is MHA?• MHA for MySQL: Master High AvailabilityManager tools for MySQL• Goal: automating master failover & slavepromotion with minimal downtime• Set of Perl scripts•
  • 8. Why MHA?1. Automating monitoring of your replicationtopology for master failover2. Scheduled online master switching to a differenthost for online maintenance• switch back after OPTIMIZE/ALTER table,software or hardware upgrade3. Interactive/non-interactive master failover (justfor failover, with detection of master failure +VIPtakeover to Pacemaker)
  • 9. Why is master failoverhard?• Whenmaster fails,no morewrites tillfailovercomplete• MySQLreplicationisasynchronous (MHAworks withasync +semi-syncreplication)• slave2 islatest,slave1+3have missingevents,MHA does:• copyid=10frommaster ifpossible• apply allmissingevents
  • 10. Semi-sync replication• if master crashes (or is shutdown :P), MHAcannot save binlogs, latest data is lost• minimises risk of binlog event existing only oncrashed master• guarantees at least one slave receives binlogevents at commit•
  • 11. MHA:Typical scenario• Monitor replication topology• If failure detected on master, immediatelyswitch a candidate master or the mostcurrent slave to become new master• CHANGE MASTER for all slaves to newmaster• Print (stderr)/email report, stop monitoring
  • 12. What does MHA do?
  • 13. Typical timeline• Usually no more than 10-30 seconds• 0-10s: Master failover detected in around 10seconds• (optional) 10-20s: 10 seconds to power offmaster• 10-20s: apply differential relay logs to newmaster• Practice: 4s @ DeNA, usually less than 10s
  • 14. How does MHA work?• Save binlog events from crashed master• Identify latest slave• Apply differential relay log to other slaves• Apply saved binlog events from master• Promote a slave to new master• Make other slaves replicate from new master
  • 15. GettingStarted• MHArequires nochanges toyourapplication• You are ofcourse towrite to avirtual IP(VIP) foryour master• MHA doesnot buildreplicationenvironments for you -that’s DIY
  • 16. MHA Node• Download mha4mysql-node & installthis on all machines: master, slaves, monitor• Packages (DEB, RPM) available• Manually, make sure you haveDBD::mysql & ensure it knows the pathof your MySQL
  • 17. What’s in MHA node• save_binary_logs - save & copy master’sbinlogs• apply_diff_relay_logs - find differentialrelay log events & apply missing events• purge_relay_logs - purge relay log files• filter_mysqlbinlog - OBSOLETE + notused by MHA (trims ROLLBACK statements &equivalent binlog events added by mysqlbinlog)
  • 18. MHA Manager server• Monitor server doesn’t have to be powerfulat all, just remain up• This is a single-point-of-failure so monitorthe manager server where MHA Managergets installed• If MHA Manager isn’t running, your app stillruns, but automated failover is nowdisabled
  • 19. MHA Manager• You must install mha4mysql-node thenmha4mysql-manager• Manager server has many Perl dependencies:DBD::mysql, Config::Tiny,Log::Dispatch,Parallel::ForkManager, Time::HiRes• Package management fixes dependencies, else useCPAN
  • 20. Configuring MHA• Application configuration file: seesamples/conf/app1.cnf• Place this in /etc/MHA/app1.cnf• Global configuration file: see /etc/MHA/masterha_default.cnf (seesamples/conf/masterha_default.cnf)
  • 21. app1.cnf[server default]manager_workdir=/var/log/masterha/app1manager_log=/var/log/masterha/app1/manager.log[server1]hostname=host1[server2]hostname=host2candidate_master=1[server3]hostname=host3[server4]hostname=host4no_master=1no need to specify master asMHA auto-detects thissets priority, but doesn’t necessarily mean it gets promotedas a default (say its too far behind replication).But maybe this is a more powerful box, or has a better setupwill never be the master. RAID0 instead of RAID1+0?Slave is in another data centre?
  • 22. masterha_default.cnf[server default]user=rootpassword=rootpassssh_user=rootmaster_binlog_dir= /var/lib/mysql,/var/log/mysqlremote_workdir=/data/log/masterhaping_interval=3# secondary_check_script=masterha_secondary_check -s remote_host1 -sremote_host2# master_ip_failover_script= /script/masterha/master_ip_failover# shutdown_script= /script/masterha/power_manager# report_script= /script/masterha/send_report# master_ip_online_change_script= /script/masterha/master_ip_online_changecheck master activity frommanager->remote_hostN->master (multiple hosts toensure its not a network issue)
  • 23. MHA uses SSH• MHA uses SSH actively; passphraseless login• In theory, only require Manager SSH to allnodes• however, remembermasterha_secondary_check•masterha_check_ssh --conf=/etc/MHA/app1.cnf
  • 24. Check replication• masterha_check_repl --conf=/etc/MHA/app1.cnf• If you don’t see MySQL ReplicationHealth is OK, MHA will fail• Common errors? Master binlog in differentposition, read privileges on binary/relay log notgranted, using multi-master replication w/o read-only=1 set (only 1 writable master allowed)
  • 25. MHA Manager• masterha_manager --conf=/etc/MHA/app1.cnf• Logs are printed to stderr by default, setmanager_log• Recommended running with nohup, oreven daemontools•
  • 26. So, the MHA playbook• Install MHA node, MHA manager• masterha_check_ssh --conf=/etc/app1.cnf• masterha_check_repl --conf=/etc/app1.cnf• masterha_manager --conf=/etc/app1.cnf• That’s it!
  • 27. master_ip_failover_script• Pacemaker can monitor & takeoverVIP ifrequired or you can have a catalog database• SharedVIP is easy to implement withminimal changes tomaster_ip_failover itself (however,use shutdown_script to power offmachine)
  • 28. master_ip_online_change• similar to master_ip_failover script, butused for online maintenance•masterha_master_switch --master_state=alive• MHA executes FLUSH TABLES WITHREAD LOCK after the writing freeze
  • 29. Test the failover!•masterha_check_status --conf=/etc/MHA/app1.cnf• Kill MySQL (kill -9, shutdown server, kernelpanic)• MHA should go thru failover (stderr)• parse the log as well• Upon completion, it stops running
  • 30. masterha_master_switch• manual failover• --master_state=dead• scheduled online master switchover• great for upgrades to server, etc.• masterha_master_switch --master_state=alive --conf=/etc/MHA/app1.cnf --new_master_host=host2
  • 31. HandlingVIPsmy $vip = ‘”;my $interface = “0”;my $ssh_start_vip = “sudo /sbin/ifconfig eth0:$key $vip”;my $ssh_stop_vip = “sudo /sbin/ifconfig eth0:$key down”;...sub start_vip() {`ssh $ssh_user@$new_master_host ” $ssh_start_vip ”`; }sub stop_vip() {`ssh $ssh_user@$orig_master_host ” $ssh_stop_vip ”`; }master_ip_online_changemaster_ip_failover
  • 32. Integration with otherHA solution• Pacemaker• on RHEL6, you need some HA add-on, justuse the CentOS packages• /etc/ha.d/haresources to configureVIP• A mostly 80% guide on how to get it going(using corosync+pacemaker):•
  • 33. Solaris 10• MHA tested to work on Linux and Solaris 10 & greater• Use a .pkg MySQL• Solaris 10 needs a compiler (SolarisStudio - register w/Oracle to download)• CPAN: manual builds of dependencies• Solaris doesn’t have md5sum (call md5)• Solaris ssh isn’t OpenSSH, missing some features likeConnectionTimeout
  • 34. What about replicationdelay?• by default, MHA checks to see if slave is behindmaster. By more than 100MB, it is never a candidateslave• If you have candidate_master=1 set, considersetting check_repl_delay=0• You can integrate it with pt-heartbeat fromPercona Toolkit•
  • 35. MHA deployment tips• You really should install this asroot• SSH needs to work across allhosts• If you don’t want plaintextpasswords in config files, useinit_conf_load_script• Each monitor can monitormultiple MHA pairs (henceapp1, app2, etc.)• You can have a standby master,make sure its read-only• By default, master1->master2->slave3 doesn’t work• MHA manages master1->master2 w/o issue• use multi_tier_slave=1option• Make sure replication userexists on candidate master too!
  • 36. Alternative solutions• Heartbeat + DRBD• cost $$$ -> passive master• innodb_flush_log_at_trx_commit=1,sync_binlog=1• use MariaDB/Percona Server: (MySQL 5.6 has too)
  • 37. Alternative solutions II• MySQL NDB Cluster• its not InnoDB...• Galera Cluster• Percona Replication Manager (PRM)• Tungsten Replicator• mysqlfailover from mysql-utilities usingGTIDs in 5.6+
  • 38. Where is MHA used• DeNA• Premaccess (Swiss HA hosting company)• Ireland’s national TV & radio service• Jetair Belgium (MHA + MariaDB!)• Samsung• SK Group• DAPA• Facebook
  • 39. MHA 0.55• MHA is being worked on & developedwhile Matsunobu-san is at Facebook• MHA 0.55 is released December 18 2012• It is OK for node to be 0.54 in thissituation, due to bug in manager only•
  • 40. Roadmap• Improve examples in package (0.56 isalready in works)• Make more integrations: pt-heartbeat,commit Solaris code upstream, easierpackages with Pacemaker/corosync• More automated master recovery if possible• Front-end that isn’t console
  • 41. Support• SkySQL:• Training from AccelerationDB•
  • 42. Automated tools• 4-hostVagrant setup for MySQL MHA:• Palomino Cluster Tool:• Ansible playbooks for MHA
  • 43. Video• Yoshinori Matsunobu talking about HighAvailability & MHA at Oracle MySQL day:• Alex Alexander (AccelerationDB) talksabout MHA, with an example of failover, andhow it compares to Tungsten:
  • 44. References• Design document:• Configuration parameters:• JetAir MHA use case:• MySQL binary log:
  • 45. SkySQL Solutions Day• TOMORROW at Hyatt• FREE + FREE BEER• MHA creator talks about MHA and othertalks•
  • 46. Q& | byte@bytebot.net | @bytebot | url: