Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MHA, Murakumo & Me

8,310 views

Published on

Using MHA (Master High Availability Manager for MySQL) and Murakumo (distributed internal dns) to achieve a simple highly-available environment.

#autotechcasual

Published in: Technology
  • Be the first to comment

MHA, Murakumo & Me

  1. 1. MHA, MURAKUMO, & ME Automation Tech Casual Talks #1 7/17/2012
  2. 2. 自己紹介 (@IJIN)• Michael H. Oshita• Japanese American• In Japan since 2002• Software/Infra Engineer• http://ijin.github.com
  3. 3. DBの冗長化• Master - Slave• DRBD• MySQL Cluster• MMM• RDS
  4. 4. DBの冗長化 (MASTER/SLAVE)• シンプル• 手動フェールオーバー対応• 検知・対応まで数十分単位
  5. 5. DBの冗長化 (DBRD)• 複雑• sync_binlogでwrite IO低下
  6. 6. DBの冗長化 (MYSQL CLUSTER)• 複雑• NDB storage engine。。
  7. 7. DBの冗長化 (MMM)• メンテされてない• 過去ケースを見ても不安定(Percona) • http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/
  8. 8. DBの冗長化 (RDS)• 楽で好き• IOがちょっと。• 政治的な理由で使えなかったり。。
  9. 9. MHA (Master High Availability Manager for MySQL)• シンプル (Master-slaveそのまま)• 自動フェールオーバー (各サーバで最新のbinlogを適用)• 安定性(DeNAさんで数百台のDBで運用)
  10. 10. MHA Masterで障害発生 Slave ManagerMaster Slave
  11. 11. MHA各slaveに最新binlogを適用し、Master昇格 binlog binlog Master Manager Master binlog Slave
  12. 12. MHA FAILOVERシーケンス"Starting master failover.""* Phase 1: Configuration Check Phase..n""* Phase 2: Dead Master Shutdown Phase..n" ==> ここで旧Master情報削除 (master_ip_failover)"* Phase 3: Master Recovery Phase..n" "* Phase 3.1: Getting Latest Slaves Phase..n" "* Phase 3.2: Saving Dead Masters Binlog Phase..n" "* Phase 3.3: Determining New Master Phase..n" ==> ここで新Master情報登録 (master_ip_failover)"* Phase 4: Slaves Recovery Phase..n"
  13. 13. MHA 切替方法は2種類 (master_ip_failoverで定義)• Virtual IP• Catalog DB Server (DNS)
  14. 14. MHA 切替方法は2種類• Virtual IP → 使えない場合も。。• Catalog DB Server (DNS) → 内部DNSサーバ運用面倒。。
  15. 15. そこで
  16. 16. By Genki Sugawara @ Cookpad
  17. 17. MURAKUMO 分散内部DNS• Gossip Protocolで拡散• ruby製 (1.9.2まで)• 操作が簡単• 余分なサーバがいらない
  18. 18. MURAKUMO Gossip Protocolnode nodenode node
  19. 19. MURAKUMO Syntax=== addition of a record shell> mrkmctl -A foo.bar,300,master,100 shell> mrkmctl -L IP address TTL Priority Weight Activity Hostname --------------- ------ -------- ------ -------- ---------- 10.11.12.13 60 Origin - Active my-host 10.11.12.13 300 Master 100 Active foo.bar=== deletion of a record shell> mrkmctl -D foo.bar shell> mrkmctl -L IP address TTL Priority Weight Activity Hostname --------------- ------ -------- ------ -------- ---------- 10.11.12.13 60 Origin - Active my-host
  20. 20. MURAKUMO chefで一発登録もknife bootstrap 10.119.45.233 -N db2 -j {"set_fqdn":"db2"} -r recipe[hostname], role[base], role[db]
  21. 21. DEMO

×