MHA, Murakumo & Me

MHA, MURAKUMO, & ME Automation Tech Casual Talks #1 7/17/2012

自己紹介 (@IJIN)• Michael H. Oshita• Japanese American• In Japan since 2002• Software/Infra Engineer• http://ijin.github.com

DBの冗長化• Master - Slave• DRBD• MySQL Cluster• MMM• RDS

DBの冗長化 (MASTER/SLAVE)• シンプル• 手動フェールオーバー対応• 検知・対応まで数十分単位

DBの冗長化 (DBRD)• 複雑• sync_binlogでwrite IO低下

DBの冗長化 (MYSQL CLUSTER)• 複雑• NDB storage engine。。

DBの冗長化 (MMM)• メンテされてない• 過去ケースを見ても不安定（Percona） • http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/

DBの冗長化 (RDS)• 楽で好き• IOがちょっと。• 政治的な理由で使えなかったり。。

MHA (Master High Availability Manager for MySQL)• シンプル (Master-slaveそのまま）• 自動フェールオーバー (各サーバで最新のbinlogを適用）• 安定性（DeNAさんで数百台のDBで運用）

MHA Masterで障害発生 Slave ManagerMaster Slave

MHA各slaveに最新binlogを適用し、Master昇格 binlog binlog Master Manager Master binlog Slave

MHA FAILOVERシーケンス"Starting master failover.""* Phase 1: Configuration Check Phase..n""* Phase 2: Dead Master Shutdown Phase..n" ==> ここで旧Master情報削除 (master_ip_failover)"* Phase 3: Master Recovery Phase..n" "* Phase 3.1: Getting Latest Slaves Phase..n" "* Phase 3.2: Saving Dead Masters Binlog Phase..n" "* Phase 3.3: Determining New Master Phase..n" ==> ここで新Master情報登録 (master_ip_failover)"* Phase 4: Slaves Recovery Phase..n"

MHA 切替方法は2種類 (master_ip_failoverで定義)• Virtual IP• Catalog DB Server (DNS)

MHA 切替方法は2種類• Virtual IP → 使えない場合も。。• Catalog DB Server (DNS) → 内部DNSサーバ運用面倒。。

そこで

By Genki Sugawara @ Cookpad

MURAKUMO 分散内部DNS• Gossip Protocolで拡散• ruby製 (1.9.2まで）• 操作が簡単• 余分なサーバがいらない

MURAKUMO Gossip Protocolnode nodenode node

MURAKUMO Syntax=== addition of a record shell> mrkmctl -A foo.bar,300,master,100 shell> mrkmctl -L IP address TTL Priority Weight Activity Hostname --------------- ------ -------- ------ -------- ---------- 10.11.12.13 60 Origin - Active my-host 10.11.12.13 300 Master 100 Active foo.bar=== deletion of a record shell> mrkmctl -D foo.bar shell> mrkmctl -L IP address TTL Priority Weight Activity Hostname --------------- ------ -------- ------ -------- ---------- 10.11.12.13 60 Origin - Active my-host

MURAKUMO chefで一発登録もknife bootstrap 10.119.45.233 -N db2 -j {"set_fqdn":"db2"} -r recipe[hostname], role[base], role[db]

http://blog.livedoor.jp	3510
http://spring-mt.tumblr.com	1100
http://webcache.googleusercontent.com	8
https://confluence.livesense.jp	7
http://getpocket.com	6
https://si0.twimg.com	1
http://localhost	1
http://us-w1.rockmelt.com	1
http://cache.yahoofs.jp	1
https://twitter.com	1

MHA, Murakumo & Me

by Michael H. Oshita on Jul 17, 2012

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 4,636

Statistics

MHA, Murakumo & Me Presentation Transcript