by Rob Kaufman [email_address] MySQL S&M

Slaves and Masters What is Master Slave Replication One database instance (the Master) that does both reads and writes This guy is in charge Should do mostly writing One or more database instances that only do reads (the Slaves) Only know what the Master tells them Never except writes

What is Master Slave Replication

One database instance (the Master) that does both reads and writes

This guy is in charge

Should do mostly writing

One or more database instances that only do reads (the Slaves)

Only know what the Master tells them

Never except writes

Master Slave Diagram Master Slave 1 Slave 2 Slave 3 App Server App Server Key: Arrow represents data flow direction

Master Slave: Why What will a Master Slave setup do for you? Redundancy Hot spares of data Any slave can become the master if the master fails Efficiency If you data is read bound Many reads to a few writes

What will a Master Slave setup do for you?

Redundancy

Hot spares of data

Any slave can become the master if the master fails

Efficiency

If you data is read bound

Many reads to a few writes

Master Slave: Why Not What draw backs does a Master Slave setup have Not a backup! Your backing up your data... right? If not, go do it now Yes I really mean it... get up and go NOW Won't help the write bound Many web apps are about creating data, not just consuming it

What draw backs does a Master Slave setup have

Not a backup!

Your backing up your data... right?

If not, go do it now

Yes I really mean it... get up and go NOW

Won't help the write bound

Many web apps are about creating data, not just consuming it

Master Slave: How Create slave user GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%.mydomain.com' IDENTIFIED BY 'slavepass'; Configure Master Configure Slave Copy The Data Use Innodb

Create slave user

GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%.mydomain.com' IDENTIFIED BY 'slavepass';

Configure Master

Configure Slave

Copy The Data

Use Innodb

Be the Master Master Configuration my.cnf: [mysqld] log-bin=mysql-bin server-id=1 Watch out for the skip-networking option Make a data backup mysqldump --all-databases --lock-all-tables --master-data >dbdump.db

Master Configuration

my.cnf:

[mysqld]

log-bin=mysql-bin

server-id=1

Watch out for the skip-networking option

Make a data backup

mysqldump --all-databases --lock-all-tables --master-data >dbdump.db

Be the Slave Slave Setup Don't have to do log-bin, but you should my.cnf: [mysqld] log-bin=mysql-bin server-id=2 master-host = host_ip_address master-user = replication master-password = slavepass master-port = 3306

Slave Setup

Don't have to do log-bin, but you should

my.cnf:

[mysqld]

log-bin=mysql-bin

server-id=2

master-host = host_ip_address

master-user = replication

master-password = slavepass

master-port = 3306

Whip that Slave Import the data Start slave with --skip-slave option mysql < dbdump.db Start slave with mysql command &quot;start slave;&quot; check with &quot;show slave status;&quot; and &quot;show master status;&quot;

Import the data

Start slave with --skip-slave option

mysql < dbdump.db

Start slave with mysql command &quot;start slave;&quot;

check with &quot;show slave status;&quot; and &quot;show master status;&quot;

Slaves in Rails How do I do this in Rails Masochism Hey, I didn't pick the name ;-) Plugin by Rick Olsen piston import http://ar-code.svn.engineyard.com/plugins/masochism/ vendor/plugins add master_database entry into your database.yml set production entry to point at slave add to production.rb config.after_initialize do ActiveReload::ConnectionProxy.setup! end

How do I do this in Rails

Masochism

Hey, I didn't pick the name ;-)

Plugin by Rick Olsen

piston import http://ar-code.svn.engineyard.com/plugins/masochism/ vendor/plugins

add master_database entry into your database.yml

set production entry to point at slave

add to production.rb

config.after_initialize do

ActiveReload::ConnectionProxy.setup!

end

Slaves in Rails See Robby Russell for more info http://www.robbyonrails.com/articles/2007/11/15/master-slave-databases-with-ruby-on-rails

See Robby Russell for more info

http://www.robbyonrails.com/articles/2007/11/15/master-slave-databases-with-ruby-on-rails

Other Slave in Rails Options Active Delegate Multiple Database Connections in Ruby on Rails From Robby Russell (http://www.robbyonrails.com/articles/2007/10/05/multiple-database-connections-in-ruby-on-rails) Likely abandoned magic_multi_connections From Dr Nic (http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/) This gem is also able to handle master/slave setups but also could be used to do data partitioning across multiple databases. Admittedly has issues

Active Delegate

Multiple Database Connections in Ruby on Rails

From Robby Russell (http://www.robbyonrails.com/articles/2007/10/05/multiple-database-connections-in-ruby-on-rails)

Likely abandoned

magic_multi_connections

From Dr Nic (http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/)

This gem is also able to handle master/slave setups but also could be used to do data partitioning across multiple databases.

Admittedly has issues

Other Slave in Rails Options ActsAsReadonlyable From the Revolution Health team (http://revolutiononrails.blogspot.com/2007/04/plugin-release-actsasreadonlyable.html) DynamicDatabaseChanger Rails Database Loadbalancing http://rubyforge.org/projects/ddcplugin/ May work for multi-master set ups too (http://scriptserver.blogspot.com/2007/06/rails-database-loadbalancing-with.html) Does not do failover if instance goes down

ActsAsReadonlyable

From the Revolution Health team (http://revolutiononrails.blogspot.com/2007/04/plugin-release-actsasreadonlyable.html)

DynamicDatabaseChanger

Rails Database Loadbalancing

http://rubyforge.org/projects/ddcplugin/

May work for multi-master set ups too (http://scriptserver.blogspot.com/2007/06/rails-database-loadbalancing-with.html)

Does not do failover if instance goes down

Master Master Replication What is Master Master Replication All Master database instances can both read and write They are set up in a &quot;circle&quot; that passes data from one instance to another Every master is one machines slave Every slave is another machines master Can be mixed with Master Slave Each Master has multiple slaves One slave of each master is the next master in the chain

What is Master Master Replication

All Master database instances can both read and write

They are set up in a &quot;circle&quot; that passes data from one instance to another

Every master is one machines slave

Every slave is another machines master

Can be mixed with Master Slave

Each Master has multiple slaves

One slave of each master is the next master in the chain

Master Master Diagram Master 2 Master 1 Master 3 App Server App Server Key: Arrow represents data flow direction App Server

Master Master: Why What will a Master Master setup do for you? Redundancy Hot spares of data If a master fails it can be removed from the loop its app servers can be redirected elsewhere data will stop moving around the circle till it is fixed or removed Efficiency If you are either write or read bound Can scale cleanly up to 10 instances w/ fast connection between servers

What will a Master Master setup do for you?

Redundancy

Hot spares of data

If a master fails

it can be removed from the loop

its app servers can be redirected elsewhere

data will stop moving around the circle till it is fixed or removed

Efficiency

If you are either write or read bound

Can scale cleanly up to 10 instances w/ fast connection between servers

Master Master: Why Not What drawbacks does a Master Master setup have Not a backup! Can propagate DROP and DELETE just as fast as it can UPDATE and INSERT Can get behind It is possible that the data might not all be propagated to every instance at a given moment If a db instance goes down If network latency is really high If traffic is higher than the db instance can handle

What drawbacks does a Master Master setup have

Not a backup!

Can propagate DROP and DELETE just as fast as it can UPDATE and INSERT

Can get behind

It is possible that the data might not all be propagated to every instance at a given moment

If a db instance goes down

If network latency is really high

If traffic is higher than the db instance can handle

Master Master: How Create slave user GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%.mydomain.com' IDENTIFIED BY 'slavepass'; Make a data backup of master1 mysqldump --all-databases --lock-all-tables --master-data >dbdump.db

Create slave user

GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%.mydomain.com' IDENTIFIED BY 'slavepass';

Make a data backup of master1

mysqldump --all-databases --lock-all-tables --master-data >dbdump.db

Be the Master Master 1 configuration my.cnf: [mysqld] log-bin=mysql-bin log-slave-updates replicate-same-server-id = 0 server-id=1 master-host = master2_ip_address master-user = replication master-password = slavepass master-port = 3306 auto_increment_increment = 10 auto_increment_offset = 1

Master 1 configuration

my.cnf:

[mysqld]

log-bin=mysql-bin

log-slave-updates

replicate-same-server-id = 0

server-id=1

master-host = master2_ip_address

master-user = replication

master-password = slavepass

master-port = 3306

auto_increment_increment = 10

auto_increment_offset = 1

Be the Other Master Master 2 configuration my.cnf: [mysqld] log-bin=mysql-bin log-slave-updates replicate-same-server-id = 0 server-id=2 master-host = master1_ip_address master-user = replication master-password = slavepass master-port = 3306 auto_increment_increment = 10 auto_increment_offset = 2

Master 2 configuration

my.cnf:

[mysqld]

log-bin=mysql-bin

log-slave-updates

replicate-same-server-id = 0

server-id=2

master-host = master1_ip_address

master-user = replication

master-password = slavepass

master-port = 3306

auto_increment_increment = 10

auto_increment_offset = 2

Whip that... Master? Import the data into master2 Start master2 with --skip-slave option mysql < dbdump.db Start master2 as slave with mysql command &quot;start slave;&quot; Get master2 position with mysql command &quot;show master status&quot; Write down the info under File and Position, use for FILE and POSITION below Back on master1 Start master1 with --skip-slave option Set the master position mysql command CHANGE MASTER TO MASTER_LOG_FILE='FILE', MASTER_LOG_POS=POSITION Start master1 as slave with mysql command &quot;start slave;&quot;

Import the data into master2

Start master2 with --skip-slave option

mysql < dbdump.db

Start master2 as slave with mysql command &quot;start slave;&quot;

Get master2 position with mysql command &quot;show master status&quot;

Write down the info under File and Position, use for FILE and POSITION below

Back on master1

Start master1 with --skip-slave option

Set the master position mysql command

CHANGE MASTER TO MASTER_LOG_FILE='FILE', MASTER_LOG_POS=POSITION

Start master1 as slave with mysql command &quot;start slave;&quot;

Master Master in Rails Step 1: Setup Master Master Replication for your production enviroment Step 2: Point different clusters of application servers at each different master Step 3: There IS NO STEP 3!

Step 1: Setup Master Master Replication for your production enviroment

Step 2: Point different clusters of application servers at each different master

Step 3: There IS NO STEP 3!

What you want load balancing? We can do load balancing for either Master Slave or Master Master SQLRelay http://sqlrelay.sourceforge.net/documentation.html Stable and well tested solution If one db instance goes down, will server requests to other instances MysqlRelay http://forge.mysql.com/wiki/MySQL_Proxy Kind of the new kid on the block MySQL specific Has other functions besides load balance and failover support

We can do load balancing for either Master Slave or Master Master

SQLRelay

http://sqlrelay.sourceforge.net/documentation.html

Stable and well tested solution

If one db instance goes down, will server requests to other instances

MysqlRelay

http://forge.mysql.com/wiki/MySQL_Proxy

Kind of the new kid on the block

MySQL specific

Has other functions besides load balance and failover support

Know When Something is Wrong You need to monitor the replication &quot;show slave status&quot; and &quot;show master status&quot; Munin http://munin.projects.linpro.no/wiki/plugin-mysql_slave_status MMM http://code.google.com/p/mysql-master-master/ PDI

You need to monitor the replication

&quot;show slave status&quot; and &quot;show master status&quot;

Munin

http://munin.projects.linpro.no/wiki/plugin-mysql_slave_status

MMM

http://code.google.com/p/mysql-master-master/

PDI

Fix An Error Now what Most of the time what went wrong was the bin log got off by position. This happens sometimes when a db instance crashes or powerfails If so you can use mysql command &quot;STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;&quot; Can be greater than 1 if you want, but watch for danger Sometimes (rarely) it will get ahead Only ever seen this on a disk full Can go back to an old position on the relay log &quot;CHANGE MASTER TO RELAY_LOG_FILE='slave-bin.006', RELAY_LOG_POS=4025;

Now what

Most of the time what went wrong was the bin log got off by position. This happens sometimes when a db instance crashes or powerfails

If so you can use mysql command

&quot;STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;&quot;

Can be greater than 1 if you want, but watch for danger

Sometimes (rarely) it will get ahead

Only ever seen this on a disk full

Can go back to an old position on the relay log

&quot;CHANGE MASTER TO RELAY_LOG_FILE='slave-bin.006', RELAY_LOG_POS=4025;

Thanks Thanks to Robby Russell and Dr Nic They have both done quite a bit of blogging in the Master Slave area Special thanks to RailsMagnet for having links to all the current master-slave plugins for Rails http://railsmagnet.com/2007/12/using-multiple-databases-rails-application MySQL www.mysql.com for its cool Master Master setup Munin http://munin.projects.linpro.no/ for watching my back

Thanks to Robby Russell and Dr Nic

They have both done quite a bit of blogging in the Master Slave area

Special thanks to RailsMagnet

for having links to all the current master-slave plugins for Rails

http://railsmagnet.com/2007/12/using-multiple-databases-rails-application

MySQL

www.mysql.com

for its cool Master Master setup

Munin

http://munin.projects.linpro.no/

for watching my back