Advanced MySQLReplication TechniquesGiuseppe Maxia, QA Director, Continuent, Inc     Facts. And Demos. Possibly fun
about me - Giuseppe Maxia•   a.k.a. The Data Charmer•   QA Director at Continuent, Inc•   Long time hacking with MySQL fea...
Agenda•   Replication basics•   Replication blues    •   Master switch and failover    •   Slave lagging    •   Gotchas•  ...
DAAG  EN           Replication basics       • Principles       • How to set up replication       • Monitoring replication
DAAG  EN     Principles
database server                               a simple web                                application                     ...
database server                             scaling web                              requests r/w requests                ...
read/write  master                               a web                                     application                    ...
client                                                         master                      transaction                    ...
DAAG  EN     replication setup
1   SHUT DOWN THE DATABASE SERVER              Master                                    11
2   MAKE A BACKUP COPY        Master                         12
3           ENABLE THE MASTER                 Master           Configuration file    [mysqld]    log-bin=mysql-bin    serv...
4   RESTART THE MASTER        Master                         14
5        CREATE REPLICATION USER                Master              SQL command    GRANT REPLICATION SLAVE ON *.*    to sl...
6            INSTALL MySQL on the slave                Slave 1Make sure that:• Youre using the same version of MySQL• You ...
7   COPY THE MASTER DATA to the slave             Slave 1                                        17
8            ENABLE THE SLAVE               Slave 1           Configuration file    [mysqld]    server-id=2    relay-log=m...
9   START THE SLAVE SERVER        Slave 1                             19
10          INITIALIZE THE SLAVE               Slave 1             SQL command     CHANGE MASTER TO     MASTER_HOST=master...
11       START THE SLAVE SERVICE              Slave 1             SQL command     START SLAVE;                            ...
12          CHECK THE SLAVE              Slave 1             SQL command     SHOW SLAVE STATUS G     ...      Slave_IO_Run...
Troubleshooting• SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No •   Make sure that the slave host can connect     to the maste...
Testing the slave• Create a table in the master.• Make sure that the slave has replicated  the table.• Insert data in the ...
DAAG  EN     Monitoring
Sample                       get slave                               statusmonitoring                                     ...
monitoring replicationmaster> SHOW MASTER STATUSslave> SHOW SLAVE STATUSFULL SCRIPTS:http://datacharmer.blogspot.com/2011/...
show master status           File: mysql-bin.000002       Position: 78045744   Binlog_Do_DB:Binlog_Ignore_DB:
show slave status       Slave_IO_State: Waiting for master to send event          Master_Host: 127.0.0.1          Master_U...
show slave status...               Replicate_Do_DB:           Replicate_Ignore_DB:            Replicate_Do_Table:        R...
show slave status...                 Last_Errno: 0                 Last_Error:               Skip_Counter: 0        Exec_M...
DAAG  EN              Replication blues         •   Master switch and failover         •   Slave lagging         •   Gotchas
DAAG  EN     Master switch and failover
Replacing the               Master  master                   crashe                                  s        Let all slav...
Planned master switch• Stop accepting writes• Wait until all slaves have caught up• Stop replication in all slaves• Promot...
Changing a failed master• Pre-requisite:• log_bin and log_slave_updates must be  enabled in all the slaves• If not, there ...
Changing a failed master (1)• Wait until all slaves have caught up• Identify the most advanced slave• Make that slave the ...
Changing a failed master (2)    • For each remaining slave:    • Find the missing statements•    Find the LAST statement r...
Changing a failed master (3)• For each remaining slave:• Apply the missing statements •   CHANGE MASTER TO     master_host...
Reasons for complexity• No global transaction ID
What is a global transaction ID • A unique identifier of a transaction • Unique for the whole cluster, not for each node •...
Why should you care• Failure recovery• MySQL DOES NOT have it
Defining the problem                                              Slave 3 Master          Replication          Slave 2    ...
The master crashes                                             Slave 3Master          Replication          Slave 2        ...
Where do the slaves                 stand? Slave 1         Slave 2        Slave 3 Binary log      Binary log     Binary lo...
Slave 3 becomes                    master                                  New master Slave 1            Slave 2          ...
Need to synch the        missing transactions                              New master Slave 1        Slave 2              ...
Transaction position in     the new master binlog                                      New master Slave 1            Slave...
An existing solution• Google has made a patch to implement global  transaction IDs• HOWEVER• It only works with 5.0• It do...
Hackahead!
CREATE TABLE Global_Trans_ID (   number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY) ENGINE = MyISAM;CREATE TABLE Last_Exec_Tr...
CREATE PROCEDURE commit_trans ()BEGIN  DECLARE trans_id, server_id INT UNSIGNED;  SET SQL_LOG_BIN = 0;  INSERT INTO global...
Hackingthe hack
Simplifying the scheme• No MyISAM table and LAST_INSERT_ID()• Use UUID_SHORT() instead• Replace the InnoDB table with a Bl...
Advantages• 10 times faster• No additional tables and storage required
Disadvantages• The master SERVER ID must be < 256
Another solution• We’ll see that in the second part of the  talk
DAAG  EN     Slave lagging
Slave lagging• Common causes: •   Excess of traffic •   Slave restart •   Long DDL operations
Slave lagging                                               transaction                                                 tr...
Slave lagging remedies• row-based replication (helps in some  cases)• Expensive query split• Double apply (dangerous)• Sla...
statement-based replicationmaster> set global binlog_format=‘statement’;master> insert into test.t1 select count(*) from  ...
row-based replicationmaster> set global binlog_format=‘row’;master> insert into test.t1 select count(*) from  information_...
Expensive query split# instead of this:insert into t1 select benchmark(10000000, sha(abc));Query OK, 1 row affected (4.19 ...
Expensive query split# at 471#110410 8:21:21 server id 1 end_log_pos 518User_varSET @`result`:=0/*!*/;# at 518#110410 8:21...
Slave query prefetch• What is it? •   a technique that fetches queries from the     relay logs, and runs them as SELECT   ...
DAAG  EN     Gotchas
Gotchas• When you thought that you had figured  out everything ...
Gotchas: LOAD DATA• LOAD DATA INFILE is replicated as a  temporary file• The slave needs three times the size of  the data...
Gotchas: default engine• If you define storage_engine in the  master, it does not get replicated
Gotchas: binlog format• In circular replication, and relay slaves• Changes to binlog_format are not  propagated
Gotchas: binlog format  server id: 101       insert into t1format: statement   values (@@server_id) 101 server id: 102  fo...
Gotchas: set global not really global• set global binlog_format=row• Works for all new connections after the  change• It d...
Gotchas: triggers and row replication• In statement based replication, triggers  are fired both on master and slave• in ro...
DAAG  EN   Replication power techniques                and features         •   row-based         •   semi-synch         •...
DAAG  EN     row-based replication
row-based replication• Available in 5.1 and later• can be changed using “binlog_format” •   row •   mixed •   statement (d...
row-based replication• DO •   when you have expensive queries in the     master •   when you are using non-deterministic  ...
DAAG  EN     semi-synch replication
semi-synchronous          replication• Available in 5.5 and higher• Makes sure that at least one slave has  copied the dat...
client                                       master   transaction                           1                    with regu...
client                                     transaction                                       master                       ...
semi-synchronous      replication in practice• installation:  •   it’s a plugin.  •   Actually, two plugins
semi-synch replication install# in the masterplugin-load=rpl_semi_sync_master=semisync_master.sorpl_semi_sync_master_enabl...
semi-synch replication check# in the mastershow variables like rpl_semi%;+------------------------------------+-------+| V...
semi-synch replication checkshow status like "rpl_semi_%tx";+-----------------------------+-------+| variable_name        ...
semi-synch replication testmaster> create table t1 ( i int);Query OK, 0 rows affected (0.01 sec)master> show status like "...
disabling semi-synch# for each slaveset global rpl_semi_sync_slave_enabled=0;stop slave io_thread;start slave io_thread;
disabled semi-synch replication testmaster> insert into t1 values (1);Query OK, 1 row affected (10.00 sec)master> show sta...
disabled semi-synch replication testmaster> insert into t1 values (2);Query OK, 1 row affected (0.01 sec)master> show stat...
re-enabling semi-synch# in one slaveset global rpl_semi_sync_slave_enabled=1;stop slave io_thread;start slave io_thread;
reenabled semi-synch replication testmaster> insert into t1 values (3);Query OK, 1 row affected (0.01 sec)master> show sta...
DAAG  EN     delayed replication
delayed replication in practiceSTOP SLAVE;change master to master_delay=60;START SLAVE;
delayed replication• DEMO
DAAG  EN     Circular replication
Circular A A      MySQL                           B        DBMS                   MySQL                               DBMS...
DAAG  EN     bi-directional replication
bi-directional replication
bi-directional replication                 MySQL   MySQL                 DBMS    DBMSLos Angeles                          ...
DAAG  EN     Leveraging replication
Overview of MySQL partitions
Partition pruning1a - unpartitioned table - SINGLE RECORD             ID                          select *             N  ...
Partition pruning2a - table partitioned by colx - SINGLE REC      1-99    100-199                  select *               ...
Partition pruning1a - unpartitioned table - RANGE                               select *              I                fro...
Partition pruning2a - table partitioned by colx - RANGE      1-99                                select *    100-199      ...
Storage comparison engine                       storage                                (GB) innodb (with PK)              ...
Benchmarking resultsengine                         6 month rangeInnoDB                              4 min 30sMyISAM       ...
Partitions limits Little concurrency Need of using the partitioning column
The ARCHIVE storage engine
Size mattersInnodb                    Archive
REPLICATION PARTITIONING      large       dataARCHIVE
Replication                =let the slave do the dirty work
masterbackup                                               slaves                            STOP SLAVE         remove sla...
master  makesummary tables                                           slaves                            STOP SLAVE         ...
masterPartitionsfor heavy                                   innodb                                         non partitioned...
Simulating         master  multiple                                 innodb                                        non part...
DAAG  EN            Tools and tips       • MySQL Sandbox        • Replaying binary logs        • Filtering binary logs    ...
DAAG  EN     MySQL Sandbox
MySQL Sandbox     http://mysqlsandbox.net• Free software (Perl under GPL)• One (unix) host• Many database servers• Single ...
overviewMySQL                     MySQLserver                    server         DATA DIRECTORY              PORT          ...
overview MySQL                                MySQL server                               server                     SAME  ...
overview MySQL                               MySQL server                              server                     SAME    ...
The hard way (1)
the hard way (2)
The easy way$ make_sandbox     /path/to/mysql-5.1.54_linux.tar.gz# it should work always
The easier way$ make_sandbox 5.1.54# Needs some preliminary work
The easiest way$ sb 5.1.54# Needs the same preliminary work
MySQL Sandbox                        VERSION               MySQL               server$SANDBOX_HOME/msb_VERSION/data       ...
MySQL Sandbox                       5.1.54              MySQL              server$SANDBOX_HOME/msb_5_1_54/data            ...
MySQL Sandbox                       5.5.9              MySQL              server$SANDBOX_HOME/msb_5_5_09/data             ...
Single Sandbox  MySQL        customized scripts  server   start    stop restart  status   clearsend_kill     use
Multiple Sandbox MySQL         customized scripts server  start_all   stop_allrestart_all m n1 status_all s1 n2  clear_all...
Where do you get it•from CPAN sudo cpan MySQL::Sandbox•from launchpad http://launchpad.net/mysql-sandbox
The easy replication way$ make_replication_sandbox     /path/to/mysql-5.1.54_linux.tar.gz# or, after some preparation$ mak...
default architecture                            $HOME               /sandboxes            opt                             ...
default architecture                          $HOME             /sandboxes            optmsb_5_0_91                       ...
creating a single sanboxmake_sandbox    /path/to/mysql-X.X.XX-OS.tar.gz
using a single sanbox# after# make_sandbox #   /path/to/mysql-X.X.XX-OS.tar.gz$ cd $SANDBOX_HOME/msb_X_X_XX$ ./use
creating a single sanbox   with a specific options filemake_sandbox    /path/to/mysql-X.X.XX-OS.tar.gz    --my_file=/path/to...
easily create a sandbox after the first                   one              The long way$ cd $HOME/opt/mysql   # $SANDBOX_BI...
easily create a sandbox after the first                   one             The short way$ make_sandbox   path/to/mysql-5.1.3...
starting a single sanbox$ cd $SANDBOX_HOME/msb_X_X_XX$ ./start
starting a single sanbox    with temporary options$ cd $SANDBOX_HOME/msb_X_X_XX$ ./start --option=value$ ./restart --optio...
creating a sandbox with custom port            and directory  $ make_sandbox 5.1.34      --sandbox_port=7800      --sandbo...
creating a sandbox with automatic           port checking $ make_sandbox 5.1.34 --check_port # if 5.1.34 is free #    port...
create a replication sandbox$ make_replication_sandbox   path/to/mysql-5.1.34-osx10.5-x86.tar.gz
create a circular replication sandbox$ make_replication_sandbox   --circular=4   path/to/mysql-5.1.34-osx10.5-x86.tar.gz
changing port to an existing sandbox$ sbtool -o port     -s /path/to/source/sandbox     --new_port=XXXX
installing the innodb plugin$ sbtool -o plugin     --plugin=innodb     -s /path/to/source/sandbox
creating a replication sandbox with               new base port$ make_replication_sandbox     --replication_directory=neww...
DAAG  EN     Replaying binary logs
DAAG  EN     Filtering binary logs
DAAG  EN     Maatkit
maatkit•   mk-heartbeat Monitor MySQL replication delay.•   mk-purge-logs Purge binary logs on a master based on purge rul...
Facebook Online Schema Change    DA  ENAG
Facebook Online Schema Change•   Based on one of Shlomi Noach OpenArk utilities•   Developed in PHP by Facebook•   http://...
DAAG  EN     OpenArk
openark•   oak-get-slave-lag: print slave replication lag and    terminate with respective exit code.•   oak-online-alter-...
DAAG  EN     Check replication speed
check replication speed• Write a time to the master• Retrieve it from the slave• Compare times• http://forge.mysql.com/too...
DAAG  EN     Replication outside the box       • Seamless failover       • Parallel replication       • Multiple source re...
Advanced MySQLReplication for the      masses
Once Upon ATime, In The Life Of A Database  Consultant ...
The story of a steel       foundry• Used MySQL databases to store  production monitoring data• Inserted a zillion records ...
transaction                                             transaction                                   MySQL       transact...
The story of a shoe        maker• Had a successful business, spread to a  dozen stores.• Needed to aggregate the data from...
MySQL   MySQL                           DBMS     MySQL   DBMS                                    DBMS                     ...
The story of a widgets        seller• Had a successful business, designed for one  server.• Products were created in sever...
MySQL   MySQL DBMS    DBMSmaster   master MySQL   MySQL DBMS    DBMSmaster   master
All these stories tell         us: Nice dream, but MySQL can’t do it
Enter Tungsten Replicator
Tungsten Replicator 2.0• What is it?
http://code.google.com/p/tungsten-replicator
http://code.google.com/p/tungsten-replicator                 Open Source
http://code.google.com/p/tungsten-replicator                 Open Source                 100% GPL v2
What can it do?• Easy failover• Multiple masters• Multiple sources to a single slave• Parallel replication• Replicate to O...
MySQL to foreign services                      MySQL setup                        to run as                      MySQL mas...
From the beginning ...
Fail-over (1)
Fail-over (2)
Fail-over (3)
Fail-over (4)
Fail-over (5)
Fail-over (6)
Fail-over (7)
Fail-over (8)
Failover• DEMO
master/slave with an attitude
The steel foundry dream or parallel replication             From here ...
The steel foundry dream or parallel replication              To here.
Parallel replication facts• Sharded by database• Good choice for slave lag problems• Bad choice for single database projects
Testing parallel  replication
db0                        preparation (1) db1                                 sysbench      db2                          ...
preparation (2)
before the test (1)
before the test (2)   SQL thread  RELAY logs                    IO thread                                          binary ...
starting the test   SQL thread  RELAY logs                    IO thread                                         binary log...
MySQL native      replicationslave catch up in 00:02:30
Tungsten parallel       replicationslave catch up in 00:01:20
The widget sellerdream, or multi masters• Tungsten Replicator recipe: use more  services
Bi-directional replication
Bi-directional replication with slaves
True multiple master We’ll see that in a moment.           But first
The shoe maker dream, or multiple sources• Tungsten Replicator recipe is still valid: use  more services
Multiple source replication
Multiple masters replication: 3 nodes
Multiple masters replication: 4 nodes
Updating 4 masters : 1 flow
Updating 4 masters : 2 flows
Updating 4 masters : 3 flows
Updating 4 masters : 4 flows
Tungsten in practice• installation
Installation:         the long way• Get the binaries• Expand the tarball• Check the requirements in the manual• Run ./confi...
Installation:         the quick way• Get the tarball• Get the tungsten deployer from Google  Code• Sit back and enjoy!
Tools• replicator• trepctl• thl
replicator• It’s the service provider• You launch it once when you start• You may restart it when you change config
trepctl• Tungsten Replicator ConTroLler• It’s the driving seat for your replication• You can start, update, and stop servi...
thl• Transaction History List• Gives you access to the Tungsten relay logs
DAAG  EN     More info on replication
Books
High Performance MySQL
MySQL High Availability
Web Operations
Cloud Application Architectures
Talks•   Monday, 1:30pm “Learn how to cure replication    deprivation with Tungsten”•   Tuesday 10:50am “Replication Updat...
ADVERTISING
WE ARE HIRING!QA and support  engineershttp://www.continuent.com/about/careers
http://opendbcamp.org         Open Database Camp               Sardinia           Technology Park           6-7-8 May 2011
Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 ...
My sql replication advanced techniques presentation
Upcoming SlideShare
Loading in …5
×

My sql replication advanced techniques presentation

6,581 views

Published on

Published in: Technology
1 Comment
6 Likes
Statistics
Notes
  • Hi

    I am kindly asking that you please send me a step by step tutorial on how to set up fan in replication with two separate databases on two master servers being replicated onto one shared database on a slave machine using tungsten replicator. It is my first time to use tungsten so I need a detailed approach right from installation to configuration and lastly testing.

    I have searched the internet and all I can find is the fact that it is possible but not how exactly to do so.

    Your assistance will be greatly appreciated.

    Rumbi
    mukrue@yahoo.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
6,581
On SlideShare
0
From Embeds
0
Number of Embeds
107
Actions
Shares
0
Downloads
342
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • Common web applications connect the web servers through load balancers. These applications are easy to scale at the web server level. They simply update the load balancer list, and the application is able to serve web pages at a reasonable speed. The database server is usually able to handle the load of many web servers when the application is created. However, soon enough it becomes a SPOF\n
  • Common web applications connect the web servers through load balancers. These applications are easy to scale at the web server level. They simply update the load balancer list, and the application is able to serve web pages at a reasonable speed. The database server is usually able to handle the load of many web servers when the application is created. However, soon enough it becomes a SPOF\n
  • The way MySQL database evolves is using the &quot;scale out&quot; technique.\nInstead of buying more expensive servers, you run a master and several slaves on commodity hardware. The master handles the write requests, and a small portion of the reads (usually when they come within transactions) and the slaves handle the read requests.\nThe downside of this scheme is that your application must be able to split write requests from read ones.\n\n
  • How does replication work? The writes go to the master. As soon as the transaction is successfully executed, the master writes it to the binary load.\nEach slave has two parallel threads dedicated to replication. The IO thread will fetch the events from the master binary log, saving them into a relay log. The SQL thread will take the events from the relay log and reproduce them in the slave\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • First of all, you must make sure that replication is working.\nIf you monitor the master, it won&amp;#x2019;t tell you anything about replication problems.\nYou need to keep an eye on the whole system.\nMonitoring SHOW SLAVE STATUS will tell you if replication is running, and it will give you some clues on whether the process is going well.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Replacing the master, instead, is probably the most complex operation in replication.\nFinding the transactions that are present in the most up to date slave (the one promoted to master) but missing in the other slaves, and making sure that each of the other slaves catches up the missing ones is a long manual chore.\nThe reason for this difficulty is that MySQL does not have a global transaction ID.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • With regular replication, you commit a transaction, and get back a result from the server. For you, that transaction is stored, and you walk away pleased of your work.\nHowever, if the master crashes after it has sent you the result and before the slave has had a chance to get the record, you have SILENTLY lost a record, which is the worst possible scenario, because you don&amp;#x2019;t know you miss it.\n
  • WIth semi-synchronous replication, instead, the master does not send the result to the client right away. It will first alert the slave that a new transaction is ready in the binary log. The slave will get the transaction and store it in its relay log, and tells the master that it has got a copy of the transaction. Only then the master tells the client that the commit was successful. If the master crashes before returning to the client, the client will get an error, and will not miss a transaction.\nNotice that the operation tells only that the transaction has reached the slave, not that it has been committed to the slave. It is not a two-phase commit.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • MySQL Sandbox is a tool that installs an additional MySQL server in a few seconds.\nYou can have more than one database servers in your host, all of them independent from each other. Installing a new server with MySQL Sandbox is a matter of seconds. You can install either a single sandbox or a system including multiple ones. Sandboxed servers can be stand-alone or related in standard or circular replication systems.\nIt is free software, and it works on most every unix OS. In combination with Gearman, it can install and setup replication systems across multiple hosts.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How do I get an eval copy of the software\n\nCan you share any whitepapers on this technology?\n\nWhat is the price?\nOur dedicated server price is based upon CPU&amp;#x2019;s and the size of the cluster\nThe monthly subscription starts at $125 list for a 2 node.\nVirtual Private Clusters are priced differently and I would be glad to discuss this off-line\n\nIs the entire database kept in memory?\n\nCan I make schema changes while the cluster is running\n\nHow do you differ from Veritas Cluster Server?\n
  • My sql replication advanced techniques presentation

    1. 1. Advanced MySQLReplication TechniquesGiuseppe Maxia, QA Director, Continuent, Inc Facts. And Demos. Possibly fun
    2. 2. about me - Giuseppe Maxia• a.k.a. The Data Charmer• QA Director at Continuent, Inc• Long time hacking with MySQL features• Formerly, community manager,db consultant, designer, coder.• A passion for QA and open source• Blogger• http://datacharmer.blogspot.com
    3. 3. Agenda• Replication basics• Replication blues • Master switch and failover • Slave lagging • Gotchas• Replication power techniques and features • row-based, semi-synch, delayed • Circular, bi-directional • leveraging replication• Tools and tips• Replication outside the box • seamless failover, parallel, multi-master
    4. 4. DAAG EN Replication basics • Principles • How to set up replication • Monitoring replication
    5. 5. DAAG EN Principles
    6. 6. database server a simple web application scheme r/w requests web server clients 6
    7. 7. database server scaling web requests r/w requests web servers load balancer clients 7
    8. 8. read/write master a web application read/only slaves scheme with replication load balancerR/W R/O web servers load balancer clients 8
    9. 9. client master transaction binary log readsslave IO thread relay log replication concepts SQL thread reads 9
    10. 10. DAAG EN replication setup
    11. 11. 1 SHUT DOWN THE DATABASE SERVER Master 11
    12. 12. 2 MAKE A BACKUP COPY Master 12
    13. 13. 3 ENABLE THE MASTER Master Configuration file [mysqld] log-bin=mysql-bin server-id=1 13
    14. 14. 4 RESTART THE MASTER Master 14
    15. 15. 5 CREATE REPLICATION USER Master SQL command GRANT REPLICATION SLAVE ON *.* to slave_user@10.10.100.% IDENTIFIED BY slave_pass; 15
    16. 16. 6 INSTALL MySQL on the slave Slave 1Make sure that:• Youre using the same version of MySQL• You have the same directory structure• The server is not started yet 16
    17. 17. 7 COPY THE MASTER DATA to the slave Slave 1 17
    18. 18. 8 ENABLE THE SLAVE Slave 1 Configuration file [mysqld] server-id=2 relay-log=mysql-relay read-only # optional: log-bin=mysql-bin 18
    19. 19. 9 START THE SLAVE SERVER Slave 1 19
    20. 20. 10 INITIALIZE THE SLAVE Slave 1 SQL command CHANGE MASTER TO MASTER_HOST=master_IP, MASTER_PORT=3306, MASTER_USER=slave_user, MASTER_PASSWORD=slave_pwd; 20
    21. 21. 11 START THE SLAVE SERVICE Slave 1 SQL command START SLAVE; 21
    22. 22. 12 CHECK THE SLAVE Slave 1 SQL command SHOW SLAVE STATUS G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... 22
    23. 23. Troubleshooting• SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No • Make sure that the slave host can connect to the master • Make sure that master and slave have different Server-id • Check the error log of both master and slave
    24. 24. Testing the slave• Create a table in the master.• Make sure that the slave has replicated the table.• Insert data in the master• read that data in the slave
    25. 25. DAAG EN Monitoring
    26. 26. Sample get slave statusmonitoring slave Running? No Yes master Get master binlog and position Same or later No Yes binlog/position? check table contents alert 26
    27. 27. monitoring replicationmaster> SHOW MASTER STATUSslave> SHOW SLAVE STATUSFULL SCRIPTS:http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-mysql.htmlhttp://forge.mysql.com/tools/tool.php?id=6
    28. 28. show master status File: mysql-bin.000002 Position: 78045744 Binlog_Do_DB:Binlog_Ignore_DB:
    29. 29. show slave status Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: rsandbox Master_Port: 27371 Connect_Retry: 60 Master_Log_File: mysql-bin.000002 Read_Master_Log_Pos: 78045744 Relay_Log_File: mysql_sandbox27372-relay-bin.000055 Relay_Log_Pos: 78045889Relay_Master_Log_File: mysql-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes
    30. 30. show slave status... Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: ...
    31. 31. show slave status... Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 78045744 Relay_Log_Space: 78046100... Seconds_Behind_Master: 0... Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:
    32. 32. DAAG EN Replication blues • Master switch and failover • Slave lagging • Gotchas
    33. 33. DAAG EN Master switch and failover
    34. 34. Replacing the Master master crashe s Let all slaves catch up with execution STOP replication in all slaves FIND the most make it the up to date slave master FIND which run missing connect all transactions are transactions slaves to the missing from to other new master other slaves slaves Stop 34
    35. 35. Planned master switch• Stop accepting writes• Wait until all slaves have caught up• Stop replication in all slaves• Promote a slave to master• Point all slaves to the new master
    36. 36. Changing a failed master• Pre-requisite:• log_bin and log_slave_updates must be enabled in all the slaves• If not, there is more manual labor
    37. 37. Changing a failed master (1)• Wait until all slaves have caught up• Identify the most advanced slave• Make that slave the new master• ... so far, so good
    38. 38. Changing a failed master (2) • For each remaining slave: • Find the missing statements• Find the LAST statement replicated by the slave• Find the same statement in the new master binlog (*)• get the position of the NEXT statement (*) if log_slave_updates was not enabled, you need to convert the relay log statements to SQL and do the next step manually
    39. 39. Changing a failed master (3)• For each remaining slave:• Apply the missing statements • CHANGE MASTER TO master_host=”new_master_hostname”, master_port=new_master_port, master_log_file=”mysql-bin.xxxxxx”, master_log_pos=YYYY
    40. 40. Reasons for complexity• No global transaction ID
    41. 41. What is a global transaction ID • A unique identifier of a transaction • Unique for the whole cluster, not for each node • Generated by the ultimate source (the master) • Does not change when the transaction goes through an intermediate master
    42. 42. Why should you care• Failure recovery• MySQL DOES NOT have it
    43. 43. Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- -------- Transactions Transactions -------- -------- ======== -------- ======== -------- ======== -------- ========
    44. 44. The master crashes Slave 3Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- -------- Transactions Transactions -------- -------- ======== -------- ======== -------- ======== -------- ========
    45. 45. Where do the slaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary logTransactions Transactions Transactions======== ======== ================ ======== ================ ======== ================ ======== ========
    46. 46. Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ========Transactions Transactions ================ ================ ================ ================ ========
    47. 47. Need to synch the missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ========Transactions Transactions ================ ================ ================ ================ ========
    48. 48. Transaction position in the new master binlog New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ========Transactions Transactions ================ ================ ================ ? ======== ?======== ========
    49. 49. An existing solution• Google has made a patch to implement global transaction IDs• HOWEVER• It only works with 5.0• It doesnt work with row based replication
    50. 50. Hackahead!
    51. 51. CREATE TABLE Global_Trans_ID ( number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY) ENGINE = MyISAM;CREATE TABLE Last_Exec_Trans ( server_id INT UNSIGNED, trans_id INT UNSIGNED) ENGINE = InnoDB;
    52. 52. CREATE PROCEDURE commit_trans ()BEGIN DECLARE trans_id, server_id INT UNSIGNED; SET SQL_LOG_BIN = 0; INSERT INTO global_trans_id() values (); SELECT LAST_INSERT_ID() INTO trans_id, @@server_id INTO server_id; DELETE FROM global_trans_id where number < trans_Id; SET SQL_LOG_BIN = 1; INSERT INTO last_exec_trans(server_id, trans_id) VALUES (server_id, trans_id);COMMIT;END
    53. 53. Hackingthe hack
    54. 54. Simplifying the scheme• No MyISAM table and LAST_INSERT_ID()• Use UUID_SHORT() instead• Replace the InnoDB table with a BlackHole one
    55. 55. Advantages• 10 times faster• No additional tables and storage required
    56. 56. Disadvantages• The master SERVER ID must be < 256
    57. 57. Another solution• We’ll see that in the second part of the talk
    58. 58. DAAG EN Slave lagging
    59. 59. Slave lagging• Common causes: • Excess of traffic • Slave restart • Long DDL operations
    60. 60. Slave lagging transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio ntra csio s ac ctio ac ion ion n an n s an nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio t tratraREPLICATION MySQL replication is SINGLE THREAD transaction transaction MySQL DBMS
    61. 61. Slave lagging remedies• row-based replication (helps in some cases)• Expensive query split• Double apply (dangerous)• Slave query prefetch (requires external app)• Parallel replication (requires external app)
    62. 62. statement-based replicationmaster> set global binlog_format=‘statement’;master> insert into test.t1 select count(*) from information_schema.columns;master> show global status like opened_tables;+---------------+-------+| Variable_name | Value |+---------------+-------+| Opened_tables | 41 |+---------------+-------+slave> show global status like opened_tables;+---------------+-------+| Variable_name | Value |+---------------+-------+| Opened_tables | 41 |+---------------+-------+
    63. 63. row-based replicationmaster> set global binlog_format=‘row’;master> insert into test.t1 select count(*) from information_schema.columns;master> show global status like opened_tables;+---------------+-------+| Variable_name | Value |+---------------+-------+| Opened_tables | 41 |+---------------+-------+slave> show global status like opened_tables;+---------------+-------+| Variable_name | Value |+---------------+-------+| Opened_tables | 17 |+---------------+-------+
    64. 64. Expensive query split# instead of this:insert into t1 select benchmark(10000000, sha(abc));Query OK, 1 row affected (4.19 sec)Records: 1 Duplicates: 0 Warnings: 0# you may do thisset @result=(select benchmark(10000000, sha(abc)));Query OK, 0 rows affected (4.19 sec)insert into t1 values (@result);Query OK, 1 row affected (0.00 sec)
    65. 65. Expensive query split# at 471#110410 8:21:21 server id 1 end_log_pos 518User_varSET @`result`:=0/*!*/;# at 518#110410 8:21:21 server id 1 end_log_pos 612 Querythread_id=3 exec_time=0 error_code=0SET TIMESTAMP=1302448881/*!*/;insert into t1 values (@result)
    66. 66. Slave query prefetch• What is it? • a technique that fetches queries from the relay logs, and runs them as SELECT statements, to warm up the indexes• How do you do it? • mk-slave-prefetch (http://www.maatkit.org) • Slave readahead http://sourceforge.net/ projects/slavereadahead/
    67. 67. DAAG EN Gotchas
    68. 68. Gotchas• When you thought that you had figured out everything ...
    69. 69. Gotchas: LOAD DATA• LOAD DATA INFILE is replicated as a temporary file• The slave needs three times the size of the data: • the relay log • the temporary file • the data itself
    70. 70. Gotchas: default engine• If you define storage_engine in the master, it does not get replicated
    71. 71. Gotchas: binlog format• In circular replication, and relay slaves• Changes to binlog_format are not propagated
    72. 72. Gotchas: binlog format server id: 101 insert into t1format: statement values (@@server_id) 101 server id: 102 format: row 102 server id: 103 102 format: row
    73. 73. Gotchas: set global not really global• set global binlog_format=row• Works for all new connections after the change• It does not work for the event scheduler!
    74. 74. Gotchas: triggers and row replication• In statement based replication, triggers are fired both on master and slave• in row-based replication, triggers are fired on master only, and the resulting rows are moved to the slave
    75. 75. DAAG EN Replication power techniques and features • row-based • semi-synch • delayed • Circular • bi-directional • Leveraging replication
    76. 76. DAAG EN row-based replication
    77. 77. row-based replication• Available in 5.1 and later• can be changed using “binlog_format” • row • mixed • statement (default)
    78. 78. row-based replication• DO • when you have expensive queries in the master • when you are using non-deterministic functions• DON’T • When you are updating a lot of rows
    79. 79. DAAG EN semi-synch replication
    80. 80. semi-synchronous replication• Available in 5.5 and higher• Makes sure that at least one slave has copied the data.• Increases reliability
    81. 81. client master transaction 1 with regular commit replication execute 2 slave returns4 to client binary log 3 5 replication 81
    82. 82. client transaction master 1 with semi- commit synchronous replication execute 2 slave7 returns to client binary log 3 sends 4 transaction relay log 5 to slave gets 6 acknowledgement 82
    83. 83. semi-synchronous replication in practice• installation: • it’s a plugin. • Actually, two plugins
    84. 84. semi-synch replication install# in the masterplugin-load=rpl_semi_sync_master=semisync_master.sorpl_semi_sync_master_enabled=1# in each slaveplugin-load=rpl_semi_sync_slave=semisync_slave.sorpl_semi_sync_slave_enabled=1# restar all servers
    85. 85. semi-synch replication check# in the mastershow variables like rpl_semi%;+------------------------------------+-------+| Variable_name | Value |+------------------------------------+-------+| rpl_semi_sync_master_enabled | ON || rpl_semi_sync_master_timeout | 10000 || rpl_semi_sync_master_trace_level | 32 || rpl_semi_sync_master_wait_no_slave | ON |+------------------------------------+-------+
    86. 86. semi-synch replication checkshow status like "rpl_semi_%tx";+-----------------------------+-------+| variable_name | value |+-----------------------------+-------+| RPL_SEMI_SYNC_MASTER_NO_TX | 0 || RPL_SEMI_SYNC_MASTER_YES_TX | 0 |+-----------------------------+-------+
    87. 87. semi-synch replication testmaster> create table t1 ( i int);Query OK, 0 rows affected (0.01 sec)master> show status like "rpl_semi_%tx";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| Rpl_semi_sync_master_no_tx | 0 || Rpl_semi_sync_master_yes_tx | 1 |+-----------------------------+-------+
    88. 88. disabling semi-synch# for each slaveset global rpl_semi_sync_slave_enabled=0;stop slave io_thread;start slave io_thread;
    89. 89. disabled semi-synch replication testmaster> insert into t1 values (1);Query OK, 1 row affected (10.00 sec)master> show status like "rpl_semi_%tx";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| Rpl_semi_sync_master_no_tx | 1 || Rpl_semi_sync_master_yes_tx | 1 |+-----------------------------+-------+2 rows in set (0.00 sec)
    90. 90. disabled semi-synch replication testmaster> insert into t1 values (2);Query OK, 1 row affected (0.01 sec)master> show status like "rpl_semi_%tx";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| Rpl_semi_sync_master_no_tx | 2 || Rpl_semi_sync_master_yes_tx | 1 |+-----------------------------+-------+2 rows in set (0.00 sec)
    91. 91. re-enabling semi-synch# in one slaveset global rpl_semi_sync_slave_enabled=1;stop slave io_thread;start slave io_thread;
    92. 92. reenabled semi-synch replication testmaster> insert into t1 values (3);Query OK, 1 row affected (0.01 sec)master> show status like "rpl_semi_%tx";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| Rpl_semi_sync_master_no_tx | 2 || Rpl_semi_sync_master_yes_tx | 2 |+-----------------------------+-------+2 rows in set (0.00 sec)
    93. 93. DAAG EN delayed replication
    94. 94. delayed replication in practiceSTOP SLAVE;change master to master_delay=60;START SLAVE;
    95. 95. delayed replication• DEMO
    96. 96. DAAG EN Circular replication
    97. 97. Circular A A MySQL B DBMS MySQL DBMS B master master MySQLDD DBMS MySQL DBMS C C master
    98. 98. DAAG EN bi-directional replication
    99. 99. bi-directional replication
    100. 100. bi-directional replication MySQL MySQL DBMS DBMSLos Angeles New York MySQL MySQL MySQL DBMS DBMS DBMS MySQL DBMS
    101. 101. DAAG EN Leveraging replication
    102. 102. Overview of MySQL partitions
    103. 103. Partition pruning1a - unpartitioned table - SINGLE RECORD ID select * N from table_nameA D where colx =T E 120A X
    104. 104. Partition pruning2a - table partitioned by colx - SINGLE REC 1-99 100-199 select * from table_name 200-299 where colx = 120 300-399 400-499 500-599
    105. 105. Partition pruning1a - unpartitioned table - RANGE select * I fromD N table_nameA where colx D between 120T E and 230A X
    106. 106. Partition pruning2a - table partitioned by colx - RANGE 1-99 select * 100-199 from table_name 200-299 where colx between 120 and 230 300-399 400-499 500-599
    107. 107. Storage comparison engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13
    108. 108. Benchmarking resultsengine 6 month rangeInnoDB 4 min 30sMyISAM 25.03sArchive 22 min 25sInnoDB partitioned by month 13.19MyISAM partitioned by year 6.31MyISAM partitioned by month 4.45Archive partitioned by year 16.67Archive partitioned by month 8.97
    109. 109. Partitions limits Little concurrency Need of using the partitioning column
    110. 110. The ARCHIVE storage engine
    111. 111. Size mattersInnodb Archive
    112. 112. REPLICATION PARTITIONING large dataARCHIVE
    113. 113. Replication =let the slave do the dirty work
    114. 114. masterbackup slaves STOP SLAVE remove slave perform from load backup balancer attach slave START Let slave to load SLAVE catch up balancer
    115. 115. master makesummary tables slaves STOP SLAVE calculate remove slave summary from load tables balancer attach slave Let slave START SLAVE to load catch up balancer
    116. 116. masterPartitionsfor heavy innodb non partitioned statistics slave slave innodb innodb non partitioned partitioned by range slave MyISAM partitioned by range
    117. 117. Simulating master multiple innodb non partitioneddimensions slave slave innodb non partitioned ARCHIVE partitioned by range (date) slave slave ARCHIVE partitioned by range ARCHIVE (location) partitioned by range (product)
    118. 118. DAAG EN Tools and tips • MySQL Sandbox • Replaying binary logs • Filtering binary logs • Baron Schwartz’s Maatkit • Shlomi Noach’s openark kit • Facebook’s Online Schema Change • Measuring replication speed • Detecting if replication is on
    119. 119. DAAG EN MySQL Sandbox
    120. 120. MySQL Sandbox http://mysqlsandbox.net• Free software (Perl under GPL)• One (unix) host• Many database servers• Single or multiple sandboxes• Customized scripts to use the servers• Standard or circular replication• Installs IN SECONDS
    121. 121. overviewMySQL MySQLserver server DATA DIRECTORY PORT SOCKET
    122. 122. overview MySQL MySQL server server SAME DATA DIRECTORY?/var/lib/mysql /var/lib/mysql DATA CORRUPTION
    123. 123. overview MySQL MySQL server server SAME PORT or SOCKET? 3306 3306/tmp/mysql.sock /tmp/mysql.sock DOES NOT START
    124. 124. The hard way (1)
    125. 125. the hard way (2)
    126. 126. The easy way$ make_sandbox /path/to/mysql-5.1.54_linux.tar.gz# it should work always
    127. 127. The easier way$ make_sandbox 5.1.54# Needs some preliminary work
    128. 128. The easiest way$ sb 5.1.54# Needs the same preliminary work
    129. 129. MySQL Sandbox VERSION MySQL server$SANDBOX_HOME/msb_VERSION/data VERSION /tmp/mysql_VERSION.sock
    130. 130. MySQL Sandbox 5.1.54 MySQL server$SANDBOX_HOME/msb_5_1_54/data 5154 /tmp/mysql_5154.sock
    131. 131. MySQL Sandbox 5.5.9 MySQL server$SANDBOX_HOME/msb_5_5_09/data 5509 /tmp/mysql_5509.sock
    132. 132. Single Sandbox MySQL customized scripts server start stop restart status clearsend_kill use
    133. 133. Multiple Sandbox MySQL customized scripts server start_all stop_allrestart_all m n1 status_all s1 n2 clear_all s2 n3send_kill_a ll use_all
    134. 134. Where do you get it•from CPAN sudo cpan MySQL::Sandbox•from launchpad http://launchpad.net/mysql-sandbox
    135. 135. The easy replication way$ make_replication_sandbox /path/to/mysql-5.1.54_linux.tar.gz# or, after some preparation$ make_replication_sandbox 5.1.54
    136. 136. default architecture $HOME /sandboxes opt expanded tarballs mysql installedsandboxes
    137. 137. default architecture $HOME /sandboxes optmsb_5_0_91 mysqlmsb_5_1_48 5.0.91rsandbox_5_1_48 5.1.45 master 5.1.48 node1 5.5.4 node2
    138. 138. creating a single sanboxmake_sandbox /path/to/mysql-X.X.XX-OS.tar.gz
    139. 139. using a single sanbox# after# make_sandbox # /path/to/mysql-X.X.XX-OS.tar.gz$ cd $SANDBOX_HOME/msb_X_X_XX$ ./use
    140. 140. creating a single sanbox with a specific options filemake_sandbox /path/to/mysql-X.X.XX-OS.tar.gz --my_file=/path/to/my.cnf
    141. 141. easily create a sandbox after the first one The long way$ cd $HOME/opt/mysql # $SANDBOX_BINARY$ gunzip -c /path/to/mysql-5.1.34-osx10.5-x86.tar.gz | tar -xf -$ mv mysql-5.1.34-osx10.5-x86 5.1.34$ make sandbox 5.1.34
    142. 142. easily create a sandbox after the first one The short way$ make_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gz --export_binaries
    143. 143. starting a single sanbox$ cd $SANDBOX_HOME/msb_X_X_XX$ ./start
    144. 144. starting a single sanbox with temporary options$ cd $SANDBOX_HOME/msb_X_X_XX$ ./start --option=value$ ./restart --option=value$ ./start --key-buffer=20000000
    145. 145. creating a sandbox with custom port and directory $ make_sandbox 5.1.34 --sandbox_port=7800 --sandbox_directory=mickeymouse
    146. 146. creating a sandbox with automatic port checking $ make_sandbox 5.1.34 --check_port # if 5.1.34 is free # port=5134 # directory=msb_5_1_34 # else # port=5135 (or the first free) # directory=msb_5_1_34_a
    147. 147. create a replication sandbox$ make_replication_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gz
    148. 148. create a circular replication sandbox$ make_replication_sandbox --circular=4 path/to/mysql-5.1.34-osx10.5-x86.tar.gz
    149. 149. changing port to an existing sandbox$ sbtool -o port -s /path/to/source/sandbox --new_port=XXXX
    150. 150. installing the innodb plugin$ sbtool -o plugin --plugin=innodb -s /path/to/source/sandbox
    151. 151. creating a replication sandbox with new base port$ make_replication_sandbox --replication_directory=newwdir --check_base_port 5.0.79# Creates a replication directory under# $SANDBOX_HOME/newdir# The previous one is preserved.# No conflicts happen
    152. 152. DAAG EN Replaying binary logs
    153. 153. DAAG EN Filtering binary logs
    154. 154. DAAG EN Maatkit
    155. 155. maatkit• mk-heartbeat Monitor MySQL replication delay.• mk-purge-logs Purge binary logs on a master based on purge rules.• mk-slave-delay Make a MySQL slave server lag behind its master.• mk-slave-find Find and print replication hierarchy tree of MySQL slaves.• mk-slave-move Move a MySQL slave around in the replication hierarchy.• mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm caches.• mk-slave-restart Watch and restart MySQL replication after errors.• mk-table-checksum Perform an online replication consistency check, or checksum MySQL tables efficiently on one or many servers.• mk-table-sync Synchronize MySQL table data efficiently.• http://www.maatkit.org/
    156. 156. Facebook Online Schema Change DA ENAG
    157. 157. Facebook Online Schema Change• Based on one of Shlomi Noach OpenArk utilities• Developed in PHP by Facebook• http://www.facebook.com/note.php?note_id=430801045932
    158. 158. DAAG EN OpenArk
    159. 159. openark• oak-get-slave-lag: print slave replication lag and terminate with respective exit code.• oak-online-alter-table: perform a non-blocking ALTER TABLE operation.• oak-purge-master-logs: purge master logs, depending on the state of replicating slaves.• oak-show-replication-status: show how far behind are replicating slaves on a given master.• http://code.openark.org/forge/openark-kit
    160. 160. DAAG EN Check replication speed
    161. 161. check replication speed• Write a time to the master• Retrieve it from the slave• Compare times• http://forge.mysql.com/tools/tool.php? id=5
    162. 162. DAAG EN Replication outside the box • Seamless failover • Parallel replication • Multiple source replication • Multiple master replication
    163. 163. Advanced MySQLReplication for the masses
    164. 164. Once Upon ATime, In The Life Of A Database Consultant ...
    165. 165. The story of a steel foundry• Used MySQL databases to store production monitoring data• Inserted a zillion records per second.• Slaves often lagged behind
    166. 166. transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio ntra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio ct tratraREPLICATION transaction transaction MySQL DBMS
    167. 167. The story of a shoe maker• Had a successful business, spread to a dozen stores.• Needed to aggregate the data from the stores in his headquarters database.
    168. 168. MySQL MySQL DBMS MySQL DBMS DBMS storestore store MySQL DBMS headquarters
    169. 169. The story of a widgets seller• Had a successful business, designed for one server.• Products were created in several sites.• Needed to allow insertions from more than one master at the time
    170. 170. MySQL MySQL DBMS DBMSmaster master MySQL MySQL DBMS DBMSmaster master
    171. 171. All these stories tell us: Nice dream, but MySQL can’t do it
    172. 172. Enter Tungsten Replicator
    173. 173. Tungsten Replicator 2.0• What is it?
    174. 174. http://code.google.com/p/tungsten-replicator
    175. 175. http://code.google.com/p/tungsten-replicator Open Source
    176. 176. http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2
    177. 177. What can it do?• Easy failover• Multiple masters• Multiple sources to a single slave• Parallel replication• Replicate to Oracle and PostgreSQL database
    178. 178. MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle Bin Logs (Binlogs enabled) Slave DB© Continuent 2010
    179. 179. From the beginning ...
    180. 180. Fail-over (1)
    181. 181. Fail-over (2)
    182. 182. Fail-over (3)
    183. 183. Fail-over (4)
    184. 184. Fail-over (5)
    185. 185. Fail-over (6)
    186. 186. Fail-over (7)
    187. 187. Fail-over (8)
    188. 188. Failover• DEMO
    189. 189. master/slave with an attitude
    190. 190. The steel foundry dream or parallel replication From here ...
    191. 191. The steel foundry dream or parallel replication To here.
    192. 192. Parallel replication facts• Sharded by database• Good choice for slave lag problems• Bad choice for single database projects
    193. 193. Testing parallel replication
    194. 194. db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbench
    195. 195. preparation (2)
    196. 196. before the test (1)
    197. 197. before the test (2) SQL thread RELAY logs IO thread binary logsMySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alpha
    198. 198. starting the test SQL thread RELAY logs IO thread binary logsMySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alpha
    199. 199. MySQL native replicationslave catch up in 00:02:30
    200. 200. Tungsten parallel replicationslave catch up in 00:01:20
    201. 201. The widget sellerdream, or multi masters• Tungsten Replicator recipe: use more services
    202. 202. Bi-directional replication
    203. 203. Bi-directional replication with slaves
    204. 204. True multiple master We’ll see that in a moment. But first
    205. 205. The shoe maker dream, or multiple sources• Tungsten Replicator recipe is still valid: use more services
    206. 206. Multiple source replication
    207. 207. Multiple masters replication: 3 nodes
    208. 208. Multiple masters replication: 4 nodes
    209. 209. Updating 4 masters : 1 flow
    210. 210. Updating 4 masters : 2 flows
    211. 211. Updating 4 masters : 3 flows
    212. 212. Updating 4 masters : 4 flows
    213. 213. Tungsten in practice• installation
    214. 214. Installation: the long way• Get the binaries• Expand the tarball• Check the requirements in the manual• Run ./configure• Run ./configure-service
    215. 215. Installation: the quick way• Get the tarball• Get the tungsten deployer from Google Code• Sit back and enjoy!
    216. 216. Tools• replicator• trepctl• thl
    217. 217. replicator• It’s the service provider• You launch it once when you start• You may restart it when you change config
    218. 218. trepctl• Tungsten Replicator ConTroLler• It’s the driving seat for your replication• You can start, update, and stop services• You can get specific info
    219. 219. thl• Transaction History List• Gives you access to the Tungsten relay logs
    220. 220. DAAG EN More info on replication
    221. 221. Books
    222. 222. High Performance MySQL
    223. 223. MySQL High Availability
    224. 224. Web Operations
    225. 225. Cloud Application Architectures
    226. 226. Talks• Monday, 1:30pm “Learn how to cure replication deprivation with Tungsten”• Tuesday 10:50am “Replication Update”• Tuesday 2:00pm “Diagnosing replication failures”• Tuesday 3:05 “Advanced replication monitoring”• Thursday 10:50am “Replication for Availability & Durability with MySQL and Amazon RDS”• Thursday 11:55am “Build your own PaaS for MySQL with Tungsten Enterprise”• Thursday 2:00pm “Error detection and correction with MySQL Replication”• Thursday 2:50pm “Performance comparisons and trade-offs for various MySQL replication schemes”
    227. 227. ADVERTISING
    228. 228. WE ARE HIRING!QA and support engineershttp://www.continuent.com/about/careers
    229. 229. http://opendbcamp.org Open Database Camp Sardinia Technology Park 6-7-8 May 2011
    230. 230. Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://code.google.com/p/tungsten-replicator© Continuent 2011

    ×