Advanced MySQL                     Replication Techniques                         Giuseppe Maxia, QA Director, Continuent,...
about me - Giuseppe Maxia •       a.k.a. The Data Charmer •       QA Director at Continuent, Inc •       Long time hacking...
Agenda        •       Replication basics        •       Replication blues              •          Master switch and failov...
DA     AG        EN                   Replication basics                     • Principles                     • How to set...
DA     AG        EN               PrinciplesMonday, April 11, 2011                5
database server                                                        a simple web                                       ...
database server                                                      scaling web                                          ...
read/write                           master                               a web                                           ...
client                                                         master                                               transa...
DA     AG        EN               replication setupMonday, April 11, 2011                       10
1                   SHUT DOWN THE DATABASE SERVER                                   Master                                ...
2                   MAKE A BACKUP COPY                             Master                                              12M...
3                           ENABLE THE MASTER                                      Master                                C...
4                   RESTART THE MASTER                             Master                                              14M...
5                        CREATE REPLICATION USER                                     Master                               ...
6                   INSTALL MySQL on the slave                           Slave 1Make sure that:• Youre using the same vers...
7                   COPY THE MASTER DATA to the slave                                  Slave 1                            ...
8                            ENABLE THE SLAVE                                    Slave 1                                Co...
9                   START THE SLAVE SERVER                             Slave 1                                            ...
10                           INITIALIZE THE SLAVE                                   Slave 1                               ...
11                        START THE SLAVE SERVICE                                  Slave 1                                ...
12                           CHECK THE SLAVE                                  Slave 1                                 SQL ...
Troubleshooting                     • SHOW SLAVE STATUS says                         SLAVE_IO_RUNNING=No                  ...
Testing the slave                     • Create a table in the master.                     • Make sure that the slave has r...
DA     AG        EN               MonitoringMonday, April 11, 2011                25
Sample                                      get slave                                                 status   monitoring ...
monitoring replication  master> SHOW MASTER STATUS  slave> SHOW SLAVE STATUS  FULL SCRIPTS:  http://datacharmer.blogspot.c...
show master status                             File: mysql-bin.000002                         Position: 78045744          ...
show slave status                                Slave_IO_State: Waiting for master to send event                         ...
show slave status     ...                             Replicate_Do_DB:                         Replicate_Ignore_DB:       ...
show slave status  ...                                    Last_Errno: 0                                    Last_Error:    ...
DA     AG        EN                        Replication blues                         •   Master switch and failover       ...
DA     AG        EN               Master switch and failoverMonday, April 11, 2011                                33
Replacing the                              Master     master                                  crashe                      ...
Planned master switch                     • Stop accepting writes                     • Wait until all slaves have caught ...
Changing a failed master                     • Pre-requisite:                     • log_bin and log_slave_updates must be ...
Changing a failed master (1)                     • Wait until all slaves have caught up                     • Identify the...
Changing a failed master (2)                     • For each remaining slave:                     • Find the missing statem...
Changing a failed master (3)                     • For each remaining slave:                     • Apply the missing state...
Reasons for complexity                     • No global transaction IDMonday, April 11, 2011                            40
What is a global transaction ID                     • A unique identifier of a transaction                     • Unique fo...
Why should you care                     • Failure recovery                     • MySQL DOES NOT have itMonday, April 11, 2...
Defining the problem                                                                       Slave 3                         ...
The master crashes                                                                      Slave 3                         Ma...
Where do the slaves                               stand?          Slave 1              Slave 2        Slave 3          Bin...
Slave 3 becomes                              master                                            New master          Slave 1...
Need to synch the                         missing transactions                                              New master    ...
Transaction position in                  the new master binlog                                                New master  ...
An existing solution                     • Google has made a patch to implement global                       transaction I...
Hack                         ahead!Monday, April 11, 2011            50
CREATE TABLE Global_Trans_ID (                         number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY            ) ENGINE ...
CREATE PROCEDURE commit_trans ()  BEGIN        DECLARE trans_id, server_id INT UNSIGNED;        SET SQL_LOG_BIN = 0;      ...
Hacking                         the hackMonday, April 11, 2011              53
Simplifying the scheme                     • No MyISAM table and LAST_INSERT_ID()                     • Use UUID_SHORT() i...
Advantages                     • 10 times faster                     • No additional tables and storage requiredMonday, Ap...
Disadvantages               • The master SERVER ID must be < 256Monday, April 11, 2011                                56
Another solution                     • We’ll see that in the second part of the                         talkMonday, April ...
DA     AG        EN               Slave laggingMonday, April 11, 2011                   58
Slave lagging                     • Common causes:                         •   Excess of traffic                         •...
Slave lagging                                                                        transaction                          ...
Slave lagging remedies                     • row-based replication (helps in some                         cases)          ...
statement-based replication  master> set global binlog_format=‘statement’;  master> insert into test.t1 select count(*) fr...
row-based replication  master> set global binlog_format=‘row’;  master> insert into test.t1 select count(*) from    inform...
Expensive query split  # instead of this:  insert into t1 select benchmark(10000000, sha(abc));  Query OK, 1 row affected ...
Expensive query split  # at 471  #110410 8:21:21 server id 1 end_log_pos 518  User_var  SET @`result`:=0/*!*/;  # at 518  ...
Slave query prefetch                     • What is it?                         •   a technique that fetches queries from t...
DA     AG        EN               GotchasMonday, April 11, 2011             67
Gotchas                     • When you thought that you had figured                         out everything ...Monday, Apri...
Gotchas: LOAD DATA                     • LOAD DATA INFILE is replicated as a                         temporary file       ...
Gotchas: default engine                     • If you define storage_engine in the                         master, it does ...
Gotchas: binlog format                     • In circular replication, and relay slaves                     • Changes to bi...
Gotchas: binlog format              server id: 101        insert into t1            format: statement    values (@@server_...
Gotchas: set global not really global                     • set global binlog_format=row                     • Works for a...
Gotchas: triggers and row replication                     • In statement based replication, triggers                      ...
DA     AG        EN           Replication power techniques                              and features                      ...
DA     AG        EN               row-based replicationMonday, April 11, 2011                           76
row-based replication                     • Available in 5.1 and later                     • can be changed using “binlog_...
row-based replication                     • DO                         •   when you have expensive queries in the         ...
DA     AG        EN               semi-synch replicationMonday, April 11, 2011                            79
semi-synchronous                                replication                     • Available in 5.5 and higher             ...
client                                                            master   transaction                                    ...
client                                     transaction                                                            master  ...
semi-synchronous                             replication in practice                     • installation:                  ...
semi-synch replication install  # in the master  plugin-load=rpl_semi_sync_master=semisync_master.so  rpl_semi_sync_master...
semi-synch replication check  # in the master  show variables like rpl_semi%;  +------------------------------------+-----...
semi-synch replication check  show status like "rpl_semi_%tx";  +-----------------------------+-------+  | variable_name  ...
semi-synch replication test  master> create table t1 ( i int);  Query OK, 0 rows affected (0.01 sec)  master> show status ...
disabling semi-synch  # for each slave  set global rpl_semi_sync_slave_enabled=0;  stop slave io_thread;  start slave io_t...
disabled semi-synch replication test  master> insert into t1 values (1);  Query OK, 1 row affected (10.00 sec)  master> sh...
disabled semi-synch replication test  master> insert into t1 values (2);  Query OK, 1 row affected (0.01 sec)  master> sho...
re-enabling semi-synch  # in one slave  set global rpl_semi_sync_slave_enabled=1;  stop slave io_thread;  start slave io_t...
reenabled semi-synch replication test  master> insert into t1 values (3);  Query OK, 1 row affected (0.01 sec)  master> sh...
DA     AG        EN               delayed replicationMonday, April 11, 2011                         93
delayed replication in practice  STOP SLAVE;  change master to master_delay=60;  START SLAVE;Monday, April 11, 2011       ...
delayed replication                     • DEMOMonday, April 11, 2011                         95
DA     AG        EN               Circular replicationMonday, April 11, 2011                          96
Circular                   A                   A           MySQL                           B                              ...
DA     AG        EN               bi-directional replicationMonday, April 11, 2011                                98
bi-directional replication               MySQL                            MySQL               DBMS                        ...
bi-directional replication                                          MySQL   MySQL                                         ...
DA     AG        EN               Leveraging replicationMonday, April 11, 2011                            101
Overview of MySQL partitionsMonday, April 11, 2011                       102
Partition pruning       1a - unpartitioned table - SINGLE RECORD                         I        D                       ...
Partition pruning       2a - table partitioned by colx - SINGLE REC                          1-99                         ...
Partition pruning       1a - unpartitioned table - RANGE                                      select *                    ...
Partition pruning       2a - table partitioned by colx - RANGE                          1-99                              ...
Storage comparison        engine                       storage                                       (GB)        innodb (w...
Benchmarking results      engine                         6 month range      InnoDB                              4 min 30s ...
Partitions limits      § Little concurrency      § Need of using the partitioning columnMonday, April 11, 2011          ...
The ARCHIVE storage engineMonday, April 11, 2011                            110
Size matters                         Innodb                                             ArchiveMonday, April 11, 2011     ...
REPLICATION PARTITIONING                               large                                data                         A...
Replication                                   =                   let the slave do the dirty workMonday, April 11, 2011   ...
masterbackup                                                               slaves                                         ...
master  makesummary tables                                                          slaves                                ...
master         Partitions         for heavy                                             innodb                            ...
Simulating                          master     multiple                                                  innodb           ...
DA     AG        EN                    Tools and tips                     • MySQL Sandbox                      • Replaying...
DA     AG        EN               MySQL SandboxMonday, April 11, 2011                   119
MySQL Sandbox                         http://mysqlsandbox.net             • Free software (Perl under GPL)             • O...
overview         MySQL                                         MySQL         server                                       ...
overview         MySQL                                               MySQL         server                                 ...
overview         MySQL                                       MySQL         server                                      ser...
The hard way (1)Monday, April 11, 2011                      124
the hard way (2)Monday, April 11, 2011                      125
The easy way     $ make_sandbox                      /path/to/mysql-5.1.54_linux.tar.gz     # it should work alwaysMonday,...
The easier way     $ make_sandbox 5.1.54     # Needs some preliminary workMonday, April 11, 2011                    127
The easiest way     $ sb 5.1.54     # Needs the same preliminary workMonday, April 11, 2011                     128
MySQL Sandbox                                                       VERSION                                              M...
MySQL Sandbox                                                      5.1.54                                             MySQ...
MySQL Sandbox                                                      5.5.9                                             MySQL...
Single Sandbox             MySQL           customized scripts             server     start     stop    restart    status  ...
Multiple Sandbox            MySQL             customized scripts            server      start_all      stop_all     restar...
Where do you get it        •from CPAN              sudo cpan MySQL::Sandbox        •from launchpad              http://lau...
The easy replication way     $ make_replication_sandbox                      /path/to/mysql-5.1.54_linux.tar.gz     # or, ...
default architecture                                           $HOME                              /sandboxes            op...
default architecture                                                   $HOME                                      /sandbox...
Monday, April 11, 2011   138
creating a single sanbox                  make_sandbox                      /path/to/mysql-X.X.XX-OS.tar.gzMonday, April 1...
using a single sanbox                  # after                  # make_sandbox                   #   /path/to/mysql-X.X.XX...
creating a single sanbox                         with a specific options file                  make_sandbox                 ...
easily create a sandbox after the first                       one                         The long way $ cd $HOME/opt/mysql...
easily create a sandbox after the first                       one                         The short way $ make_sandbox    p...
starting a single sanbox                  $ cd $SANDBOX_HOME/msb_X_X_XX                  $ ./startMonday, April 11, 2011  ...
starting a single sanbox                         with temporary options                  $ cd $SANDBOX_HOME/msb_X_X_XX    ...
creating a sandbox with custom port                 and directory                  $ make_sandbox 5.1.34                  ...
creating a sandbox with automatic                      port checking                  $ make_sandbox 5.1.34 --check_port  ...
create a replication sandbox $ make_replication_sandbox    path/to/mysql-5.1.34-osx10.5-x86.tar.gzMonday, April 11, 2011  ...
create a circular replication sandbox $ make_replication_sandbox    --circular=4    path/to/mysql-5.1.34-osx10.5-x86.tar.g...
changing port to an existing sandbox $ sbtool -o port      -s /path/to/source/sandbox      --new_port=XXXXMonday, April 11...
installing the innodb plugin $ sbtool -o plugin      --plugin=innodb      -s /path/to/source/sandboxMonday, April 11, 2011...
creating a replication sandbox with                     new base port $ make_replication_sandbox      --replication_direct...
DA     AG        EN               Replaying binary logsMonday, April 11, 2011                           153
DA     AG        EN               Filtering binary logsMonday, April 11, 2011                           154
DA     AG        EN               MaatkitMonday, April 11, 2011             155
maatkit     •       mk-heartbeat Monitor MySQL replication delay.     •       mk-purge-logs Purge binary logs on a master ...
Facebook Online Schema Change       DA      N   AGEMonday, April 11, 2011                      157
Facebook Online Schema Change     •       Based on one of Shlomi Noach OpenArk utilities     •       Developed in PHP by F...
DA     AG        EN               OpenArkMonday, April 11, 2011             159
openark                     •   oak-get-slave-lag: print slave replication lag and                         terminate with ...
DA     AG        EN               Check replication speedMonday, April 11, 2011                             161
check replication speed                     • Write a time to the master                     • Retrieve it from the slave ...
DA     AG        EN               Replication outside the box                     • Seamless failover                     ...
Advanced MySQL                         Replication for the                               massesMonday, April 11, 2011     ...
Once Upon A Time,                         In The Life Of A                            Database                          Co...
The story of a steel                              foundry                     • Used MySQL databases to store             ...
transaction                                                                      transaction                              ...
The story of a shoe                               maker                     • Had a successful business, spread to a      ...
MySQL                         MySQL                                                 DBMS     MySQL                        ...
The story of a widgets                             seller                     • Had a successful business, designed for on...
MySQL   MySQL                          DBMS    DBMS                         master   master                          MySQL...
All these stories tell us:  Nice dream, but MySQL                         can’t do itMonday, April 11, 2011               ...
Enter Tungsten ReplicatorMonday, April 11, 2011                               173
Tungsten Replicator 2.0                     • What is it?Monday, April 11, 2011                      174
http://code.google.com/p/tungsten-replicator                         Open Source                         100% GPL v2Monday...
What can it do?                • Easy failover                • Multiple masters                • Multiple sources to a si...
MySQL to foreign services                             MySQL setup                               to run as                 ...
From the beginning ...Monday, April 11, 2011                            178
Fail-over (1)Monday, April 11, 2011                   179
Fail-over (2)Monday, April 11, 2011                   180
Fail-over (3)Monday, April 11, 2011                   181
Fail-over (4)Monday, April 11, 2011                   182
Fail-over (5)Monday, April 11, 2011                   183
Fail-over (6)Monday, April 11, 2011                   184
Fail-over (7)Monday, April 11, 2011                   185
Fail-over (8)Monday, April 11, 2011                   186
Failover                     • DEMOMonday, April 11, 2011                   187
master/slave with an attitudeMonday, April 11, 2011        188
The steel foundry dream               or parallel replication                           From here ...Monday, April 11, 201...
The steel foundry dream               or parallel replication                            To here.Monday, April 11, 2011   ...
Parallel replication facts                     • Sharded by database                     • Good choice for slave lag probl...
Testing parallel                           replicationMonday, April 11, 2011                      192
db0                                                 preparation (1)                          db1                          ...
preparation (2)                  db0                    db1                         db2                          db3      ...
before the test (1)                  db0                    db1                         db2                          db3  ...
before the test (2)              SQL thread          RELAY logs                              IO thread                    ...
starting the test               SQL thread          RELAY logs                              IO thread                     ...
MySQL native                          replication              slave catch up in 00:02:30Monday, April 11, 2011           ...
Tungsten parallel                            replication              slave catch up in 00:01:20Monday, April 11, 2011    ...
The widget seller                 dream, or multi masters                     • Tungsten Replicator recipe: use more      ...
Bi-directional replicationMonday, April 11, 2011                                201
Bi-directional replication with slavesMonday, April 11, 2011                          202
True multiple master                          We’ll see that in a moment.                                    But firstMonda...
The shoe maker dream,                   or multiple sources                     • Tungsten Replicator recipe is still vali...
Multiple source replicationMonday, April 11, 2011                                 205
Multiple masters replication: 3 nodesMonday, April 11, 2011                       206
Multiple masters replication: 4 nodesMonday, April 11, 2011                       207
Updating 4 masters : 1 flowMonday, April 11, 2011                                208
Updating 4 masters : 2 flowsMonday, April 11, 2011                                 209
Updating 4 masters : 3 flowsMonday, April 11, 2011                                 210
Updating 4 masters : 4 flowsMonday, April 11, 2011                                 211
Tungsten in practice                     • installationMonday, April 11, 2011                          212
Installation:                              the long way                     • Get the binaries                     • Expan...
Installation:                                the quick way                     • Get the tarball                     • Get...
Tools                     • replicator                     • trepctl                     • thlMonday, April 11, 2011      ...
replicator                     • It’s the service provider                     • You launch it once when you start        ...
trepctl                     • Tungsten Replicator ConTroLler                     • It’s the driving seat for your replicat...
thl                     • Transaction History List                     • Gives you access to the Tungsten relay logsMonday...
DA     AG        EN               More info on replicationMonday, April 11, 2011                              219
BooksMonday, April 11, 2011           220
High Performance MySQLMonday, April 11, 2011                            221
MySQL High AvailabilityMonday, April 11, 2011                             222
Web OperationsMonday, April 11, 2011                    223
Cloud Application ArchitecturesMonday, April 11, 2011                       224
Talks                     •   Monday, 1:30pm “Learn how to cure replication                         deprivation with Tungs...
ADVERTISINGMonday, April 11, 2011                 226
WE ARE HIRING!          QA and support            engineers            http://www.continuent.com/about/careersMonday, Apri...
http://opendbcamp.org                                  Open Database Camp                                        Sardinia ...
Contact Information       Worldwide       560 S. Winchester Blvd., Suite 500       San Jose, CA 95128       Tel (866) 998-...
Upcoming SlideShare
Loading in...5
×

Advanced mysql replication techniques

14,538

Published on

Advanced MySQL Replication techniques

Published in: Technology
4 Comments
24 Likes
Statistics
Notes
No Downloads
Views
Total Views
14,538
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
690
Comments
4
Likes
24
Embeds 0
No embeds

No notes for slide

Advanced mysql replication techniques

  1. 1. Advanced MySQL Replication Techniques Giuseppe Maxia, QA Director, Continuent, Inc Facts. And Demos. Possibly funMonday, April 11, 2011 1
  2. 2. about me - Giuseppe Maxia • a.k.a. The Data Charmer • QA Director at Continuent, Inc • Long time hacking with MySQL features • Formerly, community manager,db consultant, designer, coder. • A passion for QA and open source • Blogger • http://datacharmer.blogspot.comMonday, April 11, 2011 2
  3. 3. Agenda • Replication basics • Replication blues • Master switch and failover • Slave lagging • Gotchas • Replication power techniques and features • row-based, semi-synch, delayed • Circular, bi-directional • leveraging replication • Tools and tips • Replication outside the box •Monday, April 11, 2011 seamless failover, parallel, multi-master 3
  4. 4. DA AG EN Replication basics • Principles • How to set up replication • Monitoring replicationMonday, April 11, 2011 4
  5. 5. DA AG EN PrinciplesMonday, April 11, 2011 5
  6. 6. database server a simple web application scheme r/w requests web server clients 6Monday, April 11, 2011 6
  7. 7. database server scaling web requests r/w requests web servers load balancer clients 7Monday, April 11, 2011 7
  8. 8. read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clients 8Monday, April 11, 2011 8
  9. 9. client master transaction binary log reads slave IO thread relay log replication concepts SQL thread reads 9Monday, April 11, 2011 9
  10. 10. DA AG EN replication setupMonday, April 11, 2011 10
  11. 11. 1 SHUT DOWN THE DATABASE SERVER Master 11Monday, April 11, 2011 11
  12. 12. 2 MAKE A BACKUP COPY Master 12Monday, April 11, 2011 12
  13. 13. 3 ENABLE THE MASTER Master Configuration file [mysqld] log-bin=mysql-bin server-id=1 13Monday, April 11, 2011 13
  14. 14. 4 RESTART THE MASTER Master 14Monday, April 11, 2011 14
  15. 15. 5 CREATE REPLICATION USER Master SQL command GRANT REPLICATION SLAVE ON *.* to slave_user@10.10.100.% IDENTIFIED BY slave_pass; 15Monday, April 11, 2011 15
  16. 16. 6 INSTALL MySQL on the slave Slave 1Make sure that:• Youre using the same version of MySQL• You have the same directory structure• The server is not started yet 16Monday, April 11, 2011 16
  17. 17. 7 COPY THE MASTER DATA to the slave Slave 1 17Monday, April 11, 2011 17
  18. 18. 8 ENABLE THE SLAVE Slave 1 Configuration file [mysqld] server-id=2 relay-log=mysql-relay read-only # optional: log-bin=mysql-bin 18Monday, April 11, 2011 18
  19. 19. 9 START THE SLAVE SERVER Slave 1 19Monday, April 11, 2011 19
  20. 20. 10 INITIALIZE THE SLAVE Slave 1 SQL command CHANGE MASTER TO MASTER_HOST=master_IP, MASTER_PORT=3306, MASTER_USER=slave_user, MASTER_PASSWORD=slave_pwd; 20Monday, April 11, 2011 20
  21. 21. 11 START THE SLAVE SERVICE Slave 1 SQL command START SLAVE; 21Monday, April 11, 2011 21
  22. 22. 12 CHECK THE SLAVE Slave 1 SQL command SHOW SLAVE STATUS G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... 22Monday, April 11, 2011 22
  23. 23. Troubleshooting • SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No • Make sure that the slave host can connect to the master • Make sure that master and slave have different Server-id • Check the error log of both master and slaveMonday, April 11, 2011 23
  24. 24. Testing the slave • Create a table in the master. • Make sure that the slave has replicated the table. • Insert data in the master • read that data in the slaveMonday, April 11, 2011 24
  25. 25. DA AG EN MonitoringMonday, April 11, 2011 25
  26. 26. Sample get slave status monitoring slave Running? No Yes master Get master binlog and position Same or later No Yes binlog/position? check table contents alert 26Monday, April 11, 2011 26
  27. 27. monitoring replication master> SHOW MASTER STATUS slave> SHOW SLAVE STATUS FULL SCRIPTS: http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans- mysql.html http://forge.mysql.com/tools/tool.php?id=6Monday, April 11, 2011 27
  28. 28. show master status File: mysql-bin.000002 Position: 78045744 Binlog_Do_DB: Binlog_Ignore_DB:Monday, April 11, 2011 28
  29. 29. show slave status Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: rsandbox Master_Port: 27371 Connect_Retry: 60 Master_Log_File: mysql-bin.000002 Read_Master_Log_Pos: 78045744 Relay_Log_File: mysql_sandbox27372-relay-bin.000055 Relay_Log_Pos: 78045889 Relay_Master_Log_File: mysql-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: YesMonday, April 11, 2011 29
  30. 30. show slave status ... Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: ...Monday, April 11, 2011 30
  31. 31. show slave status ... Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 78045744 Relay_Log_Space: 78046100 ... Seconds_Behind_Master: 0 ... Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:Monday, April 11, 2011 31
  32. 32. DA AG EN Replication blues • Master switch and failover • Slave lagging • GotchasMonday, April 11, 2011 32
  33. 33. DA AG EN Master switch and failoverMonday, April 11, 2011 33
  34. 34. Replacing the Master master crashe s Let all slaves catch up with execution STOP replication in all slaves FIND the most make it the up to date slave master FIND which run missing connect all transactions are transactions slaves to the missing from to other new master other slaves slaves Stop 34Monday, April 11, 2011 34
  35. 35. Planned master switch • Stop accepting writes • Wait until all slaves have caught up • Stop replication in all slaves • Promote a slave to master • Point all slaves to the new masterMonday, April 11, 2011 35
  36. 36. Changing a failed master • Pre-requisite: • log_bin and log_slave_updates must be enabled in all the slaves • If not, there is more manual laborMonday, April 11, 2011 36
  37. 37. Changing a failed master (1) • Wait until all slaves have caught up • Identify the most advanced slave • Make that slave the new master • ... so far, so goodMonday, April 11, 2011 37
  38. 38. Changing a failed master (2) • For each remaining slave: • Find the missing statements • Find the LAST statement replicated by the slave • Find the same statement in the new master binlog (*) • get the position of the NEXT statement (*) if log_slave_updates was not enabled, you need to convert the relay log statements to SQL and do the next step manuallyMonday, April 11, 2011 38
  39. 39. Changing a failed master (3) • For each remaining slave: • Apply the missing statements • CHANGE MASTER TO master_host=”new_master_hostname”, master_port=new_master_port, master_log_file=”mysql-bin.xxxxxx”, master_log_pos=YYYYMonday, April 11, 2011 39
  40. 40. Reasons for complexity • No global transaction IDMonday, April 11, 2011 40
  41. 41. What is a global transaction ID • A unique identifier of a transaction • Unique for the whole cluster, not for each node • Generated by the ultimate source (the master) • Does not change when the transaction goes through an intermediate masterMonday, April 11, 2011 41
  42. 42. Why should you care • Failure recovery • MySQL DOES NOT have itMonday, April 11, 2011 42
  43. 43. Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Monday, April 11, 2011 43
  44. 44. The master crashes Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Monday, April 11, 2011 44
  45. 45. Where do the slaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary log Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 45
  46. 46. Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 46
  47. 47. Need to synch the missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 47
  48. 48. Transaction position in the new master binlog New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ? ======== ? ======== ========Monday, April 11, 2011 48
  49. 49. An existing solution • Google has made a patch to implement global transaction IDs • HOWEVER • It only works with 5.0 • It doesnt work with row based replicationMonday, April 11, 2011 49
  50. 50. Hack ahead!Monday, April 11, 2011 50
  51. 51. CREATE TABLE Global_Trans_ID ( number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY ) ENGINE = MyISAM; CREATE TABLE Last_Exec_Trans ( server_id INT UNSIGNED, trans_id INT UNSIGNED ) ENGINE = InnoDB;Monday, April 11, 2011 51
  52. 52. CREATE PROCEDURE commit_trans () BEGIN DECLARE trans_id, server_id INT UNSIGNED; SET SQL_LOG_BIN = 0; INSERT INTO global_trans_id() values (); SELECT LAST_INSERT_ID() INTO trans_id, @@server_id INTO server_id; DELETE FROM global_trans_id where number < trans_Id; SET SQL_LOG_BIN = 1; INSERT INTO last_exec_trans(server_id, trans_id) VALUES (server_id, trans_id); COMMIT; ENDMonday, April 11, 2011 52
  53. 53. Hacking the hackMonday, April 11, 2011 53
  54. 54. Simplifying the scheme • No MyISAM table and LAST_INSERT_ID() • Use UUID_SHORT() instead • Replace the InnoDB table with a BlackHole oneMonday, April 11, 2011 54
  55. 55. Advantages • 10 times faster • No additional tables and storage requiredMonday, April 11, 2011 55
  56. 56. Disadvantages • The master SERVER ID must be < 256Monday, April 11, 2011 56
  57. 57. Another solution • We’ll see that in the second part of the talkMonday, April 11, 2011 57
  58. 58. DA AG EN Slave laggingMonday, April 11, 2011 58
  59. 59. Slave lagging • Common causes: • Excess of traffic • Slave restart • Long DDL operationsMonday, April 11, 2011 59
  60. 60. Slave lagging transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra csio s ac ctio ac ion ion n an n s an nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio t tra tra REPLICATION MySQL replication is SINGLE THREAD transaction transaction MySQL DBMSMonday, April 11, 2011 60
  61. 61. Slave lagging remedies • row-based replication (helps in some cases) • Expensive query split • Double apply (dangerous) • Slave query prefetch (requires external app) • Parallel replication (requires external app)Monday, April 11, 2011 61
  62. 62. statement-based replication master> set global binlog_format=‘statement’; master> insert into test.t1 select count(*) from information_schema.columns; master> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+ slave> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+Monday, April 11, 2011 62
  63. 63. row-based replication master> set global binlog_format=‘row’; master> insert into test.t1 select count(*) from information_schema.columns; master> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+ slave> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 17 | +---------------+-------+Monday, April 11, 2011 63
  64. 64. Expensive query split # instead of this: insert into t1 select benchmark(10000000, sha(abc)); Query OK, 1 row affected (4.19 sec) Records: 1 Duplicates: 0 Warnings: 0 # you may do this set @result=(select benchmark(10000000, sha(abc))); Query OK, 0 rows affected (4.19 sec) insert into t1 values (@result); Query OK, 1 row affected (0.00 sec)Monday, April 11, 2011 64
  65. 65. Expensive query split # at 471 #110410 8:21:21 server id 1 end_log_pos 518 User_var SET @`result`:=0/*!*/; # at 518 #110410 8:21:21 server id 1 end_log_pos 612 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP=1302448881/*!*/; insert into t1 values (@result)Monday, April 11, 2011 65
  66. 66. Slave query prefetch • What is it? • a technique that fetches queries from the relay logs, and runs them as SELECT statements, to warm up the indexes • How do you do it? • mk-slave-prefetch (http://www.maatkit.org) • Slave readahead http://sourceforge.net/ projects/slavereadahead/Monday, April 11, 2011 66
  67. 67. DA AG EN GotchasMonday, April 11, 2011 67
  68. 68. Gotchas • When you thought that you had figured out everything ...Monday, April 11, 2011 68
  69. 69. Gotchas: LOAD DATA • LOAD DATA INFILE is replicated as a temporary file • The slave needs three times the size of the data: • the relay log • the temporary file • the data itselfMonday, April 11, 2011 69
  70. 70. Gotchas: default engine • If you define storage_engine in the master, it does not get replicatedMonday, April 11, 2011 70
  71. 71. Gotchas: binlog format • In circular replication, and relay slaves • Changes to binlog_format are not propagatedMonday, April 11, 2011 71
  72. 72. Gotchas: binlog format server id: 101 insert into t1 format: statement values (@@server_id) 101 server id: 102 format: row 102 server id: 103 102 format: rowMonday, April 11, 2011 72
  73. 73. Gotchas: set global not really global • set global binlog_format=row • Works for all new connections after the change • It does not work for the event scheduler!Monday, April 11, 2011 73
  74. 74. Gotchas: triggers and row replication • In statement based replication, triggers are fired both on master and slave • in row-based replication, triggers are fired on master only, and the resulting rows are moved to the slaveMonday, April 11, 2011 74
  75. 75. DA AG EN Replication power techniques and features • row-based • semi-synch • delayed • Circular • bi-directional • Leveraging replicationMonday, April 11, 2011 75
  76. 76. DA AG EN row-based replicationMonday, April 11, 2011 76
  77. 77. row-based replication • Available in 5.1 and later • can be changed using “binlog_format” • row • mixed • statement (default)Monday, April 11, 2011 77
  78. 78. row-based replication • DO • when you have expensive queries in the master • when you are using non-deterministic functions • DON’T • When you are updating a lot of rowsMonday, April 11, 2011 78
  79. 79. DA AG EN semi-synch replicationMonday, April 11, 2011 79
  80. 80. semi-synchronous replication • Available in 5.5 and higher • Makes sure that at least one slave has copied the data. • Increases reliabilityMonday, April 11, 2011 80
  81. 81. client master transaction 1 with regular commit replication execute 2 slave returns 4 to client binary log 3 5 replication 81Monday, April 11, 2011 81
  82. 82. client transaction master 1 with semi- commit synchronous replication execute 2 slave 7 returns to client binary log 3 sends 4 transaction relay log 5 to slave gets 6 acknowledgement 82Monday, April 11, 2011 82
  83. 83. semi-synchronous replication in practice • installation: • it’s a plugin. • Actually, two pluginsMonday, April 11, 2011 83
  84. 84. semi-synch replication install # in the master plugin-load=rpl_semi_sync_master=semisync_master.so rpl_semi_sync_master_enabled=1 # in each slave plugin-load=rpl_semi_sync_slave=semisync_slave.so rpl_semi_sync_slave_enabled=1 # restar all serversMonday, April 11, 2011 84
  85. 85. semi-synch replication check # in the master show variables like rpl_semi%; +------------------------------------+-------+ | Variable_name | Value | +------------------------------------+-------+ | rpl_semi_sync_master_enabled | ON | | rpl_semi_sync_master_timeout | 10000 | | rpl_semi_sync_master_trace_level | 32 | | rpl_semi_sync_master_wait_no_slave | ON | +------------------------------------+-------+Monday, April 11, 2011 85
  86. 86. semi-synch replication check show status like "rpl_semi_%tx"; +-----------------------------+-------+ | variable_name | value | +-----------------------------+-------+ | RPL_SEMI_SYNC_MASTER_NO_TX | 0 | | RPL_SEMI_SYNC_MASTER_YES_TX | 0 | +-----------------------------+-------+Monday, April 11, 2011 86
  87. 87. semi-synch replication test master> create table t1 ( i int); Query OK, 0 rows affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 0 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+Monday, April 11, 2011 87
  88. 88. disabling semi-synch # for each slave set global rpl_semi_sync_slave_enabled=0; stop slave io_thread; start slave io_thread;Monday, April 11, 2011 88
  89. 89. disabled semi-synch replication test master> insert into t1 values (1); Query OK, 1 row affected (10.00 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 1 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 89
  90. 90. disabled semi-synch replication test master> insert into t1 values (2); Query OK, 1 row affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 2 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 90
  91. 91. re-enabling semi-synch # in one slave set global rpl_semi_sync_slave_enabled=1; stop slave io_thread; start slave io_thread;Monday, April 11, 2011 91
  92. 92. reenabled semi-synch replication test master> insert into t1 values (3); Query OK, 1 row affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 2 | | Rpl_semi_sync_master_yes_tx | 2 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 92
  93. 93. DA AG EN delayed replicationMonday, April 11, 2011 93
  94. 94. delayed replication in practice STOP SLAVE; change master to master_delay=60; START SLAVE;Monday, April 11, 2011 94
  95. 95. delayed replication • DEMOMonday, April 11, 2011 95
  96. 96. DA AG EN Circular replicationMonday, April 11, 2011 96
  97. 97. Circular A A MySQL B DBMS MySQL DBMS B master master MySQL DD DBMS MySQL DBMS C CMonday, April 11, 2011 master 97
  98. 98. DA AG EN bi-directional replicationMonday, April 11, 2011 98
  99. 99. bi-directional replication MySQL MySQL DBMS DBMSMonday, April 11, 2011 99
  100. 100. bi-directional replication MySQL MySQL DBMS DBMS Los Angeles New York MySQL MySQL MySQL DBMS DBMS DBMS MySQL DBMSMonday, April 11, 2011 100
  101. 101. DA AG EN Leveraging replicationMonday, April 11, 2011 101
  102. 102. Overview of MySQL partitionsMonday, April 11, 2011 102
  103. 103. Partition pruning 1a - unpartitioned table - SINGLE RECORD I D select * N from table_name A D where colx = T E 120 A XMonday, April 11, 2011 103
  104. 104. Partition pruning 2a - table partitioned by colx - SINGLE REC 1-99 100-199 select * from table_name 200-299 where colx = 120 300-399 400-499 500-599Monday, April 11, 2011 104
  105. 105. Partition pruning 1a - unpartitioned table - RANGE select * I from D N table_name A where colx D between 120 T E and 230 A XMonday, April 11, 2011 105
  106. 106. Partition pruning 2a - table partitioned by colx - RANGE 1-99 select * 100-199 from table_name 200-299 where colx between 120 and 230 300-399 400-499 500-599Monday, April 11, 2011 106
  107. 107. Storage comparison engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13Monday, April 11, 2011 107
  108. 108. Benchmarking results engine 6 month range InnoDB 4 min 30s MyISAM 25.03s Archive 22 min 25s InnoDB partitioned by month 13.19 MyISAM partitioned by year 6.31 MyISAM partitioned by month 4.45 Archive partitioned by year 16.67 Archive partitioned by month 8.97Monday, April 11, 2011 108
  109. 109. Partitions limits § Little concurrency § Need of using the partitioning columnMonday, April 11, 2011 109
  110. 110. The ARCHIVE storage engineMonday, April 11, 2011 110
  111. 111. Size matters Innodb ArchiveMonday, April 11, 2011 111
  112. 112. REPLICATION PARTITIONING large data ARCHIVEMonday, April 11, 2011 112
  113. 113. Replication = let the slave do the dirty workMonday, April 11, 2011 113
  114. 114. masterbackup slaves STOP SLAVE remove slave perform from load backup balancer attach slave START Let slave to load SLAVE catch up balancerMonday, April 11, 2011 114
  115. 115. master makesummary tables slaves STOP SLAVE calculate remove slave summary from load tables balancer attach slave Let slave START SLAVE to load catch up balancerMonday, April 11, 2011 115
  116. 116. master Partitions for heavy innodb non partitioned statistics slave slave innodb innodb non partitioned partitioned by range slave MyISAM partitioned by rangeMonday, April 11, 2011 116
  117. 117. Simulating master multiple innodb non partitioned dimensions slave slave innodb non partitioned ARCHIVE partitioned by range (date) slave slave ARCHIVE partitioned by range ARCHIVE (location) partitioned by range (product)Monday, April 11, 2011 117
  118. 118. DA AG EN Tools and tips • MySQL Sandbox • Replaying binary logs • Filtering binary logs • Baron Schwartz’s Maatkit • Shlomi Noach’s openark kit • Facebook’s Online Schema Change • Measuring replication speed • Detecting if replication is onMonday, April 11, 2011 118
  119. 119. DA AG EN MySQL SandboxMonday, April 11, 2011 119
  120. 120. MySQL Sandbox http://mysqlsandbox.net • Free software (Perl under GPL) • One (unix) host • Many database servers • Single or multiple sandboxes • Customized scripts to use the servers • Standard or circular replication • Installs IN SECONDSMonday, April 11, 2011 120
  121. 121. overview MySQL MySQL server server Data DB1 Data DB1 DB2 DB3 DATA DIRECTORY DB2 DB3 PORT SOCKETMonday, April 11, 2011 121
  122. 122. overview MySQL MySQL server server Data DB1 SAME Data DB1 DB2 DB3 DATA DB2 DB3 DIRECTORY? /var/lib/mysql /var/lib/mysql DATA CORRUPTIONMonday, April 11, 2011 122
  123. 123. overview MySQL MySQL server server SAME PORT or SOCKET? 3306 3306 /tmp/mysql.sock /tmp/mysql.sock DOES NOT STARTMonday, April 11, 2011 123
  124. 124. The hard way (1)Monday, April 11, 2011 124
  125. 125. the hard way (2)Monday, April 11, 2011 125
  126. 126. The easy way $ make_sandbox /path/to/mysql-5.1.54_linux.tar.gz # it should work alwaysMonday, April 11, 2011 126
  127. 127. The easier way $ make_sandbox 5.1.54 # Needs some preliminary workMonday, April 11, 2011 127
  128. 128. The easiest way $ sb 5.1.54 # Needs the same preliminary workMonday, April 11, 2011 128
  129. 129. MySQL Sandbox VERSION MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_VERSION/data VERSION /tmp/mysql_VERSION.sockMonday, April 11, 2011 129
  130. 130. MySQL Sandbox 5.1.54 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_1_54/data 5154 /tmp/mysql_5154.sockMonday, April 11, 2011 130
  131. 131. MySQL Sandbox 5.5.9 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_5_09/data 5509 /tmp/mysql_5509.sockMonday, April 11, 2011 131
  132. 132. Single Sandbox MySQL customized scripts server start stop restart status clear send_kill useMonday, April 11, 2011 132
  133. 133. Multiple Sandbox MySQL customized scripts server start_all stop_all restart_all m n1 status_all s1 n2 clear_all s2 n3 send_kill_a ll use_allMonday, April 11, 2011 133
  134. 134. Where do you get it •from CPAN sudo cpan MySQL::Sandbox •from launchpad http://launchpad.net/mysql-sandboxMonday, April 11, 2011 134
  135. 135. The easy replication way $ make_replication_sandbox /path/to/mysql-5.1.54_linux.tar.gz # or, after some preparation $ make_replication_sandbox 5.1.54Monday, April 11, 2011 135
  136. 136. default architecture $HOME /sandboxes opt expanded tarballs $SANDBOX_HOME mysql $SANDBOX_BINARY installed sandboxesMonday, April 11, 2011 136
  137. 137. default architecture $HOME /sandboxes opt msb_5_0_91 mysql msb_5_1_48 5.0.91 rsandbox_5_1_48 5.1.45 master 5.1.48 node1 5.5.4 node2Monday, April 11, 2011 137
  138. 138. Monday, April 11, 2011 138
  139. 139. creating a single sanbox make_sandbox /path/to/mysql-X.X.XX-OS.tar.gzMonday, April 11, 2011 139
  140. 140. using a single sanbox # after # make_sandbox # /path/to/mysql-X.X.XX-OS.tar.gz $ cd $SANDBOX_HOME/msb_X_X_XX $ ./useMonday, April 11, 2011 140
  141. 141. creating a single sanbox with a specific options file make_sandbox /path/to/mysql-X.X.XX-OS.tar.gz --my_file=/path/to/my.cnfMonday, April 11, 2011 141
  142. 142. easily create a sandbox after the first one The long way $ cd $HOME/opt/mysql # $SANDBOX_BINARY $ gunzip -c /path/to/mysql-5.1.34-osx10.5-x86.tar.gz | tar -xf - $ mv mysql-5.1.34-osx10.5-x86 5.1.34 $ make sandbox 5.1.34Monday, April 11, 2011 142
  143. 143. easily create a sandbox after the first one The short way $ make_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gz --export_binariesMonday, April 11, 2011 143
  144. 144. starting a single sanbox $ cd $SANDBOX_HOME/msb_X_X_XX $ ./startMonday, April 11, 2011 144
  145. 145. starting a single sanbox with temporary options $ cd $SANDBOX_HOME/msb_X_X_XX $ ./start --option=value $ ./restart --option=value $ ./start --key-buffer=20000000Monday, April 11, 2011 145
  146. 146. creating a sandbox with custom port and directory $ make_sandbox 5.1.34 --sandbox_port=7800 --sandbox_directory=mickeymouseMonday, April 11, 2011 146
  147. 147. creating a sandbox with automatic port checking $ make_sandbox 5.1.34 --check_port # if 5.1.34 is free # port=5134 # directory=msb_5_1_34 # else # port=5135 (or the first free) # directory=msb_5_1_34_aMonday, April 11, 2011 147
  148. 148. create a replication sandbox $ make_replication_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gzMonday, April 11, 2011 148
  149. 149. create a circular replication sandbox $ make_replication_sandbox --circular=4 path/to/mysql-5.1.34-osx10.5-x86.tar.gzMonday, April 11, 2011 149
  150. 150. changing port to an existing sandbox $ sbtool -o port -s /path/to/source/sandbox --new_port=XXXXMonday, April 11, 2011 150
  151. 151. installing the innodb plugin $ sbtool -o plugin --plugin=innodb -s /path/to/source/sandboxMonday, April 11, 2011 151
  152. 152. creating a replication sandbox with new base port $ make_replication_sandbox --replication_directory=newwdir --check_base_port 5.0.79 # Creates a replication directory under # $SANDBOX_HOME/newdir # The previous one is preserved. # No conflicts happenMonday, April 11, 2011 152
  153. 153. DA AG EN Replaying binary logsMonday, April 11, 2011 153
  154. 154. DA AG EN Filtering binary logsMonday, April 11, 2011 154
  155. 155. DA AG EN MaatkitMonday, April 11, 2011 155
  156. 156. maatkit • mk-heartbeat Monitor MySQL replication delay. • mk-purge-logs Purge binary logs on a master based on purge rules. • mk-slave-delay Make a MySQL slave server lag behind its master. • mk-slave-find Find and print replication hierarchy tree of MySQL slaves. • mk-slave-move Move a MySQL slave around in the replication hierarchy. • mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm caches. • mk-slave-restart Watch and restart MySQL replication after errors. • mk-table-checksum Perform an online replication consistency check, or checksum MySQL tables efficiently on one or many servers. • mk-table-sync Synchronize MySQL table data efficiently. • http://www.maatkit.org/Monday, April 11, 2011 156
  157. 157. Facebook Online Schema Change DA N AGEMonday, April 11, 2011 157
  158. 158. Facebook Online Schema Change • Based on one of Shlomi Noach OpenArk utilities • Developed in PHP by Facebook • http://www.facebook.com/note.php?note_id=430801045932Monday, April 11, 2011 158
  159. 159. DA AG EN OpenArkMonday, April 11, 2011 159
  160. 160. openark • oak-get-slave-lag: print slave replication lag and terminate with respective exit code. • oak-online-alter-table: perform a non-blocking ALTER TABLE operation. • oak-purge-master-logs: purge master logs, depending on the state of replicating slaves. • oak-show-replication-status: show how far behind are replicating slaves on a given master. • http://code.openark.org/forge/openark-kitMonday, April 11, 2011 160
  161. 161. DA AG EN Check replication speedMonday, April 11, 2011 161
  162. 162. check replication speed • Write a time to the master • Retrieve it from the slave • Compare times • http://forge.mysql.com/tools/tool.php? id=5Monday, April 11, 2011 162
  163. 163. DA AG EN Replication outside the box • Seamless failover • Parallel replication • Multiple source replication • Multiple master replicationMonday, April 11, 2011 163
  164. 164. Advanced MySQL Replication for the massesMonday, April 11, 2011 164
  165. 165. Once Upon A Time, In The Life Of A Database Consultant ...Monday, April 11, 2011 165
  166. 166. The story of a steel foundry • Used MySQL databases to store production monitoring data • Inserted a zillion records per second. • Slaves often lagged behindMonday, April 11, 2011 166
  167. 167. transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio ct tra tra REPLICATION transaction transaction MySQL DBMSMonday, April 11, 2011 167
  168. 168. The story of a shoe maker • Had a successful business, spread to a dozen stores. • Needed to aggregate the data from the stores in his headquarters database.Monday, April 11, 2011 168
  169. 169. MySQL MySQL DBMS MySQL DBMS DBMS store store store MySQL DBMS headquartersMonday, April 11, 2011 169
  170. 170. The story of a widgets seller • Had a successful business, designed for one server. • Products were created in several sites. • Needed to allow insertions from more than one master at the timeMonday, April 11, 2011 170
  171. 171. MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master masterMonday, April 11, 2011 171
  172. 172. All these stories tell us: Nice dream, but MySQL can’t do itMonday, April 11, 2011 172
  173. 173. Enter Tungsten ReplicatorMonday, April 11, 2011 173
  174. 174. Tungsten Replicator 2.0 • What is it?Monday, April 11, 2011 174
  175. 175. http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2Monday, April 11, 2011 175
  176. 176. What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Parallel replication • Replicate to Oracle and PostgreSQL databaseMonday, April 11, 2011 176
  177. 177. MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle Bin Logs (Binlogs enabled) Slave DB © Continuent 2010Monday, April 11, 2011 177
  178. 178. From the beginning ...Monday, April 11, 2011 178
  179. 179. Fail-over (1)Monday, April 11, 2011 179
  180. 180. Fail-over (2)Monday, April 11, 2011 180
  181. 181. Fail-over (3)Monday, April 11, 2011 181
  182. 182. Fail-over (4)Monday, April 11, 2011 182
  183. 183. Fail-over (5)Monday, April 11, 2011 183
  184. 184. Fail-over (6)Monday, April 11, 2011 184
  185. 185. Fail-over (7)Monday, April 11, 2011 185
  186. 186. Fail-over (8)Monday, April 11, 2011 186
  187. 187. Failover • DEMOMonday, April 11, 2011 187
  188. 188. master/slave with an attitudeMonday, April 11, 2011 188
  189. 189. The steel foundry dream or parallel replication From here ...Monday, April 11, 2011 189
  190. 190. The steel foundry dream or parallel replication To here.Monday, April 11, 2011 190
  191. 191. Parallel replication facts • Sharded by database • Good choice for slave lag problems • Bad choice for single database projectsMonday, April 11, 2011 191
  192. 192. Testing parallel replicationMonday, April 11, 2011 192
  193. 193. db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbenchMonday, April 11, 2011 193
  194. 194. preparation (2) db0 db1 db2 db3 db4 db5 db6 db7 db8 db9Monday, April 11, 2011 194
  195. 195. before the test (1) db0 db1 db2 db3 db4 db5 db6 db7 db8 db9Monday, April 11, 2011 195
  196. 196. before the test (2) SQL thread RELAY logs IO thread binary logs MySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alphaMonday, April 11, 2011 196
  197. 197. starting the test SQL thread RELAY logs IO thread binary logs MySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alphaMonday, April 11, 2011 197
  198. 198. MySQL native replication slave catch up in 00:02:30Monday, April 11, 2011 198
  199. 199. Tungsten parallel replication slave catch up in 00:01:20Monday, April 11, 2011 199
  200. 200. The widget seller dream, or multi masters • Tungsten Replicator recipe: use more servicesMonday, April 11, 2011 200
  201. 201. Bi-directional replicationMonday, April 11, 2011 201
  202. 202. Bi-directional replication with slavesMonday, April 11, 2011 202
  203. 203. True multiple master We’ll see that in a moment. But firstMonday, April 11, 2011 203
  204. 204. The shoe maker dream, or multiple sources • Tungsten Replicator recipe is still valid: use more servicesMonday, April 11, 2011 204
  205. 205. Multiple source replicationMonday, April 11, 2011 205
  206. 206. Multiple masters replication: 3 nodesMonday, April 11, 2011 206
  207. 207. Multiple masters replication: 4 nodesMonday, April 11, 2011 207
  208. 208. Updating 4 masters : 1 flowMonday, April 11, 2011 208
  209. 209. Updating 4 masters : 2 flowsMonday, April 11, 2011 209
  210. 210. Updating 4 masters : 3 flowsMonday, April 11, 2011 210
  211. 211. Updating 4 masters : 4 flowsMonday, April 11, 2011 211
  212. 212. Tungsten in practice • installationMonday, April 11, 2011 212
  213. 213. Installation: the long way • Get the binaries • Expand the tarball • Check the requirements in the manual • Run ./configure • Run ./configure-serviceMonday, April 11, 2011 213
  214. 214. Installation: the quick way • Get the tarball • Get the tungsten deployer from Google Code • Sit back and enjoy!Monday, April 11, 2011 214
  215. 215. Tools • replicator • trepctl • thlMonday, April 11, 2011 215
  216. 216. replicator • It’s the service provider • You launch it once when you start • You may restart it when you change configMonday, April 11, 2011 216
  217. 217. trepctl • Tungsten Replicator ConTroLler • It’s the driving seat for your replication • You can start, update, and stop services • You can get specific infoMonday, April 11, 2011 217
  218. 218. thl • Transaction History List • Gives you access to the Tungsten relay logsMonday, April 11, 2011 218
  219. 219. DA AG EN More info on replicationMonday, April 11, 2011 219
  220. 220. BooksMonday, April 11, 2011 220
  221. 221. High Performance MySQLMonday, April 11, 2011 221
  222. 222. MySQL High AvailabilityMonday, April 11, 2011 222
  223. 223. Web OperationsMonday, April 11, 2011 223
  224. 224. Cloud Application ArchitecturesMonday, April 11, 2011 224
  225. 225. Talks • Monday, 1:30pm “Learn how to cure replication deprivation with Tungsten” • Tuesday 10:50am “Replication Update” • Tuesday 2:00pm “Diagnosing replication failures” • Tuesday 3:05 “Advanced replication monitoring” • Thursday 10:50am “Replication for Availability & Durability with MySQL and Amazon RDS” • Thursday 11:55am “Build your own PaaS for MySQL with Tungsten Enterprise” • Thursday 2:00pm “Error detection and correction with MySQL Replication” • Thursday 2:50pm “Performance comparisons and trade-offs for various MySQL replication schemes”Monday, April 11, 2011 225
  226. 226. ADVERTISINGMonday, April 11, 2011 226
  227. 227. WE ARE HIRING! QA and support engineers http://www.continuent.com/about/careersMonday, April 11, 2011 227
  228. 228. http://opendbcamp.org Open Database Camp Sardinia Technology Park 6-7-8 May 2011Monday, April 11, 2011 228
  229. 229. Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://code.google.com/p/tungsten-replicator © Continuent 2011Monday, April 11, 2011 229
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×