• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Advanced mysql replication techniques
 

Advanced mysql replication techniques

on

  • 12,608 views

Advanced MySQL Replication techniques

Advanced MySQL Replication techniques

Statistics

Views

Total Views
12,608
Views on SlideShare
12,606
Embed Views
2

Actions

Likes
20
Downloads
580
Comments
4

1 Embed 2

https://twimg0-a.akamaihd.net 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

14 of 4 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Advanced mysql replication techniques Advanced mysql replication techniques Presentation Transcript

    • Advanced MySQL Replication Techniques Giuseppe Maxia, QA Director, Continuent, Inc Facts. And Demos. Possibly funMonday, April 11, 2011 1
    • about me - Giuseppe Maxia • a.k.a. The Data Charmer • QA Director at Continuent, Inc • Long time hacking with MySQL features • Formerly, community manager,db consultant, designer, coder. • A passion for QA and open source • Blogger • http://datacharmer.blogspot.comMonday, April 11, 2011 2
    • Agenda • Replication basics • Replication blues • Master switch and failover • Slave lagging • Gotchas • Replication power techniques and features • row-based, semi-synch, delayed • Circular, bi-directional • leveraging replication • Tools and tips • Replication outside the box •Monday, April 11, 2011 seamless failover, parallel, multi-master 3
    • DA AG EN Replication basics • Principles • How to set up replication • Monitoring replicationMonday, April 11, 2011 4
    • DA AG EN PrinciplesMonday, April 11, 2011 5
    • database server a simple web application scheme r/w requests web server clients 6Monday, April 11, 2011 6
    • database server scaling web requests r/w requests web servers load balancer clients 7Monday, April 11, 2011 7
    • read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clients 8Monday, April 11, 2011 8
    • client master transaction binary log reads slave IO thread relay log replication concepts SQL thread reads 9Monday, April 11, 2011 9
    • DA AG EN replication setupMonday, April 11, 2011 10
    • 1 SHUT DOWN THE DATABASE SERVER Master 11Monday, April 11, 2011 11
    • 2 MAKE A BACKUP COPY Master 12Monday, April 11, 2011 12
    • 3 ENABLE THE MASTER Master Configuration file [mysqld] log-bin=mysql-bin server-id=1 13Monday, April 11, 2011 13
    • 4 RESTART THE MASTER Master 14Monday, April 11, 2011 14
    • 5 CREATE REPLICATION USER Master SQL command GRANT REPLICATION SLAVE ON *.* to slave_user@10.10.100.% IDENTIFIED BY slave_pass; 15Monday, April 11, 2011 15
    • 6 INSTALL MySQL on the slave Slave 1Make sure that:• Youre using the same version of MySQL• You have the same directory structure• The server is not started yet 16Monday, April 11, 2011 16
    • 7 COPY THE MASTER DATA to the slave Slave 1 17Monday, April 11, 2011 17
    • 8 ENABLE THE SLAVE Slave 1 Configuration file [mysqld] server-id=2 relay-log=mysql-relay read-only # optional: log-bin=mysql-bin 18Monday, April 11, 2011 18
    • 9 START THE SLAVE SERVER Slave 1 19Monday, April 11, 2011 19
    • 10 INITIALIZE THE SLAVE Slave 1 SQL command CHANGE MASTER TO MASTER_HOST=master_IP, MASTER_PORT=3306, MASTER_USER=slave_user, MASTER_PASSWORD=slave_pwd; 20Monday, April 11, 2011 20
    • 11 START THE SLAVE SERVICE Slave 1 SQL command START SLAVE; 21Monday, April 11, 2011 21
    • 12 CHECK THE SLAVE Slave 1 SQL command SHOW SLAVE STATUS G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... 22Monday, April 11, 2011 22
    • Troubleshooting • SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No • Make sure that the slave host can connect to the master • Make sure that master and slave have different Server-id • Check the error log of both master and slaveMonday, April 11, 2011 23
    • Testing the slave • Create a table in the master. • Make sure that the slave has replicated the table. • Insert data in the master • read that data in the slaveMonday, April 11, 2011 24
    • DA AG EN MonitoringMonday, April 11, 2011 25
    • Sample get slave status monitoring slave Running? No Yes master Get master binlog and position Same or later No Yes binlog/position? check table contents alert 26Monday, April 11, 2011 26
    • monitoring replication master> SHOW MASTER STATUS slave> SHOW SLAVE STATUS FULL SCRIPTS: http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans- mysql.html http://forge.mysql.com/tools/tool.php?id=6Monday, April 11, 2011 27
    • show master status File: mysql-bin.000002 Position: 78045744 Binlog_Do_DB: Binlog_Ignore_DB:Monday, April 11, 2011 28
    • show slave status Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: rsandbox Master_Port: 27371 Connect_Retry: 60 Master_Log_File: mysql-bin.000002 Read_Master_Log_Pos: 78045744 Relay_Log_File: mysql_sandbox27372-relay-bin.000055 Relay_Log_Pos: 78045889 Relay_Master_Log_File: mysql-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: YesMonday, April 11, 2011 29
    • show slave status ... Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: ...Monday, April 11, 2011 30
    • show slave status ... Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 78045744 Relay_Log_Space: 78046100 ... Seconds_Behind_Master: 0 ... Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:Monday, April 11, 2011 31
    • DA AG EN Replication blues • Master switch and failover • Slave lagging • GotchasMonday, April 11, 2011 32
    • DA AG EN Master switch and failoverMonday, April 11, 2011 33
    • Replacing the Master master crashe s Let all slaves catch up with execution STOP replication in all slaves FIND the most make it the up to date slave master FIND which run missing connect all transactions are transactions slaves to the missing from to other new master other slaves slaves Stop 34Monday, April 11, 2011 34
    • Planned master switch • Stop accepting writes • Wait until all slaves have caught up • Stop replication in all slaves • Promote a slave to master • Point all slaves to the new masterMonday, April 11, 2011 35
    • Changing a failed master • Pre-requisite: • log_bin and log_slave_updates must be enabled in all the slaves • If not, there is more manual laborMonday, April 11, 2011 36
    • Changing a failed master (1) • Wait until all slaves have caught up • Identify the most advanced slave • Make that slave the new master • ... so far, so goodMonday, April 11, 2011 37
    • Changing a failed master (2) • For each remaining slave: • Find the missing statements • Find the LAST statement replicated by the slave • Find the same statement in the new master binlog (*) • get the position of the NEXT statement (*) if log_slave_updates was not enabled, you need to convert the relay log statements to SQL and do the next step manuallyMonday, April 11, 2011 38
    • Changing a failed master (3) • For each remaining slave: • Apply the missing statements • CHANGE MASTER TO master_host=”new_master_hostname”, master_port=new_master_port, master_log_file=”mysql-bin.xxxxxx”, master_log_pos=YYYYMonday, April 11, 2011 39
    • Reasons for complexity • No global transaction IDMonday, April 11, 2011 40
    • What is a global transaction ID • A unique identifier of a transaction • Unique for the whole cluster, not for each node • Generated by the ultimate source (the master) • Does not change when the transaction goes through an intermediate masterMonday, April 11, 2011 41
    • Why should you care • Failure recovery • MySQL DOES NOT have itMonday, April 11, 2011 42
    • Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Monday, April 11, 2011 43
    • The master crashes Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Monday, April 11, 2011 44
    • Where do the slaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary log Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 45
    • Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 46
    • Need to synch the missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Monday, April 11, 2011 47
    • Transaction position in the new master binlog New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ? ======== ? ======== ========Monday, April 11, 2011 48
    • An existing solution • Google has made a patch to implement global transaction IDs • HOWEVER • It only works with 5.0 • It doesnt work with row based replicationMonday, April 11, 2011 49
    • Hack ahead!Monday, April 11, 2011 50
    • CREATE TABLE Global_Trans_ID ( number INT UNSIGNED AUTO_INCREMENT PRIMARY KEY ) ENGINE = MyISAM; CREATE TABLE Last_Exec_Trans ( server_id INT UNSIGNED, trans_id INT UNSIGNED ) ENGINE = InnoDB;Monday, April 11, 2011 51
    • CREATE PROCEDURE commit_trans () BEGIN DECLARE trans_id, server_id INT UNSIGNED; SET SQL_LOG_BIN = 0; INSERT INTO global_trans_id() values (); SELECT LAST_INSERT_ID() INTO trans_id, @@server_id INTO server_id; DELETE FROM global_trans_id where number < trans_Id; SET SQL_LOG_BIN = 1; INSERT INTO last_exec_trans(server_id, trans_id) VALUES (server_id, trans_id); COMMIT; ENDMonday, April 11, 2011 52
    • Hacking the hackMonday, April 11, 2011 53
    • Simplifying the scheme • No MyISAM table and LAST_INSERT_ID() • Use UUID_SHORT() instead • Replace the InnoDB table with a BlackHole oneMonday, April 11, 2011 54
    • Advantages • 10 times faster • No additional tables and storage requiredMonday, April 11, 2011 55
    • Disadvantages • The master SERVER ID must be < 256Monday, April 11, 2011 56
    • Another solution • We’ll see that in the second part of the talkMonday, April 11, 2011 57
    • DA AG EN Slave laggingMonday, April 11, 2011 58
    • Slave lagging • Common causes: • Excess of traffic • Slave restart • Long DDL operationsMonday, April 11, 2011 59
    • Slave lagging transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra csio s ac ctio ac ion ion n an n s an nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio t tra tra REPLICATION MySQL replication is SINGLE THREAD transaction transaction MySQL DBMSMonday, April 11, 2011 60
    • Slave lagging remedies • row-based replication (helps in some cases) • Expensive query split • Double apply (dangerous) • Slave query prefetch (requires external app) • Parallel replication (requires external app)Monday, April 11, 2011 61
    • statement-based replication master> set global binlog_format=‘statement’; master> insert into test.t1 select count(*) from information_schema.columns; master> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+ slave> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+Monday, April 11, 2011 62
    • row-based replication master> set global binlog_format=‘row’; master> insert into test.t1 select count(*) from information_schema.columns; master> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 41 | +---------------+-------+ slave> show global status like opened_tables; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Opened_tables | 17 | +---------------+-------+Monday, April 11, 2011 63
    • Expensive query split # instead of this: insert into t1 select benchmark(10000000, sha(abc)); Query OK, 1 row affected (4.19 sec) Records: 1 Duplicates: 0 Warnings: 0 # you may do this set @result=(select benchmark(10000000, sha(abc))); Query OK, 0 rows affected (4.19 sec) insert into t1 values (@result); Query OK, 1 row affected (0.00 sec)Monday, April 11, 2011 64
    • Expensive query split # at 471 #110410 8:21:21 server id 1 end_log_pos 518 User_var SET @`result`:=0/*!*/; # at 518 #110410 8:21:21 server id 1 end_log_pos 612 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP=1302448881/*!*/; insert into t1 values (@result)Monday, April 11, 2011 65
    • Slave query prefetch • What is it? • a technique that fetches queries from the relay logs, and runs them as SELECT statements, to warm up the indexes • How do you do it? • mk-slave-prefetch (http://www.maatkit.org) • Slave readahead http://sourceforge.net/ projects/slavereadahead/Monday, April 11, 2011 66
    • DA AG EN GotchasMonday, April 11, 2011 67
    • Gotchas • When you thought that you had figured out everything ...Monday, April 11, 2011 68
    • Gotchas: LOAD DATA • LOAD DATA INFILE is replicated as a temporary file • The slave needs three times the size of the data: • the relay log • the temporary file • the data itselfMonday, April 11, 2011 69
    • Gotchas: default engine • If you define storage_engine in the master, it does not get replicatedMonday, April 11, 2011 70
    • Gotchas: binlog format • In circular replication, and relay slaves • Changes to binlog_format are not propagatedMonday, April 11, 2011 71
    • Gotchas: binlog format server id: 101 insert into t1 format: statement values (@@server_id) 101 server id: 102 format: row 102 server id: 103 102 format: rowMonday, April 11, 2011 72
    • Gotchas: set global not really global • set global binlog_format=row • Works for all new connections after the change • It does not work for the event scheduler!Monday, April 11, 2011 73
    • Gotchas: triggers and row replication • In statement based replication, triggers are fired both on master and slave • in row-based replication, triggers are fired on master only, and the resulting rows are moved to the slaveMonday, April 11, 2011 74
    • DA AG EN Replication power techniques and features • row-based • semi-synch • delayed • Circular • bi-directional • Leveraging replicationMonday, April 11, 2011 75
    • DA AG EN row-based replicationMonday, April 11, 2011 76
    • row-based replication • Available in 5.1 and later • can be changed using “binlog_format” • row • mixed • statement (default)Monday, April 11, 2011 77
    • row-based replication • DO • when you have expensive queries in the master • when you are using non-deterministic functions • DON’T • When you are updating a lot of rowsMonday, April 11, 2011 78
    • DA AG EN semi-synch replicationMonday, April 11, 2011 79
    • semi-synchronous replication • Available in 5.5 and higher • Makes sure that at least one slave has copied the data. • Increases reliabilityMonday, April 11, 2011 80
    • client master transaction 1 with regular commit replication execute 2 slave returns 4 to client binary log 3 5 replication 81Monday, April 11, 2011 81
    • client transaction master 1 with semi- commit synchronous replication execute 2 slave 7 returns to client binary log 3 sends 4 transaction relay log 5 to slave gets 6 acknowledgement 82Monday, April 11, 2011 82
    • semi-synchronous replication in practice • installation: • it’s a plugin. • Actually, two pluginsMonday, April 11, 2011 83
    • semi-synch replication install # in the master plugin-load=rpl_semi_sync_master=semisync_master.so rpl_semi_sync_master_enabled=1 # in each slave plugin-load=rpl_semi_sync_slave=semisync_slave.so rpl_semi_sync_slave_enabled=1 # restar all serversMonday, April 11, 2011 84
    • semi-synch replication check # in the master show variables like rpl_semi%; +------------------------------------+-------+ | Variable_name | Value | +------------------------------------+-------+ | rpl_semi_sync_master_enabled | ON | | rpl_semi_sync_master_timeout | 10000 | | rpl_semi_sync_master_trace_level | 32 | | rpl_semi_sync_master_wait_no_slave | ON | +------------------------------------+-------+Monday, April 11, 2011 85
    • semi-synch replication check show status like "rpl_semi_%tx"; +-----------------------------+-------+ | variable_name | value | +-----------------------------+-------+ | RPL_SEMI_SYNC_MASTER_NO_TX | 0 | | RPL_SEMI_SYNC_MASTER_YES_TX | 0 | +-----------------------------+-------+Monday, April 11, 2011 86
    • semi-synch replication test master> create table t1 ( i int); Query OK, 0 rows affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 0 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+Monday, April 11, 2011 87
    • disabling semi-synch # for each slave set global rpl_semi_sync_slave_enabled=0; stop slave io_thread; start slave io_thread;Monday, April 11, 2011 88
    • disabled semi-synch replication test master> insert into t1 values (1); Query OK, 1 row affected (10.00 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 1 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 89
    • disabled semi-synch replication test master> insert into t1 values (2); Query OK, 1 row affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 2 | | Rpl_semi_sync_master_yes_tx | 1 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 90
    • re-enabling semi-synch # in one slave set global rpl_semi_sync_slave_enabled=1; stop slave io_thread; start slave io_thread;Monday, April 11, 2011 91
    • reenabled semi-synch replication test master> insert into t1 values (3); Query OK, 1 row affected (0.01 sec) master> show status like "rpl_semi_%tx"; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | Rpl_semi_sync_master_no_tx | 2 | | Rpl_semi_sync_master_yes_tx | 2 | +-----------------------------+-------+ 2 rows in set (0.00 sec)Monday, April 11, 2011 92
    • DA AG EN delayed replicationMonday, April 11, 2011 93
    • delayed replication in practice STOP SLAVE; change master to master_delay=60; START SLAVE;Monday, April 11, 2011 94
    • delayed replication • DEMOMonday, April 11, 2011 95
    • DA AG EN Circular replicationMonday, April 11, 2011 96
    • Circular A A MySQL B DBMS MySQL DBMS B master master MySQL DD DBMS MySQL DBMS C CMonday, April 11, 2011 master 97
    • DA AG EN bi-directional replicationMonday, April 11, 2011 98
    • bi-directional replication MySQL MySQL DBMS DBMSMonday, April 11, 2011 99
    • bi-directional replication MySQL MySQL DBMS DBMS Los Angeles New York MySQL MySQL MySQL DBMS DBMS DBMS MySQL DBMSMonday, April 11, 2011 100
    • DA AG EN Leveraging replicationMonday, April 11, 2011 101
    • Overview of MySQL partitionsMonday, April 11, 2011 102
    • Partition pruning 1a - unpartitioned table - SINGLE RECORD I D select * N from table_name A D where colx = T E 120 A XMonday, April 11, 2011 103
    • Partition pruning 2a - table partitioned by colx - SINGLE REC 1-99 100-199 select * from table_name 200-299 where colx = 120 300-399 400-499 500-599Monday, April 11, 2011 104
    • Partition pruning 1a - unpartitioned table - RANGE select * I from D N table_name A where colx D between 120 T E and 230 A XMonday, April 11, 2011 105
    • Partition pruning 2a - table partitioned by colx - RANGE 1-99 select * 100-199 from table_name 200-299 where colx between 120 and 230 300-399 400-499 500-599Monday, April 11, 2011 106
    • Storage comparison engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13Monday, April 11, 2011 107
    • Benchmarking results engine 6 month range InnoDB 4 min 30s MyISAM 25.03s Archive 22 min 25s InnoDB partitioned by month 13.19 MyISAM partitioned by year 6.31 MyISAM partitioned by month 4.45 Archive partitioned by year 16.67 Archive partitioned by month 8.97Monday, April 11, 2011 108
    • Partitions limits § Little concurrency § Need of using the partitioning columnMonday, April 11, 2011 109
    • The ARCHIVE storage engineMonday, April 11, 2011 110
    • Size matters Innodb ArchiveMonday, April 11, 2011 111
    • REPLICATION PARTITIONING large data ARCHIVEMonday, April 11, 2011 112
    • Replication = let the slave do the dirty workMonday, April 11, 2011 113
    • masterbackup slaves STOP SLAVE remove slave perform from load backup balancer attach slave START Let slave to load SLAVE catch up balancerMonday, April 11, 2011 114
    • master makesummary tables slaves STOP SLAVE calculate remove slave summary from load tables balancer attach slave Let slave START SLAVE to load catch up balancerMonday, April 11, 2011 115
    • master Partitions for heavy innodb non partitioned statistics slave slave innodb innodb non partitioned partitioned by range slave MyISAM partitioned by rangeMonday, April 11, 2011 116
    • Simulating master multiple innodb non partitioned dimensions slave slave innodb non partitioned ARCHIVE partitioned by range (date) slave slave ARCHIVE partitioned by range ARCHIVE (location) partitioned by range (product)Monday, April 11, 2011 117
    • DA AG EN Tools and tips • MySQL Sandbox • Replaying binary logs • Filtering binary logs • Baron Schwartz’s Maatkit • Shlomi Noach’s openark kit • Facebook’s Online Schema Change • Measuring replication speed • Detecting if replication is onMonday, April 11, 2011 118
    • DA AG EN MySQL SandboxMonday, April 11, 2011 119
    • MySQL Sandbox http://mysqlsandbox.net • Free software (Perl under GPL) • One (unix) host • Many database servers • Single or multiple sandboxes • Customized scripts to use the servers • Standard or circular replication • Installs IN SECONDSMonday, April 11, 2011 120
    • overview MySQL MySQL server server Data DB1 Data DB1 DB2 DB3 DATA DIRECTORY DB2 DB3 PORT SOCKETMonday, April 11, 2011 121
    • overview MySQL MySQL server server Data DB1 SAME Data DB1 DB2 DB3 DATA DB2 DB3 DIRECTORY? /var/lib/mysql /var/lib/mysql DATA CORRUPTIONMonday, April 11, 2011 122
    • overview MySQL MySQL server server SAME PORT or SOCKET? 3306 3306 /tmp/mysql.sock /tmp/mysql.sock DOES NOT STARTMonday, April 11, 2011 123
    • The hard way (1)Monday, April 11, 2011 124
    • the hard way (2)Monday, April 11, 2011 125
    • The easy way $ make_sandbox /path/to/mysql-5.1.54_linux.tar.gz # it should work alwaysMonday, April 11, 2011 126
    • The easier way $ make_sandbox 5.1.54 # Needs some preliminary workMonday, April 11, 2011 127
    • The easiest way $ sb 5.1.54 # Needs the same preliminary workMonday, April 11, 2011 128
    • MySQL Sandbox VERSION MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_VERSION/data VERSION /tmp/mysql_VERSION.sockMonday, April 11, 2011 129
    • MySQL Sandbox 5.1.54 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_1_54/data 5154 /tmp/mysql_5154.sockMonday, April 11, 2011 130
    • MySQL Sandbox 5.5.9 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_5_09/data 5509 /tmp/mysql_5509.sockMonday, April 11, 2011 131
    • Single Sandbox MySQL customized scripts server start stop restart status clear send_kill useMonday, April 11, 2011 132
    • Multiple Sandbox MySQL customized scripts server start_all stop_all restart_all m n1 status_all s1 n2 clear_all s2 n3 send_kill_a ll use_allMonday, April 11, 2011 133
    • Where do you get it •from CPAN sudo cpan MySQL::Sandbox •from launchpad http://launchpad.net/mysql-sandboxMonday, April 11, 2011 134
    • The easy replication way $ make_replication_sandbox /path/to/mysql-5.1.54_linux.tar.gz # or, after some preparation $ make_replication_sandbox 5.1.54Monday, April 11, 2011 135
    • default architecture $HOME /sandboxes opt expanded tarballs $SANDBOX_HOME mysql $SANDBOX_BINARY installed sandboxesMonday, April 11, 2011 136
    • default architecture $HOME /sandboxes opt msb_5_0_91 mysql msb_5_1_48 5.0.91 rsandbox_5_1_48 5.1.45 master 5.1.48 node1 5.5.4 node2Monday, April 11, 2011 137
    • Monday, April 11, 2011 138
    • creating a single sanbox make_sandbox /path/to/mysql-X.X.XX-OS.tar.gzMonday, April 11, 2011 139
    • using a single sanbox # after # make_sandbox # /path/to/mysql-X.X.XX-OS.tar.gz $ cd $SANDBOX_HOME/msb_X_X_XX $ ./useMonday, April 11, 2011 140
    • creating a single sanbox with a specific options file make_sandbox /path/to/mysql-X.X.XX-OS.tar.gz --my_file=/path/to/my.cnfMonday, April 11, 2011 141
    • easily create a sandbox after the first one The long way $ cd $HOME/opt/mysql # $SANDBOX_BINARY $ gunzip -c /path/to/mysql-5.1.34-osx10.5-x86.tar.gz | tar -xf - $ mv mysql-5.1.34-osx10.5-x86 5.1.34 $ make sandbox 5.1.34Monday, April 11, 2011 142
    • easily create a sandbox after the first one The short way $ make_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gz --export_binariesMonday, April 11, 2011 143
    • starting a single sanbox $ cd $SANDBOX_HOME/msb_X_X_XX $ ./startMonday, April 11, 2011 144
    • starting a single sanbox with temporary options $ cd $SANDBOX_HOME/msb_X_X_XX $ ./start --option=value $ ./restart --option=value $ ./start --key-buffer=20000000Monday, April 11, 2011 145
    • creating a sandbox with custom port and directory $ make_sandbox 5.1.34 --sandbox_port=7800 --sandbox_directory=mickeymouseMonday, April 11, 2011 146
    • creating a sandbox with automatic port checking $ make_sandbox 5.1.34 --check_port # if 5.1.34 is free # port=5134 # directory=msb_5_1_34 # else # port=5135 (or the first free) # directory=msb_5_1_34_aMonday, April 11, 2011 147
    • create a replication sandbox $ make_replication_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gzMonday, April 11, 2011 148
    • create a circular replication sandbox $ make_replication_sandbox --circular=4 path/to/mysql-5.1.34-osx10.5-x86.tar.gzMonday, April 11, 2011 149
    • changing port to an existing sandbox $ sbtool -o port -s /path/to/source/sandbox --new_port=XXXXMonday, April 11, 2011 150
    • installing the innodb plugin $ sbtool -o plugin --plugin=innodb -s /path/to/source/sandboxMonday, April 11, 2011 151
    • creating a replication sandbox with new base port $ make_replication_sandbox --replication_directory=newwdir --check_base_port 5.0.79 # Creates a replication directory under # $SANDBOX_HOME/newdir # The previous one is preserved. # No conflicts happenMonday, April 11, 2011 152
    • DA AG EN Replaying binary logsMonday, April 11, 2011 153
    • DA AG EN Filtering binary logsMonday, April 11, 2011 154
    • DA AG EN MaatkitMonday, April 11, 2011 155
    • maatkit • mk-heartbeat Monitor MySQL replication delay. • mk-purge-logs Purge binary logs on a master based on purge rules. • mk-slave-delay Make a MySQL slave server lag behind its master. • mk-slave-find Find and print replication hierarchy tree of MySQL slaves. • mk-slave-move Move a MySQL slave around in the replication hierarchy. • mk-slave-prefetch Pipeline relay logs on a MySQL slave to pre-warm caches. • mk-slave-restart Watch and restart MySQL replication after errors. • mk-table-checksum Perform an online replication consistency check, or checksum MySQL tables efficiently on one or many servers. • mk-table-sync Synchronize MySQL table data efficiently. • http://www.maatkit.org/Monday, April 11, 2011 156
    • Facebook Online Schema Change DA N AGEMonday, April 11, 2011 157
    • Facebook Online Schema Change • Based on one of Shlomi Noach OpenArk utilities • Developed in PHP by Facebook • http://www.facebook.com/note.php?note_id=430801045932Monday, April 11, 2011 158
    • DA AG EN OpenArkMonday, April 11, 2011 159
    • openark • oak-get-slave-lag: print slave replication lag and terminate with respective exit code. • oak-online-alter-table: perform a non-blocking ALTER TABLE operation. • oak-purge-master-logs: purge master logs, depending on the state of replicating slaves. • oak-show-replication-status: show how far behind are replicating slaves on a given master. • http://code.openark.org/forge/openark-kitMonday, April 11, 2011 160
    • DA AG EN Check replication speedMonday, April 11, 2011 161
    • check replication speed • Write a time to the master • Retrieve it from the slave • Compare times • http://forge.mysql.com/tools/tool.php? id=5Monday, April 11, 2011 162
    • DA AG EN Replication outside the box • Seamless failover • Parallel replication • Multiple source replication • Multiple master replicationMonday, April 11, 2011 163
    • Advanced MySQL Replication for the massesMonday, April 11, 2011 164
    • Once Upon A Time, In The Life Of A Database Consultant ...Monday, April 11, 2011 165
    • The story of a steel foundry • Used MySQL databases to store production monitoring data • Inserted a zillion records per second. • Slaves often lagged behindMonday, April 11, 2011 166
    • transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n tio ct tra tra REPLICATION transaction transaction MySQL DBMSMonday, April 11, 2011 167
    • The story of a shoe maker • Had a successful business, spread to a dozen stores. • Needed to aggregate the data from the stores in his headquarters database.Monday, April 11, 2011 168
    • MySQL MySQL DBMS MySQL DBMS DBMS store store store MySQL DBMS headquartersMonday, April 11, 2011 169
    • The story of a widgets seller • Had a successful business, designed for one server. • Products were created in several sites. • Needed to allow insertions from more than one master at the timeMonday, April 11, 2011 170
    • MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master masterMonday, April 11, 2011 171
    • All these stories tell us: Nice dream, but MySQL can’t do itMonday, April 11, 2011 172
    • Enter Tungsten ReplicatorMonday, April 11, 2011 173
    • Tungsten Replicator 2.0 • What is it?Monday, April 11, 2011 174
    • http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2Monday, April 11, 2011 175
    • What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Parallel replication • Replicate to Oracle and PostgreSQL databaseMonday, April 11, 2011 176
    • MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle Bin Logs (Binlogs enabled) Slave DB © Continuent 2010Monday, April 11, 2011 177
    • From the beginning ...Monday, April 11, 2011 178
    • Fail-over (1)Monday, April 11, 2011 179
    • Fail-over (2)Monday, April 11, 2011 180
    • Fail-over (3)Monday, April 11, 2011 181
    • Fail-over (4)Monday, April 11, 2011 182
    • Fail-over (5)Monday, April 11, 2011 183
    • Fail-over (6)Monday, April 11, 2011 184
    • Fail-over (7)Monday, April 11, 2011 185
    • Fail-over (8)Monday, April 11, 2011 186
    • Failover • DEMOMonday, April 11, 2011 187
    • master/slave with an attitudeMonday, April 11, 2011 188
    • The steel foundry dream or parallel replication From here ...Monday, April 11, 2011 189
    • The steel foundry dream or parallel replication To here.Monday, April 11, 2011 190
    • Parallel replication facts • Sharded by database • Good choice for slave lag problems • Bad choice for single database projectsMonday, April 11, 2011 191
    • Testing parallel replicationMonday, April 11, 2011 192
    • db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbenchMonday, April 11, 2011 193
    • preparation (2) db0 db1 db2 db3 db4 db5 db6 db7 db8 db9Monday, April 11, 2011 194
    • before the test (1) db0 db1 db2 db3 db4 db5 db6 db7 db8 db9Monday, April 11, 2011 195
    • before the test (2) SQL thread RELAY logs IO thread binary logs MySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alphaMonday, April 11, 2011 196
    • starting the test SQL thread RELAY logs IO thread binary logs MySQL slave RELAY logs Tungsten slave direct: alpha (slave) replicator alphaMonday, April 11, 2011 197
    • MySQL native replication slave catch up in 00:02:30Monday, April 11, 2011 198
    • Tungsten parallel replication slave catch up in 00:01:20Monday, April 11, 2011 199
    • The widget seller dream, or multi masters • Tungsten Replicator recipe: use more servicesMonday, April 11, 2011 200
    • Bi-directional replicationMonday, April 11, 2011 201
    • Bi-directional replication with slavesMonday, April 11, 2011 202
    • True multiple master We’ll see that in a moment. But firstMonday, April 11, 2011 203
    • The shoe maker dream, or multiple sources • Tungsten Replicator recipe is still valid: use more servicesMonday, April 11, 2011 204
    • Multiple source replicationMonday, April 11, 2011 205
    • Multiple masters replication: 3 nodesMonday, April 11, 2011 206
    • Multiple masters replication: 4 nodesMonday, April 11, 2011 207
    • Updating 4 masters : 1 flowMonday, April 11, 2011 208
    • Updating 4 masters : 2 flowsMonday, April 11, 2011 209
    • Updating 4 masters : 3 flowsMonday, April 11, 2011 210
    • Updating 4 masters : 4 flowsMonday, April 11, 2011 211
    • Tungsten in practice • installationMonday, April 11, 2011 212
    • Installation: the long way • Get the binaries • Expand the tarball • Check the requirements in the manual • Run ./configure • Run ./configure-serviceMonday, April 11, 2011 213
    • Installation: the quick way • Get the tarball • Get the tungsten deployer from Google Code • Sit back and enjoy!Monday, April 11, 2011 214
    • Tools • replicator • trepctl • thlMonday, April 11, 2011 215
    • replicator • It’s the service provider • You launch it once when you start • You may restart it when you change configMonday, April 11, 2011 216
    • trepctl • Tungsten Replicator ConTroLler • It’s the driving seat for your replication • You can start, update, and stop services • You can get specific infoMonday, April 11, 2011 217
    • thl • Transaction History List • Gives you access to the Tungsten relay logsMonday, April 11, 2011 218
    • DA AG EN More info on replicationMonday, April 11, 2011 219
    • BooksMonday, April 11, 2011 220
    • High Performance MySQLMonday, April 11, 2011 221
    • MySQL High AvailabilityMonday, April 11, 2011 222
    • Web OperationsMonday, April 11, 2011 223
    • Cloud Application ArchitecturesMonday, April 11, 2011 224
    • Talks • Monday, 1:30pm “Learn how to cure replication deprivation with Tungsten” • Tuesday 10:50am “Replication Update” • Tuesday 2:00pm “Diagnosing replication failures” • Tuesday 3:05 “Advanced replication monitoring” • Thursday 10:50am “Replication for Availability & Durability with MySQL and Amazon RDS” • Thursday 11:55am “Build your own PaaS for MySQL with Tungsten Enterprise” • Thursday 2:00pm “Error detection and correction with MySQL Replication” • Thursday 2:50pm “Performance comparisons and trade-offs for various MySQL replication schemes”Monday, April 11, 2011 225
    • ADVERTISINGMonday, April 11, 2011 226
    • WE ARE HIRING! QA and support engineers http://www.continuent.com/about/careersMonday, April 11, 2011 227
    • http://opendbcamp.org Open Database Camp Sardinia Technology Park 6-7-8 May 2011Monday, April 11, 2011 228
    • Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://code.google.com/p/tungsten-replicator © Continuent 2011Monday, April 11, 2011 229