Replication 101

2,246 views
2,137 views

Published on

beginners' guide to MySQL replication

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,246
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
149
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Replication 101

  1. 1. MySQL Replication 101 Giuseppe Maxia Continuent, Inc ©Continuent 2012.Tuesday, April 10, 12 1
  2. 2. about me - Giuseppe Maxia • a.k.a. The Data Charmer • QA Director at Continuent, Inc • Long time hacking with MySQL features • Formerly, community manager,db consultant, designer, coder. • A passion for QA and open source • Blogger • http://datacharmer.blogspot.comTuesday, April 10, 12 2
  3. 3. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 3Tuesday, April 10, 12 3
  4. 4. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 4Tuesday, April 10, 12 4
  5. 5. www $$$ The world today is dominated by the web economy 5Tuesday, April 10, 12 5
  6. 6. www $$$ Databases are the backbone of the web economy 6Tuesday, April 10, 12 6
  7. 7. What database for the web? 7Tuesday, April 10, 12 7
  8. 8. What database for the web? The most powerful database 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. 7Tuesday, April 10, 12 7
  9. 9. What database for the web? The most powerful database 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. The most advanced open source database 7Tuesday, April 10, 12 7
  10. 10. What database for the web? The most powerful database 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. The most advanced open source database The most deployed open source database 7Tuesday, April 10, 12 7
  11. 11. What database for the web? The most powerful database 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. The most advanced open source database The most deployed open source database The most popular open source database 7Tuesday, April 10, 12 7
  12. 12. www $$$ Actually, MySQL databases are the backbone of the web economy 8Tuesday, April 10, 12 8
  13. 13. What database for the web? No built-in replication ✗ ✗ 1 Copyright © 2011, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. No built-in replication No built-in replication ✗ Built-in replication ✔ 9Tuesday, April 10, 12 9
  14. 14. www $$$ More precisely, MySQL REPLICATION is the backbone of the web economy 10Tuesday, April 10, 12 10
  15. 15. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 11Tuesday, April 10, 12 11
  16. 16. database server a simple web application scheme r/w requests web server clients 12Tuesday, April 10, 12 12
  17. 17. database server scaling web requests r/w requests web servers load balancer clients 13Tuesday, April 10, 12 13
  18. 18. database load on a simple web application r e write a d 85% 15% 14Tuesday, April 10, 12 14
  19. 19. write read 20% 80% database load on a successful web application 15Tuesday, April 10, 12 15
  20. 20. database server scaling up means buying a bigger ✘ r/w requests database server web servers load balancer clients 16Tuesday, April 10, 12 16
  21. 21. write read 20% 80% the bigger database server will eventually have the same problem 17Tuesday, April 10, 12 17
  22. 22. read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clients 18Tuesday, April 10, 12 18
  23. 23. r e write a read d 85% 15% 100% read/write master read/only slaves database load with replication 19Tuesday, April 10, 12 19
  24. 24. r e write a read d 85% 15% 100% read/write master read/only slaves scaling database load with replication 20Tuesday, April 10, 12 20
  25. 25. Replication assessment without replication with replication database handling easy harder performance high lower (binary logs) Point in Time recovery none easy failover none possible write scaling none minimal backup with downtime without downtime read scaling none easy 21Tuesday, April 10, 12 21
  26. 26. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 22Tuesday, April 10, 12 22
  27. 27. client master transaction binary log reads slave IO thread relay log replication concepts SQL thread reads 23Tuesday, April 10, 12 23
  28. 28. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 24Tuesday, April 10, 12 24
  29. 29. 1 SHUT DOWN THE DATABASE SERVER Master 25Tuesday, April 10, 12 25
  30. 30. 2 MAKE A BACKUP COPY Master 26Tuesday, April 10, 12 26
  31. 31. 3 ENABLE THE MASTER Master Configuration file [mysqld] log-bin=mysql-bin server-id=1 27Tuesday, April 10, 12 27
  32. 32. 4 RESTART THE MASTER Master 28Tuesday, April 10, 12 28
  33. 33. 5 CREATE REPLICATION USER Master SQL command GRANT REPLICATION SLAVE ON *.* to slave_user@10.10.100.% IDENTIFIED BY slave_pass; 29Tuesday, April 10, 12 29
  34. 34. 6 INSTALL MySQL on the slave Slave 1Make sure that:• Youre using the same version of MySQL• You have the same directory structure• The server is not started yet 30Tuesday, April 10, 12 30
  35. 35. 7 COPY THE MASTER DATA to the slave Slave 1 31Tuesday, April 10, 12 31
  36. 36. 8 ENABLE THE SLAVE Slave 1 Configuration file [mysqld] server-id=2 relay-log=mysql-relay read-only # optional: log-bin=mysql-bin 32Tuesday, April 10, 12 32
  37. 37. 9 START THE SLAVE SERVER Slave 1 33Tuesday, April 10, 12 33
  38. 38. 10 INITIALIZE THE SLAVE Slave 1 SQL command SET MASTER TO MASTER_HOST=master_IP, MASTER_PORT=3306, MASTER_USER=slave_user, MASTER_PASSWORD=slave_pwd; 34Tuesday, April 10, 12 34
  39. 39. 11 START THE SLAVE SERVICE Slave 1 SQL command START SLAVE; 35Tuesday, April 10, 12 35
  40. 40. 12 CHECK THE SLAVE Slave 1 SQL command SHOW SLAVE STATUS G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... 36Tuesday, April 10, 12 36
  41. 41. Troubleshooting • SHOW SLAVE STATUS says SLAVE_IO_RUNNING=No • Make sure that the slave host can connect to the master • Make sure that master and slave have different Server-id • Check the error log of both master and slave 37Tuesday, April 10, 12 37
  42. 42. Testing the slave • Create a table in the master. • Make sure that the slave has replicated the table. • Insert data in the master • read that data in the slave 38Tuesday, April 10, 12 38
  43. 43. What if the master was already logging? • You have two options: • Physical copy • Logical copy 39Tuesday, April 10, 12 39
  44. 44. Physical copy • stop master • make copy • remove binary log files and index • start master • (alternative: use xtrabackup) 40Tuesday, April 10, 12 40
  45. 45. Logical copy • mysqldump --all-databases --master-data 41Tuesday, April 10, 12 41
  46. 46. Common replication commands • CHANGE MASTER TO • SHOW MASTER STATUS • SHOW SLAVE STATUS • START SLAVE • STOP SLAVE • RESET MASTER (caution!) • RESET SLAVE (caution!) 42Tuesday, April 10, 12 42
  47. 47. CHANGE MASTER TO basic syntax STOP SLAVE; CHANGE MASTER TO MASTER_HOST=hostname, # or IP MASTER_PORT=3306, MASTER_USER=slaveuser, MASTER_PASSWORD=slavepassword, MASTER_LOG_FILE=filename, # default: first binlog MASTER_LOG_POS=123456; # default: from the beginning START SLAVE;Tuesday, April 10, 12 43
  48. 48. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 44Tuesday, April 10, 12 44
  49. 49. 1 NO NEED TO STOP THE MASTER! Master 45Tuesday, April 10, 12 45
  50. 50. 2 STOP THE SLAVE Slave 1 SQL command STOP SLAVE IO_THREAD; # wait until the SQL_THREAD # has done everything STOP SLAVE SQL_THREAD; # STOP THE SERVER 46Tuesday, April 10, 12 46
  51. 51. 3 MAKE A COPY OF THE DATA DIRECTORY Slave 1 47Tuesday, April 10, 12 47
  52. 52. 4 RESTART THE SLAVE Slave 1 48Tuesday, April 10, 12 48
  53. 53. 5 INSTALL MySQL on the new slave Slave 2 Make sure that: • Youre using the same version of MySQL • You have the same directory structure • The server is not started yet 49Tuesday, April 10, 12 49
  54. 54. 6 COPY THE old slave DATA on the slave Slave 2 50Tuesday, April 10, 12 50
  55. 55. 7 ENABLE THE NEW SLAVE Slave 2 Configuration file uni que! [mysqld] mus t be server-id=3 relay-log=mysql-relay read-only # optional: log-bin=mysql-bin 51Tuesday, April 10, 12 51
  56. 56. 8 START THE NEW SLAVE Slave 2 52Tuesday, April 10, 12 52
  57. 57. 9 CHECK THE SLAVE Slave 2 SQL command SHOW SLAVE STATUS G ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... 53Tuesday, April 10, 12 53
  58. 58. Why it works • No need to issue a CHANGE MASTER TO command. • Because we cloned the old slave • The new slave gets its parameters from the .info files in the data directory 54Tuesday, April 10, 12 54
  59. 59. Starting and stopping replicationTuesday, April 10, 12 55
  60. 60. Starting and stopping replication START SLAVE;Tuesday, April 10, 12 55
  61. 61. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD;Tuesday, April 10, 12 55
  62. 62. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD; START SLAVE IO_THREAD;Tuesday, April 10, 12 55
  63. 63. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD; START SLAVE IO_THREAD; START SLAVE UNTIL MASTER_LOG_FILE=filename, MASTER_LOG_POS=xxxx;Tuesday, April 10, 12 55
  64. 64. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD; START SLAVE IO_THREAD; START SLAVE UNTIL MASTER_LOG_FILE=filename, MASTER_LOG_POS=xxxx; STOP SLAVE;Tuesday, April 10, 12 55
  65. 65. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD; START SLAVE IO_THREAD; START SLAVE UNTIL MASTER_LOG_FILE=filename, MASTER_LOG_POS=xxxx; STOP SLAVE; STOP SLAVE SQL_THREAD;Tuesday, April 10, 12 55
  66. 66. Starting and stopping replication START SLAVE; START SLAVE SQL_THREAD; START SLAVE IO_THREAD; START SLAVE UNTIL MASTER_LOG_FILE=filename, MASTER_LOG_POS=xxxx; STOP SLAVE; STOP SLAVE SQL_THREAD; STOP SLAVE IO_THREAD;Tuesday, April 10, 12 55
  67. 67. DA EN Hijacking the agenda AG • MySQL Sandbox 56Tuesday, April 10, 12 56
  68. 68. MySQL Sandbox http://mysqlsandbox.net • Free software (Perl under GPL) • One (unix) host • Many database servers • Single or multiple sandboxes • Customized scripts to use the servers • Standard or circular replication • Installs IN SECONDSTuesday, April 10, 12 57
  69. 69. overview MySQL MySQL server server Data DB1 Data DB1 DB2 DB3 DATA DIRECTORY DB2 DB3 PORT SOCKETTuesday, April 10, 12 58
  70. 70. overview MySQL MySQL server server Data DB1 SAME Data DB1 DB2 DB3 DATA DB2 DB3 DIRECTORY? /var/lib/mysql /var/lib/mysql DATA CORRUPTIONTuesday, April 10, 12 59
  71. 71. overview MySQL MySQL server server SAME PORT or SOCKET? 3306 3306 /tmp/mysql.sock /tmp/mysql.sock DOES NOT STARTTuesday, April 10, 12 60
  72. 72. The hard wayTuesday, April 10, 12 61
  73. 73. The hard way Read the manualTuesday, April 10, 12 61
  74. 74. The hard way try to figure out Read the manual what to changeTuesday, April 10, 12 61
  75. 75. The hard way try to figure out Read the manual Install what to changeTuesday, April 10, 12 61
  76. 76. The easy way MySQL Sandbox $ make_sandbox /path/to/mysql-5.1.54_linux.tar.gz # it should work alwaysTuesday, April 10, 12 62
  77. 77. The easier way Prepare once Install many times # some $ make_sandbox 5.1.54 # preliminary # workTuesday, April 10, 12 63
  78. 78. The easiest way Prepare once Install many times # some # preliminary $ sb 5.1.54 # workTuesday, April 10, 12 64
  79. 79. MySQL Sandbox VERSION MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_VERSION/data VERSION /tmp/mysql_VERSION.sockTuesday, April 10, 12 65
  80. 80. MySQL Sandbox 5.1.54 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_1_54/data 5154 /tmp/mysql_5154.sockTuesday, April 10, 12 66
  81. 81. MySQL Sandbox 5.5.9 MySQL server Data DB1 DB2 DB3 $SANDBOX_HOME/msb_5_5_09/data 5509 /tmp/mysql_5509.sockTuesday, April 10, 12 67
  82. 82. Single Sandbox MySQL customized scripts server start stop restart status clear send_kill useTuesday, April 10, 12 68
  83. 83. Multiple Sandbox MySQL customized scripts server start_all stop_all restart_all m n1 status_all s1 n2 clear_all s2 n3 send_kill_a ll use_allTuesday, April 10, 12 69
  84. 84. Where do you get it •from CPAN sudo su - cpan MySQL::Sandbox •from launchpad http://launchpad.net/mysql-sandboxTuesday, April 10, 12 70
  85. 85. The easy replication way MySQL Sandbox $ make_replication_sandbox /path/to/mysql-5.1.54_linux.tar.gz Prepare once Install many times # some $ make_replication_sandbox # preparation 5.1.54Tuesday, April 10, 12 71
  86. 86. default architecture $HOME /sandboxes opt expanded tarballs $SANDBOX_HOME mysql $SANDBOX_BINARY installed sandboxesTuesday, April 10, 12 72
  87. 87. default architecture $HOME /sandboxes opt msb_5_0_91 mysql msb_5_1_48 5.0.91 rsandbox_5_1_48 5.1.45 master 5.1.48 node1 5.5.4 node2Tuesday, April 10, 12 73
  88. 88. Tuesday, April 10, 12 74
  89. 89. creating a single sanbox make_sandbox /path/to/mysql-X.X.XX-OS.tar.gzTuesday, April 10, 12 75
  90. 90. using a single sanbox # after # make_sandbox # /path/to/mysql-X.X.XX-OS.tar.gz $ cd $SANDBOX_HOME/msb_X_X_XX $ ./useTuesday, April 10, 12 76
  91. 91. creating a single sanbox with a specific options file make_sandbox /path/to/mysql-X.X.XX-OS.tar.gz --my_file=/path/to/my.cnfTuesday, April 10, 12 77
  92. 92. easily create a sandbox after the first one The long way $ cd $HOME/opt/mysql # $SANDBOX_BINARY $ gunzip -c /path/to/mysql-5.1.34-osx10.5-x86.tar.gz | tar -xf - $ mv mysql-5.1.34-osx10.5-x86 5.1.34 $ make sandbox 5.1.34Tuesday, April 10, 12 78
  93. 93. easily create a sandbox after the first one The short way $ make_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gz --export_binariesTuesday, April 10, 12 79
  94. 94. starting a single sanbox $ cd $SANDBOX_HOME/msb_X_X_XX $ ./startTuesday, April 10, 12 80
  95. 95. starting a single sanbox with temporary options $ cd $SANDBOX_HOME/msb_X_X_XX $ ./start --option=value $ ./restart --option=value $ ./start --key-buffer=20000000Tuesday, April 10, 12 81
  96. 96. creating a sandbox with custom port and directory $ make_sandbox 5.1.34 --sandbox_port=7800 --sandbox_directory=mickeymouseTuesday, April 10, 12 82
  97. 97. creating a sandbox with automatic port checking $ make_sandbox 5.1.34 --check_port # if 5.1.34 is free # port=5134 # directory=msb_5_1_34 # else # port=5135 (or the first free) # directory=msb_5_1_34_aTuesday, April 10, 12 83
  98. 98. create a replication sandbox $ make_replication_sandbox path/to/mysql-5.1.34-osx10.5-x86.tar.gzTuesday, April 10, 12 84
  99. 99. create a circular replication sandbox $ make_replication_sandbox --circular=4 path/to/mysql-5.1.34-osx10.5-x86.tar.gzTuesday, April 10, 12 85
  100. 100. changing port to an existing sandbox $ sbtool -o port -s /path/to/source/sandbox --new_port=XXXXTuesday, April 10, 12 86
  101. 101. installing the innodb plugin $ sbtool -o plugin --plugin=innodb -s /path/to/source/sandboxTuesday, April 10, 12 87
  102. 102. creating a replication sandbox with new base port $ make_replication_sandbox --replication_directory=newwdir --check_base_port 5.0.79 # Creates a replication directory under # $SANDBOX_HOME/newdir # The previous one is preserved. # No conflicts happenTuesday, April 10, 12 88
  103. 103. DA EN How to set up replication AG DEMO 89Tuesday, April 10, 12 89
  104. 104. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 90Tuesday, April 10, 12 90
  105. 105. Binary log formats • Statement based replication • default • available since 3.23 • Row based replication • introduced in 5.1 • ROW or MIXED 91Tuesday, April 10, 12 91
  106. 106. Viewing the binary logs # statement based replication $ mysqlbinlog binary-log-name # or, as a SQL command SHOW BINLOG EVENTS IN binary-log-name; # row based replication $ mysqlbinlog --verbose --base64-output=decode-rows binary-log-nameTuesday, April 10, 12 92
  107. 107. binary log examples # statement based replication mysqlbinlog binary-log-name # row based replication mysqlbinlog --verbose --base64-output=decode-rows binary-log-nameTuesday, April 10, 12 93
  108. 108. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 94Tuesday, April 10, 12 94
  109. 109. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  110. 110. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  111. 111. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  112. 112. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  113. 113. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  114. 114. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  115. 115. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  116. 116. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  117. 117. what gets replicated What Statement-based row-based insert/update/delete statement affected records schema/table/view statement statement creation/drop stored routine/trigger statement statement creation/drop statements executed stored procedure call affected records inside SP stored function call function call affected records none: the slave runs the affected records trigger execution trigger moved. No trigger runs slaveside disabled slaveside disabled event creation event event event execution statement affected records 95Tuesday, April 10, 12 95
  118. 118. DA EN Binary logs in practice AG DEMO 96Tuesday, April 10, 12 96
  119. 119. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 97Tuesday, April 10, 12 97
  120. 120. From single server application r/w requests 98Tuesday, April 10, 12 98
  121. 121. To replication-aware application read/write master read/only slaves R/W load balancer R/O 99Tuesday, April 10, 12 99
  122. 122. Single server application $link = mysql_connect( $server_IP,  mysql_user,  mysql_password ); $result = mysql_query( INSERT INTO table_name (x) VALUES (1), $link ); $result = mysql_query( SELECT * FROM table_name WHERE x=1, $link ); 100Tuesday, April 10, 12 100
  123. 123. Making an application aware of replication <<database handling>> db <<R/W database handling>> IP user db password IP connect user password connect <<read-only database <<R/W database handling>> handling>> read db db write IP IP user user password password read connect read write 101Tuesday, April 10, 12 101
  124. 124. Using replication: the WRONG way No Write Yes statement? Connect to the Connect to next available the master slave Read from Write to the slave master R/W split by Stop statement 102Tuesday, April 10, 12 102
  125. 125. Why statement split is wrong • Breaks transactions • High risk of inconsistency • Loses or corrupts data 103Tuesday, April 10, 12 103
  126. 126. Using replication: the RIGHT way No Write Yes function? Connect to the Connect to next available the master slave Read from Read and write slave from master more Yes more queries? queries? Yes R/W split No No by function Stop 104Tuesday, April 10, 12 104
  127. 127. Replication is single threaded • Let’s assume you have two queries on the master • one query takes 3 minutes to complete • the other takes 5 minutes • Both start at 12:00 noon 105Tuesday, April 10, 12 105
  128. 128. Single thread replication MASTER 12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 12:09 12:10 12:11 SLAVE 106Tuesday, April 10, 12 106
  129. 129. Single thread replication MASTER QUERY 1 QUERY 2 12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 12:09 12:10 12:11 (does nothing) SLAVE 106Tuesday, April 10, 12 106
  130. 130. Single thread replication MASTER QUERY 1 QUERY 2 12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 12:09 12:10 12:11 (does nothing) QUERY 1 SLAVE 106Tuesday, April 10, 12 106
  131. 131. Single thread replication MASTER QUERY 1 QUERY 2 12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 12:09 12:10 12:11 (does nothing) QUERY 1 QUERY 2 SLAVE 106Tuesday, April 10, 12 106
  132. 132. Single thread replication MASTER QUERY 1 QUERY 3 QUERY 5 QUERY 2 QUERY 4 QUERY 6 12:00 12:01 12:02 12:03 12:04 12:05 12:06 12:07 12:08 12:09 12:10 12:11 does nothing QUERY 1 QUERY 2 SLAVE 107Tuesday, April 10, 12 107
  133. 133. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 108Tuesday, April 10, 12 108
  134. 134. Sample get slave status monitoring slave Running? No Yes master Get master binlog and position Same or later No Yes binlog/position? check table contents alert 109Tuesday, April 10, 12 109
  135. 135. commands for checking replication master> SHOW MASTER STATUS master> SHOW PROCESSLIST slave> SHOW SLAVE STATUS slave> SHOW PROCESSLISTTuesday, April 10, 12 110
  136. 136. replication user seen in the master master > show processlistG *************************** 1. row *************** Id: 2 User: rsandbox Host: localhost:57011 db: NULL Command: Binlog Dump Time: 81625 State: Master has sent all binlog to slave; waiting for binlog to be updated Info: NULL [...]Tuesday, April 10, 12 111
  137. 137. IO-thread in the slave slave > show processlistG *************************** 1. row *************** Id: 2 User: system user Host: db: NULL Command: Connect Time: 124801 State: Waiting for master to send event Info: NULL [...]Tuesday, April 10, 12 112
  138. 138. SQL thread in the slave slave > show processlistG [...] *************************** 2. row *************** Id: 3 User: system user Host: db: NULL Command: Connect Time: 124712 State: Slave has read all relay log; waiting for the slave I/O thread to update it Info: NULL [...]Tuesday, April 10, 12 113
  139. 139. monitoring replication master> SHOW MASTER STATUS slave> SHOW SLAVE STATUS FULL SCRIPTS: http://datacharmer.blogspot.com/2011/04/refactored-again-poor-mans- mysql.html http://forge.mysql.com/tools/tool.php?id=6Tuesday, April 10, 12 114
  140. 140. show master status File: mysql-bin.000002 Position: 78045744 Binlog_Do_DB: Binlog_Ignore_DB:Tuesday, April 10, 12 115
  141. 141. show slave status Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: rsandbox Master_Port: 27371 Connect_Retry: 60 Master_Log_File: mysql-bin.000002 Read_Master_Log_Pos: 78045744 Relay_Log_File: mysql_sandbox27372-relay-bin.000055 Relay_Log_Pos: 78045889 Relay_Master_Log_File: mysql-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: YesTuesday, April 10, 12 116
  142. 142. show slave status ... Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: ...Tuesday, April 10, 12 117
  143. 143. show slave status ... Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 78045744 Relay_Log_Space: 78046100 ... Seconds_Behind_Master: 0 ... Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:Tuesday, April 10, 12 118
  144. 144. DA EN Breaking (and fixing) replication AG DEMO 119Tuesday, April 10, 12 119
  145. 145. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 120Tuesday, April 10, 12 120
  146. 146. Logs rotation # server variables max-binlog-size expire-log-days # logs commands SHOW MASTER LOGS; PURGE MASTER LOGS TO filename; FLUSH [BINARY] LOGS;Tuesday, April 10, 12 121
  147. 147. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 122Tuesday, April 10, 12 122
  148. 148. Replacing Slave a slave crashe s No are there Yes more slaves? STOP the master STOP one slave add the first add another slave slave Stop 123Tuesday, April 10, 12 123
  149. 149. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 124Tuesday, April 10, 12 124
  150. 150. Replacing the Master master crashe s Let all slaves catch up with execution STOP replication in all slaves FIND the most make it the up to date slave master FIND which run missing connect all transactions are transactions slaves to the missing from to other new master other slaves slaves Stop 125Tuesday, April 10, 12 125
  151. 151. Planned master switch • Stop accepting writes • Wait until all slaves have caught up • Stop replication in all slaves • Promote a slave to master • Point all slaves to the new masterTuesday, April 10, 12 126
  152. 152. Changing a failed master • Pre-requisite: • log_bin and log_slave_updates must be enabled in all the slaves • If not, there is more manual laborTuesday, April 10, 12 127
  153. 153. Changing a failed master (1) • Wait until all slaves have caught up • Identify the most advanced slave • Make that slave the new master • ... so far, so goodTuesday, April 10, 12 128
  154. 154. Changing a failed master (2) • For each remaining slave: • Find the missing statements • Find the LAST statement replicated by the slave • Find the same statement in the new master binlog (*) • get the position of the NEXT statement (*) if log_slave_updates was not enabled, you need to convert the relay log statements to SQL and do the next step manuallyTuesday, April 10, 12 129
  155. 155. Changing a failed master (3) • For each remaining slave: • Apply the missing statements • CHANGE MASTER TO master_host=”new_master_hostname”, master_port=new_master_port, master_log_file=”mysql-bin.xxxxxx”, master_log_pos=YYYYTuesday, April 10, 12 130
  156. 156. Reasons for complexity • No global transaction IDTuesday, April 10, 12 131
  157. 157. What is a global transaction ID • A unique identifier of a transaction • Unique for the whole cluster, not for each node • Generated by the ultimate source (the master) • Does not change when the transaction goes through an intermediate masterTuesday, April 10, 12 132
  158. 158. Why should you care • Failure recovery • MySQL DOES NOT have it • What can you do? Either wait for MySQL 5.6 or use Tungsten Replicator.Tuesday, April 10, 12 133
  159. 159. DA Why replication • The web economy EN • Scaling out AG • From single server to replication How to set replication • Adding a slave • Binary log formats Using Replication • What gets replicated and how • Replication awareness • Monitoring • Log management Managing replication • Replacing a slave • Replacing a master • master replacement What Replication is for • backup • better reads Gotchas, tips, and tricks • What to read More info • More replication sessions 134Tuesday, April 10, 12 134
  160. 160. Slave = master replacement • if: • there are no filters; • you are monitoring replication • you make sure data is consistent 135Tuesday, April 10, 12 135

×