Mysql replication outside the box

2,296 views

Published on

MySQL Replication using multiple master, parallel replication, seamless fail over, and conflict prevention using Tungsten Replicator

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,296
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
532
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Mysql replication outside the box

  1. 1. MySQL replication outside the box Giuseppe Maxia QA Director Continuent, Inc @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.Tuesday, October 25, 11 1
  2. 2. about me -Giuseppe Maxia § a.k.a. The Data Charmer § QA Director, Continuent, Inc § Long time hacking with database features § Formerly, database consultant, designer, coder. § A passion for QA § An even greater passion for open source § ... and community § Passionate blogger § http://datacharmer.blogspot.comTuesday, October 25, 11 2
  3. 3. Why Tungsten?Tuesday, October 25, 11 3
  4. 4. transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n MySQL replication tio ct tra tra REPLICATION transaction transaction is single threaded MySQL DBMSTuesday, October 25, 11 4
  5. 5. master master master master MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS slave slave slave slave single source multi source (fan-in) multiple sources?Tuesday, October 25, 11 5
  6. 6. from this to this MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master master multiple masters?Tuesday, October 25, 11 6
  7. 7. INSERT INSERT RECORD RECORD A A MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS MODIFY master master MODIFY RECORD RECORD B B Avoiding conflicts?Tuesday, October 25, 11 7
  8. 8. master Seamless failover? MySQL DBMS master MySQL DBMS master MySQL DBMS MySQL MySQL DBMS DBMS slave slave MySQL DBMS MySQL DBMS slave slave MySQL MySQL DBMS DBMS slave slaveTuesday, October 25, 11 8
  9. 9. Replicating to something else? mysql master MySQL DBMS MySQL DBMS mysql postgresql oracle mongodbTuesday, October 25, 11 9
  10. 10. All these examples tell us: Nice dream, but MySQL can’t do itTuesday, October 25, 11 10
  11. 11. Enter Tungsten ReplicatorTuesday, October 25, 11 11
  12. 12. http://code.google.com/p/tungsten-replicatorTuesday, October 25, 11 12
  13. 13. http://code.google.com/p/tungsten-replicator Open SourceTuesday, October 25, 11 12
  14. 14. http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2Tuesday, October 25, 11 12
  15. 15. What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Conflict prevention • Parallel replication • Replicate to Oracle and PostgreSQL databaseTuesday, October 25, 11 13
  16. 16. MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle or to NoSQL entities Bin Logs (Binlogs enabled) Slave DB © Continuent 2010Tuesday, October 25, 11 14
  17. 17. Tungsten Replicator in a nutshell host1 host2 master slave binlog THL global transaction ID THL trep_commit_seqno origin seqno eventid trep_commit_seqno origin seqno eventidTuesday, October 25, 11 15
  18. 18. Main components • Transaction History Logs (THL) • roughly corresponding to MySQL relay logs • have a lot of metadata • Service database • contains metadata for latest transactions • Metadata is committed together with data • Makes slaves crash proofTuesday, October 25, 11 16
  19. 19. A failover scenario 1: MySQL native replicationTuesday, October 25, 11 17
  20. 20. 1. one Master, two slaves / Loading the “employees” test database © Continuent 2011Tuesday, October 25, 11 18
  21. 21. 2. Master goes away / Stop replication / Slaves are updated at different levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,308 © Continuent 2011Tuesday, October 25, 11 19
  22. 22. 3. Look into Slave #2 binary logs / find the last transaction © Continuent 2011Tuesday, October 25, 11 20
  23. 23. 4. Look into Slave #3 binary logs 1. find the transaction that was last in slave #2 2. Recognize that last transaction in the log of slave #3 (This can actually take you a LOOOONG TIME) 3. Get the position immediately after this transaction 4. (e.g. 134000 in file mysql-bin.000018) © Continuent 2011Tuesday, October 25, 11 21
  24. 24. 5. promote Slave #3 to master / in slave #2 CHANGE MASTER TO master_host=‘slave_3_IP’, master_user=‘slavename’, master_password=‘slavepassword’, master_log_file=‘mysql-bin.000018’, master_log_pos=134000; © Continuent 2011Tuesday, October 25, 11 22
  25. 25. A failover scenario 1: Tungsten ReplicatorTuesday, October 25, 11 23
  26. 26. 1. one master, two slaves / loading the ‘employees’ test database © Continuent 2011Tuesday, October 25, 11 24
  27. 27. 2. Master goes away / Stop replication / Slaves are updated at different levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,308 © Continuent 2011Tuesday, October 25, 11 25
  28. 28. 3. no need to find the last transaction / simply change roles trepctl -host slave3 setrole -role master trepctl -host slave2 setrole -role slave -uri thl://slave3 trepctl -host slave3 online State: ONLINE trepctl -host slave2 online State: GOING-ONLINE:SYNCHRONIZING © Continuent 2011Tuesday, October 25, 11 26
  29. 29. 4. Check that the slave has synchronized # new master select seqno from tungsten.trep_commit_seqno; 78 # new slave select seqno from tungsten.trep_commit_seqno; 64 © Continuent 2011Tuesday, October 25, 11 27
  30. 30. 4. Tell the replicator to hurry up # new master trepctl -node slave3 flush Master log is synchronized with database at log sequence number: 78 # new slave trepctl host slave2 wait -applied 78 ONLINE select seqno from tungsten.trep_commit_seqno; 78 © Continuent 2011Tuesday, October 25, 11 28
  31. 31. 4. ... and we’re done # new master select count(*) from employees.titles count(*) 443308 # new slave: count(*) 443308 © Continuent 2011Tuesday, October 25, 11 29
  32. 32. Parallel replicationTuesday, October 25, 11 30
  33. 33. parallel replication From here ...Tuesday, October 25, 11 31
  34. 34. parallel replication To here.Tuesday, October 25, 11 32
  35. 35. Parallel replication facts ✓Sharded by database ✓Good choice for slave lag problems ❖Bad choice for single database projectsTuesday, October 25, 11 33
  36. 36. Benchmarking parallel replicationTuesday, October 25, 11 34
  37. 37. db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbenchTuesday, October 25, 11 35
  38. 38. before the test (2) STOPPED binary logs MySQL slave OFFLINE Tungsten slave direct: alpha (slave) replicator alphaTuesday, October 25, 11 36
  39. 39. starting the test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases running for 1 hour OFFLINE TOTAL DATA: 130 GB Tungsten slave direct: alpha RAM per server: 20GB (slave) replicator alpha Slaves will have 1 hour lagTuesday, October 25, 11 37
  40. 40. measuring results START binary logs MySQL slave ONLINE Recording catch-up time Tungsten slave direct: alpha (slave) replicator alphaTuesday, October 25, 11 38
  41. 41. MySQL native replication slave catch up in 04:29:30Tuesday, October 25, 11 39
  42. 42. Tungsten parallel replication slave catch up in 00:55:40Tuesday, October 25, 11 40
  43. 43. Parallel replication made simpler FROM HERE ....Tuesday, October 25, 11 41
  44. 44. Parallel replication made simpler TO HERETuesday, October 25, 11 42
  45. 45. Parallel replication made simplerTuesday, October 25, 11 43
  46. 46. parallel replication direct slave factsTuesday, October 25, 11 44
  47. 47. parallel replication direct slave facts ✓No need to install Tungsten on the masterTuesday, October 25, 11 44
  48. 48. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slaveTuesday, October 25, 11 44
  49. 49. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave)Tuesday, October 25, 11 44
  50. 50. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slavesTuesday, October 25, 11 44
  51. 51. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves ❖Failover (either native or Tungsten) becomes a manual taskTuesday, October 25, 11 44
  52. 52. Multiple mastersTuesday, October 25, 11 45
  53. 53. multiple masters • Tungsten Replicator recipe: use more servicesTuesday, October 25, 11 46
  54. 54. Bi-directional replicationTuesday, October 25, 11 47
  55. 55. Bi-directional replication with slavesTuesday, October 25, 11 48
  56. 56. Multiple sourcesTuesday, October 25, 11 49
  57. 57. multiple sources • Tungsten Replicator recipe is still valid: use more servicesTuesday, October 25, 11 50
  58. 58. Multiple source replicationTuesday, October 25, 11 51
  59. 59. Multiple masters replication: 3 nodesTuesday, October 25, 11 52
  60. 60. Multiple masters replication: 4 nodesTuesday, October 25, 11 53
  61. 61. Conflict preventionTuesday, October 25, 11 54
  62. 62. Conflict prevention factsTuesday, October 25, 11 55
  63. 63. Conflict prevention facts • Sharded by databaseTuesday, October 25, 11 55
  64. 64. Conflict prevention facts • Sharded by database • Defined dynamicallyTuesday, October 25, 11 55
  65. 65. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slaveTuesday, October 25, 11 55
  66. 66. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods:Tuesday, October 25, 11 55
  67. 67. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication failTuesday, October 25, 11 55
  68. 68. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silentlyTuesday, October 25, 11 55
  69. 69. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently • drop with warningTuesday, October 25, 11 55
  70. 70. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 56
  71. 71. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 56
  72. 72. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 56
  73. 73. Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 57
  74. 74. INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 57
  75. 75. INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 57
  76. 76. Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 58
  77. 77. INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 58
  78. 78. INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 58
  79. 79. Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M STuesday, October 25, 11 59
  80. 80. INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zTuesday, October 25, 11 59
  81. 81. INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zTuesday, October 25, 11 59
  82. 82. Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 60
  83. 83. INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M STuesday, October 25, 11 60
  84. 84. INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 60
  85. 85. Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,zTuesday, October 25, 11 60
  86. 86. Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M STuesday, October 25, 11 61
  87. 87. INSERT A x,y Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zTuesday, October 25, 11 61
  88. 88. Prevention methods: DROP on slave host1 (Multiple sources) M host2 INSERT A host3 x,y master INSERT A M x,z STuesday, October 25, 11 62
  89. 89. Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master INSERT A M x,z STuesday, October 25, 11 62
  90. 90. Tungsten in practice InstallationTuesday, October 25, 11 63
  91. 91. Installation • Check the requirements • Get the binaries • Expand the tarball • Run ./tools/tungsten-installerTuesday, October 25, 11 64
  92. 92. REQUIREMENTS • Java JRE or JDK (Sun/Oracle or Open-jdk) • Ruby 1.8 (only during installation) • ssh access to the same user in all nodes • MySQL user with all privilegesTuesday, October 25, 11 65
  93. 93. Installation types • master / slave • slave - directTuesday, October 25, 11 66
  94. 94. master/slave host2 host1 master slave THL binlog THL host3 slave THLTuesday, October 25, 11 67
  95. 95. slave direct host2 host1 slave master relay log THL binlog host3 slave relay log THLTuesday, October 25, 11 68
  96. 96. sample installation • 4 servers • qa.r1.continuent.com r1 • qa.r2.continuent.com r2 • qa.r3.continuent.com r3 • qa.r4.continuent.com r4 • MySQL 5.1 • tungsten-replicator-2.0.4Tuesday, October 25, 11 69
  97. 97. Installation (1) # starting at node 4, but any would do for N in 1 2 3 4 do ssh r$N mkdir tinstall done cd tinstall tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4Tuesday, October 25, 11 70
  98. 98. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall 1 ./tools/tungsten-installer 2 --master-slave 3 --master-host=r1 4 --datasource-user=tungsten 5 --datasource-password=secret 6 --service-name=dragon 7 --home-directory=$TUNGSTEN_BASE 8 --cluster-hosts=r1,r2,r3,r4 9 --startTuesday, October 25, 11 71
  99. 99. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall ./tools/tungsten-installer --master-slave # installation mode --master-host=r1 # who’s the master --datasource-user=tungsten # mysql username --datasource-password=secret # mysql password --service-name=dragon # name of the service --home-directory=$TUNGSTEN_BASE # where we install --cluster-hosts=r1,r2,r3,r4 # hosts in cluster --start # start replicator after installingTuesday, October 25, 11 72
  100. 100. What does the installation do 1: Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Report all errorsTuesday, October 25, 11 73
  101. 101. What does the installation do 1 (again): Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔Tuesday, October 25, 11 74
  102. 102. What does the installation do 2: install Tungsten in all servers host4 $HOME/ host1 tinstall/ host2 config/ host3 releases/ relay/ logs/ tungsten/Tuesday, October 25, 11 75
  103. 103. example ssh r2 chmod 444 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeableTuesday, October 25, 11 76
  104. 104. example ssh r2 chmod 755 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start # no errorsTuesday, October 25, 11 77
  105. 105. Installation - direct ./tools/tungsten-installer --direct --master-host=r1 --slave-host=r4 --master-user=tungsten --slave-user=tungsten --master-password=secret --slave-password=secret --service-name=Castor --channels=1 --home-directory=$TUNGSTEN_BASE --start-and-reportTuesday, October 25, 11 78
  106. 106. Tungsten in practice AdminTuesday, October 25, 11 79
  107. 107. Main components • THL (Tungsten generated Transaction History Log) • service databaseTuesday, October 25, 11 80
  108. 108. Tools • replicator • trepctl • thlTuesday, October 25, 11 81
  109. 109. replicator • It’s the service provider • You launch it once when you start • You may restart it when you change configTuesday, October 25, 11 82
  110. 110. trepctl • Tungsten Replicator ConTroLler • It’s the driving seat for your replication • You can start, update, and stop services • You can get specific infoTuesday, October 25, 11 83
  111. 111. thl • Transaction History List • Gives you access to the Tungsten relay logsTuesday, October 25, 11 84
  112. 112. http://bit.ly/tr20_cookbookTuesday, October 25, 11 85
  113. 113. Tungsten Replicator vs. Enterprise Feature Replicator Enterprise Simple admin with trepctl X X MySQL Row/Statement 5.0/5.1/5.5 X X Multi-master replication X X Parallel apply (2.0) X X Distributed management with cctrl X Automated failover and recovery X Zero-downtime upgrade X SQL routing and load-balancing X Floating IP support X © Continuent 2011Tuesday, October 25, 11 86
  114. 114. Conclusion and Q&ATuesday, October 25, 11 87
  115. 115. Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 @datacharmer Tel (866) 998-3642 Fax (408) 668-1009 @continuent e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://tungsten-replicator.org Getting started with Tungsten Replicator: http://datacharmer.blogspot.com © Continuent 2011Tuesday, October 25, 11 88
  116. 116. Tuesday, October 25, 11 89

×