Jailbreaking mysql replication

  • 1,809 views
Uploaded on

Working around MySQL replication shortcoming, such as lack of parallel apply, multiple masters, multi source slaves, seamless failover, using Tungsten Replicator.

Working around MySQL replication shortcoming, such as lack of parallel apply, multiple masters, multi source slaves, seamless failover, using Tungsten Replicator.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,809
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
54
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Jailbreaking MySQL replication Giuseppe Maxia QA Director Continuent, Inc @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.Saturday, October 1, 11
  • 2. about me -Giuseppe Maxia § a.k.a. The Data Charmer § QA Director, Continuent, Inc § Long time hacking with database features § Formerly, database consultant, designer, coder. § A passion for QA § An even greater passion for open source § ... and community § Passionate blogger § http://datacharmer.blogspot.comSaturday, October 1, 11
  • 3. In jail?Saturday, October 1, 11
  • 4. transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n MySQL replication tio ct tra tra REPLICATION transaction transaction is single threaded MySQL DBMSSaturday, October 1, 11
  • 5. master master master master MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS slave slave slave slave single source multi source (fan-in) multiple sources?Saturday, October 1, 11
  • 6. from this from this MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master master multiple masters?Saturday, October 1, 11
  • 7. master Seamless failover? MySQL DBMS master MySQL DBMS master MySQL DBMS MySQL MySQL DBMS DBMS slave slave MySQL DBMS MySQL DBMS slave slave MySQL MySQL DBMS DBMS slave slaveSaturday, October 1, 11
  • 8. All these examples tell us: Nice dream, but MySQL can’t do itSaturday, October 1, 11
  • 9. Enter Tungsten ReplicatorSaturday, October 1, 11
  • 10. Tungsten Replicator 2.0 • What is it?Saturday, October 1, 11
  • 11. http://code.google.com/p/tungsten-replicatorSaturday, October 1, 11
  • 12. http://code.google.com/p/tungsten-replicator Open SourceSaturday, October 1, 11
  • 13. http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2Saturday, October 1, 11
  • 14. What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Parallel replication • Replicate to Oracle and PostgreSQL databaseSaturday, October 1, 11
  • 15. MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle or to NoSQL entities Bin Logs (Binlogs enabled) Slave DB © Continuent 2010Saturday, October 1, 11
  • 16. MySQL to Oracle replication / Today at 2:45pm / Moscone West 2016 / Full session about MySQL to Oracle replication / with demos © Continuent 2010Saturday, October 1, 11
  • 17. Tungsten Replicator in a nutshell host1 host2 master slave binlog THL global transaction ID THL trep_commit_seqno origin seqno eventid trep_commit_seqno origin seqno eventidSaturday, October 1, 11
  • 18. Main components • Transaction History Logs (THL) • roughly corresponding to MySQL relay logs • have a lot of metadata • Service database • contains metadata for latest transactions • Metadata is committed together with data • Makes slaves crash proofSaturday, October 1, 11
  • 19. A failover scenario 1: MySQL native replicationSaturday, October 1, 11
  • 20. 1. one Master, two slaves / Loading the “employees” test database © Continuent 2011Saturday, October 1, 11
  • 21. 2. Master goes away / Stop replication / Slaves are updated at different levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,308 © Continuent 2011Saturday, October 1, 11
  • 22. 3. Look into Slave #2 binary logs / find the last transaction © Continuent 2011Saturday, October 1, 11
  • 23. 4. Look into Slave #3 binary logs / find the transaction that was last in slave #2 / (This can actually take you a LOOOONG TIME) / Get the position immediately after this transaction / (e.g. 134000 in file mysql-bin.000018) © Continuent 2011Saturday, October 1, 11
  • 24. 5. promote Slave #3 to master / in slave #2 CHANGE MASTER TO master_host=‘slave_3_IP’, master_user=‘slavename’, master_password=‘slavepassword’, master_log_file=‘mysql-bin.000018’, master_log_pos=134000; © Continuent 2011Saturday, October 1, 11
  • 25. A failover scenario 1: Tungsten ReplicatorSaturday, October 1, 11
  • 26. 1. one master, two slaves / loading the ‘employees’ test database © Continuent 2011Saturday, October 1, 11
  • 27. 2. Master goes away / Stop replication / Slaves are updated at different levels # 2 select count(*) from titles 333,145 # 3 select count(*) from titles 443,308 © Continuent 2011Saturday, October 1, 11
  • 28. 3. no need to find the last transaction / simply change roles trepctl -host slave3 setrole -role master trepctl -host slave2 setrole -role slave -uri thl://slave3 trepctl -host slave3 online State: ONLINE trepctl -host slave2 online State: GOING-ONLINE:SYNCHRONIZING © Continuent 2011Saturday, October 1, 11
  • 29. 4. Check that the slave has synchronized # new master select seqno from tungsten.trep_commit_seqno; 78 # new slave select seqno from tungsten.trep_commit_seqno; 64 © Continuent 2011Saturday, October 1, 11
  • 30. 4. Tell the replicator to hurry up # new master trepctl -node slave3 flush Master log is synchronized with database at log sequence number: 78 # new slave trepctl host slave2 wait -applied 78 ONLINE select seqno from tungsten.trep_commit_seqno; 78 © Continuent 2011Saturday, October 1, 11
  • 31. 4. ... and we’re done # new master select count(*) from employees.titles count(*) 443308 # new slave: count(*) 443308 © Continuent 2011Saturday, October 1, 11
  • 32. Failover • DEMOSaturday, October 1, 11
  • 33. master/slave with an attitudeSaturday, October 1, 11
  • 34. Parallel replicationSaturday, October 1, 11
  • 35. parallel replication From here ...Saturday, October 1, 11
  • 36. parallel replication To here.Saturday, October 1, 11
  • 37. Parallel replication facts • Sharded by database • Good choice for slave lag problems • Bad choice for single database projectsSaturday, October 1, 11
  • 38. Testing parallel replicationSaturday, October 1, 11
  • 39. db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbenchSaturday, October 1, 11
  • 40. before the test (2) STOPPED binary logs MySQL slave OFFLINE Tungsten slave direct: alpha (slave) replicator alphaSaturday, October 1, 11
  • 41. starting the test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases OFFLINE running for 1 hour Tungsten slave direct: alpha (slave) Slaves will have 1 replicator alpha hour lagSaturday, October 1, 11
  • 42. measuring results START binary logs MySQL slave ONLINE Recording catch-up time Tungsten slave direct: alpha (slave) replicator alphaSaturday, October 1, 11
  • 43. MySQL native replication slave catch up in 00:59:30Saturday, October 1, 11
  • 44. Tungsten parallel replication slave catch up in 00:25:40Saturday, October 1, 11
  • 45. Parallel replication made simpler FROM HERE ....Saturday, October 1, 11
  • 46. Parallel replication made simpler TO HERESaturday, October 1, 11
  • 47. Parallel replication made simplerSaturday, October 1, 11
  • 48. parallel replication direct slave factsSaturday, October 1, 11
  • 49. parallel replication direct slave facts • No need to install Tungsten on the masterSaturday, October 1, 11
  • 50. parallel replication direct slave facts • No need to install Tungsten on the master • Tungsten runs only on the slaveSaturday, October 1, 11
  • 51. parallel replication direct slave facts • No need to install Tungsten on the master • Tungsten runs only on the slave • Replication can revert to native slave with two commands (trepctl offline; start slave)Saturday, October 1, 11
  • 52. parallel replication direct slave facts • No need to install Tungsten on the master • Tungsten runs only on the slave • Replication can revert to native slave with two commands (trepctl offline; start slave) • Native replication can continue on other slavesSaturday, October 1, 11
  • 53. parallel replication direct slave facts • No need to install Tungsten on the master • Tungsten runs only on the slave • Replication can revert to native slave with two commands (trepctl offline; start slave) • Native replication can continue on other slaves • Failover (either native or Tungsten) becomes a manual taskSaturday, October 1, 11
  • 54. Multiple mastersSaturday, October 1, 11
  • 55. multiple masters • Tungsten Replicator recipe: use more servicesSaturday, October 1, 11
  • 56. Bi-directional replicationSaturday, October 1, 11
  • 57. Bi-directional replication with slavesSaturday, October 1, 11
  • 58. Multiple sourcesSaturday, October 1, 11
  • 59. multiple sources • Tungsten Replicator recipe is still valid: use more servicesSaturday, October 1, 11
  • 60. Multiple source replicationSaturday, October 1, 11
  • 61. Multiple masters replication: 3 nodesSaturday, October 1, 11
  • 62. Multiple masters replication: 4 nodesSaturday, October 1, 11
  • 63. Tungsten in practice InstallationSaturday, October 1, 11
  • 64. Installation • Check the requirements • Get the binaries • Expand the tarball • Run ./tools/tungsten-installerSaturday, October 1, 11
  • 65. REQUIREMENTS • Java JRE or JDK (Sun/Oracle or Open-jdk) • Ruby 1.8 (only during installation) • ssh access to the same user in all nodes • sudo access in all nodes (only for some backups) • MySQL user with all privilegesSaturday, October 1, 11
  • 66. Installation types • multiple hosts • master / slave • slave - direct • sandbox • master/slave • slave-directSaturday, October 1, 11
  • 67. master/slave host2 host1 master slave THL binlog THL host3 slave THLSaturday, October 1, 11
  • 68. master/slave - sandbox host1 slave master binlog THL THL slave THLSaturday, October 1, 11
  • 69. slave direct host2 host1 slave master relay log THL binlog host3 slave relay log THLSaturday, October 1, 11
  • 70. slave direct - sandbox host1 slave master relay log binlog THL slave relay log THLSaturday, October 1, 11
  • 71. sample installation • 4 servers • qa.r1.continuent.com r1 • qa.r2.continuent.com r2 • qa.r3.continuent.com r3 • qa.r4.continuent.com r4 • MySQL 5.1 • tungsten-replicator-2.0.4Saturday, October 1, 11
  • 72. Installation (1) # starting at node 4, but any would do for N in 1 2 3 4 do ssh r$N mkdir tinstall done cd tinstall tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4Saturday, October 1, 11
  • 73. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall 1 ./tools/tungsten-installer 2 --master-slave 3 --master-host=r1 4 --datasource-user=tungsten 5 --datasource-password=secret 6 --service-name=dragon 7 --home-directory=$TUNGSTEN_BASE 8 --cluster-hosts=r1,r2,r3,r4 9 --startSaturday, October 1, 11
  • 74. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall ./tools/tungsten-installer --master-slave # installation mode --master-host=r1 # who’s the master --datasource-user=tungsten # mysql username --datasource-password=secret # mysql password --service-name=dragon # name of the service --home-directory=$TUNGSTEN_BASE # where we install --cluster-hosts=r1,r2,r3,r4 # hosts in cluster --start # start replicator after installingSaturday, October 1, 11
  • 75. What does the installation do 1: Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Report all errorsSaturday, October 1, 11
  • 76. What does the installation do 1 (again): Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔Saturday, October 1, 11
  • 77. What does the installation do 2: install Tungsten in all servers host4 $HOME/ host1 tinstall/ host2 config/ host3 releases/ relay/ logs/ tungsten/Saturday, October 1, 11
  • 78. example ssh r2 chmod 444 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeableSaturday, October 1, 11
  • 79. example ssh r2 chmod 755 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start # no errorsSaturday, October 1, 11
  • 80. Installation - direct ./tools/tungsten-installer --direct --master-host=r1 --slave-host=r4 --master-user=tungsten --slave-user=tungsten --master-password=secret --slave-password=secret --service-name=Castor --channels=1 --home-directory=$TUNGSTEN_BASE --start-and-reportSaturday, October 1, 11
  • 81. MySQL Sandbox http://mysqlsandbox.net • Free software (GPL) • One (unix) host • Many database servers • Single or multiple sandboxes • Customized scripts to use the servers • Standard or circular replication • Installs IN SECONDSSaturday, October 1, 11
  • 82. installation - direct - sandbox ./tools/tungsten-installer --direct --master-host=127.0.0.1 --master-port=7101 --slave-host=127.0.0.1 --slave-port=7102 --master-user=msandbox --slave-user=msandbox --master-password=msandbox --slave-password=msandbox --master-log-directory=$SANDBOX/tr_dbs/node1/data --service-name=Castor --channels=5 --home-directory=$TUNGSTEN_BASE --startSaturday, October 1, 11
  • 83. installation - master/slave - sandbox ./tools/tungsten-installer --master-slave --cluster-hosts=127.0.0.1 --master-host=127.0.0.1 --datasource-port=7001 --datasource-user=msandbox --datasource-password=msandbox --home-directory=$TUNGSTEN_BASE/db1 --datasource-log-directory=$SBOX/tr/node1/data --service-name=tsandbox ...Saturday, October 1, 11
  • 84. installation - master/slave - sandbox $ ./tungsten-sandbox -h USAGE: ./tungsten-sandbox [flags] args flags: -n,--nodes: how many nodes to install (default: 3) -m,--mysql_version: which MySQL version to use (default: 5.1.56) -t,--tungsten_base: where to install the sandbox (default: $HOME/tsb2) -d,--group_dir: sandbox group directory name (default: tr_dbs) -s,--service: how the service is named (default: tsandbox) -P,--base_port: port base for MySQL sandbox nodes (default: 710) -l,--thl_port: port for the THL service (default: 1211) -r,--rmi_port: port for the RMI service (default: 1010) -v,--[no]version: show Tungsten sandbox version (default: false) -h,--[no]help: show this help (default: false)Saturday, October 1, 11
  • 85. installation - master/slave - sandbox tungsten-sandbox.sh --nodes=3 --mysql_version=5.1.57 # Will install 3 nodes (1 master and two slaves) # with Tungsten replicator, using MySQL 5.1.57 # It will also create utility scriptsSaturday, October 1, 11
  • 86. an installed tungsten sandbox inside $HOME/tsb2 # MySQL sandbox commands send_kill_all start_all status_all stop_all clear_all use_all n1 n2 n3 # invoke node 1, 2, and 3 # home of the replicators db1 db2 db3 # runs replicator [args] for all nodes replicator_all # runs trepctl [args] for all nodes trepctl_allSaturday, October 1, 11
  • 87. inside a single sandbox node inside $HOME/tsb2/db1 replicator # shortcut to the replicator thl # shortcut to the thl command trepctl # shortcut to the trepctl command show_log # shows the replicator logSaturday, October 1, 11
  • 88. Tungsten in practice AdminSaturday, October 1, 11
  • 89. Main components • THL (Tungsten generated Transaction History Log) • service databaseSaturday, October 1, 11
  • 90. Tools • replicator • trepctl • thlSaturday, October 1, 11
  • 91. replicator • It’s the service provider • You launch it once when you start • You may restart it when you change configSaturday, October 1, 11
  • 92. trepctl • Tungsten Replicator ConTroLler • It’s the driving seat for your replication • You can start, update, and stop services • You can get specific infoSaturday, October 1, 11
  • 93. thl • Transaction History List • Gives you access to the Tungsten relay logsSaturday, October 1, 11
  • 94. http://bit.ly/tr20_cookbookSaturday, October 1, 11
  • 95. Tungsten Replicator vs. Enterprise Feature Replicator Enterprise Simple admin with trepctl X X MySQL Row/Statement 5.0/5.1/5.5 X X Multi-master replication X X Parallel apply (2.0) X X Distributed management with cctrl X Automated failover and recovery X Zero-downtime upgrade X SQL routing and load-balancing X Floating IP support X © Continuent 2011Saturday, October 1, 11
  • 96. Conclusion and Q&ASaturday, October 1, 11
  • 97. Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://tungsten-replicator.org Getting started with Tungsten Replicator: http://datacharmer.blogspot.com © Continuent 2011Saturday, October 1, 11