Successfully reported this slideshow.

Moving data for the masses

2,774 views

Published on

How data replication has changed the world, and how you can move your data dynamically using Tungsten Replicator

Published in: Technology
  • Be the first to comment

Moving data for the masses

  1. 1. MOVING DATA FOR THE MASSESGiuseppe MaxiaQA DirectorContinuent, Inc @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.Friday, November 11, 11 1
  2. 2. Databases are the backbone of the web economy www $$$Friday, November 11, 11 2
  3. 3. Actually, database replication is the backbone of the web economyFriday, November 11, 11 3
  4. 4. Actually, database replication is the backbone of the web economy WHY?Friday, November 11, 11 3
  5. 5. Why moving data? • Scaling reads (load balancing) • Scaling writes (sharding) • Parallelize computation (sharding+map/reduce) • High availability (replacing master on failure) • Creating information (feeding data warehouses)Friday, November 11, 11 4
  6. 6. Scaling reads M S S SFriday, November 11, 11 5
  7. 7. Scaling writes (sharding) M1 M3 M2 S S S S S SFriday, November 11, 11 6
  8. 8. High availability M S S SFriday, November 11, 11 7
  9. 9. High availability S ✘ M S SFriday, November 11, 11 7
  10. 10. High availability S S MFriday, November 11, 11 8
  11. 11. Creating new information S M M MFriday, November 11, 11 9
  12. 12. Creating new information S S M S M M MFriday, November 11, 11 9
  13. 13. More about data replicationFriday, November 11, 11 10
  14. 14. database server r/w requests web server clientsFriday, November 11, 11 11
  15. 15. database server r/w requests web servers load balancer clientsFriday, November 11, 11 12
  16. 16. database load on a simple web application r e write a dFriday, November 11, 11 13
  17. 17. write read database load on a successful web applicationFriday, November 11, 11 14
  18. 18. database server ✘ r/w requests web servers load balancer clientsFriday, November 11, 11 15
  19. 19. read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clientsFriday, November 11, 11 16
  20. 20. r e write a read d read/write master read/only slaves database load with replicationFriday, November 11, 11 17
  21. 21. r e write a read d read/write master read/only slaves scaling database load with replicationFriday, November 11, 11 18
  22. 22. • If you want a successful companyFriday, November 11, 11 19
  23. 23. • If you want a successful company • in the 21st centuryFriday, November 11, 11 19
  24. 24. • If you want a successful company • in the 21st century • you need to move dataFriday, November 11, 11 19
  25. 25. • If you want a successful company • in the 21st century • you need to move data • fastFriday, November 11, 11 19
  26. 26. • If you want a successful company • in the 21st century • you need to move data • fast • and oftenFriday, November 11, 11 19
  27. 27. MOVING DATA FOR THE MASSESFriday, November 11, 11 20
  28. 28. How do you move data?Friday, November 11, 11 21
  29. 29. How do you move data? • Commodity hardwareFriday, November 11, 11 21
  30. 30. How do you move data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replicationFriday, November 11, 11 21
  31. 31. How do you move data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replication • Advanced • MySQL and other DBMS • Powerful tools (Tungsten Replicator)Friday, November 11, 11 21
  32. 32. Why Tungsten?Friday, November 11, 11 22
  33. 33. transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n MySQL replication tio ct tra tra REPLICATION transaction transaction is single threaded MySQL DBMSFriday, November 11, 11 23
  34. 34. master master master master MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS slave slave slave slave single source multi source (fan-in) multiple sources?Friday, November 11, 11 24
  35. 35. from this to this MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master master multiple masters?Friday, November 11, 11 25
  36. 36. INSERT INSERT RECORD RECORD A A MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS MODIFY master master MODIFY RECORD RECORD B B Avoiding conflicts?Friday, November 11, 11 26
  37. 37. master Seamless failover? MySQL DBMS master MySQL DBMS master MySQL DBMS MySQL MySQL DBMS DBMS slave slave MySQL DBMS MySQL DBMS slave slave MySQL MySQL DBMS DBMS slave slaveFriday, November 11, 11 27
  38. 38. Replicating to something else? mysql master MySQL DBMS MySQL DBMS mysql postgresql oracle mongodbFriday, November 11, 11 28
  39. 39. All these examples tell us: Nice dream, but MySQL can’t do itFriday, November 11, 11 29
  40. 40. Enter Tungsten ReplicatorFriday, November 11, 11 30
  41. 41. http://code.google.com/p/tungsten-replicatorFriday, November 11, 11 31
  42. 42. http://code.google.com/p/tungsten-replicator Open SourceFriday, November 11, 11 31
  43. 43. http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2Friday, November 11, 11 31
  44. 44. What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Conflict prevention • Parallel replication • Replicate to Oracle and PostgreSQL databaseFriday, November 11, 11 32
  45. 45. MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle or to NoSQL entities Bin Logs (Binlogs enabled) Slave DB © Continuent 2010Friday, November 11, 11 33
  46. 46. Main components • Transaction History Logs (THL) • roughly corresponding to MySQL relay logs • have a lot of metadata • Service database • contains metadata for latest transactions • Metadata is committed together with data • Makes slaves crash proofFriday, November 11, 11 34
  47. 47. A failover scenario 1: MySQL native replicationFriday, November 11, 11 35
  48. 48. Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Friday, November 11, 11 36
  49. 49. The master crashes Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ========Friday, November 11, 11 37
  50. 50. Where do the slaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary log Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 38
  51. 51. Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 39
  52. 52. Need to synch the missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 40
  53. 53. Transaction position in the new master binlog may not match positions in the old master binlog New master Slave 1 Slave 2 Binary log Transactions ? ? ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ? ======== ? ======== ========Friday, November 11, 11 41
  54. 54. A failover scenario 1: Tungsten ReplicatorFriday, November 11, 11 42
  55. 55. Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ======== Includes Global Transaction IDFriday, November 11, 11 43
  56. 56. The master crashes Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ========Friday, November 11, 11 44
  57. 57. Where do the slaves stand? Slave 1 Slave 2 Slave 3 THL THL THL Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 45
  58. 58. Slave 3 becomes master New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 46
  59. 59. Need to synch the missing transactions New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ========Friday, November 11, 11 47
  60. 60. Transaction IDs in the new master THL are understood immediately by the slaves New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== Give me ======== Give me ======== seqno #51 ======== seqno #67Friday, November 11, 11 48
  61. 61. Parallel replicationFriday, November 11, 11 49
  62. 62. parallel replication From here ...Friday, November 11, 11 50
  63. 63. parallel replication To here.Friday, November 11, 11 51
  64. 64. Parallel replication facts ✓Sharded by database ✓Good choice for slave lag problems ❖Bad choice for single database projectsFriday, November 11, 11 52
  65. 65. Benchmarking parallel replicationFriday, November 11, 11 53
  66. 66. db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbenchFriday, November 11, 11 54
  67. 67. before the test (2) STOPPED binary logs MySQL slave OFFLINE Tungsten slave direct: alpha (slave) replicator alphaFriday, November 11, 11 55
  68. 68. starting the test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases running for 1 hour OFFLINE TOTAL DATA: 130 GB Tungsten slave direct: alpha RAM per server: 20GB (slave) replicator alpha Slaves will have 1 hour lagFriday, November 11, 11 56
  69. 69. measuring results START binary logs MySQL slave ONLINE Recording catch-up time Tungsten slave direct: alpha (slave) replicator alphaFriday, November 11, 11 57
  70. 70. MySQL native replication slave catch up in 04:29:30Friday, November 11, 11 58
  71. 71. Tungsten parallel replication slave catch up in 00:55:40Friday, November 11, 11 59
  72. 72. Parallel replication made simpler FROM HERE ....Friday, November 11, 11 60
  73. 73. Parallel replication made simpler TO HEREFriday, November 11, 11 61
  74. 74. Parallel replication made simplerFriday, November 11, 11 62
  75. 75. parallel replication direct slave factsFriday, November 11, 11 63
  76. 76. parallel replication direct slave facts ✓No need to install Tungsten on the masterFriday, November 11, 11 63
  77. 77. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slaveFriday, November 11, 11 63
  78. 78. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave)Friday, November 11, 11 63
  79. 79. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slavesFriday, November 11, 11 63
  80. 80. parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves ❖Failover (either native or Tungsten) becomes a manual taskFriday, November 11, 11 63
  81. 81. Multiple mastersFriday, November 11, 11 64
  82. 82. multiple masters • Tungsten Replicator recipe: use more servicesFriday, November 11, 11 65
  83. 83. Bi-directional replicationFriday, November 11, 11 66
  84. 84. Bi-directional replication with slavesFriday, November 11, 11 67
  85. 85. Multiple sourcesFriday, November 11, 11 68
  86. 86. multiple sources • Tungsten Replicator recipe is still valid: use more servicesFriday, November 11, 11 69
  87. 87. Multiple source replicationFriday, November 11, 11 70
  88. 88. Multiple masters replication: 3 nodesFriday, November 11, 11 71
  89. 89. Multiple masters replication: 4 nodesFriday, November 11, 11 72
  90. 90. 4 nodes multiple masters simplifiedFriday, November 11, 11 73
  91. 91. star topologiesFriday, November 11, 11 74
  92. 92. Conflict preventionFriday, November 11, 11 75
  93. 93. Conflict prevention factsFriday, November 11, 11 76
  94. 94. Conflict prevention facts • Sharded by databaseFriday, November 11, 11 76
  95. 95. Conflict prevention facts • Sharded by database • Defined dynamicallyFriday, November 11, 11 76
  96. 96. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slaveFriday, November 11, 11 76
  97. 97. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods:Friday, November 11, 11 76
  98. 98. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication failFriday, November 11, 11 76
  99. 99. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silentlyFriday, November 11, 11 76
  100. 100. Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently • drop with warningFriday, November 11, 11 76
  101. 101. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 77
  102. 102. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 77
  103. 103. Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 77
  104. 104. Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 78
  105. 105. INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 78
  106. 106. INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 78
  107. 107. Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 79
  108. 108. INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 79
  109. 109. INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 79
  110. 110. Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M SFriday, November 11, 11 80
  111. 111. INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zFriday, November 11, 11 80
  112. 112. INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zFriday, November 11, 11 80
  113. 113. Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 81
  114. 114. INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M SFriday, November 11, 11 81
  115. 115. INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 81
  116. 116. Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,zFriday, November 11, 11 81
  117. 117. Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M SFriday, November 11, 11 82
  118. 118. INSERT A x,y Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,zFriday, November 11, 11 82
  119. 119. Prevention methods: DROP on slave host1 (Multiple sources) M host2 INSERT A host3 x,y master INSERT A M x,z SFriday, November 11, 11 83
  120. 120. Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master INSERT A M x,z SFriday, November 11, 11 83
  121. 121. Tungsten in practice InstallationFriday, November 11, 11 84
  122. 122. Installation • Check the requirements • Get the binaries • Expand the tarball • Run ./tools/tungsten-installerFriday, November 11, 11 85
  123. 123. REQUIREMENTS • Java JRE or JDK (Sun/Oracle or Open-jdk) • Ruby 1.8 (only during installation) • ssh access to the same user in all nodes • MySQL user with all privilegesFriday, November 11, 11 86
  124. 124. Installation types • master / slave • slave - directFriday, November 11, 11 87
  125. 125. master/slave host2 host1 master slave THL binlog THL host3 slave THLFriday, November 11, 11 88
  126. 126. slave direct host2 host1 slave master relay log THL binlog host3 slave relay log THLFriday, November 11, 11 89
  127. 127. sample installation • 4 servers • qa.r1.continuent.com r1 • qa.r2.continuent.com r2 • qa.r3.continuent.com r3 • qa.r4.continuent.com r4 • MySQL 5.1 • tungsten-replicator-2.0.4Friday, November 11, 11 90
  128. 128. Installation (1) # starting at node 4, but any would do for N in 1 2 3 4 do ssh r$N mkdir tinstall done cd tinstall tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4Friday, November 11, 11 91
  129. 129. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall 1 ./tools/tungsten-installer 2 --master-slave 3 --master-host=r1 4 --datasource-user=tungsten 5 --datasource-password=secret 6 --service-name=dragon 7 --home-directory=$TUNGSTEN_BASE 8 --cluster-hosts=r1,r2,r3,r4 9 --startFriday, November 11, 11 92
  130. 130. Installation (2) export TUNGSTEN_BASE=$HOME/tinstall ./tools/tungsten-installer --master-slave # installation mode --master-host=r1 # who’s the master --datasource-user=tungsten # mysql username --datasource-password=secret # mysql password --service-name=dragon # name of the service --home-directory=$TUNGSTEN_BASE # where we install --cluster-hosts=r1,r2,r3,r4 # hosts in cluster --start # start replicator after installingFriday, November 11, 11 93
  131. 131. What does the installation do 1: Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Report all errorsFriday, November 11, 11 94
  132. 132. What does the installation do 1 (again): Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔Friday, November 11, 11 95
  133. 133. What does the installation do 2: install Tungsten in all servers host4 $HOME/ host1 tinstall/ host2 config/ host3 releases/ relay/ logs/ tungsten/Friday, November 11, 11 96
  134. 134. example ssh r2 chmod 444 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeableFriday, November 11, 11 97
  135. 135. example ssh r2 chmod 755 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start # no errorsFriday, November 11, 11 98
  136. 136. Installation - direct ./tools/tungsten-installer --direct --master-host=r1 --slave-host=r4 --master-user=tungsten --slave-user=tungsten --master-password=secret --slave-password=secret --service-name=Castor --channels=1 --home-directory=$TUNGSTEN_BASE --start-and-reportFriday, November 11, 11 99
  137. 137. http://bit.ly/tr20_cookbookFriday, November 11, 11 100
  138. 138. Tungsten Replicator vs. Enterprise Feature Replicator Enterprise Simple admin with trepctl X X MySQL Row/Statement 5.0/5.1/5.5 X X Multi-master replication X X Parallel apply (2.0) X X Distributed management with cctrl X Automated failover and recovery X Zero-downtime upgrade X SQL routing and load-balancing X Floating IP support X © Continuent 2011Friday, November 11, 11 101
  139. 139. Conclusion and Q&AFriday, November 11, 11 102
  140. 140. Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 @datacharmer Tel (866) 998-3642 Fax (408) 668-1009 @continuent e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://tungsten-replicator.org Getting started with Tungsten Replicator: http://datacharmer.blogspot.com © Continuent 2011Friday, November 11, 11 103
  141. 141. Friday, November 11, 11 104

×