Juggle your data with Tungsten Replicator

856 views
758 views

Published on

Juggle your data in different replication topologies, using Tungsten replicator. Make all-masters, fan-in, star topologies, and more complex ones

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
856
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
9
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Juggle your data with Tungsten Replicator

  1. 1. ©Continuent 2013 Juggle your data with Tungsten Replicator Giuseppe Maxia, Director of QA Continuent, Inc 1Sunday, September 22, 13
  2. 2. ©Continuent 2013 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement 2Sunday, September 22, 13
  3. 3. ©Continuent 2013 A Review of Tungsten Replicator 3 3Sunday, September 22, 13
  4. 4. ©Continuent 2013 Tungsten Replicator Overview 4 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC 4Sunday, September 22, 13
  5. 5. ©Continuent 2013 Master Replication Service 5 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip 5Sunday, September 22, 13
  6. 6. ©Continuent 2013 Slave Replication Service 6 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip 6Sunday, September 22, 13
  7. 7. ©Continuent 2013 star master-slave Heterogeneous fan-in slave all-masters MySQL Oracle Oracle MySQL Oracle Oracle MySQL MySQL 7Sunday, September 22, 13
  8. 8. ©Continuent 2013 Tungsten Replicator 2.1.1 8 8Sunday, September 22, 13
  9. 9. ©Continuent 2013 Tungsten Replicator 2.1.2 • Releases as GA • Main features: • installation with tpm • security layer • full support for MySQL 5.6 • improved management tools • trepctl properties • trepctl services [-full] [-json] • trepctl clients • trepctl status [-json] • thl list [-headers] [-json] 9 9Sunday, September 22, 13
  10. 10. ©Continuent 2013 Working with Tungsten 10 10Sunday, September 22, 13
  11. 11. ©Continuent 2013 Overview of Installation Process 1. Set up hosts 2. Prepare MySQL replicas 3. Download software 4. Install using tungsten-installer & con!gure- service tpm 11 Amazon Setup: https://docs.continuent.com/wiki/display/TEDOC/ Preparing+EC2+Servers 11Sunday, September 22, 13
  12. 12. ©Continuent 2013 How tungsten-installer Works for Basic Master/Slave Deployment 12 db1 db2 db3 Staging copy of files check prereqs copy code configure 12Sunday, September 22, 13
  13. 13. ©Continuent 2013 Tungsten master/slave replication 13 host1 host2 host3 alpha alpha alpha THL THL THL installer Old installation and New installation 13Sunday, September 22, 13
  14. 14. ©Continuent 2013 Bi-directional replication 14 host1 host2 alpha alpha bravo bravo Install master on host1 installer install master on host2 installer install slave service on host1 installer install slave service on host2 installer OLD INSTALLATION 14Sunday, September 22, 13
  15. 15. ©Continuent 2013 Bi-directional replication 15 host1 host2 alpha alpha bravo bravo NEW INSTALLATION Install all master and slave services on all hosts at once installer 15Sunday, September 22, 13
  16. 16. ©Continuent 2013 16 ./tools/tungsten-installer --master-slave --master-host=$MASTER2 --datasource-user=tungsten --datasource-password=secret --service-name=bravo --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER2 --start $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER1 --datasource=$MASTER1 --local-service-name=alpha --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER2 --master-thl-port=2112 --svc-start bravo ./tools/tungsten-installer --master-slave --master-host=$MASTER1 --datasource-user=tungsten --datasource-password=secret --service-name=alpha --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER1 --start $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER2 --datasource=$MASTER2 --local-service-name=bravo --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER1 --master-thl-port=2112 --svc-start alpha 1 2 3 4 Multi-master installation (2 nodes) until TR 2.1.0 ~ 3 minutes using tungsten-installer 16Sunday, September 22, 13
  17. 17. ©Continuent 2013 4 nodes all-masters 17 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha 17Sunday, September 22, 13
  18. 18. ©Continuent 2013 18 Multi-master installation (4 nodes) until TR 2.1.0 ~ 8 minutes tungsten-installer host1 - alpha 1 tungsten-installer host2 - bravo 2 tungsten-installer host3 - charlie 3 tungsten-installer host4 - delta 4 con!gure-service host1 - bravo 5 con!gure-service host1 - charlie 6 con!gure-service host1 - delta 7 con!gure-service host2 - alpha 8 con!gure-service host2 - charlie 9 con!gure-service host2 - delta 10 con!gure-service host3 - alpha 11 con!gure-service host3 - bravo 12 con!gure-service host3 - delta 13 con!gure-service host4 - alpha 14 con!gure-service host4 - bravo 15 con!gure-service host4 - charlie 16 using tungsten-installer 18Sunday, September 22, 13
  19. 19. ©Continuent 2013 19 ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start Multi-master installation (2 nodes) with TR 2.1.1 ~ 40 seconds using tpm 19Sunday, September 22, 13
  20. 20. ©Continuent 2013 20 ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --start Multi-master installation (4 nodes) with TR 2.1.1 ~ 45 seconds using tpm 20Sunday, September 22, 13
  21. 21. ©Continuent 2013 One more reason why tpm is better • Security: • all data in transit (transaction history logs, or THL) is encrypted using SSL • all administrative tra"c is encrypted with SSL • Transparent to the user • Independent of the database server (works also for heterogeneous replication) 21 21Sunday, September 22, 13
  22. 22. ©Continuent 2013 Tungsten replicator without security 22 replicator services host1 host4 host2 host3 alpha alpha alpha alpha master slave THL THL THLTHL plain text 22Sunday, September 22, 13
  23. 23. ©Continuent 2013 Tungsten Replicator with security 23 replicator services host1 host4 host2 host3 alpha alpha alpha alpha master slave THL THL THLTHL SSL SSL SSL SSL 23Sunday, September 22, 13
  24. 24. ©Continuent 2013 Installing with security 1: create keys 24 ./cookbook/create-security-keys HarryPotter ... ... # Options for tpm (stored in ./cookbook/security.options) --thl-ssl=true --rmi-ssl=true --rmi-authentication=true --rmi-user=cookbook --java-keystore-password=HarryPotter --java-truststore-password=HarryPotter --java-truststore-path=$PWD/cookbook/truststore.ts --java-keystore-path=$PWD/cookbook/keystore.jks --java-jmxremote-access-path=$PWD/cookbook/jmxremote.access --java-passwordstore-path=$PWD/cookbook/passwords.store 24Sunday, September 22, 13
  25. 25. ©Continuent 2013 Installing with security - 2: run tpm 25 ./tools/tpm install alpha --topology=master-slave --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --master=host1 --slaves=host2,host3,host4 $(cat ./cookbook/security.options) --start ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 $(cat ./cookbook/security.options) --master-services=alpha,bravo,charlie,delta --start 25Sunday, September 22, 13
  26. 26. ©Continuent 2013 Demo : security 26 26Sunday, September 22, 13
  27. 27. ©Continuent 2013 Advanced Installation using Cookbook 27 27Sunday, September 22, 13
  28. 28. ©Continuent 2013 Installing Master/Slave Replication ... 28 ./cookbook/install_master_slave host1 host2 host3 alpha alpha alpha THL THL THL 28Sunday, September 22, 13
  29. 29. ©Continuent 2013 master/slave using tpm 29 ./tools/tpm install alpha --topology=master-slave --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --master=host1 --slaves=host2,host3,host4 --start 29Sunday, September 22, 13
  30. 30. ©Continuent 2013 Installing Fan-In Replication 30 ./cookbook/install_fan_in host1 host4 host2 host3 alpha bravo charlie bravo charlie alpha 30Sunday, September 22, 13
  31. 31. ©Continuent 2013 fan-in using tpm 31 ./tools/tpm install many_towns --replication-user=tungsten --replication-password=secret --home-directory=/opt/continuent/replication --masters=host1,host2,host3 --slaves=host4 --master-services=alpha,bravo,charlie --topology=fan-in --start 31Sunday, September 22, 13
  32. 32. ©Continuent 2013 Install Multi-Master replication 32 ./cookbook/install_all_masters host1 host2 alpha alpha bravo bravo 32Sunday, September 22, 13
  33. 33. ©Continuent 2013 multi-master using tpm 33 ../tools/tpm install musketeers --reset --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start 33Sunday, September 22, 13
  34. 34. ©Continuent 2013 Extend multi-master-replication 34 host1 host2 alpha alpha bravo bravo host1 host2 host3 alpha alpha bravo bravo charlie charlie bravo alpha charlie 34Sunday, September 22, 13
  35. 35. ©Continuent 2013 extending multi-master (1:install) 35 ../tools/tpm install musketeers --reset --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start 35Sunday, September 22, 13
  36. 36. ©Continuent 2013 extending multi-master (2: add a node) 36 ./tools/tpm configure musketeers --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3 --master-services=alpha,bravo,charlie --start ./tools/tpm update 36Sunday, September 22, 13
  37. 37. ©Continuent 2013 Extending further ... 37 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host1 host2 host3 alpha alpha bravo bravo charlie charlie bravo alpha charlie 37Sunday, September 22, 13
  38. 38. ©Continuent 2013 extending multi-master (3: add one more node) 38 ./tools/tpm configure musketeers --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --start ./tools/tpm update 38Sunday, September 22, 13
  39. 39. ©Continuent 2013 Install a Star Topology 39 ./cookbook/install_star host1 host4 host2 host3 alpha bravo charlie charlie delta bravo alpha delta charlie charlie hub 39Sunday, September 22, 13
  40. 40. ©Continuent 2013 star using tpm 40 ./tools/tpm install constellation --replication-user=tungsten --replication-password=secret --home-directory=/opt/continuent/replication --masters=host1,host2,host4 --hub=host3 --hub-service=charlie --master-services=alpha,bravo,delta --topology=star --start 40Sunday, September 22, 13
  41. 41. ©Continuent 2012. More complex topologies 41 41Sunday, September 22, 13
  42. 42. ©Continuent 2013 all-masters with attached slaves 42 host1 host4 host2 host3 alpha alpha bravo bravo bravo alpha data from goes to host1 host2,host3 host2 host1,host4 42Sunday, September 22, 13
  43. 43. ©Continuent 2013 all-masters with attached slaves 43 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host6host5 deltacharlie data from goes to host1 host2,host3,host4 host2 host1,host3,host4 host3 host1,host2,host4,host5 host4 host1,host2,host3,host6 43Sunday, September 22, 13
  44. 44. ©Continuent 2013 four all-masters with attached slaves 1: con!gure defaults 44 #1 ./tools/tpm configure defaults --reset --replication-user=tungsten --replication-password=secret --home-directory=/home/tungsten/installs/cookbook --start 44Sunday, September 22, 13
  45. 45. ©Continuent 2013 four all-masters with attached slaves 2: con!gure main all-masters 45 #2 ./tools/tpm configure four_musketeers --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --topology=all-masters 45Sunday, September 22, 13
  46. 46. ©Continuent 2013 four all-masters with attached slaves 3: con!gure !rst slave 46 #3 ./tools/tpm configure charlie --hosts=host3,host5 --slaves=host5 --master=host3 46Sunday, September 22, 13
  47. 47. ©Continuent 2013 four all-masters with attached slaves 4: con!gure second slave 47 #4 ./tools/tpm configure delta --hosts=host4,host6 --slaves=host6 --master=host4 #5 ./tools/tpm install 47Sunday, September 22, 13
  48. 48. ©Continuent 2013 four all-masters with attached slaves 5: !nal step: install 48 #5 ./tools/tpm install 48Sunday, September 22, 13
  49. 49. ©Continuent 2013 all-masters with attached mongoDB slave 49 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host5 alpha 49Sunday, September 22, 13
  50. 50. ©Continuent 2013 all-masters with mongodb slave 1: install masters with heterogeneous options 50 ./tools/tpm configure all_masters --reset --log=timestamp --replication-port=3306 --replication-user=tungsten --replication-password=secret --home-directory=/home/tungsten/installs/cookbook --repl-disable-relay-logs=true -a --masters=ubuntu1,ubuntu2,ubuntu3,ubuntu4 --master-services=alpha,bravo,charlie,delta --topology=all-masters --java-file-encoding=UTF8 --mysql-use-bytes-for-string=false --mysql-enable-enumtostring=true --mysql-enable-settostring=true --svc-extractor-filters=colnames,pkey --start-and-report 50Sunday, September 22, 13
  51. 51. ©Continuent 2013 all-masters with mongodb slave 2: add mongodb slave and install 51 ./tools/tpm configure alpha --datasource-type=mongodb --hosts=host1 --master=ubuntu1 --replication-port=6700 --home-directory=~/deploy/mongodb --slaves=host1 --role=slave --master-thl-port=2112 --thl-port=2113 --rmi-port=10100 --start-and-report ./tools/tpm install --verbose 51Sunday, September 22, 13
  52. 52. ©Continuent 2013 DEMO: multi master installation 52 52Sunday, September 22, 13
  53. 53. ©Continuent 2012. Overview of Tungsten Cookbook 53 53Sunday, September 22, 13
  54. 54. ©Continuent 2013 Tungsten Cookbook tungsten-replicator-2.1.1-230 | +--/cluster-home +--/cookbook +--/tools +--/tungsten-replicator 54 54Sunday, September 22, 13
  55. 55. ©Continuent 2013 Tungsten Cookbook tungsten-replicator-2.1.1-230 | +--/cookbook | +--COMMON_NODES.sh +--USER_VALUES.sh +--show_cluster +--test_cluster +--clear_cluster ... 55 55Sunday, September 22, 13
  56. 56. ©Continuent 2013 Tungsten Cookbook $ cat COMMON_NODES.sh export NODE1=host1 export NODE2=host2 export NODE3=host3 export NODE4=host4 56 56Sunday, September 22, 13
  57. 57. ©Continuent 2013 Tungsten Cookbook $ cat USER_VALUES.sh # User defined values for the cluster to be installed. export TUNGSTEN_BASE=$HOME/installs/cookbook export DATABASE_USER=tungsten export BINLOG_DIRECTORY=/var/lib/mysql export MY_CNF=/etc/my.cnf export DATABASE_PASSWORD=secret export DATABASE_PORT=3306 export TUNGSTEN_SERVICE=cookbook export RMI_PORT=10000 export THL_PORT=2112 export START_OPTION=start 57 57Sunday, September 22, 13
  58. 58. ©Continuent 2013 Sample all-masters Installation • edit cookbook/COMMON_NODES.sh • edit cookbook/USER_VALUES.sh • run cookbook/install_all_masters • and then: • run cookbook/show_cluster • run cookbook/test_cluster 58 58Sunday, September 22, 13
  59. 59. ©Continuent 2013 DEMO: cookbook main tools * show status * show logs * edit con!guration * show services 59 59Sunday, September 22, 13
  60. 60. ©Continuent 2013 Joining the Community 60 60Sunday, September 22, 13
  61. 61. ©Continuent 2013 Tungsten Replicator is Open Source 61 • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, !nd builds, post in replicator discussion group • Documentation: https://docs.continuent.com/wiki/display/TEDOC/ Tungsten+Documentation+Home 61Sunday, September 22, 13
  62. 62. ©Continuent 2013 Wrapping Up 62 62Sunday, September 22, 13
  63. 63. ©Continuent 2012. Continuent Web Page: http://www.continuent.com Tungsten Replicator 2.1: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://www.continuent.com/news/blogs 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com 63Sunday, September 22, 13

×