Tungsten University: MySQL Multi-Master Operations Made Simple With Tungsten Replicator 2.1.1
•
0 likes•1,192 views
Deployment of MySQL multi-master topologies with Tungsten Replicator has been constantly improving. Yet, earlier there were some heavy operations to sustain, and unfriendly commands to perform. The latest version of Tungsten Replicator delivers all the topologies of its predecessors, with an improved installation tool that cuts down the deployment time to half in simple topologies, and to 1/10th in complex ones. Now you can install master/slave, multi-master, fan-in, and star topologies in less than a minute. But there is more. Thanks to a versatile Tungsten Replicator installation tool, you can define your own deployment on-the-fly, and get creative: you can have stars with satellites, all-masters with fan-in slaves, and other customized clusters. We will also cover other enhancements in Tungsten Replicator 2.1.1, such as full integration with MySQL 5.6, enhanced output from administrative tools, a few more goodies.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Tungsten University: MySQL Multi-Master Operations Made Simple With Tungsten Replicator 2.1.1
Similar to Tungsten University: MySQL Multi-Master Operations Made Simple With Tungsten Replicator 2.1.1 (20)
More from Continuent
More from Continuent (20)
Recently uploaded
Recently uploaded (20)
Tungsten University: MySQL Multi-Master Operations Made Simple With Tungsten Replicator 2.1.1
- 1. ©Continuent 2013 MySQL Multi-Master Operations Made Simple Featuring Tungsten Replicator 2.1.1 Giuseppe Maxia, Director of QA - Continuent, Inc Robert Hodges, CEO - Continuent, Inc
- 2. ©Continuent 2013 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
- 3. ©Continuent 2013 A Review of Tungsten Replicator 3
- 4. ©Continuent 2013 Tungsten Replicator Overview 4 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
- 5. ©Continuent 2013 Master Replication Service 5 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
- 6. ©Continuent 2013 Slave Replication Service 6 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
- 7. ©Continuent 2013 star master-slave Heterogeneous fan-in slave all-masters MySQL Oracle Oracle MySQL Oracle Oracle MySQL MySQL
- 8. ©Continuent 2013 Tungsten Replicator 2.1.1 8
- 9. ©Continuent 2013 Tungsten Replicator 2.1.1 • Releases as GA on August 21, 2013 • Main features: • installation with tpm • security layer • full support for MySQL 5.6 • improved management tools • trepctl properties • trepctl services [-full] [-json] • trepctl clients • trepctl status [-json] • thl list [-headers] [-json] 9
- 10. ©Continuent 2013 Working with Tungsten 10
- 11. ©Continuent 2013 Overview of Installation Process 1. Set up hosts 2. Prepare MySQL replicas 3. Download software 4. Install using tungsten-installer & con!gure- service tpm 11 Amazon Setup: https://docs.continuent.com/wiki/display/TEDOC/ Preparing+EC2+Servers
- 12. ©Continuent 2013 How tungsten-installer Works for Basic Master/Slave Deployment 12 db1 db2 db3 Staging copy of files check prereqs copy code configure
- 13. ©Continuent 2013 Tungsten master/slave replication 13 host1 host2 host3 alpha alpha alpha THL THL THL installer Old installation and New installation
- 14. ©Continuent 2013 Bi-directional replication 14 host1 host2 alpha alpha bravo bravo Install master on host1 installer install master on host2 installer install slave service on host1 installer install slave service on host2 installer OLD INSTALLATION
- 15. ©Continuent 2013 Bi-directional replication 15 host1 host2 alpha alpha bravo bravo NEW INSTALLATION Install all master and slave services on all hosts at once installer
- 16. ©Continuent 2013 16 ./tools/tungsten-installer --master-slave --master-host=$MASTER2 --datasource-user=tungsten --datasource-password=secret --service-name=bravo --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER2 --start $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER1 --datasource=$MASTER1 --local-service-name=alpha --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER2 --master-thl-port=2112 --svc-start bravo ./tools/tungsten-installer --master-slave --master-host=$MASTER1 --datasource-user=tungsten --datasource-password=secret --service-name=alpha --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER1 --start $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER2 --datasource=$MASTER2 --local-service-name=bravo --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER1 --master-thl-port=2112 --svc-start alpha 1 2 3 4 Multi-master installation (2 nodes) until TR 2.1.0 ~ 3 minutes using tungsten-installer
- 17. ©Continuent 2013 4 nodes all-masters 17 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha
- 18. ©Continuent 2013 18 Multi-master installation (4 nodes) until TR 2.1.0 ~ 8 minutes tungsten-installer host1 - alpha 1 tungsten-installer host2 - bravo 2 tungsten-installer host3 - charlie 3 tungsten-installer host4 - delta 4 con!gure-service host1 - bravo 5 con!gure-service host1 - charlie 6 con!gure-service host1 - delta 7 con!gure-service host2 - alpha 8 con!gure-service host2 - charlie 9 con!gure-service host2 - delta 10 con!gure-service host3 - alpha 11 con!gure-service host3 - bravo 12 con!gure-service host3 - delta 13 con!gure-service host4 - alpha 14 con!gure-service host4 - bravo 15 con!gure-service host4 - charlie 16 using tungsten-installer
- 19. ©Continuent 2013 19 ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start Multi-master installation (2 nodes) with TR 2.1.1 ~ 40 seconds using tpm
- 20. ©Continuent 2013 20 ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --start Multi-master installation (4 nodes) with TR 2.1.1 ~ 45 seconds using tpm
- 21. ©Continuent 2013 One more reason why tpm is better • Security: • all data in transit (transaction history logs, or THL) is encrypted using SSL • all administrative tra"c is encrypted with SSL • Transparent to the user • Independent of the database server (works also for heterogeneous replication) 21
- 22. ©Continuent 2013 Tungsten replicator without security 22 replicator services host1 host4 host2 host3 alpha alpha alpha alpha master slave THL THL THLTHL plain text
- 23. ©Continuent 2013 Tungsten Replicator with security 23 replicator services host1 host4 host2 host3 alpha alpha alpha alpha master slave THL THL THLTHL SSL SSL SSL SSL
- 24. ©Continuent 2013 Installing with security 1: create keys 24 ./cookbook/create-security-keys HarryPotter ... ... # Options for tpm (stored in ./cookbook/security.options) --thl-ssl=true --rmi-ssl=true --rmi-authentication=true --rmi-user=cookbook --java-keystore-password=HarryPotter --java-truststore-password=HarryPotter --java-truststore-path=$PWD/cookbook/truststore.ts --java-keystore-path=$PWD/cookbook/keystore.jks --java-jmxremote-access-path=$PWD/cookbook/jmxremote.access --java-passwordstore-path=$PWD/cookbook/passwords.store
- 25. ©Continuent 2013 Installing with security - 2: run tpm 25 ./tools/tpm install alpha --topology=master-slave --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --master=host1 --slaves=host2,host3,host4 $(cat ./cookbook/security.options) --start ./tools/tpm install my_cluster --topology=all-masters --home-directory=$TUNGSTEN_HOME --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 $(cat ./cookbook/security.options) --master-services=alpha,bravo,charlie,delta --start
- 26. ©Continuent 2013 Demo : security 26
- 27. ©Continuent 2013 Advanced Installation using Cookbook 27
- 28. ©Continuent 2013 Installing Master/Slave Replication ... 28 ./cookbook/install_master_slave host1 host2 host3 alpha alpha alpha THL THL THL
- 29. ©Continuent 2013 master/slave using tpm 29 ./tools/tpm install alpha --topology=master-slave --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --master=host1 --slaves=host2,host3,host4 --start
- 30. ©Continuent 2013 Installing Fan-In Replication 30 ./cookbook/install_fan_in host1 host4 host2 host3 alpha bravo charlie bravo charlie alpha
- 31. ©Continuent 2013 fan-in using tpm 31 ./tools/tpm install many_towns --replication-user=tungsten --replication-password=secret --home-directory=/opt/continuent/replication --masters=host1,host2,host3 --slaves=host4 --master-services=alpha,bravo,charlie --topology=fan-in --start
- 32. ©Continuent 2013 Install Multi-Master replication 32 ./cookbook/install_all_masters host1 host2 alpha alpha bravo bravo
- 33. ©Continuent 2013 multi-master using tpm 33 ../tools/tpm install musketeers --reset --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start
- 35. ©Continuent 2013 extending multi-master (1:install) 35 ../tools/tpm install musketeers --reset --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2 --master-services=alpha,bravo --start
- 36. ©Continuent 2013 extending multi-master (2: add a node) 36 ./tools/tpm configure musketeers --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3 --master-services=alpha,bravo,charlie --start ./tools/tpm update
- 37. ©Continuent 2013 Extending further ... 37 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host1 host2 host3 alpha alpha bravo bravo charlie charlie bravo alpha charlie
- 38. ©Continuent 2013 extending multi-master (3: add one more node) 38 ./tools/tpm configure musketeers --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --start ./tools/tpm update
- 39. ©Continuent 2013 Install a Star Topology 39 ./cookbook/install_star host1 host4 host2 host3 alpha bravo charlie charlie delta bravo alpha delta charlie charlie hub
- 40. ©Continuent 2013 star using tpm 40 ./tools/tpm install constellation --replication-user=tungsten --replication-password=secret --home-directory=/opt/continuent/replication --masters=host1,host2,host4 --hub=host3 --hub-service=charlie --master-services=alpha,bravo,delta --topology=star --start
- 41. ©Continuent 2012. More complex topologies 41
- 42. ©Continuent 2013 all-masters with attached slaves 42 host1 host4 host2 host3 alpha alpha bravo bravo bravo alpha data from goes to host1 host2,host3 host2 host1,host4
- 43. ©Continuent 2013 all-masters with attached slaves 43 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host6host5 deltacharlie data from goes to host1 host2,host3,host4 host2 host1,host3,host4 host3 host1,host2,host4,host5 host4 host1,host2,host3,host6
- 44. ©Continuent 2013 four all-masters with attached slaves 1: con!gure defaults 44 #1 ./tools/tpm configure defaults --reset --replication-user=tungsten --replication-password=secret --home-directory=/home/tungsten/installs/cookbook --start
- 45. ©Continuent 2013 four all-masters with attached slaves 2: con!gure main all-masters 45 #2 ./tools/tpm configure four_musketeers --masters=host1,host2,host3,host4 --master-services=alpha,bravo,charlie,delta --topology=all-masters
- 46. ©Continuent 2013 four all-masters with attached slaves 3: con!gure !rst slave 46 #3 ./tools/tpm configure charlie --hosts=host3,host5 --slaves=host5 --master=host3
- 47. ©Continuent 2013 four all-masters with attached slaves 4: con!gure second slave 47 #4 ./tools/tpm configure delta --hosts=host4,host6 --slaves=host6 --master=host4 #5 ./tools/tpm install
- 48. ©Continuent 2013 four all-masters with attached slaves 5: !nal step: install 48 #5 ./tools/tpm install
- 49. ©Continuent 2013 all-masters with attached mongoDB slave 49 host1 host4 host2 host3 alpha alpha bravo bravo charlie charlie delta delta bravobravo alpha delta delta charlie charlie alpha host5 alpha
- 50. ©Continuent 2013 all-masters with mongodb slave 1: install masters with heterogeneous options 50 ./tools/tpm configure all_masters --reset --log=timestamp --replication-port=3306 --replication-user=tungsten --replication-password=secret --home-directory=/home/tungsten/installs/cookbook --repl-disable-relay-logs=true -a --masters=ubuntu1,ubuntu2,ubuntu3,ubuntu4 --master-services=alpha,bravo,charlie,delta --topology=all-masters --java-file-encoding=UTF8 --mysql-use-bytes-for-string=false --mysql-enable-enumtostring=true --mysql-enable-settostring=true --svc-extractor-filters=colnames,pkey --start-and-report
- 51. ©Continuent 2013 all-masters with mongodb slave 2: add mongodb slave and install 51 ./tools/tpm configure alpha --datasource-type=mongodb --hosts=host1 --master=ubuntu1 --replication-port=6700 --home-directory=~/deploy/mongodb --slaves=host1 --role=slave --master-thl-port=2112 --thl-port=2113 --rmi-port=10100 --start-and-report ./tools/tpm install --verbose
- 52. ©Continuent 2013 DEMO: multi master installation 52
- 53. ©Continuent 2012. Overview of Tungsten Cookbook 53
- 56. ©Continuent 2013 Tungsten Cookbook $ cat COMMON_NODES.sh export NODE1=host1 export NODE2=host2 export NODE3=host3 export NODE4=host4 56
- 57. ©Continuent 2013 Tungsten Cookbook $ cat USER_VALUES.sh # User defined values for the cluster to be installed. export TUNGSTEN_BASE=$HOME/installs/cookbook export DATABASE_USER=tungsten export BINLOG_DIRECTORY=/var/lib/mysql export MY_CNF=/etc/my.cnf export DATABASE_PASSWORD=secret export DATABASE_PORT=3306 export TUNGSTEN_SERVICE=cookbook export RMI_PORT=10000 export THL_PORT=2112 export START_OPTION=start 57
- 58. ©Continuent 2013 Sample all-masters Installation • edit cookbook/COMMON_NODES.sh • edit cookbook/USER_VALUES.sh • run cookbook/install_all_masters • and then: • run cookbook/show_cluster • run cookbook/test_cluster 58
- 59. ©Continuent 2013 DEMO: cookbook main tools * show status * show logs * edit con!guration * show services 59
- 60. ©Continuent 2013 Joining the Community 60
- 61. ©Continuent 2013 Tungsten Replicator is Open Source 61 • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, !nd builds, post in replicator discussion group • Documentation: https://docs.continuent.com/wiki/display/TEDOC/ Tungsten+Documentation+Home
- 63. ©Continuent 2013 Tungsten University Sessions 63 • Con!gure & Provision Tungsten Clusters - Thursday September 5th - Tuesday September 10th • Setup & Operate Tungsten Replicator - Thursday October 17th - Tuesday October 22nd • Send any feedback to: tu@continuent.com
- 64. ©Continuent 2012. Continuent Web Page: http://www.continuent.com Tungsten Replicator 2.1.1: http://code.google.com/p/tungsten-replicator Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://www.continuent.com/news/blogs 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com