• Share
  • Email
  • Embed
  • Like
  • Private Content
Tungsten University: Set Up And Manage Advanced Replication Topologies
 

Tungsten University: Set Up And Manage Advanced Replication Topologies

on

  • 506 views

Do you know how to set up Tungsten Replication to handle multi-master topologies? Do you know how to replicate transactions from multiple servers into a single slave? Do you know how to replicate ...

Do you know how to set up Tungsten Replication to handle multi-master topologies? Do you know how to replicate transactions from multiple servers into a single slave? Do you know how to replicate between Tungsten clusters? In this course we show you how to set up and manage complex replication topologies using Tungsten.

Course Topics
- Overview of Tungsten features for complex replication configurations
- Basic installation commands
- Tungsten cookbook tools for fast setup of standard topologies
- Setting up master/slave topology
- Taking over from native MySQL replication
- Adding a node to a master/slave topology
- Switching between nodes in master/slave topology
- Setting up an all-masters topology
- Setting up star topologies
- Adding a node in a star topology
- Setting up fan-in replication from multiple masters
- Standard problems with complex topologies and how to handle them

Statistics

Views

Total Views
506
Views on SlideShare
504
Embed Views
2

Actions

Likes
1
Downloads
14
Comments
0

1 Embed 2

http://www.linkedin.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Tungsten University: Set Up And Manage Advanced Replication Topologies Tungsten University: Set Up And Manage Advanced Replication Topologies Presentation Transcript

    • Tungsten University:  Set Up and Manage Advanced Replication Topologies Robert Hodges, CEO Giuseppe Maxia, Director of QA©Continuent 2013
    • Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement©Continuent 2013 2
    • A Review of Tungsten Replicator©Continuent 2013 3
    • Tungsten Replicator Overview Master Replicator THL Download transactions via network (Transactions + Metadata) DBMS Logs Slave Replicator THL Apply using JDBC (Transactions + Metadata)©Continuent 2013 4
    • Master Replication Service Pipeline Stage Stage Extract Filter Apply Extract Filter Apply Slave Binlog tcp/ip Replicators In-Memory Transaction MySQL History Log Queue Master©Continuent 2013 5
    • Slave Replication Service Pipeline Stage Stage Stage Extract Filter Apply Extract Filter Apply Extract Filter Apply tcp/ip Master Replicator Transaction In-Memory Slave History Log Queue DBMS©Continuent 2013 6
    • Overview of Installation Process 1. Set up hosts 2. Prepare MySQL replicas 3. Download software 4. Install using tungsten-installer & con"gure- service Amazon Setup: https://docs.continuent.com/wiki/display/TEDOC/ Preparing+EC2+Servers©Continuent 2013 7
    • How tungsten-installer Works for Basic Master/Slave Deployment Staging copy db1 of files db2 check prereqs copy code db3 configure©Continuent 2013 8
    • From Master/Slave Replication ... db2 Replicator db1 Replicator Service alpha Service alpha db3 Replicator Service alpha tungsten- installer Install master and slaves on the whole cluster©Continuent 2013 9
    • To Multi-Master db2 db1 Replicator Replicator Service alpha Service alpha Service bravo Service bravo tungsten- tungsten- con!gure- con!gure- installer installer service serviceInstall master on install master install slave install slave db1 on db2 service on db1 service on db2©Continuent 2013 10
    • tungsten-installer master 1 TUNGSTEN_HOME=/home/tungsten/installs/cookbook ./tools/tungsten-installer --master-slave --master-host=$MASTER1 --datasource-port=3306 --datasource-user=tungsten --datasource-password=secret --datasource-log-directory=/var/lib/mysql --service-name=alpha --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER1 --start creating service alpha Notice: --cluster-hosts has only one host©Continuent 2013 11
    • tungsten-installer master 2 TUNGSTEN_HOME=/home/tungsten/installs/cookbook ./tools/tungsten-installer --master-slave --master-host=$MASTER2 --datasource-port=3306 --datasource-user=tungsten --datasource-password=secret --datasource-log-directory=/var/lib/mysql --service-name=bravo --home-directory=$TUNGSTEN_HOME --cluster-hosts=$MASTER2 --start creating service bravo Notice: --cluster-hosts has only one host©Continuent 2013 12
    • Con"gure Service master 1 TUNGSTEN_HOME=/home/tungsten/installs/cookbook $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER1 --datasource=$MASTER1 --local-service-name=alpha --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER2 --master-thl-port=2112 --svc-start bravo Notice: bravo is the master service in host 2©Continuent 2013 13
    • Con"gure Service master 2 TUNGSTEN_HOME=/home/tungsten/installs/cookbook $TUNGSTEN_HOME/tungsten/tools/configure-service -C --quiet --host=$MASTER2 --datasource=$MASTER2 --local-service-name=bravo --role=slave --service-type=remote --release-directory=$TUNGSTEN_HOME/tungsten --skip-validation-check=THLStorageCheck --master-thl-host=$MASTER1 --master-thl-port=2112 --svc-start alpha Notice: alpha is the master service in host 1©Continuent 2013 14
    • Advanced Installation using Cookbooks©Continuent 2013 15
    • From Master/Slave Replication ... db2 Replicator db1 Replicator Service db1 Service db1 db3 Replicator Service db1 ./cooobook/install_master_slave.sh©Continuent 2013 16
    • How Do I Install Fan-In Replication? db1 Replicator Service db1 Replicator Service db1 Service db2 db2 Replicator Service db2 db3 ./cooobook/install_fan_in.sh©Continuent 2013 17
    • How Do I Install Multi-Master? db1 Replicator Service db1 Service db2 db2 Service db1 Service db2 Replicator ./cooobook/install_all_masters.sh©Continuent 2013 18
    • How Do I Extend Multi-Master? db1 Replicator Service db1 Service db2 Replicator db3 Service db3 Service db1 Service db2 db2 Service db3 Service db1 Service db2 Service db3 Replicator©Continuent 2013 19
    • How Do I Extend Multi-Master? Replicator Replicator db1 db3 Service db1 Service db1 Service db2 Service db2 Service db3 Service db3 Service db4 Service db4 db2 db4 Service db1 Service db1 Service db2 Service db2 Service db3 Service db3 Service db4 Service db4 Replicator Replicator©Continuent 2013 20
    • How Do I Install a Star Topology? db1 Replicator Service db1 Replicator db3 Service db3 Service db1 HUB Service db2 db2 Service db3 Service db2 Service db3 Replicator ./cooobook/install_star.sh©Continuent 2013 21
    • How Do I Extend a Star Topology? Replicator db1 Service db1 db3 Service db1 Service db3 Service db2 HUB Service db3 Replicator db2 Service db4 Service db2 Service db3 db4 Service db3 Service db4 Replicator©Continuent 2013 22
    • How Do I Extend a Star Topology? Replicator db1 Service db1 db3 Service db1 Service db3 Service db2 HUB Service db3 Replicator db2 Service db4 Service db2 Service db5 Service db3 Replicator db4 db5 Service db5 Service db3 Service db3 Service db4 Replicator©Continuent 2013 23
    • Overview of Tungsten Cookbook©Continuent 2012. 24
    • Tungsten Cookbook tungsten-replicator-2.0.7-278 | +--/cluster-home +--/cookbook +--/tools +--/tungsten-replicator©Continuent 2013 25
    • Tungsten Cookbook tungsten-replicator-2.0.7-278 | +--/cookbook | +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_MASTER_SLAVE.sh +--show_master_slave.sh +--test_master_slave.sh +--clear_cluster_master_slave.sh ...©Continuent 2013 26
    • Tungsten Cookbook tungsten-replicator-2.0.7-278 | +--/cookbook | +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_STAR.sh +--show_star.sh +--test_star.sh +--clear_cluster_star.sh ...©Continuent 2013 27
    • Tungsten Cookbook tungsten-replicator-2.0.7-278 | +--/cookbook | +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_ALL_MASTERS.sh +--show_all_masters.sh +--test_all_masters.sh +--clear_cluster_all_masters.sh ...©Continuent 2013 28
    • Tungsten Cookbook tungsten-replicator-2.0.7-278 | +--/cookbook | +--COMMON_NODES.sh +--USER_VALUES.sh +--NODES_FAN_IN.sh +--show_fan_in.sh +--test_fan_in.sh +--clear_cluster_fan_in.sh ...©Continuent 2013 29
    • Tungsten Cookbook $ cat COMMON_NODES.sh export NODE1=host1 export NODE2=host2 export NODE3=host3 export NODE4=host4©Continuent 2013 30
    • Tungsten Cookbook $ cat USER_VALUES.sh # User defined values for the cluster to be installed. export TUNGSTEN_BASE=$HOME/installs/cookbook export DATABASE_USER=tungsten export BINLOG_DIRECTORY=/var/lib/mysql export MY_CNF=/etc/my.cnf export DATABASE_PASSWORD=secret export DATABASE_PORT=3306 export TUNGSTEN_SERVICE=cookbook export RMI_PORT=10000 export THL_PORT=2112 export START_OPTION=start©Continuent 2013 31
    • Sample Star Installation • edit cookbook/COMMON_NODES.sh • edit cookbook/USER_VALUES.sh • run cookbook/install_star.sh • and then: • run cookbook/show_star.sh • run cookbook/test_star.sh©Continuent 2013 32
    • CONFLICTS©Continuent 2013 33
    • Con#ict Handling Strategies • resolving planned for future • after the fact • Needs information that is missing in async replication • avoiding • requires synchronous replication with 2pc • preventing available now • setting and enforcing a split sources policy • transforming and resolving planned for • all records are converted to INSERTs future • con"icts are resolved within a given time window©Continuent 2013 34
    • Multi-Master: Con#ict Prevention©Continuent 2012. 35
    • Tungsten Con#ict Prevention Facts • Sharded by database • De"ned dynamically • Applied on the slave services • methods: • error: make replication fail • drop: drop silently • warn: drop with warning©Continuent 2013 36
    • Tungsten Con#ict Prevention Applicability • unknown shards • The schema being updated is not planned • actions: accept, drop, warn, error • unwanted shards • the schema is updated from the wrong master • actions: accept, drop, warn, error • whitelisted shards • can be updated by any master©Continuent 2013 37
    • Con#ict Prevention Directives--svc-extractor-filters=shardfilterreplicator.filter.shardfilter.unknownShardPolicy=errorreplicator.filter.shardfilter.unwantedShardPolicy=errorreplicator.filter.shardfilter.enforceHomes=falsereplicator.filter.shardfilter.allowWhitelisted=false©Continuent 2013 38
    • Con#ict Prevention in a Star Topology Host1 A master: alpha C database: employees A Host3 master: charlie (hub) B C database: vehiclesHost2master: bravodatabase: buildings B C©Continuent 2013 39
    • Setting Con#ict Prevention Rules trepctl -host host1 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 1©Continuent 2013 40
    • Sample Correct Operation (1) mysql #1> create table employees.names( ... ) # all servers receive the table # all servers keep working well©Continuent 2013 41
    • Sample Correct Operation (2) mysql #2> create table buildings.homes( ... ) # all servers receive the table # all servers keep working well©Continuent 2013 42
    • Sample Incorrect Operation (1) mysql #2> create table employees.nicknames( ... ) # Only server #2 receives the table # slave service in hub gets an error # slave service in #1 does not receive anything©Continuent 2013 43
    • Sample Incorrect Operation (2) #3 $ trepct services | simple_services alpha [slave] seqno: 7 - latency: 0.136 - ONLINE bravo [slave] seqno: -1 - latency: -1.000 - OFFLINE:ERROR charlie [master] seqno: 66 - latency: 0.440 - ONLINE©Continuent 2013 44
    • Sample Incorrect Operation (3) #3 $ trepct -service bravo status NAME VALUE ---- ----- appliedLastEventId : NONE appliedLastSeqno : -1 appliedLatency : -1.0 (...) offlineRequests : NONE pendingError : Stage task failed: q-to-dbms pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000002:0000000000001241;0 pendingErrorSeqno : 7 pendingExceptionMessage: Rejected event from unknown shard: seqno=7 shard ID=employees (...)©Continuent 2013 45
    • Fixing the Issue mysql #1> drop table if exists employees.nicknames; mysql #1> create table if exists employees.nicknames ( ... ) ; #3 $ trepct -service bravo online -skip-seqno 7 # all servers receive the new table©Continuent 2013 46
    • Joining the Community©Continuent 2013 47
    • Tungsten Replicator is Open Source • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, "nd builds, post in replicator discussion group • Documentation: https://docs.continuent.com/wiki/display/TEDOC/ Tungsten+Documentation+Home©Continuent 2013 48
    • Wrapping Up©Continuent 2013 49
    • Tungsten University Sessions • Zero-Downtime Maintenance And Upgrade - Thursday March 21st - Tuesday March 26th • Geographically Distributed Tungsten Clusters - Thursday April 11th - Tuesday April 16th • Replicate Between MySQL And Oracle - Thursday May 2nd - Tuesday May 7th • Send any feedback to: tu@continuent.com©Continuent 2013 50
    • 560 S. Winchester Blvd., Suite 500 Our Blogs:San Jose, CA 95128 http://scale-out-blog.blogspot.comTel +1 (866) 998-3642 http://datacharmer.org/blogFax +1 (408) 668-1009 http://www.continuent.com/news/blogse-mail: sales@continuent.com Continuent Web Page: http://www.continuent.com Tungsten Replicator 2.0: http://code.google.com/p/tungsten-replicator©Continuent 2012.