Solving MySQL replication problems with Tungsten

Using Tungsten Replicator
to solve replication
problems
Neil Armitage, Cluster implementation Engineer, Continuent
Giuseppe Maxia, QA Director, Continuent

©Continuent 2012. 1

Monday, December 03, 12 1

ABOUT US

• Neil Armitage
• Continuent Tungsten Deployment and Support
Engineer, Continuent, Inc
• 20 years development and DB experience
• Giuseppe Maxia, a.k.a. "The Data Charmer"
• QA Director, Continuent, Inc
• 25 years development and DB experience
• long timer MySQL community member. Oracle
ACE Director



What are we talking about?

• Requirements

• Components

• Installation

• Topologies

• Administration

• Troubleshooting



Tungsten Replicator Concepts

Replicator The replication engine

Role Master, slave, direct slave

service A.k.a. "pipeline"

stage extract,queue,apply



Tungsten Replicator Components

THL Transaction History Log

service schema Makes the node crash proof

properties file service deﬁnition

Ruling from a centralized
tools
location



Tungsten Replicator in a nutshell
host1
host2
master
slave

binlog THL
global
transaction ID THL

trep_commit_seqno
origin seqno eventid
trep_commit_seqno
origin seqno eventid



Planning

• Hosts

• Topology

• Stand-alone or taking over



master-slave Heterogeneous Direct slave
Regular
MySQL
MySQL Oracle
Master

Oracle MySQL

fan-in slave all-masters star-schema

©Continuent 2012.


Installation



Installation

• System Requirements

• Validate !rst

• Deploying from a single location



Installation - tools

• tools/ tungsten-installer

• tools/ con!gure-service

• tools/update

• (Using the cookbook recipes, you hardly see
them)



Tungsten in practice
Installation



Installation

• Check the requirements

• Get the binaries

• Expand the tarball

• Run cookbook



REQUIREMENTS

• Java JRE or JDK (Sun/Oracle or Open-jdk)

• Ruby 1.8 (only during installation)

• ssh access to the same user in all nodes

• MySQL user with all privileges



Installation - Choices

• --master-slave

• --direct



master-slave

host2
host1

master slave

THL
binlog THL

host3

slave

THL



direct

host2
host1
slave
master
relay log

THL
binlog
host3

slave

relay log

THL


• Overview of VM’s

• overview of cookbook



Requirements : how to

• step by step: how it happened



installing VMs

• Step-by-step demo



Overview of Tungsten
cookbook



tungsten cookbook
tungsten-replicator-2.0.7-xx
|
+--/cluster-home
+--/cookbook
+--/tools
+--/tungsten-replicator



tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_MASTER_SLAVE.sh
+--show_master_slave.sh
+--test_master_slave.sh
+--clear_cluster_master_slave.sh
...



tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_STAR.sh
+--show_star.sh
+--test_star.sh
+--clear_cluster_star.sh
...



tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_ALL_MASTERS.sh
+--show_all_masters.sh
+--test_all_masters.sh
+--clear_cluster_all_masters.sh
...



tungsten cookbook
|
+--/cookbook
|
+--COMMON_NODES.sh
+--USER_VALUES.sh
+--NODES_FAN_IN.sh
+--show_fan_in.sh
+--test_fan_in.sh
+--clear_cluster_fan_in.sh
...



tungsten cookbook
$ cat COMMON_NODES.sh

export NODE1=host1
export NODE2=host2
export NODE3=host3
export NODE4=host4



tungsten cookbook
$ cat USER_VALUES.sh
# User defined values for the cluster to be
installed.

export TUNGSTEN_BASE=$HOME/installs/cookbook
export DATABASE_USER=tungsten
export BINLOG_DIRECTORY=/var/lib/mysql
export MY_CNF=/etc/my.cnf
export DATABASE_PASSWORD=secret
export DATABASE_PORT=3306
export TUNGSTEN_SERVICE=cookbook
export RMI_PORT=10000
export THL_PORT=2112
export START_OPTION=start


sample master-slave installation

• edit cookbook/COMMON_NODES.sh

• edit cookbook/USER_VALUES.sh

• run cookbook/install_master_slave.sh

• and then:

• run cookbook/show_master_slave.sh

• run cookbook/test_master_slave.sh



What does the installation do

1: Validate all servers

host4 host1 host2 host3

Report all errors




1: (again) Validate all servers

host4 host1 host2 host3




2: install Tungsten in all servers
host4
$HOME/ host1
tinstall/ host2
config/ host3
releases/
relay/
thl/
tungsten/



example (from manual installation)
ssh r2 chmod 444 $HOME/tinstall
./tools/tungsten-installer
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable



example
ssh r2 chmod 755 $HOME/tinstall
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

# no errors



Master - to - master

• Bi-directional installation

• Operational steps:
• install a master service in both servers
• install the corresponding slave service in the
other server

master-alpha
slave-alpha
slave-bravo
master-bravo



BI-DIR: the painless way

• edit cookbook/COMMON_NODES.sh

• edit cookbook/USER_VALUES.sh
• remove two nodes
• edit the variables in cookbook/
NODES_ALL_MASTERS.sh

• cookbook/install_all_masters.sh



A master-alpha

slave-bravo
B master-bravo all-masters
slave-charlie
slave-alpha
A slave-delta
slave-charlie

slave-delta
B

D C
D master-delta C master-charlie

slave-alpha slave-alpha

slave-bravo slave-bravo

slave-charlie slave-delta



Multiple masters

• fan-in

• Steps:
• install a master service in each node
• install a slave service for each master in the fan-
in node
• or :
• cookbook/install_fan_in.sh



Master - to - master

• star schema

• Steps:
• install a master service in each server
• in the hub, install a slave service for each spoke
• in each spoke, install a slave service for the hub,
using bypass option
• cookbook/install_star.sh



Taking Over from Standard
Replication
• cookbook/install_standard_replicaton.sh

• cookbook/takeover.sh



Monitoring and Alerting

• Using trepctl, thl



Replication Management



Common Commands

• replicator

• trepctl

• thl

• the Tungsten service schema



replicator

• It’s the service provider

• You launch it once when you start

• You may restart it when you change con!g



trepctl

• Tungsten Replicator ConTroLler

• It’s the driving seat for your replication

• You can start, update, and stop services

• You can get speci!c info



trepctl
• Tungsten Replicator Controller
• put services online or o"ine
• check status
• skip events
• inspect internals
• change roles
• heartbeat
• backup/restore
• ... and a lot more


thl

• Transaction History List

• Gives you access to the Tungsten relay logs



thl

• Transaction History Log
• info
• index
• list (total or a speci!c event, or by range)
• purge



Tungsten service schema

• one for each service

• named "tungsten_SERVICE_NAME"

• e.g. tungsten_alpha, tungsten_dragon

• Most important table: trep_commit_seqno



Looking at the tungsten service db
select * from tungsten_dragon.trep_commit_seqnoG
******************* 1. row *******************
task_id: 0
seqno: 102
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.
000002:0000000000018903;0
applied_latency: 0
update_timestamp: 2012-02-06 05:56:12
shard_id: tungsten_dragon
extract_timestamp: 2012-02-06 05:56:09



Where are the tools
in the tungsten directory:

$TUNGSTEN_BASE/tungsten/tungsten-replicator/bin

replicator # the daemon

trepctl # replicator controller

thl # transaction history log tool



Starting and stopping the replicator
cd $TUNGSTEN_BASE/tungsten/tungsten-replicator/bin

./replicator status
Tungsten Replicator Service is running (PID:32400).

./replicator stop
Stopping Tungsten Replicator Service...
Stopped Tungsten Replicator Service.

./replicator start
Starting Tungsten Replicator Service...



checking replicator vitals
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: -1 # bad sign?
appliedLatency : -1.0
role : slave
serviceName : dragon
serviceType : local
started : true
state : ONLINE
Finished services command...



sending a heartbeat
trepctl -host $MASTER_HOST heartbeat
trepctl services
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 102
appliedLatency : 3.139
role : slave
serviceType : local
started : true
state : ONLINE
Finished services command...



replicator status (1)
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000018903;0
appliedLastSeqno : 102
clusterName : default
currentEventId : NONE
currentTimeMillis : 1328504342058
dataServerHost : qa.r4.continuent.com
extensions :
latestEpochNumber : 0
masterConnectUri : thl://qa.r1.continuent.com:2112/
masterListenUri : thl://qa.r4.continuent.com:2112/
maximumStoredSeqNo : 102
minimumStoredSeqNo : 0
[...]



replicator status (2)
[...]
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceType : local
simpleServiceName : dragon
siteName : default
sourceId : qa.r4.continuent.com
state : ONLINE
timeInStateSeconds : 245.215
uptimeSeconds : 245.539
Finished status command...


A failover scenario
1: MySQL native replication



1. one Master, two slaves

• Loading the “employees” test database



2. Master goes away
* Stop replication
* Slaves are updated at di#erent levels

# 2
select count(*) from titles
333,145

# 3
443,308



3. Look into Slave #2 binary logs

• !nd the last transaction



4. Look into Slave #3 binary logs

1. !nd the transaction that was last in slave #2
2. Recognize that last transaction in the log of
slave #3 (This can actually take you a
LOOOONG TIME)
3. Get the position immediately after this
transaction
4. (e.g. 134000 in !le mysql-bin.000018)



5. promote Slave #3 to master
* in slave #2

CHANGE MASTER TO
master_host=‘slave_3_IP’,
master_user=‘slavename’,
master_password=‘slavepassword’,
master_log_file=‘mysql-bin.000018’,
master_log_pos=134000;



A failover scenario
1I: Tungsten Replicator



1. one master, two slaves

• loading the ‘employees’ test database



2. Master goes away
* Stop replication
* Slaves are updated at di#erent levels

# 2
333,145

# 3
443,308



3. no need to !nd the last transaction
# simply change roles

trepctl -host slave3 setrole -role master

trepctl -host slave2 setrole
-role slave -uri thl://slave3

trepctl -host slave3 online
State: ONLINE

trepctl -host slave2 online
State: GOING-ONLINE:SYNCHRONIZING



4. Check that the slave has
synchronized
# new master
select seqno from tungsten.trep_commit_seqno;
78

# new slave
64



4. Tell the replicator to hurry up
# new master
trepctl -node slave3 flush
Master log is synchronized with database at log
sequence number: 78

# new slave
trepctl host slave2 wait -applied 78
ONLINE
78



4. ... and we’re done
# new master
select count(*) from employees.titles
count(*)
443308

# new slave:
count(*)
443308



planned role switch
cookbook/install_master_slave.sh

cookbook/switch



Switch: 0

✔ online
A master

M:A M:A

slave B C slave

✔ online ✔ online


Switch: 1

✗ o"ine
A master
Master goes
o"ine

M:A M:A

slave B C slave



Switch: 2

✗ o"ine
Wait for
A master
latest
transaction
to be applied

M:A M:A

slave B C slave



Switch: 3

✗ o"ine
A master

slaves go
o"ine

M:A M:A

slave B C slave

✗ o"ine ✗ o"ine


Switch: 4

✗ o"ine
A master Slave is
promoted.
Notice: 2
masters, but
o"ine

M:A

slave B C master

✗ o"ine ✗ o"ine


Switch: 5

✗ o"ine old master
M:C
and
A slave
remaining
slave are
linked to new
master

M:C

slave B C master

✗ o"ine ✗ o"ine


Switch: 6

✔ online old master
M:C
and
A slave
remaining
slave connect
to new
master

M:C

slave B C master



Tungsten GTID vs MySQL 5.6 GTID

• What is GTID

• How it works in Tungsten

• How it works (or not) in MySQL 5.6



without global transaction ID

binlog
commit
commit
commit
commit
A master

position

binlog
binlog

slave B C slave

position 79
position
©Continuent 2012.


with global transaction ID

commit
commit
commit
commit
A master

id#200

slave B C slave

©Continuent 2012. id#200 80 id#200

Tungsten and global transaction ID:
activation
(none)
active by default



status
trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysql-bin.000002:0000000000001442;0
appliedLastSeqno : 6
clusterName : default
currentEventId : NONE
currentTimeMillis : 1354304680923
dataServerHost : qa.r4.continuent.com



seeing transactions
thl list -seqno 6
SEQ# = 6 / FRAG# = 0 (last frag)
- TIME = 2012-11-30 20:44:35.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000001442;0
- SOURCEID = qa.r1.continuent.com
- SQL(0) = insert into test.v1 values (1, 'inserted
by node #1') /* ___SERVICE___ = [cookbook] */



changing master connection
trepctl offline
trepctl online -seqno 105



Tungsten and Global transaction ID:
crash-safe slave tables
mysql -e 'select * from tungsten_cookbook.trep_commit_seqnoG'
*************************** 1. row ***************************
task_id: 0
seqno: 6
fragno: 0
last_frag: 1
source_id: qa.r1.continuent.com
epoch_number: 0
eventid: mysql-bin.000002:0000000000001442;0
applied_latency: 0
update_timestamp: 2012-11-30 20:44:35
shard_id: test
extract_timestamp: 2012-11-30 20:44:35



Tungsten and Global transaction ID:
crash-safe tables and parallel replication
mysql -e 'select seqno, source_id, shard_id,update_timestamp from
tungsten_cookbook.trep_commit_seqno'
+-------+----------------------+----------+---------------------+
| seqno | source_id | shard_id | update_timestamp |
+-------+----------------------+----------+---------------------+
| 7 | qa.r1.continuent.com | db1 | 2012-11-30 20:54:14 |
+-------+----------------------+----------+---------------------+



MySQL 5.6 and global transaction ID
activation
mysqld --log-slave-updates
--gtid-mode=on
--disable-gtid-unsafe-statements

WARNING: in MySQL 5.6.9, this will change to
--enforce-gtid-consistency



status
show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: rsandbox
Master_Port: 13233
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 1837
Relay_Log_File: mysql_sandbox13234-relay-bin.000005
Relay_Log_Pos: 2047
Relay_Master_Log_File: mysql-bin.000002
...
Retrieved_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7
Executed_Gtid_Set: 46E13434-3B28-11E2-BF47-6C626DA07446:1-7



changing master connection

CHANGE MASTER TO master_log_file='mysql-bin-000003',
master_log_pos='1234'

# No global transaction ID is used



crash-safe slave table
select * from slave_relay_log_infoG
********************* 1. row ********************
Number_of_lines: 7
Relay_log_name: ./mysql_sandbox13234-relay-bin.000005
Relay_log_pos: 2047
Master_log_name: mysql-bin.000002
Master_log_pos: 1837
Sql_delay: 0
Number_of_workers: 5
Id: 1

# NO Global transaction ID is used!



crash-safe slave table + parallel
select * from mysql.slave_worker_infoG
Id: 12
Relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Relay_log_pos: 4299
Master_log_name: mysql-bin.000002
Master_log_pos: 7155
Checkpoint_relay_log_name: ./mysql_sandbox13234-relay-bin.000007
Checkpoint_relay_log_pos: 1786
Checkpoint_master_log_name: mysql-bin.000002
Checkpoint_master_log_pos: 4642
Checkpoint_seqno: 9
Checkpoint_group_size: 64
Checkpoint_group_bitmap: ?

# NO Global transaction ID is used!



Filters



Replicator Pipeline Architecture

Tungsten Replicator Process
Pipeline
Stage Stage Stage
Assign
Extract Shard Apply Extract Apply Extract Apply
ID

Transaction
History Log

MySQL THL
Binlog Slave
DBMS



Pipeline
Stage Stage Stage
Assign
ID

Transaction
History Log

MySQL ﬁlter THL ﬁlter
Binlog Slave
DBMS


Restrict replication to some schemas
and tables
--master-slave -a
...
--svc-extractor-filters=replicate
"--property=replicator.filter.replicate.do=test,*.foo"
...
--start-and-report

# test="test.*" -> same drawback as binlog-do-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way



Exclude some schemas and tables
from replication
--master-slave -a
...
--svc-extractor-filters=replicate
"--property=replicator.filter.replicate.ignore=test,*.foo"
...
--start-and-report

# test="test.*" -> same drawback as binlog-ignore-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way

# DO NOT MIX .do and .ignore!
# (you can do it, but it may not do what you mean)



Change name of replicated schema
-a --svc-applier-filters=dbtransform
--property=replicator.filter.dbtransform.from_regex1=stores
--property=replicator.filter.dbtransform.to_regex1=playground

# from_regex1=stores -> name of the schema in the master
# to_regex1=playground -> name of the schema in the slave

# WARNING: requires "USE schema_name" to work properly.



Multi-master:
Con$ict prevention



Con$ict prevention facts




• Sharded by database





• De!ned dynamically






• Applied either at the master or at the slave







• methods:







• methods:
• make replication fail







• methods:
• drop silently







• methods:
• drop silently
• drop with warning



Tungsten Conﬂict prevention points
host1
host3

M S
S

host2 host4
master

S M S


Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
©Continuent 2012.
x,z 100


Prevention methods: Fail on slave
host1
host3

M S
S

host2 host4
master

S M S


host1
(Multiple sources)

M

host2 host3
master

M S


Prevention methods: DROP on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT B
x,y
host1
host3

M S
S

host2 host4
master

S M S


Prevention methods: DROP on slave
host1
(Multiple sources)

M

host2 host3
master

M S


host1
(Multiple sources)

M

host2 INSERT B host3
x,y
master
INSERT B
M x,z S


host1
(Multiple sources)

M

host2 host3
master
INSERT B
M x,z S


Con$ict prevention syntax during
installation
--svc-extractor-filters=shardfilter
--property=replicator.stage.q-to-
dbms.filters=mysqlsessions,pkey,bidiSlave,shardfilter

--property=replicator.filter.shardfilter.unknownShardPolicy=warn

# OR:
--property=replicator.filter.shardfilter.unknownShardPolicy=drop

--property=replicator.filter.shardfilter.unknownShardPolicy=error



con$ict prevention in a star topology
H
Host1 m
A master: alpha d
C database: employees
H
m
d
A Host3
master: charlie (hub)
B C
database: buildings

Host2
master: bravo
database: vehicles B C



con$ict prevention rules
trepctl -host host1 -service charlie
shard -insert < shards_charlie.map

cat shards_charlie.map
shard_id master critical
tungsten_charlie charlie false
buildings charlie false
test charlie false



trepctl -host host2 -service charlie
shard -insert < shards_charlie.map

cat shards_charlie.map
tungsten_charlie charlie false
buildings charlie false
test charlie false



trepctl -host host3 -service alpha
shard -insert < shards_alpha.map

cat shards_alpha.map
tungsten_alpha alpha false
employees alpha false
test alpha false



trepctl -host host3 -service bravo
shard -insert < shards_bravo.map

cat shards_alpha.map
tungsten_bravo bravo false
vehicles bravo false
test bravo false



Con$ict prevention demo



administration



Viewing THL Events
thl info
log directory = /home/tungsten/installs/master_slave/thl/dragon/
min seq# = 0
max seq# = 101
events = 101



viewing THL events
thl index
LogIndexEntry thl.data.0000000001(0:102)



viewing THL events
thl index
[...]



viewing THL events
thl list -seqno 102
[...]
SEQ# = 102 / FRAG# = 0 (last frag)
- TIME = 2012-02-06 05:56:09.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000018903;0
- SOURCEID = qa.r1.continuent.com
- METADATA =
[mysql_server_id=10;is_metadata=true;service=dragon;shard=tung
sten_dragon;heartbeat=NONE]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = ISO8859_1, autocommit = 1, sql_auto_is_null = 1, foreign_key_checks = 1,
unique_checks = 1, sql_mode = 'IGNORE_SPACE', character_set_client = 8, collation_connection = 8,
collation_server = 8]
- SCHEMA = tungsten_dragon
- SQL(0) = UPDATE tungsten_dragon.heartbeat SET source_tstamp=
"2012-02-06 05:56:09", salt= 2, name= "NONE" WHERE id= 1 /*
___SERVICE___ = [dragon] */



Skipping a THL Event
trepctl online -skip-seqno 1092

trepctl online -skip-seqno 1092,1093,1094

# see example



Adding a Member

• Let's see the cookbook, and use it



parallel replication




Pipeline
Stage Stage Stage
Extract Apply
Assign Parallel
Queue
ID Extract Apply

“channels”
Transaction
History Log
shard.list
MySQL THL ﬁle
Binlog Slave
DBMS


Parallel replication facts

✓ Sharded by database
✓ Good choice for slave lag problems
❖ Bad choice for single database projects



Parallel Replication test

STOPPED

binary logs

MySQL slave Concurrent sysbench
on 30 databases
running for 1 hour
OFFLINE
TOTAL DATA: 130 GB
Tungsten slave direct:
alpha
RAM per server: 20GB
(slave)

replicator alpha

Slaves will have 1 hour lag

measuring results

START

binary logs

MySQL slave

ONLINE Recording
catch-up time
Tungsten slave direct:
alpha
(slave)

replicator alpha


MySQL native
replication

slave catch up in 04:29:30


Tungsten parallel
replication

slave catch up in 00:55:40


Parallel replication made simpler

FROM HERE ....


TO HERE

direct slave facts



direct slave facts

✓ No need to install Tungsten on the master



direct slave facts

✓ Tungsten runs only on the slave



direct slave facts

✓ Replication can revert to native slave with two
commands (trepctl offline; start slave)



direct slave facts


✓ Native replication can continue on other slaves



direct slave facts


✓ Native replication can continue on other slaves
❖ Failover (either native or Tungsten) becomes a manual
task



installing parallel replication

• MORE_OPTIONS='--channels=10'

• ./cookbook/install_master_slave.sh



Checking parallel replication
trepctl status

trepctl status -name tasks

trepctl status -name shards

trepctl status -name stores



Troubleshooting



Identify the Failed Component

• Steps
1. trepctl services
2. trepctl -service SVC_NAME status
3. look at the logs
4. Take action



reading the logs
ls $TUNGSTEN_BASE/tungsten/tungsten-replicator/logs/
trepsvc.log user.log

# let's see it in practice



Parting thoughts



We are hiring QA engineers

http://continuent.com/about/careers



560 S. Winchester Blvd., Suite 500 Our Blogs:
San Jose, CA 95128 http://scale-out-blog.blogspot.com
Tel +1 (866) 998-3642 http://datacharmer.blogspot.com
Fax +1 (408) 668-1009 http://ﬂyingclusters.blogspot.com
e-mail: sales@continuent.com

Continuent Website:
http://www.continuent.com

Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator

©Continuent 2012 138


Solving MySQL replication problems with Tungsten

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Solving MySQL replication problems with Tungsten

Similar to Solving MySQL replication problems with Tungsten (20)

More from Giuseppe Maxia

More from Giuseppe Maxia (20)

Recently uploaded

Recently uploaded (20)

Solving MySQL replication problems with Tungsten