The document discusses multi-master replication using Continuent Tungsten and how it prevents data conflicts. It describes how Tungsten allows defining rules to specify which database can be updated by each master node. Tungsten then enforces these rules on slave nodes to prevent "unknown" or "unwanted" shards from being updated, which could lead to data duplication, inconsistency, loss or break replication. Examples are provided showing how conflict prevention works in different replication topologies.

1. Preventing con!icts in
Multi-master replication
with Tungsten
Giuseppe Maxia, QA Director, Continuent
©Continuent 2012. 1
Sunday, February 03, 13

2. Introducing Continuent
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Open source, Flexible, high-
performance data movement
©Continuent 2012 2
Sunday, February 03, 13

3. What is Tungsten Replicator?
• Open source drop-in replacement for MySQL replication,
providing:
• Global transaction ID
• Multiple masters
• Multiple sources
• Flexible topologies
• Heterogeneous replication
• Parallel replication
• ... and more
©Continuent 2012. 3
Sunday, February 03, 13

4. Tungsten Replicator Overview
Master
Replicator THL
Download
transactions
via network (Transactions + Metadata)
DBMS
Logs
Slave Replicator THL
Apply using JDBC (Transactions + Metadata)
©Continuent 2012 4
Sunday, February 03, 13

5. Tungsten Replication Service
Pipeline
Stage Stage Stage
Extract Filter Apply Extract Filter Apply Extract Filter Apply
Master Transaction In-Memory Slave
DBMS History Log Queue DBMS
©Continuent 2012 5
Sunday, February 03, 13

6. Multiple Services Per Replicator
NYC
Replicator
Service nyc
Replicator
Service nyc
Service fra
Frankfurt
Replicator
Service fra London
©Continuent 2012 6
Sunday, February 03, 13

7. Multi-Master Replication
• Updates on 2+ sites (active-active mode)
• Enables geographic distribution of data
• No failover necessary if network fails or site
becomes unavailable
• Not all applications can handle multi-master
• Applications must avoid con!icts
• Careful testing required
• Restoration of broken systems may not be easy
©Continuent 2012 7
Sunday, February 03, 13

8. Simple Multi-Master Con!guration
NYC Replicator Frankfurt
Replicator
fra (slave) fra (master)
nyc (master) nyc (slave)
Database-to-Database
©Continuent 2012 8
Sunday, February 03, 13

9. Three-node
Multi-Master Con!guration
NYC Replicator Replicator
Frankfurt
fra (slave) fra (master)
nyc (master) nyc (slave)
lon (slave) lon (slave)
Database-to-Database
lon (master)
fra (slave)
nyc (slave)
London
Replicator
©Continuent 2012 9
Sunday, February 03, 13

10. multi-node star Con!guration
NYC Replicator Replicator
Frankfurt
nyc (master) fra (master)
lon (slave) lon (slave)
Replicator
lon (master) Singapore
sin (master)
fra (slave)
lon (slave)
nyc (slave)
London (hub) sin (slave)
Replicator
©Continuent 2012 10
Sunday, February 03, 13

11. CONFLICTS
©Continuent 2012 11
Sunday, February 03, 13

12. What's a con"ict
• Data modi!ed by several sources (masters)
• Creates one or more :
• data loss (unwanted delete)
• data inconsistency (unwanted update)
• duplicated data (unwanted insert)
• replication break
©Continuent 2012. 12
Sunday, February 03, 13

13. Data duplication
4 Matt 140
id name amount bravo
alpha 1 Joe 100
2 Frank 110
3 Sue 100
BREAKS
REPLICATION
4 Matt 130
charlie
©Continuent 2012. 13
Sunday, February 03, 13

14. auto_increment o#sets
are not a remedy
• A popular recipe
• auto_increment_increment +
auto_increment_offset
• They don't prevent con"icts
• They hide duplicates
©Continuent 2012. 14
Sunday, February 03, 13

15. Hidden data duplication
11 Matt 140
INSERT
bravo
alpha id name amount o#set 2
o#set 1 1 Joe 100
2 Frank 110
3 Sue 100
INSERT
charlie 13 Matt 130
o#set 3
©Continuent 2012. 15
Sunday, February 03, 13

16. Data inconsistency
3 Sue 108
UPDATE
id name amount bravo
alpha 1 Joe 100
2 Frank 110
3 Sue 100
UPDATE
3 Sue 105
charlie
©Continuent 2012. 16
Sunday, February 03, 13

17. Data loss
3 Sue 108
UPDATE
id name amount bravo
alpha 1 Joe 100
2 Frank 110
3 Sue 100
DELETE MAY BREAK
REPLICATION
record #3
charlie
©Continuent 2012. 17
Sunday, February 03, 13

18. con"ict handling strategies
• resolving
• after the fact
• Needs information that is missing in async replication
• avoiding
• requires synchronous replication with 2pc
• preventing used by Tungsten
• setting and enforcing a split sources policy
• Transforming and resolving planned for
• all records are converted to INSERTs future use
• con!icts are resolved within a given time window
©Continuent 2012 18
Sunday, February 03, 13

19. Tungsten Replicator Filters
©Continuent 2012. 19
Sunday, February 03, 13

20. Replicator Pipeline Architecture
Tungsten Replicator Process
Pipeline
Stage Stage Stage
Assign
Extract Shard Apply Extract Apply Extract Apply
ID
Transaction
History Log
MySQL filter THL filter filter
Binlog Slave
DBMS
Sunday, February 03, 13

21. Restrict replication to some schemas
and tables
./tools/tungsten-installer
--master-slave -a
...
--svc-extractor-filters=replicate
"--property=replicator.filter.replicate.do=test,*.foo"
...
--start-and-report
# test="test.*" -> same drawback as binlog-do-db in MySQL
# *.foo = table 'foo' in any database
# employees.dept_codes,employees.salaries => safest way
©Continuent 2012. 21
Sunday, February 03, 13

22. Multi-master:
Con!ict prevention
©Continuent 2012. 22
Sunday, February 03, 13

23. Tungsten con"ict prevention
in a nutshell
1. de!ne the rules
(which master can update which database)
2. tell Tungsten the rules
3. de!ne the policy
(error, drop, warn, or accept)
4. Let Tungsten enforce your rules
©Continuent 2012. 23
Sunday, February 03, 13

24. Tungsten Con"ict prevention facts
• Sharded by database
• De!ned dynamically
• Applied on the slave services
• methods:
• error: make replication fail
• drop: drop silently
• warn: drop with warning
©Continuent 2012. 24
Sunday, February 03, 13

25. Tungsten con"ict prevention
applicability
• unknown shards
• The schema being updated is not planned
• actions: accept, drop, warn, error
• unwanted shards
• the schema is updated from the wrong master
• actions: accept, drop, warn, error
• whitelisted shards
• can be updated by any master
©Continuent 2012. 25
Sunday, February 03, 13

26. Con"ict prevention directives
--svc-extractor-filters=shardfilter
replicator.filter.shardfilter.unknownShardPolicy=error
replicator.filter.shardfilter.unwantedShardPolicy=error
replicator.filter.shardfilter.enforceHomes=false
replicator.filter.shardfilter.allowWhitelisted=false
©Continuent 2012. 26
Sunday, February 03, 13

27. con"ict prevention in a star topology
alpha updates H
employees Host1 m
A master: alpha d
C database: employees
H
m
d
✔ A Host3
master: charlie (hub)
B C
database: vehicles
Host2
✔
master: bravo
database: buildings B C
©Continuent 2012. 27
Sunday, February 03, 13

28. con"ict prevention in a star topology
alpha updates H
vehicles Host1 m
A master: alpha d
C database: employees
H
m
d
✗ A Host3
master: charlie (hub)
B C
database: vehicles
Host2
master: bravo
database: buildings B C
©Continuent 2012. 28
Sunday, February 03, 13

29. con"ict prevention in a all-masters
topology
H
alpha updates Host1 m
employees A master: alpha d
B
C database: employees
H
m
d
✔ A Host3
master: charlie (hub)
C
✔
B
database: vehicles
A
Host2
master: bravo
database: buildings B C
©Continuent 2012. 29
Sunday, February 03, 13

30. con"ict prevention in a all-masters
topology
H
charlie updates Host1 m
vehicles A master: alpha d
B
C database: employees
H
m
d
✔ A Host3
master: charlie (hub)
C
✔
B
database: vehicles
A
Host2
master: bravo
database: buildings B C
©Continuent 2012. 30
Sunday, February 03, 13

31. con"ict prevention in a all-masters
topology
H
alpha updates Host1 m
buildings A master: alpha d
B
C database: employees
H
✗
m
d
A Host3
Host2
✗ A
B C master: charlie (hub)
database: vehicles
master: bravo
database: buildings B C
©Continuent 2012. 31
Sunday, February 03, 13

32. con"ict prevention in a all-masters
topology
H
charlie updates Host1 m
employees A master: alpha d
B
C database: employees
H
✗
m
d
A Host3
Host2
✗ A
B C master: charlie (hub)
database: vehicles
master: bravo
database: buildings B C
©Continuent 2012. 32
Sunday, February 03, 13

33. setting con"ict prevention rules
trepctl -host host1 -service charlie
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 1
©Continuent 2012. 33
Sunday, February 03, 13

34. setting con"ict prevention rules
trepctl -host host2 -service charlie
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# charlie is slave service in host 2
©Continuent 2012. 34
Sunday, February 03, 13

35. setting con"ict prevention rules
trepctl -host host3 -service alpha
shard -insert < shards.map
trepctl -host host3 -service bravo
shard -insert < shards.map
cat shards.map
shard_id master critical
personnel alpha false
buildings bravo false
vehicles charlie false
test whitelisted false
# alpha and bravo are slave services in host 3
©Continuent 2012. 35
Sunday, February 03, 13

36. Con"ict prevention demo
• reminder
• Server #1 can update "employees"
• Server #2 can update "buildings"
• Server #3 can update "vehicles"
©Continuent 2012. 36
Sunday, February 03, 13

37. Sample correct operation (1)
mysql #1> create table employees.names( ... )
# all servers receive the table
# all servers keep working well
©Continuent 2012. 37
Sunday, February 03, 13

38. Sample correct operation (2)
mysql #2> create table buildings.homes( ... )
# all servers receive the table
# all servers keep working well
©Continuent 2012. 38
Sunday, February 03, 13

39. Sample incorrect operation (1)
mysql #2> create table employees.nicknames( ... )
# Only server #2 receives the table
# slave service in hub gets an error
# slave service in #1 does not receive anything
©Continuent 2012. 39
Sunday, February 03, 13

40. sample incorrect operation (2)
#3 $ trepct services | simple_services
alpha [slave]
seqno: 7 - latency: 0.136 - ONLINE
bravo [slave]
seqno: -1 - latency: -1.000 - OFFLINE:ERROR
charlie [master]
seqno: 66 - latency: 0.440 - ONLINE
©Continuent 2012. 40
Sunday, February 03, 13

41. sample incorrect operation (3)
#3 $ trepct -service bravo status
NAME VALUE
---- -----
appliedLastEventId : NONE
appliedLastSeqno : -1
appliedLatency : -1.0
(...)
offlineRequests : NONE
pendingError : Stage task failed: q-to-dbms
pendingErrorCode : NONE
pendingErrorEventId : mysql-bin.000002:0000000000001241;0
pendingErrorSeqno : 7
pendingExceptionMessage: Rejected event from wrong shard:
seqno=7 shard ID=employees shard master=alpha service=bravo
(...)
©Continuent 2012. 41
Sunday, February 03, 13

42. Fixing the issue
mysql #1> drop table if exists employees.nicknames;
mysql #1> create table if exists employees.nicknames ( ... ) ;
#3 $ trepct -service bravo online -skip-seqno 7
# all servers receive the new table
©Continuent 2012. 42
Sunday, February 03, 13

43. Sample whitelisted operation
mysql #2> create table test.hope4best( ... )
mysql #1> insert into test.hope4best values ( ... )
# REMEMBER: 'test' was explicitly whitelisted
# All servers get the new table and records
# But there is no protection against conflicts
©Continuent 2012. 43
Sunday, February 03, 13

44. Parting thoughts
We are hiring !
http://continuent.com/about/careers
©Continuent 2012. 44
Sunday, February 03, 13

45. blog: http://datacharmer.blogspot.com
twitter: @datacharmer
Continuent Website:
http://www.continuent.com
Tungsten Replicator 2.0:
http://tungsten-replicator.org
©Continuent 2012 45
Sunday, February 03, 13