Preventing con!icts in                          Multi-master replication                               with Tungsten      ...
Introducing Continuent                    •     The leading provider of clustering and                          replicatio...
What is Tungsten Replicator?       •      Open source drop-in replacement for MySQL replication,              providing:  ...
Tungsten Replicator Overview                                                 Master                                       ...
Tungsten Replication Service                                              Pipeline                      Stage             ...
Multiple Services Per Replicator               NYC                          Replicator                            Service ...
Multi-Master Replication                    •     Updates on 2+ sites (active-active mode)                    •     Enable...
Simple Multi-Master Con!guration              NYC         Replicator                          Frankfurt                   ...
Three-node               Multi-Master Con!guration            NYC           Replicator                      Replicator    ...
multi-node star Con!guration               NYC            Replicator                       Replicator                     ...
CONFLICTS   ©Continuent 2012           11Sunday, February 03, 13
Whats a con"ict                    •     Data modi!ed by several sources (masters)                    •     Creates one or...
Data duplication                                4        Matt         140                                    id      name ...
auto_increment o#sets               are not a remedy                    •     A popular recipe                          • ...
Hidden data duplication                          11        Matt         140                                INSERT         ...
Data inconsistency                                3        Sue         108                                    UPDATE      ...
Data loss                                3        Sue        108                                    UPDATE                ...
con"ict handling strategies                    •     resolving                          •   after the fact                ...
Tungsten Replicator Filters   ©Continuent 2012.         19Sunday, February 03, 13
Replicator Pipeline Architecture                                   Tungsten Replicator Process                            ...
Restrict replication to some schemas               and tables  ./tools/tungsten-installer      --master-slave -a     ...  ...
Multi-master:                Con!ict prevention   ©Continuent 2012.      22Sunday, February 03, 13
Tungsten con"ict prevention               in a nutshell                1. de!ne the rules                          (which ...
Tungsten Con"ict prevention facts                    •     Sharded by database                    •     De!ned dynamically...
Tungsten con"ict prevention               applicability                    •     unknown shards                          •...
Con"ict prevention directives  --svc-extractor-filters=shardfilter  replicator.filter.shardfilter.unknownShardPolicy=error...
con"ict prevention in a star topology        alpha updates                                               H          employ...
con"ict prevention in a star topology        alpha updates                                               H           vehic...
con"ict prevention in a all-masters               topology                                                                ...
con"ict prevention in a all-masters               topology                                                                ...
con"ict prevention in a all-masters               topology                                                                ...
con"ict prevention in a all-masters               topology                                                                ...
setting con"ict prevention rules          trepctl -host host1 -service charlie             shard -insert < shards.map     ...
setting con"ict prevention rules          trepctl -host host2 -service charlie             shard -insert < shards.map     ...
setting con"ict prevention rules          trepctl         -host host3 -service alpha             shard         -insert < s...
Con"ict prevention demo       •       reminder       •       Server #1 can update "employees"       •       Server #2 can ...
Sample correct operation (1)          mysql #1> create table employees.names( ... )          # all servers receive the tab...
Sample correct operation (2)          mysql #2> create table buildings.homes( ... )          # all servers receive the tab...
Sample incorrect operation (1)          mysql #2> create table employees.nicknames( ... )          # Only server #2 receiv...
sample incorrect operation (2)          #3 $ trepct services | simple_services          alpha    [slave]          seqno:  ...
sample incorrect operation (3)          #3 $ trepct -service bravo status          NAME                     VALUE         ...
Fixing the issue          mysql #1> drop table if exists employees.nicknames;          mysql #1> create table if exists em...
Sample whitelisted operation          mysql #2> create table test.hope4best( ... )          mysql #1> insert into test.hop...
Parting thoughts                We are hiring !                http://continuent.com/about/careers   ©Continuent 2012.    ...
blog: http://datacharmer.blogspot.com                          twitter: @datacharmer                                 Conti...
Upcoming SlideShare
Loading in …5
×

Preventing multi master conflicts with tungsten

1,555 views

Published on

Preventing replication conflicts in MySQL multi master deployments with Tungsten Replicator

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,555
On SlideShare
0
From Embeds
0
Number of Embeds
71
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Preventing multi master conflicts with tungsten

  1. 1. Preventing con!icts in Multi-master replication with Tungsten Giuseppe Maxia, QA Director, Continuent ©Continuent 2012. 1Sunday, February 03, 13
  2. 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Open source, Flexible, high- performance data movement ©Continuent 2012 2Sunday, February 03, 13
  3. 3. What is Tungsten Replicator? • Open source drop-in replacement for MySQL replication, providing: • Global transaction ID • Multiple masters • Multiple sources • Flexible topologies • Heterogeneous replication • Parallel replication • ... and more ©Continuent 2012. 3Sunday, February 03, 13
  4. 4. Tungsten Replicator Overview Master Replicator THL Download transactions via network (Transactions + Metadata) DBMS Logs Slave Replicator THL Apply using JDBC (Transactions + Metadata) ©Continuent 2012 4Sunday, February 03, 13
  5. 5. Tungsten Replication Service Pipeline Stage Stage Stage Extract Filter Apply Extract Filter Apply Extract Filter Apply Master Transaction In-Memory Slave DBMS History Log Queue DBMS ©Continuent 2012 5Sunday, February 03, 13
  6. 6. Multiple Services Per Replicator NYC Replicator Service nyc Replicator Service nyc Service fra Frankfurt Replicator Service fra London ©Continuent 2012 6Sunday, February 03, 13
  7. 7. Multi-Master Replication • Updates on 2+ sites (active-active mode) • Enables geographic distribution of data • No failover necessary if network fails or site becomes unavailable • Not all applications can handle multi-master • Applications must avoid con!icts • Careful testing required • Restoration of broken systems may not be easy ©Continuent 2012 7Sunday, February 03, 13
  8. 8. Simple Multi-Master Con!guration NYC Replicator Frankfurt Replicator fra (slave) fra (master) nyc (master) nyc (slave) Database-to-Database ©Continuent 2012 8Sunday, February 03, 13
  9. 9. Three-node Multi-Master Con!guration NYC Replicator Replicator Frankfurt fra (slave) fra (master) nyc (master) nyc (slave) lon (slave) lon (slave) Database-to-Database lon (master) fra (slave) nyc (slave) London Replicator ©Continuent 2012 9Sunday, February 03, 13
  10. 10. multi-node star Con!guration NYC Replicator Replicator Frankfurt nyc (master) fra (master) lon (slave) lon (slave) Replicator lon (master) Singapore sin (master) fra (slave) lon (slave) nyc (slave) London (hub) sin (slave) Replicator ©Continuent 2012 10Sunday, February 03, 13
  11. 11. CONFLICTS ©Continuent 2012 11Sunday, February 03, 13
  12. 12. Whats a con"ict • Data modi!ed by several sources (masters) • Creates one or more : • data loss (unwanted delete) • data inconsistency (unwanted update) • duplicated data (unwanted insert) • replication break ©Continuent 2012. 12Sunday, February 03, 13
  13. 13. Data duplication 4 Matt 140 id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 BREAKS REPLICATION 4 Matt 130 charlie ©Continuent 2012. 13Sunday, February 03, 13
  14. 14. auto_increment o#sets are not a remedy • A popular recipe • auto_increment_increment + auto_increment_offset • They dont prevent con"icts • They hide duplicates ©Continuent 2012. 14Sunday, February 03, 13
  15. 15. Hidden data duplication 11 Matt 140 INSERT bravo alpha id name amount o#set 2 o#set 1 1 Joe 100 2 Frank 110 3 Sue 100 INSERT charlie 13 Matt 130 o#set 3 ©Continuent 2012. 15Sunday, February 03, 13
  16. 16. Data inconsistency 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 UPDATE 3 Sue 105 charlie ©Continuent 2012. 16Sunday, February 03, 13
  17. 17. Data loss 3 Sue 108 UPDATE id name amount bravo alpha 1 Joe 100 2 Frank 110 3 Sue 100 DELETE MAY BREAK REPLICATION record #3 charlie ©Continuent 2012. 17Sunday, February 03, 13
  18. 18. con"ict handling strategies • resolving • after the fact • Needs information that is missing in async replication • avoiding • requires synchronous replication with 2pc • preventing used by Tungsten • setting and enforcing a split sources policy • Transforming and resolving planned for • all records are converted to INSERTs future use • con!icts are resolved within a given time window ©Continuent 2012 18Sunday, February 03, 13
  19. 19. Tungsten Replicator Filters ©Continuent 2012. 19Sunday, February 03, 13
  20. 20. Replicator Pipeline Architecture Tungsten Replicator Process Pipeline Stage Stage Stage Assign Extract Shard Apply Extract Apply Extract Apply ID Transaction History Log MySQL filter THL filter filter Binlog Slave DBMSSunday, February 03, 13
  21. 21. Restrict replication to some schemas and tables ./tools/tungsten-installer --master-slave -a   ...   --svc-extractor-filters=replicate   "--property=replicator.filter.replicate.do=test,*.foo"   ...   --start-and-report # test="test.*" -> same drawback as binlog-do-db in MySQL # *.foo = table foo in any database # employees.dept_codes,employees.salaries => safest way ©Continuent 2012. 21Sunday, February 03, 13
  22. 22. Multi-master: Con!ict prevention ©Continuent 2012. 22Sunday, February 03, 13
  23. 23. Tungsten con"ict prevention in a nutshell 1. de!ne the rules (which master can update which database) 2. tell Tungsten the rules 3. de!ne the policy (error, drop, warn, or accept) 4. Let Tungsten enforce your rules ©Continuent 2012. 23Sunday, February 03, 13
  24. 24. Tungsten Con"ict prevention facts • Sharded by database • De!ned dynamically • Applied on the slave services • methods: • error: make replication fail • drop: drop silently • warn: drop with warning ©Continuent 2012. 24Sunday, February 03, 13
  25. 25. Tungsten con"ict prevention applicability • unknown shards • The schema being updated is not planned • actions: accept, drop, warn, error • unwanted shards • the schema is updated from the wrong master • actions: accept, drop, warn, error • whitelisted shards • can be updated by any master ©Continuent 2012. 25Sunday, February 03, 13
  26. 26. Con"ict prevention directives --svc-extractor-filters=shardfilter replicator.filter.shardfilter.unknownShardPolicy=error replicator.filter.shardfilter.unwantedShardPolicy=error replicator.filter.shardfilter.enforceHomes=false replicator.filter.shardfilter.allowWhitelisted=false ©Continuent 2012. 26Sunday, February 03, 13
  27. 27. con"ict prevention in a star topology alpha updates H employees Host1 m A master: alpha d C database: employees H m d ✔ A Host3 master: charlie (hub) B C database: vehicles Host2 ✔ master: bravo database: buildings B C ©Continuent 2012. 27Sunday, February 03, 13
  28. 28. con"ict prevention in a star topology alpha updates H vehicles Host1 m A master: alpha d C database: employees H m d ✗ A Host3 master: charlie (hub) B C database: vehicles Host2 master: bravo database: buildings B C ©Continuent 2012. 28Sunday, February 03, 13
  29. 29. con"ict prevention in a all-masters topology H alpha updates Host1 m employees A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 29Sunday, February 03, 13
  30. 30. con"ict prevention in a all-masters topology H charlie updates Host1 m vehicles A master: alpha d B C database: employees H m d ✔ A Host3 master: charlie (hub) C ✔ B database: vehicles A Host2 master: bravo database: buildings B C ©Continuent 2012. 30Sunday, February 03, 13
  31. 31. con"ict prevention in a all-masters topology H alpha updates Host1 m buildings A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 31Sunday, February 03, 13
  32. 32. con"ict prevention in a all-masters topology H charlie updates Host1 m employees A master: alpha d B C database: employees H ✗ m d A Host3 Host2 ✗ A B C master: charlie (hub) database: vehicles master: bravo database: buildings B C ©Continuent 2012. 32Sunday, February 03, 13
  33. 33. setting con"ict prevention rules trepctl -host host1 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 1 ©Continuent 2012. 33Sunday, February 03, 13
  34. 34. setting con"ict prevention rules trepctl -host host2 -service charlie shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # charlie is slave service in host 2 ©Continuent 2012. 34Sunday, February 03, 13
  35. 35. setting con"ict prevention rules trepctl -host host3 -service alpha shard -insert < shards.map trepctl -host host3 -service bravo shard -insert < shards.map cat shards.map shard_id master critical personnel alpha false buildings bravo false vehicles charlie false test whitelisted false # alpha and bravo are slave services in host 3 ©Continuent 2012. 35Sunday, February 03, 13
  36. 36. Con"ict prevention demo • reminder • Server #1 can update "employees" • Server #2 can update "buildings" • Server #3 can update "vehicles" ©Continuent 2012. 36Sunday, February 03, 13
  37. 37. Sample correct operation (1) mysql #1> create table employees.names( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 37Sunday, February 03, 13
  38. 38. Sample correct operation (2) mysql #2> create table buildings.homes( ... ) # all servers receive the table # all servers keep working well ©Continuent 2012. 38Sunday, February 03, 13
  39. 39. Sample incorrect operation (1) mysql #2> create table employees.nicknames( ... ) # Only server #2 receives the table # slave service in hub gets an error # slave service in #1 does not receive anything ©Continuent 2012. 39Sunday, February 03, 13
  40. 40. sample incorrect operation (2) #3 $ trepct services | simple_services alpha [slave] seqno: 7 - latency: 0.136 - ONLINE bravo [slave] seqno: -1 - latency: -1.000 - OFFLINE:ERROR charlie [master] seqno: 66 - latency: 0.440 - ONLINE ©Continuent 2012. 40Sunday, February 03, 13
  41. 41. sample incorrect operation (3) #3 $ trepct -service bravo status NAME VALUE ---- ----- appliedLastEventId : NONE appliedLastSeqno : -1 appliedLatency : -1.0 (...) offlineRequests : NONE pendingError : Stage task failed: q-to-dbms pendingErrorCode : NONE pendingErrorEventId : mysql-bin.000002:0000000000001241;0 pendingErrorSeqno : 7 pendingExceptionMessage: Rejected event from wrong shard: seqno=7 shard ID=employees shard master=alpha service=bravo (...) ©Continuent 2012. 41Sunday, February 03, 13
  42. 42. Fixing the issue mysql #1> drop table if exists employees.nicknames; mysql #1> create table if exists employees.nicknames ( ... ) ; #3 $ trepct -service bravo online -skip-seqno 7 # all servers receive the new table ©Continuent 2012. 42Sunday, February 03, 13
  43. 43. Sample whitelisted operation mysql #2> create table test.hope4best( ... ) mysql #1> insert into test.hope4best values ( ... ) # REMEMBER: test was explicitly whitelisted # All servers get the new table and records # But there is no protection against conflicts ©Continuent 2012. 43Sunday, February 03, 13
  44. 44. Parting thoughts We are hiring ! http://continuent.com/about/careers ©Continuent 2012. 44Sunday, February 03, 13
  45. 45. blog: http://datacharmer.blogspot.com twitter: @datacharmer Continuent Website: http://www.continuent.com Tungsten Replicator 2.0: http://tungsten-replicator.org ©Continuent 2012 45Sunday, February 03, 13

×