How Safe is Asynchronous Master-Master
February, 01, 2020
Sveta Smirnova
• MySQL Support engineer
• Author of
• MySQL Troubleshooting
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova
2
•What Happens with Two Active Nodes?
•Why to Write to Multiple Nodes?
•Galera, PXC, InnoDB Cluster
•How to Setup Safe Asynchronous Master-Master
Table of Contents
3
Master Slave
← Initiates
Asynchronous Built-In Replication
4
Master Slave
← Initiates
← Requests a packet
Asynchronous Built-In Replication
4
Master
Sends the packet →
Slave
← Initiates
← Requests a packet
Asynchronous Built-In Replication
4
Master
Sends the packet →
Slave
← Initiates
← Requests a packet
... ?
Asynchronous Built-In Replication
4
What Happens with Two Active Nodes?
Node1
INSERT SET id=42
Node2
INSERT SET id=42
INSERT
6
Node1
INSERT SET id=42
Sends the update →
Node2
INSERT SET id=42
← Sends the update
INSERT
6
Node1
INSERT SET id=42
Sends the update →
Receives the update →
Node2
INSERT SET id=42
← Sends the update
← Receives the update
INSERT
6
Node1
INSERT SET id=42
Sends the update →
Receives the update →
Duplicate key error!
Node2
INSERT SET id=42
← Sends the update
← Receives the update
Duplicate key error!
INSERT
6
Node1
DELETE WHERE id=42
Node2
DELETE WHERE id=42
DELETE
7
Node1
DELETE WHERE id=42
Sends the update →
Node2
DELETE WHERE id=42
← Sends the update
DELETE
7
Node1
DELETE WHERE id=42
Sends the update →
Receives the update →
Node2
DELETE WHERE id=42
← Sends the update
← Receives the update
DELETE
7
Node1
DELETE WHERE id=42
Sends the update →
Receives the update →
Key not found error!
Node2
DELETE WHERE id=42
← Sends the update
← Receives the update
Key not found error!
DELETE
7
Node1
UPDATE SET i=42
Node2
UPDATE SET i=25
UPDATE
8
Node1
UPDATE SET i=42
Sends the update →
Node2
UPDATE SET i=25
← Sends the update
UPDATE
8
Node1
UPDATE SET i=42
Sends the update →
Receives the update →
Node2
UPDATE SET i=25
← Sends the update
← Receives the update
UPDATE
8
Node1
UPDATE SET i=42
Sends the update →
Receives the update →
i=25
Node2
UPDATE SET i=25
← Sends the update
← Receives the update
i=42
UPDATE
8
Why to Write to Multiple Nodes?
• Each node for its own purpose
Application 1 Application 2
Data Distribution
10
• Multiple machines for better scalability
Node 1 Node 2
[Read] Scale
11
• Nodes in different geographical regions
Belgium Brazil
GEO Distribution
12
• Waits when the main master deads
• Then accepts writes
Active Hot Standby
Hot Standby
13
Galera, PXC, InnoDB Cluster
Node1
UPDATE ...
Node2,3,...
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Node2,3,...
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Waits
Node2,3,...
← Receives the packet
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Waits
Waits
Node2,3,...
← Receives the packet
Applies changes
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Node2,3,...
← Receives the packet
Applies changes
← Confirms
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Receives confirmation →
Node2,3,...
← Receives the packet
Applies changes
← Confirms
Fully Synchronous Cluster
15
Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Receives confirmation →
Ready for new update
Node2,3,...
← Receives the packet
Applies changes
← Confirms
Fully Synchronous Cluster
15
• Speed of the slowest member
Synchronization Penalty
16
• Speed of the slowest member
• Writes or reads should wait
Synchronization Penalty
16
• Speed of the slowest member
• Writes or reads should wait
• Tunable
Synchronization Penalty
16
• Speed of the slowest member
• Writes or reads should wait
• Tunable
• Galera
• wsrep sync wait
• InnoDB Cluster
• group replication consistency
Synchronization Penalty
16
• Synchronization takes too much time
• Nodes often disconnect
GEO Replication is Painful
17
How to Setup Safe Asynchronous Master-Master
• Business logic
• Databases
• Tables
• Rows
Divide and Rule
19
• Each server writes to its own set of tables
Node 1 Node 2
D1
D2
D3
D4
D1
D2
D3
D4
Databases and Tables
20
• Agreement on which rows
Node 1 Node 2
Row 1
Row 2
Row 3
Row 4
Row 1
Row 2
Row 3
Row 4
Rows
21
• Agreement on which rows
•
E.g. AUTO INCREMENT
• auto increment increment=NUMBER OF SERVERS
• auto increment offset = server id
• Custom index
• Create your own unique pattern for the each node
Rows
21
• Hot Standby
•
Easy to setup and maintain
•
Separating writes by database/table
• Comparatively easy to setup and maintain
• Separating writes by row
• Should be used very carefully
Conclusion
22
MySQL User Reference Manual
Why MySQL Replication Fails, and How to...
MySQL High Availability
More information
23
www.slideshare.net/SvetaSmirnova
twitter.com/svetsmirnova
github.com/svetasmirnova
Thank you!
24
DATABASE PERFORMANCE
MATTERS

How Safe is Asynchronous Master-Master Setup?