This document discusses redundancy models for MySQL, MariaDB, MongoDB and TokuMX databases. It covers asynchronous replication used in MySQL replication and MongoDB/TokuMX compared to synchronous replication in Galera and NDB Cluster. The document then zooms into recovery procedures for Galera clusters and discusses how to prevent split-brain situations in multi-datacenter setups through the use of additional nodes and assigning node weights.

1. Repair and Recovery for your
MySQL, MariaDB and MongoDB/
TokuMX clusters
January 2013
Johan Andersson
Severalnines AB
johan@severalnines.com
Mobile +46 73 073 60 99

2. Copyright 2014 Severalnines AB
Agenda
! Redundancy models for Galera, NDB and MongoDB/
TokuMX
! Failover & Recovery (Automatic vs Manual)
! Zooming into Galera recovery procedures
! Split brains in multi-datacenter setups
2

3. Copyright 2014 Severalnines AB
Redundancy models for Galera, NDB
and MongoDB/TokuMX
! Database Clustering can be divided in two areas
! Asynchronously replicated data
! MySQL Replication
! TokuMX/MongoDB
! Synchronously replicated data
! Galera for MySQL
! NDB Cluster (aka MySQL Cluster)
3

4. Copyright 2014 Severalnines AB
Asynchronous
1
2
3

5. Copyright 2014 Severalnines AB
Asynchronous
Pros
Cons
! Read scaling is easy
! Master handles all writes
! Read write splitting
! Lower write latency
! Automated failover is hard:
! Elect a new master
! Slave lag & Data loss
! Has everything been sent
to the slave when the
master crashes?
! Has the slave applied all
data?
5

6. Copyright 2014 Severalnines AB
Synchronous
1
4
2
3

7. Copyright 2014 Severalnines AB
Synchronous
Pros
Cons
! All servers are masters
! Higher write latency
! Application can read and
write on any server
! Sensitive to link latency
! No data loss
! Network partitioning
! Simple failover for apps
7

8. Copyright 2014 Severalnines AB
Redundancy Models
TokuMX/MongoDB ReplicaSet (1/2)
! Up to 12 members
! Max 7 voting numbers
! Primary, Secondary, Arbiter
! Slave Delay
! Chaining Allowed
! Heartbeats every 2 seconds
! 5 missed heartbeats: node dead
Primary
Secondary
Arbiter
8

9. Copyright 2014 Severalnines AB
Redundancy Models
TokuMX/MongoDB ReplicaSet (2/2)
! Automatic failover
! Priority (between 0 and 100.0)
! Primary has highest priority
! Influences election of new primary
Data Center 1
Primary
Secondary
Data Center 2
Secondary
9

10. Copyright 2014 Severalnines AB
Redundancy Models
TokuMX/MongoDB Sharded Cluster
! Built-in support for sharding
! Min 2 Query Routers
! 3 Config Servers
Primary Secondary
Arbiter
Primary Secondary
Arbiter
10

11. Copyright 2014 Severalnines AB
Redundancy Models
MySQL Replication
! No hard limits on no. of slaves
! Replication chaining
! Delayed slave
! Manual failover
Slave
Master
Slave
11

12. Copyright 2014 Severalnines AB
Redundancy Models
MySQL Cluster
! In-built sharding
! Separate SQL and Data layers
! Up to 4 replicas/node group
! Max 48 data nodes
! Auto-failover
Node Group 1
Node Group 2
12

13. Copyright 2014 Severalnines AB
Redundancy Models
Galera Cluster
! Synchronous replication plugin
! All masters have same dataset
! Slowest node determines
speed of cluster
! No hard limits on no. of nodes
! Auto-failover
! Galera arbitrator (garbd)
Master
Master
Master
13

14. Copyright 2014 Severalnines AB
Synchronous Replication
Linear Two-phase commit
! Used by NDB data nodes
! Pessimistic locking
! Works well for cases
with update
contention
1
4
6
3
2
5
1. Prepare transaction
4. commit
14

15. Copyright 2014 Severalnines AB
Synchronous Replication
Certificate-based
! Used by Galera/WSREP
! Optimistic locking
! Not optimal for cases
with conflicting
updates
1
2
3
4
1. begin transaction
2. commit
15

16. Copyright 204Severalnines AB
Zooming into Galera recovery
procedures
By Seppo Jaakola @ Codership
16

17. Copyright 2014 Severalnines AB
Split-brain
! Network partitioning (aka) split brain. N nodes divided
into M equal parts
Galera Servers - SYNCED
DC1
WAN
Galera Servers - SYNCED
DC2
17

18. Copyright 2014 Severalnines AB
Split-brain
! WAN breaks - 2 equal splits (Network partitioning)
! Non-Primary – cannot handle transactions
Galera – Non-Primary
DC1
WAN
Galera – Non-Primary
DC2
18

19. Copyright 2014 Severalnines AB
Split-brain prevention
! Solution: The side with the majority of nodes will win:
! Add garbd or a galera node to DC1 or DC2
! Add weight to existing node in DC1:
wsrep_provider_options=“pc.weight=N;” (default = 1 )
Galera – Synced
DC1
WAN
Galera – Synced
DC2
19

20. Copyright 2014 Severalnines AB
Split-brain prevention
! Add weight to existing node in DC1:
wsrep_provider_options=“pc.weight=2;”
Galera – Synced
DC1
pc.weight=2
WAN
1
3
Galera – Synced
DC2
1
1
2
20

21. Copyright 2014 Severalnines AB
Split-brain prevention
! Final Two Data center setup with an additional node.
Galera – Synced
DC1
WAN
Galera – Synced
DC2
21

22. Copyright 2014 Severalnines AB
Split-brain prevention
! Three data centers, split brain unlikely.
! Galera 3.0 supports segments (gmcast.segment)
! Only one node in each segment will relay to only one other
node in another segment
! Segments also applies to Two Data centers, and between
racks in one DC.
WAN
Galera – Segment 3
DC2
Galera – Segment 1
DC1
Galera – Segment 2
DC3

23. Copyright 2014 Severalnines AB
Q&A
23

24. Copyright 2014 Severalnines AB
Thank you.
For further questions please contact John-Jerome Schmidt:
jj@severalnines.com
24