Quite often, a single MySQL instance isn't enough, it rather should be several. The typical reasons are increased throughput, by distributing the load on more than one node, or high availability, by avoiding a single point of failure. These goals can be achieved by setting up replication among several servers running MySQL, or by combining them to form a Galera cluster. This talk will describe both approaches, compare them to each other, and assist with the decision which of these approaches will fit better to the user's goals and situation.

1. www.fromdual.com
1 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Servers Working as a Team -
Replication or Galera Cluster
Open Source Data Center Conference, April 26th
- 28th
, Berlin
Jörg Brühe
Senior Support Engineer, FromDual GmbH
joerg.bruehe@fromdual.com
CC-BY-SA

2. www.fromdual.com
2 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
FromDual GmbH
Support
remote-DBA
Training
Consulting

3. www.fromdual.com
3 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
About Me
● Development distributed SQL-DBMS
Porting mainframe -> Unix,
Interface to archiver tools (ADSM, NetWorker)
● MySQL Build Team
Release builds incl. tests, packaging, scripts, ...
● DBA
MySQL running a web platform
(master-master-replication)
● Support-Engineer (FromDual)
Support + Remote-DBA for MySQL / MariaDB / Percona
both with and without Galera Cluster

4. www.fromdual.com
4 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Contents
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

5. www.fromdual.com
5 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
General Remarks
● Concepts rather than details:
”the forest, not the trees“
● MySQL 5.6 (established GA version)
● Also valid for Percona and MariaDB
● Not applicable to ”embedded“ MySQL
● Not considered: NDB = ”MySQL Cluster“

6. www.fromdual.com
6 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

7. www.fromdual.com
7 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Client-Server-DBMS
App App App
Server
Disk
Client (application)
local or remote
Socket, LAN or internet
Server is separate process,
multi-threaded:
1 thread per user session
Disk / SSD, local or SAN

8. www.fromdual.com
8 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Inside the Server
mysqld
MySQL ist eine multi-Thread
und NICHT eine multi-Prozess
Applikation!
Application / Client
Thread
Cache
Connection
Manager
User Au-
thentication
Command
Dispatcher
Logging
Query Cache
Module
Query
Cache
Parser
Optimizer
Access Control
Table Manager
Table Open
Cache (.frm, fh)
Table Definition
Cache (tbl def.)
Handler Interface
MyISAM Memory NDB PBXTInnoDB ...Aria XtraDB Federated-X

9. www.fromdual.com
9 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

10. www.fromdual.com
10 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Layers + Binlog
MyISAM
InnoDB
...
Userthread2
Userthread1
...
SQL layer:
- Parser
- Optimizer
- Privileges
- Query Cache
- …
Handler Interface
File layer:
- Table Handler
- InnoDB:
- Row Access
- Row Locks
- Recovery
- ...
Binlog:
All data
and schema
changes

11. www.fromdual.com
11 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Binlog
● All data changes executed
● All schema changes executed
● Timestamps
● Essential for Point-in-Time-Recovery ”PITR“
● Independent of table handler
● Formats ”statement“, ”row“, and ”mixed“
● Segments of configurable size
● Numbered sequentially

12. www.fromdual.com
12 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

13. www.fromdual.com
13 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Replication
● Applications communicate with ”Master“
● ”Master“ logs all changes
● ”Slave“ has identical initial state
● Slave fetches all changes from master
and applies them locally
● Replication is running asynchronous
● Slave stops replication on difference

14. www.fromdual.com
14 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Slave fetches binlog
Userthread2
Userthread1
...
Userthread3
MyISAM
InnoDB
...
Userthread2
Userthread1
...
SQLthread
IOthread
BinlogRelay
Log
MyISAM
InnoDB
...
Slave:
“log-bin = FILE”, or else no binlog
“log_slave_updates = 1” for forwarding
Master:
“log-bin = FILE”, or else no binlog
(no master function)
Binlog

15. www.fromdual.com
15 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Typical Usage
● ”High Availability“
● Geographic redundancy
● Support higher read load
(= ”read scale-out“)
● Read-only instance(s)
e.g. for backup or reports
● Intentional delay is possible
● Filtering (by DB or table) is possible

16. www.fromdual.com
16 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Replication Cascade
Userthread2
Userthread1
...
Userthread3
MyISAM
InnoDB
...
MyISAM
InnoDB
...
MyISAM
InnoDB
...
SQLthread
IOthread
Userthread2
Userthread1
...
SQLthread
IOthread
Userthread2
Userthread1
...
● Recommended: ”read-only = 1“ on slave
”log_slave_updates = 1“
● Multiple slaves per master are possible

17. www.fromdual.com
17 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Entries in Binlog
Original:
● Identification by file name and position
● Replication: ”change master to ...“ specifying
host, port, user, password, file, position
● See also: ”mysqldump ­­master­data“
From MySQL 5.6 also:
● GTID = ”Global Transaction ID“
● Replication: ”change master to ...“ specifying
host, port, user, password, ”auto_position = 1“

18. www.fromdual.com
18 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Master-Master-Replication
Userthread2
Userthread1
...
Userthread3
MyISAM
InnoDB
...
Userthread2
Userthread1
...
SQLthread
IOthread
BinlogRelay
Log
MyISAM
InnoDB
...
Binlog
SQLthread
IOthread
Relay
Log
● Overlapping changes are fatal!

19. www.fromdual.com
19 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Notes about Replication
● Master-Master is controversial, be careful!
● Replication increases read throughput,
but not/barely write throughput
● Replication causes file IO und network load
● Format ”row“ is more efficient, but less readable
● Multi-threaded replication since MySQL 5.6,
multi-master (”multi-source“) coming in MySQL 5.7
● Big installation: booking.com
● Recommended: datacharmer.blogspot.de
(Giuseppe Maxia, August 2015)

20. www.fromdual.com
20 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

21. www.fromdual.com
21 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Replication Weaknesses
● Asynchronous
● Asymmetrical
● Only one write node
● Parallel writes may cause breakage
● HA needs failover after node crash
● Each node is SPOF for its slaves,
breakdown requires structure change
● Dynamic changes are complicated

22. www.fromdual.com
22 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Better Alternative
● Synchronous transfer
● Symmetrical cluster
● Write accesses on all nodes
● Distributed conflict analysis and handling
● HA by continuity after node outage
● Dynamic entry / exit of nodes supported

23. www.fromdual.com
23 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Galera Cluster
App App App
Load balancing (LB)
Node 2 Node 3Node 1
wsrep
Galera replication
wsrep wsrep
Inclusiding outage detection
and redirection for HA
“Working Set Replication”
Dedicated network preferred
Locale disks,
each holding all data
= =
“shared nothing” architecture

24. www.fromdual.com
24 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Galera Properties (1)
+ Based on InnoDB (due to transactions and rollback)
+ Also transfers user definitions, privileges, ...
+ Quasi-synchronous transfer on commit,
check for conflicts, efficient
+ Symmetrical, HA without server failover, quorum
+ No loss of transactions
+ Brings read scale-out, also some write increase
+ Dynamical entry / exit possible,
synchronisation is automated

25. www.fromdual.com
25 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Order of Events
Graph by
Vadim Tkachenko
(Percona):
http://www.mysqlperformanceblog.com/2012/01/19/
percona-xtradb-cluster-feature-2-multi-master-replication/

26. www.fromdual.com
26 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Galera Properties (2)
- MySQL sources need patching
(Codership offers binaries, ditto MariaDB and Percona)
- Beware of hot spots (rows)
- Conflict detection is late, full rollback
(Check postponed till commit)
- Minimum size is 3 nodes
- Synchronisation time for large DB
(mysqldump -> xtrabackup or rsync)

27. www.fromdual.com
27 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Certifiation at Commit
http://galeracluster.com/documentation-webpages/certificationbasedreplication.html

28. www.fromdual.com
28 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

29. www.fromdual.com
29 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Team of MySQL Servers
● Alternatives: Replication or Galera Cluster
● Redundancy of machine and storage
● HA
● Scale-out, esp. for read load
● Instances for reports, analysis, backup
● Data available locally (branch offices, ...)

30. www.fromdual.com
30 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Comparison (1)
Replication Galera
Standard Add-on product
All handlers InnoDB only
Upwards compatible Same versions
Minimum 2 nodes Minimum 3 nodes
HA by failover HA without changes
Communication:
Hierarchical, chain Symmetrical, parallel
Asynchronous Quasi-synchronous
Delay is configurable Immediate
Filtering is configurable Complete

31. www.fromdual.com
31 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Comparison (2)
Replication Galera
Read scale-Out Read scale-Out
Write unchanged Write increased
1 Master:
1* write 1* write
Local conflict :
Error on statement Error on statement
n Master:
n* write n* write
Distributed conflict:
Replication breakdown Rollback on commit

32. www.fromdual.com
32 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Comparison (3)
Replication Galera
Short interruption:
Replication resumes IST (incremental transfer)
Long interruption:
Replication resumes SST (full transfer)
Structure change:
Manual / separate Tool Automatic / dynamic
Initial setup:
Snapshot, Full transfer,
master remains available donor may be blocked

33. www.fromdual.com
33 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
CAP Theorem
”For a distributed computer system,
it is impossible to simultaneously provide
all three of the following guarantees:
● C = Consistency (identical data throughout)
● A = Availability (system is operational)
● P = Partition Tolerance (network outage)“
Eric Brewer, 1998 ff.
https://en.wikipedia.org/wiki/CAP_theorem

34. www.fromdual.com
34 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Prospect: MySQL 5.7
● MySQL 5.7 is GA (5.7.9, 2015-Oct-21)
● Replication like in MySQL 5.6,
added: multi-source replication
(one slave reading from several masters)
● Codership is working on adding
Galera Cluster to MySQL 5.7
● Oracle is working on ”Group replication“,
currently available as ”labs release“
(= ”not fit for production“)

35. www.fromdual.com
35 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
MySQL Server: Architecture
Binlog
Replication
Galera Cluster
Comparison
Examples / When (not) to Choose Which

36. www.fromdual.com
36 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Communication Outage (1)
Galera Cluster:
● Isolated node has no quorum
=> will not serve applications
● Quorum is at risk!
● Active nodes write ”gcache“ to files,
storage period?
● Switch to SST threatens

37. www.fromdual.com
37 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Communication Outage (2)
Replication:
● Master writes log segments to files
● IO-Thread asks to read from binlog position /
GTID, retries periodical until successful
● Avoid ”purge log“!
Replication is more tolerant
than Galera Cluster !

38. www.fromdual.com
38 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Global Production
Solution:
Replication with
filtering
Requirement:
Head office (D) and
factories (BR, CN, ...)
with selective transfer

39. www.fromdual.com
39 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Parallel Writes + Conflict
Galera:
● Retry of autocommit statements configurable
● Transaction conflict causes rollback
=> Application repeats complete transaction
Replication:
● Slave detects conflict, no contact to application
=> Replication stops
Replication needs admin action on conflict !

40. www.fromdual.com
40 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Hot Spot
● Replication: frequent aborts
● Galera: frequent rollbacks
=> Agree on a single write node !

41. www.fromdual.com
41 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
High Availability
Replication:
● Failover manual (reaction time) or automated
(correct?)
● Slave lag, selection of new master
Galera:
● Symmetrical, no change of roles
● Virtually synchronous replication (no lag)
=> Advantage Galera

42. www.fromdual.com
42 / 42MySQL Teamwork: Replication or Galera Cluster, joerg.bruehe@fromdual.com, 2016 April, CC-BY-SA
Q & A
Questions ?
Discussion?
● FromDual provides neutral and independent:
● Consulting
● Remote-DBA
● Support for MySQL, Galera, Percona Server and MariaDB
● Training
www.fromdual.com/presentations