In the first part of Galera Cluster best practices series, we will discuss the following topics: * ongoing monitoring of the cluster and detection of bottlenecks; * fine-tuning the configuration based on the actual database workload; * selecting the optimal State Snapshot Transfer (SST) method; * backup strategies (video:http://galeracluster.com/videos/2159/)

1. Galera Cluster Best Practices
for DBAs and DevOps
Philip Stoev
Codership Oy

2. Agenda
• A very quick overview of Galera Cluster
• Ongoing monitoring of the cluster and detection of
bottlenecks
• Backup strategies
• Selecting the optimal State Snapshot Transfer (SST)
method

3. Galera Cluster Overview
Synchronous
– each transaction is immediately replicated on all nodes at commit
– no stale slaves
Multi-Master
– read from and write to any node
– automatic transaction conflict detection
Replication
– a copy of the entire dataset is available on all nodes
– new nodes can join automatically
For MySQL
– based on a modified version of MySQL (5.5, 5.6 with 5.7 coming up)
– InnoDB storage engine

4. And more …
• Recovers from node failures within seconds
• Data consistency protections
– avoids reading stale data
– prevents unsafe data modifications
• Cloud and WAN support

5. Monitoring Galera Cluster

6. General Principles of Monitoring
• Galera Cluster is first and foremost MySQL + InnoDB:
– all traditional advice still applies (e.g. slow query log)
– InnoDB still does most of the heavy lifting (e.g. I/O)
• Galera reports metrics via SHOW GLOBAL STATUS
– FLUSH STATUS resets counters
• Galera reports events via the MySQL server error log
– good idea to check you logs or log rotation scripts
• Monitor each node separately for maximum visibility

7. Monitoring the Network
Especially in WAN or complex network environments:
• monitor the health of the network using a third-party
tool:
– round-trip (ping) times
– bandwidth utilization
• monitor each network link between any two segments
separately
– Galera assumes all links perform equally well

8. Detecting Issues With Cluster Health
• SHOW GLOBAL STATUS variables:
MySQL [test]> show status like '%wsrep%';
+------------------------------+-------------------------------------------+
| Variable_name | Value |
+------------------------------+-------------------------------------------+
| wsrep_ready | ON |
| wsrep_cluster_status | Primary (or non-Primary) |
| wsrep_local_state_comment | Synced (or Donor/Desynced, Joiner) |
| wsrep_cluster_size | 3 |
+------------------------------+-------------------------------------------+

9. Detecting Galera-Specific Bottlenecks
MySQL [test]> show global status like '%wsrep%';
+------------------------------+--------------------------------------------
+
| Variable_name | Value
|
+------------------------------+--------------------------------------------
+
| wsrep_flow_control_paused | 0.100000
|
| wsrep_flow_control_sent | 123
|
| wsrep_local_bf_aborts | 77
|
| wsrep_local_cert_failures | 24
|
+------------------------------+--------------------------------------------
+
In case of BF aborts or cert failures:

10. Detecting Galera-Specific Bottlenecks #2
• SHOW PROCESSLIST:
MySQL [test]> show processlist;
+----+------+-----------+------+---------+------+----------+---------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------+------+---------+------+----------+---------------------------+
...
| 5 | root | localhost | test | Query | 10 | query end| INSERT INTO t1 VALUES (1) |
...
3 rows in set (0.05 sec)

11. Backups

12. Backups
• Multiple nodes do not remove the need for backups
– backups also help in case of human error
• All nodes in Galera have the same information, so
backing up one node is sufficient
• Galera does not use the binary log files, so generate
and back those up only if you have a specific need
• Practicing the restore operation is highly recommended

13. Performance Considerations
• Backups slow down the node being backed up
– which in turn can cause the entire cluster to slow down
• or reduce the number of available nodes in the cluster
– can affect quorum calculations
• you can have an async slave and take backups from it
– Just do not let it fall far behind the master cluster

14. XtraBackup
Non-blocking backup for InnoDB
• --galera-info option records the precise position
when the backup was taken in a file named
xtrabackup_galera_info
• a short lock is still taken

15. Backup Procedure with a Dedicated Node
• Make sure backup node will get no write traffic
– make sure all existing sessions are disconnected
• Temporarily disconnect a node from the cluster
– SET GLOBAL wsrep_provider = ‘none’
• Perform backup
• Restore the value of wsrep_provider; check for
wsrep_ready=ON

16. Restoring Single Node from Backup
Option #1:
– wipe the data directory clean and restart
– let it rejoin the cluster via SST
– the operation will involve a donating node as well
Option #2:
– use a backup that was recently taken
– wipe the data directory clean and restore data
– put the information from xtrabackup_galera_info into grastate.dat
# GALERA saved state
version: 2.1
uuid: 021a77ed-80cf-11e6-9e8e-6249ad0d3a57
seqno: 1234
cert_index:
– restart node and it will catch up via IST

17. Restoring Entire Cluster #1
Note: a new logical cluster is being created
Easiest approach:
1. Restore data on one node and restart first node with –
wsrep-new-cluster
2. Start one more node. Node rejoins via SST, with first
node as donor
3. Restart two more nodes, first two nodes serve as
donors, and so forth.

18. Restoring Entire Cluster #2
Optimized approach:
1. Restore backup on first node, bootstrap 1-node
cluster. Prevent new transactions from being issued.
2. Restore backup on other nodes and create
grastate.dat on them:
# GALERA saved state
version: 2.1
uuid: <new-cluster-uuid>
seqno: 0
cert_index:

19. SST

20. General Principles
• It is always best to avoid SST altogether by sizing
gcache.size appropriately
• SST choice is determined by:
– availability of a dedicated donating node
– encryption requirements (SST encryption is configured
separately from other cluster operations)
– bandwidth utilization

21. SST Methods
• Rsync (the default)
– causes the donor to block new transactions
• XtraBackup
– Donor remains operational
• Mysqldump
– logical, rather than physical database copy

22. Configuration without a Dedicated Donor
• Use xtrabackup-v2 SST method
– requires the installation of XtraBackup
– there is still performance impact on the donor
– encryption, compression are available

23. Configuration with Dedicated Donor
• rsync may be the best method to use:
– available on any machine
– compression and partial file transfers are possible
• use wsrep_sst_donor to specify donor node to use:
– takes a list of node names as configured with
wsrep_node_name
– if the list terminates with a comma, other nodes can also be
used for donors if none of the nodes from the list are available
• if using an arbitrator, consider switching it to a full
Galera node that can then act as a dedicated donor

24. Questions
• Please use the Question/Chat box in the GoToWebinar
panel
• Ideas welcome for future webinars

25. Thank You
http://www.galeracluster.com
Discussion group:
codership-team@googlegroups.com