Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Galera Cluster Best Practices for DBA's and DevOps Part 1


Published on

In the first part of Galera Cluster best practices series, we will discuss the following topics:

* ongoing monitoring of the cluster and detection of bottlenecks;
* fine-tuning the configuration based on the actual database workload;
* selecting the optimal State Snapshot Transfer (SST) method;
* backup strategies


Published in: Software
  • Be the first to comment

  • Be the first to like this

Galera Cluster Best Practices for DBA's and DevOps Part 1

  1. 1. Galera Cluster Best Practices for DBAs and DevOps Philip Stoev Codership Oy
  2. 2. Agenda • A very quick overview of Galera Cluster • Ongoing monitoring of the cluster and detection of bottlenecks • Backup strategies • Selecting the optimal State Snapshot Transfer (SST) method
  3. 3. Galera Cluster Overview Synchronous – each transaction is immediately replicated on all nodes at commit – no stale slaves Multi-Master – read from and write to any node – automatic transaction conflict detection Replication – a copy of the entire dataset is available on all nodes – new nodes can join automatically For MySQL – based on a modified version of MySQL (5.5, 5.6 with 5.7 coming up) – InnoDB storage engine
  4. 4. And more … • Recovers from node failures within seconds • Data consistency protections – avoids reading stale data – prevents unsafe data modifications • Cloud and WAN support
  5. 5. Monitoring Galera Cluster
  6. 6. General Principles of Monitoring • Galera Cluster is first and foremost MySQL + InnoDB: – all traditional advice still applies (e.g. slow query log) – InnoDB still does most of the heavy lifting (e.g. I/O) • Galera reports metrics via SHOW GLOBAL STATUS – FLUSH STATUS resets counters • Galera reports events via the MySQL server error log – good idea to check you logs or log rotation scripts • Monitor each node separately for maximum visibility
  7. 7. Monitoring the Network Especially in WAN or complex network environments: • monitor the health of the network using a third-party tool: – round-trip (ping) times – bandwidth utilization • monitor each network link between any two segments separately – Galera assumes all links perform equally well
  8. 8. Detecting Issues With Cluster Health • SHOW GLOBAL STATUS variables: MySQL [test]> show status like '%wsrep%'; +------------------------------+-------------------------------------------+ | Variable_name | Value | +------------------------------+-------------------------------------------+ | wsrep_ready | ON | | wsrep_cluster_status | Primary (or non-Primary) | | wsrep_local_state_comment | Synced (or Donor/Desynced, Joiner) | | wsrep_cluster_size | 3 | +------------------------------+-------------------------------------------+
  9. 9. Detecting Galera-Specific Bottlenecks MySQL [test]> show global status like '%wsrep%'; +------------------------------+-------------------------------------------- + | Variable_name | Value | +------------------------------+-------------------------------------------- + | wsrep_flow_control_paused | 0.100000 | | wsrep_flow_control_sent | 123 | | wsrep_local_bf_aborts | 77 | | wsrep_local_cert_failures | 24 | +------------------------------+-------------------------------------------- + In case of BF aborts or cert failures:
  10. 10. Detecting Galera-Specific Bottlenecks #2 • SHOW PROCESSLIST: MySQL [test]> show processlist; +----+------+-----------+------+---------+------+----------+---------------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+----------+---------------------------+ ... | 5 | root | localhost | test | Query | 10 | query end| INSERT INTO t1 VALUES (1) | ... 3 rows in set (0.05 sec)
  11. 11. Backups
  12. 12. Backups • Multiple nodes do not remove the need for backups – backups also help in case of human error • All nodes in Galera have the same information, so backing up one node is sufficient • Galera does not use the binary log files, so generate and back those up only if you have a specific need • Practicing the restore operation is highly recommended
  13. 13. Performance Considerations • Backups slow down the node being backed up – which in turn can cause the entire cluster to slow down • or reduce the number of available nodes in the cluster – can affect quorum calculations • you can have an async slave and take backups from it – Just do not let it fall far behind the master cluster
  14. 14. XtraBackup Non-blocking backup for InnoDB • --galera-info option records the precise position when the backup was taken in a file named xtrabackup_galera_info • a short lock is still taken
  15. 15. Backup Procedure with a Dedicated Node • Make sure backup node will get no write traffic – make sure all existing sessions are disconnected • Temporarily disconnect a node from the cluster – SET GLOBAL wsrep_provider = ‘none’ • Perform backup • Restore the value of wsrep_provider; check for wsrep_ready=ON
  16. 16. Restoring Single Node from Backup Option #1: – wipe the data directory clean and restart – let it rejoin the cluster via SST – the operation will involve a donating node as well Option #2: – use a backup that was recently taken – wipe the data directory clean and restore data – put the information from xtrabackup_galera_info into grastate.dat # GALERA saved state version: 2.1 uuid: 021a77ed-80cf-11e6-9e8e-6249ad0d3a57 seqno: 1234 cert_index: – restart node and it will catch up via IST
  17. 17. Restoring Entire Cluster #1 Note: a new logical cluster is being created Easiest approach: 1. Restore data on one node and restart first node with – wsrep-new-cluster 2. Start one more node. Node rejoins via SST, with first node as donor 3. Restart two more nodes, first two nodes serve as donors, and so forth.
  18. 18. Restoring Entire Cluster #2 Optimized approach: 1. Restore backup on first node, bootstrap 1-node cluster. Prevent new transactions from being issued. 2. Restore backup on other nodes and create grastate.dat on them: # GALERA saved state version: 2.1 uuid: <new-cluster-uuid> seqno: 0 cert_index:
  19. 19. SST
  20. 20. General Principles • It is always best to avoid SST altogether by sizing gcache.size appropriately • SST choice is determined by: – availability of a dedicated donating node – encryption requirements (SST encryption is configured separately from other cluster operations) – bandwidth utilization
  21. 21. SST Methods • Rsync (the default) – causes the donor to block new transactions • XtraBackup – Donor remains operational • Mysqldump – logical, rather than physical database copy
  22. 22. Configuration without a Dedicated Donor • Use xtrabackup-v2 SST method – requires the installation of XtraBackup – there is still performance impact on the donor – encryption, compression are available
  23. 23. Configuration with Dedicated Donor • rsync may be the best method to use: – available on any machine – compression and partial file transfers are possible • use wsrep_sst_donor to specify donor node to use: – takes a list of node names as configured with wsrep_node_name – if the list terminates with a comma, other nodes can also be used for donors if none of the nodes from the list are available • if using an arbitrator, consider switching it to a full Galera node that can then act as a dedicated donor
  24. 24. Questions • Please use the Question/Chat box in the GoToWebinar panel • Ideas welcome for future webinars
  25. 25. Thank You Discussion group: