Galera cluster for MySQL - Introduction Slides
•
50 likes•144,259 views
This set of slides gives you an overview of Galera, configuration basics and deployment best practices. The following topics are covered: - Concepts - Node provisioning - Network partitioning - Configuration example - Benchmarks - Deployment best practices - Galera monitoring and management
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Galera cluster for MySQL - Introduction Slides
Similar to Galera cluster for MySQL - Introduction Slides (20)
More from Severalnines
More from Severalnines (20)
Recently uploaded
Recently uploaded (20)
Galera cluster for MySQL - Introduction Slides
- 1. Multi-Master Synchronous Replication Galera Cluster for MySQL August 2012 Alex Yu VP Products alex@severalnines.com Confidential
- 2. Copyright Severalnines AB Agenda About Severalnines What is Galera Replication? Galera Concepts Node Provisioning Network partitioning/Split brain Configuration Example Benchmarks & Performance Metrics Best Practices Monitoring and Management Confidential 2
- 3. Copyright Severalnines AB About Us Stockholm, Tokyo and Singapore Database Automation and DBaaS software vendor Over 7,000 deployments to date Commercial product launched Q1 2011 Winner Best Startup EuroCloud Europe 2011 Launched Europe’s first Data Cloud in Nov 2011 Press coverage 2011: CIO Magazine, eWeek, PC-World, IDG News, Le Figaro, LeMondeInformatique, heise.de, Computerwelt, silicon.de, etc … Confidential 3
- 4. Copyright Severalnines AB What is Galera Replication? Synchronous (Virtually) Multi-Master Replication Read and Write on any Node No Master Failover! No Slave Lag! Application MySQL Server Guaranteed write consistency WSREP API WSREP API Cluster wide conflicts resolution (certification) WSREP Provider wsrep plugin Highly Available and Scalable Replication Replication No SPOF Read and Write (Parallel Applier threads) scalability Geographical Replication (Mix MySQL Async & Galera Sync) Cluster (Group Communication Protocol) Automatic Node Provisioning, QoS Confidential 4
- 5. Copyright Severalnines AB Galera Cluster for MySQL Codership patches for MySQL Binaries and source available at launchpad InnoDB (& MyISAM experimental) Client Client Client No need to change DB schema/queries Local queries LB Parallel Replication! Multiple Applier Threads (1-512) R/W R/W R/W MySQL MySQL MySQL Row events, row level locks [WSREP] [WSREP] [WSREP] Asynchronous Replication Galera Replication (Synchronous) In/Out of the cluster Confidential 5
- 6. Copyright Severalnines AB Galera Cluster for MySQL cont. Higher probability for “deadlocks” Cluster wide optimistic locking Locking conflicts detected at commit Client Client Client First to commit succeeds Minimum 3 nodes required LB “Donor” node blocks writes during full synch of joining/recovering node R/W R/W R/W 3rd node then is available for service MySQL MySQL MySQL [WSREP] Gotchas: 2 recovering nodes will block the last node [WSREP] [WSREP] Replication performance dependent on Galera Replication (Synchronous) Network latency Performance of the “slowest” or the farthest Node (RTT) Number of deployed nodes Confidential 6
- 7. Copyright Severalnines AB Synchronous Replication Transaction t1 Node 1 BEGIN COMMIT (REQ) COMMIT (ACK/returns) Statements Commit response time time COMMIT or Rollback WS Replication event OK or Conflict Node 2 Transaction applied (virtually synchronous) WS time Certification Apply event Node 3 Transaction applied (virtually synchronous) WS time Certification Apply event All nodes 100% sync Confidential 7
- 8. Copyright Severalnines AB Galera Concepts Application State A set of data that application decides to replicate Default is the whole MySQL databases. Every node is a complete replica Application state is identified by a Global Transaction ID Global Transaction ID (GTID) f7720ae0-6f9b-11e1-0800-598d1b386dce:32520198989 CLUSTER/HISTORY/STATE UUID:TRX/STATE/SEQNO All replicated transactions can be uniquely referenced in any node Initial state: f7720ae0-6f9b-11e1-0800-598d1b386dce:0 Undefined state: 00000000-0000-0000-0000-000000000000:-1 Confidential 8
- 9. Copyright Severalnines AB Galera Concepts cont. MySQL [WSREP] Primary Component - PC The whole cluster is a PC during normal operation Node and network failures MySQL [WSREP] MySQL [WSREP] Splits clusters into several components Primary Component Only PC can continue to modify state Quorum algorithm invoked to select a PC during cluster partitioning Majority rules Minority tries to reconnect with PC Confidential 9
- 10. Copyright Severalnines AB Galera Concepts cont. State Snapshot Transfer - SST A transfer of a consistent snapshot of a node state corresponding to a certain GTID Initialize the state of a newly joining cluster node from an already initialized node (donor) Incremental State Transfer - IST Catch up with the cluster by replaying missing transactions Known initial node state Enough transactions cached at the donor Confidential 10
- 11. Copyright Severalnines AB Galera Concepts cont. Node Failures A peer crash is indistinguishable from network failure A node is considered failed when it no longer can be communicated with Node health verified by receiving messages or keepalives evs.inactive_timeout sets the timeout after which node is considered inactive (dead) evs.suspect_timeout sets the timeout after which the node can be pronounced dead if everyone else agrees Confidential 11
- 12. Copyright Severalnines AB Galera Concepts cont. LAN vs WAN replication No notion of local or remote node Works as long as TCP works May need tuning to be more tolerant to network latency/issues Network params sample evs.keepalive_period = PT3S evs.inactive_check_period = PT10S evs.suspect_timeout = PT30S evs.inactive_timeout = PT1M evs.consensus_timeout = PT1M Confidential 12
- 13. Copyright Severalnines AB Node Provisioning Automatic node (re)synchronization A ‘donor’ is chosen to provision a ‘joiner’ node ‘Donor’ node is blocked (write operations) until SST completes State Snapshot Transfer - SST Scriptable interface mysqldump (slow) rsync (fast) Percona Xtrabackup (faster and non-blocking) Confidential 13
- 14. Copyright Severalnines AB Node Provisioning cont. Client Client Client Load balancer Node 1 MySQL [WSREP] Node 2 MySQL [WSREP] Confidential 14
- 15. Copyright Severalnines AB Node Provisioning cont. Client Client Client Load balancer Node 1 MySQL [WSREP] MySQL Node 2 MySQL [WSREP] [WSREP] ‘Joiner’ Node 3 Confidential 15
- 16. Copyright Severalnines AB Node Provisioning cont. Client Client Client Load balancer Node 1 MySQL [WSREP] ‘Joiner’ Node 3 MySQL Node 2 MySQL [WSREP] [WSREP] rsync receive wsrep_cluster_address=Node 2 SST Request Confidential 16
- 17. Copyright Severalnines AB Node Provisioning cont. Client Client Client Load balancer Node 1 MySQL [WSREP] ‘Joiner’ Node 3 MySQL Node 2 MySQL [WSREP] [WSREP] rsync receive rsync send Node 2 in ‘donor mode’. Write operations blocked Confidential 17
- 18. Copyright Severalnines AB Node Provisioning cont. Client Client Client Load balancer Node 1 MySQL [WSREP] Catch up MySQL Node 2 MySQL [WSREP] [WSREP] Node 3 Confidential 18
- 19. Copyright Severalnines AB Network Partitioning/Split Brain Quorum based system “Majority >50%” partition continues operation “Minority” partition blocks operations Until reconnected with Primary Component Use odd number of nodes Minimum 3 (5, 7, 9 etc) Galera Arbitrator (garbd) Useful if you have even number of nodes Nodes across DCs Replication relay Confidential 19
- 20. Copyright Severalnines AB Network Partitioning/Split Brain cont. Client Client Client Load balancer MySQL [WSREP] MySQL 1 Primary Component [WSREP] MySQL [WSREP] DC1 DC2 Confidential 20
- 21. Copyright Severalnines AB Network Partitioning/Split Brain cont. Client Client Client Load balancer MySQL [WSREP] MySQL Block operations until Primary Component ? [WSREP] reconnected with PC MySQL [WSREP] DC1 DC2 Confidential 21
- 22. Copyright Severalnines AB Network Partitioning/Split Brain cont. Client Client Client Load balancer MySQL [WSREP] MySQL [WSREP] MySQL [WSREP] DC1 DC2 Galera Arbitrator DC3 Confidential 22
- 23. Copyright Severalnines AB Network Partitioning/Split Brain cont. Client Client Client Load balancer MySQL [WSREP] MySQL [WSREP] MySQL [WSREP] Replication Relay DC1 DC2 Galera Arbitrator DC3 Confidential 23
- 24. Copyright Severalnines AB Network Partitioning/Split Brain cont. Client Client Client Load balancer MySQL [WSREP] MySQL Primary Component ? [WSREP] MySQL [WSREP] DC1 DC2 Galera Arbitrator DC3 Confidential 24
- 25. Copyright Severalnines AB Galera Configuration Example [mysqld] wsrep_cluster_address=/usr/lib64/libgalera_smm.so wsrep_node_address=gcomm:// # NOTE: This must be changed to peer address ASAP! wsrep_node_name=node1 wsrep_provider='/usr/lib64/galera/libgalera_smm.so' wsrep_provider_options='gcache.size=1G;socket.ssl_key=my_key;socket.ssl_cert=my_cert ' wsrep_slave_threads=16 wsrep_sst_method=xtrabackup wsrep_sst_auth=root: innodb_buffer_pool_size=1G innodb_log_file_size=256M innodb_autoinc_lock_mode=2 innodb_flush_log_at_trx_commit=0 innodb_doublewrite=0 innodb_file_per_table=1 binlog_format=ROW datadir=/var/lib/mysql log-bin = mysql-bin server-id = 2 relay-log = mysql-relay-bin #read-only = 1 log-slave-updates = 1 Confidential 25
- 26. Copyright Severalnines AB wsrep variables wsrep_provider Path to wsrep provider library wsrep_cluster_address URI form:'gcomm://another_node_address?opt1=val1&opt2=val2 'gcomm://' special meaning. Initialize the cluster (never leave it in my.cnf) wsrep_node_address An optional address of the node. A short-cut way to configure listen addresses for replication and state transfers By default it will be initialized to the first network interface returned by ifconfig. This could be unreliable. For best results initialize it explicitly Confidential 26
- 27. Copyright Severalnines AB wsrep variables cont. wsrep_node_name An optional name for the node. It will be used in logging and to identify the desired donor for state transfer Default it will be initialized to hostname wsrep_provider_options Semicolon-separated list of options specific to provider Ex: gcache.size – a size of the permanent transaction on-disk cache socket.ssl_key, socket.ssl_cert – SSL key and certificate files Confidential 27
- 28. Copyright Severalnines AB wsrep variables cont. wsrep_slave_threads Parallel applying threads (1-512) >1 requires certain InnoDB settings. Applying of STATEMENT-based events is always serialized wsrep_sst_method Base package contains scripts for mysqldump, rsync and xtrabackup based state snapshot transfers. Own scripts can be used Default is mysqldump Confidential 28
- 29. Copyright Severalnines AB Performance Metrics wsrep_flow_control_paused Fraction of the time replication was paused wsrep_flow_control_sent How many times this node paused replication wsrep_local_recv_queue_avg Average length of slave trx queue – a sign of slave side bottleneck wsrep_cert_deps_distance How many transactions can be applied in parallel wsrep_local_send_queue_avg A sign of network bottleneck Confidential 29
- 30. Copyright Severalnines AB Number of conflicts/”deadlocks” wsrep_last_committed Last committed transaction wsrep_local_cert_failures, wsrep_local_bf_aborts Rollbacks, conflicts detected Confidential 30
- 31. Copyright Severalnines AB Benchmarks: sysbench, tps http://codership.com/content/whats-difference-kenneth Confidential 31
- 32. Copyright Severalnines AB Benchmarks: sysbench, latency http://codership.com/content/whats-difference-kenneth Confidential 32
- 33. Copyright Severalnines AB Benchmarks: Comparing NDB vs Galera Note: No optimizations done for the NDB storage engine (DB schema nor queries) http://codership.com/content/whats-difference-kenneth Confidential 33
- 34. Copyright Severalnines AB Benchmarks: Comparing NDB vs Galera Note: No optimizations done for the NDB storage engine (DB schema nor queries) http://codership.com/content/whats-difference-kenneth Confidential 34
- 35. Copyright Severalnines AB Best Practices Dedicated switch/network for Galera Nodes (1 GBit min) Connection pools/Load balancing with applications Gives best performance Use static/elastic IPs for the Galera nodes Con: Need to handle node membership changes Con: JDBC/PHP etc are not aware of Galera specific Node states Load Balancers Hardware, e.g., IP5 SW load balancer HAProxy with Galera specific health check scripts IP dispatching in the kernal for example Linux LVS GLB (Galera Load Balancer) Con: Need to setup LB redundancy Confidential 35
- 36. Copyright Severalnines AB Best Practices cont. Reference Node Client Client Client Act as a ‘donor’ node Backup node No client connections LB R/W R/W R/W MySQL [WSREP] ... MySQL [WSREP] MySQL [WSREP] Donor & Backup Node Confidential 36
- 37. Copyright Severalnines AB Best Practices cont. Minimize probability of deadlocks Writes go only to 1 Node Applications use connection pool or Client Client Client load balancer on read only nodes Have 1 “reference” Node for write failover LB and donor R R W MySQL [WSREP] ... MySQL [WSREP] MySQL [WSREP] “Master” Node Confidential 37
- 38. Copyright Severalnines AB Galera Limitations MyISAM replication is experimental DDL statements are replicated in statement level Any writes to other table types, including system (mysql.*) tables are not replicated CREATE USER..., but issuing: INSERT INTO mysql.user..., will not be replicated Non-deterministic functions like NOW() are not supported Query log cannot be directed to table LOCK/UNLOCK TABLES cannot be supported in multi-master setups lock functions (GET_LOCK(), RELEASE_LOCK()... ) Maximum allowed transaction size is defined by wsrep_max_ws_rows and wsrep_max_ws_size XA transactions can not be supported due to possible rollback on commit Confidential 38
- 39. Copyright Severalnines AB Monitoring and Management Confidential 39
- 40. Copyright Severalnines AB ClusterControl Host Monitoring (CPU, RAM, Disk, Network) Configuration Management DB Metrics Monitoring Performance Management DB Resources Monitoring Database Upgrades/Downgrades Cluster-wide Query Analyzer Online Scaling of MySQL Servers Schema Management Configurable Resource Thresholds Replication Fail-over Alarms and Email Notifications Clusterware – Process Management and Automated Recovery Backup Scheduling Manual start/stop of Nodes Real-time Performance Probes Confidential 40
- 41. Copyright Severalnines AB Configurators Confidential 41
- 42. Copyright Severalnines AB Galera Configurator Confidential 42
- 43. Copyright Severalnines AB Galera Configurator cont. Confidential 43
- 44. Copyright Severalnines AB Galera Configurator cont. Confidential 44
- 45. Copyright Severalnines AB Deploy Galera Cluster with HAProxy cd ~/s9s-galera-2.10/mysql/scripts/install ./deploy.sh &> | tee -a cc.log wget http://severalnines.com/downloads/s9s-haproxy.tar.gz tar zxvf s9s-haproxy.tar.gz cd haproxy ./install-haproxy.sh <lb host> <rhel|debian> galera done... Confidential 45
- 46. Copyright Severalnines AB Confidential 46
- 47. Copyright Severalnines AB Confidential 47
- 48. Copyright Severalnines AB Resources Severalnines MySQL Galera Configurator http://www.severalnines.com/resources/configurator Supported platforms (MySQL Galera) http://support.severalnines.com/entries/21589522-verified-and-supported-operating- systems Galera limitations http://support.severalnines.com/entries/21692388-limitations-in-galera-replication-for- mysql ClusterControl server requirements http://support.severalnines.com/entries/20614858-server-requirements-on-premise- amis-other-imageshttp://support.severalnines.com/entries/20614858-server- requirements-on-premise-amis-other-images Confidential 48