Percona xtra db cluster(pxc) non blocking operations, what you need to know to avoid pitfalls
•
0 likes•69 views
Performing simple DDL operations as ADD/DROP INDEX in a tightly connected cluster as PXC, can become a nightmare. Metalock will prevent Data modifications for long period of time and to bypass this, we need to become creative, like using Rolling schema upgrade or Percona online-schema-change. With NBO, we will be able to avoid such craziness at least for a simple operation like adding an index. In this brief talk I will illustrate what you should do to see the negative effect of NON using NBO, as well what you should do to use it correctly and what to expect out of it.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Percona xtra db cluster(pxc) non blocking operations, what you need to know to avoid pitfalls
Similar to Percona xtra db cluster(pxc) non blocking operations, what you need to know to avoid pitfalls (20)
More from Marco Tusa
More from Marco Tusa (20)
Recently uploaded
Recently uploaded (20)
Percona xtra db cluster(pxc) non blocking operations, what you need to know to avoid pitfalls
- 2. 2 • Open source enthusiast • MySQL tech lead; principal architect • Working in DB/development world over 35 years (yes, I am that old) • Open source developer and community contributor About Me
- 3. 3 Why In standard MySQL the online DDL feature provides support for instant and in-place table alterations and concurrent DML. ● Improved responsiveness and availability in busy production environments, where making a table unavailable for minutes or hours is not practical. ● For in-place operations, the ability to adjust the balance between performance and concurrency during DDL operations using the LOCK clause. ● Less disk space usage and I/O overhead than the table-copy method. However, in the Galera/PXC context the ONLINE DDL feature was not respected and a a simple ADD INDEX cause a full write stop on the executing node. We will review what we are doing to leverage this limitation and we also do very fast comparison with Group Replication.
- 4. 4 What online DDLs cover in MySQL All operations: ● Index Operations ● Primary Key Operations ● Column Operations ● Generated Column Operations ● Foreign Key Operations ● Table Operations ● Tablespace Operations ● Partitioning Operations The less impacting: ● Index Operations ● Column Operations ● Foreign Key Operations ● Tablespace Operations
- 5. 5 What methods we had in PXC for DDL ● Total Order Isolation: Abbreviated as TOI, these are schema changes made on all cluster nodes in the same total order sequence, preventing other transactions from committing for the duration of the operation. ● Rolling Schema Upgrade: Known also as RSU, these are schema changes run locally, affecting only the node on which they are run. The changes do not replicate to the rest of the cluster. ● pt-online-schema-change: works by creating an empty copy of the table to alter, modifying it as desired, and then copying rows from the original table into the new table. When the copy is complete, it moves away the original table and replaces it with the new one.
- 8. 7 Quick look how each method works - TOI Given a cluster Primary Secondary Secondary TOI Primary Secondary Secondary DDL is ececuted As soon as DDL is received. Metalock is raised and command distributed/executed on all nodes
- 9. 7 Quick look how each method works - TOI Given a cluster Primary Secondary Secondary TOI Primary Secondary Secondary DDL is ececuted As soon as DDL is received. Metalock is raised and command distributed/executed on all nodes Any write is put on hold on the whole cluster However, cluster remain always consistent
- 11. 8 Quick look how each method works - RSU Given a cluster Primary Secondary Secondary RSU Secondary DDL is ececuted As soon as DDL is received. Metalock is raised and command executed only locally Writes can be moved to another node. Primary Secondary
- 12. 8 Quick look how each method works - RSU Given a cluster Primary Secondary Secondary RSU Secondary DDL is ececuted As soon as DDL is received. Metalock is raised and command executed only locally Writes can be moved to another node. Primary Secondary Action needs to be repeated on each node, one by one. In the meanwhile cluster state is not consistent.
- 19. 10 What is NBO and what it covers and what it doesn’t The new Non Blocking Operation (NBO) method is to help to reduce the impact on the cluster and make it easier to perform some DDL operations. At the moment we only support a limited set of operations with NBO like: ● ALTER INDEX ● CREATE INDEX ● DROP INDEX Any other command will result in an error message ER_NOT_SUPPORTED_YET. And we are still talking of a technical preview.
- 21. 12 The test - 1 DC1-1(root@localhost) [windmills_s]>show create table windmills_testG *************************** 1. row *************************** Table: windmills_test Create Table: CREATE TABLE `windmills_test` ( `id` bigint NOT NULL AUTO_INCREMENT, `uuid` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, `millid` smallint NOT NULL, `kwatts_s` int NOT NULL, `date` date NOT NULL, `location` varchar(50) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, `active` tinyint NOT NULL DEFAULT '1', `time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `strrecordtype` char(3) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, PRIMARY KEY (`id`), KEY `IDX_millid` (`millid`,`active`), KEY `IDX_active` (`id`,`active`), KEY `kuuid_x` (`uuid`), KEY `millid_x` (`millid`), KEY `active_x` (`active`) ) ENGINE=InnoDB AUTO_INCREMENT=8199260 DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC 1 row in set (0.00 sec) DC1-1(root@localhost) [windmills_s]>select count(*) from windmills_test; +----------+ | count(*) | +----------+ | 5002909 | +----------+
- 22. 13 The test - 2 Connection 1: ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE; ALTER TABLE windmills_test drop INDEX idx_1, ALGORITHM=INPLACE; Connection 2: while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./ my.cnf -uroot -D windmills_s -e "insert into windmills_test select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done Connection 3: while [ 1 = 1 ];do da=$(date +'%s.%3N');/opt/mysql_templates/PXC8P/bin/mysql --defaults-file=./ my.cnf -uroot -D windmills_s -e "insert into windmills8 select null,uuid,millid,kwatts_s,date,location,active,time,strrecordtype from windmills7 limit 1;" -e "select count(*) from windmills_s.windmills_test;" > /dev/null;db=$(date +'%s.%3N'); echo "$(echo "($db - $da)"|bc)";sleep 1;done Connections 4-5: while [ 1 = 1 ];do echo "$(date +'%T.%3N')";/opt/mysql_templates/PXC8P/bin/mysql --defaults- file=./my.cnf -uroot -D windmills_s -e "show full processlist;"|egrep -i -e "(windmills_test| windmills_s)"|grep -i -v localhost;sleep 1;done
- 23. 14 The test - 3 Operations: start inserts from connections start commands in connections 4 - 5 on the other nodes execute: For TOI DC1-1(root@localhost) [windmills_s]>SET SESSION wsrep_OSU_method=TOI; For NBO DC1-1(root@localhost) [windmills_s]>SET SESSION wsrep_OSU_method=NBO; For both DC1-1(root@localhost) [windmills_s]>ALTER TABLE windmills_test ADD INDEX idx_1 (`uuid`,`active`), ALGORITHM=INPLACE, LOCK=shared;
- 24. 15 The test – 4 Results TOI Inserts in the altering table (connection 2): .450 .492 64.993 <--- Alter blocks all inserts on the table we are altering .788 .609 Inserts on the other table (connection 3): .455 .461 64.161 <--- Alter blocks all inserts on all the other tables as well .641 .483 NBO Inserts in the altering table (connection 2): .437 .487 120.758 <---- Execution time increase .617 .510 Inserts on the other table (connection 3): .468 .485 25.061 <---- still a metalock, but not locking the other tables for the whole duration .494 .471 TOI NBO Time on hold for insert for altering table ~64 sec ~120 sec Time on hold for insert for another table ~64 sec ~25 sec metalock whole time only at the end
- 25. 16 What is doing and why? ● TOI: when you issue a DDL like ADD INDEX a metadata lock is taken on the table and it will be released only at the end of the operation. During this time, you cannot: ○ Perform DMLs on any cluster node ○ Alter another table in the cluster ● NBO: the metadata lock is taken at the start and at the end for a very brief period of time. The ADD INDEX operation will then work on each node independently. The lock taken at the end is to have all the nodes agree on the operation and commit or roll back (using cluster error voting). This final phase costs a bit more in time and is what adds a few seconds to the operation execution. But during the operation: ○ You can alter another table (using NBO) ○ You can continue to insert data, except in the table(s) you are altering. ○ On node crash the operation will continue on the other nodes, and if successful it will persist.
- 30. 18 Quick comparison with GR - 2 EVENTUAL (on the primary only) Node 1 same table: .184 .186 <--- no locking during alter on the same node <snip> .184 .217 <--- moment of commit .185 Node 1 another table : .189 .198 <--- no locking during alter on the same node .188 <snip> .191 .211 <--- moment of commit .194 Action Group Replication PXC (NBO) Time on hold for insert in altering table ~0.217 sec ~ 120 sec Time on hold for insert in another table ~0.211 sec ~ 25 sec
- 31. 18 Quick comparison with GR - 2 EVENTUAL (on the primary only) Node 1 same table: .184 .186 <--- no locking during alter on the same node <snip> .184 .217 <--- moment of commit .185 Node 1 another table : .189 .198 <--- no locking during alter on the same node .188 <snip> .191 .211 <--- moment of commit .194 Action Group Replication PXC (NBO) Time on hold for insert in altering table ~0.217 sec ~ 120 sec Time on hold for insert in another table ~0.211 sec ~ 25 sec However, yes there is a however, PXC was maintaining consistency between the different nodes during the DDL execution, while MySQL 8.0.27 with Group Replication was postponing consistency on the secondaries, thus Primary and Secondary were not in sync until full DDL finalization on the secondaries.
- 32. 19 Conclusion ● As raised in one internal discussion we should call this LBO, Less blocking Operation. ● NBO can be significantly helpful to reduce the impact of DDL on the cluster, for now limited to the widely used creation/modification/drop of an index. But in the future … we may expand it. ● A small cost still exists but PXC offers full cluster consistency all time vs GR. ● The feature is still a technology preview, so do not trust in production, but test it and let us know what you think. ● More will come… ● Final comment. Another distribution has introduced NBO, but only if you buy the enterprise version. However Percona, which is truly open source with facts not just words, has implemented NBO in standard PXC, and the code is fully open source. This is not the first one, but just another of the many features Percona is offering for free, while others ask you to buy the enterprise version.