M|18 Battle of the Online Schema Change Methods

1. Battle of the online schema change methods Ivan Groenewold Valerie Parham-Thompson 27 February 2018
2. ● Definition of “Online” ● Native Online DDL ● Alternate Methods ● Alternate Tools ● Battle! Special Replication/Cluster Cases ● Battle! Special Workload Cases ● Q&A Agenda
3. What is “Online”?
4. ● “Online” can mean several things: ● Changes only happen to metadata. ● Changes are made in-place, and the software allows concurrent DML. ● Table is rebuilt with table copy, but the software allows concurrent DML. ● In some cases (e.g., TokuDB), changes are made in memory and not pushed to disk until the row is accessed. More on that later! Definition
5. ● Blocking ● Locking ● Application failures Why do we care?
6. Native Online DDL
7. Native Online DDL - InnoDB/XtraDB - Overview ● Supported since MariaDB 10.0 / MySQL 5.6 ● Information about progress is limited ● MariaDB >= 10.1.1: SHOW STATUS LIKE 'innodb_onlineddl%'; and error log ● MySQL >= 5.7.6: Performance schema stage/innodb/alter%
8. Native Online DDL - InnoDB/XtraDB - Settings ● Algorithm and locking methods ● e.g., ALTER TABLE test ADD INDEX name_idx (name), ALGORITHM=inplace, LOCK=none; ● You get an error message if the specified algorithm/locking cannot be used ● If not explicitly specified e.g., ALTER TABLE test ADD INDEX name_idx (name), the less expensive combination will be used ● ALGORITHM=copy, LOCK=shared is the old behaviour (only reads allowed)
9. ● Some operations still require a table copy ● Change column data type i. from MariaDB 10.2.2, increasing the size of a VARCHAR column can be done in-place ● Charset change ● Primary Key change ● DATE, DATETIME, TIMESTAMP columns created before MariaDB 10.1.2 / MySQL 5.6 ● Online DDL not really “online” on replicas ● Operation needs to finish on the master before being replicated ● SQL thread is “blocked” while replica is executing the DDL operation ● Lag spike at the end Native Online DDL - InnoDB/XtraDB - Limitations
10. ● Works only with InnoDB ● Can’t pause an online DDL operation ● Rollback can be expensive ● LOCK=none not supported with fulltext indexes ● https://bugs.mysql.com/bug.php?id=81819 ● Instant ADD COLUMN for InnoDB in MariaDB 10.3+ is in RC ● https://jira.mariadb.org/browse/MDEV-11369 https://mariadb.com/kb/en/library/online-ddl-overview/ https://dev.mysql.com/doc/refman/5.7/en/innodb-create-index-overview.html Native Online DDL - InnoDB/XtraDB - More Limitations
11. ● “TokuDB enables you to add or delete columns in an existing table, expand char, varchar, varbinary, and integer type columns in an existing table, or rename an existing column in a table with little blocking of other updates and queries. … The work of altering the table for column addition, deletion, or expansion is performed as subsequent operations touch parts of the Fractal Tree, both in the primary index and secondary indexes.” Native Online DDL - TokuDB - Overview Note: tokudb_version=5.6.37-82.2
12. ● Don’t change too much at once. For example, in changing column name, don’t change other elements, or it will revert to a regular mysql (not hot copy) alter. This was originally intcol1 int(11) default null. Native Online DDL - TokuDB - Limitations ALTER TABLE test.t1 CHANGE intcol1 intone INT(11) DEFAULT NULL; Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE test.t1 CHANGE intcol1 inttwo INT(11) NOT NULL; Query OK, 56777 rows affected (1.52 sec) Records: 56777 Duplicates: 0 Warnings: 0 Note: tokudb_version=5.6.37-82.2
13. ● Don’t try to do multiple types of changes in one statement, or multiple renames. Native Online DDL - TokuDB - HCADER - Limitations These are not executed online: ALTER TABLE test.t2 DROP number4, ADD number7 int(11) default null; Query OK, 56777 rows affected (0.93 sec) Records: 56777 Duplicates: 0 Warnings: 0 ALTER TABLE test.t2 CHANGE number4 num4 int(11) default null, CHANGE number7 num7 int(11) default null; Query OK, 56777 rows affected (1.53 sec) Records: 56777 Duplicates: 0 Warnings: 0 Although these are (because natively supported): ALTER TABLE test.t2 ADD number5 int(11) default null, ADD number6 int(11) default null; Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE test.t2 DROP number5, DROP number6; Query OK, 0 rows affected (0.18 sec) Records: 0 Duplicates: 0 Warnings: 0
14. ● If dropping a column with an index, drop the index first, then drop the column. Native Online DDL - TokuDB - HCADER - Limitations This is not executed online, because number4 has an associated index: ALTER TABLE test.t1 DROP column number4; alter table test.t1 drop column number4; Query OK, 159002 rows affected (4.52 sec) Records: 159002 Duplicates: 0 Warnings: 0 Separating the statements allows the drop to happen online: ALTER TABLE test.t1 DROP index idx4; Query OK, 0 rows affected (0.37 sec) Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE test.t1 DROP column number4; Query OK, 0 rows affected (0.18 sec) Records: 0 Duplicates: 0 Warnings: 0
15. ● Changing the size of an int, char, varchar, or varbinary is supported online, but only if the column doesn’t have an index attached to it. Native Online DDL - TokuDB - HCADER - Limitations This is not executed online, because string1 has an associated index: ... `string1` char(2) DEFAULT NULL, ... KEY `idxstr1` (`string1`) ALTER TABLE test.t2 change column string1 string1 char(3) default null; Query OK, 56777 rows affected (6.39 sec) Records: 56777 Duplicates: 0 Warnings: 0 This is executed online, because string2 has no index: ... `string2` char(2) DEFAULT NULL, ALTER TABLE test.t2 change column string2 string2 char(3) default null; Query OK, 0 rows affected (0.26 sec) Records: 0 Duplicates: 0 Warnings: 0
16. ● A note about performance: “The time that the table lock is held can vary. The table-locking time for HCADER is dominated by the time it takes to flush dirty pages, because MySQL closes the table after altering it. If a checkpoint has happened recently, this operation is fast (on the order of seconds). However, if the table has many dirty pages, then the flushing stage can take on the order of minutes.” ● Another note about performance: “The work of altering the table for column addition, deletion, or expansion is performed as subsequent operations touch parts of the Fractal Tree, both in the primary index and secondary indexes.” Think about very busy workloads. ● Take this very seriously, especially in the presence of replication! Test early and often. Native Online DDL - TokuDB - HCADER - Limitations
17. ● Adding indexes is done online, allowing concurrent inserts and updates by: 1. Setting the variable: set tokudb_create_index_online=on 2. Using the syntax: create index index_name on table_name (column_name); BUT will block replication on replicas: it is long-running even though non-blocking Native Online DDL - TokuDB - Adding Indexes
18. All changes require a table copy, except create or drop secondary indexes. More may be supported soon. https://github.com/facebook/mysql-5.6/issues/47 Native Online DDL - Native Online DDL - RocksDB
19. ● Traditional online DDL ● Aurora Fast DDL ● updates INFORMATION_SCHEMA system table with the new schema ● records the old schema into a Schema Version Table with the timestamp ● change is propagated to read replicas ● on subsequent DML operations, check to see if the affected data page has a pending schema operation by comparing the log sequence number (LSN) timestamp for the page with the LSN timestamp of schema changes. ● update the page to the new schema before applying the DML statement https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-fast-ddl/ Native Online DDL: RDS MySQL / Aurora - Overview
20. ● Problems with traditional online DDL ● Temp table can fill ephemeral storage ● Fast DDL ● only supports adding nullable columns to the end of a table ● no support for partitioned tables ● no support for the older REDUNDANT row format ● Only usable if the max possible record size is smaller than half the page size Native Online DDL: RDS MySQL / Aurora - Limitations
21. Alternate Methods: Rolling Schema Updates
22. ● Stop replica ● Perform the alter on the replica ● Start the replica and allow it to catch up ● (repeat...) ● Promote replica as master + deploy new code release ● Perform the alter on the old master ● Promote the old master back (optional) Rolling Schema Updates - Overview
23. ● Time-consuming manual process ● Complexity proportional to number of slaves ● Schema is inconsistent ● If using slaves for reads, the application needs to be forward/backward compatible ● Some changes can break replication ● Potential problems with GTID Rolling Schema Updates - Limitations
24. ● Run a DDL change on slave A ● The DDL change is applied to other slaves (B, C, …) ● Promote slave A as master ● Possible outcomes: ● Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires ● If server A still has the binlog with the DDL change, the transaction is sent (again) to all current slaves Rolling Schema Updates - Limitations - Errant Tx
25. ● Avoid errant transactions ● Disable binlog for the session ● Run the changes ● How to spot them ● Check executed_gtid_set in the slave (show slave status) ● Check executed_gtid_set in the master (show master status) ● Use gtid_substract(‘slave_set’, ‘master_set’) function Rolling Schema Updates - Limitations - Errant Tx
26. Alternate Methods: Downtime
27. ● Schedule application downtime ● Do changes during code rollout maintenance ● Rarely used these days, but sometimes can choose to do impactful changes during relatively quiet times to reduce length of running change Downtime - Overview
28. Alternate Tools: pt-online-schema-change
29. ● Create an empty copy of the table to alter ● Modify the new table ● Create triggers from old to new table (_ins, _upd, _del) ● Copy rows from the original table into the new table (in chunks) ● Move away the original table and replace it with the new one pt-osc - Overview https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html
30. ● Chunk size controls ● --chunk-time i. Adjusts chunk size dynamically based on row copy time ● --chunk-size i. Specify a fixed chunk size ● Triggers ● --preserve-triggers (MySQL 5.7 only) ● Finding replicas ● --recursion-method i. processlist ii. hosts (Uses SHOW SLAVE HOSTS) iii. dsn=DSN iv. none pt-osc - Settings
31. ● Throttling ● Master i. --max-load Threads_connected=800,Threads_running=80 ii. --critical-load iii.--pause-file ● Slave lag i. --max-lag ii. --check-interval iii.--check-slave-lag or DSN to monitor specific replica(s) pt-osc - Settings
32. ● Potential stalls after dropping a big table ● Use --no-drop-old-table ● Trigger-related ● Tool cannot be completely paused ● Adding/dropping triggers is expensive! i. Control metadata lock time 1. --set-vars=lock_wait_timeout=10 ii. Use retries 1. --tries create_triggers:5:0.5,drop_triggers:5:0.5 (try 5 times, wait 0.5 sec between tries) pt-osc - Limitations
33. ● Dealing with foreign keys ● --alter-foreign-keys-method i. drop_swap: no atomic rename ii. rebuild_constraints: rebuild is done using normal blocking DDL iii. auto: let pt-osc decide ● FK Names: adds/removes underscores at the beginning based on the first FK of the table ● Be careful with underscores in FK names ● Try to avoid (in the same table) things like: i. _FK1 ii. __FK2 iii. FK3 ● https://bugs.launchpad.net/percona-toolkit/+bug/1428812 pt-osc - More Limitations
34. Alternate Tools: Facebook OSC
35. ● Create an empty copy (“shadow”) of the table to alter with the new structure ● Create change capture (“deltas”) table ● All columns of the source table ● Extra integer auto-increment column to track order of changes ● Extra integer column to track DML type ● Create 3 triggers from old to “deltas” table ● Inserts ● Deletes ● Updates Facebook OSC - Overview
36. ● Dump chunks of rows from the original table & load to new table ● Replay changes from deltas table ● Checksum ● Cut-over ● Lock the existing table ● Final round of replay ● Swap the tables Facebook OSC - Overview
37. Facebook OSC - Unique features ● Developed for rolling schema updates ● Tool runs with sql-log-bin=0 ● Change replay is asynchronous ● Use of SELECT INTO OUTFILE / LOAD DATA INFILE to avoid gap lock ● Manage schema via source control ● Relies on a file containing CREATE TABLE to run the schema change ● Supports MyRocks ● Implemented as standalone Python class ● Interact with OSC from your code
38. Facebook OSC - Limitations ● No support for triggers, FK, rename column or RBR ● Only one OSC can run at a time ● Requires Python 2.7 ● Difficult to install in older distributions ● No atomic rename ● 'table not found' errors can happen for a short period ● Implications of using sql-log-bin=0 https://github.com/facebookincubator/OnlineSchemaChange
39. Alternate Tools: gh-ost
40. ● Create an empty copy of the table to alter ● Modify the new table ● Hook up as a MySQL replica (stream binlog events) ● Copy rows from the original table into the new table in chunks ● Apply events on the new table ● When the copy is complete, move away the original table and replace it with the new one gh-ost - Overview
41. ● No triggers! ● Can be completely paused ● Low overhead and less locking ● Multiple concurrent migrations ● Dynamic reconfiguration ● Can offload some operations to a slave ● reading binlog events ● queries required for (optional) accurate progress counter ● Supports testing or migrating on a slave ● --test-on-replica ● --migrate-on-replica gh-ost - Unique features
42. ● Throttling ● Master i. --max-load=Threads_connected=800,Threads_running=80 ii. --critical-load iii.--critical-load-interval-millis=10000 (2nd chance) iv. --critical-load-hibernate-seconds=300 (don’t panic, hibernate instead) ● Slave lag i. --max-lag-millis ii. --throttle-control-replicas ● Pause i. echo throttle | nc -U /tmp/gh-ost.test.sock gh-ost - Settings
43. ● Send commands via Unix socket ● echo ‘status’ | nc -U /tmp/gh-ost.test.sock ● echo chunk-size=? | nc -U /tmp/gh-ost.test.sock ● echo ‘chunk-size=2500’ | nc -U /tmp/gh-ost.test.sock ● echo ‘[no-]throttle’ | nc -U /tmp/gh-ost.test.sock ■ Make sure binlogs don’t get purged! ● Delay cutover ■ --postpone-cut-over-flag-file=/tmp/file ■ echo unpostpone | nc -U /tmp/gh-ost.test.sock https://github.com/github/gh-ost/blob/master/doc/interactive-commands.md gh-ost - Interactive commands
44. gh-ost - Limitations ● No support for FK or triggers ● Needs row based binlogs on at least one slave ● No support for new MySQL 5.7 columns ● Generated ● POINT ● No JSON PKs ● Master-master is supported but only active-passive
45. Special Replication / Cluster Cases: RDS / Aurora
46. Use Case: RDS MySQL / Aurora - Overview ● Black-box approach ● RDS replicas similar to traditional slaves ● Aurora replicas share the same storage with the master ● STOP/START SLAVE done through procedures ● mysql.rds_stop_replication ● mysql.rds_start_replication
47. BATTLE!
48. Use Case: RDS MySQL / Aurora - Complications ● Privileges are limited ● No SUPER ● Detecting replicas is tricky ● processlist gives you the internal IP ● Aurora read replicas are “special” ● Read_only can’t be modified ● Replication filters are used ● Replicate_Ignore_Table: mysql.plugin,mysql.rds_monitor ● Binlogs are disabled by default
49. gh-ost wins!
50. Use Case: Aurora - gh-ost on master ● Make sure binary logs are enabled ● Set backup retention >= 1 day ● set binlog_format=ROW in parameter group ● use --assume-rbr ● Specify the Aurora writer endpoint in --host ● use --allow-on-master
51. ./gh-ost … --alter=”add column b int” --assume-rbr --allow-on-master --host=<aurora writer endpoint> --database="testdb" --table="mytable" --verbose --execute Use Case: Aurora - gh-ost on master
52. Use Case: RDS MySQL - gh-ost on replica ● Make sure binary logs are enabled on the replica ● Set backup retention >= 1 day ● Set binlog_format=ROW in the parameter group ● Use --assume-rbr ● Specify the replica endpoint in --host ● Specify the master’s endpoint in --assume-master-host
53. Use Case: RDS MySQL - gh-ost on replica ./gh-ost … --alter=”add column b int” --assume-rbr --assume-master-host=<master endpoint> --host=<replica endpoint> --max-lag-millis=5000 --database="testdb" --table="mytable" --verbose --execute
54. Special Replication / Cluster Cases: Tungsten
55. ● Tungsten uses an external replicator process ● Reads from the binary logs (row or statement-based data) ● Converts transaction data to db-agnostic format ● Writes into Transaction History Log (THL) files ● Hosts are not aware they are replicating ● SHOW SLAVE STATUS is empty Use case: Tungsten - Overview
56. BATTLE!
57. ● Issues with pt-osc ● Replicas have to be manually specified (DSN table) ● Binlog format considerations (triggers) ■ STATEMENT: works out of the box ■ ROW: use Tungsten plugin for Percona toolkit ■ MIXED: do not use! ● Issues with gh-ost ● Can’t detect the master/replicas automatically ● Can’t start/stop replication ● Can’t switch binlog format Use case: Tungsten - Complications
58. gh-ost wins!
59. MySQL binlogs Tungsten Replicator (master) THL MySQL Tungsten Replicator (slave) THL master slave gh-ost Read changes Copy rows and apply changes to gh-ost table binlogs read write read write Use case: Tungsten - gh-ost using a replica
60. ● Pass the slave to --host ● Use --assume-master-host to specify the actual master ● Specify --tungsten argument ● Slave needs binlog_format=ROW ● Use Tungsten log-slave-updates parameter ○ tpm configure property=log-slave-updates=true ○ hang at INFO Waiting for tables to be in place otherwise Use case: Tungsten - gh-ost using a slave
61. ./gh-ost --assume-master-host=<master_ip> --tungsten --host=<slave1_ip> --database="test" --table="test_t" --verbose --alter="add column b int" --execute Use case: Tungsten - gh-ost using a slave
62. Special Replication / Cluster Cases: Galera
63. ● Can’t benefit from online DDL ● Total order isolation (TOI) ● Blocks all writes to the cluster - no thanks! ● Rolling schema updates (RSU) ● Block one node at a time ● New and old schema definitions need to be compatible ● Manual operation ● Adjust gcache big enough to allow IST after Use case: Galera - Overview
64. BATTLE!
65. ● pt-osc ● Only InnoDB tables are supported ● gh-ost ● No official support for Galera/XtraDB Cluster i. Works in Galera 5.6 using --cut-over=two-step ii. Doesn’t work with Galera 5.7 iii. https://github.com/github/gh-ost/issues/224 Use case: Galera - Complications
66. pt-osc wins!
67. ● Set wsrep_OSU_method to TOI ● ALTER of _new table needs to replicate to all nodes at the same time ● No other special arguments are required ● Throttling ● No replication lag in Galera ● --max-flow-ctl (percentage) i. Check average time cluster spent pausing for Flow Control and make pt-osc pause if it goes over the percentage indicated ii. available for Galera >=5.6 Use case: Galera - pt-osc
68. Special Replication / Cluster Cases: Multisource Replication
69. Multisource replication is commonly found in MariaDB because it has been supported in the product for a while. One replica supports multiple sources. Use case: Multisource Replication - Overview m1 m2 s1
70. BATTLE!
71. gh-ost wins!
72. Use case: Multisource Replication Previous version required running gh-ost on the master. See https://github.com/github/gh- ost/issues/225. Now we can use assume-master-host. m1 m2 s1 gh-ost ./gh-ost --max-load=Threads_running=25 --critical-load=Threads_running=1000 --chunk-size=1000 --throttle-control-replicas="192.168.56.12" --max-lag-millis=1500 --user="ghost" --password="ghost" --allow-master-master --assume-master-host=192.168.56.10 --database="d1" --table="t1" --verbose --alter="change column name name char(50)" --cut-over=default --default-retries=120 --panic-flag-file=/tmp/ghost.panic.flag --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --initially-drop-ghost-table --initially-drop-old-table --execute
73. Special Replication / Cluster Cases: Daisy-Chained Replication
74. Daisy-chained replicas are commonly used in complex topologies to scale a large number of second-level replicas while avoiding load on the primary from the replication threads. The key is to use “log-slave-updates” so that changes are passed to the second-level replicas. Use case: Daisy-chained replication - Overview m1 s1 s1a log-slave-updates
75. BATTLE!
76. pt-osc wins!
77. Run pt-osc on master. Use case: Daisy-chained replication m1 s1 pt-osc s1a
78. Special Workload Cases: Very Large Tables
79. Use case: Very large tables - Overview ● Definition of big is environment-dependent ● Many rows ● Disk space used/free
80. BATTLE!
81. Use case: Very large tables - Complications ● Problems with DDL changes are exacerbated ● Replication lag ● Expensive rollback ● Info about progress can be limited ● Long queries ● Disk space ● Memory usage
82. pt-osc wins!
83. Use case: Very large tables ● External tools are more desirable than online DDL ● pt-osc is usually faster than gh-ost ● Experiment with --chunk-size and --chunk-time vs default auto ● Set high timeouts ● Retry failed operations ● Throttle ● If table is extremely big, consider dump/load into an altered table
84. Use case: Very large tables - pt-osc pt-online-schema-change --set- vars=lock_wait_timeout=10,innodb_lock_wait_timeout=10 --critical-load Threads_running=1000 --max-load Threads_running=50 --max-lag 30 --alter-foreign-keys-method=drop_swap --chunk-time 1 --alter '...' D=my_db,t=my_table --execute
85. Special Workload Cases: Very Busy Tables
86. Use case: Very busy tables - Overview ● Busy tables receive many changes to data over short period of time ● Often combined with very large tables use case
87. BATTLE!
88. Use case: Very busy tables - Complications ● Usual problems with online DDL ● No throttling ● Progress info is limited ● Long queries, IO contention ● Problem with buffering of ongoing changes ● Innodb_online_alter_log_max_size parameter ■ stores data inserted, updated, or deleted in the table during the DDL operation ■ If a temporary log file exceeds the upper size limit, operation fails ■ A large value extends the period of time at the end of the DDL operation when the table is locked to apply the data from the log
89. gh-ost wins!
90. Use case: Very busy tables ● Use a replica to offload traffic from the master ● Use small chunks ● Set low timeouts to reduce impact ● Configure retry of failed operations ● Configure throttling ● Trigger the cut-over manually on a low traffic period
91. Use case: Very busy tables - gh-ost ./gh-ost --max-load=Threads_running=50 --critical-load=Threads_running=500 --chunk-size=250 --max-lag-millis=5000 --host=... --database="..." --table="..." --verbose --default-retries=200 --cut-over-lock-timeout-seconds=5 --critical-load-interval-millis=1000 --critical-load-hibernate-seconds=30 --execute
92. Special Cases: Metadata-only
93. Use case: Metadata-only - Overview ● Metadata-only changes touch the .frm -- the table definition -- not the data files ● Examples ● dropping an index ● set column default value ● change auto_increment ● change column name ● FK changes with foreign_key_checks=0
94. BATTLE!
95. Native online DDL wins!
96. Thank you Robots photos from Ariel Waldman https://www.flickr.com/photos/ariels_photos
97. © 2017 Pythian. Confidential 97 About Pythian Pythian’s 400+ IT professionals help companies adopt and manage disruptive technologies to better compete

Editor's Notes

(Valerie and Ivan)

Hi, I’m Valerie. I’m in the open source database cluster at Pythian.

Hi, I’m Ivan...

Today we’re here to referee a battle of the online schema change methods.

(Valerie starts)
First we’ll walk through a few definitions,

give a basic overview of online DDL,

and then discuss alternate methods and tools.

Then we’ll engage in battle!

What does it mean when we say we want to do an “online” schema change?

Online can mean several things.

The change may be happening only to metadata of the table, the data definition files.

The change might be happening in-place (not a table copy per se, but sometimes requiring reorganization), and allowing changes to the data (like inserts and updates) to happen at the same time.

The change might require a full table copy on disk, but also not block statements.

With some engines, this can even mean the changes are made in temporary memory structures, and written by row to the main storage as data is accessed.

So it means a lot of things at the database level. From the application’s perspective, though, it means statements aren’t blocked while your schema change is happening.

Why do we care about changes being online? Again, from the perspective of the application, statements can continue to run against the database, reading and writing without issue, while the change is happening. That means no downtime for your application.

There’s no promise that performance might not dip some, and there may be a brief pause at the beginning or end of a long-running change, but in general there will be no application failures.

As database technologies have matured, they’ve all started to allow more native capabilities for online changes.

Native online schema changes have been supported in MariaDB since version 10.0, and in MySQL since version 5.6.

Status output about progress of long-running changes has improved, but remains limited. Remember this when we get to the battle: when a change has been running for 2 days, a manager might ask you when it’s going to be done.

You can specify three alter algorithms: default, copy, or inplace. If nothing is specified, then the default behavior will occur for this type of change.

You might specify “copy” if you want to force a table rebuild, but in general, you’ll want to choose inplace. If the alter doesn’t support inplace, you’ll receive an error message. If you get into the habit of specifying algorithm=inplace, you’ll be prepared if a change will require a table copy.

You can specify four alter locks: default, none, shared, exclusive. Again, get into the habit of using “none” to specify that that alter won’t block operations.

Her are some changes remaining that require table copies.

A critical thing to consider is how much replication lag will be incurred on any single-threaded replicas. That SQL thread will be blocked while the replica is running the change. If it’s a long-running change, that’s a lot of lag.

Another important thing to highlight is that you can’t pause a native online alter. Once it starts, you’re committed to completing it, or stopping it and waiting for what can be an expensive rollback.

Information about native online alters above was specific to InnoDB.

A newer engine, TokuDB, offers unique online alter functionality. The several available alters are completed in memory and then pushed down to storage as a part of regular Toku operations.

You just need to be careful about how you plan changes in Toku. Do changes one at a time; for example, don’t change names and data types at the same time.

If you need to do something like add a column and drop a column, separate those alters.

If you need to drop a column that has an index, drop the index online first, then drop the column online.

Changing the size of a basic datatype will be done online, but only if the column doesn’t have an index attached.

The most critical limitations of online alters in TokuDB are centered around performance.

Two things stand out: A table lock with a flush is done at the end of the change, so if there are many dirty pages to be flushed, that process can take several minutes. And, because Toku will push the changes down as part of its regular operations; if you have a very busy workload (or even a batch workload, if timed poorly), this background work can use a lot of system resources.

Take these limitations very seriously, especially in the presence of replication. Test your workload very well.

Adding indexes in TokuDB is a specific online alter case. It’s important to use the correct syntax, highlighted here.

Again, we will see a block with single-threaded replication, causing replication lag.

For another newer engine, RocksDB, all changes require a table copy, except creating/dropping indexes.

Community support may soon be saving the day here.

With Amazon RDS, traditional native online alter functionality is available.

In addition, Aurora Fast DDL allows online changes because of versioning. Consider overhead of these additional background processes.

Specific to RDS, if you have ephemeral storage, and you’re doing any change that requires temp space, the required temp space can fill your allocation.

Aurora Fast DDL is also limited in nonstandard cases, like on tables with partitions.

(Ivan takes over)

Okay. Online alters are pretty robust these days, with most standard table definitions and changes covered.

Alternate methods and tools exist to help with changes not covered by online alter functionality, and, for all changes, to allow for better insight and control.

We’ll cover two alternate methods briefly.

A common method used in the past often for large tables, or for replication scenarios requiring read replicas to be completely up-to-date, is to do a rolling schema update.

In this method, the replica is stopped and changes are made there. Then a failover is done in conjunction with the application change requiring the new schema (if the application code isn’t forward- and backward-compatible), the change is done on the old primary, and replication is restored.

The major limitation here is the time-consuming manual process, compounded in the presence of complex replication setups.

Any writes to the replica can cause serious data inconsistency when GTIDs are used, resulting in errant transactions that need to be manually resolved.

Here’s an example of how GTID modifies the rolling schema approach. If we are not careful we can end up with duplicate statements or broken slaves after we promote a new master

Just as a public service announcement, In case errant transactions occur, here’s how this would be resolved.

Another commonly used schema change process in the past was just to take complete downtime. The downside of downtime is… well, downtime, and it doesn’t work well in today 24x7 environments.

A “light” way to do this is to do changes during relative downtime. That is, do changes when the application is relatively quiet, to avoid performance degradation or replication lag.

Alternate tools were developed to address the limitations of online alters, before the native online DDL was quite robust enough.

These tools remain useful, however.

pt-online-schema-change is a well-known and widely used tool.

With this tool, a new table is created, rows are written in chunks, and concurrent changes are applied via triggers. At the end of the alter, a table rename is done.

pt-online-schema-change can be controlled in various ways when starting the change, including controlling the size of the chunks, and specifying which replicas will be monitored for replication lag.

When starting the tool, you can also specify the thresholds of variables related to workload and replication lag.

To avoid a major limitation, don’t drop the old table right after the table rename. Do this at a later time when load is quiet, to avoid stalls.

The use of triggers has been the most widely critiqued feature of pt-online-schema-change, that being a poorly optimized part of the database code.

Other limitations exist around foreign keys, but this can be configured to a certain degree.

Especially note the bug around foreign keys including an underscore in the name.

Another tool that is similar to pt-osc is Facebook’s OSC, created specifically for their environment, but open-sourced and available to all, so I think it’s interesting to look at.

Facebook osc works similarly to pt-osc. We create a table with the new structure and then add some triggers on the old table.

We start dumping rows in chunks from the source table and loading onto the new table, while replaying changes.

There is a checksum performed, doing a consistent snapshot at the beginning, and then a quick replay to fill the gap between shadow and existing table.

At the end, the old and new tables are swapped.

The most important thing to keep in mind is the tool is designed to run rolling schema updates, as changes do not propagate to the slaves.

The main difference with pt-osc is the change capture table that is written to by the triggers, rather than the target table itself, which allows the change replay to be asynchronous.

Also rather than insert/select as pt-osc does, the tool uses select into outfile and load data infile. This avoids the use of gap-lock mechanism thus reducing locking.

Since the tool relies on a file with a create table statement, you can easily integrate changes with your source control mechanism.

The main limitations of Facebook online schema change are no support for triggers and foreign keys.

Only one migration can be run at a time.

Python 2.7 support is tricky if you are running an older distribution.

Your application needs to be able to tolerate “table not found” errors for a short-period.

As mentioned before, the tool is meant to be run on each of your servers individually.

Issues with trigger-based tools led to the development of Github’s gh-ost, which is a triggerless online schema migration solution.

It is testable and provides the ability to pause changes, dynamic control/reconfiguration, auditing, and many operational perks.

Gh-ost works in the same way as other tools by copying the table rows in chunks, but rather than using triggers, gh-ost will attach itself as a virtual replica, and capture ongoing changes directly from the binary logs.

In addition to the benefits of not having triggers, gh-ost can be reconfigured on the fly by sending commands to a UNIX socket.

Gh-ost can attach to either the master or a slave, as long as log-slave-updates is set. An accurate progress counter, which requires a full scan of the table is also available to be offloaded to a slave.

gh-ost can operate directly on a slave, and also do a test migration. In that mode, the replication is left stopped so you can manually compare and checksum the old and new tables.

One of the available mechanisms for throttling is the usual check for threads connected/running. If you hit max-load, the operation is paused, while critical load is used to abort the change. There are some thresholds available to do a second check before aborting.

The throttling configuration can be set to check for slave lag; you can also specify which slaves are monitored.

Since there are no triggers, you can interactively pause an ongoing change by sending the command to the unix socket.

The dynamic reconfiguration is possible by sending the commands via unix socket. You can check the status and also query or modify the chunk size. The operation can also be throttled manually.

The cutover can be delayed by specifying a command line argument, and manually triggered on a low traffic period.

If the interactive commands don’t work because the -U flag is not available, be sure you have the correct version of netcat installed.

The main thing that prevents using gh-ost is If you have foreign keys or triggers.

You also need at least a slave running row-based replication, although gh-ost can flip the format for you if you let it.

The newer columns are also not supported yet, and if you have a master-master setup where you write to both masters at the same time that is also a problem.

Now we move on to some use cases, and let the battle begin!

RDS and Aurora take away many complexities of db administration, so you can think of your database as a black box

Unique to Aurora, consider that the underlying storage for nodes is shared.

Any online alter method or tool must take these two constraints into account.

Related, stopping and starting the replica is done through procedures, not directly in the database.

Let’s battle!

Given the differences in database administration statements, there are several complications.

In Amazon’s version of MySQL, either RDS MySQL or Aurora, privileges are limited. This often turns out to be an issue with operational tasks, and online schema changes are no exception.

Commands using “super” are disabled, processlist doesn’t give accessible IP information, filters are in place, read_only can’t be modified.

Also, binlogs aren’t enabled by default, but these can be configured by setting the backup retention appropriately.

With Amazon, gh-ost wins! The additional flexibility provided by dynamic reconfiguration, plus reduced overhead due to no triggers gives gh-ost the crown here, as long as you don’t have Foreign Keys

For Aurora, the only available option is to run against the master, as read replicas are effectively using the same underlying database files

Enable binlogs, and set binlog format to “row” in the parameter group.

For gh-ost, use assume RBR so it does not get stuck on trying to set the binlog format, and use allow on master (specify the master in the host flag).

And here’s an example of the gh-ost statement.

For rds the recommended way is to run the tool against a replica. We also need to manually give the master external ip address

And here’s an example of the gh-ost statement.

Next contestant up is Tungsten.

Tungsten uses an external replication process, as a replacement to the native built-in replication.

Notably when doing operational tasks such as online schema changes, replication status via “show slave status” is not seen inside the database.

Let’s battle!

The complications with native online DDL remain with Tungsten, and additional complications arise with external tools.

With pt-osc, because of the lack of replication information, the replicas have to be specified manually via the DSN table. And because of the triggers used in pt-osc, binlog format matters. With statement-based replication, pt-osc works out of the box. With row-based replication, you’ll need to use a Tungsten plugin. Mixed replication is not recommended.

With gh-ost, you can’t detect replication because of the lack of status within the database, and you can’t start or stop replication as well.

For Tungsten, gh-ost wins. It’s the least difficult to configure correctly for Tungsten, and allows more control than pt-osc.

Gh-ost can be configured with Tungsten to run against the master or a replica. Here’s an overview of the process when running against a replica, which is what we’ll recommend here.

Some important configuration notes: You will need to set log-slave-updates in the Tungsten configuration. That’s in the Tungsten configuration, distinct from the database configuration.

In the gh-ost statement, specify the master with assume-master-host, and use the tungsten flag.

Gh-ost requires binlog format to be set to row wherever it is running, so in this case, you’ll set the replica’s binlog-format to row. If this isn’t already set, you will need to restart the Tungsten replicator as well.

And here’s an example of the gh-ost configuration.

Up next, Galera.

Galera has built-in methods for making schema changes, but completely online DDL is not one of them.

You can choose total order isolation, but this blocks all writes to the cluster. Without taking application downtime, we don’t want this.

You can use rolling schema updates, removing one node at a time as described earlier in alternate methods. This method incurs the same complications as described earlier: it is a manual process and requires the application to be forward- and backward-compatible.

Maybe we should use an alternate tool.

Let’s battle!

Alternate tools have their own complications, but do a better job with online alters than the built-in methods.

With pt-osc, only innodb is supported, and the online schema upgrade configuration has to be set to total order isolation, so there will be minimal blocking.

With gh-ost, there’s no official support, so you can run into untested or undocumented issues. In Galera 5.6, there was a workaround to use the two-step cutover, but this does not work in Galera 5.7.

And pt-osc wins! It is the more straight forward option for Galera cluster.

To use pt-osc with Galera, set the online schema upgrade setting to total order isolation. This does not cause downtime, but indicates that Galera will process the changes on all nodes.

Other than that, no special arguments are required. You won’t see any replication lag.

(Valerie takes over)

Multisource replication can introduce several operational complexities, online schema change among them.

Here is what multisource replication looks like. It’s often found in environments running MariaDB, because it has been available for a while.

The main disadvantage to using native online alter with this setup is the space on the replica, which is already at a premium. If the change is not metadata-only, then you can get a surprise in disk utilization on the replica.

Let’s battle!

Gh-ost wins here, for the ability to control the change in what can be a complex replication environment. If we run into any surprises with disk utilization or otherwise, we can pause and sort it out.

Run gh-ost on the replica, specifying assume-master-host so gh-ost knows which master the table change should be run on.

Daisy-chained replication introduces similar complexities with operational tasks.

Here’s an example of daisy-chained replication. The middle replica has log-slave-updates turned on, and changes are sent down the line to another replica.

Let’s battle!

Because gh-ost is unwilling to set RBR when it detects a slave, you would need to set binlog_format=ROW on the intermediate slave beforehand.

To avoid that additional change, pt-osc wins here for changes that aren’t metadata only.

Run pt online schema change against the master, of course.

Daisy-chained replication can incur replication lag on a downstream replica as a matter of course. If that’s the case in your environment, exclude the downstream replica in any replication lag checks in the pt-osc configuration to avoid unnecessary pauses in the change.

Now for two use cases that are fairly common in our experience: very large tables, and very busy tables.

The first we’ll tackle is very large tables.

What is very large? Let’s define that somewhat vaguely as having many rows of a wide table. That means: it depends. But you know very large when you see it in your environment.

(We will exclude for the moment the once-in-a-lifetime very-very-very large table situation, for which there’s no canned recommendation.)

Let’s battle!

All the things that can complicate a schema change become exacerbated with a very large table. Replication lag becomes laggier. Expensive rollbacks become unaffordable. Disk space headroom is likely not available. You’re getting major resource contention.

In this case, pt-online-schema-change wins, because in both benchmarks and real world cases it has been proven to be faster.

When you’re considering all the complications above, reducing the time a change runs by even 12 hours can be positive.

Be absolutely sure you have the disk space required before starting, though (see very-very-very large table disclaimer above).

Again, test in your environment with your workload. Experiment with setting flags in a test environment (you won’t be able to change those on the fly).

Set high timeouts and retries to allow the tool to work without constant throttling.

Here are some pt-osc typical settings for a very large table.

The other common scenario is a very busy table. Unlike the case with very large tables, recommendations here can cover very-very-very busy tables as well.

What is “busy”? Busy tables are those with many changes in a short period of time. Think of an e-commerce site on Black Friday.

Busy tables can be large tables, and so this use case is often intertwined with the previous one. There’s a particular feature of busy tables to take into consideration, though, when making a recommendation.

Let’s battle!

Again, the typical complications with online changes become exacerbated with very busy tables.

There is an additional problem with native online DDL. If the change buffer used fills up, the operation will fail. If you configure this to be too high, the table lock at the end of the change will be extended.

In the case of very busy tables, gh-ost wins. This is even true when the table is very large, because when a table is very busy, there are typically times when it is more busy (think about the workload in the daytime versus middle-of-the-night), and you will want the ability to pause a change on a very busy table during a very-very-very busy time. So if you have a change against a very large table that will require pausing during a very busy time during the day, you’ll want to pause it.

Plan ahead, and use your testing to predict how long this change will take. Time it so that the cutover is done manually after the change is complete, to time it with any application changes and to avoid doing that during a very-very-very busy time.

REDUNDANT -- fix Valerie

Again, external tools are going to work better than native online DDL.

Configure the change to allow for small chunks. In this case, you’ll want low timeouts to avoid performance degradation, and configure for retries.

As mentioned above, schedule the cutover to happen manually at a time when there is low traffic workload.

And here’s how you would configure gh-ost to run a change with a very busy table.

We’ve refereed a battle for several setups you’re likely to run into. What we haven’t talked about is a change that isn’t going to require a table copy.

A handful of changes are only going to change the metadata. It’s useful to check documentation for your version of MariaDB or MySQL to see if a change is done only to that metadata, before engaging one of the complex processes described above.

A common example is dropping an index. This works on metadata only in most recent versions of both MariaDB and MySQL.

Let’s battle! We’ll say our battle involves a change to drop an index.

And in this case, native online DDL wins.

So keep an eye on release notes and documentation. Database products continue to mature, and continue to allow more online features.

OK, that wraps up our online schema change battle.

Are there any questions?

M|18 Battle of the Online Schema Change Methods

You are reading a preview.

M|18 Battle of the Online Schema Change Methods

Editor's Notes

Editor's Notes

More Related Content

Slideshows for you

Similar to M|18 Battle of the Online Schema Change Methods

More from MariaDB Corporation

Featured

Related Books

Related Audiobooks

M|18 Battle of the Online Schema Change Methods

You are reading a preview.

M|18 Battle of the Online Schema Change Methods

Editor's Notes

Editor's Notes

More Related Content

Slideshows for you

Similar to M|18 Battle of the Online Schema Change Methods

More from MariaDB Corporation

Featured

Related Books

Related Audiobooks

Just for you: FREE 60-day trial to the world’s largest digital library.