This document discusses Spirit, an online schema change utility for MySQL 8.0. It begins by covering the state of DDL operations in MySQL and how Spirit works to perform schema changes without blocking reads or writes. It then discusses optimizations Spirit uses and features like checkpointing. Finally, it outlines some feature requests to make more operations instant or inplace in MySQL to reduce the need for Spirit in many cases.

1. Spirit
Online Schema
Change Utility
for MySQL 8.0
1
Morgan Tocker
MySQL Belgian Days 2024

2. Agenda.
1. State of DDL
2. How Spirit works
3. Optimizations
4. Feature requests for MySQL DDL
2

3. ALTER TABLE t1 _
3
* Simplified
ADD COLUMN, DROP COLUMN, RENAME COLUMN,
CHANGE DEFAULT VALUE, CHANGE ENUM/SET, RENAME TABLE
INSTANT
INPLACE
COPY
DROP INDEX, RENAME INDEX, ADD INDEX, ADD PARTITION, DROP
PARTITION, TRUNCATE PARTITION, CHANGE NULL/NOT NULL,
INCREASE VARCHAR (same ptr len)
CHANGE DATA TYPE, ADD PRIMARY KEY,
DROP PRIMARY KEY, CHANGE CHARACTER SET

4. INSTANT
■ New to MySQL 8.0
■ Works 99% as described
■ Nuanced exceptions:
■ Max 64 instant ADD/DROP
columns before a fully copy
■ Compressed tables
■ Temporary tables
■ Tables with FULLTEXT
indexes
4
Production
Safe
Assert with:
ALTER TABLE t1 ADD newcol
INT, ALGORITHM=INSTANT;

5. INPLACE
Time varies between operations
Blocks InnoDB Purge
5
Production
Safe
Statement Algorithm Time
Drop Index DROP INDEX b INPLACE 0.00s
Extend Varchar (same ptr len) MODIFY b VARCHAR(61) INPLACE 0.00s
Add Column ADD COLUMN d INT INSTANT 0.08s
Add Index ADD INDEX(b) INPLACE 8.49s
Extend Varchar (larger ptr len) MODIFY b VARCHAR(70) COPY 16.23s
Set NULL MODIFY b VARCHAR(60) NULL INPLACE 18.74s
Set NOT NULL MODIFY b VARCHAR(60) NOT NULL INPLACE 18.83s
NULL ALTER ENGINE=InnoDB COPY 18.88s

6. COPY
■ Rebuilds the whole table
■ Some operations are also
blocking:
“Permits Concurrent DML: No”
ALTER TABLE t1
ALGORITHM=copy, LOCK=none;
6
Production
Safe

7. How do you make schema changes then?
7
1. Once upon a time:
replication flipping
(aka blue green)
■ Not safe with Row-based
Replication
2. Use an external tool:
■ Pt-online-schema-change
■ Gh-ost
■ Spirit (new!)

8. Agenda.
1. State of DDL
2. How Spirit works
3. Optimizations
4. Feature requests for MySQL DDL
8

9. Basic Algorithm
9
■ Copy table definition to a
new table
■ Alter the new table
■ Start subscribing to
replication changes
■ Copy rows from origin to
new
■ Apply changes from
replication
■ Cutover

10. Nuanced Algorithm
10
■ Attempt INSTANT DDL
■ Parse DDL and apply
INPLACE if only
DROP/RENAME index
■ Copy table definition to a
new table
■ Alter the new table
■ Start subscribing to
replication changes
■ Copy rows from origin to
new
■ Apply changes from
replication
■ Checksum
■ Cutover

11. Agenda.
1. State of DDL
2. How Spirit works
3. Optimizations
4. Feature requests for MySQL DDL
11

12. Limitations (for some)
12
■ Only MySQL 8.0
■ Must have a Primary Key
■ No FOREIGN KEY support
■ No RENAME column
■ No Triggers
■ No lossy conversions
■ Replication Throttling to
minimum of 10s fidelity

13. Dynamic Chunking
Optimizations: Copying Rows
13
Parallel Copy
Batching based
on time not rows
(default: 500ms)
Copied in N
threads
(default: 4)

14. Optimizations: Apply Changes
14
■ Delta Map / Parallel Apply
■ Requires memory comparable
Primary Key
■ Key Above Watermark
■ Currently requires
auto-increment Primary Key

15. Features: Checkpointing
15
■ Every ~1m save point the
copy/apply has safely
completed
■ Allows for resume of
changes

16. Agenda.
1. State of DDL
2. How Spirit works
3. Optimizations
4. Feature requests for MySQL DDL
16

17. BUG #113355
17
ADD COLUMN, DROP COLUMN, RENAME COLUMN,
CHANGE DEFAULT VALUE, CHANGE ENUM/SET, RENAME TABLE
INSTANT
INPLACE
COPY
DROP INDEX, RENAME INDEX, ADD INDEX, ADD PARTITION, DROP
PARTITION, TRUNCATE PARTITION, CHANGE NULL/NOT NULL,
INCREASE VARCHAR (same ptr len)
CHANGE DATA TYPE, ADD PRIMARY KEY,
DROP PRIMARY KEY, CHANGE CHARACTER SET
DROP/RENAME INDEX as INSTANT

18. BUG #113476
18
ADD COLUMN, DROP COLUMN, RENAME COLUMN,
CHANGE DEFAULT VALUE, CHANGE ENUM/SET, RENAME TABLE
INSTANT
INPLACE
COPY
DROP INDEX, RENAME INDEX, ADD INDEX, ADD PARTITION, DROP
PARTITION, TRUNCATE PARTITION, CHANGE NULL/NOT NULL,
INCREASE VARCHAR (same ptr len)
CHANGE DATA TYPE, ADD PRIMARY KEY,
DROP PRIMARY KEY, CHANGE CHARACTER SET
Long running INPLACE grows HLL

19. BUG #113355
19
ADD COLUMN, DROP COLUMN, RENAME COLUMN,
CHANGE DEFAULT VALUE, CHANGE ENUM/SET, RENAME TABLE
INSTANT
INPLACE
COPY
DROP INDEX, RENAME INDEX, ADD INDEX, ADD PARTITION, DROP
PARTITION, TRUNCATE PARTITION, CHANGE NULL/NOT NULL,
INCREASE VARCHAR (same ptr len)
CHANGE DATA TYPE, ADD PRIMARY KEY,
DROP PRIMARY KEY, CHANGE CHARACTER SET
Support INSTANT for some change type

20. The Potential Future
20
ADD COLUMN, DROP COLUMN, DROP INDEX, CHANGE DATA TYPE
(expansion), RENAME COLUMN, CHANGE DEFAULT VALUE, CHANGE
ENUM/SET
INSTANT
INPLACE
SPIRIT
ADD INDEX
CHANGE DATA TYPE (other), CHANGE NULL/NOT NULL
vs OUR MOST COMMON DDLs

21. Conclusion
Instant DDL is awesome
Large schema changes #1
blocker to larger databases
Spirit is symbiotic
It attempts MySQL DDL
first, falls back as required
As more changes become
instant, Spirit will take
advantage
21
github.com/cashapp/spirit
Apache 2.0 License