We all have tasks from time to time for bulk-loading external data into MySQL. What's the best way of doing this? That's the task I faced recently when I was asked to help benchmark a multi-terrabyte database. We had to find the most efficient method to reload test data repeatedly without taking days to do it each time. In my presentation, I'll show you several alternative methods for bulk data loading, and describe the practical steps to use them efficiently. I'll cover SQL scripts, the mysqlimport tool, MySQL Workbench import, the CSV storage engine, and the Memcached API. I'll also give MySQL tuning tips for data loading, and how to use multi-threaded clients.
2. Bill Karwin
Software developer, consultant, trainer
Using MySQL since 2000
Senior Database Architect at SchoolMessenger
SQL Antipatterns: Avoiding the Pitfalls of Database
Programming
https://pragprog.com/titles/bksqla/sql-antipatterns
Oracle ACE Director
3. Load Data Fast!
Common chores
§ Dump and restore
§ Import third-party data
§ Extract, Transfer, Load (ETL)
§ Test data that needs to be reloaded
repeatedly
https://commons.wikimedia.org/wiki/File:Kitten_with_laptop_-_278017185.jpg
Is it done yet?
4. How to Speed This Up?
1. Query Solutions
2. Schema Solutions
3. Configuration Solutions
4. Parallel Execution Solutions
5. Example Table
CREATE TABLE TestTable (
id INT UNSIGNED NOT NULL PRIMARY KEY,
intCol INT UNSIGNED DEFAULT NULL,
stringCol VARCHAR(100) DEFAULT NULL,
textCol TEXT
) ENGINE=InnoDB;
Let’s load 1 million rows!
6. Best Case Performance
Running a test script to loop over 1 million rows, without inserting to a database.
$ php test-bulk-insert.php --total-rows 1000000 --noop
This should have a speed that is the upper bound for any subsequent test.
Time: 2 seconds (00:00:02)
1000000 rows = 432435.24 rows/sec
1000000 stmt = 432435.24 stmt/sec
1000000 txns = 432435.24 txns/sec
1000000 conn = 432435.24 conn/sec
7. Worst Case Performance
INSERT INTO TestTable (id, intCol, stringCol, textCol) VALUES
(?, ?, ?, ?);
Run a test script that executes one INSERT, commits, reconnects.
$ php test-bulk-insert.php --total-rows 10000
Time: 34 seconds (00:00:34)
10000 rows = 290.29 rows/sec
10000 stmt = 290.29 stmt/sec
10000 txns = 290.29 txns/sec
10000 conn = 290.29 conn/sec
10. Inserting One Row at a Time
INSERT INTO TestTable (id, intCol, stringCol, textCol) VALUES
(?, ?, ?, ?);
Run a test script that executes one INSERT, commits using a single connection.
$ php test-bulk-insert.php --total-rows 1000000
--txns-per-conn 1000000
Time: 527 seconds (00:08:47)
1000000 rows = 1894.67 rows/sec
1000000 stmt = 1894.67 stmt/sec
1000000 txns = 1894.67 txns/sec
1 conn = 0.00 conn/sec
15. Transactions
BEGIN TRANSACTION;
INSERT INTO TestTable …
INSERT INTO TestTable …
INSERT INTO TestTable …
INSERT INTO TestTable …
INSERT INTO TestTable …
INSERT INTO TestTable …
COMMIT;
Q: How many statements can you do in one transaction?
A: In theory this is constrained by undo log segments, but it's a lot.
17. Inserting with Prepared Queries
BEGIN TRANSACTION;
PREPARE INSERT INTO TestTable …
EXECUTE …
EXECUTE …
EXECUTE …
EXECUTE …
COMMIT;
Q: How many times can you execute a given prepared statement?
A: There is no limit, as far as I can tell.
20. Load Data in File: Results
mysql> LOAD DATA LOCAL INFILE 'TestTable.csv'
INTO TABLE TestTable;
https://dev.mysql.com/doc/refman/8.0/en/load-data.html
Flat-file data load in a single transaction.
Works with replication.
21. Overhead: Load Data Infile
0
50
100
150
200
250
Sending query Parsing LOAD DATA INFILE Closing query
23. Load XML in File: Results
LOAD XML LOCAL INFILE 'TestTable.xml'
INTO TABLE TestTable;
https://dev.mysql.com/doc/refman/8.0/en/load-xml.html
$ php test-bulk-insert.php --total-rows 1000000 --load-xml
Time: 77 seconds (00:01:17)
1000000 rows = 12858.16 rows/sec
1 stmt = 0.01 stmt/sec
1 txns = 0.01 txns/sec
1 conn = 0.01 conn/sec
24. What about Load JSON in File?
Sorry, the hypothetical LOAD JSON INFILE is not supported by MySQL yet.
😭
But it has been proposed as a feature request:
https://bugs.mysql.com/bug.php?id=79209
Go vote for it!
Or better yet, implement it and contribute a patch!
26. Indexes
How much overhead for one index? Two indexes?
1. mysql> ALTER TABLE TestTable ADD INDEX (intCol);
2. mysql> ALTER TABLE TextTable ADD INDEX (stringCol);
29. Index Deferral
What if we insert with no indexes, and build indexes at the end?
§ Thi is what Percona’s mysqldump --innodb-optimize-keys does.
§ Load time is like when you have no indexes:
Then create indexes after data load. This reduces the effective rate of rows/second:
mysql> ALTER TABLE TestTable ADD INDEX (intCol);
Query OK, 0 rows affected (7.02 sec)
mysql> ALTER TABLE TestTable ADD INDEX (stringCol);
Query OK, 0 rows affected (8.54 sec)
Time: 63 seconds (00:01:03)
1000000 rows = 15744.53 rows/sec
Time: 63 + 7 + 8.5 seconds (00:01:35)
1000000 rows = 12738.85 rows/sec
effective data
load rate
30. Triggers
How much overhead for a trigger?
mysql> CREATE TRIGGER TestTrigger
BEFORE INSERT ON TestTable
FOR EACH ROW
SET NEW.stringCol = UPPER(NEW.stringCol);
This is a very simple trigger. If you have more complex code, like subordinate
INSERT statements, the cost will be higher.
46. Parallel Import
Like LOAD DATA INFILE but supports multi-threaded import:
$ mysqlimport --local --use-threads 4
dbname table1 table2 table3 table4
Runs a fixed number of threads, imports one table per thread.
If an import finishes and there are more tables, first available thread does it.
https://dev.mysql.com/doc/refman/8.0/en/mysqlimport.html
47. Parallel Import
Connecting to localhost
Connecting to localhost
Connecting to localhost
Connecting to localhost
Selecting database test
Selecting database test
Selecting database test
Selecting database test
Loading data from LOCAL file: TestTable2.csv into TestTable2
Loading data from LOCAL file: TestTable3.csv into TestTable3
Loading data from LOCAL file: TestTable1.csv into TestTable1
Loading data from LOCAL file: TestTable4.csv into TestTable4
test.TestTable3: Records: 250000 Deleted: 0 Skipped: 0 Warnings: 0
Disconnecting from localhost
test.TestTable1: Records: 250000 Deleted: 0 Skipped: 0 Warnings: 0
Disconnecting from localhost
test.TestTable2: Records: 250000 Deleted: 0 Skipped: 0 Warnings: 0
Disconnecting from localhost
test.TestTable4: Records: 250000 Deleted: 0 Skipped: 0 Warnings: 0
Disconnecting from localhost
51. Want to Try The Tests Yourself?
The test-bulk-insert.php script is available here:
https://github.com/billkarwin/bk-tools
52. One Last Thing…
What Was Our Solution?
We cheated:
§ Load database once.
§ Take a filesystem snapshot.
§ Run tests.
§ Restore from snapshot.
§ Re-run tests.
§ etc.
This is not a good solution for everyone. It worked for one specific use case.
53. License and Copyright
Copyright 2017 Bill Karwin
http://www.slideshare.net/billkarwin
Released under a Creative Commons 3.0 License:
http://creativecommons.org/licenses/by-nc-nd/3.0/
You are free to share—to copy, distribute,
and transmit this work, under the following conditions:
Attribution.
You must attribute this
work to Bill Karwin.
Noncommercial.
You may not use this work
for commercial purposes.
No Derivative Works.
You may not alter,
transform, or build upon
this work.