Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Big Tables and You
Keeping DDL operations fast
So You Want To Add a New Column
class AddFoo < ActiveRecord::Migration
def self.change
add_column :foo, :bar, :string
end
...
What is this doing?
“ALTER TABLE makes a temporary copy of the original
table...waits for other operations that are modify...
Where Did Production Go?!
Whats wrong with this approach?
Write operations are stalled and you’ve just crashed
production
Multiple ALTER statements ...
Demo!
ruby> File.open('/tmp/foo','w') {|f| (1..10_000_000).to_a.
each{|r|f.puts(r)} } # 10 million rows
mysql> CREATE DATA...
Demo! (Continued)
mysql> ALTER TABLE foo ADD COLUMN baz varchar
(256);
Query OK, 10000000 rows affected (42.97 sec)
Record...
Rethinking DDL Changes
“ALTER TABLE makes a temporary copy of the original
table...waits for other operations that are mod...
DDL Plan of Attack
CREATE TABLE foo_temp LIKE foo;
ALTER TABLE foo_temp ADD COLUMN baz varchar
(256);
INSERT INTO foo_temp...
What Changes?
90% of the time in “copy to tmp table”
to
90% of our time in “Sending data” (non
blocking)
This means record...
Enter MySQL Big Table Migration
A Rails plugin that adds methods to
ActiveRecord::Migration to allow columns
and indexes t...
Example
class AddBazToFoo < ActiveRecord::Migration
def self.up
add_column_using_tmp_table :foo, :baz, :string
end
end
Additional Methods
● add_column_using_tmp_table
● remove_column_using_tmp_table
● rename_column_using_tmp_table
● change_c...
When Should This Be Used?
A good rule of thumb is any table already in
production
Another rule of thumb is any table with ...
The “Meat”
def with_tmp_table(table_name)
say "Creating temporary table #{new_table_name} like #
{table_name}..."
# DDL op...
Demo!
rails new temp_table_demo
# Gemfile
gem 'mysql_big_table_migration', git: 'git@github.com:
thickpaddy/mysql_big_tabl...
Questions from the Audience
Q&A
Upcoming SlideShare
Loading in …5
×

Big tables and you - Keeping DDL operatations fast

1,932 views

Published on

A discussion on applying DDLs using a temp table pattern to allow for non blocking updates to the database. Includes a demonstration of the gem mysql_big_table_migration

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Big tables and you - Keeping DDL operatations fast

  1. 1. Big Tables and You Keeping DDL operations fast
  2. 2. So You Want To Add a New Column class AddFoo < ActiveRecord::Migration def self.change add_column :foo, :bar, :string end end ALTER TABLE foo ADD COLUMN bar varchar(256);
  3. 3. What is this doing? “ALTER TABLE makes a temporary copy of the original table...waits for other operations that are modifying the table...incorporates the alteration into the copy, deletes the original table, and renames the new one. While ALTER TABLE is executing, the original table is readable… ...writes to the table that begin after the ALTER TABLE operation begins are stalled until the new table is ready…”
  4. 4. Where Did Production Go?!
  5. 5. Whats wrong with this approach? Write operations are stalled and you’ve just crashed production Multiple ALTER statements are applied separately making the time to execute T(n*rows) Worse with indexes
  6. 6. Demo! ruby> File.open('/tmp/foo','w') {|f| (1..10_000_000).to_a. each{|r|f.puts(r)} } # 10 million rows mysql> CREATE DATABASE temp_table_demo; mysql> USE temp_table_demo; mysql> CREATE TABLE foo (id int PRIMARY KEY AUTO_INCREMENT, bar VARCHAR(256)); mysql> LOAD DATA INFILE "/tmp/foo" INTO TABLE foo;
  7. 7. Demo! (Continued) mysql> ALTER TABLE foo ADD COLUMN baz varchar (256); Query OK, 10000000 rows affected (42.97 sec) Records: 10000000 Duplicates: 0 Warnings: 0 mysql> SHOW PROCESSLIST; “State” => “copy to tmp table” ~90% of the execution time
  8. 8. Rethinking DDL Changes “ALTER TABLE makes a temporary copy of the original table...waits for other operations that are modifying the table...incorporates the alteration into the copy, deletes the original table, and renames the new one. While ALTER TABLE is executing, the original table is readable… We can 1) make a temporary copy 2) incorporate changes 3) sync 4) delete 5) rename
  9. 9. DDL Plan of Attack CREATE TABLE foo_temp LIKE foo; ALTER TABLE foo_temp ADD COLUMN baz varchar (256); INSERT INTO foo_temp (id,bar) SELECT * FROM foo; # Syncing checks here for records modified during change DROP TABLE foo; RENAME TABLE foo_temp TO foo;
  10. 10. What Changes? 90% of the time in “copy to tmp table” to 90% of our time in “Sending data” (non blocking) This means records can be inserted, updated, deleted without waiting for table metadata lock
  11. 11. Enter MySQL Big Table Migration A Rails plugin that adds methods to ActiveRecord::Migration to allow columns and indexes to be added to and removed from large tables with millions of rows in MySQL, without leaving processes seemingly stalled in state "copy to tmp table".
  12. 12. Example class AddBazToFoo < ActiveRecord::Migration def self.up add_column_using_tmp_table :foo, :baz, :string end end
  13. 13. Additional Methods ● add_column_using_tmp_table ● remove_column_using_tmp_table ● rename_column_using_tmp_table ● change_column_using_tmp_table ● add_index_using_tmp_table ● remove_index_using_tmp_table
  14. 14. When Should This Be Used? A good rule of thumb is any table already in production Another rule of thumb is any table with more than 1 million rows Not necessary for small, or new tables
  15. 15. The “Meat” def with_tmp_table(table_name) say "Creating temporary table #{new_table_name} like # {table_name}..." # DDL operations performed on temp table say "Inserting into temporary table in batches of #{batch_size}..." say "Replacing source table with temporary table..." say "Cleaning up, checking for rows created/updated during migration, dropping old table..." end
  16. 16. Demo! rails new temp_table_demo # Gemfile gem 'mysql_big_table_migration', git: 'git@github.com: thickpaddy/mysql_big_table_migration.git' Run DDLs with and without temp table pattern
  17. 17. Questions from the Audience Q&A

×