Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dd and atomic ddl pl17 dublin

1,262 views

Published on

MySQL 8.0 implements atomic DDLs.

Published in: Software
  • Be the first to comment

Dd and atomic ddl pl17 dublin

  1. 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL 8.0: Atomic DDLs – Implementation and Impact Ståle Deraas, Senior Development Manager Oracle, MySQL 26 Sept 2017 Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
  2. 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  3. 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Agenda Requirements for a Data Dictionary Storage Engine Requirements for an SE implementing Atomic DDLs Effects of new INFORMATION_SCHEMA in MySQL 8.0 1 2 3
  4. 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Requirements for a Data Dictionary Storage Engine
  5. 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Data Dictionary before MySQL 8.0 5 Data Dictionary Files FRM TRG OPT System Tables (mysql.) user procevents InnoDB System Tables MyISAM File system InnoDB SQL
  6. 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 6 Data Dictionary DD SE SQL DD TableDD TableDD Table
  7. 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Requirements for Data Dictionary Storage Engine • Some special care need to be taken at initialize/bootstrap • Must support transactions • Must support “attachable transactions” – To open a table, the DD SE needs to open DD tables • In MySQL 8.0 InnoDB has dual role – MySQL Data Dictionary Storage Engine (DD SE) – Storage Engine that supports Atomic DDL 7
  8. 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | The Data Dictionary Storage Engine API • An addition to the Storage Engine API • Not strictly separated from SE API • Some examples of functionality that the DD SE must implement: - Initialize dictionary support - Initiate recovery of dictionary transactions - Reset of the DD SE internal cache of meta data 8
  9. 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Server Data Dictionary Storage Engine 9 Query Executor Optimizer Storage Engine SQL statement Client Parser Result Data Dictionary DD Storage Engine
  10. 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Example - DROP SCHEMA at high level 10 MySQL 5.7 • Delete tables – Metadata, TRN/TRG/FRM files – Data, InnoDB tables • Delete stored programs – Metadata, rows in MyISAM (non- transactional) • Delete schema – Metadata, DB.OPT file Mix of filesystem, non- transactional/transactional storage and multiple commits MySQL 8.0 • Delete tables – Metadata, rows in InnoDB – Data, InnoDB tables • Delete stored programs – Metadata, rows in InnoDB • Delete schema – Metadata, rows in InnoDB Updates to transactional storage, one commit
  11. 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Requirements for an SE implementing Atomic DDLs
  12. 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Why Atomic DDL support in MySQL 8.0 • Ensure we have a consistent state after a DDL operation – Prevent slave drift – Prevent internal inconsistencies • At last possible due to transactional storage of meta data • Enables implemention of crash-safe DDL • Enables addressing longstanding issues 12
  13. 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | User visible changes to Atomic TABLE DDL in MySQL 8.0 • DROP TABLES , all tables dropped or none • DROP SCHEMA, all entities in the schema are dropped, or none • Note that atomic DDL statements will be rolled back or committed even in case of crash, e.g. RENAME TABLES • CREATE TABLE would be successfully committed or rolled back (no orphan ibd left) • TRUNCATE TABLE (including InnoDB tables with FTS AUX tables) would be successfully committed or rolled back • RENAME TABLES, all or none • ALTER TABLE successful or not done 13
  14. 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Storing SE private data (prerequisite) • An SE which will support atomic DDL can store private data in SE private columns of Data Dictionary tables • InnoDB has chosen to store key-value pairs in the SE private data • Examples: index root page number, InnoDB table ID, tablespace ID etc. 14
  15. 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How do we implement Atomic and Crash-Safe TABLE DDL • At high level, create a single atomic transaction: – Do updates to the Data Dictionary (DD SE) – Do necessary changes in the Storage Engine – Do necessary writes to binary log – Since it is a single transaction, all updates to Data Dictionary can be rolled back and are crash safe 15
  16. 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How do we implement Atomic and Crash-Safe TABLE DDL • At more detailed level: – Make sure there are no intermediate commits on SQL layer during DDL – SE methods should have no intermediate commits, and register as part of trx – SE can do redo/rollback of DDL , implemented with DDL_LOG in InnoDB – SQL layer will invoke a post_ddl() hook, so SE can do post commit/rollback work – As part of the transaction, write to binary log – Introduced the SE property HTON_SUPPORTS_ATOMIC_DDL, that can be tested for in the source code 16
  17. 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB: Features in SE for Atomic and Crash-Safe TABLE DDL • DDL_LOG – Protected table in mysql tablespace, so no DDL and no user DML allowed – Add trx Ids to all entries, which will be deleted by post_ddl() hook – Changes to DDL_LOG are persisted ASAP. Will prevent data files updates and no flushing of DDL_LOG entries (exempted from innodb_flush_log_at_trx_commit control for delayed flush) • Issue with physical files and index tree: – Before MySQL 8.0, could leave “orphan” InnoDB ibd files, “orphan” index trees/pages – Orphan files/trees cause problems for subsequent DDLs and waste tablespace – With MySQL 8.0, DDLs “crash safe”, no ibd files/index trees orphaned if no commit – At commit, the old ibd files/index trees will be dropped 17
  18. 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - CREATE TABLE, SQL Perspective 18 MySQL 5.7 • At SQL layer: create table obj • Store obj in DD tables, commit • «Open» table, construct all internal structures • Call to SE handler::create(name,...) MySQL 8.0 • At SQL layer: create table obj • Call to SE, adding SE private data • Store obj in DD tables * • «Open» table, construct all internal structures • Call to SE handler::create(name,...), which can update SE private data • Store obj in DD tables * * = commit for non-atomic SE
  19. 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - CREATE TABLE, SQL Perspective CONT 19 MySQL 5.7 • SE creates physical files, its own artifacts such as InnoDB tablespaces. InnoDB stores data in own system tables. • Write statement to binary log MySQL 8.0 • Write statement to binary log • Commit/Rollback, post_ddl() hook
  20. 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - CREATE TABLE, SE perspective 20 MySQL 5.7 • Call to SE handler::create(name,.) • SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees • SE starts its own transaction, insert the new table/indexes's metadata to InnoDB own System tables MySQL 8.0 • Call to SE handler::create(name,...) • SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees • SE logs above physical file and indexes creation • Note: No separate SE transaction, all info for new tablespace/indexes are passed back to server with DD objects
  21. 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - CREATE TABLE, SE perspective cont 21 MySQL 5.7 • Note: If crash happens before above trx commit, there could be orphan files or index trees left MySQL 8.0 • SQL layer commits or rollback • Call to SE post_ddl(). If rollback, the post_ddl() physically delete the tablespace/ibd (file-per-table) and drop the index trees for the table • Note: If crash, SE will recover info in DDL_LOG to delete the ibd/index trees
  22. 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - DROP TABLES, SQL Perspective 22 MySQL 5.7 • For each table in table list – Call SE handler:ha_delete_table() and if error do appropriate action – Remove for Data Dictionary and commit • Write up to 3 artifical DROP TABLES to binary log for dropped tables (temp tables *2 + all base tables) MySQL 8.0 – For each table in table list • Check for 5 types of tables – Err out if non-existing tables and no IF EXISTS clause – Handle tables in non-atomic DDL SE • Delete Table in SE, and del in DD, commit – Handle tables in atomic DDL SE • Delete Table in SE, and del in DD, no commit – Write DROP TABLES statment to binary log incl all table (in «no GTID» mode) – Commit/rollback , post_ddl() hook • Concurrent DDL on table blocked
  23. 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - DROP TABLES, SQL Perspective cont 23 MySQL 5.7 MySQL 8.0 – If GTID mode, and write to binlog postponed, write DROP TABLES stmt with all tables we dropped to binlog • To handle GTID mode from older servers – For each temp table in non- transactional SE • Drop table, call SE handler::delete_table() – Write DROP TEMPORARY TABLES for all temp tables above to binary log – For each temp table in transactional SE • Drop table, call SE handler::delete_table() – Write DROP TEMPORARY TABLES for all temp tables above to binary log
  24. 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - DROP TABLES, SE perspective 24 MySQL 5.7 • Call to SE handler:ha_delete_table() • InnoDB starts its own transaction and delete metadata from the InnoDB own system tables • InnoDB commit trx • InnoDB deletes physical files/ibd (in file-per-table case) • Note: If crash happens after InnoDB commit trx, there could be orphaned files/ibd or trees MySQL 8.0 • Call to SE handler:ha_delete_table() • InnoDB only logs actions to delete ibd file or index trees in DDL_LOG, so no physical action (No InnoDB system tables needs to be updated) • Back to SQL layer to commit - Delete metadata in DD system tables and commit
  25. 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL: Use Case - DROP TABLES, SE perspective cont 25 MySQL 5.7 MySQL 8.0 • SQL layer calls post_ddl(). If commit, InnoDB physically deletes ibd and drop index trees. If rollback, delete entries in DDL_LOG, files and trees left as they are • Note: If crash and DROP trx committed, recover info from DDL_log, and delete files/trees during crash recovery
  26. 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How do we implement Atomic and Crash-Safe non-TABLE DDL • At slightly detailed level (this is all at SQL layer): – Make sure there are no intermediate commits during DDL – Write to binary log as part of the DDL transaction – Ensure caches for Data Dictionary, routines, events and UDFs are consistent with DDL status – Ensure user visible behaviour is atomic – NOTE: Behaviour of DROP VIEW changes, for list of views • Pre-MySQL 8.0: Giving an error for non-existent views and removing existing views • MySQL 8.0: Giving an error for non-existent views 26
  27. 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL Non-TABLE: Use case - CREATE ROUTINE 27 MySQL 5.7 • Create routine obj • Store obj in DD tables, commit • For SF, find VIEWs usage and update view cols metadata for each view. Commit for each view. • Invalidate cache • Write stored routine statement to binary log MySQL 8.0 • Create routine obj • Store obj in DD tables • For SF, find VIEWs usage and update view cols metadata for each view • Write stored routine event to binary log • Commit and invalidate cache, or on error, rollback and give msg
  28. 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL for user management: Overview 28 MySQL 5.7 • Partial execution is allowed – Make persistent changes to ACL tables/caches – Throw error • Error propagation to binary log – Required for HA setup – Other nodes expect same error while executing the statement MySQL 8.0 • Execution is atomic • A statement either fully succeeds or is rolled back • No error propagation to binary log • Only a successful statment makes it way to binary log • Simpler handling in HA setup
  29. 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL for user management: Use Case - CREATE USER 29 MySQL 5.7 • Result – foo@mysql.com is created – Error is thrown for bar@mysql.com for invalid authentication plugin • Binary log – Statement is written to binary log with password transformation – Expected error for bar@localhost.com is written MySQL 8.0 • Result • Error because of invalid authentication plugin for bar@localhost.com • foo@mysql.com is not created • Binary log • No statement is written to binary log CREATE USER `foo`@`mysql.com` IDENTIFIED BY ‘haha’, `bar`@`mysql.com` IDENTIFIED WITH ‘non_existing_plugin’ BY ‘hoho’;
  30. 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL for user management: Use Case - GRANT 30 MySQL 5.7 • Result – foo@mysql.com get ability to SELECT from all tables/views. – Error is thrown for non existing user • Binary log – Statement is written to binary log – Expected error for `non_existing`@`mysql.com` MySQL 8.0 • Result • Error because of invalid user • foo@mysql.com is not granted SELECT • Binary log • No statement is written to binary log GRANT SELECT ON *.* TO `foo`@`mysql.com`, `non_existing@mysql.com`;
  31. 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Atomic DDL for user management: Note that... 31 • IF EXISTS extensions (ALTER USER | DROP USER) – Will continue to ignore errors related to non-existing authorization IDs – All other errors will cause statement rollback • IF NOT EXISTS extensions ( CREATE USER | CREATE ROLE) – Will continue to ignore errors related to existing authorization IDs – All other errors will cause statement rollback • New MDL type to protect access to in-memory caches for user management
  32. 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Effects of new INFORMATION_SCHEMA in MySQL 8.0
  33. 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | NEW INFORMATION_SCHEMA in MySQL 8.0 33 Uniform, simpler implemention makes it a lot faster MySQL Client I_S Query Results MySQL Server Optimizer prepares execution plan. Executor reads metadata from data dictionary tables. InnoDB storage engine Return rows to user. INFORMATION_SCHEMA in 8.0 MySQL Client I_S Query Results MySQL Server Create temporary table. Heuristic optimization. Read metadata from File system or from MyISAM/InnoDB engine. . TEMP TABLE Return rows to user. INFORMATION_SCHEMA in 5.7 File system / MyISAM / InnoDB engine
  34. 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION_SCHEMA Performance and Scalability • Typically 30X performance improvements over MySQL 5.7 • More than 100X for some queries like: List all InnoDB table columns 34 I_S queries scale, both with database size and query load 0 20 40 60 80 100 120 140 160 List all InnoDB tables columns 5k tables List all InnoDB tables columns 10k tables MySQL 8.0 MySQL 5.7 Time in Seconds (Lower is better)
  35. 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 35 100 schemas times 50 tables (5000 tables) INFORMATION_SCHEMA Performance 0 0.5 1 1.5 2 2.5 3 3.5 4 Count All Schemas Schema aggregate size stats All Dynamic Table Info All Static Table Info Auto Increments Near Limit Count All Columns Count All Indexes MySQL 8.0 MySQL 5.7 Time in Seconds (Lower is better)
  36. 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION_SCHEMA in MySQL 8.0, community feedback • In MySQL 8.0 DMRs : Option “information_schema_stats” introduced – TABLES.TABLE_ROWS, TABLES.DATA_FREE, … • Handled by opening table and retrieving data from SE in MySQL 5.7 – expensive!!! • Bugreport: Nicolai Plum bug#83957 and Peter Brawley bug#87548 • Feedback from several MySQL ACEs, that found inital solution confusing • For MySQL 8.0.3 RC1 We have created an improved solution 36
  37. 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION_SCHEMA in MySQL 8.0.3 • New option: information_schema_stats_expiry – Default 24h – Cache dynamic values in mysql.index_stats and mysql.table_stats – Set to 0, always fetch data from SE and no «stats» will be stored – ANALYZE will update mysql.index_stats and mysql.table_stats • Improved performance fetching dynamic data from SE (InnoDB) – In 8.0.1 DMR, typically 2-3 times faster than 5.7 – In 8.0.3 RC1, typically 4-5 times faster than 5.7 • Option “information_schema_stats” removed 37
  38. 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Try for yourself! • Downloadable 8.0.3 RC1 – dev.mysql.com • Enjoy and give us your feedback! • Thank you for listening • http://mysqlserverteam.com • For details on atomic DDL: – WL#7743, WL#9173, WL#9045, WL#9175, WL#9045, WL#9536 38

×