Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0:
An Inte...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
What is a Data Dictionary?
Data Dictionary b...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What is a Data Dictionary?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data Dictionary
• Metadata is information about dat...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Server
Data Dictionary
6
Role in Server
Query...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
CREATE PROCEDURE p1(v INT)
SQL SECURITY INVOKER
BEG...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data Dictionary before MySQL 8.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
• Metadata s...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
10
Data Dict...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
• INFORMATIO...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary
MySQL 8.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
13
Data ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
• All me...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
15
Overv...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
• The da...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
InnoDB
Transactional Data Dictionary in MySQL 8.0
1...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary API in MySQL 8.0
• A ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
SDI: Ser...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
What’s i...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary Benefits
• INFORMATIO...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 22
MySQLClient
I_S Query Results
MySQL Server
Creat...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 23
MySQL Client
I_S Query Results
MySQL Server
Opti...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA Performance
Confidential – Oracl...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA Performance and Scalability
• Ty...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 26
100 schemas times 50 tables (5000 tables)
Inform...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA in MySQL 8.0: Dynamic values
• T...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION SCHEMA longstanding issues solved in 8....
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Atomic DDL
• All metadata stored in I...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Atomic DDL : Use Case - DROP DATABASE...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability – Disaster Recovery
• MySQL 8.0, metada...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL before 8.0
• Moving .FRM and data files aroun...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reliability and Ease of Use – Automated Dictionary ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Extending Dictionary Tables from Plugins
• API for ...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Try for yourself!
• Downloadable 8.0.1 DMR
– dev.my...
Upcoming SlideShare
Loading in …5
×

Data dictionary pl17

257 views

Published on

Transactional Data Dictionary in MySQL 8.0

Published in: Software
  • Be the first to comment

Data dictionary pl17

  1. 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0: An Internal Server ComponentThat Matters Ståle Deraas, Senior Development Manager Oracle, MySQL 27 Apr 2017 Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
  2. 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  3. 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Agenda What is a Data Dictionary? Data Dictionary before MySQL 8.0 Transactional Data Dictionary in MySQL 8.0 What’s in it for you? 1 2 3 4
  4. 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | What is a Data Dictionary?
  5. 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Data Dictionary • Metadata is information about data in an RDBMS – Column definitions, Index definitions, Foreign key definitions ... • Data Dictionary is a collection of metadata for all data in an RDBMS ID NAME WEIGHT HEIGHT GENDER 3 Bob 80 185 M 5 Liz 55 165 F Metadata Data
  6. 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Server Data Dictionary 6 Role in Server Query Executor Optimizer Storage Engine SQL statement Client Parser Result Data Dictionary
  7. 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | CREATE PROCEDURE p1(v INT) SQL SECURITY INVOKER BEGIN ... END Data Dictionary Types of Metadata Data Dictionary Table Definitions SP Definitions View Definitions Schemas ACL CREATE TABLE customers( id INT AUTO_INCREMENT ... PRIMARY KEY (id), INDEX ... FOREIGN KEY ... )
  8. 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Data Dictionary before MySQL 8.0
  9. 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Data Dictionary before MySQL 8.0 • Metadata stored in a mix of files and tables : – File based • FRM, TRN, TRG, OPT, PAR ... – Table based: non-transactional • mysql.proc ... – Table based: transactional • Innodb.SYS
  10. 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Data Dictionary before MySQL 8.0 10 Data Dictionary Files FRM TRG OPT System Tables (mysql.) user procevents InnoDB System Tables MyISAM File system InnoDB SQL
  11. 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Data Dictionary before MySQL 8.0 • INFORMATION_SCHEMA queries are slow • Inconsistencies due to non-transactional storage of metadata • Inconsistencies between InnoDB metadata and Server metadata • Showstopper for crash-safe / transactional DDL • Replication is challenging, as DDLs are not atomic • Difficult to extend • No uniform API 11 Problems
  12. 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary MySQL 8.0
  13. 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 13 Data Dictionary InnoDB SQL DD TableDD TableDD Table
  14. 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 • All metadata stored in tables • Single metadata repository for all MySQL server subsystems • Reliable, crash-safe InnoDB tables • INFORMATION_SCHEMA implemented as views over DD tables – Queries can be optimized – Improved performance – Simpler, uniform implementation, easier to maintain Main features
  15. 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 15 Overview InnoDB Data Dictionary DD Table User Table INFORMATION SCHEMA Views Archive User Table CSV User Table Optimizer
  16. 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 • The data dictionary schema is based on SQL standard definitions • Designed for automated upgrade of metadata • The data dictionary is designed to be easily extended for new requirements • Designed to allow plugins/components to add INFORMATION_SCHEMA views and PERFORMANCE_SCHEMA tables Main features, cont’ed
  17. 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Transactional Data Dictionary in MySQL 8.0 17 Architecture Query Executor Parser Optimizer Data Dictionary Tablespace Data Dictionary API Storage Engine PluginUser TableUser Table
  18. 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary API in MySQL 8.0 • A single way to deal with Data Dictionary – For the server core – For Storage Engines • A uniform API for all types of metadata • Provide a way to handle Storage Engine private data – Storage Engines no longer have to store their own metadata – The API enables SEs to store key-value pairs 18 Design goals
  19. 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 SDI: Serialized Dictionary Information - Copy of metadata from the Data Dictionary in JSON format - Stored in data tablespaces(InnoDB) or .sdi file (MyISAM) - Used for data migration and redundancy The InnoDB Data Dictionary tablespace is the metadata storage 19 Reliability and Redundancy InnoDB User Tablespaces User Table InnoDB Data Dictionary tablespace mysql.tables SDI ID Name 1 User Table 1 2 User Table 2
  20. 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary in MySQL 8.0 What’s in it for you?
  21. 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Transactional Data Dictionary Benefits • INFORMATION SCHEMA – Improved Performance and Scalability – Solves longstanding issues • Reliability – Atomic DDL – Serialized Dictionary Information – Automated Data Dictionary upgrade • +++ 21
  22. 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 22 MySQLClient I_S Query Results MySQL Server Create temporary table. Heuristic optimization Read metadata from File system or from MyISAM/InnoDB engine. . TEMP TABLE Return rows to user. File system / MyISAM / InnoDB engine INFORMATION SCHEMA MySQL 5.7: Multiple sources, heuristics, creation of temp tables
  23. 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 23 MySQL Client I_S Query Results MySQL Server Optimizer prepares execution plan Executor reads metadata from data dictionary tables InnoDB storage engine Return rows to user INFORMATION SCHEMA MySQL 8.0: Uniform, simple and using server standard features
  24. 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION SCHEMA Performance Confidential – Oracle Internal/Restricted/Highly Restricted 24 Uniform, simpler implemention makes it a lot faster MySQL Client I_S Query Results MySQL Server Optimizer prepares execution plan. Executor reads metadata from data dictionary tables. InnoDB storage engine Return rows to user. INFORMATION_SCHEMA in 8.0 MySQL Client I_S Query Results MySQL Server Create temporary table. Heuristic optimization. Read metadata from File system or from MyISAM/InnoDB engine. . TEMP TABLE Return rows to user. INFORMATION_SCHEMA in 5.7 File system / MyISAM / InnoDB engine
  25. 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION SCHEMA Performance and Scalability • Typically 30X performance improvements over MySQL 5.7 • More than 100X for some queries like: List all InnoDB table columns 25 I_S queries scale, both with database size and query load 0 20 40 60 80 100 120 140 160 List all InnoDB tables columns 5k tables List all InnoDB tables columns 10k tables MySQL 8.0 MySQL 5.7 Time in Seconds (Lower is better)
  26. 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 26 100 schemas times 50 tables (5000 tables) Information Schema Performance 0 0.5 1 1.5 2 2.5 3 3.5 4 Count All Schemas Schema aggregate size stats All Dynamic Table Info All Static Table Info Auto Increments Near Limit Count All Columns Count All Indexes MySQL 8.0 MySQL 5.7 Time in Seconds (Lower is better)
  27. 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION SCHEMA in MySQL 8.0: Dynamic values • TABLES.TABLE_ROWS, TABLES.DATA_FREE, … – Handled by opening table and retrieving data from SE in MySQL 5.7 – expensive!!! • Two approaches in MySQL 8.0 – information_schema_stats=cached • Default • Stored in auxiliary tables • ANALYZE TABLE – information_schema_stats=latest • Native functions in the I_S views • Uses special SE API (InnoDB) or falls back to opening tables (MyISAM) 27
  28. 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | INFORMATION SCHEMA longstanding issues solved in 8.0 – Bug#34921 comparison with information schema tables don’t honor collation – Bug#48445 Inconsistency with SHOW and SELECT FROM I_S – Bug#61846 SHOW FULL COLUMNS displays incorrect privileges for table – Bug#65121 Inconsistent result for select on INFORMATION_SCHEMA.STATISTICS – Bug#75532 Join between I_S schema tables is case insensitive/returns wrong value – Bug#81347 unnecessary scanned all databases for information schema – ..... 28
  29. 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Reliability – Atomic DDL • All metadata stored in InnoDB (transactional SE) • DDL code changed to avoid intermediate commits – At server level – At InnoDB level – At replication level 29 Transactional behavior
  30. 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Reliability – Atomic DDL : Use Case - DROP DATABASE 30 MySQL 5.7 • Delete tables – Metadata, TRN/TRG/FRM files – Data, InnoDB tables • Delete stored programs – Metadata, rows in MyISAM (non- transactional) • Delete schema – Metadata, DB.OPT file Mix of filesystem, non- transactional/transactional storage and multiple commits MySQL 8.0 • Delete tables – Metadata, rows in InnoDB – Data, InnoDB tables • Delete stored programs – Metadata, rows in InnoDB • Delete schema – Metadata, rows in InnoDB Updates to transactional storage, one commit
  31. 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Reliability – Disaster Recovery • MySQL 8.0, metadata and data stored in InnoDB (transactional SE) – Metadata copy stored in SDI embedded in .ibd – Data stored in .ibd • MySQL before 8.0, metadata in files, data in InnoDB – Metadata stored in .FRM – Data stored in .ibd 31 Use case – IMPORT using Serialized Dictionary Information, InnoDB
  32. 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL before 8.0 • Moving .FRM and data files around (MyISAM) • Create tables and do import using SQL (InnoDB) CREATE TABLE t1 (...) ALTER TABLE t1 DISCARD TABLESPACE ALTER TABLE t1 IMPORT TABLESPACE ... • Easy to do mistakes MySQL 8.0 • Self-descriptive tablespaces (SDI) • New IMPORT statement will import the tablespace: – Read metadata from the SDI, and create the tables – Import the data into these tables • Worst case, if SDI is corrupt, you can extract it (JSON format) and edit 32 Reliability – Disaster Recovery • Use case – IMPORT using Serialized Dictionary Information
  33. 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Reliability and Ease of Use – Automated Dictionary Upgrade • The Data Dictionary is versioned • The MySQL Server will support upgrading dictionary tables automatically – Done directly by the executable and not in a script – Reduces user and privilege issues for the process doing the upgrade • The Data Dictionary will be upgraded atomically 33
  34. 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Extending Dictionary Tables from Plugins • API for plugins to extend INFORMATION_SCHEMA & PERFORMANCE_SCHEMA – Add new virtual tables/views • API for plugins to store their specific data in the Data Dictionary 34
  35. 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Try for yourself! • Downloadable 8.0.1 DMR – dev.mysql.com • A «hot» labs release right off the press: 8.0.1 + Unified Data Dictionary • Enjoy and give us your feedback! • Thank you for listening • http://mysqlserverteam.com 35

×