New awesome features in MySQL 5.7

1. New Awesome Features
in MySQL 5.7
— by royalwzy

3. Feedbacks

4. Personal Profile
More than 5 years experience in database use and management
Proficient in synchronizing data between heterogeneous database
Skilled in designing and optimizing database
Install and configure MySQL cluster and high-availability cluster environment
Synchronize and merge data from Oracle to MySQL
mysqlexp
mysqlclone
Oracle 10g/11g OCM
MySQL 5.6 Database Administrator
MCITP/MCTS DBA(MSSQL2008)
RHCE
Java Programmer
Oracle ACEA, Oracle Young Expert
ACOUG/SHOUG/OCMU core member
ITPUB Moderator and Community Expert
Senior lecturer in "ENMO" and “UPLOOKING"
Active in OTN & ITPUB and other forums
My blog:royalwzy.com

5. MySQL5.7 Released Versions
❖ MySQL 5.7.9 was GA on 2015-10-21
❖ MySQL 5.7.10 was GA on 2015-12-07
❖ MySQL 5.7.11 was GA on 2016-02-05

6. New Features in MySQL 5.7
❖ Performance Improvements
❖ Performance Schema
❖ SYS Schema
❖ Optimizer Improvements
❖ Optimizer Cost Model
❖ JSON Explain
❖ New and Improved Optimizer Hints
❖ Query Rewrite Plugin
❖ Replication Improvements
❖ Multi-Source Replication
❖ Transaction Based Parallel Replication
❖ Online Replication Changes
❖ Semi-Sync Replication Enhancements
❖ Enhanced Monitoring
❖ Group Replication
❖ InnoDB Improvements
❖ InnoDB Online Operations
❖ InnoDB General Tablespaces
❖ Transparent Page Level Compression
❖ InnoDB Native Partitioning
❖ InnoDB Native Full Text Search
❖ InnoDB Cache Retention
❖ Security Improvements
❖ AES 256 Encryption
❖ Password Rotation Policies
❖ MySQL Secure Installation
❖ Comfortable User Account Management
❖ Native JSON Support
❖ Generated Columns
❖ High Availability Improvements
❖ GIS Improvements
❖ GeoJSON
❖ GeoHash
❖ Spatial Aware Client Tools

7. Performance Improvements
❖ 3x Faster than MySQL 5.6: SQL Point Selects

8. Performance Schema Enhancements
❖ Metadata Locking
❖ Stage Tracking
❖ Transactions
❖ Memory Usage
❖ Stored Programs
❖ Prepared Statements

9. SYS Schema
❖ MySQL sys schema is included by default
in 5.7.7
❖ What's its role?
❖ Who is taking up all the resources on my database server?
❖ Which hosts are hitting my database server those most?
❖ Which objects are accessed the most, and how?
❖ Which statements are using temporary tables on disk?
❖ Which tables take the most space in my InnoDB buffer pool?
❖ Where is all the memory going on my instance?
❖ Where is the time being spent the most within my instance?

10. Optimizer Improvements
❖ New MySQL Parser & New MySQL Optimizer

11. Optimizer Improvements
❖ The Optimizer Cost Model
❖ JSON Explain
❖ Optimizer Hints
❖ Query Rewrite Plugin

12. Optimizer - The Optimizer Cost Model
❖ The cost based optimizer has been
improved to make dynamic, intelligent,
and ultimately better cost based
decisions
❖ These estimates are stored in the
server_cost and engine_cost tables
within the mysql system database.
❖ Dynamically reload or execute FLUSH
OPTIMIZER_COSTS statement

13. Optimizer - The Cost Model Project
❖ Relate cost to more things
❖ Make the cost model aware of the
underlying hardware
❖ Make use of the cache information
❖ Improve the precision of the statistics
❖ Prioritize histograms
❖ Add networking cost to the model

14. Optimizer - JSON Explain
❖ Use EXPLAIN FORMAT=JSON to extend
query execution plan
❖ See Visual Explain in Workbench

15. Optimizer - JSON Explain

16. Optimizer - Optimizer Hints
❖ The disadvantage of optimizer_switch:
❖ It needs an extra step to change it's value
❖ It affects the entire statement
❖ New parser rules to support hint
syntax
❖ Server side statement timeouts

17. Optimizer - Hints Rules
❖ Comment syntax /*+ */ is used for the new hints
❖ Multiple hints can be specified within the same
comment
❖ A query block can have only one comment
containing hints
❖ Incorrect and/or invalid syntax within a hint
will cause a warning
❖ The first conflicting hint will take effect
❖ Multilevel hints are supported

18. Optimizer - Query Rewrite Plugin
❖ Why Query Rewrites?
❖ How Query Rewrites?
❖ Install Query Rewrite Plugin
❖ A glance with a demo

19. Replication Improvements
❖ Multi-Source Replication(Flexibility)
❖ Transaction Based Parallel Replication(Slave
Throughput)
❖ Online Replication Changes(Convenience)
❖ Semi-Sync Replication Enhancements(Consistency)
❖ Enhanced Monitoring(Monitoring)
❖ Group Replication(HA)

20. Replication - Replication Components
SESSION DUMP I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
❖ Session thread
❖ Dump thread
Master Log
Binary
Log
❖ I/O thread
❖ SQL thread
❖ Binary log
❖ Master log
❖ Relay log

21. Replication - Why Use Replication
❖ Read Scale-Out

22. Replication - Why Use Replication
❖ Write Scale-Out
❖ MySQL Fabric helps sharding you data
with MySQL

23. Replication - Why Use Replication
❖ Switching Masters During Failover
❖ Use STOP SLAVE & CHANGE MASTER TO MASTER_HOST='Slave1'

24. Replication - Why Use Replication
❖ Replicating Different Databases to
Different Slaves
MySQL
Slave1
Master
Database
MySQL
Slave2
MySQL
Slave3

25. Replication - Multi-Source Replication
❖ Complex queries for analytics purposes
❖ Data HUB for inter-cluster replication
❖ Merge table shards
❖ Integrated backup

26. Replication - Multi-Source Replication
❖ A server(slave) can replicate from multiple
sources(masters)
❖ Multiple channels(channel:connection thread,
relay log, applier threads) that can be
stopped/started individually
❖ Virtually no limit on the number of
sources(capped to 256, but can be changed if
server binary is rebuilt)
❖ Able to manage each source separately
CHANGE MASTER TO [...] FOR CHANNEL = 'name'

27. Replication - Multi-Source Replication
❖ Integrated with GTIDs
❖ Integrated with Multi-threaded Slave
❖ each channel has its own multi-threaded applier set of threads
❖ Integrated with the new P_S tables
❖ replication_execute_status_by_coordinator shows multiple entries, one per
channel/source applier
❖ replication_connection_status shows multiple entries, one per connection to
different sources
❖ Integrated with crash-safe tables
❖ Progress state is stored in these tables for recoverability purposes

28. Transactions
logged in
sequence
Transactions
applied in
parallel
Replication - Transaction Based Parallel
Replication
❖ Before MySQL5.6:Single-Threaded Slave
SESSION DUMP I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
Master Log
Binary
Log
Transactions
applied in
sequence
Bottleneck

29. Replication - Transaction Based Parallel
Replication
❖ MySQL5.6:Multi-Threaded Slave by database
❖ Different databases go to different threads
❖ Great for some applications, BUT:
❖ No improvement if there is only one database
❖ May break cross-database consistency
Transactions
logged in
sequence
Transactions
applied in
parallel
SESSION DUMP I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
Master Log
Binary
Log
Transactions
applied in
parallel
DB3
DB2
DB1

30. Replication - Transaction Based Parallel
Replication
❖ MySQL 5.7.2: Multi-threaded slave by master
concurrency
❖ Transactions that were prepared at the same time on master
cannot conflict
❖ Master stores a logical timestamp in the binary log
❖ Slave applies transactions with same timestamp in parallel
❖ Works always
❖ Even for one-database applications
❖ Consistent
❖ To enable:slave> SET GLOBAL slave_parallel_type =
logical_clock(DEFAULT)(database for MySQL5.6);

31. Replication - Transaction Based Parallel
Replication
❖ Multi-threaded slave by master concurrency

32. Replication - Online Replication Changes
❖ This procedure is suited to use in production
❖ Pre-conditions
❖ All servers in your topology must use MySQL 5.7.6 or later
❖ All servers have gtid_mode set to the default value OFF

33. Replication - Semi-Sync Replication
Enhancements
❖ By default, replication is asynchronous
❖ In parallel: Master acks to app and sends transaction to
slave
❖ Fast
❖ Changes lost if master dies
❖ MySQL 5.5: semi-synchronous replication
possible
❖ In sequence: slave receives transaction, then master
acks to app
❖ Slower: Master waits for slave
❖ Less risk for lost updates

34. Replication - Semi-Sync Replication
Enhancements
SESSION DUMP I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
Master Log
Binary
Log
1
2
4
6
3
5
SESSION DUMP I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
Master Log
Binary
Log
1
2
3
4
4

35. Replication - Semi-Sync Replication
Enhancements
❖ MySQL 5.5: semi-
synchronous
replication
❖ Master commit
❖ Slave receive
❖ Client ack
❖ If master crashes
between 1 and 2,
committed data is lost
❖ Concurrent clients may
have seen the
transaction
❖ MySQL 5.7.2:
❖ loss-less semi-sync replication
❖ Slave receive
❖ Master commit
❖ Client ack
❖ If master crashes,
all committed data
is on slave

36. Replication - Semi-Sync Replication
Enhancements
❖ rpl_semi_sync_master_wait_point
❖ AFTER_SYNC(Default)
❖ AFTER_COMMIT
SESSION
DUMP
I/O SQL
NETWORK
Master
Database
Salve
Database
Binary
Log
Relay
Log
Master Log
Binary
Log
1
2
5
6
3
4ACK

37. Replication - Enhanced Monitoring
❖ Traditional replication monitoring:
SHOW SLAVE STATUS
❖ Simple
❖ Not SQL friendly – no WHERE, no joins, etc
❖ Multi-source has per-source status
❖ Multi-threaded slave has per-applier status
❖ Performance_Schema tables for
replication slave in 5.7.2

38. Replication - Enhanced Monitoring
I/O SQL
NETWORK
Salve
Database
Relay
Log
Master Log
Binary
Log
execute
statusexecute
configuration
connection
configuration
connection
status
execute status by
coordinator execute status
by worker
❖ 6 tables in performance_schema database:
❖ replication_connection_configuration
❖ replication_connection_status
❖ replication_execute_configuration
❖ replication_execute_status
❖ replication_execute_status_by_coordinator
❖ replication_execute_status_by_worker
❖ Consistent semantics across tables
❖ Consistent naming across tables

39. Replication - Enhanced Monitoring
❖ Example:Connection status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME: CHANNEL1
SOURCE_UUID: 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID: 13
SERVICE_STATE: ON
RECEIVED_TRANSACTION_SET: 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME: CHANNEL2
.
.
.

40. Replication - Group Replication
❖ Introduction

41. Replication - Group Replication
❖ Monitoring and Inspecting the System Through
Performance_Schema
❖ Self-healing, Elasticity, Redundancy
❖ Update Everywhere
❖ It's available on labs

42. Replication - Group Replication
❖ vs Percona XtraDB Cluster

43. High Availability Improvements
❖ A new data migration tool
❖ mysqlpump
❖ Backup Examples
❖ Restore Example

44. Generated Columns
❖ The syntax
❖ As a materialized cache for often used
expressions
❖ Function based indexes
❖ Limitations:
❖ Only use native functions
❖ Can’t refer to itself or to other that are later
defined

45. Generated Columns

46. InnoDB Improvements
❖ InnoDB Online Operations
❖ InnoDB General Tablespaces
❖ InnoDB Transparent Page Level
❖ InnoDB Native Partitioning
❖ InnoDB Native Full Text Search
❖ InnoDB Cache Retention

47. InnoDB - Online Operations
❖ Resizing the InnoDB Buffer Pool
Online
❖ Truncating Undo Logs Online
❖ Renaming Index Online
❖ Enlarging VARCHAR column size Online

48. InnoDB - Resizing the InnoDB Buffer Pool
Online
❖ Activing transactions MUST be completed and waited
mysql> SET GLOBAL innodb_buffer_pool_size=<chunk_size*N>;
❖ Configuring InnoDB buffer pool
❖ Modifing InnoDB buffer pool chunk size
❖ Configuring InnoDB buffer pool chunk size
mysql> SHOW VARIABLES LIKE ‘innodb_buffer_pool_size’;
❖ Command line:
❖ Configuration file:
./mysqld --innodb_buffer_pool_chunk_size=134217728
innodb_buffer_pool_chunk_size=134217728

49. InnoDB - Resizing the InnoDB Buffer Pool
Online
❖ Monitoring online buffer pool resize progress
[Note] InnoDB: Resizing buffer pool from 134217728 to 4294967296. (unit=134217728)
[Note] InnoDB: disabled adaptive hash index.
[Note] InnoDB: buffer pool 0 : 31 chunks (253952 blocks) was added.
[Note] InnoDB: buffer pool 0 : hash tables were resized.
[Note] InnoDB: Resized hash tables at lock_sys, adaptive hash index, dictionary.
[Note] InnoDB: completed to resize buffer pool from 134217728 to 4294967296.
[Note] InnoDB: re-enabled adaptive hash index.
mysql> SHOW STATUS WHERE Variable_name='InnoDB_buffer_pool_resize_status';
+----------------------------------+----------------------------------+
| Variable_name | Value |
+----------------------------------+----------------------------------+
| Innodb_buffer_pool_resize_status | Resizing also other hash tables. |
+----------------------------------+----------------------------------+

50. ❖ Buffer Pool Resizing Internals(Increasing)
❖ 1.Adds pages in chunks
❖ 2.Coverts hash tables, lists, and pointers to use new addresses in memory
❖ 3.Adds new pages to the free list
❖ 4.While these operations are in progress, other threads are blocked from
accessing the buffer pool.
❖ Buffer Pool Resizing Internals(Decreasing)
❖ 1.Defragments the buffer pool and withdraws (frees) pages
❖ 2.Removes pages in chunks
❖ 3.Converts hash tables, lists, and pointers to use new addresses in
memory
❖ 4.Of these operations, only defragmenting the buffer pool and withdrawing
pages allow other threads to access to the buffer pool concurrently.
InnoDB - Resizing the InnoDB Buffer Pool
Online

51. InnoDB - Truncating Undo Logs Online
❖ Truncating Undo Logs That Reside in Undo Tablespaces
❖ innodb_undo_tablespaces=2
❖ innodb_undo_logs=35(or greater)
mysql> SET GLOBAL innodb_undo_log_truncate=ON;
❖ Enabling Truncation of Undo Tablespaces
❖ Expediting Truncation of Undo Tablespace Files
❖ Performance Impact of Truncating Undo Tablespace
Files Online
mysql> SET GLOBAL innodb_purge_rseg_truncate_frequency=32;

52. InnoDB - Renaming Index Online
❖ Three processes
❖ Renaming in InnoDB data dictionary (SYS_INDEXES)
❖ Renaming in InnoDB data dictionary cache (the
dict_table_t/dict_index_t objects)
❖ Renaming in InnoDB persistent stats storage
❖ Online DDL
mysql> ALTER TABLE <tbl_name> RENAME INDEX xxx to yyy;

53. InnoDB - Enlarging VARCHAR column size
Online
❖ Enlarging VARCHAR column size Online

54. ❖ InnoDB Tablespace Types:
❖ InnoDB System Tablespace
❖ InnoDB File-Per-Table Tablespace(Single)
❖ InnoDB Undo Tablespace
❖ InnoDB General Tablespace
InnoDB - General Tablespaces
“A tablespace consists of a chain of files. The size of the files does not have to be
divisible by the database block size, because we may just leave the last incomplete
block unused. When a new file is appended to the tablespace, the maximum size of the
file is also specified. At the moment, we think that it is best to extend the file to
its maximum size already at the creation of the file, because then we can avoid
dynamically extending the file when more space is needed for the tablespace.”

55. InnoDB - General Tablespaces
❖ Shared tablespaces that can store data for multiple tables
❖ General tablespaces support both Antelope and Barracuda file formats
❖ Data files may be placed in a directory relative to or
independent of the MySQL data directory
❖ Have a potential memory advantage over file-per-table
tablespaces
mysql> CREATE TABLESPACE `ts1` ADD DATAFILE 'ts1.ibd' Engine=InnoDB;
mysql> CREATE TABLESPACE `ts1` ADD DATAFILE '/path/ts1.ibd' Engine=InnoDB;
mysql> CREATE TABLESPACE `ts1` ADD DATAFILE '../path/ts1.ibd' Engine=InnoDB;

56. InnoDB - General Tablespaces
❖ Adding tables to a general tablespace
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) TABLESPACE ts1 ROW_FORMAT=COMPACT;
❖ Moving tables between tablespaces using ALTER TABLE
ALTER TABLE tbl_name TABLESPACE [=] tablespace_name;
mysql> ALTER TABLE t2 TABLESPACE ts1;
ALTER TABLE tbl_name ... TABLESPACE [=] innodb_system;
ALTER TABLE tbl_name ... TABLESPACE [=] innodb_file_per_table;

57. InnoDB - General Tablespaces
❖ General Tablespace Limitations
❖ A generated or existing tablespace cannot be
changed to a general tablespace
❖ Creation of temporary general tablespaces is
not supported
❖ General tablespaces do not support temporary
tables
❖ ALTER TABLE ... DISCARD TABLESPACE and ALTER
TABLE ...IMPORT TABLESPACE are not supported
for tables that belong to a general tablespace

58. InnoDB - Transparent Page Level
Compression
Database
Logical Physical
Tablespace Data file
Segment
Extent
Pages
Storage System
• SAN
• NAS
• File System
• NFS
• RAW
OS block
❖ Logical and Physical Database Structures in InnoDB

59. InnoDB - Transparent Page Level
Compression
❖ InnoDB table & page level compression
❖ Zlib & LZ4 compression algorithms
Enable:CREATE TABLE t1 (c1 INT) COMPRESSION="zlib";
Disable:ALTER TABLE t1 COMPRESSION="None";
❖ 32K & 64K page sizes
❖ The extent size is 2M for 32k page sizes
❖ The extent size is 4M for 64k page sizes
❖ Only support file_per_table tablespace

60. InnoDB - Native Partitioning
❖ Background
❖ Tables
❖ Storage engines
❖ Handlers
❖ Saving up to 90% memory
❖ Parallel query processing
❖ Foreign key support
❖ Partition pruning improvement
❖ Global secondary indexes
❖ Full-text searches

61. InnoDB - Native Full Text Search
❖ Supporting external parsers
❖ N-gram parser plugin for CJK
❖ MeCab full-text parser plugin for
Japanese

62. InnoDB - Cache Retention
❖ Displaying Buffer Pool Status
❖ Saving the Buffer Pool State Now

63. InnoDB - Cache Retention
❖ Displaying Buffer Pool Dump/Load Progress
❖ Restoring the Buffer Pool State Now

64. InnoDB - Cache Retention

65. InnoDB - Cache Retention
❖ Monitoring Buffer Pool Load Progress Using P_S Tables
mysql> SELECT EVENT_NAME, WORK_COMPLETED, WORK_ESTIMATED FROM events_stages_current;
+-------------------------------+----------------+----------------+
| EVENT_NAME | WORK_COMPLETED | WORK_ESTIMATED |
+-------------------------------+----------------+----------------+
| stage/innodb/buffer pool load | 5353 | 7167 |
+-------------------------------+----------------+----------------+
mysql> SELECT EVENT_NAME, WORK_COMPLETED, WORK_ESTIMATED FROM events_stages_history;
+-------------------------------+----------------+----------------+
| EVENT_NAME | WORK_COMPLETED | WORK_ESTIMATED |
+-------------------------------+----------------+----------------+
| stage/innodb/buffer pool load | 7167 | 7167 |
+-------------------------------+----------------+----------------+

66. Native JSON Support
❖ Native JSON data type was introduced in
MySQL 5.7.7
❖ It provides some major benefits compared
with TEXT or VARCHAR column
❖ Document Validation
❖ Efficient Access
❖ Performance
❖ Convenience
❖ You can seamlessly blend RDBMS and NoSQL in
a single tool

67. Native JSON Support - Document
Validation
❖ Insert valid data
❖ Insert invalid data

68. Native JSON Support - Efficient Access
❖ Add a column within JSON, and update the value
❖ Retrieve data from JSON column

69. Security Improvements
❖ AES 256 Encryption
❖ Password Rotation Policies
❖ MySQL Secure Installation
❖ Comfortable User Account Management

70. Security - AES 256 Encryption
❖ Supporting multiple AES Encryption
modes
❖ Deprecating ENCODE()/DECODE() and
replacing by AES based functions
❖ AES_ENCRYPT()
❖ AES_DECRYPT()
❖ Supporting for larger key sizes

71. Security - Password Rotation Policies
❖ MySQL5.6
❖ validate_password plugin
❖ password expiration mechanism
❖ MySQL5.7.4
❖ password lifetime policy
❖ default_password_lifetime=360
❖ New Columns
❖ authentication_string
❖ password_last_changed
❖ password_lifetime
❖ password_expired

72. Security - MySQL Secure Installation
❖ MySQL deployments installed using mysqld --
initialize are secure by default.
❖ Random account passwords on install
❖ Deploy without tests and demos
❖ No anonymous accounts
❖ Limit the scope of the FILE privilege
❖ Stricter permissions on installation files
❖ Remove Server Package’s Perl Dependencies
❖ mysql_install_db converted to a C program
❖ mysql_secure_install converted to a C program
❖ mysql_upgrade now doesn’t need to call external binaries

73. Security - Comfortable User Account
Management
❖ Fully functional ALTER USER
❖ IF [NOT] EXISTS clause to user account
statements
❖ Temporarily disable user accounts
❖ Authentication plugin that disables logins
❖ Time-based password expiration policy
❖ “Offline” server mode
❖ User account names now 32 characters long

74. GIS Improvements
❖ New & Improved Spatial Extensions
❖ InnoDB Spatial Indexes
❖ GeoJSON
❖ ST_GeomFromGeoJson()
❖ ST_AsGeoJson()
❖ GeoHash
❖ Spatial Aware Client Tools

75. GIS Improvements

76. Thank you for using MySQL!

77. Q & A