MySQL Performance Tuning: The Perfect Scalability (OOW2019)

Copyright © 2019 Oracle and/or its affiliates.
MySQL Performance Tuning:
The Perfect Scalability
(HOL3025)
MySQL Principal Technical Support Engineer
Oracle MySQL Support
September 17, 2019
Mirko Ortensi

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

Our Goal
When dataset and concurrency increase,
horizontal and vertical scaling can address
service growth. But in order to achieve
perfect scalability, I/O, CPU and Network
bound workloads can meet additional
improvements at no expense to get the
most out of MySQL Server.

Why This Matters
Scaling up and out a database implies more
performing storage, more processing
capacity, additional memory and higher
bandwidth. Keeping adjustements to
infrastructure as little as possible helps
keeping operational expenditure limited.

“Query is slow
Anonymous

Mirko Ortensi

Agenda
• Horizontal vs Vertical scaling
• I/O Bound Workload
• CPU Bound Workload
• Network Bound Workload
• OS-wise
• Conclusions

Data and Concurrency grow
Service is growing: more data
and more active connections
will grow the database «old»
THREAD
THREAD
THREAD
THREAD
CLIENT
CLIENT
CLIENT
CLIENT
THREAD
THREAD

I/O, CPU and NETWORK bound
More locks + scans Buffer pool should be
increased
More connections
threads
I/O throughput

Scaling out READS
With additional slaves
MySQL Router
WRITE
READ
REPL

Scaling out READS
With additional slaves
• Create slave server (clone(), MEB, mysqldump)
• Distribute read and write operations accordingly
• Slaves can be delayed

Scaling out READS
With InnoDB Cluster
MySQL Router
WRITE
READ
REPL

Scaling out READS
With InnoDB Cluster
• Highly available with automatic failover
• Integrated solution to scale and balance reads
• Up to 9 replicas
• Writes do not scale (single or multi master)
• Read your own writes

Scaling out WRITES
Sharding
• MySQL Server does not support sharding
• Only auto-sharding solution is NDB Cluster
• Custom / third party
ID SERVICE DATA
1773467253 chat xxx
6257346892 location xxx
1773467253 chat xxx
7234782739 photos xxx
8437829249 location xxx

Scaling Up?
• More sockets / cores
• Faster disks / SSD
• More memory
• More bandwidth

MySQL Multiplies!
Take advantage of:
• Multiple storages
• Thread pooling
• Parallel slave
And tune as usual:
• Configuration
• Queries
CLIENT
CLIENT
CLIENT
CLIENT
THREAD
THREAD
THREAD
CLIENT
CLIENT
WORKER
WORKER
MASTER

1 2 3
If Data and Concurrency grows, "MySQL multiplies"
IO bound
• Single storage
• Temporary tables
• Set to prevent
data loss
• Insufficient
Buffer pool
• Write workload
CPU bound
• Active
connections
• Redundant
indexes
• sort/ joins
• Scans
• Background tasks
Network bound
• Uncompressed
• Huge recordsets

• Linux (native/VM)
• MySQL 8
• sysstat
• world.sql https://bit.ly/2kO70nY
HOL3025

I/O Bound
Monitoring
• iostat
• SHOW ENGINE INNODB STATUS
• Performance schema (MEM)
• SYS schema
select * from sys.io_by_thread_by_latency;
select * from sys.io_global_by_file_by_bytes;
select * from sys.io_global_by_file_by_latency;
select * from sys.waits_global_by_latency;
select * from sys.statements_with_temp_tables;

Reconfigure Move ib_logfile files Permissions START MySQL
I/O Bound
REDO Log
Stop the instance, edit
Custom
innodb_log_group_ho
me_dir and save
changes.
From data directory to
the new directory, The
files are usually
ib_logfile0 and
ib_logfile1, but they
could be more,
depending on the
value of
innodb_log_files_in_g
roup
Make sure that the
MySQL server has the
proper access rights
to read/create files in
the log directory.
It will read the
configuration file with
the new location.

On storage devices with cache, data loss is possible if data files
and redo log files reside on different storage devices, and a crash
occurs before data file writes are flushed from the device cache.
If you use or intend to use different storage devices for redo logs and
data files, use O_DIRECT instead.
I/O Bound
REDO Log

I/O Bound
Binary Logs
Custom log-bin
1. Stop: systemctl stop mysqld
2. Create: /opt/mysql/binlog
3. Edit my.cnf: log_bin=/opt/mysql/binlog/binlog
4. Move binlog.* to /opt/mysql/binlog
5. Edit: binlog.index to adopt absolute paths
(e.g. /opt/mysql/binlog/binlog.000011)
6. Start: systemctl start mysqld

I/O Bound
Binary Logs
TEST
1. FLUSH BINARY LOGS;
2. SELECT @@log_bin_basename;
3. RESET MASTER;

I/O Bound
UNDO Logs
Custom innodb_undo_directory
1. Stop: systemctl stop mysqld
2. Create: /opt/mysql/undo
3. Relocate undo_00*
4. Start: systemctl start mysqld
5. Check:
1. select @@innodb_undo_directory

I/O Bound
EXTERNAL tablespace
• Creating a file-per-table tablespace outside of the data directory.
• The DATA DIRECTORY clause cannot be used with ALTER TABLE
to change the location later.
• CREATE TABLE..DATA DIRECTORY
TEST
mysql> USE test;
Database changed
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA
DIRECTORY = '/opt/mysql/tablespaces';

I/O Bound
GENERAL tablespace
• Acts as a container of tablespaces, same as system tablespace
• Can be located outside of the data directory
• From file-per-table or system to general tablespace (ALTER)
• From system or general (shared) to file-per-table (ALTER)
• Directory must be known: set innodb_directories (read-only)
TEST
CREATE TABLESPACE `ts1` ADD DATAFILE
'/opt/mysql/tablespaces/ts1.ibd' Engine=InnoDB;
CREATE TABLE t1 (c1 INT PRIMARY KEY) TABLESPACE ts1;

I/O Bound
Partitioning
• Horizontal partitioning—that is, different rows of a table may be assigned to
different physical partitions.
• Partitioning makes it possible to store more data in one table than can be held
on a single disk or file system partition.
TEST
CREATE TABLE th (id INT, name VARCHAR(30), adate DATE)
PARTITION BY LIST(YEAR(adate))
(
PARTITION p1999 VALUES IN (1995, 1999, 2003)
DATA DIRECTORY = '/opt/mysql/tablespaces/',
PARTITION p2000 VALUES IN (1996, 2000, 2004)
DATA DIRECTORY = '/opt/mysql/tablespaces/'
);

I/O Bound
Session and Global Temporary Tablespace
• If no value is specified for innodb_temp_data_file_path, the default behavior is
to create an auto-extending data file named ibtmp1 in the
innodb_data_home_dir (system tablespace location)
• Not a dynamic setting
• At database start a pool of session 10 temporary tables is created
• At database start, ibtmp1 is created to
• Temp tables rollback segments in ibtmp1
• Example:
--innodb-temp-data-file-
path=../../../tmp/ibtmp1:12M:autoextend

I/O Bound
Session and Global Temporary Tablespace
TEST
• mysql> select @@innodb_temp_data_file_path;
• mysql> SELECT * FROM INFORMATION_SCHEMA.FILES
WHERE TABLESPACE_NAME='innodb_temporary'G
• shell> ls –l <location>
• Stop Server and list directory again

I/O Bound
Logging
General log
general_log_file
Slow Query Log
slow_query_log_file
Audit Log
audit_log_file

I/O Bound
Flush method
Data can travel through different layer before reaching the
durable storage.

I/O Bound
Flush method
Flush method: innodb_flush_method (not dynamic)
• fsync is default: fsync is used for both data and log file
• O_DIRECT can help to avoid double buffering between the InnoDB buffer
pool and the operating system file system cache. But is buffer pool is not
enough, buffer cache won’t come to help. fsync is used for both data and log
file
• O_DIRECT_NO_FSYNC same as before, but skips fsyncs
• O_DSYNC: InnoDB uses O_SYNC to open and flush the log files, and fsync()
to flush the data files, only. Based on O_SYNC: nfsync(), write() calls go to
buffer cache, blocks until writes are into the physical storage.

I/O Bound
Flush method monitoring
TEST 1
1. sync; echo 1 > /proc/sys/vm/drop_caches
2. vmstat 1 30
3. mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --
concurrency=20 --iterations=100 —number-of-queries=10
TEST 2
At different innodb_flush_method:
• mysql> show status like "Innodb_data_fsyncs";
• mysql> show status like "Innodb_data_pending_fsyncs";
• bash> free -m

I/O Bound
Miscellaneous
An all-comprehensive discussion in our official online documentation.
• Disable logging of compressed pages
• Act on I/O capacity for background tasks: innodb_io_capacity and
innodb_flush_sync
• Threshold on write buffer before fsync(): innodb_fsync_threshold
• Data size
• ...

CPU Bound
Monitoring
• vmstat / mpstat
• Status
SHOW STATUS LIKE "Handler%";
SHOW STATUS LIKE "select_full_join";
• sys schema
select * from sys.schema_redundant_indexes;
select * from sys.schema_unused_indexes;
select * from sys.statements_with_sorting;
CALL sys.diagnostics(60, 60, 'current');

CPU Bound
Thread Pool
High CPU may also be a good sign, no long IO wait or heavy locking. But
can also be for context switches by CPU scheduler.
1. Too many thread stacks make CPU caches almost useless in highly
parallel execution workloads
2. With too many threads executing in parallel, context switching
overhead is high
Thread pool promotes thread stack reuse
The thread pool controls when transactions start to ensure that not too
many execute in parallel.

CPU Bound
Query Tuning
TEST
1. Import world.sql
2. Monitor: mpstat 1 30
3. mysqlslap -uroot -p --no-drop --create-schema=world --
query="select * from world.city where Name='Kabul';" --
concurrency=10 --iterations=1000
4. EXPLAIN SELECT * FROM world.city WHERE Name='Kabul';
5. SELECT * FROM sys.schema_tables_with_full_table_scans;
6. CREATE INDEX city_idx ON world.city(Name);
7. Repeat the test

CPU Bound
Replication
• In 5.6 we introduced parallel slave, that could run load for different
databases in parallel. With operations split on several DBs, this
could be a simple option.
• In 5.7 we introduced LOGICAL_CLOCK algorithm, that allows
running many transactions in parallel if they were run in parallel on
master.
• In 8.0 version we have introduced write sets, that allows to run
some statements in parallel, even if they were executed by the
same thread on master.

CPU Bound
Miscellaneous
• Tablespace compression
• Protocol compression
• Thread cache
• SSL
• Background
Checkpoint
Purge
Automatic Statistics Calculation

NETWORK Bound
Compression
--compress: Compress all information sent between the client and the server if
possible
TEST
• mysql –uroot –compress –p
• mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --
concurrency=20 --iterations=100 —number-of-queries=10 –compress
• sar -n DEV 3 30
• mpstat 1 30
Look at averages

OS-wise
SWAP, SWAPPINESS and AFFINITY
SWAP & SWAPPINESS
cat /proc/sys/vm/swappiness
AFFINITY
At the linux OS level, you can limit the number of processors that the
kernel will utilize for a user space process using numactl, or more
commonly taskset.

OS-wise
SWAP, SWAPPINESS and AFFINITY
TEST
• Identify systemd configuration file:
/etc/systemd/system/multi-user.target.wants/mysqld.service
• Modify:
[Service]
CPUAffinity=1
• bash> systemctl daemon-reload

CPU monitor Press f Last CPU used Esc
OS-wise
AFFINITY
top -H -p <pid> To get the list of
options
Scroll down and select
"P = Last Used Cpu
(SMP)", press space
bar
Column on the right
side will show what
CPU has just been
used

Conclusion
Setup the most
reliable
production-like
traffic generator
Test and monitor
at different
concurrencies
Experiment!

Thank You
Oracle MySQL Support
September 17, 2019
Mirko Ortensi

Session Survey
Help us make the content
even better. Please complete
the session survey in the
Mobile App.

What’s Ahead
Tuesday
5pm-5.45pm
Thursday
NDB, Simply the World’s Highest-
Throughput Transactional Datastore
[DEV2554]
Moscone South - Room 205
10am-10:45am What’s New in MySQL Optimizer and
Executor? [DEV2077]
Moscone South - Room 205

The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

MySQL Performance Tuning: The Perfect Scalability (OOW2019)

More Related Content

What's hot

Similar to MySQL Performance Tuning: The Perfect Scalability (OOW2019)

Recently uploaded

MySQL Performance Tuning: The Perfect Scalability (OOW2019)