MySQL 优化

MySQL Optimization
(A couple of different ways.)
Morgan Tocker
morgan@percona.com
1
Monday, March 7, 2011

★ InnoDB?
★ MyISAM?
★ XtraDB?
★ MySQL 4.x?
★ MySQL 5.0.x?
★ MySQL 5.1?
★ MySQL 5.5?
★ Something else?
2
Show of Hands

About this presentation
★ Q: “Would you like to talk about MySQL optimization?”
★ A: Sure.
★ Optimization can be made at many levels.
★ I’m going to show 2-3 parts of the puzzle:
✦
Query Optimization.
✦
How InnoDB works.
✦
(Time Permitting) Our work at Percona on improving MySQL.
3

It all starts with EXPLAIN
★ Bookmark this manual page:
http://dev.mysql.com/doc/refman/5.1/en/using-
explain.html
★ It is the best source for anyone getting started.
4

Examples come from:
★ IMDB database loaded into
InnoDB tables (~5G).
★ Download it and import it for
yourself using imdbpy2sql.py:
http://imdbpy.sourceforge.net/
5

First Example
6

Find the Title Bambi
7
ALL means
tablescan
In this case a sort is
required because of the
order by
Anticipated number
of rows to be
examined
Additional filtering may
be possible before
passing to sort.
3.09s

Aha! Now add an index:
8

We must revisit...
9
Using = for comparison, but not
primary key lookup.
Size of the index
used (in bytes)
Anticipated number of
rows to be examined
dropped considerably.
Identified title as a
candidate index,
chose to use it.
0.00s

Other ways of accessing
10
Better type of ‘const’. We can
stop when we find one row.
0.00s
At most one
matching row.
In InnoDB the primary key is
often much faster than all
other keys.

And..
11
Ignore the time with
EXPLAIN. Only look at
the time for a query.
Type is range. BETWEEN, IN()
and < > are also ranges.
Number of rows to be examined
has increased - we are not
specific enough.
0.00s

Why’s that a range?
★ We're looking for titles between BambA and BambZ*
★ When we say index in MySQL, we mean trees.
✦
That is, B-Tree/B+Tree/T-Tree.
✦
Pretend they're all the same (for simpliﬁcation).
✦
There's no radically different indexing methods in MySQL
unless you play storage engine Bingo**.
12
* In reality the range is a little wider
** The memory storage engine supports hash indexes

What’s that?
13

Could this be a range?
14
3.2s

No, we can’t traverse.
15
Do we head left or right here?

LIKE ‘Z%’
16
0.05s

LIKE ‘T%’
17
3.13s

LIKE ‘The %’
18
3.07s

MySQL is (reasonably) smart.
★ It dynamically samples the data to choose which is the
better choice - or in some cases uses static statistics*.
★ This helps the optimizer choose:
✦
Which indexes will be useful.
✦
Which indexes should be avoided.
✦
Which is the better index when there is more than one.
19 * To refresh statistics run ANALYZE TABLE table_name;

Why avoid indexes?
★ B-Trees work like humans search a phone book;
✦
Use an index if you want just a few rows.
✦
Scan cover-to-cover if you want a large percentage.
20

★ We benchmarked this on a different schema:
Why avoid indexes (cont.)
21
Table scan has a relatively
fixed cost (red line).
The index has completely
different effectiveness
depending on how much it
can filter.
Hopefully MySQL switches
at the right point.

What you should take away:
★ Data is absolutely critical.
✦
Development environments should contain sample data
exported from production systems.
★ Input values are absolutely critical.
✦
Between two seemingly identical queries, execution plans
may be very different.
22
See also: http://www.mysqlperformanceblog.com
/2009/10/16/how-not-to-find-unused-indexes/

Improve this Query
23
3.41s
This number of rows is a
guess. It keeps changing
between examples.

We’re Spoiled for Choice.
24

Index on production_year
25
3.53s

Might work if...
26
0.92s

Index on title(50)
27
0.02s

Comparing the two:
28
★ mysql> EXPLAIN SELECT * from title WHERE title = 'Pilot' AND
production_year BETWEEN 2006 and 2009G

Composite Indexes.
★ What is better?
✦
INDEX py_t (production_year, title)
✦
INDEX t_py (title, production_year)
29

Index on py_t
30
0.02s
http://www.mysqlperformanceblog.com/2010/01/09/getting-around-
optimizer-limitations-with-an-in-list/

Index on t_py
31
0.00s

Recommendations
★ Index over multiple columns if it can improve ﬁltering. i.e.
✦
GOOD: Only some pilots made between 2006-2009.
✦
BAD: All pilots made between 2006-2009.
32

Recommendations (cont.)
★ Don't know what order to specify the columns?
✦
RULE: Think how to ﬁlter the fastest. Use that order left to
right.
✦
EXCEPTION: If there's a range (><, BETWEEN, %). Those
always go to the RIGHT.
33
http://www.mysqlperformanceblog.com/2010/01/09/getting-around-
optimizer-limitations-with-an-in-list/

Query Tuning

How InnoDB Works

“Numbers everyone should know”
36
L1 cache reference 0.5 ns
Branch mispredict 5 ns
L2 cache reference 7 ns
Mutex lock/unlock 25 ns
Main memory reference 100 ns
Compress 1K bytes with Zippy 3,000 ns
Send 2K bytes over 1 Gbps network 20,000 ns
Read 1 MB sequentially from memory 250,000 ns
Round trip within same datacenter 500,000 ns
Disk seek 10,000,000 ns
Read 1 MB sequentially from disk 20,000,000 ns
Send packet CA->Netherlands->CA 150,000,000 ns
See: http://www.linux-mag.com/cache/7589/1.html and Google http://
www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

About Disks.
★ 10,000,000 ns = 10ms = 100 operations/second.
✦
This is about the average for a 7200RPM drive.
★ The actual time has dramatic variation.
✦
The variation is because disks are mechanical.
✦
We can much write faster sequentially than randomly.
37

[default] Everything is buffered!
★ When you write to a ﬁle, here’s what happens in the
Operating System:
38
Block 9, 10, 1, 4, 200, 5.
Block 1, 4, 5, 9, 10, 200
What happens to this
buffer if we loose power?

The OS provides a way!
★ $ man fsync
39
Synopsis
#include <unistd.h>
int fsync(int fd);
int fdatasync(int fd);
Description
fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache
pages for) the file referred to by the file descriptor fd to the disk device (or other
permanent storage device) where that file resides. The call blocks until the device
reports that the transfer has completed. It also flushes metadata information
associated with the file (see stat(2)).
Hint: MyISAM just writes to the
OS buffer and has no durability.
http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/

Knowing this:
★ InnoDB wants to try and reduce random IO.
★ It can not (safely) rely on the operating system’s write
buffering and be ACID compliant.
✦
.. and InnoDB algorithms have to compensate.
40

Basic Operation (High Level)
Log Files
41
SELECT * FROM City
WHERE CountryCode=ʼAUSʼ
Buffer Pool
Tablespace

Basic Operation (cont.)
42
Log Files
UPDATE City SET
name = 'Morgansville'
WHERE name = 'Brisbane'
AND CountryCode='AUS'
Buffer Pool
Tablespace

Basic Operation (cont.)
42
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace

Why don’t we update?
★ This is an optimization!
✦
The log ﬁle IO is sequential and much cheaper than live
updates.
✦
The IO for the eventual updates to the tablespace can be
optimized as well.
★ Provided that you saved enough to recover, this
shouldn’t matter should it?
43

More on Logs...
★ Logs are only used during recovery.
✦
Not even read when we need to write down
dirty pages!
★ To ﬁgure out which pages need to be evicted we have two
lists - the ﬂush list and the LRU.
★ Log activities are all assigned a LSN (log sequence
number).
44
Log Files

Log Files, Checkpoints, etc.
★ Most database systems work this way:
✦
In Oracle the transaction logs are called “Redo Logs”.
★ The background process of syncing dirty pages is
normally referred to as a “Checkpoint”.
★ InnoDB has fuzzy checkpointing.
45

Log Writing
★ You can change increase innodb_log_file_size.
This will allow InnoDB to “smooth out” background IO
for longer.
✦
Tip: Optionally you could change
innodb_log_files_in_group as well. Be aware that
your effective log ﬁle is innodb_log_file_size *
innodb_log_files_in_group
46

Log Writing (cont.)
★ You can also change
innodb_flush_log_at_trx_commit to 0 or 2 to
reduce the durability requirements of this write.
✦
Requires less ﬂushing - particularly helpful on systems
without writeback caches!
★ innodb_log_buffer_size may also help buffer
changes for longer before writing to the logs.
✦
Very workload dependent - tends to be more helpful for
writing big TEXT/BLOB changes.
47

In summary
★ On commit, the log has to be ﬂushed to guarantee
changes.
✦
Nothing else has to be done.
★ What visibility do we have into these operations?
★ How do we decide how much background work to do per
second?
★ What happens if we fall behind in background work?
48

Enhancements via Percona

Terminology
50
Oracle Product License Percona Equivalent
Product
License
MySQL Server GPL Percona Server GPL
The InnoDB Storage
Engine (Plugin edition)
GPL The XtraDB Storage
Engine
GPL
InnoDB Hot Backup Commercial XtraBackup GPL
(GPL = Completely free for you to use. Support not included.)

Performance Improvements
51
Improved Buffer
Pool Scalability
Improved IO Path
+ adaptive checkpointing
Improved Rollback
Segment Scalability*Separate purge
thread
Data dictionary
memory
consumption
controls
Increased number of
undo slots*
Faster page
checksums*
Support for different
page sizes*
* Changes on disk format (not backwards compatible)
Insert buffer controls
Completely
disable the query
cache.
Remove excess fcntl calls
Per session configuration of
innodb_flush_log_at_trx_commit
Separate location
of double write
buffer*
Strip comments before
using query cache
Transaction logs
larger than 4G
supported.

Improved Buffer Pool Scalability
★ Additional patches to what is already available in the
InnoDB Plugin.
✦
Splits the buffer pool mutex into smaller mutexes:
• Flush list mutex
• LRU mutex
• Free mutex
• hash mutex
52

Data Dictionary control
★ Once an InnoDB table is opened it is never freed from
the in-memory data dictionary (which is unlimited in
size).
★ XtraDB introduces:
✦
--innodb_dict_size_limit - a conﬁguration item in
bytes.
✦
innodb_dict_tables - a status variable for number
entries in the cache.
53

Undo Slots
★ In the built-in InnoDB, the number of undo slots is
limited to 1024.
✦
This means the number of open transactions is limited to
1023. See: http://bugs.mysql.com/bug.php?id=26590
✦
Some statements require 2 undo slots.
★ In XtraDB, this is expanded to 4072 with --
innodb_extra_undoslots=1.
54 Warning: This is binary format incompatible!

Rollback Segments
★ In XtraDB, it’s also possible to have more than one
rollback segment.
✦
Each segment contains undo slots.
★ Conﬁguration is --innodb-extra-rsegments=N
★ This has the added effect of reducing mutex contention
on the rollback segment:
✦
“Mutex at 0×1b3e3e78 created ﬁle trx/trx0rseg.c line 167″
55
http://www.percona.com/docs/wiki/percona-
xtradb:patch:innodb_extra_rseg
http://www.mysqlperformanceblog.com/2009/10/14/tuning-
for-heavy-writing-workloads/
Warning: This is binary format incompatible!

Fast Checksums
★ The InnoDB page checksum computation is slower than
it needs to be.
★ XtraDB has the option to use a faster checksum format.

Different Page Sizes
★ XtraDB now has support for different page sizes - 4K,
8K, 16K.

Separate Purge Thread
★ Cleans up a long history list length faster:
58
See: http://www.mysqlperformanceblog.com/2009/10/14/
tuning-for-heavy-writing-workloads/

Usability Enhancements
59
Show contents of
the buffer pool
Import / export of
innodb_file_per_table
tables
Import / export of
buffer pool
contents
Transactional
Replication
Better handling of
corrupted tables
Store buffer pool
in shared memory
segment
Save index statistics
between restarts
Advise in
processlist when
waiting on Query
cache mutex.
Improved slow
query log
User / Index / Table statistics
Disable automatic
statistics
regeneration
Show data
dictionary
Deadlock counter
Show Temporary Tables
Log connection
errors
Retain query
response time
distribution.

Show Buffer Pool Contents
60
mysql> SELECT d.*,round(100*cnt*16384/(data_length+index_length),2) fit FROM
(SELECT schema_name,table_name,count(*) cnt,sum(dirty),sum(hashed) FROM
INNODB_BUFFER_POOL_PAGES_INDEX GROUP BY schema_name,table_name ORDER BY cnt DESC
LIMIT 20) d JOIN TABLES ON (TABLES.table_schema=d.schema_name AND
TABLES.table_name=d.table_name);
+-------------+---------------------+---------+------------+-------------+--------+
| schema_name | table_name | cnt | sum(dirty) | sum(hashed) | fit |
+-------------+---------------------+---------+------------+-------------+--------+
| db | table1 | 1699133 | 13296 | 385841 | 87.49 |
| db | table2 | 1173272 | 17399 | 11099 | 98.42 |
| db | table3 | 916641 | 7849 | 15316 | 94.77 |
| db | table4 | 86999 | 1555 | 75554 | 87.42 |
| db | table5 | 32701 | 7997 | 30082 | 91.61 |
| db | table6 | 31990 | 4495 | 25681 | 102.97 |
| db | table7 | 1 | 0 | 0 | 100.00 |
+-------------+---------------------+---------+------------+-------------+--------+
7 rows in set (26.45 sec)
Source: http://www.mysqlperformanceblog.com/
2010/03/26/tables-fit-buffer-poo/
* Command may differ slightly between versions.

Save Buffer Pool Contents
★ Export the contents of the buffer pool to a ﬁle called
‘ib_lru_dump’ in the data directory:
✦
SELECT * FROM
information_schema.XTRADB_ADMIN_COMMAND /*!
XTRA_LRU_DUMP*/;
★ Restored ib_lru_dump:
✦
SELECT * FROM
information_schema.XTRADB_ADMIN_COMMAND /*!
XTRA_LRU_RESTORE*/;
61
Note: Not the actual contents - it takes 8 bytes to
remember the address of a 16K page.

Import/Export tables
★ Because --innodb-file-per-table still has
information (data dictionary, undo) in the global
tablespace you can’t just back it up by itself.
★ With a new setting, --innodb_expand_import=1,
this is no longer the case.
★ Tip: The import/export still has to be done with
XtraBackup. Documentation available here:
http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_expand_import
62

The Slow Query Log
63
$ mysql -e “SET GLOBAL log_slow_verbosity = ‘full’;”
$ tail /var/log/mysql.slow
..
# Time: 100924 13:58:47
# User@Host: root[root] @ localhost []
# Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0
# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 Rows_affected: 0 Rows_read: 1
# Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923
# InnoDB_trx_id: 1403
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5
# InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 65329
SET timestamp=1285336727;
select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON
(cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;
$ tail /var/log/mysql.slow
..
# Time: 100924 13:58:47
# User@Host: root[root] @ localhost []
# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608
SET timestamp=1285336727;
select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON
(cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;
MySQL Server
Percona Server

(Related) Xtrabackup Features
★ Report on fragmentation of indexes:
★ $ xtrabackup --stats --tables=art.link* --
datadir=/mnt/data/mysql/
...
table: art/link_out104, index: PRIMARY, space
id: 12, root page 3
leaf pages: recs=25958413, pages=497839,
data=7492026403 bytes, data/pages=91%
...
65
http://www.mysqlperformanceblog.com/2009/09/14/statistics-
of-innodb-tables-and-indexes-available-in-xtrabackup/

Basic Operation (again.)
66
Log Files
UPDATE City SET
Buffer Pool
Tablespace

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
Set innodb_buffer_pool_size to
“50-80%” of memory.

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
Increase the size of the log
files (innodb_log_file_size
and
innodb_log_files_in_group)

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
Set
innodb_flush_log_at_trx_commit=2
if durability is not as important.
and

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
Typically use
innodb_flush_method=O_DIRECT
if using a Hardware RAID controller.
Set
and

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
(Possibly) Move the log
files to separate spindles
(sequential IO)
Typically use
Set
and

66
Log Files
UPDATE City SET
01010
Buffer Pool
Tablespace
(Possibly) Move the log
files to separate spindles
(sequential IO)
Typically use
Set
If innodb_log_waits > 0 the
buffer being filled
(innodb_log_buffer_size)
before writing to the log files
may be too small.
and

The End.
★ Questions?
67

MySQL 优化

More Related Content

Similar to MySQL 优化

More from YUCHENG HU

Recently uploaded

MySQL 优化