MySQL optimisation Percona LeMug.fr

MySQL Optimization (A couple of different ways.) Morgan Tocker morgan@percona.com 1Tuesday, March 8, 2011

Show of Hands ★ InnoDB? ★ MyISAM? ★ XtraDB? ★ MySQL 4.x? ★ MySQL 5.0.x? ★ MySQL 5.1? ★ MySQL 5.5? ★ Something else? 2Tuesday, March 8, 2011

About this presentation ★ Q: “Would you like to talk about MySQL optimization?” ★ A: Sure. ★ Optimization can be made at many levels. ★ I’m going to show 2-3 parts of the puzzle: ✦ Query Optimization. ✦ How InnoDB works. ✦ (Time Permitting) Our work at Percona on improving MySQL. 3Tuesday, March 8, 2011

It all starts with EXPLAIN ★ Bookmark this manual page: http://dev.mysql.com/doc/refman/5.1/en/using- explain.html ★ It is the best source for anyone getting started. 4Tuesday, March 8, 2011

Examples come from: ★ IMDB database loaded into InnoDB tables (~5G). ★ Download it and import it for yourself using imdbpy2sql.py: http://imdbpy.sourceforge.net/ 5Tuesday, March 8, 2011

First Example 6Tuesday, March 8, 2011

Find the Title Bambi 3.09s ALL means tablescan Anticipated number In this case a sort is of rows to be required because of the examined order by Additional filtering may be possible before passing to sort. 7Tuesday, March 8, 2011

Aha! Now add an index: 8Tuesday, March 8, 2011

We must revisit... 0.00s Using = for comparison, but not primary key lookup. Identified title as a candidate index, chose to use it. Size of the index used (in bytes) Anticipated number of rows to be examined dropped considerably. 9Tuesday, March 8, 2011

Other ways of accessing 0.00s At most one Better type of ‘const’. We can matching row. stop when we find one row. In InnoDB the primary key is often much faster than all other keys. 10Tuesday, March 8, 2011

And.. 0.00s Type is range. BETWEEN, IN() and < > are also ranges. Number of rows to be examined has increased - we are not specific enough. Ignore the time with EXPLAIN. Only look at the time for a query. 11Tuesday, March 8, 2011

Why’s that a range? ★ Were looking for titles between BambA and BambZ* ★ When we say index in MySQL, we mean trees. ✦ That is, B-Tree/B+Tree/T-Tree. ✦ Pretend theyre all the same (for simpliﬁcation). ✦ Theres no radically different indexing methods in MySQL unless you play storage engine Bingo**. * In reality the range is a little wider 12 ** The memory storage engine supports hash indexesTuesday, March 8, 2011

What’s that? 13Tuesday, March 8, 2011

Could this be a range? 3.2s 14Tuesday, March 8, 2011

No, we can’t traverse. Do we head left or right here? 15Tuesday, March 8, 2011

LIKE ‘Z%’ 0.05s 16Tuesday, March 8, 2011

LIKE ‘T%’ 3.13s 17Tuesday, March 8, 2011

LIKE ‘The %’ 3.07s 18Tuesday, March 8, 2011

MySQL is (reasonably) smart. ★ It dynamically samples the data to choose which is the better choice - or in some cases uses static statistics*. ★ This helps the optimizer choose: ✦ Which indexes will be useful. ✦ Which indexes should be avoided. ✦ Which is the better index when there is more than one. 19 * To refresh statistics run ANALYZE TABLE table_name;Tuesday, March 8, 2011

Why avoid indexes? ★ B-Trees work like humans search a phone book; ✦ Use an index if you want just a few rows. ✦ Scan cover-to-cover if you want a large percentage. 20Tuesday, March 8, 2011

Why avoid indexes (cont.) ★ We benchmarked this on a different schema: Table scan has a relatively fixed cost (red line). The index has completely different effectiveness depending on how much it can filter. Hopefully MySQL switches at the right point. 21Tuesday, March 8, 2011

What you should take away: ★ Data is absolutely critical. ✦ Development environments should contain sample data exported from production systems. ★ Input values are absolutely critical. ✦ Between two seemingly identical queries, execution plans may be very different. See also: http://www.mysqlperformanceblog.com 22 /2009/10/16/how-not-to-find-unused-indexes/Tuesday, March 8, 2011

Improve this Query 3.41s This number of rows is a guess. It keeps changing between examples. 23Tuesday, March 8, 2011

We’re Spoiled for Choice. 24Tuesday, March 8, 2011

Index on production_year 3.53s 25Tuesday, March 8, 2011

Might work if... 0.92s 26Tuesday, March 8, 2011

Index on title(50) 0.02s 27Tuesday, March 8, 2011

Comparing the two: ★ mysql> EXPLAIN SELECT * from title WHERE title = Pilot AND production_year BETWEEN 2006 and 2009G 28Tuesday, March 8, 2011

Composite Indexes. ★ What is better? ✦ INDEX py_t (production_year, title) ✦ INDEX t_py (title, production_year) 29Tuesday, March 8, 2011

Index on py_t 0.02s http://www.mysqlperformanceblog.com/2010/01/09/getting-around- 30 optimizer-limitations-with-an-in-list/Tuesday, March 8, 2011

Index on t_py 0.00s 31Tuesday, March 8, 2011

Recommendations ★ Index over multiple columns if it can improve ﬁltering. i.e. ✦ GOOD: Only some pilots made between 2006-2009. ✦ BAD: All pilots made between 2006-2009. 32Tuesday, March 8, 2011

Recommendations (cont.) ★ Dont know what order to specify the columns? ✦ RULE: Think how to ﬁlter the fastest. Use that order left to right. ✦ EXCEPTION: If theres a range (><, BETWEEN, %). Those always go to the RIGHT. http://www.mysqlperformanceblog.com/2010/01/09/getting-around- 33 optimizer-limitations-with-an-in-list/Tuesday, March 8, 2011

How InnoDB WorksTuesday, March 8, 2011

“Numbers everyone should know” L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns Send 2K bytes over 1 Gbps network 20,000 ns Read 1 MB sequentially from memory 250,000 ns Round trip within same datacenter 500,000 ns Disk seek 10,000,000 ns Read 1 MB sequentially from disk 20,000,000 ns Send packet CA->Netherlands->CA 150,000,000 ns See: http://www.linux-mag.com/cache/7589/1.html and Google http:// 35 www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdfTuesday, March 8, 2011

About Disks. ★ 10,000,000 ns = 10ms = 100 operations/second. ✦ This is about the average for a 7200RPM drive. ★ The actual time has dramatic variation. ✦ The variation is because disks are mechanical. ✦ We can much write faster sequentially than randomly. 36Tuesday, March 8, 2011

[default] Everything is buffered! ★ When you write to a ﬁle, here’s what happens in the Operating System: Block 9, 10, 1, 4, 200, 5. Block 1, 4, 5, 9, 10, 200 What happens to this buffer if we loose power? 37Tuesday, March 8, 2011

The OS provides a way! ★ $ man fsync Hint: MyISAM just writes to the OS buffer and has no durability. Synopsis #include <unistd.h> int fsync(int fd); int fdatasync(int fd); Description fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)). 38 http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/Tuesday, March 8, 2011

Knowing this: ★ InnoDB wants to try and reduce random IO. ★ It can not (safely) rely on the operating system’s write buffering and be ACID compliant. ✦ .. and InnoDB algorithms have to compensate. 39Tuesday, March 8, 2011

Basic Operation (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 40Tuesday, March 8, 2011

Basic Operation (cont.) Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Tablespace Buffer Pool 41Tuesday, March 8, 2011

Basic Operation (cont.) 01010 Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Tablespace Buffer Pool 41Tuesday, March 8, 2011

Why don’t we update? ★ This is an optimization! ✦ The log ﬁle IO is sequential and much cheaper than live updates. ✦ The IO for the eventual updates to the tablespace can be optimized as well. ★ Provided that you saved enough to recover, this shouldn’t matter should it? 42Tuesday, March 8, 2011

More on Logs... ★ Logs are only used during recovery. ✦ Not even read when we need to write down dirty pages! Log Files ★ To ﬁgure out which pages need to be evicted we have two lists - the ﬂush list and the LRU. ★ Log activities are all assigned a LSN (log sequence number). 43Tuesday, March 8, 2011

Log Files, Checkpoints, etc. ★ Most database systems work this way: ✦ In Oracle the transaction logs are called “Redo Logs”. ★ The background process of syncing dirty pages is normally referred to as a “Checkpoint”. ★ InnoDB has fuzzy checkpointing. 44Tuesday, March 8, 2011

Log Writing ★ You can change increase innodb_log_file_size. This will allow InnoDB to “smooth out” background IO for longer. ✦ Tip: Optionally you could change innodb_log_files_in_group as well. Be aware that your effective log ﬁle is innodb_log_file_size * innodb_log_files_in_group 45Tuesday, March 8, 2011

Log Writing (cont.) ★ You can also change innodb_flush_log_at_trx_commit to 0 or 2 to reduce the durability requirements of this write. ✦ Requires less ﬂushing - particularly helpful on systems without writeback caches! ★ innodb_log_buffer_size may also help buffer changes for longer before writing to the logs. ✦ Very workload dependent - tends to be more helpful for writing big TEXT/BLOB changes. 46Tuesday, March 8, 2011

In summary ★ On commit, the log has to be ﬂushed to guarantee changes. ✦ Nothing else has to be done. ★ What visibility do we have into these operations? ★ How do we decide how much background work to do per second? ★ What happens if we fall behind in background work? 47Tuesday, March 8, 2011

Enhancements via PerconaTuesday, March 8, 2011

Terminology Oracle Product License Percona Equivalent License Product MySQL Server GPL Percona Server GPL The InnoDB Storage GPL The XtraDB Storage GPL Engine (Plugin edition) Engine InnoDB Hot Backup Commercial XtraBackup GPL (GPL = Completely free for you to use. Support not included.) 49Tuesday, March 8, 2011

Performance Improvements Transaction logs Improved IO Path larger than 4G + adaptive checkpointing supported. Improved Buffer Separate location Pool Scalability of double write buffer* Remove excess fcntl calls Insert buffer controls Improved Rollback Separate purge Strip comments before Segment Scalability* thread using query cache Completely Data dictionary disable the query memory Increased number of cache. consumption undo slots* controls Per session configuration of Faster page Support for different innodb_flush_log_at_trx_commit checksums* page sizes* 50 * Changes on disk format (not backwards compatible)Tuesday, March 8, 2011

Improved Buffer Pool Scalability ★ Additional patches to what is already available in the InnoDB Plugin. ✦ Splits the buffer pool mutex into smaller mutexes: • Flush list mutex • LRU mutex • Free mutex • hash mutex 51Tuesday, March 8, 2011

Data Dictionary control ★ Once an InnoDB table is opened it is never freed from the in-memory data dictionary (which is unlimited in size). ★ XtraDB introduces: ✦ --innodb_dict_size_limit - a conﬁguration item in bytes. ✦ innodb_dict_tables - a status variable for number entries in the cache. 52Tuesday, March 8, 2011

Undo Slots ★ In the built-in InnoDB, the number of undo slots is limited to 1024. ✦ This means the number of open transactions is limited to 1023. See: http://bugs.mysql.com/bug.php?id=26590 ✦ Some statements require 2 undo slots. ★ In XtraDB, this is expanded to 4072 with -- innodb_extra_undoslots=1. 53 Warning: This is binary format incompatible!Tuesday, March 8, 2011

Rollback Segments ★ In XtraDB, it’s also possible to have more than one rollback segment. ✦ Each segment contains undo slots. ★ Conﬁguration is --innodb-extra-rsegments=N ★ This has the added effect of reducing mutex contention on the rollback segment: ✦ “Mutex at 0×1b3e3e78 created ﬁle trx/trx0rseg.c line 167″ http://www.percona.com/docs/wiki/percona- xtradb:patch:innodb_extra_rseg http://www.mysqlperformanceblog.com/2009/10/14/tuning- for-heavy-writing-workloads/ 54 Warning: This is binary format incompatible!Tuesday, March 8, 2011

Fast Checksums ★ The InnoDB page checksum computation is slower than it needs to be. ★ XtraDB has the option to use a faster checksum format. 55 Warning: This is binary format incompatible!Tuesday, March 8, 2011

Different Page Sizes ★ XtraDB now has support for different page sizes - 4K, 8K, 16K. 56 Warning: This is binary format incompatible!Tuesday, March 8, 2011

Separate Purge Thread ★ Cleans up a long history list length faster: See: http://www.mysqlperformanceblog.com/2009/10/14/ 57 tuning-for-heavy-writing-workloads/Tuesday, March 8, 2011

Usability Enhancements Show data User / Index / Table statistics dictionary Show contents of Log connection the buffer pool errors Disable automatic Import / export of statistics Save index statistics innodb_file_per_table regeneration between restarts tables Transactional Import / export of Deadlock counter Replication buffer pool Advise in contents Retain query processlist when response time waiting on Query Improved slow distribution. cache mutex. query log Store buffer pool in shared memory segment Better handling of Show Temporary Tables corrupted tables 58Tuesday, March 8, 2011

Show Buffer Pool Contentsmysql> SELECT d.*,round(100*cnt*16384/(data_length+index_length),2) fit FROM(SELECT schema_name,table_name,count(*) cnt,sum(dirty),sum(hashed) FROMINNODB_BUFFER_POOL_PAGES_INDEX GROUP BY schema_name,table_name ORDER BY cnt DESCLIMIT 20) d JOIN TABLES ON (TABLES.table_schema=d.schema_name ANDTABLES.table_name=d.table_name);+-------------+---------------------+---------+------------+-------------+--------+| schema_name | table_name | cnt | sum(dirty) | sum(hashed) | fit |+-------------+---------------------+---------+------------+-------------+--------+| db | table1 | 1699133 | 13296 | 385841 | 87.49 || db | table2 | 1173272 | 17399 | 11099 | 98.42 || db | table3 | 916641 | 7849 | 15316 | 94.77 || db | table4 | 86999 | 1555 | 75554 | 87.42 || db | table5 | 32701 | 7997 | 30082 | 91.61 || db | table6 | 31990 | 4495 | 25681 | 102.97 || db | table7 | 1 | 0 | 0 | 100.00 |+-------------+---------------------+---------+------------+-------------+--------+7 rows in set (26.45 sec) Source: http://www.mysqlperformanceblog.com/ 2010/03/26/tables-fit-buffer-poo/ 59 * Command may differ slightly between versions.Tuesday, March 8, 2011

Save Buffer Pool Contents ★ Export the contents of the buffer pool to a ﬁle called ‘ib_lru_dump’ in the data directory: ✦ SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_DUMP*/; ★ Restored ib_lru_dump: ✦ SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_RESTORE*/; Note: Not the actual contents - it takes 8 bytes to 60 remember the address of a 16K page.Tuesday, March 8, 2011

Import/Export tables ★ Because --innodb-file-per-table still has information (data dictionary, undo) in the global tablespace you can’t just back it up by itself. ★ With a new setting, --innodb_expand_import=1, this is no longer the case. ★ Tip: The import/export still has to be done with XtraBackup. Documentation available here: http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_expand_import 61Tuesday, March 8, 2011

The Slow Query Log $ tail /var/log/mysql.slow .. # Time: 100924 13:58:47 MySQL Server # User@Host: root[root] @ localhost [] # Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 SET timestamp=1285336727; select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1; $ mysql -e “SET GLOBAL log_slow_verbosity = ‘full’;” $ tail /var/log/mysql.slow Percona Server .. # Time: 100924 13:58:47 # User@Host: root[root] @ localhost [] # Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0 # Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 Rows_affected: 0 Rows_read: 1 # Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923 # InnoDB_trx_id: 1403 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes # Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5 # InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 65329 SET timestamp=1285336727; select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1; 62Tuesday, March 8, 2011

(Related) Xtrabackup Features ★ Report on fragmentation of indexes: ★ $ xtrabackup --stats --tables=art.link* -- datadir=/mnt/data/mysql/ ... table: art/link_out104, index: PRIMARY, space id: 12, root page 3 leaf pages: recs=25958413, pages=497839, data=7492026403 bytes, data/pages=91% ... http://www.mysqlperformanceblog.com/2009/09/14/statistics- of-innodb-tables-and-indexes-available-in-xtrabackup/ 64Tuesday, March 8, 2011

Basic Operation (again.) Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) 01010 Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) Set innodb_buffer_pool_size to “50-80%” of memory. 01010 Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) Set innodb_buffer_pool_size to “50-80%” of memory. 01010 Log Files UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) Set innodb_buffer_pool_size to “50-80%” of memory. 01010 Log Files Set innodb_flush_log_at_trx_commit=2 if durability is not as important. UPDATE City SET name = Morgansville WHERE name = Brisbane AND CountryCode=AUS Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) Set innodb_buffer_pool_size to “50-80%” of memory. 01010 Log Files Set innodb_flush_log_at_trx_commit=2 if durability is not as important. UPDATE City SET name = Morgansville WHERE name = Brisbane Typically use AND CountryCode=AUS innodb_flush_method=O_DIRECT if using a Hardware RAID controller. Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) (Possibly) Move the log Set innodb_buffer_pool_size to files to separate spindles “50-80%” of memory. (sequential IO) 01010 Log Files Set innodb_flush_log_at_trx_commit=2 if durability is not as important. UPDATE City SET name = Morgansville WHERE name = Brisbane Typically use AND CountryCode=AUS innodb_flush_method=O_DIRECT if using a Hardware RAID controller. Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Tablespace Buffer Pool 65Tuesday, March 8, 2011

Basic Operation (again.) (Possibly) Move the log Set innodb_buffer_pool_size to files to separate spindles “50-80%” of memory. (sequential IO) 01010 Log Files Set If innodb_log_waits > 0 the innodb_flush_log_at_trx_commit=2 buffer being filled if durability is not as important. (innodb_log_buffer_size) UPDATE City SET beforeMorgansville the log files name = writing to WHERE name = Brisbane Typically use may be too small. AND CountryCode=AUS innodb_flush_method=O_DIRECT if using a Hardware RAID controller. Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Tablespace Buffer Pool 65Tuesday, March 8, 2011

The End. ★ Questions? 66Tuesday, March 8, 2011

http://www.lemug.fr	204
http://static.slidesharecdn.com	1
http://www.techgig.com	1

MySQL optimisation Percona LeMug.fr

by cyruss666 on Mar 12, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 206

Statistics

MySQL optimisation Percona LeMug.fr — Presentation Transcript