DPC Tutorial

Optimizing MySQL Essentials

About Me• Ligaya Turmelle• Senior Technical Support Engineer• MySQL Now (AKA Oracle)• ~3 years

Agenda• Basics• Hardware and OS• Server • variables• Queries • slow query log and EXPLAIN • example query

Basics1+1 = 3

• Deﬁnition of Performance: Characterized by the amount of useful work accomplished by a computer system compared to the time and resources used.

This can cover a lot of different areas(potentially at the same time) in a LAMPstack• Application• Network• Hardware• Database

Learn how to benchmark• There are lots of benchmarking tools out there.• Use what is appropriate to measure what you are trying to improve

OS and Hardware

OS• Linux vs. Windows• Linux vs. Linux

Linux• swappiness = 0• IO scheduler • HDD = deadline • SSD or you have a good RAID controller = noop• You can use ext3 - but not the best

Hardware• RAM• RAM• RAM• Disks• CPU• And did I mention RAM

Basic Linux commands• top• iostat• vmstat• dstat

Server Settings So what should i tweak?

Knobs, Levers and Buttons• mysqld• MyISAM• InnoDB• Others

MySQLDwhat knobs to turn?

MySQLD Many think that yum install / apt-get install and they’re done.• No, we’re just beginning! • Logging • Memory Usage: • Global • Session • Character Sets

MySQLD ( logging )• log_bin = <filename>• general_log = 0 general_log_file = <filename>• slow_query_log = 1 slow_log_file = <filename> long_query_time = 1• log_queries_not_using_indexes min_examined_row_limit = 100

MySQLD ( memory usage )• Memory is allocated... • Globally - server wide, usually allocated only at server startup and exist only “once” • Session - each connection or event • Both - In these cases the global variable value ( join_buffer_size ) is used if a new value is not set during the session.

MySQLD ( Memory / global )• connection_timeout ( 10 )• expire_log_days ( 7 )• query_cache_type = 0 ( yes, 0! )• wait_timeout = 3• skip_name_resolve = 1• performance_schema = 1

MySQLD ( memory / global )• max_heap_table_size / tmp_table_size• table_open_cache• thead_cache_size }• read_buffer_size / read_rnd_buffer_size Also• sort_buffer_size conﬁgurable per session• join_buffer_size

MyISAMwhat levers to pull

MyISAM

MyISAM• Switch to using InnoDB

MyISAM• Switch to using InnoDB • Not an option?

MyISAM• Switch to using InnoDB • Not an option?• Switch to using InnoDB

MyISAM• Switch to using InnoDB • Not an option?• Switch to using InnoDB • Still not an option?

MyISAM• Switch to using InnoDB • Not an option?• Switch to using InnoDB • Still not an option?• The BLACKHOLE storage engine is really fast

MyISAM• Switch to using InnoDB • Not an option?• Switch to using InnoDB • Still not an option?• The BLACKHOLE storage engine is really fast• Still stuck?

MyISAM• switch to mongodb, it’s webscale! http://nosql.mypopescu.com/post/1016320617/mongodb-is-web-scale

MyISAM• concurrent_insert • 0 - Never, 1 - Hole free, 2 - Holes• key_buffer_size • Key_reads / Key_read_requests • Key_writes / Key_write_requests

InnoDBwhat buttons to press

InnoDB Many options only a few covered• innodb_buffer_pool_size - caches both indexes and data - upwards of 85% available RAM• innodb_flush_log_at_trx_commit - 0 - written and flushed to disk once per second - 1 - written and flushed to disk at each transaction commit - 2 - written to disk at each commit, flush one a second

InnoDB• innodb_log_buffer_size - Amount of data written to ib_logfile - The larger, the less disk I/O• innodb_log_file_size - The larger the ﬁle, the more InnoDB can do - < 5.5 the larger, the longer crash recovery time• innodb_double_write_buffer - data integrity checking, costs IOPSs• innodb_file_per_table - data and indexes written to their own .ibd ﬁle

Examine, Testand Measure “Dr. Watson I Presume”

MySQLD ( quick check )Quick Check - mysql-tuning-primerhttps://launchpad.net/mysql-tuning-primer

MySQLD ( in depth 1/11 ) Getting In depth - SHOW GLOBAL STATUS+--------------------------+------------+| Variable_name | Value |+--------------------------+------------+ | Com_replace | 37372 || Com_alter_table | 184 | | Com_replace_select | 0 || Com_begin | 103281460 | | Com_rollback | 148312594 || Com_change_db | 631 | | Com_select | 7164448573 || Com_commit | 669825867 | | Com_set_option | 1877614377 || Com_create_index | 0 | | Com_show_collations | 130652029 || Com_create_table | 354 | | Com_show_fields | 694 || Com_delete | 1235248406 | | Com_show_grants | 36 || Com_delete_multi | 353434291 | | Com_show_processlist | 33184 || Com_drop_table | 359 | | Com_show_slave_hosts | 1 || Com_flush | 3 | | Com_show_variables | 130885618 || Com_insert | 2725147239 | | Com_show_warnings | 99619 || Com_insert_select | 9 | | Com_truncate | 11 || Com_kill | 3642 | | Com_unlock_tables | 18 || Com_load | 15 | | Com_update | 1348570903 || Com_lock_tables | 18 | | Com_update_multi | 8 | +--------------------------+------------+

MySQLD ( in depth 2/11 ) Values with out timeframe are meaningless.10:21:08 rderoo@mysql09:mysql [1169]> SHOW GLOBAL STATUS LIKE Uptime;+---------------+----------+| Variable_name | Value |+---------------+----------+| Uptime | 12973903 |+---------------+----------+1 row in set (0.00 sec) That’s about 150 days. :)

MySQLD ( in depth 3/11 ) By using mysqladmin we can observe changes over short periods of time:$ mysqladmin -u rderoo -p -c 2 -i 10 -r extended-status > review.txt Output is the same as SHOW GLOBAL STATUS with two copies appearing in review.txt the ﬁrst has values since the last server restart / FLUSH COUNTERS the second output are the delta between the ﬁrst and 10 seconds later.

MySQLD ( in depth 4/11 ) Temporary Tables: How big?17:54:33 rderoo@mysql09:event [1185]> SHOW GLOBAL VARIABLES LIKE %_table_size;+---------------------+----------+| Variable_name | Value |+---------------------+----------+| max_heap_table_size | 67108864 || tmp_table_size | 67108864 |+---------------------+----------+2 rows in set (0.00 sec) How many?17:55:09 rderoo@mysql09:event [1186]> SHOW GLOBAL STATUS LIKE %_tmp%tables;+-------------------------+-----------+| Variable_name | Value |+-------------------------+-----------+| Created_tmp_disk_tables | 156 || Created_tmp_tables | 278190736 |+-------------------------+-----------+2 rows in set (0.00 sec)

MySQLD ( in depth 5/11 )The max temporary table size is 64mb.In 150 days 278 million temp tables wherecreated.With only 156 going to disk.This is well tuned!

MySQLD ( in depth 6/11 ) open_table_cache| Open_tables | 4094 || Opened_tables | 12639 || Uptime | 12980297 | • This yields a rate of ~85 table opens per day, a bit high...20:52:22 rderoo@mysql09:mysql [1190]> SHOW GLOBAL VARIABLES LIKE table_cache;+---------------+-------+| Variable_name | Value |+---------------+-------+| table_cache | 4096 |+---------------+-------+ • Increasing the table_cache would be advisable.

MySQLD ( in depth 7/11 ) thread_cache_size • To determine the effectiveness of the thread_cache as a hit ratio use the following formula: 100-((Threads_created/Connections)*100)| Connections | 131877233 || Threads_created | 20014624 |100 - ( ( 20014624 / 131877233 ) * 100 ) = 84.8% • Increasing the thread_cache_size would seem to be advisable.

MySQLD ( in depth 8/11 ) read_buffer_size • The effectiveness can be found by examining the Select_scan :| Select_scan | 304864263 | • This should be set to a small value globally and increased on a per session basis • Increase when performing full table scans

MySQLD ( in depth 9/11)read_rnd_buffer_size• Used when reading sorted results after a key sort has taken place• This should be set to a small value globally and increased on a per session basis• Usually should be set ( per session ) no high than 8M

MySQLD ( in depth 10/11) sort_buffer_size • The effectiveness can be found by examining the Sort_merge_passes :| Sort_merge_passes | 7 | • This should be set to a small value globally and increased on a per session basis • Increase GROUP BY, ORDER BY, SELECT DISTINCT and UNION DISTINCT performance

MySQLD ( in depth 11/11 ) join_buffer_size • The effectiveness can be found by examining the Select_full_join :| Select_full_join | 5369 | • A crutch used when proper indexing is not in place • Will help in cases of full table scans • This should be set to a small value globally and increased on a per session basis • Increase GROUP BY and ORDER BY performance

Tuning Queries Where the magic happens

Tuning your queries is the single biggestperformance improvement you can do toyour server• efﬁcient resource usage• results returned quickerPer QUERY!

Where do I start?

slow query log

Slow query log• A log of all the “slow” queries• Deﬁne “slow” • took longer then long_query_time to run AND at least min_examined_row_limit rows are looked at • log_queries_not_using_indexes • log_slow_admin_statements

Turning it on• Option is slow_query_log • no argument or value of 1 - enabled • argument of 0 - disabled (default)

Output Types• Use log_output • write to a ﬁle • write to a table in the mysql database

slow log at startup• Place the slow_query_log and optional log_output options in my.cnf• log_output acceptable values • TABLE, FILE, NONE• Can also use slow_query_log_file to specify a log ﬁle name

slow log at runtime• All the below global variables can be changed at run time • log_output • slow_query_log • slow_query_log_file SET GLOBAL slow_query_log = 1;

/usr/sbin/mysqld, Version: 5.0.67-0ubuntu6.1-log ((Ubuntu)). started with:Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sockTime Id Command Argument# Time: 110328 13:15:05# User@Host: homer[homer] @ localhost []# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 4452238use confs;select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> 6198 and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = 6198 and t.ID = ts.talk_id ) order by rand() limit 15;

slow query log EXPLAINed

There are multiple ways of gettinginformation out of the slow query log.• just open it up to see the raw data• usually more helpful to summarize/ aggregate the raw data

Tools for aggregating the slow log data• mk-query-digest • third party • part of the maatkit utilities• mysqldumpslow • comes with MySQL in the bin dir

Macintosh-7:joindin-for-lig ligaya$ perl -laF ~/mysql_installs/maatkit/mk-query-digest mysql-slow.log# 23s user time, 240ms system time, 17.53M rss, 86.51M vsz# Current date: Mon May 16 11:08:24 2011# Hostname: Macintosh-7.local# Files: mysql-slow.log# Overall: 73.72k total, 14 unique, 0.02 QPS, 0.04x concurrency __________# Time range: 2011-03-28 13:15:05 to 2011-05-13 02:33:52# Attribute total min max avg 95% stddev median# ============ ======= ======= ======= ======= ======= ======= =======# Exec time 160130s 0 180s 2s 3s 819ms 2s# Lock time 11s 0 1s 149us 0 12ms 0# Rows sent 387.64k 0 255 5.38 14.52 6.86 0# Rows examine 338.66G 0 5.35M 4.70M 4.93M 387.82k 4.70M# Query size 30.19M 17 8.35k 429.47 420.77 29.33 420.77# Profile# Rank Query ID Response time Calls R/Call Apdx V/M Item# ==== ================== ================= ===== ======= ==== ===== =====# 1 0x2AAA5EC420D7E2A2 159878.0000 99.8% 73682 2.1698 0.50 0.12 SELECTuser talks talk_speaker talks# 3 0x0A2584C03C8614A9 50.0000 0.0% 16 3.1250 0.44 0.87 SELECTwp_comments# MISC 0xMISC 202.0000 0.1% 20 10.1000 NS 0.0 <12ITEMS>

Count: 1 Time=180.00s (180s) Lock=0.00s (0s) Rows=1.0 (1), XXXXX[xxxxx]@localhost SELECT SLEEP(N)Count: 16 Time=3.12s (50s) Lock=0.00s (0s) Rows=0.9 (14), XXXXX[xxxxx]@localhost SELECT comment_date_gmt FROM wp_comments WHERE comment_author_IP = S ORcomment_author_email = S ORDER BY comment_date DESC LIMIT NCount: 5 Time=2.80s (14s) Lock=0.00s (0s) Rows=0.0 (0), XXXXX[xxxxx]@localhost SELECT comment_ID FROM wp_comments WHERE comment_post_ID = S AND ( comment_author= S OR comment_author_email = S ) AND comment_content = S LIMIT NCount: 73682 Time=2.17s (159867s) Lock=0.00s (11s) Rows=5.4 (396223),XXXXX[xxxxx]@localhost select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> N and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = N and t.ID = ts.talk_id ) order by rand() limit N

Tuning a query• Tables • SHOW CREATE TABLE `talks`; • SHOW TABLE STATUS LIKE ‘talks’;• Indexes • SHOW INDEXES FROM ‘talks’;• EXPLAIN

EXPLAIN

EXPLAIN Basics• Syntax: EXPLAIN [EXTENDED] SELECT select_options• Displays information from the optimizer about the query execution plan• Works only with SELECT statements

EXPLAIN Output• Each row provides information on one table• Output columns: id key_length select_type ref table rows type Filtered (new to 5.1) possible_keys Extra key

EXPLAIN Output

id• Select identiﬁer• Only if there are subqueries, derived tables or unions is this incremented• Number reﬂects the order that the SELECT/FROM was done in

select_type• Type of SELECT • SIMPLE • PRIMARY • DERIVED • UNION • SUBQUERY

table• The table (or alias) that is used• Read down - these are the order the tables are JOINed• DERIVED tables are noted and numbered

type• Access type• Various types: preference access type BEST system/const eq_ref ref ref_or_null index_merge range index WORST ALL

possible_keys• List of all possible indexes (or NULL) that optimizer considered using

key• Index used for the query or NULL• Look at key and possible_keys to consider your index strategy

key_len• shows number of bytes MySQL will use from the index• Possible that only part of the index will be used NOTE: a character set may use more then one byte per character

ref• Very different then the access type ‘ref’• Show which columns or a constant within the index will be used for the access

rows• Number of rows MySQL expects to ﬁnd based on statistics • statistics can be updated with ANALYZE TABLE

ﬁltered• New in 5.1.12• Estimated number of rows ﬁltered by the table condition• Shows up if you use EXPLAIN EXTENDED

Extra• All the other stuff • Using index - using a covering index • Using ﬁlesort - sorting in a temporary table • Using temporary - a temporary table was made. • Using where - ﬁltering outside the storage engine.

Query Tuning Example

Query we are working with:EXPLAIN select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> 11945 and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = 11945 and t.ID = ts.talk_id ) order by rand() limit 15G

Full EXPLAIN plan*************************** 1. row *************************** id: 1 select_type: PRIMARY table: t type: indexpossible_keys: PRIMARY key: idx_event key_len: 9 ref: NULL rows: 3119 Extra: Using where; Using index; Using temporary; Using filesort*************************** 2. row *************************** id: 1 select_type: PRIMARY table: ts type: refpossible_keys: talk_id key: talk_id key_len: 5 ref: joindin.t.ID rows: 1 Extra: Using where*************************** 3. row *************************** id: 1 select_type: PRIMARY table: u type: eq_refpossible_keys: PRIMARY key: PRIMARY key_len: 4 ref: joindin.ts.speaker_id rows: 1 Extra:

Full EXPLAIN plan (con’t)*************************** 4. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: t type: refpossible_keys: PRIMARY,idx_event key: idx_event key_len: 5 ref: func rows: 24 Extra: Using where; Using index; Using temporary*************************** 5. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: ts type: refpossible_keys: talk_id key: talk_id key_len: 5 ref: joindin.t.ID rows: 1 Extra: Using where5 rows in set (0.00 sec)

What was wrongwith that query plan?

HintEXPLAIN select distinct u.ID as user_id, *************************** 4. row *********************** t.event_id, id: 2 u.username, select_type: DEPENDENT SUBQUERY u.full_name table: t from user u, type: ref talks t, possible_keys: PRIMARY,idx_event talk_speaker ts key: idx_event where u.ID <> 11945 and key_len: 5 u.ID = ts.speaker_id and ref: func t.ID = ts.talk_id and rows: 24 t.event_id in ( Extra: Using where; Using index; Using temporary select distinct t.event_id *************************** 5. row ********************** from talk_speaker ts, id: 2 talks t select_type: DEPENDENT SUBQUERY where ts.speaker_id = 11945 table: tsand type: ref t.ID = ts.talk_id possible_keys: talk_id ) key: talk_id order by rand() key_len: 5 limit 15G ref: joindin.t.ID rows: 1 Extra: Using where 5 rows in set (0.00 sec)

Solution1:SELECT DISTINCT u.ID as user_id, t.event_id, u.username, u.full_nameFROM user u JOIN talk_speaker ts ON u.ID = ts.speaker_id JOIN talks t ON t.ID = ts.talk_id JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G

Solution1: reminder: CREATE TABLE `talk_speaker` (SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL, t.event_id, `speaker_name` varchar(200) DEFAULT NULL, u.username, `ID` int(11) NOT NULL AUTO_INCREMENT, u.full_name `speaker_id` int(11) DEFAULT NULL,FROM user u `status` varchar(10) DEFAULT NULL, JOIN talk_speaker ts PRIMARY KEY (`ID`), ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763 ON t.ID = ts.talk_id DEFAULT CHARSET=utf8; JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G

Solution1: reminder: CREATE TABLE `talk_speaker` (SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL, t.event_id, `speaker_name` varchar(200) DEFAULT NULL, u.username, `ID` int(11) NOT NULL AUTO_INCREMENT, u.full_name `speaker_id` int(11) DEFAULT NULL,FROM user u `status` varchar(10) DEFAULT NULL, JOIN talk_speaker ts PRIMARY KEY (`ID`), ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763 ON t.ID = ts.talk_id DEFAULT CHARSET=utf8; JOIN -- find the events the speaker has talks in ( SELECT t1.event_id 2: FROM talk_speaker ts1 ALTER TABLE `talk_speaker` JOIN talks t1 ADD INDEX (speaker_id); ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 OR ) as e ALTER TABLE `talk_speaker` ON t.event_id = e.event_id ADD INDEX (speaker_id, talk_id);WHERE u.ID <> 11945LIMIT 15G

Thank youhttp://joind.in/3215

DPC Tutorial

by Ligaya Turmelle on May 26, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

DPC Tutorial Presentation Transcript