Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tek tutorial

31,815 views

Published on

Tutorial given solo. minor alterations made.

Published in: Technology
  • Be the first to comment

Tek tutorial

  1. 1. Optimizing MySQL Essentials http://joind.in/3420
  2. 2. About Me• Ligaya Turmelle• Senior Technical Support Engineer• MySQL Now (AKA Oracle)• ~3 years
  3. 3. Agenda• Basics• Hardware and OS• Server • variables• Queries • slow query log and EXPLAIN • example query
  4. 4. Basics1+1 = 3
  5. 5. • Definition of Performance: Characterized by the amount of useful work accomplished by a computer system compared to the time and resources used.
  6. 6. This can cover a lot of different areas(potentially at the same time) in a LAMPstack• Application• Network• Hardware• Database
  7. 7. Learn how to benchmark• There are lots of benchmarking tools out there.• Use what is appropriate to measure what you are trying to improve
  8. 8. OS and Hardware
  9. 9. OS• Linux vs. Windows• Linux vs. Linux
  10. 10. Linux• swappiness = 0• IO scheduler • HDD = deadline • SSD or you have a good RAID controller = noop
  11. 11. Hardware• RAM• Disks• CPU
  12. 12. Basic Linux commands• top• iostat• vmstat
  13. 13. Server Settings So what should i tweak?
  14. 14. Knobs, Levers and Buttons• mysqld• MyISAM• InnoDB• Others
  15. 15. MySQLDwhat knobs to turn?
  16. 16. MySQLD Many think that yum install / apt-get install and they’re done.• No, we’re just beginning! • Logging • Memory Usage: • Global • Session • etc
  17. 17. MySQLD( logging )
  18. 18. MySQLD ( logging )• log_bin = <filename>
  19. 19. MySQLD ( logging )• log_bin = <filename>• general_log = 0 general_log_file = <filename>
  20. 20. MySQLD ( logging )• log_bin = <filename>• general_log = 0 general_log_file = <filename>• slow_query_log = 1 slow_log_file = <filename> long_query_time = 1 • log_queries_not_using_indexes min_examined_row_limit = 100
  21. 21. MySQLD ( memory usage )• Memory is allocated... • Globally - server wide, usually allocated only at server startup and exist only “once” • Session - each connection or event
  22. 22. MySQLD ( Timers )
  23. 23. MySQLD ( Timers )• connect_timeout ( 10 )
  24. 24. MySQLD ( Timers )• connect_timeout ( 10 )• wait_timeout = 3
  25. 25. MySQLD ( Timers )• connect_timeout ( 10 )• wait_timeout = 3• expire_log_days ( 7 )
  26. 26. MySQLD( Memory / global )
  27. 27. MySQLD ( Memory / global )• query_cache_type = 0 ( yes, 0! ) • query_cache_size
  28. 28. MySQLD ( Memory / global )• query_cache_type = 0 ( yes, 0! ) • query_cache_size• max_heap_table_size / tmp_table_size
  29. 29. MySQLD( Memory / global )
  30. 30. MySQLD ( Memory / global )• table_open_cache
  31. 31. MySQLD ( Memory / global )• table_open_cache• thead_cache_size
  32. 32. MySQLD ( Memory / global )• table_open_cache• thead_cache_size• skip_name_resolve = 1
  33. 33. MySQLD ( Memory / global )• table_open_cache• thead_cache_size• skip_name_resolve = 1• performance_schema = 1
  34. 34. MySQLD ( memory / global )• read_buffer_size / read_rnd_buffer_size• sort_buffer_size• join_buffer_size } Also configurable per session
  35. 35. MyISAMwhat levers to pull
  36. 36. MyISAM
  37. 37. MyISAM• key_buffer_size • Key_reads / Key_read_requests • Key_writes / Key_write_requests
  38. 38. MyISAM• concurrent_insert • 0 - Never, 1 - Hole free, 2 - Holes
  39. 39. InnoDBwhat buttons to press
  40. 40. InnoDB Many options only a few covered• innodb_buffer_pool_size - caches both indexes and data - upwards of 85% available RAM• innodb_flush_log_at_trx_commit - 0 - written and flushed to disk once per second - 1 - written and flushed to disk at each transaction commit - 2 - written to disk at each commit, flush one a second
  41. 41. InnoDB• innodb_log_buffer_size - Amount of data written to ib_logfile - The larger, the less disk I/O• innodb_log_file_size - The larger the file, the more InnoDB can do - < 5.5 the larger, the longer crash recovery time• innodb_double_write_buffer - data integrity checking• innodb_file_per_table - data and indexes written to their own .ibd file
  42. 42. InnoDB• Efficiency: innodb_buffer_pool_reads / innodb_buffer_pool_read_requests
  43. 43. Examine, Testand Measure “Dr. Watson I Presume”
  44. 44. MySQLD ( quick check )Quick Check - mysql-tuning-primerhttps://launchpad.net/mysql-tuning-primer
  45. 45. MySQLD ( in depth 1/11 ) Getting In depth - SHOW GLOBAL STATUS+--------------------------+------------+| Variable_name | Value |+--------------------------+------------+ | Com_replace | 37372 || Com_alter_table | 184 | | Com_replace_select | 0 || Com_begin | 103281460 | | Com_rollback | 148312594 || Com_change_db | 631 | | Com_select | 7164448573 || Com_commit | 669825867 | | Com_set_option | 1877614377 || Com_create_index | 0 | | Com_show_collations | 130652029 || Com_create_table | 354 | | Com_show_fields | 694 || Com_delete | 1235248406 | | Com_show_grants | 36 || Com_delete_multi | 353434291 | | Com_show_processlist | 33184 || Com_drop_table | 359 | | Com_show_slave_hosts | 1 || Com_flush | 3 | | Com_show_variables | 130885618 || Com_insert | 2725147239 | | Com_show_warnings | 99619 || Com_insert_select | 9 | | Com_truncate | 11 || Com_kill | 3642 | | Com_unlock_tables | 18 || Com_load | 15 | | Com_update | 1348570903 || Com_lock_tables | 18 | | Com_update_multi | 8 | +--------------------------+------------+
  46. 46. MySQLD ( in depth 2/11 ) Values with out timeframe are meaningless.10:21:08 rderoo@mysql09:mysql [1169]> SHOW GLOBAL STATUS LIKE Uptime;+---------------+----------+| Variable_name | Value |+---------------+----------+| Uptime | 12973903 |+---------------+----------+1 row in set (0.00 sec) That’s about 150 days. :)
  47. 47. MySQLD ( in depth 3/11 ) By using mysqladmin we can observe changes over short periods of time:$ mysqladmin -u rderoo -p -c 2 -i 10 -r extended-status > review.txt Output is the same as SHOW GLOBAL STATUS with two copies appearing in review.txt the first has values since the last server restart / FLUSH COUNTERS the second output are the delta between the first and 10 seconds later.
  48. 48. MySQLD ( in depth 4/11 )Temporary Tables:How rderoo@mysql09:event [1185]> SHOW GLOBAL VARIABLES LIKE17:54:33 big?%_table_size;+---------------------+----------+| Variable_name | Value |+---------------------+----------+| max_heap_table_size | 67108864 || tmp_table_size | 67108864 |+---------------------+----------+2 rows in set (0.00 sec)How many?17:55:09 rderoo@mysql09:event [1186]> SHOW GLOBAL STATUS LIKE %_tmp%tables;+-------------------------+-----------+| Variable_name | Value |+-------------------------+-----------+| Created_tmp_disk_tables | 156 || Created_tmp_tables | 278190736 |+-------------------------+-----------+2 rows in set (0.00 sec)
  49. 49. MySQLD ( in depth 6/11 ) open_table_cache | Open_tables | 4094 | | Opened_tables | 12639 | | Uptime | 12980297 |• This yields a rate of ~85 table opens per day, a bit high... 20:52:22 rderoo@mysql09:mysql [1190]> SHOW GLOBAL VARIABLES LIKE table_cache; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | table_cache | 4096 | +---------------+-------+• Increasing the table_cache would be advisable.
  50. 50. MySQLD ( in depth 7/11 )thread_cache_size• To determine the effectiveness of the thread_cache as a hit ratio use the following formula:100-((Threads_created/Connections)*100) | Connections | 131877233 | | Threads_created | 20014624 | 100 - ( ( 20014624 / 131877233 ) * 100 ) = 84.8%
  51. 51. MySQLD ( in depth 8/11 )read_buffer_size• The effectiveness can be found by examining the Select_scan : | Select_scan | 304864263 |• This should be set to a small value globally and increased on a per session basis• Increase when performing full table scans
  52. 52. MySQLD ( in depth 9/11)read_rnd_buffer_size• Used when reading sorted results after a key sort has taken place• This should be set to a small value globally and increased on a per session basis• Usually should be set ( per session ) no high than 8M
  53. 53. MySQLD ( in depth 10/11)sort_buffer_size• The effectiveness can be found by examining the Sort_merge_passes : | Sort_merge_passes | 7 |• This should be set to a small value globally and increased on a per session basis• Increase GROUP BY, ORDER BY, SELECT DISTINCT and UNION DISTINCT performance
  54. 54. MySQLD ( in depth 11/11 )join_buffer_size• The effectiveness can be found by examining the Select_full_join : | Select_full_join | 5369 |• A crutch used when proper indexing is not in place• Will help in cases of full table scans• This should be set to a small value globally and increased on a per session basis• Increase GROUP BY and ORDER BY performance
  55. 55. Tuning Queries Where the magic happens
  56. 56. Tuning your queries is the single biggestperformance improvement you can do toyour server• efficient resource usage• results returned quickerPer QUERY!
  57. 57. Where do I start?
  58. 58. slow query log
  59. 59. Slow query log
  60. 60. Slow query log• A log of all the “slow” queries
  61. 61. Slow query log• A log of all the “slow” queries• Define “slow” • took longer then long_query_time to run AND at least min_examined_row_limit rows are looked at • log_queries_not_using_indexes • log_slow_admin_statements
  62. 62. Turning it on• Option is slow_query_log • no argument or value of 1 - enabled • argument of 0 - disabled (default)
  63. 63. Output Types• Use log_output • write to a file • write to a table in the mysql database
  64. 64. slow log at startup• Place the slow_query_log and optional log_output options in my.cnf• log_output acceptable values • TABLE, FILE, NONE• Can also use slow_query_log_file to specify a log file name
  65. 65. slow log at runtime• All the below global variables can be changed at run time • log_output • slow_query_log • slow_query_log_file SET GLOBAL slow_query_log = 1;
  66. 66. /usr/sbin/mysqld, Version: 5.0.67-0ubuntu6.1-log ((Ubuntu)). started with:Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sockTime Id Command Argument# Time: 110328 13:15:05# User@Host: homer[homer] @ localhost []# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 4452238use confs;select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> 6198 and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = 6198 and t.ID = ts.talk_id ) order by rand() limit 15;
  67. 67. slow query log EXPLAINed
  68. 68. There are multiple ways of gettinginformation out of the slow query log.• just open it up to see the raw data• usually more helpful to summarize/ aggregate the raw data
  69. 69. Tools for aggregating the slow log data• mk-query-digest • third party • part of the maatkit utilities• mysqldumpslow • comes with MySQL in the bin dir
  70. 70. Macintosh-7:joindin-for-lig ligaya$ perl -laF ~/mysql_installs/maatkit/mk-query-digest mysql-slow.log# 23s user time, 240ms system time, 17.53M rss, 86.51M vsz# Current date: Mon May 16 11:08:24 2011# Hostname: Macintosh-7.local# Files: mysql-slow.log# Overall: 73.72k total, 14 unique, 0.02 QPS, 0.04x concurrency __________# Time range: 2011-03-28 13:15:05 to 2011-05-13 02:33:52# Attribute total min max avg 95% stddev median# ============ ======= ======= ======= ======= ======= ======= =======# Exec time 160130s 0 180s 2s 3s 819ms 2s# Lock time 11s 0 1s 149us 0 12ms 0# Rows sent 387.64k 0 255 5.38 14.52 6.86 0# Rows examine 338.66G 0 5.35M 4.70M 4.93M 387.82k 4.70M# Query size 30.19M 17 8.35k 429.47 420.77 29.33 420.77# Profile# Rank Query ID Response time Calls R/Call Apdx V/M Item# ==== ================== ================= ===== ======= ==== ===== =====# 1 0x2AAA5EC420D7E2A2 159878.0000 99.8% 73682 2.1698 0.50 0.12 SELECTuser talks talk_speaker talks# 3 0x0A2584C03C8614A9 50.0000 0.0% 16 3.1250 0.44 0.87 SELECTwp_comments# MISC 0xMISC 202.0000 0.1% 20 10.1000 NS 0.0 <12ITEMS>
  71. 71. Count: 1 Time=180.00s (180s) Lock=0.00s (0s) Rows=1.0 (1), XXXXX[xxxxx]@localhost SELECT SLEEP(N)Count: 16 Time=3.12s (50s) Lock=0.00s (0s) Rows=0.9 (14), XXXXX[xxxxx]@localhost SELECT comment_date_gmt FROM wp_comments WHERE comment_author_IP = S ORcomment_author_email = S ORDER BY comment_date DESC LIMIT NCount: 5 Time=2.80s (14s) Lock=0.00s (0s) Rows=0.0 (0), XXXXX[xxxxx]@localhost SELECT comment_ID FROM wp_comments WHERE comment_post_ID = S AND ( comment_author= S OR comment_author_email = S ) AND comment_content = S LIMIT NCount: 73682 Time=2.17s (159867s) Lock=0.00s (11s) Rows=5.4 (396223),XXXXX[xxxxx]@localhost select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> N and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = N and t.ID = ts.talk_id ) order by rand() limit N
  72. 72. Tuning a query• Tables • SHOW CREATE TABLE `talks`; • SHOW TABLE STATUS LIKE ‘talks’;• Indexes • SHOW INDEXES FROM ‘talks’;• EXPLAIN
  73. 73. EXPLAIN
  74. 74. EXPLAIN Basics• Syntax: EXPLAIN [EXTENDED] SELECT select_options• Displays information from the optimizer about the query execution plan• Works only with SELECT statements
  75. 75. EXPLAIN Output• Each row provides information on one table• Output columns: id key_length select_type ref table rows type Filtered (new to 5.1) possible_keys Extra key
  76. 76. EXPLAIN Output
  77. 77. id• Select identifier• Only if there are subqueries, derived tables or unions is this incremented• Number reflects the order that the SELECT/FROM was done in
  78. 78. select_type• Type of SELECT • SIMPLE • PRIMARY • DERIVED • UNION • SUBQUERY
  79. 79. table• The table (or alias) that is used• Read down - these are the order the tables are JOINed• DERIVED tables are noted and numbered
  80. 80. type• Access type• Various types: preference access type BEST system/const eq_ref ref ref_or_null index_merge range index WORST ALL
  81. 81. possible_keys• List of all possible indexes (or NULL) that optimizer considered using
  82. 82. key• Index used for the query or NULL• Look at key and possible_keys to consider your index strategy
  83. 83. key_len• shows number of bytes MySQL will use from the index• Possible that only part of the index will be used NOTE: a character set may use more then one byte per character
  84. 84. ref• Very different then the access type ‘ref’• Show which columns or a constant within the index will be used for the access
  85. 85. rows• Number of rows MySQL expects to find based on statistics • statistics can be updated with ANALYZE TABLE
  86. 86. filtered• New in 5.1.12• Estimated number of rows filtered by the table condition• Shows up if you use EXPLAIN EXTENDED
  87. 87. Extra• All the other stuff • Using index - using a covering index • Using filesort - sorting in a temporary table • Using temporary - a temporary table was made. • Using where - filtering outside the storage engine.
  88. 88. Query Tuning Example
  89. 89. Query we are working with:EXPLAIN select distinct u.ID as user_id, t.event_id, u.username, u.full_name from user u, talks t, talk_speaker ts where u.ID <> 11945 and u.ID = ts.speaker_id and t.ID = ts.talk_id and t.event_id in ( select distinct t.event_id from talk_speaker ts, talks t where ts.speaker_id = 11945 and t.ID = ts.talk_id ) order by rand() limit 15G
  90. 90. Full EXPLAIN plan*************************** 1. row *************************** id: 1 select_type: PRIMARY table: t type: indexpossible_keys: PRIMARY key: idx_event key_len: 9 ref: NULL rows: 3119 Extra: Using where; Using index; Using temporary; Using filesort*************************** 2. row *************************** id: 1 select_type: PRIMARY table: ts type: refpossible_keys: talk_id key: talk_id key_len: 5 ref: joindin.t.ID rows: 1 Extra: Using where*************************** 3. row *************************** id: 1 select_type: PRIMARY table: u type: eq_refpossible_keys: PRIMARY key: PRIMARY key_len: 4 ref: joindin.ts.speaker_id rows: 1 Extra:
  91. 91. Full EXPLAIN plan (con’t)*************************** 4. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: t type: refpossible_keys: PRIMARY,idx_event key: idx_event key_len: 5 ref: func rows: 24 Extra: Using where; Using index; Using temporary*************************** 5. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: ts type: refpossible_keys: talk_id key: talk_id key_len: 5 ref: joindin.t.ID rows: 1 Extra: Using where5 rows in set (0.00 sec)
  92. 92. What was wrongwith that query plan?
  93. 93. HintEXPLAIN select distinct u.ID as user_id, *************************** 4. row *********************** t.event_id, id: 2 u.username, select_type: DEPENDENT SUBQUERY u.full_name table: t from user u, type: ref talks t, possible_keys: PRIMARY,idx_event talk_speaker ts key: idx_event where u.ID <> 11945 and key_len: 5 u.ID = ts.speaker_id and ref: func t.ID = ts.talk_id and rows: 24 t.event_id in ( Extra: Using where; Using index; Using temporary select distinct t.event_id *************************** 5. row ********************** from talk_speaker ts, id: 2 talks t select_type: DEPENDENT SUBQUERY where ts.speaker_id = 11945 table: tsand type: ref t.ID = ts.talk_id possible_keys: talk_id ) key: talk_id order by rand() key_len: 5 limit 15G ref: joindin.t.ID rows: 1 Extra: Using where 5 rows in set (0.00 sec)
  94. 94. Solution1:SELECT DISTINCT u.ID as user_id, t.event_id, u.username, u.full_nameFROM user u JOIN talk_speaker ts ON u.ID = ts.speaker_id JOIN talks t ON t.ID = ts.talk_id JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G
  95. 95. Solution1:SELECT DISTINCT u.ID as user_id, t.event_id, u.username, u.full_nameFROM user u JOIN talk_speaker ts ON u.ID = ts.speaker_id JOIN talks t ON t.ID = ts.talk_id JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G
  96. 96. Solution1:SELECT DISTINCT u.ID as user_id, t.event_id, u.username, u.full_nameFROM user u JOIN talk_speaker ts ON u.ID = ts.speaker_id JOIN talks t ON t.ID = ts.talk_id JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G
  97. 97. Solution1: reminder: CREATE TABLE `talk_speaker` (SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL, t.event_id, `speaker_name` varchar(200) DEFAULT NULL, u.username, `ID` int(11) NOT NULL AUTO_INCREMENT, u.full_name `speaker_id` int(11) DEFAULT NULL,FROM user u `status` varchar(10) DEFAULT NULL, JOIN talk_speaker ts PRIMARY KEY (`ID`), ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763 ON t.ID = ts.talk_id DEFAULT CHARSET=utf8; JOIN -- find the events the speaker has talks in ( SELECT t1.event_id FROM talk_speaker ts1 JOIN talks t1 ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 ) as e ON t.event_id = e.event_idWHERE u.ID <> 11945LIMIT 15G
  98. 98. Solution1: reminder: CREATE TABLE `talk_speaker` (SELECT DISTINCT u.ID as user_id, `talk_id` int(11) DEFAULT NULL, t.event_id, `speaker_name` varchar(200) DEFAULT NULL, u.username, `ID` int(11) NOT NULL AUTO_INCREMENT, u.full_name `speaker_id` int(11) DEFAULT NULL,FROM user u `status` varchar(10) DEFAULT NULL, JOIN talk_speaker ts PRIMARY KEY (`ID`), ON u.ID = ts.speaker_id KEY `talk_id` (`talk_id`) USING BTREE JOIN talks t ) ENGINE=MyISAM AUTO_INCREMENT=3763 ON t.ID = ts.talk_id DEFAULT CHARSET=utf8; JOIN -- find the events the speaker has talks in ( SELECT t1.event_id 2: FROM talk_speaker ts1 ALTER TABLE `talk_speaker` JOIN talks t1 ADD INDEX (speaker_id); ON t1.ID = ts1.talk_id WHERE ts1.speaker_id = 11945 OR ) as e ALTER TABLE `talk_speaker` ON t.event_id = e.event_id ADD INDEX (speaker_id, talk_id);WHERE u.ID <> 11945LIMIT 15G
  99. 99. Q&A
  100. 100. Thank youhttp://joind.in/3420

×