Performance Schema in MySQL (Danil Zburivsky)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Performance Schema in MySQL (Danil Zburivsky)

on

  • 1,080 views

 

Statistics

Views

Total Views
1,080
Views on SlideShare
961
Embed Views
119

Actions

Likes
1
Downloads
38
Comments
0

3 Embeds 119

http://www.highload.ru 108
http://www.megaindex.tv 10
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Performance Schema in MySQL (Danil Zburivsky) Presentation Transcript

  • 1. Performance Schema in MySQL Danil Zburivsky MySQL DBA and Team Lead at PythianTuesday, October 23, 12
  • 2. About myself • MySQL DBA and Team Lead at Pythian • Managing dozens of customers and thousands of MySQL servers • http://www.pythian.com/ news/author/zburivsky/ • @zburivskyTuesday, October 23, 12
  • 3. Debugging MySQL PerformanceTuesday, October 23, 12
  • 4. MySQL is a great database, but instrumentation sucks Slow query log SHOW ENGINE INNODB STATUS OS STATS SHOW PROCESSLISTTuesday, October 23, 12
  • 5. I think there is a problem with database...Tuesday, October 23, 12
  • 6. I think there is a problem with database... • Identify timeframe (use trending tools)Tuesday, October 23, 12
  • 7. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incidentTuesday, October 23, 12
  • 8. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more dataTuesday, October 23, 12
  • 9. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can!Tuesday, October 23, 12
  • 10. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of itTuesday, October 23, 12
  • 11. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of it • Tune settings, SQL, hardware. Did it help?Tuesday, October 23, 12
  • 12. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of it • Tune settings, SQL, hardware. Did it help? • I think there is a problem with database...Tuesday, October 23, 12
  • 13. Tools that might help you • pt-stalk (http://www.percona.com/doc/ percona-toolkit/2.1/pt-stalk.html) • innotop (http://code.google.com/p/ innotop/) • Or you write your own stuffTuesday, October 23, 12
  • 14. What do “Big Boys” have? • Oracle: SQL-traces, kernel events timings, tons of books on performance tuning • SQL Server: SQL profiler, Data Management Views, Extended Events,tons of books on performance tuningTuesday, October 23, 12
  • 15. Performance schema in MySQL 5.5Tuesday, October 23, 12
  • 16. The Idea • Executing a query breaks down to hundreds of smaller tasks • There are background tasks as well • We want to instrument it all to know where server is spending timeTuesday, October 23, 12
  • 17. Implementation • Instrumentation: measuring when event begins and ends • Implemented in MySQL code on server end storage engine level • Can be enabled/disabled or customizedTuesday, October 23, 12
  • 18. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 19. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 20. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 21. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 22. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 23. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 24. Instruments +------------------------------------------------------------+ | NAME | +------------------------------------------------------------+ | wait/synch/mutex/sql/PAGE::lock | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool | ..... | | wait/synch/rwlock/sql/LOCK_grant | | wait/synch/rwlock/sql/LOGGER::LOCK_logger | ...... | wait/synch/cond/sql/COND_thread_count | | wait/synch/cond/sql/COND_thread_cache | ..... | | wait/io/file/sql/binlog | | wait/io/file/sql/binlog_index | +------------------------------------------------------------+Tuesday, October 23, 12
  • 25. Consumers SELECT * FROM setup_consumers; +----------------------------------------------+---------+ | NAME | ENABLED | +----------------------------------------------+---------+ | events_waits_current | YES | | events_waits_history | YES | | events_waits_history_long | YES | | events_waits_summary_by_thread_by_event_name | YES | | events_waits_summary_by_event_name | YES | | events_waits_summary_by_instance | YES | | file_summary_by_event_name | YES | | file_summary_by_instance | YES | +----------------------------------------------+---------+ 8 rows in set (0.00 sec)Tuesday, October 23, 12
  • 26. What are top wait events in my server?SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAITFROM events_waits_summary_by_instanceORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+---------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+---------------------------------------------+-----------------------+---------------+---------|| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 || wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 || wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 || wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 || wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 || wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 || wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |+---------------------------------------------+------------+-------------------+----------------+10 rows in set (0.03 sec)Tuesday, October 23, 12
  • 27. innodb_buffer_pool_instances=10SELECT * FROM events_waits_summary_by_instanceORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+-----------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+-----------------------------------------------+------------+-------------------+----------------+| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 || wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 || wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 || wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 || wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 || wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 || wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |+-----------------------------------------------+------------+-------------------+----------------+10 rows in set (0.02 sec) buf_pool_mutex is no longer in top 10 wait eventsTuesday, October 23, 12
  • 28. Which files are being accessed the most?mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITEFROM file_summary_by_instanceORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;+------------------------------+--------------------------+---------------------------+| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |+------------------------------+--------------------------+---------------------------+| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 || /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 || /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 || /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 || /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 || /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 || /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 || /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 || /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 || /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |+------------------------------+--------------------------+---------------------------+10 rows in set (0.01 sec)Tuesday, October 23, 12
  • 29. Threads table(performance_schema) > SELECT * FROM threads;+-----------+----------------+----------------------------------------+| THREAD_ID | PROCESSLIST_ID | NAME |+-----------+----------------+----------------------------------------+| 0 | 0 | thread/sql/main || 12 | 0 | thread/innodb/srv_lock_timeout_thread || 14 | 0 | thread/innodb/srv_monitor_thread || 16 | 0 | thread/innodb/srv_purge_thread || 13 | 0 | thread/innodb/srv_error_monitor_thread || 2 | 0 | thread/innodb/io_handler_thread || 1 | 0 | thread/innodb/io_handler_thread || 9 | 0 | thread/innodb/io_handler_thread || 25 | 8 | thread/sql/one_connection || 3 | 0 | thread/innodb/io_handler_thread || 15 | 0 | thread/innodb/srv_master_thread || 10 | 0 | thread/innodb/io_handler_thread || 5 | 0 | thread/innodb/io_handler_thread || 8 | 0 | thread/innodb/io_handler_thread || 7 | 0 | thread/innodb/io_handler_thread || 17 | 0 | thread/sql/signal_handler || 4 | 0 | thread/innodb/io_handler_thread || 6 | 0 | thread/innodb/io_handler_thread |+-----------+----------------+----------------------------------------+18 rows in set (0.00 sec)Tuesday, October 23, 12
  • 30. Average wait per thread (I)SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)FROM events_waits_summary_by_thread_by_event_name eLEFT JOIN threads t ON t.THREAD_ID = e.THREAD_IDWHERE event_name LIKE wait/synch/mutex/innodb/%AND t.NAME=thread/sql/one_connectionAND AVG_TIMER_WAIT > 0GROUP BY t.thread_id, e.EVENT_NAMEORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)Tuesday, October 23, 12
  • 31. Average wait per thread (II) +-----------+-------------------------------------------------------+-------------------------+ | THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) | +-----------+-------------------------------------------------------+-------------------------+ | 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 | | 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 | ..... ..... | 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 | | 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 | | 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 | | 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 | ---------------------------------------------------------------------------------------------- | 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 | | 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 | | 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 | ..... ..... | 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 | | 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 | | 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 | | 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 | | 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 | | 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |Tuesday, October 23, 12
  • 32. Average wait per thread (III) innodb_change_buffering = none [Don’t try this at home! ] +-----------+--------------------------------------------------+-------------------------+ | THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) | +-----------+--------------------------------------------------+-------------------------+ | 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 | | 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 | | 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 | | 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 | ..... ..... | 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 | | 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 | | 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 | | 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 | ----------------------------------------------------------------------------------------- | 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 | | 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 | | 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 | | 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 | .... .... | 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 | | 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 | | 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 | | 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 | | 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |Tuesday, October 23, 12
  • 33. Picoseconds, shmicosecond! ps_helper to rescue! • http://www.markleith.co.uk/ps_helper/ • Great examples of using performance_schema • Useful tools for converting time, bytes, paths into human readable formatTuesday, October 23, 12
  • 34. Example: top_global_io_consumers_by_bytes_usage(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latencyFROM top_global_io_consumers_by_bytes_usage;+-------------------------+------------+-----------+---------------+-------------+-------------+| event_name | total_read | avg_read | total_written | avg_written | avg_latency |+-------------------------+------------+-----------+---------------+-------------+-------------+| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms || innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms || myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs || sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs || sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs || myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs || mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs || sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs || sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs || sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs || archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs || sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs || sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |+-------------------------+------------+-----------+---------------+-------------+-------------+13 rows in set (0.00 sec)Tuesday, October 23, 12
  • 35. No free lunch: PFS overhead • Instrumentation doesn’t come for free • In CPU-bound workloads overhead can be ~15-20% • In IO-bound — 5%-8% • Significant improvements in 5.6Tuesday, October 23, 12
  • 36. Performance schema in MySQL 5.6Tuesday, October 23, 12
  • 37. What’s new in 5.6? • performance_schema enabled by default! • Less overhead: 5%-10% for CPU-bound workload • Statements, Stages, Actors and ObjectsTuesday, October 23, 12
  • 38. Actors: filter events by user(performance_schema) > SELECT * FROM setup_actors;+------+------+------+| HOST | USER | ROLE |+------+------+------+| % | % | % |+------+------+------+1 row in set (0.00 sec)Tuesday, October 23, 12
  • 39. Objects: filter out events by database/table(performance_schema) > SELECT * FROM setup_objects;+-------------+--------------------+-------------+---------+-------+| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |+-------------+--------------------+-------------+---------+-------+| TABLE | mysql | % | NO | NO || TABLE | performance_schema | % | NO | NO || TABLE | information_schema | % | NO | NO || TABLE | % | % | YES | YES |+-------------+--------------------+-------------+---------+-------+4 rows in set (0.00 sec)Tuesday, October 23, 12
  • 40. StatementsSELECT * FROM events_statements_summary_by_digestG*************************** 4. row *************************** DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY`SUM_TIMER_WAIT` DESC LIMIT ? COUNT_STAR: 1 SUM_TIMER_WAIT: 511539000 MIN_TIMER_WAIT: 511539000 AVG_TIMER_WAIT: 511539000 MAX_TIMER_WAIT: 511539000 SUM_LOCK_TIME: 72000000 SUM_ERRORS: 0 SUM_WARNINGS: 0 SUM_ROWS_AFFECTED: 0 SUM_ROWS_SENT: 5 SUM_ROWS_EXAMINED: 91SUM_CREATED_TMP_DISK_TABLES: 0 SUM_CREATED_TMP_TABLES: 0 SUM_SELECT_FULL_JOIN: 0 SUM_SELECT_FULL_RANGE_JOIN: 0 SUM_SELECT_RANGE: 0 SUM_SELECT_RANGE_CHECK: 0 SUM_SELECT_SCAN: 1 SUM_SORT_MERGE_PASSES: 0 SUM_SORT_RANGE: 0 SUM_SORT_ROWS: 5 SUM_SORT_SCAN: 1 SUM_NO_INDEX_USED: 1 SUM_NO_GOOD_INDEX_USED: 0 FIRST_SEEN: 2012-10-16 21:50:13 LAST_SEEN: 2012-10-16 21:50:13Tuesday, October 23, 12
  • 41. Summary. Pros. • Provides insight into internal server metric • Flexible • Relational model allows to build your own views on top of this dataTuesday, October 23, 12
  • 42. Summary. Cons. • Steep learning curve: not very well documented • Overhead for CPU-bound loads can be significantTuesday, October 23, 12
  • 43. Thank you!Tuesday, October 23, 12