Performance Schema in MySQL (Danil Zburivsky)

  • 925 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
925
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
40
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Performance Schema in MySQL Danil Zburivsky MySQL DBA and Team Lead at PythianTuesday, October 23, 12
  • 2. About myself • MySQL DBA and Team Lead at Pythian • Managing dozens of customers and thousands of MySQL servers • http://www.pythian.com/ news/author/zburivsky/ • @zburivskyTuesday, October 23, 12
  • 3. Debugging MySQL PerformanceTuesday, October 23, 12
  • 4. MySQL is a great database, but instrumentation sucks Slow query log SHOW ENGINE INNODB STATUS OS STATS SHOW PROCESSLISTTuesday, October 23, 12
  • 5. I think there is a problem with database...Tuesday, October 23, 12
  • 6. I think there is a problem with database... • Identify timeframe (use trending tools)Tuesday, October 23, 12
  • 7. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incidentTuesday, October 23, 12
  • 8. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more dataTuesday, October 23, 12
  • 9. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can!Tuesday, October 23, 12
  • 10. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of itTuesday, October 23, 12
  • 11. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of it • Tune settings, SQL, hardware. Did it help?Tuesday, October 23, 12
  • 12. I think there is a problem with database... • Identify timeframe (use trending tools) • Collect data during incident • Collect more data • Collect as much data as you can! • Try to make sense of it • Tune settings, SQL, hardware. Did it help? • I think there is a problem with database...Tuesday, October 23, 12
  • 13. Tools that might help you • pt-stalk (http://www.percona.com/doc/ percona-toolkit/2.1/pt-stalk.html) • innotop (http://code.google.com/p/ innotop/) • Or you write your own stuffTuesday, October 23, 12
  • 14. What do “Big Boys” have? • Oracle: SQL-traces, kernel events timings, tons of books on performance tuning • SQL Server: SQL profiler, Data Management Views, Extended Events,tons of books on performance tuningTuesday, October 23, 12
  • 15. Performance schema in MySQL 5.5Tuesday, October 23, 12
  • 16. The Idea • Executing a query breaks down to hundreds of smaller tasks • There are background tasks as well • We want to instrument it all to know where server is spending timeTuesday, October 23, 12
  • 17. Implementation • Instrumentation: measuring when event begins and ends • Implemented in MySQL code on server end storage engine level • Can be enabled/disabled or customizedTuesday, October 23, 12
  • 18. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 19. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 20. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 21. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 22. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 23. Schema and a storage engine +----------------------------------------------+ | Tables_in_performance_schema | +----------------------------------------------+ | cond_instances | | events_waits_current | | events_waits_history | | events_waits_history_long | | events_waits_summary_by_instance | | events_waits_summary_by_thread_by_event_name | | events_waits_summary_global_by_event_name | | file_instances | | file_summary_by_event_name | | file_summary_by_instance | | mutex_instances | | performance_timers | | rwlock_instances | | setup_consumers | | setup_instruments | | setup_timers | | threads | +----------------------------------------------+ 17 rows in set (0.00 sec)Tuesday, October 23, 12
  • 24. Instruments +------------------------------------------------------------+ | NAME | +------------------------------------------------------------+ | wait/synch/mutex/sql/PAGE::lock | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active | | wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool | ..... | | wait/synch/rwlock/sql/LOCK_grant | | wait/synch/rwlock/sql/LOGGER::LOCK_logger | ...... | wait/synch/cond/sql/COND_thread_count | | wait/synch/cond/sql/COND_thread_cache | ..... | | wait/io/file/sql/binlog | | wait/io/file/sql/binlog_index | +------------------------------------------------------------+Tuesday, October 23, 12
  • 25. Consumers SELECT * FROM setup_consumers; +----------------------------------------------+---------+ | NAME | ENABLED | +----------------------------------------------+---------+ | events_waits_current | YES | | events_waits_history | YES | | events_waits_history_long | YES | | events_waits_summary_by_thread_by_event_name | YES | | events_waits_summary_by_event_name | YES | | events_waits_summary_by_instance | YES | | file_summary_by_event_name | YES | | file_summary_by_instance | YES | +----------------------------------------------+---------+ 8 rows in set (0.00 sec)Tuesday, October 23, 12
  • 26. What are top wait events in my server?SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAITFROM events_waits_summary_by_instanceORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+---------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+---------------------------------------------+-----------------------+---------------+---------|| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 || wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 || wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 || wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 || wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 || wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 || wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |+---------------------------------------------+------------+-------------------+----------------+10 rows in set (0.03 sec)Tuesday, October 23, 12
  • 27. innodb_buffer_pool_instances=10SELECT * FROM events_waits_summary_by_instanceORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+-----------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+-----------------------------------------------+------------+-------------------+----------------+| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 || wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 || wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 || wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 || wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 || wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 || wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |+-----------------------------------------------+------------+-------------------+----------------+10 rows in set (0.02 sec) buf_pool_mutex is no longer in top 10 wait eventsTuesday, October 23, 12
  • 28. Which files are being accessed the most?mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITEFROM file_summary_by_instanceORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;+------------------------------+--------------------------+---------------------------+| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |+------------------------------+--------------------------+---------------------------+| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 || /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 || /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 || /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 || /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 || /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 || /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 || /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 || /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 || /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |+------------------------------+--------------------------+---------------------------+10 rows in set (0.01 sec)Tuesday, October 23, 12
  • 29. Threads table(performance_schema) > SELECT * FROM threads;+-----------+----------------+----------------------------------------+| THREAD_ID | PROCESSLIST_ID | NAME |+-----------+----------------+----------------------------------------+| 0 | 0 | thread/sql/main || 12 | 0 | thread/innodb/srv_lock_timeout_thread || 14 | 0 | thread/innodb/srv_monitor_thread || 16 | 0 | thread/innodb/srv_purge_thread || 13 | 0 | thread/innodb/srv_error_monitor_thread || 2 | 0 | thread/innodb/io_handler_thread || 1 | 0 | thread/innodb/io_handler_thread || 9 | 0 | thread/innodb/io_handler_thread || 25 | 8 | thread/sql/one_connection || 3 | 0 | thread/innodb/io_handler_thread || 15 | 0 | thread/innodb/srv_master_thread || 10 | 0 | thread/innodb/io_handler_thread || 5 | 0 | thread/innodb/io_handler_thread || 8 | 0 | thread/innodb/io_handler_thread || 7 | 0 | thread/innodb/io_handler_thread || 17 | 0 | thread/sql/signal_handler || 4 | 0 | thread/innodb/io_handler_thread || 6 | 0 | thread/innodb/io_handler_thread |+-----------+----------------+----------------------------------------+18 rows in set (0.00 sec)Tuesday, October 23, 12
  • 30. Average wait per thread (I)SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)FROM events_waits_summary_by_thread_by_event_name eLEFT JOIN threads t ON t.THREAD_ID = e.THREAD_IDWHERE event_name LIKE wait/synch/mutex/innodb/%AND t.NAME=thread/sql/one_connectionAND AVG_TIMER_WAIT > 0GROUP BY t.thread_id, e.EVENT_NAMEORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)Tuesday, October 23, 12
  • 31. Average wait per thread (II) +-----------+-------------------------------------------------------+-------------------------+ | THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) | +-----------+-------------------------------------------------------+-------------------------+ | 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 | | 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 | ..... ..... | 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 | | 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 | | 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 | | 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 | ---------------------------------------------------------------------------------------------- | 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 | | 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 | | 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 | ..... ..... | 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 | | 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 | | 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 | | 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 | | 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 | | 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |Tuesday, October 23, 12
  • 32. Average wait per thread (III) innodb_change_buffering = none [Don’t try this at home! ] +-----------+--------------------------------------------------+-------------------------+ | THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) | +-----------+--------------------------------------------------+-------------------------+ | 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 | | 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 | | 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 | | 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 | ..... ..... | 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 | | 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 | | 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 | | 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 | ----------------------------------------------------------------------------------------- | 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 | | 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 | | 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 | | 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 | .... .... | 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 | | 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 | | 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 | | 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 | | 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |Tuesday, October 23, 12
  • 33. Picoseconds, shmicosecond! ps_helper to rescue! • http://www.markleith.co.uk/ps_helper/ • Great examples of using performance_schema • Useful tools for converting time, bytes, paths into human readable formatTuesday, October 23, 12
  • 34. Example: top_global_io_consumers_by_bytes_usage(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latencyFROM top_global_io_consumers_by_bytes_usage;+-------------------------+------------+-----------+---------------+-------------+-------------+| event_name | total_read | avg_read | total_written | avg_written | avg_latency |+-------------------------+------------+-----------+---------------+-------------+-------------+| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms || innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms || myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs || sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs || sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs || myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs || mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs || sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs || sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs || sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs || archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs || sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs || sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |+-------------------------+------------+-----------+---------------+-------------+-------------+13 rows in set (0.00 sec)Tuesday, October 23, 12
  • 35. No free lunch: PFS overhead • Instrumentation doesn’t come for free • In CPU-bound workloads overhead can be ~15-20% • In IO-bound — 5%-8% • Significant improvements in 5.6Tuesday, October 23, 12
  • 36. Performance schema in MySQL 5.6Tuesday, October 23, 12
  • 37. What’s new in 5.6? • performance_schema enabled by default! • Less overhead: 5%-10% for CPU-bound workload • Statements, Stages, Actors and ObjectsTuesday, October 23, 12
  • 38. Actors: filter events by user(performance_schema) > SELECT * FROM setup_actors;+------+------+------+| HOST | USER | ROLE |+------+------+------+| % | % | % |+------+------+------+1 row in set (0.00 sec)Tuesday, October 23, 12
  • 39. Objects: filter out events by database/table(performance_schema) > SELECT * FROM setup_objects;+-------------+--------------------+-------------+---------+-------+| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |+-------------+--------------------+-------------+---------+-------+| TABLE | mysql | % | NO | NO || TABLE | performance_schema | % | NO | NO || TABLE | information_schema | % | NO | NO || TABLE | % | % | YES | YES |+-------------+--------------------+-------------+---------+-------+4 rows in set (0.00 sec)Tuesday, October 23, 12
  • 40. StatementsSELECT * FROM events_statements_summary_by_digestG*************************** 4. row *************************** DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY`SUM_TIMER_WAIT` DESC LIMIT ? COUNT_STAR: 1 SUM_TIMER_WAIT: 511539000 MIN_TIMER_WAIT: 511539000 AVG_TIMER_WAIT: 511539000 MAX_TIMER_WAIT: 511539000 SUM_LOCK_TIME: 72000000 SUM_ERRORS: 0 SUM_WARNINGS: 0 SUM_ROWS_AFFECTED: 0 SUM_ROWS_SENT: 5 SUM_ROWS_EXAMINED: 91SUM_CREATED_TMP_DISK_TABLES: 0 SUM_CREATED_TMP_TABLES: 0 SUM_SELECT_FULL_JOIN: 0 SUM_SELECT_FULL_RANGE_JOIN: 0 SUM_SELECT_RANGE: 0 SUM_SELECT_RANGE_CHECK: 0 SUM_SELECT_SCAN: 1 SUM_SORT_MERGE_PASSES: 0 SUM_SORT_RANGE: 0 SUM_SORT_ROWS: 5 SUM_SORT_SCAN: 1 SUM_NO_INDEX_USED: 1 SUM_NO_GOOD_INDEX_USED: 0 FIRST_SEEN: 2012-10-16 21:50:13 LAST_SEEN: 2012-10-16 21:50:13Tuesday, October 23, 12
  • 41. Summary. Pros. • Provides insight into internal server metric • Flexible • Relational model allows to build your own views on top of this dataTuesday, October 23, 12
  • 42. Summary. Cons. • Steep learning curve: not very well documented • Overhead for CPU-bound loads can be significantTuesday, October 23, 12
  • 43. Thank you!Tuesday, October 23, 12