MySQL Performance schema missing_manual_flossuk

2,737 views

Published on

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,737
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
67
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

MySQL Performance schema missing_manual_flossuk

  1. 1. PERFORMANCE_SCHEMA in MySQL 5.6: The Missing Manual Valerii Kravchuk, Principal Support Engineer valerii.kravchuk@percona.com
  2. 2. www.percona.com Who am I? Valerii (aka Valeriy) Kravchuk: ● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012 ○ Bugs Verification Team all this time ○ Support issues related to bugs, InnoDB, performance and more ○ Trainings (mostly informal) for new team members ○ All kinds of decision making committees… ● Principal Support Engineer in Percona, 2012 - ... ○ Doing more or less the same as before, but better (I hope)... ○ Plus I try to speak and write about MySQL in public now ● http://mysqlentomologist.blogspot.com - my blog about MySQL (mostly bugs) ● https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about MySQL (mostly bugs…) ● http://bugs.mysql.com - my personal playground. 70 bugs reported in 2014
  3. 3. www.percona.com What is this session about? ● How to find the root causes of MySQL server performance problems? ○ http://method-r.com/faq/35-what-is-method-r ○ Where time is spent? ● Proper instrumentation of the code, is it really important? ○ http://tkyte.blogspot.com/2005/06/instrumentation.html ○ http://en.wikipedia.org/wiki/Observer_effect_(physics) ● OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any performance problem in MySQL? ● Instrumentation in MySQL - PERFORMANCE_SCHEMA ○ History and engineers involved ○ Features in MySQL 5.6 ○ How to use it? ○ Any alternatives in MySQL world? ○ Sources of additional information and tools
  4. 4. www.percona.com Method R - common sense approach 1. Identify the most important task (the bottleneck) 2. Measure its response time (R) in detail a. Time spent on useful processing b. Time spent on waiting (I/O, locks, mutexes) 3. Optimize that response time in the most (economically) efficient way a. Process (do) less b. Try to reduce waits of all kinds 4. Repeat until your system is (economically) optimal
  5. 5. www.percona.com On Proper Instrumentation of Code “Code instrumentation, what is it? To me, it is the fine art of making approximately every other line of your developed code “debug” of some sort. Trace information. With timestamps. And meaningful information.” - Tom Kyte “In science, the term observer effect refers to changes that the act of observation will make on a phenomenon being observed. This is often the result of instruments that, by necessity, alter the state of what they measure in some manner.” - Wikipedia
  6. 6. www.percona.com Sources of information on performance ● Trace files from -debug binaries, optimizer trace files ● Extended slow query log ● show global status; ● show engine innodb statusG ● show engine innodb mutex; ● InnoDB-related tables in the INFORMATION_SCHEMA ● userstat (Percona Server and other builds) ● show profiles; ● Poor man’s profiler (pt-pmp) or real profilers (perf, oprofile) ● DTrace ● tcpdump analysis (see https://vividcortex.com/blog/2014/02/25/performance-schema-slow- query-log-tcp-sniffing/) ● In MySQL 5.6+ it’s time to use PERFORMANCE_SCHEMA!
  7. 7. www.percona.com Extended slow query log ● Feature of Percona Server (5.1+) ● http://www.percona.com/doc/percona-server/5.5/diagnostics/slow_extended_55.html # User@Host: mailboxer[mailboxer] @ [192.168.10.165] # Thread_id: 11167745 Schema: board # Query_time: 1.009400 Lock_time: 0.000190 Rows_sent: 4 Rows_examined: 1543719 Rows_affected: 0 Rows_read: 4 # Bytes_sent: 278 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: 1500 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 6415 InnoDB_IO_r_bytes: 105103360 InnoDB_IO_r_wait: 0.001279 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 6430 SET timestamp=1346844943; SELECT id,title,production_year FROM title WHERE title = 'Bambi';
  8. 8. www.percona.com Percona Server and userstat http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html: ● Introduced in Google patch, allows to solve many performance issues ● Additional tables in INFORMATION_SCHEMA: ○ CLIENT_STATISTICS ○ THREAD_STATISTICS ○ USER_STATISTICS ○ TABLE_STATISTICS ○ INDEX_STATISTICS ● Easy to configure: ○ http://dom.as/2013/04/17/on-performance-schemas/ ● Low performance impact as some sources say: ○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/ ● But some sources say it adds overhead: ○ “and then, once userstat=1 the regression is horrible.. over 30%... “ ● Hardly this can help with mutexes or waits…
  9. 9. www.percona.com Time to try PERFORMANCE_SCHEMA ● We know how to find functions where most of the time was spent (profiling) ● We know how to find mutex waits that were long enough to force OS wait, or how to find blocking row locks... ● If we need more details, specifically time spent waiting… it’s time to use PERFORMANCE_SCHEMA it seems ● Sample output while test (for Bug #68079) was running, naive attempt (the time is in picoseconds, 1E-12, 0.000 000 000 001): mysql> select event_name, count_star, sum_timer_wait from performance_schema. events_waits_summary_global_by_event_name where event_name like 'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5; +--------------------------------------------+------------+----------------+ | event_name | count_star | sum_timer_wait | +--------------------------------------------+------------+----------------+ | wait/synch/mutex/innodb/os_mutex | 71754726 | 11380485535846 | | wait/synch/mutex/innodb/trx_sys_mutex | 60057 | 4191727883 | | wait/synch/mutex/innodb/buf_pool_mutex | 8655 | 468303388 | | wait/synch/mutex/innodb/mutex_list_mutex | 10310 | 427385546 | | wait/synch/mutex/innodb/rw_lock_list_mutex | 8304 | 316946749 | +--------------------------------------------+------------+----------------+ 5 rows in set (0.00 sec)
  10. 10. www.percona.com Some history behind... ● Read the WorkLogs: ○ http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this” ○ http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors” ○ http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…” ● Where the name comes from? “Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL), PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan “
  11. 11. www.percona.com Basic ideas and terms PERFORMANCE_SCHEMA (aka PS, P_S, PFS): ● Inspired by Oracle Wait Interface ● Separate database and storage engine ● Records run time statistics via built-in instrumentation points ● Designed to let us get the answers to the following questions: ○ Where was time spent? ○ Who has used most of resources/time? ○ What was executed to to use these? ○ When it was executed? ● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6 (52 tables), and is documented and explained in many details already: ○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html ● Implemented by Marc Alff (http://marcalff.blogspot.com/) ● Deal with instruments (including wait events), consumers, objects (tables for now) that are monitored, threads (some of them background) that run and actors (users) that execute SQL statements (in stages - thread statuses).
  12. 12. www.percona.com Overview of PERFORMANCE_SCHEMA tables
  13. 13. www.percona.com What to do with these tables? ● Table names are lowercase and long (events_statements_summary_by_digest) ● Column names are uppercase ● Tables are “in memory”, no indexes, content disappear or is reinitiated upon restart ● In general you are supposed to SELECT … FROM ..., UPDATE selected columns (setup_* tables), INSERT or DELETE (setup_actors, setup_objects) rows or run TRUNCATE TABLE (events_* tables and *_summary_* tables) use performance_schema DESC events_waits_current; SHOW CREATE TABLE setup_timersG UPDATE setup_instruments SET ENABLED=”YES”, TIMED=”YES”; UPDATE setup_consumers SET ENABLED=”YES”; SELECT * FROM threads; TRUNCATE TABLE events_statements_summary_by_digest; ● TRUNCATE TABLE for summary tables resets summary columns to 0 or NULL ○ All but events_statements_summary_by_digest - digest rows are removed ● You need SELECT, UPDATE, … and DROP privileges, accordingly
  14. 14. www.percona.com So, how to use P_S? ● The amount of information in the manual is huge, it seems not easy to start using P_S ○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT ● Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuring- performance-schema.html: ○ “Define your goals” - for example, we have waits and we need to find out what waits are most important based on time wasted (or waited) ○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits and internal lock waits, but how to do this? Details next, but: ■ "wait/synch/mutex/%", for mutexes ■ "wait/synch/rwlock/%", for read write locks ○ “Define how much detail to collect” - depends on goals... ○ “Provide sizing data” - autosizing based on max_connections (mostly) ○ “Monitor sizing problems” - memory used vs dropped statistics and impact
  15. 15. www.percona.com Define your goals ● From casual monitoring (“non-blocking SHOW”) to debugging “hangs” and solving complex scalability problems... ● Examples include: ○ Monitoring file I/O globally for capacity planning ○ Extended threads monitoring (“better SHOW PROCESSLIST”) ○ Performance of SQL statements by stages (“extended slow query log”) ○ Load by database and by table, unused indexes etc (“userstat”, setup_objects) ○ Find InnoDB scalability bottlenecks (“hot” mutex or rwlock) ● Set realistic goals. PERFORMANCE_SCHEMA (in 5.6) is NOT any good for: ○ Checking row locks in InnoDB tables (or explaining deadlocks) ○ Checking metadata locks (until 5.7.3) ○ Finding memory leaks (until 5.7.x) ○ Keeping historical data about performance (buffers are limited, no disk storage) ○ Explaining optimizer decisions (there is optimizer trace for this in 5.6) ● Instruments to enable, level of details to collect and overhead depend on goals and deployment (production server vs development machine) ● It makes sense to concentrate on things you can control/change
  16. 16. www.percona.com Define what to instrument ● It’s controlled by the content of setup_instuments table ● By default the following instruments are enabled (for slow SQL or I/O mostly): ○ File I/O ○ Table I/O ○ Table lock ○ Statements ○ Idle ● Dynamic (all but wait/synch/%), can be set in my.cnf also mysql> select * from performance_schema.setup_instruments where enabled='YES' limit 5; +---------------------------------+---------+-------+ | NAME | ENABLED | TIMED | +---------------------------------+---------+-------+ | wait/io/file/sql/map | YES | YES | | wait/io/file/sql/binlog | YES | YES | | wait/io/file/sql/binlog_index | YES | YES | | wait/io/file/sql/relaylog | YES | YES | | wait/io/file/sql/relaylog_index | YES | YES | +---------------------------------+---------+-------+ 5 rows in set (0.00 sec)
  17. 17. www.percona.com How instruments are named? ● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrument- naming.html: ○ "wait/io/file/%" - file io ○ "wait/io/socket/%" - network io ○ "wait/io/table/%" - table io ○ "wait/lock/table/%" - table locks ○ "wait/synch/cond/%" - conditions ○ "wait/synch/mutex/%" - mutexes ○ "wait/synch/rwlock/%" - read write locks ○ "stage/%" - stages of statement execution (see show profile) ○ "idle" - clients idle time. ○ "statement/com/%" - commands in the client/server protocol ○ "statement/sql/%" - SQL queries (and some more statement/…) Hint: bold means “new in MySQL 5.6”, blue color means enabled by default
  18. 18. www.percona.com Define how much details to collect ● It’s controlled by the content of setup_consumers table ● Dynamic, can be set in my.cnf also ● By default the following instruments are enabled (global, thread, statements digest and details about statements), they are ring buffers (sizing discussed later): mysql> select * from performance_schema.setup_consumers; +--------------------------------+---------+ | NAME | ENABLED | +--------------------------------+---------+ | events_stages_current | NO | # current stage of execution per thread | events_stages_history | NO | # 10 recent stage events per thread | events_stages_history_long | NO | # 10000 recent stage events overall | events_statements_current | YES | # P_S.events_statements_current (details…) | events_statements_history | NO | # historical data, 10 recent per thread | events_statements_history_long | NO | # historical data, 10000 recent overall | events_waits_current | NO | # current wait event per thread | events_waits_history | NO | # 10 recent wait events per thread | events_waits_history_long | NO | # 10000 recent wait events overall | global_instrumentation | YES | # P_S enabled or not essentially (1 table) | thread_instrumentation | YES | # Statistics collection per thread (userstat) | statements_digest | YES | # P_S.events_statements_summary_by_digest +--------------------------------+---------+ 12 rows in set (0.00 sec)
  19. 19. www.percona.com Example: setup for checking waits... ● http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschema- to-check.html - my experience back at MySQL 5.6 RC stage... ● To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in general with MySQL 5.6.x you should do at least the following: 1. Start MySQL server with all mutex related instruments enabled at startup (performance_schema=ON by default), like this: mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' & 2. Connect to server and set up proper consumers explicitly, like this: UPDATE performance_schema.setup_consumers SET enabled = 'YES' WHERE name like 'events_waits%'; 3. Run your problematic load. 4. Check waits using whatever tables you need
  20. 20. www.percona.com Example: everything enabled in my.cnf To set up all instruments and consumers (once and forever) in my.cnf, add: performance_schema_instrument = '%=on' performance_schema_consumer_events_stages_current = ON performance_schema_consumer_events_stages_history = ON performance_schema_consumer_events_stages_history_long = ON performance_schema_consumer_events_statements_current = ON performance_schema_consumer_events_statements_history = ON performance_schema_consumer_events_statements_history_long = ON performance_schema_consumer_events_waits_current = ON performance_schema_consumer_events_waits_history = ON performance_schema_consumer_events_waits_history_long = ON performance_schema_consumer_global_instrumentation = ON performance_schema_consumer_thread_instrumentation = ON performance_schema_consumer_statements_digest = ON
  21. 21. www.percona.com Define proper sizing for data collected ● Default sizes may just work (autosizing happens): mysql> show global variables like 'performance_schema%'; +--------------------------------------------------------+-------+ | Variable_name | Value | +--------------------------------------------------------+-------+ | performance_schema | ON | | performance_schema_accounts_size | 100 | | performance_schema_digests_size | 10000 | | performance_schema_events_stages_history_long_size | 10000 | | performance_schema_events_stages_history_size | 10 | ... | performance_schema_max_mutex_instances | 15906 | ... ● The following 4 server variables define autosizing heuristic: ○ max_connections ○ table_definition_cache ○ table_open_cache ○ open_files_limit (it also depends on ulimit at OS level and defines size of 3 above if it’s not high enough to accept settings)
  22. 22. www.percona.com Monitor sizing problems ● Too big means a lot of memory used (no memory leaks though!) ● Too small means partial data collected and performance degradation ● When this happens proper status variable *_lost (23 in total) is incremented: mysql> show global status like 'performance_schema%'; +-----------------------------------------------+-------+ | Variable_name | Value | +-----------------------------------------------+-------+ | Performance_schema_accounts_lost | 0 | | Performance_schema_cond_classes_lost | 0 | | Performance_schema_cond_instances_lost | 0 | | Performance_schema_digest_lost | 1 | # <-- problem ... ● It’s easy to check the amount of memory used with SHOW ENGINE...: mysql> show engine performance_schema status; ... | performance_schema | events_statements_summary_by_digest.row_size | 1472 | | performance_schema | events_statements_summary_by_digest.row_count | 10000 | | performance_schema | events_statements_summary_by_digest.memory | 14720000 | ... | performance_schema | performance_schema.memory | 421469376 |
  23. 23. www.percona.com What you can find out with default P_S ● What every foreground thread is doing: mysql> select * from performance_schema.events_statements_currentG *************************** 1. row *************************** THREAD_ID: 32 EVENT_ID: 240 END_EVENT_ID: NULL EVENT_NAME: statement/sql/select SOURCE: mysqld.cc:939 TIMER_START: 450538953130416024 TIMER_END: NULL TIMER_WAIT: NULL LOCK_TIME: 18001000000 SQL_TEXT: select * from performance_schema.events_statements_current ... ERRORS: 0 WARNINGS: 0 ROWS_AFFECTED: 0 ROWS_SENT: 0 ROWS_EXAMINED: 0 CREATED_TMP_DISK_TABLES: 0 CREATED_TMP_TABLES: 0 ... NO_INDEX_USED: 1 NO_GOOD_INDEX_USED: 0
  24. 24. www.percona.com What you can find out with default P_S ● What threads are running: mysql> select thread_id, name, processlist_id from performance_schema.threads; +-----------+----------------------------------------+----------------+ | thread_id | name | processlist_id | +-----------+----------------------------------------+----------------+ | 1 | thread/sql/main | NULL | | 2 | thread/innodb/io_handler_thread | NULL | ... | 10 | thread/innodb/io_handler_thread | NULL | | 11 | thread/innodb/io_handler_thread | NULL | | 14 | thread/innodb/srv_lock_timeout_thread | NULL | | 15 | thread/innodb/srv_monitor_thread | NULL | | 16 | thread/innodb/srv_purge_thread | NULL | | 17 | thread/innodb/srv_error_monitor_thread | NULL | | 18 | thread/innodb/srv_master_thread | NULL | | 19 | thread/innodb/page_cleaner_thread | NULL | | 20 | thread/sql/shutdown | NULL | | 21 | thread/sql/con_sockets | NULL | | 32 | thread/sql/one_connection | 11 | +-----------+----------------------------------------+----------------+ 20 rows in set (0.00 sec)
  25. 25. www.percona.com What you can find out with default P_S ● What is in the host cache: mysql> select host, last_error_seen, count_host_acl_errors -> from performance_schema.host_cache; +-------+---------------------+-----------------------+ | host | last_error_seen | count_host_acl_errors | +-------+---------------------+-----------------------+ | pc-PC | 2014-03-02 17:42:09 | 1 | +-------+---------------------+-----------------------+ 1 row in set (0.00 sec) ● What tables are used a lot or caused a lot of waits: mysql> select object_schema, object_name, count_star, sum_timer_wait -> from performance_schema.table_io_waits_summary_by_table -> order by 4 desc limit 5; +---------------+-----------------------+------------+----------------+ | object_schema | object_name | count_star | sum_timer_wait | +---------------+-----------------------+------------+----------------+ | test | t | 19 | 41330631677220 | | test | test2 | 4 | 26094282300 | | test | tt | 4 | 122427090 | | mysql | db | 0 | 0 | | mysql | time_zone_leap_second | 0 | 0 | +---------------+-----------------------+------------+----------------+ 5 rows in set (16.33 sec)
  26. 26. www.percona.com Example 1: slow statement execution # Make sure we have historical events collected about stages mysql> select count(*) from performance_schema.events_stages_history_long; +----------+ | count(*) | +----------+ | 2893 | +----------+ 1 row in set (0.09 sec) # Find query that spent longest time on some stage, can you guess why from the below? mysql> select stg.event_name, stmt.sql_text, -> stg.timer_wait/1000000000000 wait_s -> from performance_schema.events_stages_history_long stg -> join performance_schema.events_statements_history_long stmt -> on (stg.nesting_event_id=stmt.event_id) -> where 1=1 -> /*stg.event_name like '%Sending data%'*/ -> and stmt.rows_examined < 100000 /* stmt.rows_sent < 100000 */ -> and stg.timer_wait > 10*1000000000000 -> order by stg.timer_wait desc limit 1G *************************** 1. row *************************** event_name: stage/sql/updating sql_text: update tt set c1=c1+1 wait_s: 51.0647 1 row in set (0.03 sec)
  27. 27. www.percona.com Example 2: current waits # Make sure we have all waits we care about instrumented mysql> select e.event_name, e.source, e.operation, t.processlist_info -> from performance_schema.events_waits_current e -> join performance_schema.threads t on t.thread_id=e.thread_id -> where type='foreground' and t.processlist_id <> connection_id()G *************************** 1. row *************************** event_name: wait/io/table/sql/handler source: handler.cc:2687 operation: fetch processlist_info: update tt set c1=c1+1 1 row in set (0.08 sec) 2686 MYSQL_TABLE_IO_WAIT(m_psi, PSI_TABLE_FETCH_ROW, MAX_KEY, 0, 2687 { result= rnd_next(buf); }) 2688 DBUG_RETURN(result); mysql> select e.event_name, e.source, e.operation, t.processlist_info -> from performance_schema.events_waits_current e -> join performance_schema.threads t on t.thread_id=e.thread_id -> where type='foreground' and t.processlist_id <> connection_id()G *************************** 1. row *************************** event_name: idle source: mysqld.cc:909 operation: idle processlist_info: NULL 1 row in set (0.15 sec)
  28. 28. www.percona.com Example 3: hot files for disk I/O mysql> select event_name, count_star, sum_timer_wait/1000000000000 wait_s from performance_schema.file_summary_by_event_name order by 3 desc limit 5; +---------------------------+------------+--------+ | event_name | count_star | wait_s | +---------------------------+------------+--------+ | wait/io/file/sql/FRM | 1982 | 4.2914 | | wait/io/file/sql/binlog | 136 | 1.3193 | | wait/io/file/myisam/kfile | 1608 | 0.8984 | | wait/io/file/myisam/dfile | 739 | 0.4980 | | wait/io/file/archive/data | 12 | 0.2637 | +---------------------------+------------+--------+ 5 rows in set (0.01 sec) # This is where to look for file names mysql> select file_name, sum_timer_wait/1000000000000 wait_s from performance_schema.file_summary_by_instance order by 2 desc limit 5; +-----------------------------------------------------------------+--------+ | file_name | wait_s | +-----------------------------------------------------------------+--------+ | C:ProgramDataMySQLMySQL Server 5.6datapc-PC-bin.000013 | 0.2703 | | C:ProgramDataMySQLMySQL Server 5.6datatest#sql-233c_e.frm | 0.2052 | | C:ProgramDataMySQLMySQL Server 5.6datapc-PC-bin.000010 | 0.2002 | | C:ProgramDataMySQLMySQL Server 5.6datapc-PC-bin.000014 | 0.1344 | | C:ProgramDataMySQLMySQL Server 5.6datapc-PC-bin.index | 0.1253 | +-----------------------------------------------------------------+--------+ 5 rows in set (0.00 sec)
  29. 29. www.percona.com Example 4: better PROCESSLIST mysql> select -> processlist_id as Id, processlist_user as User, -> processlist_host as Host, -> processlist_db as db, processlist_command as Command, -> processlist_time as Time, processlist_state as State, -> processlist_info as Info -> from performance_schema.threads -> where type='FOREGROUND'G *************************** 1. row *************************** Id: 1 User: root Host: localhost db: test Command: Query Time: 0 State: Sending data Info: select processlist_id as Id, processlist_user as User, processlist_host as Host, processlist_db as db, processlist_command as Command, processlist_time as Time, processlist_state as State, processlist_info as Info from performance_schema.threads where type='FOREGROUND' 1 row in set (0.00 sec)
  30. 30. www.percona.com Example 5: SHOW PROFILES remake mysql> show warningsG *************************** 1. row *************************** Level: Warning Code: 1287 Message: '@@profiling' is deprecated and will be removed in a future release. 1 row in set (0.06 sec) # we may have no SHOW PROFILES in MySQL 5.7! Get ready: mysql> select event_id as Query_ID, timer_wait/1000000000000 as Duration, sql_text as Query -> from performance_schema.events_statements_history_long statements -> join performance_schema.threads threads -> on threads.thread_id = statements.thread_id -> where threads.processlist_id = connection_id() -> order by Duration desc limit 2G *************************** 1. row *************************** Query_ID: 50 Duration: 0.8202 Query: desc information_schema.schemata *************************** 2. row *************************** Query_ID: 1333 Duration: 0.7237 Query: select event_id as Query_ID, timer_wait/1000000000000 as Duration, sql_text as Query from performance_schema.events_statements_history_long statements join performance_schema.threads threads ... 2 rows in set (0.02 sec)
  31. 31. www.percona.com Still seems complicated?
  32. 32. www.percona.com ps_helper To the Rescue! ● Developed by Mark Leith, MySQL expert @ Oracle ○ worked in MySQL Support team since 2005 ○ senior software development manager @Oracle now (MEM, maybe MEB) ○ good Oracle RDBMS background ● Easier to use comparing to “raw” PERFORMANCE_SCHEMA ● Ready to use collection of views, stored routines and... ● Recently even an easy way to create ps, P_S, PFS or whatever synonym you like! ● git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer... ● Installed in a separate schema (ps_helper) ● IO/latency/waits/statement digests ● Tools to simplify P_S configuration changes, convenient data presentation, and more Links: ● http://www.markleith.co.uk/ps_helper/ ● https://github.com/MarkLeith/dbahelper/archive/master.zip ● https://www.slideshare.net/Leithal/performance-schema-andpshelper
  33. 33. www.percona.com ps_helper: basics Install: openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot < ps_helper_56.sql Check: mysql> show tables; +-------------------------------------------------+ | Tables_in_ps_helper | +-------------------------------------------------+ | _digest_95th_percentile_by_avg_us | | _digest_avg_latency_by_avg_us | | check_lost_instrumentation | ... 53 rows in set (0,00 sec) mysql> select routine_name from information_schema.routines where routine_schema='ps_helper'; +-------------------------------+ | routine_name | +-------------------------------+ | analyze_statement_digest | ... 20 rows in set (0,00 sec)
  34. 34. www.percona.com ps_helper: configuration-related SPs ● reset_to_default(FALSE) - default configuration no matter what is in my.cnf ● save_current_config() - save current configuration in temporary tables ● reload_saved_config() - restore saved configuration ● truncate_all(FALSE) - truncate all the events and summary tables: mysql> select count(*) from performance_schema.events_statements_summary_by_digest; +----------+ | count(*) | +----------+ | 6 | +----------+ 1 row in set (0,00 sec) mysql> call ps_helper.truncate_all(FALSE); Query OK, 0 rows affected (0,03 sec) mysql> select count(*) from performance_schema.events_statements_summary_by_digest; +----------+ | count(*) | +----------+ | 1 | +----------+ 1 row in set (0,00 sec)
  35. 35. www.percona.com ps_helper: slow statement execution ● Let’s assume you’ve noted some slow enough statement: select digest, digest_text from performance_schema.events_statements_summary_by_digestG ... *************************** 16. row *************************** digest: f1a302160f89f76cbe60a86761a8c58d digest_text: INSERT INTO `t1` ( `c1` ) SELECT `c1` FROM `t1` ● You can try to analyze it in details: mysql> call ps_helper.analyze_statement_digest('f1a302160f89f76cbe60a86761a8c58d', 30, 1.0, FALSE, FALSE); ... +------------+-------------+-----------+-----------+---------------+---------------+------------ +------------+ | executions | exec_time | lock_time | rows_sent | rows_affected | rows_examined | tmp_tables | full_scans | +------------+-------------+-----------+-----------+---------------+---------------+------------ +------------+ | 1 | 00:01:08.84 | 428.00 us | 0 | 262144 | 524288 | 1 | 1 | +------------+-------------+-----------+-----------+---------------+---------------+------------ +------------+ 1 row in set (30,36 sec) ...
  36. 36. www.percona.com ps_helper.analyze_statement_digest() ... +-------------------------------------+-------+-----------+ | event_name | count | latency | +-------------------------------------+-------+-----------+ | stage/sql/Sending data | 1 | 3.53 s | | stage/sql/converting HEAP to MyISAM | 1 | 326.79 ms | | stage/sql/query end | 1 | 128.05 ms | | stage/sql/removing tmp table | 1 | 16.78 ms | | stage/sql/init | 2 | 299.73 us | | stage/sql/Creating tmp table | 1 | 109.72 us | | stage/sql/Opening tables | 1 | 104.24 us | | stage/sql/statistics | 1 | 54.66 us | | stage/sql/closing tables | 1 | 51.30 us | | stage/sql/System lock | 1 | 43.07 us | | stage/sql/freeing items | 1 | 42.29 us | | stage/sql/preparing | 1 | 36.15 us | | stage/sql/checking permissions | 2 | 18.98 us | | stage/sql/optimizing | 1 | 14.15 us | | stage/sql/end | 1 | 8.06 us | | stage/sql/executing | 1 | 5.57 us | | stage/sql/cleaning up | 1 | 3.71 us | | stage/sql/logging slow query | 1 | 2.32 us | +-------------------------------------+-------+-----------+ 18 rows in set (30,37 sec) # and more outputs...
  37. 37. www.percona.com ps_helper: hot tables... mysql> select * from ps_helper.schema_table_statistics_with_buffer limit 1G *************************** 1. row *************************** table_schema: test table_name: t1 rows_fetched: 524306 fetch_latency: 20.97 s rows_inserted: 524288 insert_latency: 00:01:47.45 rows_updated: 0 update_latency: 0 ps rows_deleted: 0 delete_latency: 0 ps io_read_requests: 14 io_read: 1.42 KiB io_read_latency: 73.96 us io_write_requests: 7992 io_write: 254.04 MiB io_write_latency: 1.11 s io_misc_requests: 253 io_misc_latency: 20.18 s innodb_buffer_allocated: 107.17 MiB # these are from I_S.INNODB_BUFFER_PAGE innodb_buffer_data: 99.16 MiB innodb_buffer_pages: 6859 innodb_buffer_pages_hashed: 6859 innodb_buffer_pages_old: 6859 innodb_buffer_rows_cached: 473493 1 row in set (2,89 sec)
  38. 38. www.percona.com ps_helper: better PROCESSLIST # smart view to get a lot of useful information easily mysql> SELECT * FROM ps_helper.processlist WHERE conn_id = CONNECTION_ID()G *************************** 1. row *************************** thd_id: 24 conn_id: 5 user: root@localhost db: test command: Query state: Sending data time: 0 current_statement: SELECT * FROM ps_helper.proces ... HERE conn_id = CONNECTION_ID() last_statement: NULL last_statement_latency: NULL lock_latency: 4.98 ms rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 1 tmp_disk_tables: 0 full_scan: YES last_wait: wait/io/file/myisam/dfile last_wait_latency: 4.74 us source: mi_dynrec.c:201 1 row in set (0,03 sec)
  39. 39. www.percona.com ps_helper: why is it useful? Have some real fun: openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper < procedures/create_synonym_db.sql Warning: Using a password on the command line interface can be insecure. mysql> call create_synonym_db('performance_schema', 'ps'); Query OK, 0 rows affected (2,35 sec) mysql> use ps Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +----------------------------------------------------+ | Tables_in_ps | +----------------------------------------------------+ | accounts | ... Use ps_helper.format_time(SUM(sum_timer_wait)): ● Check Bug #70252 to find out what may happen if you don’t use it ● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper
  40. 40. www.percona.com ps_helper: why is it useful? Saves a lot of keystrokes... More fancy views: ● statement_analysis ● statements_with_runtimes_in_95th_percentile ● statements_with_full_table_scans ● statements_with_sorting ● statements_with_temp_tables ● … More fancy stored procedures and functions: ● dump_thread_stack() - allows to produce .dot files → .png or .pdf ● format_statement() ● format_time() ● format_bytes() ● format_path()
  41. 41. www.percona.com PERFORMANCE_SCHEMA - The Good ● Monitoring is available on all platforms supported by MySQL ● Activating the Performance Schema causes no (few) changes in server behavior ○ No persistent on-disk storage ○ Memory allocation at server startup ○ Processing happens mostly at events retrieval ○ No new keywords or threads added ● Overhead is small comparing to some other ways to monitor ● Problems/failures in P_S code should not prevent server from working normally ● It is easy to add new instrumentation points (they say) ● Under active development: ○ Very few bugs - 4 active in MySQL 5.6.16 as of February 22, 2014 ○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!) ● Used by Oracle internally a lot for real things (MEM, performance improvements) ● As it’s based on proper ideas, eventually it’s going to become an ultimate solution for MySQL performance monitoring (and replace everything else)
  42. 42. www.percona.com PERFORMANCE_SCHEMA - The Bad ● Performance overhead in default configuration is notable in some cases (10-30%, see bug reports and 3rd party studies) ● Additional memory use is notable (see Bug #68514): ○ “Why is this not a bug? On my x86-64 with 5.6.12 when performance_schema=1 then the extra RAM is: max_connections * 196kb” - Mark Callaghan ● Mutex and rw-lock waits instrumentation must be enabled at server startup ● How complete is the instrumentation? ● There are bugs and missing parts in the manual ● There are (few) bugs in implementation ● Is this kind of instrumentation really good? (SQL, no trace)
  43. 43. www.percona.com Bugs in PERFORMANCE_SCHEMA Bug #68413 - Facebook: “performance_schema overhead is at least 10%” Bug #71204 - Timing for stages in P_S is different from what SHOW PROFILE reports Bug #71634 - P_S digest looks wrong for system variables, shown as @ @ variable… Bug #71682 - PROCESSLIST_ID for slave sql thread is missing Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with 5.6.16, by the way... Bug #68785 - new features (connection attributes) work well together (or not)... Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in Oracle... Bug #68514 - beware of memory usage impact with P_S
  44. 44. www.percona.com Missing manual: verified bug reports Bug #68097 - Manual does not explain that some P_S instruments must be enabled at startup Bug #71274 - Manual does not provide enough details about background threads in P_S.THREADS Bug #71276 - Manual does not explain PROCESSLIST_STATE in P_S. THREADS properly Bug #71293 - Manual page for P_S.FILE_INSTANCES table does not explain EVENT_NAME values Bug #71303 - Manual page on P_S build configuration does not provide enough details Bug #71304 - Manual does not provide enough details about automatic sizing of P_S parameters These were reported by me and tagged with “missing manual”. There are more...
  45. 45. www.percona.com PERFORMANCE_SCHEMA - The Ugly ● Names: schema name, long table names… you know whom to blame (and thank for ps_helper). I wish we have public synonyms in MySQL, so I’d create V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no? ● Why additional views and procedures (ps_helper) are not there by default (Bug #71207)? - going to be solved with SYS schema ● Some important things and storage engines are not properly instrumented yet, MDL locks, for example… ups, seems fixed in 5.7.3 ● Heuristics to control size (read Bug #68514 again)… less than obvious and I’d prefer more control ● Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS ● Want to know what else users care about? Check feature requests for P_S (11 “Verified” as of November 2, 2013), some of them are limitations “by design”: ○ Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer ○ Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” - Mark Callaghan ○ Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE performance_schema STATUS” - Peter Laursen
  46. 46. www.percona.com PERFORMANCE_SCHEMA in MySQL 5.6 Useful reading (and names to remember): ● http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark Alff on how to configure PERFORMANCE_SCHEMA ● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM ● http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schema-overhead. html - overhead explained and measured by Dimitri Kravtchuk: ● http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typical- workload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013) ● http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql. wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/ ● http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of PERFORMANCE_SCHEMA, by Todd Farmer ● http://www.slideshare.net/SvetaSmirnova/performance-schema-for-mysql-troubleshooting - practical guide from Oracle’s Senior Principal Support engineer Sveta Smirnova
  47. 47. www.percona.com Percona Live 2014, April 1-4, 2014 http://www.percona.com/live/mysql-conference-2014/home Two sessions on PERFORMANCE_SCHEMA there: ● Is performance_schema worth the overhead? ● MySQL 5.7: Performance Schema Improvements ● Many more on MySQL performance in general...
  48. 48. www.percona.com Thank you! Questions and Answers Please, report bugs at: http://bugs.mysql.com Use “Affects Me” button!

×