Performance schema in_my_sql_5.6_pluk2013
Upcoming SlideShare
Loading in...5
×
 

Performance schema in_my_sql_5.6_pluk2013

on

  • 4,046 views

 

Statistics

Views

Total Views
4,046
Views on SlideShare
4,046
Embed Views
0

Actions

Likes
1
Downloads
25
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Performance schema in_my_sql_5.6_pluk2013 Performance schema in_my_sql_5.6_pluk2013 Presentation Transcript

  • PERFORMANCE_SCHEMA in MySQL 5.6: The Good, The Bad and The Ugly Valerii Kravchuk, Principal Support Engineer valerii.kravchuk@percona.com
  • Who am I? Valerii (aka Valeriy) Kravchuk (you may call me “Larry”): ● ● ● ● ● ● MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012 ○ Bugs Verification Team all this time ○ Support issues related to bugs, InnoDB, performance and more ○ Trainings (mostly informal) for new team members ○ All kinds of decision making committees… Principal Support Engineer in Percona, 2012 - ... ○ Doing more or less the same as before, but better (I hope)... ○ Plus I speak and write about MySQL in public now! http://mysqlentomologist.blogspot.com - my blog about MySQL bugs https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about MySQL (mostly bugs…) http://bugs.mysql.com - my personal playground Yes, I am Orthodox, in many senses... www.percona.com
  • What is this session about? “The Good, the Bad and the Ugly (Italian title: Il buono, il brutto, il cattivo) is a 1966 Italian epic Spaghetti western film directed by Sergio Leone, starring Clint Eastwood, Lee Van Cleef, and Eli Wallach in the title roles respectively. ... It is the third film in the Dollars Trilogy following A Fistful of Dollars (1964) and For a Few Dollars More (1965). The plot revolves around three gunslingers competing to find a fortune in buried Confederate gold amid the violent chaos of gunfights, hangings, American Civil War battles and prison camps.” http://en.wikipedia.org/wiki/The_Good,_the_Bad_and_the_Ugly www.percona.com View slide
  • So, what is this session really about? ● ● ● ● ● How to find the root causes of MySQL server performance problems? ○ http://method-r.com/faq/35-what-is-method-r ○ Where time is spent? Proper instrumentation of the code, is it really important? ○ http://tkyte.blogspot.com/2005/06/instrumentation.html ○ http://en.wikipedia.org/wiki/Observer_effect_%28physics%29 OS-level debug, trace and profiling tools, aren’t they allow to pinpoint any performance problem in MySQL? Instrumentation in MySQL - PERFORMANCE_SCHEMA ○ History and engineers involved ○ Features in MySQL 5.6 ○ How to use it? ○ Any alternatives in MySQL world? ○ Sources of additional information and tools The Good, The Bad and The Ugly in PERFORMANCE_SCHEMA www.percona.com View slide
  • Method R - common sense approach 1. Identify the most important task (the bottleneck) 2. Measure its response time (R) in detail 3. Optimize that response time in the most (economically) efficient way 4. Repeat until your system is (economically) optimal Percona also tries to apply common sense: http://www.mysqlperformanceblog.com/2008/11/24/how-percona-does-a-mysql-performance-audit/ http://www.mysqlperformanceblog.com/2011/07/27/whats-required-to-tune-mysql/ http://www.mysqlperformanceblog.com/2011/05/05/the-two-even-more-fundamental-performance-metrics/ www.percona.com
  • Instrumentation “Code instrumentation, what is it? To me, it is the fine art of making approximately every other line of your developed code “debug” of some sort. Trace information. With timestamps. And meaningful information.” - Tom Kyte “In science, the term observer effect refers to changes that the act of observation will make on a phenomenon being observed. This is often the result of instruments that, by necessity, alter the state of what they measure in some manner.” - Wikipedia www.percona.com
  • Trace files and debug printouts? ● ● ● ● Readily available in MySQL for years... ○ http://dev.mysql.com/doc/refman/5.6/en/making-trace-files.html ○ http://dev.mysql.com/doc/refman/5.6/en/dbug-package.html ○ … --debug=d:t:i:o,/tmp/mysqld.trace ○ -debug binaries only Additional tracing features are available in MySQL 5.6 for non-debug binaries also (for optimizer decision making, for example) ○ http://dev.mysql.com/doc/internals/en/tracing-example.html Still far behind Oracle’s SQL Trace and trace events in general ○ http://www.juliandyke.com/Diagnostics/Events/EventReference.html Let’s try to produce trace while test case for Bug #68079 is running: mysql> set session debug='d:t:i:o,/tmp/mysqld.trace'; Query OK, 0 rows affected (0.00 sec) www.percona.com
  • Content of the trace file... ... T@9 : | | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60. 2/sql/sql_parse.cc:5383 T@9 : | | | | >PROFILING::status_change T@9 : | | | | <PROFILING::status_change 374 T@9 : | | | | >close_thread_tables T@9 : | | | | <close_thread_tables 1469 T@9 : | | | | >stmt_causes_implicit_commit T@9 : | | | | <stmt_causes_implicit_commit 279 T@9 : | | | | >MDL_context::release_transactional_locks T@9 : | | | | | >MDL_context::release_locks_stored_before T@9 : | | | | | <MDL_context::release_locks_stored_before 2753 T@9 : | | | | | >MDL_context::release_locks_stored_before T@9 : | | | | | <MDL_context::release_locks_stored_before 2753 T@9 : | | | | <MDL_context::release_transactional_locks 2908 T@9 : | | | <mysql_execute_command 5431 T@9 : | | | >~opt_trace_start T@9 : | | | <~opt_trace_start 230 T@9 : | | | THD::enter_stage: /home/openxs/bzr2/percona-5.6/Percona-Server-5.6.10-alpha60. 2/sql/sql_parse.cc:6676 T@9 : | | | >PROFILING::status_change T@9 : | | | <PROFILING::status_change 374 ... ● ● ● Debug printouts/traces/assertions/error log, is this enough? Think about concurrency! Too intrusive, changes behavior, serializes and wastes resources... www.percona.com
  • Example of the problem to solve: Bug #68079 http://bugs.mysql.com/bug.php?id=68079 CREATE TABLE `incident` ( `sys_id` char(32) NOT NULL DEFAULT '', `category` varchar(40) DEFAULT NULL, PRIMARY KEY (`sys_id`), KEY `incident_category` (`category`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; insert into incident(sys_id) values (rand()*1000000); insert into incident(sys_id) select rand()*1000000 from incident; -- 13 times CREATE TABLE `task` ( `sys_id` char(32) NOT NULL DEFAULT '', `u_root_cause` char(32) DEFAULT NULL, `u_business_impact_description` mediumtext, `u_business_impact_category` mediumtext, PRIMARY KEY (`sys_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; insert insert insert update into task(sys_id) values(rand()*1000000); into task(sys_id) select rand()*1000000 from task; -- 13 times into task(sys_id) select sys_id from incident limit 100; incident set category=rand()*100000; www.percona.com
  • Example of the problem to solve: Bug #68079 These queries scale well: select count(*), category from incident group by category; select * from task where sys_id=rand()*1000000; This one does not scale even to 10 threads on 12 cores: select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category; mysql> explain select count(*), category from task inner join incident on task. sys_id=incident.sys_id group by incident.category; | 1 | SIMPLE | NULL | 1 | SIMPLE | test.incident.sys_id incident | index | PRIMARY | 8210 | Using index | task | eq_ref | PRIMARY | 1 | Using index | | incident_category | 123 | | PRIMARY | | 96 www.percona.com
  • Example of the problem to solve: Bug #68079 mysqlslap -uroot --iterations=10 --concurrency=N --create-schema=test --no-drop --numberof-queries=1000 --query='select count(*), category from task inner join incident on task. sys_id=incident.sys_id group by incident.category' where N in (1, 2, 4, 8, 10, 12, 24 … 2*number_of_cores … 1024 … 8192) Results: Benchmark Average number of Minimum number of Maximum number of Number of clients Average number of … Benchmark Average number of Minimum number of Maximum number of Number of clients Average number of seconds seconds seconds running queries to run all queries: 8.512 seconds to run all queries: 8.248 seconds to run all queries: 8.844 seconds queries: 8 per client: 125 seconds seconds seconds running queries to run all queries: 9.399 seconds to run all queries: 9.137 seconds to run all queries: 9.749 seconds queries: 10 per client: 100 www.percona.com
  • Example of the problem to solve: Bug #68079 Simple (?) join does NOT scale even to 10 threads... ● Why does this happen? ● Where time is spent? ● Is there any way to tune MySQL server to improve scalability for this query? ● Maybe we need different hardware? ● Any changes at schema level that may help? ● How to get information to answer the questions above? ● This is a performance tuning task to check tools, approaches and code improvement ideas! www.percona.com
  • Sources of information on performance What sources of information and tools we can try to use? ● ● ● ● ● ● ● ● ● ● ● ● show engine innodb statusG show engine innodb mutex; InnoDB-related tables in the INFORMATION_SCHEMA (?) show global status like ‘innodb%’ (?) userstat (Percona Server and others) (?) show profiles Poor man’s profiler (pt-pmp) perf oprofile DTrace (?) Maybe… PERFORMANCE_SCHEMA? Discussion of the need for PERFORMANCE_SCHEMA and alternative approaches - hence all the above www.percona.com
  • SHOW ENGINE INNODB STATUS ● Do you know how to read that? ○ ○ ○ ● http://dev.mysql.com/doc/refman/5.6/en/innodb-monitors.html - fine manual http://www.mysqlperformanceblog.com/2006/07/17/show-innodb-status-walk-through/ - the best thing ever since Worcestershire sauce http://www.percona.com/doc/percona-server/5.6/diagnostics/innodb_show_status.html - just better in Percona Server Not very useful for Bug #68079 on my QuadCore: Per second averages calculated from the last 36 seconds ----------------BACKGROUND THREAD ----------------srv_master_thread loops: 121 srv_active, 0 srv_shutdown, 2473 srv_idle srv_master_thread log flush and writes: 2594 ---------SEMAPHORES ---------OS WAIT ARRAY INFO: reservation count 9477 OS WAIT ARRAY INFO: signal count 3703 Mutex spin waits 28946906, rounds 25108917, OS waits 8475 RW-shared spins 209, rounds 5999, OS waits 182 RW-excl spins 0, rounds 9, OS waits 0 Spin rounds per wait: 0.87 mutex, 28.70 RW-shared, 9.00 RW-excl www.percona.com
  • SHOW ENGINE INNODB STATUS ... Purge done for trx's n:o < 2384 undo n:o < 0 state: running but idle History list length 20 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 143558, not started MySQL thread id 1, OS thread handle 0x7f01cc057700, query id 10678 localhost msandbox init show engine innodb status ... Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 285 OS file reads, 49 OS file writes, 49 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.06 writes/s, 0.06 fsyncs/s Hash table size 276671, node heap has 0 buffer(s) 0.00 hash searches/s, 142759.09 non-hash searches/s ... Log sequence number 4732500 ... Last checkpoint at 4732480 0 pending log writes, 0 pending chkp writes 50 log i/o's done, 0.06 log i/o's/second ... Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 ... Main thread process no. 5202, id 139645286401792, state: sleeping Number of rows inserted 0, updated 0, deleted 0, read 89167561 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 144695.09 reads/s www.percona.com
  • SHOW ENGINE INNODB MUTEX ● ● http://dev.mysql.com/doc/refman/5.6/en/show-engine.html -debug builds give more information ○ count - how many times the mutex was requested ○ spin_waits - how many times the spinlock had to run ○ spin_rounds - the number of spinlock rounds ○ os_waits - indicates the number of operating system waits. The mutex was not unlocked during the spinlock and it was necessary to yield to the operating system and wait. You have only this value in nondebug binaries. ○ os_wait_times - indicates the amount of time (in ms) spent in operating system waits, if the timed_mutexes system variable is 1 (ON). If timed_mutexes is 0 (OFF, default), timing is disabled, so os_wait_times is 0. | InnoDB | rw_lock_mutexes | count=0, spin_waits=0, spin_rounds=0, os_waits=0, os_yields=0, os_wait_times=0 | www.percona.com
  • SHOW ENGINE INNODB MUTEX ● Looks like we have something immediately useful here for Bug #68079: mysql [localhost] {msandbox} (test) > show engine innodb mutex; +--------+-------------------------+----------------+ | Type | Name | Status | +--------+-------------------------+----------------+ | InnoDB | log0log.cc:737 | os_waits=2 | | InnoDB | buf0buf.cc:1242 | os_waits=2 | | InnoDB | combined buf0buf.cc:992 | os_waits=66475 | | InnoDB | dict0dict.cc:896 | os_waits=1 | | InnoDB | log0log.cc:799 | os_waits=16 | | InnoDB | combined buf0buf.cc:993 | os_waits=1 | +--------+-------------------------+----------------+ 6 rows in set (0.02 sec) vi +992 /home/openxs/bzr2/mysql-5.6/storage/innobase/buf/buf0buf.cc mutex_create(PFS_NOT_INSTRUMENTED, &block->mutex, SYNC_BUF_BLOCK); ● ● Hardly is maintained properly in MySQL 5.6. Check my Bug #70819 Percona Server shows mutex name, not file and line number… | InnoDB | &buf_pool->LRU_list_mutex | InnoDB | &fil_system->mutex | InnoDB | combined &block->mutex | os_waits=13 | os_waits=5073 | os_waits=162396 www.percona.com
  • INFORMATION_SCHEMA? InnoDB-related tables in the INFORMATION_SCHEMA: ● ● ● http://dev.mysql.com/doc/refman/5.6/en/innodb-i_s-tables.html - there are many... But as we know from INNODB STATUS it’s not about I/O or locks, it seems they do not add much useful for the case of Bug #68079… If only INNODB_METRICS - http://dev.mysql.com/doc/refman/5.6/en/innodbmetrics-table.html: | NAME | SUBSYSTEM | COUNT | MAX_COUNT | MIN_COUNT | AVG_COUNT ... | STATUS | TYPE | COMMENT | | | | | | varchar(193) varchar(193) bigint(21) bigint(21) bigint(21) double | | | | | | NO NO NO YES YES YES | varchar(193) | NO | varchar(193) | NO | varchar(193) | NO | | | | | | | | | | | | | | | | | | 0 NULL NULL NULL | | | | | | | | | | | | | | | | | | www.percona.com
  • INNODB_METRICS... mysql> select name, count from information_schema.innodb_metrics where status='enabled' and type='status_counter'; +---------------------------------+-----------+ | name | count | +---------------------------------+-----------+ ... | buffer_pool_reads | 271 | | buffer_pool_read_requests | 336246061 | ... | buffer_pages_read | 270 | | buffer_data_reads | 6623232 | | buffer_data_written | 141312 | | os_data_reads | 285 | | os_data_writes | 89 | | os_data_fsyncs | 89 | | os_log_bytes_written | 40960 | | os_log_fsyncs | 83 | ... | innodb_rwlock_s_spin_waits | 210 | | innodb_rwlock_x_spin_waits | 0 | | innodb_rwlock_s_spin_rounds | 6029 | | innodb_rwlock_x_spin_rounds | 9 | | innodb_rwlock_s_os_waits | 183 | | innodb_rwlock_x_os_waits | 0 | ... www.percona.com
  • SHOW GLOBAL STATUS LIKE ‘innodb%’ ● ● pt-mext helps - http://www.percona.com/doc/percona-toolkit/2.1/pt-mext.html Let’s monitor the counters (51 starts with Innodb_) over 10 seconds: [openxs@chief msb_5_6_14]$ pt-mext -r -- /home/openxs/5.6.14/bin/mysqladmin -socket=/tmp/mysql_sandbox5614.sock -uroot ext -i10 -c3 >/tmp/mext.txt Innodb_buffer_pool_read_requests ... Innodb_buffer_pool_write_requests Innodb_data_fsyncs ... Innodb_data_read Innodb_data_reads Innodb_data_writes ... Innodb_data_written ... Innodb_log_writes Innodb_num_open_files Innodb_os_log_fsyncs ... Innodb_rows_read 14726570 686556718 160 176 3 3 6623232 285 176 0 0 3 250368 1536 160 7 166 342861007 3 0 3 7353994 www.percona.com
  • Percona Server and userstat? http://www.percona.com/doc/percona-server/5.6/diagnostics/user_stats.html: ● ● ● ● ● ● Introduced in Google patch Additional tables in INFORMATION_SCHEMA: ○ CLIENT_STATISTICS ○ THREAD_STATISTICS ○ USER_STATISTICS ○ TABLE_STATISTICS ○ INDEX_STATISTICS Easy to configure: ○ http://dom.as/2013/04/17/on-performance-schemas/ Low performance impact we say: ○ http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-user_statistics/ Hardly this can help with mutexes or waits… But some says it adds overhead: ○ “and then, once userstat=1 the regression is horrible.. over 30%... “ - guess who says that and why? www.percona.com
  • SHOW PROFILE ● ● ● http://bugs.mysql.com/bug.php?id=24795 - SHOW PROFILE contribution by Jeremy Cole (MySQL 5.0) http://dev.mysql.com/doc/refman/5.6/en/show-profile.html - details… Let’s try to profile the query while test for Bug #68079 is running: mysql [localhost] {msandbox} (test) > set profiling=1; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql [localhost] {msandbox} (test) > show warningsG *************************** 1. row *************************** Level: Warning Code: 1287 Message: '@@profiling' is deprecated and will be removed in a future release. 1 row in set (0.00 sec) mysql [localhost] {msandbox} (test) > select count(*), category from task inner join incident on task.sys_id=incident.sys_id group by incident.category; ... 103 rows in set (0.08 sec) mysql [localhost] {msandbox} (test) > show profiles; mysql [localhost] {msandbox} (test) > show profile for query 3; www.percona.com
  • SHOW PROFILE ● Profile of the query while test for Bug #68079 is running: | starting | 0.000063 | | checking permissions | 0.000004 | | checking permissions | 0.000004 | | Opening tables | 0.000092 | | init | 0.000023 | | System lock | 0.000007 | | optimizing | 0.000009 | | statistics | 0.000037 | | preparing | 0.000015 | | Creating tmp table | 0.000023 | | Sorting result | 0.000003 | | executing | 0.000002 | | Sending data | 0.081722 | | Creating sort index | 0.000141 | | end | 0.000005 | | removing tmp table | 0.000008 | | end | 0.000004 | | query end | 0.000008 | | closing tables | 0.000013 | | freeing items | 0.000553 | | cleaning up | 0.000017 | +----------------------+----------+ 21 rows in set, 1 warning (0.00 sec) www.percona.com
  • Poor man’s profiler (gdb tricks) ● ● ● ● http://poormansprofiler.org/ http://www.percona.com/doc/percona-toolkit/2.1/pt-pmp.html http://www.mysqlperformanceblog.com/2011/12/02/three-ways-that-thepoor-mans-profiler-can-hurt-mysql/ ○ mysqld freezes for the duration of the process ○ They say mysqld can crash (?) ○ They say mysqld can be left in an unstable state (?) I’ve tried recently on QuadCore with test from bug #68079, not impressed: 10 libaio::??,os_aio_linux_collect,os_aio_linux_handle,fil_aio_wait,io_handler_thread,start_thread,clone 2 __memcpy_ssse3,store_key_field::copy_inner,copy,cp_buffer_from_ref,cmp_buffer_with_ref,join_read_key, sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_execute_select,mysql_select, handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command, do_handle_one_connection,handle_one_connection, pfs_spawn_thread,start_thread,clone 1 update_ref_and_keys,make_join_statistics,JOIN::optimize,mysql_execute_select,mysql_select, handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command, do_handle_one_connection,handle_one_connection, pfs_spawn_thread,start_thread,clone www.percona.com
  • Poor man’s profiler - Domas rulez! ● ● ● ● So, I was not impressed (found nothing enlightening), but it depends: ○ “Usually pmp (poor man's profiler, making stacktrace snapshots) producing more relevant information than performance schema reports.” Google for something like site:bugs.mysql.com Mark Callaghan pmp: ○ http://bugs.mysql.com/bug.php?id=58037 - fixed in 5.5+ with MDL ○ http://bugs.mysql.com/bug.php?id=57021 - it was used to find a problem in Facebook’s patch… ○ http://bugs.mysql.com/bug.php?id=61545 - still “Verified”, commit stalls when mysql_binlog_send does disk read If you are on Linux and don’t run MySQL with real profiler, try use pt-pmp when server stalls, hangs or seems overloaded and unusable Key to success for all kinds of profilers - open source of MySQL server! www.percona.com
  • Let’s try real profiler - perf ● ● ● https://perf.wiki.kernel.org/index.php/Main_Page: ○ “Among others, it provides per task, per CPU and per-workload counters, sampling on top of these and source code event annotation” http://bugs.mysql.com/bug.php?id=69236 - Single-threaded workloads can be much slower on 5.6 than on 4.0. perf was used to prove that! Sample session while test for Bug #68079 was running: [openxs@chief ~]$ perf record -p 3310 sleep 20 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 2.006 MB perf.data (~87654 samples) ] [openxs@chief ~]$ perf report ... 12.76% mysqld mysqld 12.32% mysqld mysqld 10.39% mysqld mysqld [.] btr_cur_search_to_nth_level(d [.] buf_page_get_gen(unsigned lon [.] my_strnncollsp_utf8 www.percona.com
  • perf output continued... ● Sample session while test for Bug #68079 was running: ... 7.28% 6.68% 5.76% 4.56% 4.22% 2.87% 2.79% 2.68% 2.33% 1.97% 1.86% 1.83% 1.81% mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld mysqld [.] [.] [.] [.] [.] [.] [.] [.] [.] [.] [.] [.] [.] rec_get_offsets_func(unsigned mtr_memo_slot_release_func(mt page_cur_search_with_match(bu cmp_dtuple_rec_with_match_low row_search_for_mysql(unsigned ut_delay(unsigned long) pfs_mutex_enter_func(ib_mutex innobase_mysql_cmp(int, unsig cmp_whole_field(unsigned long row_mysql_store_col_in_innoba get_internal_charset mutex_spin_wait(ib_mutex_t*, pfs_mutex_exit_func(ib_mutex_ ... ● ● This was on QuadCore with 4 concurrent threads, MySQL 5.6.14 Now tell me that PERFORMANCE_SCHEMA impact with default settings is low on MySQL 5.6... www.percona.com
  • Let’s try another profiler - oprofile ● ● http://www.fromdual.ch/mysql-oprofile: ○ Nice summary from Oli on how to use it! Sample session while test for Bug #68079 was running: [root@chief openxs]# opreport --demangle=smart --symbols --long-filenames --merge tgid /home/openxs/5.6.14/bin/mysqld | head -n 20 ... Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 samples % image name symbol name 1208437 12.5180 /home/openxs/5.6.14/bin/mysqld btr_cur_search_to_nth_level (dict_index_t*, unsigned long, dtuple_t const*, unsigned long, unsigned long, btr_cur_t*, unsigned long, char const*, unsigned long, mtr_t*) 1151258 11.9257 /home/openxs/5.6.14/bin/mysqld buf_page_get_gen(unsigned long, unsigned long, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned long, mtr_t*) 1115969 11.5601 /home/openxs/5.6.14/bin/mysqld my_strnncollsp_utf8 ... 242608 2.5131 /home/openxs/5.6.14/bin/mysqld pfs_mutex_enter_func(ib_mutex_t*, char const*, unsigned long) ... 142685 1.4780 /lib64/libc-2.13.so __memcpy_ssse3 132334 1.3708 /home/openxs/5.6.14/bin/mysqld pfs_mutex_exit_func(ib_mutex_t*) ... www.percona.com
  • DTrace? One day, maybe... ● I wish I remember how to use it and have MacOS X or Solaris, or FreeBSD 9.x (on some good hardware) at hand… ● but I don’t. So, I had not tried to use DTrace to find out what’s wrong with that Bug #68079. ● Some useful references, still, if you ever plan to try: ○ ○ http://dom.as/tag/dtrace/ - check what Domas used to do with DTrace ○ http://cdn.oreillystatic.com/en/assets/1/event/21/Deep-inspecting%20MySQL%20with% 20DTrace%20Presentation.pdf - from one of User Conferences... ○ ● http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html - it’s documented in the manual, including static probes added http://dev.mysql.com/tech-resources/articles/getting_started_dtrace_saha.html - “native” Sun engineers also contributed I really miss Sun... www.percona.com
  • Time to try PERFORMANCE_SCHEMA ● ● ● ● We know functions where most of the time was spent (profiling) We know that some mutex waits were long enough to force OS wait (switch to other thread) We need more details, specifically time spent waiting… So, time to use PERFORMANCE_SCHEMA it seems Sample output while test for Bug #68079 was running, naive attempt (the time is in picoseconds): mysql> select event_name, count_star, sum_timer_wait from performance_schema. events_waits_summary_global_by_event_name where event_name like 'wait/synch/mutex/innodb%' and count_star > 0 order by sum_timer_wait desc limit 5; +--------------------------------------------+------------+----------------+ | event_name | count_star | sum_timer_wait | +--------------------------------------------+------------+----------------+ | wait/synch/mutex/innodb/os_mutex | 71754726 | 11380485535846 | | wait/synch/mutex/innodb/trx_sys_mutex | 60057 | 4191727883 | | wait/synch/mutex/innodb/buf_pool_mutex | 8655 | 468303388 | | wait/synch/mutex/innodb/mutex_list_mutex | 10310 | 427385546 | | wait/synch/mutex/innodb/rw_lock_list_mutex | 8304 | 316946749 | +--------------------------------------------+------------+----------------+ 5 rows in set (0.00 sec) www.percona.com
  • Basic ideas and terms PERFORMANCE_SCHEMA (aka PS, P_S, PFS): ● Inspired by Oracle Wait Interface ● Separate database and storage engine ● Records run time statistics via built-in instrumentation points ● Designed to let us get the answers to the following questions: ○ Where was time spent? ○ Who has used most of resources/time? ○ What was executed to to use these? ○ When it was executed? ● Appeared in MySQL 5.5, improved and ON by default in MySQL 5.6, and is documented and explained in details already: ○ http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html ● Implemented by Marc Alff (http://marcalff.blogspot.com/) ● Deal with instruments (including wait events), consumers, objects (tables for now) that are monitored, threads (some of them background) that run and actors (users) that execute SQL statements (in stages). www.percona.com
  • So, how to use it? ● ● The amount of information in the manual is huge, it’s not easy to start using P_S ○ I wish it would be as easy as setting TIMED_STATISTICS to TRUE with ALTER SYSTEM + few queries to V$SESSTATS, V$SYSSTATS or V$SESSION_WAIT Marc Alff explains in http://marcalff.blogspot.com/2013/04/on-configuringperformance-schema.html: ○ “Define your goals” - this is clear, we have waits and we need to find out what waits are most important based on time wasted (or waited) ○ “Define what to instrument” - we need to instrument InnoDB waits mutex waits and internal lock waits, but how to do this? Details next, but: ■ "wait/synch/mutex/%", for mutexes ■ "wait/synch/rwlock/%", for read write locks ○ “Define how much detail to collect” - well, we need counts and time spent for all InnoDB-related waits ○ “Provide sizing data” - autosizing based on max_connections? | performance_schema_max_mutex_instances | performance_schema_max_rwlock_instances ○ | 15600 | | 9000 | “Monitor sizing problems” - memory used vs dropped statistics and impact www.percona.com
  • How instruments are named? ● ● For those who want solutions, not complains or comments... http://dev.mysql.com/doc/refman/5.6/en/performance-schema-instrumentnaming.html: ○ "wait/io/file/%" - file io ○ "wait/io/socket/%" - network io ○ "wait/io/table/%" - table io ○ "wait/lock/table/%" - table locks ○ "wait/synch/cond/%" - conditions ○ "wait/synch/mutex/%" - mutexes ○ "wait/synch/rwlock/%" - read write locks ○ "stage/%" - stages of statement execution (see show profile) ○ "idle" - clients idle time. ○ "statement/com/%" - commands in the client/server protocol ○ "statement/sql/%" - SQL queries (and some more statement/…) Hint: bold means “new in MySQL 5.6”, blue color means enabled by default www.percona.com
  • Setup for Bug #68079 (waits/…) ● ● http://mysqlentomologist.blogspot.com/2013/01/how-to-use-performanceschemato-check.html - my experience back at MySQL 5.6 RC stage... To set up InnoDB mutex waits monitoring via PERFORMANCE_SCHEMA in general with MySQL 5.6.x you should do at least the following: 1. Start MySQL server with all mutex related instruments enabled at startup (performance_schema=ON by default on recent 5.6.x), like this: mysqld_safe --performance_schema_instrument='wait/synch/mutex/innodb/%=on' & 2. Connect to server and set up proper consumers explicitly, like this: UPDATE performance_schema.setup_consumers SET enabled = 'YES' WHERE name like 'events_waits%'; 3. 4. Run your problematic load. Check waits using whatever tables you need www.percona.com
  • Summary of my setup experience... ● ● ● ● It's great to have Oracle engineers reading your Facebook posts and replying (especially in a real time). Thank you, Mark Leith! PERFORMANCE_SCHEMA in MySQL 5.6 can be used to study InnoDB mutex waits MySQL 5.6 manual is not clear enough about steps and details on instruments involved to do the study explained above http://dev.mysql.com/doc/refman/5.6/en/performance-schema-filtering.html: “Modifications to any of these tables affect monitoring immediately, with the exception of setup_actors. Modifications to setup_actors affect only foreground threads created thereafter.” ● ● Now compare to reality, complain in public, report a bug: ○ http://bugs.mysql.com/bug.php?id=68097 - “Manual does not explain that some P_S instruments must be enabled at startup” Get a reply from Dimitri Kravtchuk: “All instrumentation is dynamic EXCEPT mutexes which are created mainly on the MySQL server startup, so to get their instrumentation initialized you have to enable it on startup” www.percona.com
  • Not that clear and easy to use? www.percona.com
  • ps_helper To the Rescue! ps_helper: ● ● ● ● ● ● ● Developed by Mark Leith, MySQL expert @ Oracle ○ worked in MySQL Support team since 2005 ○ senior software development manager @Oracle now (MEM, maybe MEB) ○ good Oracle RDBMS background Easier to use comparing to “raw” PERFORMANCE_SCHEMA Ready to use collection of views, stored routines and... Recently even an easy way to create ps, P_S, PFS or whatever synonym you like! git clone https://github.com/MarkLeith/dbahelper.git dbahelper - if you prefer... Installed in a separate schema (ps_helper) IO/latency/waits/statement digests Links: ● http://www.markleith.co.uk/ps_helper/ ● https://github.com/MarkLeith/dbahelper/archive/master.zip ● https://www.slideshare.net/Leithal/performance-schema-andpshelper www.percona.com
  • ps_helper: basics Install: openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot test < ps_helper_56.sql Check: mysql> show tables; +-------------------------------------------------+ | Tables_in_ps_helper | +-------------------------------------------------+ | _digest_95th_percentile_by_avg_us | | _digest_avg_latency_by_avg_us | | check_lost_instrumentation | ... 53 rows in set (0,00 sec) mysql> select routine_name from information_schema.routines where routine_schema='ps_helper'; +-------------------------------+ | routine_name | +-------------------------------+ | analyze_statement_digest | ... 20 rows in set (0,00 sec) www.percona.com
  • ps_helper: why is it useful? Have some real fun: openxs@ao756:/tmp/ps_helper/dbahelper-master$ mysql -uroot -proot ps_helper < procedures/create_synonym_db.sql Warning: Using a password on the command line interface can be insecure. mysql> call create_synonym_db('performance_schema', 'ps'); Query OK, 0 rows affected (2,35 sec) mysql> use ps Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +----------------------------------------------------+ | Tables_in_ps | +----------------------------------------------------+ | accounts | ... Use ps_helper.format_time(SUM(sum_timer_wait)): ● Check Bug #70252 to find out what may happen if you don’t use it ● Check https://www.slideshare.net/Leithal/performance-schema-andpshelper www.percona.com
  • Some history behind... ● “The Good, The Bad and The Ugly”: “The film's title has entered the English language as an idiomatic expression. Typically used when describing something thoroughly, the respective phrases refer to upsides, downsides and the parts that could, or should have been done better, but were not.” - Wikipedia ● “For Three Men The Civil War Wasn’t Hell. It Was Practice!” ○ ○ Guess who are these men, in case of PERFORMANCE_SCHEMA… Maybe more than three… ● Read the WorkLogs: ○ ○ ○ http://dev.mysql.com/worklog/task/?id=2360 - “Why we're doing this” http://dev.mysql.com/worklog/task/?id=4674 - “Setup for Actors” http://dev.mysql.com/worklog/task/?id=4513 - “Implement it…” ● Where the name comes from? “Proposals for the schema name were DYNAMIC (but that is a reserved word in standard SQL), PERFORMANCE_SCHEMA (Jim Winstead's suggestion), PERFORMANCE (Peter's preference), and DBTrace because it sounds like DTrace. There was a vote (see dev-private thread "WL#2360 Vote: schema name and table name"). The winner was PERFORMANCE_SCHEMA.” - Peter Gulutzan www.percona.com
  • PERFORMANCE_SCHEMA in MySQL 5.6 Useful reading (and names to remember): ● http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html - article from Mark Alff on how to configure PERFORMANCE_SCHEMA ● http://dev.mysql.com/doc/refman/5.6/en/performance-schema-configuration.html - RTFM ● https://www.facebook.com/notes/mysql-at-facebook/my-mysql-is-faster-than-yourmysql/10151250402570933 - influence of P_S in 5.6.10 and... 5.1.63 or 5.1.52+FB patch are still fastest in many cases ● http://dimitrik.free.fr/blog/archives/2013/07/mysql-performance-why-performance-schemaoverhead.html - overhead explained and measured by Dimitri Kravtchuk: ● http://www.percona.com/webinars/using-mysql-56-performance-schema-troubleshoot-typicalworkload-bottlenecks - webinar by Peter Zaitsev (May 15, 2013) ● http://mysql.wisborg.dk/category/mysql/performance_schema/ - useful posts from Jesper Krogh on the topic. He had created yet another set of tools to make out life easier, ps_tools: http://mysql. wisborg.dk/2013/10/10/mysql-connect-2013-ps_tools/ ● http://mysqlblog.fivefarmers.com/tag/performance_schema/ - practical use of PERFORMANCE_SCHEMA, by Todd Farmer www.percona.com
  • PERFORMANCE_SCHEMA - The Good ● ● ● ● ● ● ● ● Monitoring is available on all platforms supported by MySQL Activating the Performance Schema causes no (few) changes in server behavior ○ No persistent on-disk storage ○ Memory allocation at server startup ○ Processing happens mostly at events retrieval ○ No new keywords or threads added Overhead is small comparing to some other ways to monitor Problems/failures in P_S code should not prevent server from working normally It is easy to add new instrumentation points (they say) Under active development: ○ Very few bugs - 4 active in MySQL 5.6 (of 47) as of November 1, 2013 ○ Bug #70018 (PFS Overhead on frequent connect/disconnect) fixed in 5.6.15! ○ Even more features in MySQL 5.7: check Bug #69527 (MDL locks!) Used by Oracle internally a lot for real things (MEM, performance improvements) As it’s based on proper ideas, eventually it’s going to become an ultimate solution for MySQL performance monitoring (and replace everything else) www.percona.com
  • Let me produce some flame... www.percona.com
  • PERFORMANCE_SCHEMA - The Bad ● Performance overhead in default configuration is notable in some cases (10-30%, see bug reports and 3rd party studies) ● Additional memory use is notable (see Bug #68514): ○ “Why is this not a bug? On my x86-64 with 5.6.12 when performance_schema=1 then the extra RAM is: max_connections * 196kb” - Mark Callaghan ● Mutex and rw-lock waits instrumentation must be enabled at server startup ● How complete is the instrumentation? ● There are bugs and missing parts in the manual ● There are bugs in implementation ● Is this kind of instrumentation really good? (SQL, no trace) ● Any critical posts and even some bug reports are sometimes considered as a kind of a personal insult: "godfather" case... www.percona.com
  • Bugs in P_S and fine manual Bug #68413 - Facebook: “performance_schema overhead is at least 10%” Bug #69782 - Old files not being removed from performance_schema.file_instances Bug #70628 - Wrong instrumentation interface for mysql_cond_timedwait Bug #70697 - Probably not a bug, but indication that manual for SHOW ENGINE PERFORMANCE_SCHEMA STATUS is still not clear Bug #68574 - “No instrumentation for InnoDB files in P_S on windows” - same with 5.6.14, by the way... Bug #68785 - new features (connection attributes) work well together (or not)... Bug #66589 - still not fixed (do not use mysqlhotcopy). One of the last bugs I verified in Oracle... Bug #68514 - beware of memory usage impact with P_S Bug #68097 - still some details to document in the manual clearly www.percona.com
  • PERFORMANCE_SCHEMA - The Ugly ● ● ● ● ● ● Names: schema name, long table names… you know whom to blame (and thank for ps_helper). I wish we have public synonyms in MySQL, so I’d create V$SYSSTATS (Bug #68855). You can create views, but that’s overhead, no? Why additional views and procedures (ps_helper) are not there by default? Some important things and storage engines are not properly instrumented yet, MDL locks, for example… ups, seems fixed in 5.7.3 Heuristics to control size (read Bug #68514 again)… less than obvious and I’d prefer more control Make up your mind on what should be in P_S vs I_S vs SHOW statements. Think about SHOW PROCESSLIST vs I_S.PROCESSLIST vs P_S.THREADS Want to know what else users care about? Check feature requests for P_S (11 “Verified” as of November 2, 2013), some of them are limitations “by design”: ○ ○ ○ Bug #69881 - “Try to compute elapsed time for an event which has not completed” - Todd Farmer Bug #66515 - “I have a problem with PS instrumentation that can only be enabled at server start.” Mark Callaghan Bug #70383 - “Implement a SELECT-syntax/statement as a synonym for SHOW ENGINE performance_schema STATUS” - Peter Laursen www.percona.com
  • Do we have a hope? www.percona.com
  • Do you still care about Bug #68079? “mysql-trunk (a.k.a 5.7.3): Scales up to 32 threads reasonably well, the HW has 48 logical CPUs” - Sunny Bains, October 18, 2013 “There is some confusion here, there are two issues in this bug report. One is the page mutex and the other is the page re-lock. For this query we acquire it in S mode all the time. If we remove the page mutex overhead then this S mode rwlock becomes the next bottleneck. However, this doesn't mean that the original problem is not fixed, it simply means that the fix requires further work that is more involved, that's all.“ - Sunny Bains, October 30, 2013 “These same numbers can be achieved by using "PARTITION BY KEY() PARTITIONS N". By using partitions you can reduce the contention on the block mutex.” - Sunny Bains, October 30, 2013 ● ● Does Oracle care about MySQL? Was PERFORMANCE_SCHEMA used to find the bottlenecks? www.percona.com
  • Thank you! Questions and Answers Please, report bugs to: http://bugs.mysql.com Use “Affects Me” button! www.percona.com