Sveta Smirnova, profile picture
Uploaded bySveta Smirnova
PDF, PPTX2,093 views

Troubleshooting MySQL Performance

This document discusses troubleshooting MySQL performance issues. It begins with an overview of MySQL server architecture and important components like the optimizer and storage engines. It then covers various diagnostic instruments for troubleshooting like log files, the Information Schema, and the Performance Schema. Specific issues covered include single statement performance using EXPLAIN, internal concurrency issues detected via locks diagnostics from sources like SHOW PROCESSLIST and the Performance Schema. The document provides examples of using these diagnostic tools to analyze and optimize query performance.

Embed presentation

Download as PDF, PPTX
Troubleshooting MySQL Performance Sveta Smirnova Principal Support Engineer November, 21, 2015
About Percona • Performance Consulting Experts • Free software: extended versions of MySQL and MongoDB (Percona Server), technical support, consulting, remote DBA (managed services) • Since 2006, headquarters are in USA • Distributed company with more than 100 employees in 30 countries • 3000+ customers, including Cisco and Alcatel Lucent 2 www.percona.com
Percona customers • Data retrieved from free sources 3 www.percona.com
Table of Contents •MySQL Server overview •Single statement performance •Internal concurrency issues •Environment 4 www.percona.com
MySQL Server overview 5 www.percona.com
MySQL architecture Connectors: C, JDBC, ODBC, Python, ... Connection Pool: Authentication, Caches SQL interface Parser Optimizer Caches and Buffers: Global Engine-specific Storage engines: InnoDB, TokuDB, ... File system: Data, Index, logs, other files • Base • Layout • Log files • Connectors • Optimizer • Cache and buffers • Storage engines • Management 6 www.percona.com
Important server parts • Optimizer • Storage engine • Study tools which offers your engine • Variables 7 www.percona.com
Troubleshooting instruments • Log files • Information Schema • SHOW commands • Performance Schema • Storage engine instruments 8 www.percona.com
Troubleshooting instruments • Log files • Error • Slow query • General query • Audit plugins • Your application • Operating system 8 www.percona.com
Troubleshooting instruments • Information Schema • Contains metadata information • Tables • Indexes • Other • Allows to create plugins • InnoDB plugins 8 www.percona.com
Troubleshooting instruments • Performance Schema • Monitors interval operations • Statements • Stages • Locks • Memory • Variables • Replication • IO • Mutexes and waits 8 www.percona.com
Troubleshooting instruments • Storage engine instruments • SHOW ENGINE ... STATUS • engine * variables • Engine-specific tools 8 www.percona.com
Monitoring solutions • Command-line • Percona Toolkit • MySQL Utilities • With GUI • MySQL Enterprise Monitor (MEM) • MEM plugin for Oracle Enterprise Manager • VividCortex • Many others 9 www.percona.com
What can affect performance? • Query processing • Optimization • Execution • Concurrency • Environment • Parallel processes • Hardware 10 www.percona.com
Troubleshooting workflow Statement Concurrency Environment 11 www.percona.com
Single statement performance 12 www.percona.com
Where to find slow statement? • While designing application • In slow query log • In Performance Schema mysql> select sql_text, (timer_end - timer_start)/1000000000 as time -> from events_statements_history_long -> where (timer_end - timer_start) > 4*10000000000 -> and event_name=’statement/sql/select’ and current_schema=’employees’G *************************** 1. row *************************** sql_text: select * from titles time: 1929.8997 *************************** 2. row *************************** sql_text: select count(*) from titles join employees using (emp_no) where title=’Senior Engineer’ time: 4493.3863 2 rows in set (0.01 sec) 13 www.percona.com
MySQL Indexes • B-Tree (Mostly) • Fractal Tree • R-Tree (Spatial) • Hash (Memory SE) • Engine- dependent d001 d003 d008 d009 d003****** d009****** d008****** d009****** d001****** d003****** d009****** d008****** d009****** d001****** d008****** d008****** d001****** 14 www.percona.com
More in Performance Schema • events statements * and prepared statements instances tables • Important field names • CREATED TMP DISK TABLES • CREATED TMP TABLES • SELECT FULL JOIN • SELECT RANGE CHECK • SELECT SCAN • SORT MERGE PASSES • SORT SCAN • Views in sys schema 15 www.percona.com
More in Performance Schema • events statements * and prepared statements instances tables • Views in sys schema • Important view names • statement analysis • statements with full table scans • statements with runtimes in 95th percentile • statements with sorting • statements with temp tables • statements with errors or warnings 15 www.percona.com
Stages shortcuts • Everything, related to temporary tables • EVENT NAME LIKE ’stage/sql/%tmp%’ • Everything, related to locks • EVENT NAME LIKE ’stage/sql/%lock%’ • Everything in state ”Waiting for” • EVENT NAME LIKE ’stage/%/Waiting for%’ • Frequently met issues (in Support) 16 www.percona.com
Stages shortcuts • Everything, related to temporary tables • Everything, related to locks • Everything in state ”Waiting for” • Frequently met issues (in Support) • EVENT NAME=’stage/sql/end’ • EVENT NAME=’stage/sql/freeing items’ • EVENT NAME=’stage/sql/Sending data’ • EVENT NAME=’stage/sql/cleaning up’ • EVENT NAME=’stage/sql/closing tables’ 16 www.percona.com
How to fix? Optimize! • EXPLAIN EXTENDED • Should be followed by SHOW WARNINGS • EXPLAIN PARTITIONS • EXPLAIN FORMAT=JSON • Status variables ’Handler %’ • INFORMATION SCHEMA.TRACE 17 www.percona.com
EXPLAIN in MySQL mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.userG *************************** 1. row *************************** id: 1 select_type: SIMPLE table: user type: index possible_keys: NULL key: PRIMARY key_len: 228 ref: NULL rows: 4 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) mysql> SHOW WARNINGSG ******* 1. row ******* Level: Note Code: 1003 Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mysql‘.‘user‘ 18 www.percona.com
EXPLAIN in details mysql> explain extended select * from t1 join t2 where... +----+-------------+-------+-------+*** | id | select_type | table | type |*** +----+-------------+-------+-------+*** | 1 | SIMPLE | t1 | ref |*** | 1 | SIMPLE | t2 | index |*** +----+-------------+-------+-------+*** 2 rows in set, 1 warning (0.00 sec) SIMPLE;PRIMARY;UNION;DEPENDENT UNION;UNION RESULT; SUBQUERY;DEPENDENT SUBQUERY;DERIVED;MATERIALIZED system const eq ref ref fulltext ref or null index merge unique subquery index subquery range index ALL 19 www.percona.com
EXPLAIN in details: keys mysql> explain extended select * from t1 join t2 where t1.int_key=1; ***+---------------+---------+---------+-------+*** ***| possible_keys | key | key_len | ref |*** ***+---------------+---------+---------+-------+*** ***| int_key,ik | int_key | 5 | const |*** ***| NULL | pk | 9 | NULL |*** ***+---------------+---------+---------+-------+*** 2 rows in set, 1 warning (0.00 sec) Keys, which can be used for resolving the query Only one key was actually used Actual length of the key (Important for multiple-column keys) Which columns were compared with the index Constant Numeric in our case Index used to resolve rows 20 www.percona.com
EXPLAIN in details: rows mysql> explain extended select * from t1 join t2 where t1.int_key=1; ***+-------+----------+----------------------------------------------------+ ***| rows | filtered | Extra | ***+-------+----------+----------------------------------------------------+ ***| 4 | 100.00 | NULL | ***| 6 | 100.00 | Using index; Using join buffer (Block Nested Loop) | ***+-------+----------+----------------------------------------------------+ 2 rows in set, 1 warning (0.00 sec) % of rows filtered Number of rows accessed Additional information: how query is resolved Using filesort Using temporary etc. 4X6 = 24 All rows used 21 www.percona.com
EXPLAIN FORMAT=JSON • Gives more information than regular EXPLAIN • Real execution path of the query • Pushed conditions • Query cost • Temporary table and index creation are more precise • Reflects execution order of ”group by” and ”order by” operations • Displays table materializations 22 www.percona.com
EXPLAIN FORMAT=JSON mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.userG *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "table": { "table_name": "user", "access_type": "index", "key": "PRIMARY", "used_key_parts": [ "Host", "User" ], "key_length": "228", "rows": 8, "filtered": 100, "using_index": true } } } 23 www.percona.com
EXPLAIN PARTITIONS mysql> explain partitions select count(*) -> from employees_part where hire_date > ’1991-01-01’G ************************ 1. row ************************ id: 1 select_type: SIMPLE table: employees_part partitions: p1,p2 type: index possible_keys: NULL key: PRIMARY key_len: 7 ref: NULL rows: 135214 Extra: Using where; Using index 24 www.percona.com
When EXPLAIN lies: Handler % mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select titles.* from titles join employees using (emp no) -> where title=’Senior Engineer’ and to date=’9999-01-01’ -> order by from date limit 10; +--------+-----------------+------------+------------+ | emp_no | title | from_date | to_date | +--------+-----------------+------------+------------+ | 235233 | Senior Engineer | 1985-02-01 | 9999-01-01 | ... mysql> SHOW STATUS LIKE ’Handler_read_%’; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_read_first | 1 | | Handler_read_key | 11 | ... | Handler_read_rnd_next | 443309 | +-----------------------+--------+ 7 rows in set (0.00 sec) 25 www.percona.com
I S.OPTIMIZER TRACE • Debug trace of optimizer • Prints • join preparation • join optimization • considered execution plans • refine plan • More • join execution 26 www.percona.com
I S.OPTIMIZER TRACE example mysql> SET optimizer_trace="enabled=on"; Query OK, 0 rows affected (0.00 sec) mysql> select titles.* from titles join employees using (emp_no) -> where title=’Senior Engineer’ and to_date=’9999-01-01’ -> order by from_date limit 10G *************************** 1. row *************************** emp_no: 235233 title: Senior Engineer from_date: 1985-02-01 to_date: 9999-01-01 ... 27 www.percona.com
I S.OPTIMIZER TRACE example cont. mysql> SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACEG *************************** 1. row *************************** QUERY: select titles.* from titles join employees using (emp_no) where ... TRACE: { "steps": [ { "join_preparation": { "select#": 1, "steps": [ { "expanded_query": ... { "join_optimization": { "select#": 1, "steps": [ { "condition_processing": { ... 28 www.percona.com
I S.OPTIMIZER TRACE example cont. "considered_execution_plans": [ { "plan_prefix": [ ], "table": "‘employees‘", "best_access_path": { "considered_access_paths": [ { "access_type": "ref", "index": "PRIMARY", "usable": false, "chosen": false }, { "rows_to_scan": 269763, "access_type": "scan", "resulting_rows": 269763, "cost": 54754, "chosen": true } ... 29 www.percona.com
I S.OPTIMIZER TRACE example cont. "condition_filtering_pct": 100, "rows_for_plan": 269763, "cost_for_plan": 54754, ... "rest_of_plan": [ { "plan_prefix": [ "‘titles‘" ], "table": "‘employees‘", "best_access_path": { "considered_access_paths": [ { "access_type": "eq_ref", "index": "PRIMARY", "rows": 1, "cost": 5222.9, "chosen": true, "cause": "clustered_pk_chosen_by_heuristics" }, 30 www.percona.com
I S.OPTIMIZER TRACE example cont. { "access_type": "scan", "chosen": false, "cause": "covering_index_better_than_full_scan" } ] }, "condition_filtering_pct": 100, "rows_for_plan": 4352.4, "cost_for_plan": 93459, "chosen": true ... 31 www.percona.com
Internal concurrency issues 32 www.percona.com
Common concurrency issues • Query or transaction waits a lock, held by another one • Fight for system resources • Symptoms • Many processes are waiting • You see query in the slow log, but it runs fast if single-thread environment • Query execution time varies 33 www.percona.com
Lock types and transactions • Lock types • Levels • MDL • Table-level • Row-level • What do they lock • Read locks • Block writes • Write locks • Block reads and writes • Transactions • Server-level • MDL locks • Table locks • Engine-level • Table locks • Row locks • AUTOCOMMIT • supported 34 www.percona.com
Locks diagnostic • SHOW [FULL] PROCESSLIST • SHOW ENGINE INNODB STATUS • INFORMATION SCHEMA • PROCESSLIST • InnoDB tables • Performance Schema • METADATA LOCKS • TABLE HANDLES • EVENTS TRANSACTIONS * • Both server-level and engine-level • Contain GTID information 35 www.percona.com
Locks diagnostic • SHOW [FULL] PROCESSLIST • For any kind of locks mysql> select id, state, info from information_schema.processlistG ******************** 1. row ******************** id: 5 state: info: NULL ******************** 2. row ******************** id: 4 state: Waiting for table metadata lock info: alter table titles add column f1 int ******************** 3. row ******************** id: 2 state: executing info: select id, state, info from information_schema.processlist 35 www.percona.com
Locks diagnostic • Performance Schema • Metadata locks mysql> select processlist_id, object_type, lock_type, lock_status, source -> from metadata_locks join threads on (owner_thread_id=thread_id) -> where object_schema=’employees’ and object_name=’titles’G *************************** 1. row *************************** processlist_id: 4 object_type: TABLE lock_type: EXCLUSIVE lock_status: PENDING -- waits source: mdl.cc:3263 *************************** 2. row *************************** processlist_id: 5 object_type: TABLE lock_type: SHARED_READ lock_status: GRANTED -- holds source: sql_parse.cc:5707 35 www.percona.com
Locks diagnostic • Performance Schema • Table locks mysql2> select * from table_handlesG *************************** 1. row *************************** OBJECT_TYPE: TABLE OBJECT_SCHEMA: employees OBJECT_NAME: employees OBJECT_INSTANCE_BEGIN: 140544885988272 OWNER_THREAD_ID: 23 OWNER_EVENT_ID: 818320 INTERNAL_LOCK: NULL EXTERNAL_LOCK: READ EXTERNAL -- Table lock! 1 row in set (0.00 sec) 35 www.percona.com
Locks diagnostic • Performance Schema • Transactions mysql> select thread_id, processlist_id, access_mode, gtid -> from events_transactions_current join threads using(thread_id) -> where state = ’active’G *************************** 1. row *************************** thread_id: 25 processlist_id: 2 access_mode: READ WRITE gtid: AUTOMATIC *************************** 2. row *************************** thread_id: 28 processlist_id: 5 access_mode: READ WRITE gtid: 07aff4c6-87e7-11e5-9a51-30b5c2208a0f:202 2 rows in set (0.00 sec) 35 www.percona.com
InnoDB Monitors • SHOW ENGINE INNODB STATUS • Permanent output • innodb status output • innodb status output locks • innodb-status-file - Deleted on normal shutdown! 36 www.percona.com
InnoDB Monitors • SHOW ENGINE INNODB STATUS ------------- TRANSACTIONS ------------- Trx id counter 0 26243837 Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0 History list length 2 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index mysql tables in use 1, locked 1 LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s) MySQL thread id 4485, query id 25022137 localhost root Updating update t set a=36 where a=6 36 www.percona.com
Other InnoDB diagnostic • INFORMATION SCHEMA • INNODB TRX • INNODB LOCKS • INNODB LOCK WAITS • INNODB METRICS • Options innodb monitor * • Option innodb print all deadlocks 37 www.percona.com
Other InnoDB diagnostic • INFORMATION SCHEMA • mysql> SELECT TRX_ID, TRX_REQUESTED_LOCK_ID, BLOCKING_TRX_ID, -> BLOCKING_LOCK_ID, TRX_QUERY -> FROM INNODB_TRX JOIN INNODB_LOCK_WAITS ON -> (INNODB_TRX.TRX_ID=INNODB_LOCK_WAITS.REQUESTING_TRX_ID) -> WHERE TRX_STATE = ’LOCK WAIT’G *************************** 1. row *************************** TRX_ID: 2318 TRX_REQUESTED_LOCK_ID: 2318:28:5:2 BLOCKING_TRX_ID: 2316 BLOCKING_LOCK_ID: 2316:28:5:2 TRX_QUERY: update titles set title=’Principal Engineer’ where title=’Senior Engineer’ 1 row in set (0.00 sec) mysql> select TRX_ID, trx_mysql_thread_id from INNODB_TRX WHERE TRX_ID=2316G *************************** 1. row *************************** TRX_ID: 2316 trx_mysql_thread_id: 2 1 row in set (0.00 sec) 37 www.percona.com
Locks diagnostic summary • Table-level • PROCESSLIST: ”Waiting for table lock” • P S.TABLE HANDLES • Row-level • InnoDB monitors • SHOW ENGINE INNODB STATUS • Tables in INFORMATION SCHEMA • P S.EVENTS TRANSACTIONS • Option –innodb print all deadlocks • MDL • PROCESSLIST • ”Waiting for metadata lock” • P S.METADATA LOCKS 38 www.percona.com
Environment 39 www.percona.com
Memory usage • What uses memory in MySQL • Buffers • Temporary tables • Internal structures - out of user control! • OS did not show memory as freed yet? 40 www.percona.com
Memory diagnostic before 5.7 • free • top • vmstat • Investigation • There was no way to know how exactly memory was allocated 41 www.percona.com
Memory diagnostic before 5.7 • free $free total used free shared buffers cached Mem: 16149184 6223916 9925268 317536 1048 3655160 -/+ buffers/cache: 2567708 13581476 Swap: 2110460 0 2110460 • top • vmstat • Investigation 41 www.percona.com
Memory diagnostic before 5.7 • free • top $top Tasks: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.0 us, 0.8 sy, 0.1 ni, 95.4 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 s KiB Mem: 16149184 total, 6231688 used, 9917496 free, 1048 buffers KiB Swap: 2110460 total, 0 used, 2110460 free. 3670752 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1914 mysql 20 0 670m 95m 1296 S 0.7 1.2 2:42.14 mysqld • vmstat • Investigation 41 www.percona.com
Memory diagnostic before 5.7 • free • top • vmstat $vmstat -t 5 3 procs ----------------------memory---------------------- ------swap---- ------ r b swpd free buff cache si so bi bo in cs us sy id wa.. 2 0 0 9923160 1048 3662724 0 0 168 86 167 674 3 1 87... 0 0 0 9923252 1048 3662904 0 0 30 122 1168 5264 3 1 96... 0 0 0 9922864 1048 3663120 0 0 25 128 1191 5342 2 1 96... • Investigation 41 www.percona.com
Memory diagnostic before 5.7 • free • top • vmstat • Investigation • Total size of buffers • Number of temporary tables • Number of parallel connections 41 www.percona.com
Memory diagnostic in 5.7 mysql> select thread_id tid, user, current_allocated ca, total_allocated -> from sys.memory_by_thread_by_current_bytes; +-----+-------------------------+-------------+-----------------+ | tid | user | ca | total_allocated | +-----+-------------------------+-------------+-----------------+ | 1 | sql/main | 2.53 GiB | 2.69 GiB | | 150 | root@127.0.0.1 | 4.06 MiB | 32.17 MiB | | 146 | sql/slave_sql | 1.31 MiB | 1.44 MiB | | 145 | sql/slave_io | 1.08 MiB | 2.79 MiB | ... | 60 | innodb/io_read_thread | 0 bytes | 384 bytes | | 139 | innodb/srv_purge_thread | -328 bytes | 754.21 KiB | | 69 | innodb/io_write_thread | -1008 bytes | 34.28 KiB | | 68 | innodb/io_write_thread | -1440 bytes | 298.05 KiB | | 74 | innodb/io_write_thread | -1656 bytes | 103.55 KiB | | 4 | innodb/io_log_thread | -2880 bytes | 132.38 KiB | | 72 | innodb/io_write_thread | -7632 bytes | 1.10 MiB | +-----+-------------------------+-------------+-----------------+ 145 rows in set (2.65 sec) 42 www.percona.com
Threads Statistics mysql> select * from sys.memory_by_thread_by_current_bytes -> order by current_allocated descG *************************** 1. row *************************** thread_id: 152 user: lj@127.0.0.1 current_count_used: 325 current_allocated: 36.00 GiB current_avg_alloc: 113.43 MiB current_max_alloc: 36.00 GiB total_allocated: 37.95 GiB ... 43 www.percona.com
RAW Performance Schema tables • memory summary by account by event name • memory summary by host by event name • memory summary by thread by event name • memory summary by user by event name • memory summary global by event name • sys schema also includes information about user name 44 www.percona.com
Users in sys.memory * tables • NAME@HOST - regular user • System users • sql/main • innodb/* • ... • Data comes from table THREADS 45 www.percona.com
Memory usage: how to fix? • mysqld uses to low memory • Buffers are not tuned for performance • Operating system limits resources • mysqld uses too much memory • Buffers are too large • General use buffers: allocated once for whole server • Connection buffers: allocated for every connection • Task buffers: allocated for the particular task • Prioritize! • Allocations, which cannot be controlled • Watch • Find workaround • Implement 46 www.percona.com
Disk diagnostics • df • iostat • ls -l /proc/{PID OF MYSQLD}/fd • InnoDB status • performance schema.table io waits % • sys.io % 47 www.percona.com
Disk diagnostics • df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda6 41944064 28429816 11433384 72% / devtmpfs 8067632 0 8067632 0% /dev tmpfs 8074592 41300 8033292 1% /dev/shm tmpfs 8074592 2376 8072216 1% /run tmpfs 8074592 0 8074592 0% /sys/fs/cgroup /dev/sda6 41944064 28429816 11433384 72% /.snapshots /dev/sda6 41944064 28429816 11433384 72% /var/tmp /dev/sda6 41944064 28429816 11433384 72% /var/spool /dev/sda6 41944064 28429816 11433384 72% /var/opt /dev/sda6 41944064 28429816 11433384 72% /var/log /dev/sda6 41944064 28429816 11433384 72% /var/lib/pgsql /dev/sda6 41944064 28429816 11433384 72% /var/lib/named /dev/sda6 41944064 28429816 11433384 72% /var/lib/mailman /dev/sda6 41944064 28429816 11433384 72% /var/crash /dev/sda6 41944064 28429816 11433384 72% /usr/local /dev/sda6 41944064 28429816 11433384 72% /tmp ... 47 www.percona.com
Disk diagnostics • iostat sveta@thinkie:~/src/test-db/employees_db> iostat -x Linux 3.16.7-92-desktop (thinkie) 11/11/2015 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.47 1.77 2.54 0.94 0.00 80.28 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz ... sda 0.28 0.70 4.66 21.59 93.69 346.10 33.51 4.14 ... sdb 0.00 0.00 0.00 0.00 0.02 0.00 15.69 0.00 ... 47 www.percona.com
Disk diagnostics • ls -l /proc/{PID OF MYSQLD}/fd sveta@thinkie:~/src/test-db/employees_db> ls -l /proc/26676/fd total 0 lrwx------ 1 sveta users 64 Nov 11 15:11 0 -> /dev/pts/1 l-wx------ 1 sveta users 64 Nov 11 15:11 1 -> /log/mysqld.1.err lrwx------ 1 sveta users 64 Nov 11 15:11 10 -> /data/ib_logfile1 lrwx------ 1 sveta users 64 Nov 11 15:11 11 -> /data/ibtmp1 lrwx------ 1 sveta users 64 Nov 11 15:11 12 -> /tmp/mysqld.1/ibx2zTxG (deleted ... lrwx------ 1 sveta users 64 Nov 11 15:11 27 -> /data/sys/sys_config.ibd l-wx------ 1 sveta users 64 Nov 11 15:11 28 -> /data/master-bin.000002 lrwx------ 1 sveta users 64 Nov 11 15:11 29 -> socket:[1245731] lrwx------ 1 sveta users 64 Nov 11 15:11 3 -> /data/master-bin.index ... lrwx------ 1 sveta users 64 Nov 11 15:11 5 -> /tmp/mysqld.1/ibT4odfr (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 6 -> /tmp/mysqld.1/ibJoMdZu (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 7 -> /tmp/mysqld.1/ibvUBeJy (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 8 -> /tmp/mysqld.1/ibP6JxvC (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 9 -> /data/ib_logfile0 47 www.percona.com
Disk diagnostics • InnoDB status mysql> show engine innodb statusG ... -------- FILE I/O -------- I/O thread 0 state: waiting for i/o request (insert buffer thread) I/O thread 1 state: waiting for i/o request (log thread) I/O thread 2 state: waiting for i/o request (read thread) I/O thread 3 state: waiting for i/o request (read thread) I/O thread 4 state: waiting for i/o request (write thread) I/O thread 5 state: waiting for i/o request (write thread) Pending normal aio reads: [0, 0] , aio writes: [0, 0] , ibuf aio reads:, log i/o’s:, sync i/o’s: Pending flushes (fsync) log: 0; buffer pool: 0 11468 OS file reads, 102 OS file writes, 17 OS fsyncs 11.80 reads/s, 16384 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ... 47 www.percona.com
Disk diagnostics • InnoDB status -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=26676, Main thread ID=140424906065664, state: sleeping Number of rows inserted 19, updated 0, deleted 0, read 5843307 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1584.23 reads/s 47 www.percona.com
Disk diagnostics mysql> select OBJECT_SCHEMA, OBJECT_NAME, INDEX_NAME -> from table_io_waits_summary_by_index_usage where COUNT_STAR=0; +---------------+--------------+------------+ | OBJECT_SCHEMA | OBJECT_NAME | INDEX_NAME | +---------------+--------------+------------+ | employees | departments | PRIMARY | | employees | departments | dept_name | | employees | dept_emp | PRIMARY | | employees | dept_emp | emp_no | | employees | dept_emp | dept_no | | employees | dept_manager | PRIMARY | | employees | dept_manager | emp_no | | employees | dept_manager | dept_no | | employees | salaries | PRIMARY | | employees | salaries | emp_no | | sys | sys_config | PRIMARY | +---------------+--------------+------------+ 11 rows in set (0.00 sec) 47 www.percona.com
Disk diagnostics • sys.io % mysql> select event_name, count_read, total_read, total_requested -> from io_global_by_wait_by_bytes; +-------------------------+------------+------------+-----------------+ | event_name | count_read | total_read | total_requested | +-------------------------+------------+------------+-----------------+ | innodb/innodb_data_file | 17349 | 274.36 MiB | 289.22 MiB | | sql/binlog | 16696 | 130.42 MiB | 130.42 MiB | | sql/io_cache | 1556 | 37.61 MiB | 100.49 MiB | | sql/FRM | 1137 | 626.03 KiB | 626.03 KiB | | innodb/innodb_log_file | 8 | 132.50 KiB | 141.50 KiB | | sql/ERRMSG | 3 | 72.23 KiB | 72.23 KiB | | sql/query_log | 0 | 0 bytes | 44.02 KiB | | myisam/dfile | 20 | 21.99 KiB | 21.99 KiB | | mysys/charset | 1 | 18.27 KiB | 18.27 KiB | | myisam/kfile | 24 | 6.62 KiB | 6.62 KiB | | sql/file_parser | 3 | 2.11 KiB | 2.11 KiB | | sql/slow_log | 0 | 0 bytes | 1.64 KiB | | mysys/cnf | 3 | 56 bytes | 56 bytes | ... 47 www.percona.com
Disk diagnostics • sys.io % • Other useful views • host summary by file io • host summary by file io type • io by thread by latency • io global by file by bytes - Per-file • io global by file by latency • io global by wait by bytes • io global by wait by latency • latest file io - Latest IO operations • user summary by file io • user summary by file io type 47 www.percona.com
How to fix IO issues • Optimize queries • InnoDB options, affecting writes • Binary log 48 www.percona.com
How to fix IO issues • Optimize queries • InnoDB options, affecting writes • innodb adaptive flushing • innodb doublewrite • innodb flush log at trx commit • innodb flush method - Test! • innodb io capacity - Follow docs • innodb max dirty pages pct 48 www.percona.com
How to fix IO issues • Optimize queries • Binary log • Formats • ROW • STATEMENT • MIXED • binlog row image • sync binlog 48 www.percona.com
CPU diagnostics • top • iostat • ps 49 www.percona.com
CPU diagnostics • top Tasks: 298 total, 2 running, 296 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.2 us, 2.7 sy, 1.8 ni, 79.4 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 16149184 total, 15075560 used, 1073624 free, 1596 buffers KiB Swap: 2110460 total, 8252 used, 2102208 free. 9994956 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26676 sveta 20 0 1963204 155108 23300 S 100.0 0.960 0:43.73 mysqld 10365 sveta 20 0 3601524 1.555g 108488 S 25.00 10.10 224:21.91 firefox 29915 sveta 20 0 15476 2612 2120 R 12.50 0.016 0:00.02 top ... 49 www.percona.com
CPU diagnostics • iostat sveta@thinkie:~/src/test-db/employees_db> iostat -x Linux 3.16.7-92-desktop (thinkie) 11/11/2015 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.47 1.77 2.54 0.94 0.00 80.28 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz ... sda 0.28 0.70 4.66 21.59 93.69 346.10 33.51 4.14 ... sdb 0.00 0.00 0.00 0.00 0.02 0.00 15.69 0.00 ... 49 www.percona.com
CPU diagnostics • ps sveta@thinkie> ps -eo pid,user,comm,pcpu,pmem,vsz | grep mysqld 26676 sveta mysqld 1.4 0.9 1963204 26678 sveta mysqld 0.0 0.7 1765772 49 www.percona.com
CPU: problem solving • Low load • Is your mysqld really active? • Check OS limits • ulimit • User options • Global options • Increase concurrency-related options • innodb thread concurrency • innodb read io threads • innodb write io threads • Max (2 X (number of CPU cores))! • High load • Move job to memory: increase buffers • Tune queries 50 www.percona.com
Network: what is worth attention? • Stability • Bandwidth • Speed • RTT sveta@thinkie> ping www.percona.com -c 2 PING www.percona.com (74.121.199.234) 56(84) bytes of data. 64 bytes from www.percona.com (74.121.199.234): icmp_seq=1 ttl=54 time=18 64 bytes from www.percona.com (74.121.199.234): icmp_seq=2 ttl=54 time=18 --- www.percona.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 181.380/181.685/181.991/0.524 ms 51 www.percona.com
Network:diagnostics • –log-warnings=2 • Send huge file (1G or larger) • tcpdump • P S.accounts, hosts, users • P S.host cache 52 www.percona.com
Network:diagnostics • tcpdump $tcpdump -i en1 host master.mysql.com and port 33051 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes 22:28:12.195270 IP master.mysql.com.33051 > apple.60344: P 1752426772:1752426864(92) ack 1474226199 win 91 <nop,nop,timestamp 1939999898 649946687> 22:28:12.195317 IP apple.60344 > master.mysql.com.33051: . ack 92 win 65535 <nop,nop,timestamp 649946998 1939999898> ^C 2 packets captured 37 packets received by filter 0 packets dropped by kernel 52 www.percona.com
Network:diagnostics • P S.accounts, hosts, users mysql> select user, host, current_connections as cur, -> total_connections as total from accounts; +------+-----------+-----+-------+ | user | host | cur | total | +------+-----------+-----+-------+ | foo | localhost | 0 | 3 | | root | localhost | 1 | 3 | | NULL | NULL | 14 | 17 | +------+-----------+-----+-------+ 3 rows in set (0.01 sec) 52 www.percona.com
Network:diagnostics • P S.host cache mysql> select IP, HOST, HOST_VALIDATED, SUM_CONNECT_ERRORS, -> COUNT_NAMEINFO_PERMANENT_ERRORS, COUNT_PROXY_USER_ERRORS, -> COUNT_DEFAULT_DATABASE_ERRORS, FIRST_SEEN, LAST_SEEN from host_cacheG ************************ 1. row ************************ IP: 10.159.169.27 HOST: foo.bar.com HOST_VALIDATED: YES SUM_CONNECT_ERRORS: 1 COUNT_NAMEINFO_PERMANENT_ERRORS: 0 COUNT_PROXY_USER_ERRORS: 0 COUNT_DEFAULT_DATABASE_ERRORS: 1 FIRST_SEEN: 2014-06-10 20:29:45 LAST_SEEN: 2015-11-11 15:32:44 52 www.percona.com
More information 53 www.percona.com
Further reading • MySQL Troubleshooting book • High Performance MySQL book • Planet MySQL • MySQL User Reference Manual • Bug trackers • http://bugs.mysql.com • https://bugs.launchpad.net/percona- server/ 54 www.percona.com
Current Percona vacancies www.percona.com/about-percona/careers 55 www.percona.com
Place for your questions ??? 56 www.percona.com
Thank you! http://www.slideshare.net/SvetaSmirnova https://twitter.com/svetsmirnova 57 www.percona.com

More Related Content

PDF
Introducing new SQL syntax and improving performance with preparse Query Rewr...
bySveta Smirnova
 
PDF
Troubleshooting MySQL Performance
bySveta Smirnova
 
PDF
Moving to the NoSQL side: MySQL JSON functions
bySveta Smirnova
 
PDF
Why Use EXPLAIN FORMAT=JSON?
bySveta Smirnova
 
PDF
New features in Performance Schema 5.7 in action
bySveta Smirnova
 
PDF
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
PDF
New features in Performance Schema 5.7 in action
bySveta Smirnova
 
PDF
MySQL Performance Schema in Action
bySveta Smirnova
 
Introducing new SQL syntax and improving performance with preparse Query Rewr...
bySveta Smirnova
 
Troubleshooting MySQL Performance
bySveta Smirnova
 
Moving to the NoSQL side: MySQL JSON functions
bySveta Smirnova
 
Why Use EXPLAIN FORMAT=JSON?
bySveta Smirnova
 
New features in Performance Schema 5.7 in action
bySveta Smirnova
 
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
New features in Performance Schema 5.7 in action
bySveta Smirnova
 
MySQL Performance Schema in Action
bySveta Smirnova
 

What's hot

PDF
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
PDF
Basic MySQL Troubleshooting for Oracle Database Administrators
bySveta Smirnova
 
PDF
Preparse Query Rewrite Plugins
bySveta Smirnova
 
PDF
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
PDF
MySQL Query tuning 101
bySveta Smirnova
 
PDF
UKOUG 2011: Practical MySQL Tuning
byFromDual GmbH
 
PDF
Basic MySQL Troubleshooting for Oracle DBAs
bySveta Smirnova
 
PDF
Highload Perf Tuning
byHighLoad2009
 
PDF
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
PDF
Introduction to MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
PDF
Character Encoding - MySQL DevRoom - FOSDEM 2015
bymushupl
 
PDF
Introduction into MySQL Query Tuning
bySveta Smirnova
 
PDF
Performance Schema for MySQL troubleshooting
bySveta Smirnova
 
PDF
MySQL Performance Schema in 20 Minutes
bySveta Smirnova
 
PDF
Introduction into MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
PDF
MySQL Troubleshooting with the Performance Schema
bySveta Smirnova
 
ODP
Common schema my sql uc 2012
byRoland Bouman
 
PDF
Capturing, Analyzing, and Optimizing your SQL
byPadraig O'Sullivan
 
PDF
Modern query optimisation features in MySQL 8.
byMydbops
 
PDF
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB Cluster
bySveta Smirnova
 
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
Basic MySQL Troubleshooting for Oracle Database Administrators
bySveta Smirnova
 
Preparse Query Rewrite Plugins
bySveta Smirnova
 
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
MySQL Query tuning 101
bySveta Smirnova
 
UKOUG 2011: Practical MySQL Tuning
byFromDual GmbH
 
Basic MySQL Troubleshooting for Oracle DBAs
bySveta Smirnova
 
Highload Perf Tuning
byHighLoad2009
 
Performance Schema for MySQL Troubleshooting
bySveta Smirnova
 
Introduction to MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
Character Encoding - MySQL DevRoom - FOSDEM 2015
bymushupl
 
Introduction into MySQL Query Tuning
bySveta Smirnova
 
Performance Schema for MySQL troubleshooting
bySveta Smirnova
 
MySQL Performance Schema in 20 Minutes
bySveta Smirnova
 
Introduction into MySQL Query Tuning for Dev[Op]s
bySveta Smirnova
 
MySQL Troubleshooting with the Performance Schema
bySveta Smirnova
 
Common schema my sql uc 2012
byRoland Bouman
 
Capturing, Analyzing, and Optimizing your SQL
byPadraig O'Sullivan
 
Modern query optimisation features in MySQL 8.
byMydbops
 
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB Cluster
bySveta Smirnova
 

Viewers also liked

PDF
How to Design Indexes, Really
byKarwin Software Solutions LLC
 
KEY
MySQL Performance - SydPHP October 2011
byGraham Weldon
 
PDF
The MySQL Performance Schema & New SYS Schema
byTed Wennmark
 
PDF
Performance Schema in MySQL (Danil Zburivsky)
byOntico
 
PDF
MySQL Oslayer performace optimization
byLouis liu
 
PDF
An Overview to MySQL SYS Schema
byMydbops
 
ODP
MySQL Monitoring Mechanisms
byMark Leith
 
PDF
MySQL User Camp: MySQL Cluster
byShivji Kumar Jha
 
PDF
MySQL User Camp: GTIDs
byShivji Kumar Jha
 
PDF
Mysql tech day_paris_ps_and_sys
byMark Leith
 
PPTX
MySQL 5.7 New Features for Developers
byZohar Elkayam
 
PDF
Open source India - MySQL Labs: Multi-Source Replication
byShivji Kumar Jha
 
PDF
MySQL Query Optimization.
byRemote MySQL DBA
 
PDF
My sql 5.7-upcoming-changes-v2
byMorgan Tocker
 
PDF
Join-fu: The Art of SQL Tuning for MySQL
byZendCon
 
PDF
MySQL 5.6 - Operations and Diagnostics Improvements
byMorgan Tocker
 
PPTX
MySQL Indexing - Best practices for MySQL 5.6
byMYXPLAIN
 
PDF
MySQL For Linux Sysadmins
byMorgan Tocker
 
PDF
MySQL: Indexing for Better Performance
byjkeriaki
 
PDF
Instrumenting plugins for Performance Schema
byMark Leith
 
How to Design Indexes, Really
byKarwin Software Solutions LLC
 
MySQL Performance - SydPHP October 2011
byGraham Weldon
 
The MySQL Performance Schema & New SYS Schema
byTed Wennmark
 
Performance Schema in MySQL (Danil Zburivsky)
byOntico
 
MySQL Oslayer performace optimization
byLouis liu
 
An Overview to MySQL SYS Schema
byMydbops
 
MySQL Monitoring Mechanisms
byMark Leith
 
MySQL User Camp: MySQL Cluster
byShivji Kumar Jha
 
MySQL User Camp: GTIDs
byShivji Kumar Jha
 
Mysql tech day_paris_ps_and_sys
byMark Leith
 
MySQL 5.7 New Features for Developers
byZohar Elkayam
 
Open source India - MySQL Labs: Multi-Source Replication
byShivji Kumar Jha
 
MySQL Query Optimization.
byRemote MySQL DBA
 
My sql 5.7-upcoming-changes-v2
byMorgan Tocker
 
Join-fu: The Art of SQL Tuning for MySQL
byZendCon
 
MySQL 5.6 - Operations and Diagnostics Improvements
byMorgan Tocker
 
MySQL Indexing - Best practices for MySQL 5.6
byMYXPLAIN
 
MySQL For Linux Sysadmins
byMorgan Tocker
 
MySQL: Indexing for Better Performance
byjkeriaki
 
Instrumenting plugins for Performance Schema
byMark Leith
 

Similar to Troubleshooting MySQL Performance

PDF
Query Optimization with MySQL 5.6: Old and New Tricks
byMYXPLAIN
 
PDF
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
byJaime Crespo
 
PPTX
Alkin Tezuysal "MySQL Query Optimization Best Practices and Indexing"
byFwdays
 
PPTX
Работа с индексами - лучшие практики для MySQL 5.6, Петр Зайцев (Percona)
byOntico
 
PPTX
Mysql query optimization best practices and indexing
byAlkin Tezuysal
 
PDF
MySQL Database Monitoring: Must, Good and Nice to Have
bySveta Smirnova
 
PPTX
MySQL performance tuning
byAnurag Srivastava
 
PDF
Percona Live 2012PPT: MySQL Query optimization
bymysqlops
 
PDF
Scaling MySQL Strategies for Developers
byJonathan Levin
 
PDF
Quick Wins
byHighLoad2009
 
ODP
Beyond php - it's not (just) about the code
byWim Godden
 
PDF
Webinar 2013 advanced_query_tuning
by晓 周
 
PDF
Using MySQL in a web-scale environment
byDavid Landgren
 
PDF
MySQL Performance Schema in Action: the Complete Tutorial
bySveta Smirnova
 
PPTX
Guide To Mastering The MySQL Query Execution Plan
byOptimiz DBA
 
PDF
How to Design Indexes, Really
byMYXPLAIN
 
PDF
MySQL Performance Schema in Action
bySveta Smirnova
 
PPTX
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)
byDave Stokes
 
PDF
U C2007 My S Q L Performance Cookbook
byguestae36d0
 
PDF
介绍 Percona 服务器 XtraDB 和 Xtrabackup
byYUCHENG HU
 
Query Optimization with MySQL 5.6: Old and New Tricks
byMYXPLAIN
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
byJaime Crespo
 
Alkin Tezuysal "MySQL Query Optimization Best Practices and Indexing"
byFwdays
 
Работа с индексами - лучшие практики для MySQL 5.6, Петр Зайцев (Percona)
byOntico
 
Mysql query optimization best practices and indexing
byAlkin Tezuysal
 
MySQL Database Monitoring: Must, Good and Nice to Have
bySveta Smirnova
 
MySQL performance tuning
byAnurag Srivastava
 
Percona Live 2012PPT: MySQL Query optimization
bymysqlops
 
Scaling MySQL Strategies for Developers
byJonathan Levin
 
Quick Wins
byHighLoad2009
 
Beyond php - it's not (just) about the code
byWim Godden
 
Webinar 2013 advanced_query_tuning
by晓 周
 
Using MySQL in a web-scale environment
byDavid Landgren
 
MySQL Performance Schema in Action: the Complete Tutorial
bySveta Smirnova
 
Guide To Mastering The MySQL Query Execution Plan
byOptimiz DBA
 
How to Design Indexes, Really
byMYXPLAIN
 
MySQL Performance Schema in Action
bySveta Smirnova
 
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)
byDave Stokes
 
U C2007 My S Q L Performance Cookbook
byguestae36d0
 
介绍 Percona 服务器 XtraDB 和 Xtrabackup
byYUCHENG HU
 

More from Sveta Smirnova

PDF
War Story: Removing Offensive Language from Percona Toolkit
bySveta Smirnova
 
PDF
MySQL 2024: Зачем переходить на MySQL 8, если в 5.х всё устраивает?
bySveta Smirnova
 
PDF
Database in Kubernetes: Diagnostics and Monitoring
bySveta Smirnova
 
PDF
MySQL Cookbook: Recipes for Developers
bySveta Smirnova
 
PDF
MySQL Performance for DevOps
bySveta Smirnova
 
PDF
MySQL Test Framework для поддержки клиентов и верификации багов
bySveta Smirnova
 
PDF
MySQL Cookbook: Recipes for Your Business
bySveta Smirnova
 
PDF
Производительность MySQL для DevOps
bySveta Smirnova
 
PDF
MySQL Performance for DevOps
bySveta Smirnova
 
PDF
How to migrate from MySQL to MariaDB without tears
bySveta Smirnova
 
PDF
Modern solutions for modern database load: improvements in the latest MariaDB...
bySveta Smirnova
 
PDF
How Safe is Asynchronous Master-Master Setup?
bySveta Smirnova
 
PDF
Современному хайлоду - современные решения: MySQL 8.0 и улучшения Percona
bySveta Smirnova
 
PDF
How to Avoid Pitfalls in Schema Upgrade with Galera
bySveta Smirnova
 
PDF
How Safe is Asynchronous Master-Master Setup?
bySveta Smirnova
 
PDF
Billion Goods in Few Categories: How Histograms Save a Life?
bySveta Smirnova
 
PDF
A Billion Goods in a Few Categories: When Optimizer Histograms Help and When ...
bySveta Smirnova
 
PDF
Что нужно знать о трёх топовых фичах MySQL
bySveta Smirnova
 
PDF
Billion Goods in Few Categories: How Histograms Save a Life?
bySveta Smirnova
 
PDF
Optimizer Histograms: When they Help and When Do Not?
bySveta Smirnova
 
War Story: Removing Offensive Language from Percona Toolkit
bySveta Smirnova
 
MySQL 2024: Зачем переходить на MySQL 8, если в 5.х всё устраивает?
bySveta Smirnova
 
Database in Kubernetes: Diagnostics and Monitoring
bySveta Smirnova
 
MySQL Cookbook: Recipes for Developers
bySveta Smirnova
 
MySQL Performance for DevOps
bySveta Smirnova
 
MySQL Test Framework для поддержки клиентов и верификации багов
bySveta Smirnova
 
MySQL Cookbook: Recipes for Your Business
bySveta Smirnova
 
Производительность MySQL для DevOps
bySveta Smirnova
 
MySQL Performance for DevOps
bySveta Smirnova
 
How to migrate from MySQL to MariaDB without tears
bySveta Smirnova
 
Modern solutions for modern database load: improvements in the latest MariaDB...
bySveta Smirnova
 
How Safe is Asynchronous Master-Master Setup?
bySveta Smirnova
 
Современному хайлоду - современные решения: MySQL 8.0 и улучшения Percona
bySveta Smirnova
 
How to Avoid Pitfalls in Schema Upgrade with Galera
bySveta Smirnova
 
How Safe is Asynchronous Master-Master Setup?
bySveta Smirnova
 
Billion Goods in Few Categories: How Histograms Save a Life?
bySveta Smirnova
 
A Billion Goods in a Few Categories: When Optimizer Histograms Help and When ...
bySveta Smirnova
 
Что нужно знать о трёх топовых фичах MySQL
bySveta Smirnova
 
Billion Goods in Few Categories: How Histograms Save a Life?
bySveta Smirnova
 
Optimizer Histograms: When they Help and When Do Not?
bySveta Smirnova
 

Recently uploaded

PDF
Track Phone Location on iPhone (2026)_ What Actually Works, What’s a Scam, an...
byOlive Dimitrescu
 
PPTX
Best Digital Marketing Company in Lucknow
bydigitallearning9277
 
PPTX
HR Software for Construction: Smarter Workforce & Payroll Management
byhelixtahr
 
PDF
Track Phone Location Android (2026)_ What Actually Works, What’s New, and the...
byRachel Green
 
PDF
JFokus 2026 - Please pass the salt- Serve up passwords with a side of entropy...
byOrtus Solutions, Corp
 
PDF
software architecture for busy developers
byMuhammadAwaisQaisran
 
PDF
On Demand Grocery Delivery App Development
byV3CUBE TECHNOLABS
 
PDF
Metrics, Logs, Traces, and Mayhem_ Introducing an observability adventure gam...
byImma Valls Bernaus
 
PDF
Project Tracking in Software Project Management
bySahanaBarveen1
 
PDF
Track Phone Location Free by Number (2026)_ What’s Real, What’s a Scam, and t...
byRachel Green
 
PPTX
2025 S/4HANA Cloud Release Upgrade Overview.pptx
byssuser835a7b
 
PDF
Massage Booking App Development Company
byV3CUBE TECHNOLABS
 
PPTX
BUSY Software Business Management Solution.pptx
byMoving Cloud
 
PDF
20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf
byAkihiro Suda
 
PPTX
GraphQL Day Paris 2025 | Bringing DevOps to GraphQL with the Apollo GraphOS O...
byapidays
 
PDF
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
PDF
apidays Australia 2025 | Orchestrator vs. Choreography
byapidays
 
PPTX
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
PDF
Build a Scalable On-Demand Courier Service App
byV3CUBE TECHNOLABS
 
PDF
Artificial Intelligence Planning (Harinath Palavalli)
byHarinath Palavalli
 
Track Phone Location on iPhone (2026)_ What Actually Works, What’s a Scam, an...
byOlive Dimitrescu
 
Best Digital Marketing Company in Lucknow
bydigitallearning9277
 
HR Software for Construction: Smarter Workforce & Payroll Management
byhelixtahr
 
Track Phone Location Android (2026)_ What Actually Works, What’s New, and the...
byRachel Green
 
JFokus 2026 - Please pass the salt- Serve up passwords with a side of entropy...
byOrtus Solutions, Corp
 
software architecture for busy developers
byMuhammadAwaisQaisran
 
On Demand Grocery Delivery App Development
byV3CUBE TECHNOLABS
 
Metrics, Logs, Traces, and Mayhem_ Introducing an observability adventure gam...
byImma Valls Bernaus
 
Project Tracking in Software Project Management
bySahanaBarveen1
 
Track Phone Location Free by Number (2026)_ What’s Real, What’s a Scam, and t...
byRachel Green
 
2025 S/4HANA Cloud Release Upgrade Overview.pptx
byssuser835a7b
 
Massage Booking App Development Company
byV3CUBE TECHNOLABS
 
BUSY Software Business Management Solution.pptx
byMoving Cloud
 
20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf
byAkihiro Suda
 
GraphQL Day Paris 2025 | Bringing DevOps to GraphQL with the Apollo GraphOS O...
byapidays
 
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
apidays Australia 2025 | Orchestrator vs. Choreography
byapidays
 
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
Build a Scalable On-Demand Courier Service App
byV3CUBE TECHNOLABS
 
Artificial Intelligence Planning (Harinath Palavalli)
byHarinath Palavalli
 

Troubleshooting MySQL Performance

  • 1.
    Troubleshooting MySQL Performance SvetaSmirnova Principal Support Engineer November, 21, 2015
  • 2.
    About Percona • PerformanceConsulting Experts • Free software: extended versions of MySQL and MongoDB (Percona Server), technical support, consulting, remote DBA (managed services) • Since 2006, headquarters are in USA • Distributed company with more than 100 employees in 30 countries • 3000+ customers, including Cisco and Alcatel Lucent 2 www.percona.com
  • 3.
    Percona customers • Dataretrieved from free sources 3 www.percona.com
  • 4.
    Table of Contents •MySQLServer overview •Single statement performance •Internal concurrency issues •Environment 4 www.percona.com
  • 5.
  • 6.
    MySQL architecture Connectors: C,JDBC, ODBC, Python, ... Connection Pool: Authentication, Caches SQL interface Parser Optimizer Caches and Buffers: Global Engine-specific Storage engines: InnoDB, TokuDB, ... File system: Data, Index, logs, other files • Base • Layout • Log files • Connectors • Optimizer • Cache and buffers • Storage engines • Management 6 www.percona.com
  • 7.
    Important server parts •Optimizer • Storage engine • Study tools which offers your engine • Variables 7 www.percona.com
  • 8.
    Troubleshooting instruments • Logfiles • Information Schema • SHOW commands • Performance Schema • Storage engine instruments 8 www.percona.com
  • 9.
    Troubleshooting instruments • Logfiles • Error • Slow query • General query • Audit plugins • Your application • Operating system 8 www.percona.com
  • 10.
    Troubleshooting instruments • InformationSchema • Contains metadata information • Tables • Indexes • Other • Allows to create plugins • InnoDB plugins 8 www.percona.com
  • 11.
    Troubleshooting instruments • PerformanceSchema • Monitors interval operations • Statements • Stages • Locks • Memory • Variables • Replication • IO • Mutexes and waits 8 www.percona.com
  • 12.
    Troubleshooting instruments • Storageengine instruments • SHOW ENGINE ... STATUS • engine * variables • Engine-specific tools 8 www.percona.com
  • 13.
    Monitoring solutions • Command-line •Percona Toolkit • MySQL Utilities • With GUI • MySQL Enterprise Monitor (MEM) • MEM plugin for Oracle Enterprise Manager • VividCortex • Many others 9 www.percona.com
  • 14.
    What can affectperformance? • Query processing • Optimization • Execution • Concurrency • Environment • Parallel processes • Hardware 10 www.percona.com
  • 15.
  • 16.
  • 17.
    Where to findslow statement? • While designing application • In slow query log • In Performance Schema mysql> select sql_text, (timer_end - timer_start)/1000000000 as time -> from events_statements_history_long -> where (timer_end - timer_start) > 4*10000000000 -> and event_name=’statement/sql/select’ and current_schema=’employees’G *************************** 1. row *************************** sql_text: select * from titles time: 1929.8997 *************************** 2. row *************************** sql_text: select count(*) from titles join employees using (emp_no) where title=’Senior Engineer’ time: 4493.3863 2 rows in set (0.01 sec) 13 www.percona.com
  • 18.
    MySQL Indexes • B-Tree(Mostly) • Fractal Tree • R-Tree (Spatial) • Hash (Memory SE) • Engine- dependent d001 d003 d008 d009 d003****** d009****** d008****** d009****** d001****** d003****** d009****** d008****** d009****** d001****** d008****** d008****** d001****** 14 www.percona.com
  • 19.
    More in PerformanceSchema • events statements * and prepared statements instances tables • Important field names • CREATED TMP DISK TABLES • CREATED TMP TABLES • SELECT FULL JOIN • SELECT RANGE CHECK • SELECT SCAN • SORT MERGE PASSES • SORT SCAN • Views in sys schema 15 www.percona.com
  • 20.
    More in PerformanceSchema • events statements * and prepared statements instances tables • Views in sys schema • Important view names • statement analysis • statements with full table scans • statements with runtimes in 95th percentile • statements with sorting • statements with temp tables • statements with errors or warnings 15 www.percona.com
  • 21.
    Stages shortcuts • Everything,related to temporary tables • EVENT NAME LIKE ’stage/sql/%tmp%’ • Everything, related to locks • EVENT NAME LIKE ’stage/sql/%lock%’ • Everything in state ”Waiting for” • EVENT NAME LIKE ’stage/%/Waiting for%’ • Frequently met issues (in Support) 16 www.percona.com
  • 22.
    Stages shortcuts • Everything,related to temporary tables • Everything, related to locks • Everything in state ”Waiting for” • Frequently met issues (in Support) • EVENT NAME=’stage/sql/end’ • EVENT NAME=’stage/sql/freeing items’ • EVENT NAME=’stage/sql/Sending data’ • EVENT NAME=’stage/sql/cleaning up’ • EVENT NAME=’stage/sql/closing tables’ 16 www.percona.com
  • 23.
    How to fix?Optimize! • EXPLAIN EXTENDED • Should be followed by SHOW WARNINGS • EXPLAIN PARTITIONS • EXPLAIN FORMAT=JSON • Status variables ’Handler %’ • INFORMATION SCHEMA.TRACE 17 www.percona.com
  • 24.
    EXPLAIN in MySQL mysql>EXPLAIN EXTENDED SELECT user, host FROM mysql.userG *************************** 1. row *************************** id: 1 select_type: SIMPLE table: user type: index possible_keys: NULL key: PRIMARY key_len: 228 ref: NULL rows: 4 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) mysql> SHOW WARNINGSG ******* 1. row ******* Level: Note Code: 1003 Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mysql‘.‘user‘ 18 www.percona.com
  • 25.
    EXPLAIN in details mysql>explain extended select * from t1 join t2 where... +----+-------------+-------+-------+*** | id | select_type | table | type |*** +----+-------------+-------+-------+*** | 1 | SIMPLE | t1 | ref |*** | 1 | SIMPLE | t2 | index |*** +----+-------------+-------+-------+*** 2 rows in set, 1 warning (0.00 sec) SIMPLE;PRIMARY;UNION;DEPENDENT UNION;UNION RESULT; SUBQUERY;DEPENDENT SUBQUERY;DERIVED;MATERIALIZED system const eq ref ref fulltext ref or null index merge unique subquery index subquery range index ALL 19 www.percona.com
  • 26.
    EXPLAIN in details:keys mysql> explain extended select * from t1 join t2 where t1.int_key=1; ***+---------------+---------+---------+-------+*** ***| possible_keys | key | key_len | ref |*** ***+---------------+---------+---------+-------+*** ***| int_key,ik | int_key | 5 | const |*** ***| NULL | pk | 9 | NULL |*** ***+---------------+---------+---------+-------+*** 2 rows in set, 1 warning (0.00 sec) Keys, which can be used for resolving the query Only one key was actually used Actual length of the key (Important for multiple-column keys) Which columns were compared with the index Constant Numeric in our case Index used to resolve rows 20 www.percona.com
  • 27.
    EXPLAIN in details:rows mysql> explain extended select * from t1 join t2 where t1.int_key=1; ***+-------+----------+----------------------------------------------------+ ***| rows | filtered | Extra | ***+-------+----------+----------------------------------------------------+ ***| 4 | 100.00 | NULL | ***| 6 | 100.00 | Using index; Using join buffer (Block Nested Loop) | ***+-------+----------+----------------------------------------------------+ 2 rows in set, 1 warning (0.00 sec) % of rows filtered Number of rows accessed Additional information: how query is resolved Using filesort Using temporary etc. 4X6 = 24 All rows used 21 www.percona.com
  • 28.
    EXPLAIN FORMAT=JSON • Givesmore information than regular EXPLAIN • Real execution path of the query • Pushed conditions • Query cost • Temporary table and index creation are more precise • Reflects execution order of ”group by” and ”order by” operations • Displays table materializations 22 www.percona.com
  • 29.
    EXPLAIN FORMAT=JSON mysql> EXPLAINFORMAT=JSON SELECT user, host FROM mysql.userG *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "table": { "table_name": "user", "access_type": "index", "key": "PRIMARY", "used_key_parts": [ "Host", "User" ], "key_length": "228", "rows": 8, "filtered": 100, "using_index": true } } } 23 www.percona.com
  • 30.
    EXPLAIN PARTITIONS mysql> explainpartitions select count(*) -> from employees_part where hire_date > ’1991-01-01’G ************************ 1. row ************************ id: 1 select_type: SIMPLE table: employees_part partitions: p1,p2 type: index possible_keys: NULL key: PRIMARY key_len: 7 ref: NULL rows: 135214 Extra: Using where; Using index 24 www.percona.com
  • 31.
    When EXPLAIN lies:Handler % mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select titles.* from titles join employees using (emp no) -> where title=’Senior Engineer’ and to date=’9999-01-01’ -> order by from date limit 10; +--------+-----------------+------------+------------+ | emp_no | title | from_date | to_date | +--------+-----------------+------------+------------+ | 235233 | Senior Engineer | 1985-02-01 | 9999-01-01 | ... mysql> SHOW STATUS LIKE ’Handler_read_%’; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_read_first | 1 | | Handler_read_key | 11 | ... | Handler_read_rnd_next | 443309 | +-----------------------+--------+ 7 rows in set (0.00 sec) 25 www.percona.com
  • 32.
    I S.OPTIMIZER TRACE •Debug trace of optimizer • Prints • join preparation • join optimization • considered execution plans • refine plan • More • join execution 26 www.percona.com
  • 33.
    I S.OPTIMIZER TRACEexample mysql> SET optimizer_trace="enabled=on"; Query OK, 0 rows affected (0.00 sec) mysql> select titles.* from titles join employees using (emp_no) -> where title=’Senior Engineer’ and to_date=’9999-01-01’ -> order by from_date limit 10G *************************** 1. row *************************** emp_no: 235233 title: Senior Engineer from_date: 1985-02-01 to_date: 9999-01-01 ... 27 www.percona.com
  • 34.
    I S.OPTIMIZER TRACEexample cont. mysql> SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACEG *************************** 1. row *************************** QUERY: select titles.* from titles join employees using (emp_no) where ... TRACE: { "steps": [ { "join_preparation": { "select#": 1, "steps": [ { "expanded_query": ... { "join_optimization": { "select#": 1, "steps": [ { "condition_processing": { ... 28 www.percona.com
  • 35.
    I S.OPTIMIZER TRACEexample cont. "considered_execution_plans": [ { "plan_prefix": [ ], "table": "‘employees‘", "best_access_path": { "considered_access_paths": [ { "access_type": "ref", "index": "PRIMARY", "usable": false, "chosen": false }, { "rows_to_scan": 269763, "access_type": "scan", "resulting_rows": 269763, "cost": 54754, "chosen": true } ... 29 www.percona.com
  • 36.
    I S.OPTIMIZER TRACEexample cont. "condition_filtering_pct": 100, "rows_for_plan": 269763, "cost_for_plan": 54754, ... "rest_of_plan": [ { "plan_prefix": [ "‘titles‘" ], "table": "‘employees‘", "best_access_path": { "considered_access_paths": [ { "access_type": "eq_ref", "index": "PRIMARY", "rows": 1, "cost": 5222.9, "chosen": true, "cause": "clustered_pk_chosen_by_heuristics" }, 30 www.percona.com
  • 37.
    I S.OPTIMIZER TRACEexample cont. { "access_type": "scan", "chosen": false, "cause": "covering_index_better_than_full_scan" } ] }, "condition_filtering_pct": 100, "rows_for_plan": 4352.4, "cost_for_plan": 93459, "chosen": true ... 31 www.percona.com
  • 38.
  • 39.
    Common concurrency issues •Query or transaction waits a lock, held by another one • Fight for system resources • Symptoms • Many processes are waiting • You see query in the slow log, but it runs fast if single-thread environment • Query execution time varies 33 www.percona.com
  • 40.
    Lock types andtransactions • Lock types • Levels • MDL • Table-level • Row-level • What do they lock • Read locks • Block writes • Write locks • Block reads and writes • Transactions • Server-level • MDL locks • Table locks • Engine-level • Table locks • Row locks • AUTOCOMMIT • supported 34 www.percona.com
  • 41.
    Locks diagnostic • SHOW[FULL] PROCESSLIST • SHOW ENGINE INNODB STATUS • INFORMATION SCHEMA • PROCESSLIST • InnoDB tables • Performance Schema • METADATA LOCKS • TABLE HANDLES • EVENTS TRANSACTIONS * • Both server-level and engine-level • Contain GTID information 35 www.percona.com
  • 42.
    Locks diagnostic • SHOW[FULL] PROCESSLIST • For any kind of locks mysql> select id, state, info from information_schema.processlistG ******************** 1. row ******************** id: 5 state: info: NULL ******************** 2. row ******************** id: 4 state: Waiting for table metadata lock info: alter table titles add column f1 int ******************** 3. row ******************** id: 2 state: executing info: select id, state, info from information_schema.processlist 35 www.percona.com
  • 43.
    Locks diagnostic • PerformanceSchema • Metadata locks mysql> select processlist_id, object_type, lock_type, lock_status, source -> from metadata_locks join threads on (owner_thread_id=thread_id) -> where object_schema=’employees’ and object_name=’titles’G *************************** 1. row *************************** processlist_id: 4 object_type: TABLE lock_type: EXCLUSIVE lock_status: PENDING -- waits source: mdl.cc:3263 *************************** 2. row *************************** processlist_id: 5 object_type: TABLE lock_type: SHARED_READ lock_status: GRANTED -- holds source: sql_parse.cc:5707 35 www.percona.com
  • 44.
    Locks diagnostic • PerformanceSchema • Table locks mysql2> select * from table_handlesG *************************** 1. row *************************** OBJECT_TYPE: TABLE OBJECT_SCHEMA: employees OBJECT_NAME: employees OBJECT_INSTANCE_BEGIN: 140544885988272 OWNER_THREAD_ID: 23 OWNER_EVENT_ID: 818320 INTERNAL_LOCK: NULL EXTERNAL_LOCK: READ EXTERNAL -- Table lock! 1 row in set (0.00 sec) 35 www.percona.com
  • 45.
    Locks diagnostic • PerformanceSchema • Transactions mysql> select thread_id, processlist_id, access_mode, gtid -> from events_transactions_current join threads using(thread_id) -> where state = ’active’G *************************** 1. row *************************** thread_id: 25 processlist_id: 2 access_mode: READ WRITE gtid: AUTOMATIC *************************** 2. row *************************** thread_id: 28 processlist_id: 5 access_mode: READ WRITE gtid: 07aff4c6-87e7-11e5-9a51-30b5c2208a0f:202 2 rows in set (0.00 sec) 35 www.percona.com
  • 46.
    InnoDB Monitors • SHOWENGINE INNODB STATUS • Permanent output • innodb status output • innodb status output locks • innodb-status-file - Deleted on normal shutdown! 36 www.percona.com
  • 47.
    InnoDB Monitors • SHOWENGINE INNODB STATUS ------------- TRANSACTIONS ------------- Trx id counter 0 26243837 Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0 History list length 2 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index mysql tables in use 1, locked 1 LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s) MySQL thread id 4485, query id 25022137 localhost root Updating update t set a=36 where a=6 36 www.percona.com
  • 48.
    Other InnoDB diagnostic •INFORMATION SCHEMA • INNODB TRX • INNODB LOCKS • INNODB LOCK WAITS • INNODB METRICS • Options innodb monitor * • Option innodb print all deadlocks 37 www.percona.com
  • 49.
    Other InnoDB diagnostic •INFORMATION SCHEMA • mysql> SELECT TRX_ID, TRX_REQUESTED_LOCK_ID, BLOCKING_TRX_ID, -> BLOCKING_LOCK_ID, TRX_QUERY -> FROM INNODB_TRX JOIN INNODB_LOCK_WAITS ON -> (INNODB_TRX.TRX_ID=INNODB_LOCK_WAITS.REQUESTING_TRX_ID) -> WHERE TRX_STATE = ’LOCK WAIT’G *************************** 1. row *************************** TRX_ID: 2318 TRX_REQUESTED_LOCK_ID: 2318:28:5:2 BLOCKING_TRX_ID: 2316 BLOCKING_LOCK_ID: 2316:28:5:2 TRX_QUERY: update titles set title=’Principal Engineer’ where title=’Senior Engineer’ 1 row in set (0.00 sec) mysql> select TRX_ID, trx_mysql_thread_id from INNODB_TRX WHERE TRX_ID=2316G *************************** 1. row *************************** TRX_ID: 2316 trx_mysql_thread_id: 2 1 row in set (0.00 sec) 37 www.percona.com
  • 50.
    Locks diagnostic summary •Table-level • PROCESSLIST: ”Waiting for table lock” • P S.TABLE HANDLES • Row-level • InnoDB monitors • SHOW ENGINE INNODB STATUS • Tables in INFORMATION SCHEMA • P S.EVENTS TRANSACTIONS • Option –innodb print all deadlocks • MDL • PROCESSLIST • ”Waiting for metadata lock” • P S.METADATA LOCKS 38 www.percona.com
  • 51.
  • 52.
    Memory usage • Whatuses memory in MySQL • Buffers • Temporary tables • Internal structures - out of user control! • OS did not show memory as freed yet? 40 www.percona.com
  • 53.
    Memory diagnostic before5.7 • free • top • vmstat • Investigation • There was no way to know how exactly memory was allocated 41 www.percona.com
  • 54.
    Memory diagnostic before5.7 • free $free total used free shared buffers cached Mem: 16149184 6223916 9925268 317536 1048 3655160 -/+ buffers/cache: 2567708 13581476 Swap: 2110460 0 2110460 • top • vmstat • Investigation 41 www.percona.com
  • 55.
    Memory diagnostic before5.7 • free • top $top Tasks: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.0 us, 0.8 sy, 0.1 ni, 95.4 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 s KiB Mem: 16149184 total, 6231688 used, 9917496 free, 1048 buffers KiB Swap: 2110460 total, 0 used, 2110460 free. 3670752 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1914 mysql 20 0 670m 95m 1296 S 0.7 1.2 2:42.14 mysqld • vmstat • Investigation 41 www.percona.com
  • 56.
    Memory diagnostic before5.7 • free • top • vmstat $vmstat -t 5 3 procs ----------------------memory---------------------- ------swap---- ------ r b swpd free buff cache si so bi bo in cs us sy id wa.. 2 0 0 9923160 1048 3662724 0 0 168 86 167 674 3 1 87... 0 0 0 9923252 1048 3662904 0 0 30 122 1168 5264 3 1 96... 0 0 0 9922864 1048 3663120 0 0 25 128 1191 5342 2 1 96... • Investigation 41 www.percona.com
  • 57.
    Memory diagnostic before5.7 • free • top • vmstat • Investigation • Total size of buffers • Number of temporary tables • Number of parallel connections 41 www.percona.com
  • 58.
    Memory diagnostic in5.7 mysql> select thread_id tid, user, current_allocated ca, total_allocated -> from sys.memory_by_thread_by_current_bytes; +-----+-------------------------+-------------+-----------------+ | tid | user | ca | total_allocated | +-----+-------------------------+-------------+-----------------+ | 1 | sql/main | 2.53 GiB | 2.69 GiB | | 150 | root@127.0.0.1 | 4.06 MiB | 32.17 MiB | | 146 | sql/slave_sql | 1.31 MiB | 1.44 MiB | | 145 | sql/slave_io | 1.08 MiB | 2.79 MiB | ... | 60 | innodb/io_read_thread | 0 bytes | 384 bytes | | 139 | innodb/srv_purge_thread | -328 bytes | 754.21 KiB | | 69 | innodb/io_write_thread | -1008 bytes | 34.28 KiB | | 68 | innodb/io_write_thread | -1440 bytes | 298.05 KiB | | 74 | innodb/io_write_thread | -1656 bytes | 103.55 KiB | | 4 | innodb/io_log_thread | -2880 bytes | 132.38 KiB | | 72 | innodb/io_write_thread | -7632 bytes | 1.10 MiB | +-----+-------------------------+-------------+-----------------+ 145 rows in set (2.65 sec) 42 www.percona.com
  • 59.
    Threads Statistics mysql> select* from sys.memory_by_thread_by_current_bytes -> order by current_allocated descG *************************** 1. row *************************** thread_id: 152 user: lj@127.0.0.1 current_count_used: 325 current_allocated: 36.00 GiB current_avg_alloc: 113.43 MiB current_max_alloc: 36.00 GiB total_allocated: 37.95 GiB ... 43 www.percona.com
  • 60.
    RAW Performance Schematables • memory summary by account by event name • memory summary by host by event name • memory summary by thread by event name • memory summary by user by event name • memory summary global by event name • sys schema also includes information about user name 44 www.percona.com
  • 61.
    Users in sys.memory* tables • NAME@HOST - regular user • System users • sql/main • innodb/* • ... • Data comes from table THREADS 45 www.percona.com
  • 62.
    Memory usage: howto fix? • mysqld uses to low memory • Buffers are not tuned for performance • Operating system limits resources • mysqld uses too much memory • Buffers are too large • General use buffers: allocated once for whole server • Connection buffers: allocated for every connection • Task buffers: allocated for the particular task • Prioritize! • Allocations, which cannot be controlled • Watch • Find workaround • Implement 46 www.percona.com
  • 63.
    Disk diagnostics • df •iostat • ls -l /proc/{PID OF MYSQLD}/fd • InnoDB status • performance schema.table io waits % • sys.io % 47 www.percona.com
  • 64.
    Disk diagnostics • df Filesystem1K-blocks Used Available Use% Mounted on /dev/sda6 41944064 28429816 11433384 72% / devtmpfs 8067632 0 8067632 0% /dev tmpfs 8074592 41300 8033292 1% /dev/shm tmpfs 8074592 2376 8072216 1% /run tmpfs 8074592 0 8074592 0% /sys/fs/cgroup /dev/sda6 41944064 28429816 11433384 72% /.snapshots /dev/sda6 41944064 28429816 11433384 72% /var/tmp /dev/sda6 41944064 28429816 11433384 72% /var/spool /dev/sda6 41944064 28429816 11433384 72% /var/opt /dev/sda6 41944064 28429816 11433384 72% /var/log /dev/sda6 41944064 28429816 11433384 72% /var/lib/pgsql /dev/sda6 41944064 28429816 11433384 72% /var/lib/named /dev/sda6 41944064 28429816 11433384 72% /var/lib/mailman /dev/sda6 41944064 28429816 11433384 72% /var/crash /dev/sda6 41944064 28429816 11433384 72% /usr/local /dev/sda6 41944064 28429816 11433384 72% /tmp ... 47 www.percona.com
  • 65.
    Disk diagnostics • iostat sveta@thinkie:~/src/test-db/employees_db>iostat -x Linux 3.16.7-92-desktop (thinkie) 11/11/2015 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.47 1.77 2.54 0.94 0.00 80.28 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz ... sda 0.28 0.70 4.66 21.59 93.69 346.10 33.51 4.14 ... sdb 0.00 0.00 0.00 0.00 0.02 0.00 15.69 0.00 ... 47 www.percona.com
  • 66.
    Disk diagnostics • ls-l /proc/{PID OF MYSQLD}/fd sveta@thinkie:~/src/test-db/employees_db> ls -l /proc/26676/fd total 0 lrwx------ 1 sveta users 64 Nov 11 15:11 0 -> /dev/pts/1 l-wx------ 1 sveta users 64 Nov 11 15:11 1 -> /log/mysqld.1.err lrwx------ 1 sveta users 64 Nov 11 15:11 10 -> /data/ib_logfile1 lrwx------ 1 sveta users 64 Nov 11 15:11 11 -> /data/ibtmp1 lrwx------ 1 sveta users 64 Nov 11 15:11 12 -> /tmp/mysqld.1/ibx2zTxG (deleted ... lrwx------ 1 sveta users 64 Nov 11 15:11 27 -> /data/sys/sys_config.ibd l-wx------ 1 sveta users 64 Nov 11 15:11 28 -> /data/master-bin.000002 lrwx------ 1 sveta users 64 Nov 11 15:11 29 -> socket:[1245731] lrwx------ 1 sveta users 64 Nov 11 15:11 3 -> /data/master-bin.index ... lrwx------ 1 sveta users 64 Nov 11 15:11 5 -> /tmp/mysqld.1/ibT4odfr (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 6 -> /tmp/mysqld.1/ibJoMdZu (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 7 -> /tmp/mysqld.1/ibvUBeJy (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 8 -> /tmp/mysqld.1/ibP6JxvC (deleted) lrwx------ 1 sveta users 64 Nov 11 15:11 9 -> /data/ib_logfile0 47 www.percona.com
  • 67.
    Disk diagnostics • InnoDBstatus mysql> show engine innodb statusG ... -------- FILE I/O -------- I/O thread 0 state: waiting for i/o request (insert buffer thread) I/O thread 1 state: waiting for i/o request (log thread) I/O thread 2 state: waiting for i/o request (read thread) I/O thread 3 state: waiting for i/o request (read thread) I/O thread 4 state: waiting for i/o request (write thread) I/O thread 5 state: waiting for i/o request (write thread) Pending normal aio reads: [0, 0] , aio writes: [0, 0] , ibuf aio reads:, log i/o’s:, sync i/o’s: Pending flushes (fsync) log: 0; buffer pool: 0 11468 OS file reads, 102 OS file writes, 17 OS fsyncs 11.80 reads/s, 16384 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ... 47 www.percona.com
  • 68.
    Disk diagnostics • InnoDBstatus -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=26676, Main thread ID=140424906065664, state: sleeping Number of rows inserted 19, updated 0, deleted 0, read 5843307 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1584.23 reads/s 47 www.percona.com
  • 69.
    Disk diagnostics mysql> selectOBJECT_SCHEMA, OBJECT_NAME, INDEX_NAME -> from table_io_waits_summary_by_index_usage where COUNT_STAR=0; +---------------+--------------+------------+ | OBJECT_SCHEMA | OBJECT_NAME | INDEX_NAME | +---------------+--------------+------------+ | employees | departments | PRIMARY | | employees | departments | dept_name | | employees | dept_emp | PRIMARY | | employees | dept_emp | emp_no | | employees | dept_emp | dept_no | | employees | dept_manager | PRIMARY | | employees | dept_manager | emp_no | | employees | dept_manager | dept_no | | employees | salaries | PRIMARY | | employees | salaries | emp_no | | sys | sys_config | PRIMARY | +---------------+--------------+------------+ 11 rows in set (0.00 sec) 47 www.percona.com
  • 70.
    Disk diagnostics • sys.io% mysql> select event_name, count_read, total_read, total_requested -> from io_global_by_wait_by_bytes; +-------------------------+------------+------------+-----------------+ | event_name | count_read | total_read | total_requested | +-------------------------+------------+------------+-----------------+ | innodb/innodb_data_file | 17349 | 274.36 MiB | 289.22 MiB | | sql/binlog | 16696 | 130.42 MiB | 130.42 MiB | | sql/io_cache | 1556 | 37.61 MiB | 100.49 MiB | | sql/FRM | 1137 | 626.03 KiB | 626.03 KiB | | innodb/innodb_log_file | 8 | 132.50 KiB | 141.50 KiB | | sql/ERRMSG | 3 | 72.23 KiB | 72.23 KiB | | sql/query_log | 0 | 0 bytes | 44.02 KiB | | myisam/dfile | 20 | 21.99 KiB | 21.99 KiB | | mysys/charset | 1 | 18.27 KiB | 18.27 KiB | | myisam/kfile | 24 | 6.62 KiB | 6.62 KiB | | sql/file_parser | 3 | 2.11 KiB | 2.11 KiB | | sql/slow_log | 0 | 0 bytes | 1.64 KiB | | mysys/cnf | 3 | 56 bytes | 56 bytes | ... 47 www.percona.com
  • 71.
    Disk diagnostics • sys.io% • Other useful views • host summary by file io • host summary by file io type • io by thread by latency • io global by file by bytes - Per-file • io global by file by latency • io global by wait by bytes • io global by wait by latency • latest file io - Latest IO operations • user summary by file io • user summary by file io type 47 www.percona.com
  • 72.
    How to fixIO issues • Optimize queries • InnoDB options, affecting writes • Binary log 48 www.percona.com
  • 73.
    How to fixIO issues • Optimize queries • InnoDB options, affecting writes • innodb adaptive flushing • innodb doublewrite • innodb flush log at trx commit • innodb flush method - Test! • innodb io capacity - Follow docs • innodb max dirty pages pct 48 www.percona.com
  • 74.
    How to fixIO issues • Optimize queries • Binary log • Formats • ROW • STATEMENT • MIXED • binlog row image • sync binlog 48 www.percona.com
  • 75.
    CPU diagnostics • top •iostat • ps 49 www.percona.com
  • 76.
    CPU diagnostics • top Tasks:298 total, 2 running, 296 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.2 us, 2.7 sy, 1.8 ni, 79.4 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 16149184 total, 15075560 used, 1073624 free, 1596 buffers KiB Swap: 2110460 total, 8252 used, 2102208 free. 9994956 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26676 sveta 20 0 1963204 155108 23300 S 100.0 0.960 0:43.73 mysqld 10365 sveta 20 0 3601524 1.555g 108488 S 25.00 10.10 224:21.91 firefox 29915 sveta 20 0 15476 2612 2120 R 12.50 0.016 0:00.02 top ... 49 www.percona.com
  • 77.
    CPU diagnostics • iostat sveta@thinkie:~/src/test-db/employees_db>iostat -x Linux 3.16.7-92-desktop (thinkie) 11/11/2015 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.47 1.77 2.54 0.94 0.00 80.28 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz ... sda 0.28 0.70 4.66 21.59 93.69 346.10 33.51 4.14 ... sdb 0.00 0.00 0.00 0.00 0.02 0.00 15.69 0.00 ... 49 www.percona.com
  • 78.
    CPU diagnostics • ps sveta@thinkie>ps -eo pid,user,comm,pcpu,pmem,vsz | grep mysqld 26676 sveta mysqld 1.4 0.9 1963204 26678 sveta mysqld 0.0 0.7 1765772 49 www.percona.com
  • 79.
    CPU: problem solving •Low load • Is your mysqld really active? • Check OS limits • ulimit • User options • Global options • Increase concurrency-related options • innodb thread concurrency • innodb read io threads • innodb write io threads • Max (2 X (number of CPU cores))! • High load • Move job to memory: increase buffers • Tune queries 50 www.percona.com
  • 80.
    Network: what isworth attention? • Stability • Bandwidth • Speed • RTT sveta@thinkie> ping www.percona.com -c 2 PING www.percona.com (74.121.199.234) 56(84) bytes of data. 64 bytes from www.percona.com (74.121.199.234): icmp_seq=1 ttl=54 time=18 64 bytes from www.percona.com (74.121.199.234): icmp_seq=2 ttl=54 time=18 --- www.percona.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 181.380/181.685/181.991/0.524 ms 51 www.percona.com
  • 81.
    Network:diagnostics • –log-warnings=2 • Sendhuge file (1G or larger) • tcpdump • P S.accounts, hosts, users • P S.host cache 52 www.percona.com
  • 82.
    Network:diagnostics • tcpdump $tcpdump -ien1 host master.mysql.com and port 33051 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes 22:28:12.195270 IP master.mysql.com.33051 > apple.60344: P 1752426772:1752426864(92) ack 1474226199 win 91 <nop,nop,timestamp 1939999898 649946687> 22:28:12.195317 IP apple.60344 > master.mysql.com.33051: . ack 92 win 65535 <nop,nop,timestamp 649946998 1939999898> ^C 2 packets captured 37 packets received by filter 0 packets dropped by kernel 52 www.percona.com
  • 83.
    Network:diagnostics • P S.accounts,hosts, users mysql> select user, host, current_connections as cur, -> total_connections as total from accounts; +------+-----------+-----+-------+ | user | host | cur | total | +------+-----------+-----+-------+ | foo | localhost | 0 | 3 | | root | localhost | 1 | 3 | | NULL | NULL | 14 | 17 | +------+-----------+-----+-------+ 3 rows in set (0.01 sec) 52 www.percona.com
  • 84.
    Network:diagnostics • P S.hostcache mysql> select IP, HOST, HOST_VALIDATED, SUM_CONNECT_ERRORS, -> COUNT_NAMEINFO_PERMANENT_ERRORS, COUNT_PROXY_USER_ERRORS, -> COUNT_DEFAULT_DATABASE_ERRORS, FIRST_SEEN, LAST_SEEN from host_cacheG ************************ 1. row ************************ IP: 10.159.169.27 HOST: foo.bar.com HOST_VALIDATED: YES SUM_CONNECT_ERRORS: 1 COUNT_NAMEINFO_PERMANENT_ERRORS: 0 COUNT_PROXY_USER_ERRORS: 0 COUNT_DEFAULT_DATABASE_ERRORS: 1 FIRST_SEEN: 2014-06-10 20:29:45 LAST_SEEN: 2015-11-11 15:32:44 52 www.percona.com
  • 85.
  • 86.
    Further reading • MySQLTroubleshooting book • High Performance MySQL book • Planet MySQL • MySQL User Reference Manual • Bug trackers • http://bugs.mysql.com • https://bugs.launchpad.net/percona- server/ 54 www.percona.com
  • 87.
  • 88.
    Place for yourquestions ??? 56 www.percona.com
  • 89.