This document provides an overview of basic MySQL troubleshooting techniques for Oracle database administrators. It covers MySQL server architecture including connectors, clients, APIs, storage engines, and plugins. It then discusses basic troubleshooting techniques such as error processing, access privileges, and using system variables, performance schema, and EXPLAIN to analyze query execution plans. The document is intended to help Oracle DBAs understand fundamental aspects of MySQL administration.

1. Basic MySQL Troubleshooting
for Oracle Database Administrators
Sveta Smirnova
Principal Support Engineer
October, 29, 2015

2. Table of Contents
•MySQL Server overview
•Basic troubleshooting techniques
•High concurrency issues
•Replication
2 www.percona.com

3. MySQL Server overview
3 www.percona.com

4. MySQL architecture
Connectors: C, JDBC, ODBC, Python, ...
Connection Pool: Authentication, Caches
SQL interface
Parser
Optimizer
Caches and Buffers:
Global
Engine-specific
Storage engines: InnoDB, TokuDB, ...
File system: Data, Index, logs, other files
• Base
• Installation layout
• Log files
• Connectors
• Clients, APIs
• Optimizer
• Cache and buffers
• Storage engines
• Management
4 www.percona.com

5. Typical installation layout
• datadir
• Schema directory
• Table and trigger files: *.frm, *.ibd, *.MYD,
*.par, etc.
• Schema
• ...
• InnoDB shared tablespace
• Log files
• InnoDB redo log files
• Binary, relay log files
• Error log
• Slow query log
• General query log
• Configurable
• You can setup
custom path for
each component
• Including custom
paths for tables
5 www.percona.com

6. Connectors for MySQL server
• Clients
• MySQL CLI
• MySQL Workbench
• Many other graphical and web-based
• APIs
• Exist for most popular programming languages
• C, C++, JDBC, PHP, Python, Net, ODBC, more
6 www.percona.com

7. Monitoring solutions
• Command-line
• Percona Toolkit
• MySQL Utilities
• With GUI
• MySQL Enterprise Monitor (MEM)
• MEM plugin for Oracle Enterprise Manager
• VividCortex
• Many others
7 www.percona.com

8. Plugins
• Storage engines
• Authentication
• Audit
• Query rewrite
• More
8 www.percona.com

9. Storage engines
• Own data
• Own index format
• Own locking model
• Own diagnostic
• Own log files
• CHECK TABLE
9 www.percona.com

10. Basic troubleshooting
techniques
10 www.percona.com

11. Error processing
• Warnings, errors, and notes
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist.
Check the ’Function Name Parsing and Resolution’ section
in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+-----------+--------+----------------------------------------------------+
| Level | Code | Message |
+-----------+--------+----------------------------------------------------+
| Warning | 1292 | Truncated incorrect DOUBLE value: ’f1’ |
+-----------+--------+----------------------------------------------------+
1 row in set (0.00 sec)
11 www.percona.com

12. Error handling in applications
• Error information
• mysql error
• mysql errno
• Warnings and notes
• mysql info
• mysql sqlstate
• mysql warning count
12 www.percona.com

13. Error handling in stored routines
• GET DIAGNOSTICS
• GET DIAGNOSTICS rows = ROW COUNT,
conditions = NUMBER;
• GET DIAGNOSTICS CONDITION 1 code =
RETURNED SQLSTATE, msg = MESSAGE TEXT;
• SIGNAL/RESIGNAL
• Can be used for custom errors
13 www.percona.com

14. More information about errors
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the ’Function Name Parsing and Resolution’
section in the Reference Manual
[sveta@delly ~]$ perror 1292
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE):
Truncated incorrect %.32s value: ’%.128s’
[sveta@delly ~]$ perror 2
OS error code 2: No such file or directory
[sveta@delly ~]$ perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed
14 www.percona.com

15. MySQL Access Privilege System
• No roles by default, limited user limits
• All records are in the mysql database (schema)
• Pluggable authentication since version 5.5
• Connections
• TCP/IP with login-password
• Socket (Unix)
• Named pipe (Windows)
15 www.percona.com

16. Common user access issues
• Privileged client cannot connect
• Unprivileged client can connect
• Privileged user cannot perform operation
• Unprivileged user has undesired access
16 www.percona.com

17. Common queries for access issues
mysql> select user, host from mysql.user order by user desc, host desc;
+------+------------+
| user | host |
+------+------------+
| root | localhost |
| root | delly |
| root | ::1 |
| root | 127.0.0.1 |
| foo | % |
| | localhost |
+------+------------+
6 rows in set (0.00 sec)
• Most descriptive host first, then wildcard
• Socket connection by default on Unix
17 www.percona.com

18. Wrong access checklist
• SHOW GRANTS [FOR user@host]
• Grant tables
• mysql.db
• mysql.tables priv
• mysql.columns priv
• mysql.procs priv
• mysql.proxies priv
• SELECT USER(), CURRENT USER()
18 www.percona.com

19. Wrong access example
mysql> show grants;
+--------------------------------------------------------------------+
| Grants for root@localhost |
+--------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO ’root’@’localhost’ WITH GRANT OPTION|
| GRANT PROXY ON ’’@’’ TO ’root’@’localhost’ WITH GRANT OPTION |
+--------------------------------------------------------------------+
2 rows in set (0.02 sec)
mysql> show grants for foo@’%’;
+-----------------------------------------------------+
| Grants for foo@% |
+-----------------------------------------------------+
| GRANT USAGE ON *.* TO ’foo’@’%’ |
| GRANT ALL PRIVILEGES ON ‘test‘.* TO ’foo’@’%’ |
+-----------------------------------------------------+
2 rows in set (0.02 sec)
19 www.percona.com

20. Connection statistics in P S
• Tables accounts, users, hosts
mysql> select user, host, current_connections as cur,
-> total_connections as total from performance_schema.accounts;
+------+-----------+-----+-------+
| user | host | cur | total |
+------+-----------+-----+-------+
| foo | localhost | 0 | 3 |
| root | localhost | 1 | 3 |
| NULL | NULL | 14 | 17 |
+------+-----------+-----+-------+
3 rows in set (0.01 sec)
• HOST CACHE
20 www.percona.com

21. Connection statistics in P S
• Tables accounts, users, hosts
• HOST CACHE
• Content of DNS cache
• Errors from:
• Name Server
• Connection
• Authentication
• max connect errors, max user errors, etc.
• Your first assistant in case of connection issue
20 www.percona.com

22. Performance Schema
• Monitors interval operations
• Statements
• Stages
• Locks
• Memory
• Variables
• Replication
• IO
• Mutexes and waits
• Similar to Oracle wait interface
21 www.percona.com

23. What can affect query execution?
• You run a query, it does not return an error, but still
behaves not as expected
• It can be:
• Startup options or system variables
• How optimizer creates query plan
• Storage engine used
• Parallel execution - next big section
22 www.percona.com

24. System variables and options: scope
• Global
• Control parameters, necessary for all server
processes
• Location of server files: datadir etc.
• Shared buffers
• More
• Session
• Control connection-specific parameters
• MySQL option tables
23 www.percona.com

25. System Variables: how to set
• SET [GLOBAL] var name = NEW VAL
• Command-line option
• Configuration file
• In default location
• Specified by option –defaults-file
24 www.percona.com

26. System Variables: who can change
• Global options and few session options
• A user with privilege SUPER
• Session options
• Anybody
• There are no limits!
25 www.percona.com

27. Buffers: when allocated
• Those which control behavior of whole server
• Once at server startup
• Can start with low values, then grow to specified
• Connection options
• For every connection when connection opens
• Operation-specific
• For every operation when needed
• Can be allocated more than once per query
26 www.percona.com

28. System variables: control before 5.7
• SHOW [GLOBAL] STATUS
• Tables in Information Schema
• GLOBAL ,SESSION VARIABLES
• GLOBAL ,SESSION STATUS
• GLOBAL
• Since server start
• SESSION
• For operations in current session
• Can be reset
• FLUSH STATUS
27 www.percona.com

29. System status variables: example
mysql> show global status like ’Handler_read_rnd_next’G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 27
1 row in set (0.00 sec)
mysql> show status like ’Handler_read_rnd_next’G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 7
1 row in set (0.00 sec)
28 www.percona.com

30. Information Schema
• Contains metadata information
• Tables
• Indexes
• Other
• Allows to create plugins
• InnoDB plugins
• Similar to Data Dictionary Views in Oracle
29 www.percona.com

31. System variables: control in 5.7
• Performance Schema tables
• variabes by*
• user variables by*
• status by*
• Statistics grouped by
• Global
• Session
• Thread
• Account/Host/User
30 www.percona.com

32. System variables: best practices
• Record currently used variables
• SHOW [GLOBAL] VARIABLES
• Make change dynamically if possible
• SET [GLOBAL] var name=NEW VAL
• Test in one session first
• Then change global variable
• Change configuration file after you are happy with
results
31 www.percona.com

33. When affecting option is not known
• Record currently used variables
• SHOW [GLOBAL] VARIABLES
• Start mysqld with option –no-defaults
• This option must be first one!
• Check if problem is solved
• Change variable values one-by-one until you find one
which leads to the problem
32 www.percona.com

34. MySQL Optimizer
• EXPLAIN is less powerful if compare with Oracle
• Visual EXPLAIN in MySQL Workbench
• EXPLAIN EXTENDED
• Should be followed by SHOW WARNINGS
• EXPLAIN PARTITIONS
• EXPLAIN FORMAT=JSON
• INFORMATION SCHEMA.TRACE
• Status variables ’Handler %’
33 www.percona.com

35. EXPLAIN in Oracle
• http://docs.oracle.com/cd/B10500_01/server.
920/a96533/ex_plan.htm
EXPLAIN PLAN SET statement_id = ’example_plan4’ FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
AND h.date_ordered > SYSDATE30
AND l.header_id = h.header_id ;
Plan
--------------------------------------------------
SELECT STATEMENT
NESTED LOOPS
TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
INDEX RANGE SCAN SO_HEADERS_N1
TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
INDEX RANGE SCAN SO_LINES_N1
34 www.percona.com

36. EXPLAIN in MySQL
mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.userG
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
filtered: 100.00
Extra: Using index
1 row in set, 1 warning (0.00 sec)
mysql> SHOW WARNINGSG
******* 1. row *******
Level: Note
Code: 1003
Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mys
35 www.percona.com

37. EXPLAIN: overview
mysql> explain extended select * from t1 join t2 where t1.int_key=1;
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+-----------
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | f... | Extra
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+-----------
| 1 | SIMPLE | t1 | ref | int_key,ik | int_key | 5 | const | 4 | 100. | NULL
| 1 | SIMPLE | t2 | index | NULL | pk | 9 | NULL | 6 | 100. | Using inde
Using join
(Block Nes
+----+-------------+-------+-------+---------------+---------+---------+-------+------+------+-----------
2 rows in set, 1 warning (0.00 sec)
Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘ AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,‘test
AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘ from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘
Number of select
Select type
Tables, for which information is printed
How data is accessed
Possible keys
Key, which was actually used
Length of the key
Which columns were compared with the index
Number of examined rows
% of filtered rows
rows x filtered / 100 — number of rows,
which will be joined with another table
Additional information
Table, for which information is printed
Product of rows here: how many rows in all tables will be accessed
For this example estimated value is 4*6 = 24
Actual (optimized) query as executed by MySQL Server
36 www.percona.com

38. EXPLAIN FORMAT=JSON
mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.userG
*************************** 1. row ***************************
EXPLAIN: {
"query_block": {
"select_id": 1,
"table": {
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
"used_key_parts": [
"Host",
"User"
],
"key_length": "228",
"rows": 8,
"filtered": 100,
"using_index": true
}
}
}
37 www.percona.com

39. When EXPLAIN lies: Handler % variables
mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> select count(*) from employees join titles using(emp_no)
-> where title=’Senior Engineer’G
*************************** 1. row ***************************
count(*): 97750
1 row in set (3.24 sec)
mysql> SHOW STATUS LIKE ’Handler_read_%’;
+----------------------------+--------+
| Variable_name | Value |
+----------------------------+--------+
| Handler_read_first | 1 |
| Handler_read_key | 300027 |
| Handler_read_last | 0 |
| Handler_read_next | 397774 |
...
38 www.percona.com

40. MySQL Optimizer: other tools
• INFORMATION SCHEMA.OPTIMIZER TRACE
• join preparation, join optimization, join execution
• considered execution plans, refine plan, more
• Query Analyzer in MEM
• Visual EXPLAIN in MySQL Workbench
39 www.percona.com

41. MySQL Storage Engines
• Own way to handle
• Corruption
• Index statistics
• CHECK TABLE to check for errors
• They care about physical data, so all data information is
on their level
• Options usually start from engine name
• myisam *, innodb *, custom *
40 www.percona.com

42. Storage Engines explained
• InnoDB
• TokuDB
• Popular non-transactional engines
• MyISAM - Previous default engine
• Memory
• Federated
• CSV
• Blackhole
41 www.percona.com

43. Storage Engines explained
• InnoDB
• Most used storage engine
• Physical layout
• *frm file – table definition
• Shared tablespace
• *ibd file – tablespace for individual table - –innodb file per table = 1
• Redo log files
• Automatic startup check
41 www.percona.com

44. Storage Engines explained
• InnoDB
• Own locking model: row-based
• Troubleshooting tools
• InnoDB Monitors (SHOW ENGINE InnoDB STATUS)
• Tables in Information Schema
• Instrumentation in Performance Schema
41 www.percona.com

45. Storage Engines explained
• TokuDB
• Fractal tree
• Optimized for compression and writes
• Troubleshooting tools
• SHOW ENGINE tokudb STATUS
• Information schema tables
• Status variables
41 www.percona.com

46. Storage Engines explained
• Popular non-transactional engines
• Server provides locking model
• Table-level locks
• Can have own instrumentation
• MyISAM
• Status variables
• Performance Schema
• Set of utilities, names start from myisam *
41 www.percona.com

47. High concurrency issues
42 www.percona.com

48. Common concurrency issues
• Query or transaction waits a lock, held by another one
• Fight for system resources
• Resource overload
• Resource underload
43 www.percona.com

49. Lock types and transactions
• Lock types
• Levels
• MDL
• Table-level
• Row-level
• What do they lock
• Read locks
• Block writes
• Write locks
• Block reads and writes
• Transactions
• Server-level
• MDL locks
• Table locks
• Engine-level
• Table locks
• Row locks
• AUTOCOMMIT
• supported
44 www.percona.com

50. Locks diagnostic
• SHOW [FULL] PROCESSLIST
• SHOW ENGINE INNODB STATUS
• INFORMATION SCHEMA
• PROCESSLIST
• InnoDB table
• Performance Schema
• METADATA LOCKS
• TABLE HANDLES
• EVENTS TRANSACTIONS *
• Both server-level and engine-level
• Contain GTID information
45 www.percona.com

51. Locks diagnostic
• SHOW [FULL] PROCESSLIST
mysql> select id, state, info from information_schema.processlistG
******************** 1. row ********************
id: 5
state:
info: NULL
******************** 2. row ********************
id: 4
state: Waiting for table metadata lock
info: alter table titles add column f1 int
******************** 3. row ********************
id: 2
state: executing
info: select id, state, info from information_schema.processlist
45 www.percona.com

52. Locks diagnostic
• Performance Schema
mysql> select processlist_id, object_type, lock_type, lock_status, source
-> from metadata_locks join threads on (owner_thread_id=thread_id)
-> where object_schema=’employees’ and object_name=’titles’G
*************************** 1. row ***************************
processlist_id: 4
object_type: TABLE
lock_type: EXCLUSIVE
lock_status: PENDING -- waits
source: mdl.cc:3263
*************************** 2. row ***************************
processlist_id: 5
object_type: TABLE
lock_type: SHARED_READ
lock_status: GRANTED -- holds
source: sql_parse.cc:5707
45 www.percona.com

53. Locks diagnostic
• SHOW ENGINE INNODB STATUS
-------------
TRANSACTIONS
-------------
Trx id counter 0 26243837
Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0
History list length 2
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s)
MySQL thread id 4485, query id 25022137 localhost root Updating
update t set a=36 where a=6
45 www.percona.com

54. InnoDB Monitors
• SHOW ENGINE INNODB STATUS
• Permanent output
• innodb status output
• innodb status output locks
• innodb-status-file - Deleted on normal shutdown!
46 www.percona.com

55. Other InnoDB diagnostic
• INFORMATION SCHEMA
• INNODB TRX
• INNODB LOCKS
• INNODB LOCK WAITS
• INNODB METRICS
• Options innodb monitor *
• Option innodb print all deadlocks
47 www.percona.com

56. Locks diagnostic summary
• Table-level
• PROCESSLIST: ”Waiting for table lock”
• P S.TABLE HANDLES
• Row-level
• InnoDB monitors
• SHOW ENGINE INNODB STATUS
• Tables in INFORMATION SCHEMA
• Option –innodb print all deadlocks
• MDL
• PROCESSLIST
• ”Waiting for metadata lock”
• P S.METADATA LOCKS
48 www.percona.com

57. Replication
49 www.percona.com

58. MySQL Replication Overview
• Always available, needs to be setup
• Asynchronous master-slave
• Master
• Keeps all updates in binary log file
• Two formats: ROW and STATEMENT
• Slave
• IO thread read updates from master and stores in
relay log file
• SQL thread executes updates
• Multiple SQL threads since 5.6
• Multiple masters since 5.7
• GTIDs since 5.6
50 www.percona.com

59. Main troubleshooting tools
• SHOW SLAVE STATUS
• Tables in Performance Schema and system database
• Multi-threaded slave (MTS) has additional output for
each worker
• Error log file
51 www.percona.com

60. Main troubleshooting tools
• SHOW SLAVE STATUS
mysql> show slave status G
************************ 1. row ************************
Slave_IO_State: Queueing master event to the relay log
...
Master_Log_File: master-bin.000003
Read_Master_Log_Pos: 191
Relay_Log_File: slave-relay-bin.000006
Relay_Log_Pos: 4
Relay_Master_Log_File: master-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Last_Errno: 0
Last_Error:
...
Exec_Master_Log_Pos: 319
...
51 www.percona.com

61. Main troubleshooting tools
• Tables in Performance Schema and system database
• replication connection configuration
• replication applier configuration
• replication connection status
• replication applier status
• replication applier status by coordinator
• replication applier status by worker - MTS only
• mysql.slave master info
• mysql.slave relay log info
• mysql.slave worker info
51 www.percona.com

62. Main troubleshooting tools
• Multi-threaded slave (MTS)
mysql> select * from replication_applier_status_by_workerG
*************************** 1. row ***************************
CHANNEL_NAME:
WORKER_ID: 1
THREAD_ID: 25
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
*************************** 2. row ***************************
CHANNEL_NAME:
WORKER_ID: 2
THREAD_ID: 26
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: d0753e78-14ec-11e5-b3fb-28b2bd7442fd:770
...
51 www.percona.com

63. IO thread: communication issues
• Check slave error log
• SHOW SLAVE STATUS
• P S.replication connection status
• Access error
• Try to connect using normal MySQL client using
slave credentials
• SHOW GRANTS
• Fix privileges on master
• Restart slave
52 www.percona.com

64. SQL thread: typical issues
• Simple master-slave
• Data is different on master and slave
• Replication event can not be applied
• Different errors on master and slave
• Slave lags far behind the master
• Circular replication or other writes in addition to slave
SQL thread
• Data is different on master and slave
53 www.percona.com

65. Data is different on master and slave
• Was the table modified besides the SQL thread?
• How?
• Can it affect content of the table in the wrong way?
• Are the table definitions same on master and slave?
• MySQL Utilities
• mysqlrplsync, mysqldbcompare, mysqldiff
• Maybe master events were applied in wrong order?
• Use mysqlbinlog to find queries caused the issue
• Check application, running on master
54 www.percona.com

66. Events applied in different order
• Happens only with STATEMENT format
• Row lock issues
• Triggers
• SET GLOBAL slave skip counter - No GTIDs!
• Skip transaction - GTIDs
• Synchronize tables!
• Different options
• Start slave with master’s options, then check
• Happens in older versions
55 www.percona.com

67. Slave lags behind master
• Threads
• Master runs in multiple update threads
• Slave uses single
• Seconds behind master is growing - Not 100 % reliable!
• Tune slave performance
• Multi-threaded slave
• One thread per database in 5.6
• Affected by concurrency issues
• Indexes on slave only
• For statement-based replication
56 www.percona.com

68. More information
57 www.percona.com

69. Further reading
• MySQL Troubleshooting book
• Planet MySQL
• MySQL User Reference Manual
• Bug trackers
• http://bugs.mysql.com
• https://bugs.launchpad.net/percona-server/
58 www.percona.com

70. Place for your questions
???
59 www.percona.com

71. Thank you!
http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
60 www.percona.com