Basic MySQL Troubleshooting for Oracle DBAs

Basic MySQL Troubleshooting
for Oracle DBAs
Sveta Smirnova
Senior Principal Technical Support Engineer
MySQL Support
September 29, 2014
Copyright © 2014, Oracle and/or its affiliates. All rriigghhttss rreesseerrvveedd.. ||

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Program Agenda
Introduction
Basic troubleshooting techniques
High concurrency issues
High availability solutions
More information
1
2
3
4
5

Introduction

MySQL architecture
Click icon to add picture
Base
- Installation layout
- Log files
Connectors
- Clients
- APIs
Optimizer
Caches & buffers
Storage engines
Management

Typical installation layout
• datadir
– Schema directory
– Table files: *frm, *ibd, *MYD, *MYI, *par, etc.
– Trigger files
– Schema
– …
– InnoDB shared tablespace
– Log files
– InnoDB log files
– Binary, relay logs
– Error log
– Slow query log
– General query log
Configurable
You can setup custom path for each
component
Including custom path for tables

Connectors
The way you connect to MySQL Server
• Clients
–MySQL CLI
–MySQL Workbench
–MySQL Enterprise Monitor (MEM)
–Oracle Enterprise Manager with MEM plugin
• APIs
–Exist for most popular programming languages
–C, C++, JDBC, PHP, Python, Net, ODBC

Plugins
MySQL Server is highly configurable via plugins
• Storage engines
• Full-text parsers
• Daemon
• INFORMATION_SCHEMA
• Semisynchronous Replication
• Audit
• Authentication
• Password-validation
• Protocol Trace
5.7

Storage engines
From troubleshooting point of view
• Own data
• Own index format
• Own locking model
• Own diagnostic
• Own log files
• CHECK TABLE

Basic troubleshooting techniques

MySQL Access Privilege System
• Privileged client cannot connect
• Unprivileged client can connect
• Privileged user cannot perform operation
• Unprivileged user has undesired access
Typical issues

Overview
• No roles by default, limited user limits
• All records are in the mysql database (schema)
• Pluggable authentication since version 5.5
• Connections
● TCP/IP with login-password
● Socket (Unix)
● Named pipe (Windows)

select user, host from mysql.user order by user
desc, host desc;
• Most descriptive host first, then wildcard
• Socket connection by default on Unix
Where to find out why you have connection issues?

Wrong access
SHOW GRANTS [FOR user@host]
• Grant tables
● mysql.db
● mysql.tables_priv
● mysql.columns_priv
● mysql.procs_priv
● mysql.proxies_priv
SELECT USER(), CURRENT_USER()

mysql> show grants;
+--------------------------------------------------------------------+
| Grants for root@localhost |
+--------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION|
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION |
+--------------------------------------------------------------------+
Wrong access

mysql> show grants for foo@'%';
+-----------------------------------------------------+
| Grants for foo@% |
+-----------------------------------------------------+
| GRANT USAGE ON *.* TO 'foo'@'%' |
| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%' |
+-----------------------------------------------------+
Wrong access

Error handling
Errors vs warnings
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist. Check the
'Function Name Parsing and Resolution' section in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+-----------+--------+----------------------------------------------------+
| Level | Code | Message |
+-----------+--------+----------------------------------------------------+
| Warning | 1292 | Truncated incorrect DOUBLE value: 'f1' |
+-----------+--------+----------------------------------------------------+
1 row in set (0.00 sec)

Application (C API)
• Error information
● mysql_errno
● mysql_error
• Warnings
● mysql_info
● mysql_sqlstate
● mysql_warning_count

Error handling
perror
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the 'Function Name Parsing and
Resolution' section in the Reference Manual
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated
incorrect %.32s value: '%.128s'
OS error code 2: No such file or directory
MySQL error code 150: Foreign key constraint is incorrectly formed

Error handling
In stored routines
● GET DIAGNOSTICS
● GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;
● GET DIAGNOSTICS CONDITION 1 code = RETURNED_SQLSTATE,
msg = MESSAGE_TEXT;
● http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html

What can affect query execution?
You run a query, it does not return an error, but still behaves not as expected
• Startup options or system variables
• How optimizer creates query plan
• Storage engine used
• Parallel execution next section

System variables and startup options
Scope
• Global
● Control parameters, necessary for all server processes
– Location of server files: datadir etc.
– Shared buffers
– More
• Session
● Control connection-specific parameters
• http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html

• SET [GLOBAL] var_name = NEW_VAL
• Command line option
• Configuration file
● In default location
– http://dev.mysql.com/doc/refman/5.6/en/option-files.html
● Specified by option --
defaults-file
How to set

Who can change
• Global options and few session options
● A user with privilege SUPER
• Session options
● Anybody
• There is no limits!

When allocated
• Those which control behavior of whole server
● Once at server startup
● Can start with low values, then grow to specified
• Connection options
● For every connection when connection opens
• Operation-specific
● For every operation when needed

How to control
• SHOW [GLOBAL] STATUS
• GLOBAL
● Since server start
• SESSION
● For operations in current session
● Can be reset
– FLUSH STATUS

How to control
mysql> show global status like 'Handler_read_rnd_next'G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 27
mysql> show status like 'Handler_read_rnd_next'G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 7

Troubleshooting best practices
• Record currently used variables
● SHOW [GLOBAL] VARIABLES
• Make change dynamically if possible
● SET [GLOBAL] var_name=NEW_VAL
• Test in one session first
• Then change global variable
• Change configuration file after you are happy with results

When affecting option is not known
• Record currently used variables
● SHOW [GLOBAL] VARIABLES
• Start mysqld with option --no-defaults
● This option must be first one!
• Check if problem is solved
• Change variable values one-by-one until you find one which leads to the
problem

Diagnostic tool: INFORMATION_SCHEMA
When affecting option is not known
• Contain metadata information
● Tables
● Indexes
● Other
• Allows to create plugins
● InnoDB plugins
– We will discuss them later
• Oracle analog
● Data Dictionary Views

MySQL Optimizer
Overview
• EXPLAIN is less powerful if compare with Oracle
● It is improved in version 5.7.3
● Graphic EXPLAIN in MySQL Workbench 6.0
• EXPLAIN EXTENDED
• EXPLAIN PARTITIONS
• EXPLAIN FORMAT=JSON
• INFORMATION_SCHEMA.TRACE
• Status variables 'Handler_%'
Default in 5.7

http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm
EXPLAIN in Oracle
EXPLAIN PLAN SET statement_id = 'example_plan4' FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
AND h.date_ordered > SYSDATE30
AND l.header_id = h.header_id ;
Plan
--------------------------------------------------
SELECT STATEMENT
NESTED LOOPS
TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
INDEX RANGE SCAN SO_HEADERS_N1
TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
INDEX RANGE SCAN SO_LINES_N1

EXPLAIN in MySQL
mysql> EXPLAIN SELECT user, host FROM mysql.userG
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
Extra: Using index

EXPLAIN in MySQL
Variant: EXTENDED
mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.userG
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
filtered: 100.00
Extra: Using index
1 row in set, 1 warning (0.00 sec)
mysql> SHOW WARNINGSG
******* 1. row *******
Level: Note
Code: 1003
Message: /* select#1 */
select
`mysql`.ùser`.Ùser` AS
ùser`,`mysql`.ùser`.`H
ost` AS `host` from
`mysql`.ùser`

EXPLAIN in MySQL
Variant: FORMAT=JSON
mysql> EXPLAIN FORMAT=JSON SELECT
user, host FROM mysql.userG
************** 1. row **************
EXPLAIN: {
"query_block": {
"select_id": 1,
"table": {
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
...
...
"used_key_parts": [
"Host",
"User"
],
"key_length": "228",
"rows": 4,
"filtered": 100,
"using_index": true
}
}
}
1 row in set, 1 warning (0.03 sec)

EXPLAIN in MySQL
More information
• http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
• http://dev.mysql.com/doc/refman/5.6/en/explain-output.html
• MySQL EXPLAIN in Practice [HOL9232] (past)
• MySQL 5.7: What’s New in the Parser and the Optimizer? [CON2830] (past)
• How to Analyze and Tune MySQL Queries for Better Performance
[TUT3157] (past)
• Using MySQL Workbench Performance Tools [HOL9237] (past)

MySQL Optimizer
Handler_% status variables
mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Handler_read_first | 0 |
| Handler_read_key | 0 |
| Handler_read_last | 0 |
| Handler_read_next | 0 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 0 |
| Handler_read_rnd_next | 0 |
+----------------------------+-------+

MySQL Optimizer
Handler_% status variables
mysql> select count(*) from employees join titles using(emp_no) where title='Senior Engineer'G
*************************** 1. row ***************************
count(*): 97750
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+--------+
| Variable_name | Value |
+----------------------------+--------+
| Handler_read_first | 1 |
| Handler_read_key | 300027 |
| Handler_read_last | 0 |
| Handler_read_next | 397774 |
...

MySQL Optimizer
Other tools
• INFORMATION_SCHEMA.TRACE
● http://dev.mysql.com/doc/internals/en/optimizer-tracing.html
● join_preparation, join_optimization,
join_execution
– considered_execution_plans, refine_plan, more
• Query Analyzer in MEM
● http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html
• Graphic EXPLAIN in MySQL Workbench 6.0

MySQL Storage Engines
Specifics
• Own way to handle
● Corruption
● Index statistics
● CHECK TABLE to check for errors
• They care about physical data, so all data information are on their level
• Options usually start from engine name
● myisam_*, innodb_*, custom_*

MySQL Storage Engines
InnoDB
• Transactional storage engine
• Physical layout
● *frm file – table definition
● Shared tablespace
● *ibd file – tablespace for individual table
– Optional: --innodb_file_per_table
– Recommended and default since 5.6
● Redo log files
• OPTIMIZE TABLE = ALTER + ANALYZE
• Automatic startup check

• Query or transaction waits a lock, held by another one
• Fight for system resources
• Resource overload
• Resource underload
Common problems

Transactions and their relation to locks
• Lock types
• Levels
–MDL
– Table-level
– Row-level
• What do they lock
–Read locks
– Block writes
–Write locks
– Block both reads and writes
● Transactions
● Server-level
– MDL locks
● Engine level
– Table locks
– Row locks
● AUTOCOMMIT
● Supported

Locking issues
Diagnostic tools
• SHOW [FULL] PROCESSLIST
● Universal tool which lists all currently running connections
• SHOW ENGINE INNODB STATUS
• INFORMATION SCHEMA
● PROCESSLIST
● InnoDB tables
• PERFORMANCE SCHEMA
● MDL locks
● Table locks
● Server-level transactions

Storage engine specifics
Diagnostic tools
• Transactions and engine-level locks are done at the engine level
• Storage engines have own diagnostic tools
• Use them when hit an issue with such a lock

• SHOW ENGINE INNODB STATUS
• InnoDB monitors
● Pseudo-tables
– Turn periodical logging into error log
● CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;
– Lock monitor
● Changes output format of InnoDB Monitor
● CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;
● Options
– innodb_status_output
– innodb_status_output_locks
InnoDB

InnoDB
• Tables in INFORMATION_SCHEMA
● INNODB_TRX
● INNODB_LOCKS
● INNODB_LOCK_WAITS
● INNODB_METRICS
– Options innodb_monitor_*
● Option innodb_print_all_deadlocks

Diagnostic tool: performance_schema
For whole server
• Monitors internal operations
● Events
● Waits
● Mutexes
● Statements
● Stages
● MDL, table-level locks, transactions
● Memory
● Replication
• Similar to Oracle wait interface
5.7+

Diagnosting locks
Summary
• Table-level locks
● PROCESSLIST: “Waiting for table lock”
● P_S.TABLE_HANDLES
• Row-level locks
5.7+
● InnoDB monitors, SHOW ENGINE INNODB STATUS
● Tables in INFORMATION_SCHEMA
● Option --innodb_print_all_deadlocks
• MDL locks
● PROCESSLIST: “Waiting for metadata lock”
● P_S.METADATA_LOCKS
5.7+

High availability solutions

MySQL Cluster
• Special storage engine: NDB
• Stores data on two or more physical machines
• Two or more copies
• General troubleshooting techniques
● Applicable
• Specific NDB storage engine techniques

MySQL Replication
Overview
• Always available, but you must setup it before use
• Asynchronous master-slave
• Master
● Keeps all updates in separate file: binary log
• Slave
● IO thread read updates from master and stores in relay log file
● SQL thread executes updates

Replication IO thread
Communication issues
• Access error
● Check slave error log
● SHOW SLAVE STATUS
● P_S.replication_connection_status
● Try to connect using normal MySQL client using slave credentials
– SHOW GRANTS
● Fix privileges on master
● Restart slave

Replication SQL thread
Typical issues
• Simple master-slave
● Data is different on master and slave
– Replication event can not be applied
● Different errors on master and slave
● Slave lags far behind the master
• Circular replication or other writes in addition to slave SQL thread
● Data is different on master and slave

Data is different on master and slave
• Was the table modified besides the SQL thread?
● How?
● Can it affect content of the table in the wrong way?
• Are the table definitions same on master and slave?
● MySQL Utilities
– mysqlrplsync, mysqldbcompare, mysqldiff
• Is it possible that master events was applied in wrong order?
● Use mysqlbinlog to find queries which caused the issue
● Check master's application to find what is wrong

Events from master were applied in wrong order
• Lock issues
● InnoDB
• Triggers
● SET GLOBAL slave_skip_counter
● Synchronize tables!
• Different options
● Start slave with master's options, then check

Slave lags far behind the master
• Threads
● Master runs in multiple update threads
● Slave uses single
• Seconds_behind_master is growing
• Tune slave performance
● Buffers
● Indexes (for statement-based replication)

More information

OpenWorld sessions
October, 01
• Practical MySQL Optimization [CON5522]
● Peter Zaitsev, 3:30 PM, Moscone South - 262
• What’s New in MySQL 5.7 Security [CON1985]
● Georgi Kodinov, 4:45 PM, Moscone South - 262
• MySQL Cost Model [CON3163]
● Olav Sandstå, 5:30 PM, Moscone South - 250

OpenWorld sessions
October, 02
• MySQL Troubleshooting with the Performance Schema [HOL9215]
● Sveta Smirnova, Lig Isler-turmelle
● 8:30 AM, Hotel Nikko - Monterey
• MySQL Enterprise Edition Features in Practice [HOL9216]
● Matt Lord, 10:00 AM, Hotel Nikko - Monterey
• MySQL Performance: Demystified Tuning and Best Practices [CON5097]
● Dimitri Kravtchuk, 10:45 AM, Moscone South - 252

Further reading
• MySQL Troubleshooting book
● http://shop.oreilly.com/product/0636920021964.do
• Marc Alff's Performance Schema blog
● http://marcalff.blogspot.ru/
• Planet MySQL
● http://planet.mysql.com/
● http://dev.mysql.com/support/blogs/
● http://mysqlserverteam.com/

References
• MySQL User Reference Manual
● http://dev.mysql.com/doc/refman/5.6/en/index.html
• Knowledge Management Database
• Forums
● http://forums.mysql.com
• Bug trackers
● http://bugs.mysql.com
● Oracle Internal Bugs database
• My Oracle Support
● https://support.oracle.com

You will find me at
• https://twitter.com/svetsmirnova
• https://blogs.oracle.com/svetasmirnova/
• http://www.slideshare.net/SvetaSmirnova/

?

Thank you!

Basic MySQL Troubleshooting for Oracle DBAs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Basic MySQL Troubleshooting for Oracle DBAs

Similar to Basic MySQL Troubleshooting for Oracle DBAs (20)

More from Sveta Smirnova

More from Sveta Smirnova (20)

Recently uploaded

Recently uploaded (20)

Basic MySQL Troubleshooting for Oracle DBAs