Basic MySQL Troubleshooting for Oracle DBAs

Basic MySQL
Troubleshooting for Oracle
DBAs
Sveta Smirnova
Principal Technical Support Engineer


Insert Picture Here

Program Agenda
 Introduction
 Basic single-client issues
 Concurrency issues
 High availability solutions
 Tools


Introduction
Insert Picture Here


MySQL architecture
Overview
Base
●
Installation layout
●
Log files
Connectors
●
Clients
●
APIs
Optimizer
Caches & buffers
Storage engines
Management


Insert Picture Here

Typical installation layout

Datadir
●
Schema
●
Table files: *frm, *ibd, *MYD, *MYI, *par, etc.
●

Log files

●

InnoDB shared tablespace

Configurable


Log files

General query log
Slow query log
Binary log
Relay log
Error log
InnoDB logs (and, probably, created by other storage engines)


Connectors

Clients
●
MySQL CLI
●
MySQL Workbench
●
MySQL Enterprise Monitor (MEM)
●
For administrators
APIs
●

Exist for most popular programming languages

●

C, C++, JDBC, PHP, Python, Net, ODBC


Plugins

Plugins
●
Storage engine
●
Full-text parsers
●
Daemon
●
INFORMATION_SCHEMA
●
Semisynchronous Replication
●
Audit
●
Authentication
●
Password-validation


Storage engines

From troubleshooting point of view
●

Own data/index format

●

Own locking model

●

Own diagnostic

●

Own log files

●

CHECK TABLE


Basic single-client issues
Insert Picture Here


MySQL Access Privilege System

No roles by default, limited user limits
All records are in the mysql database (schema)
Pluggable authentication since version 5.5
Connections: TCP/IP with login-password, socket (Unix), named pipe
(Windows)

SELECT user, host FROM mysql.user
SELECT USER(), CURRENT_USER()
SHOW GRANTS

Error handling
Errors vs warnings
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist. Check the
'Function Name Parsing and Resolution' section in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
++++
| Level | Code | Message |
++++
| Warning | 1292 | Truncated incorrect DOUBLE value: 'f1' |
++++
1 row in set (0.00 sec)


Error handling
Application (C API)
Error information
●
mysql_errno
●
mysql_error
Warnings
●
mysql_info
●
mysql_sqlstate
●
mysql_warning_count


Error handling
perror
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the 'Function Name Parsing and
Resolution' section in the Reference Manual
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated
incorrect %.32s value: '%.128s'
OS error code 2: No such file or directory
MySQL error code 150: Foreign key constraint is incorrectly
formed


Error handling
in stored routines
GET DIAGNOSTICS
●
GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;
●
GET DIAGNOSTICS CONDITION 1 code = RETURNED_SQLSTATE,
msg = MESSAGE_TEXT;
http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html


Options

Have global and session scope
Can be set
●
In configuration files
●
Command line parameter
●
SET [GLOBAL] var_name = NEW_VAL
●
SHOW [GLOBAL] VARIABLES
Only two built-in levels of write access
●
SUPER for GLOBAL and limited SESSION variables
●
Non-privileged user can change any session option!


Options

Allocated at
●
Server startup
●
User connection creation
●
For certain operations
To watch status
●
SHOW [GLOBAL] STATUS
http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html


INFORMATION_SCHEMA

Contain metadata information
●
Tables
●
Indexes
●
Other
Allows to create plugins
●
InnoDB plugins
●
We will discuss them later
Oracle analog: Data Dictionary Views


Optimizer

EXPLAIN is less powerful if compare with Oracle
●
It is improved in version 5.7.3
●
Graphic EXPLAIN in MySQL Workbench 6.0
●
MySQL’s EXPLAIN Command New Features [HOL9734, passed]
EXPLAIN EXTENDED
INFORMATION_SCHEMA.OPTIMIZER_TRACE
Status variables 'Handler_%'


EXPLAIN in Oracle
http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm
EXPLAIN PLAN SET statement_id = 'example_plan4' FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
  FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
   AND h.date_ordered > SYSDATE30
   AND l.header_id = h.header_id ;
Plan

SELECT STATEMENT
NESTED LOOPS
  TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
   INDEX RANGE SCAN SO_HEADERS_N1
  TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
   INDEX RANGE SCAN SO_LINES_N1


EXPLAIN in MySQL
mysql> EXPLAIN SELECT user, host FROM userG
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: user
         type: index
possible_keys: NULL
          key: PRIMARY
      key_len: 228
          ref: NULL
         rows: 4
        Extra: Using index


EXPLAIN in MySQL

EXPLAIN EXTENDED
●
Follow by SHOW WARNINGS
EXPLAIN PARTITIONS
EXPLAIN FORMAT=JSON

http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D79908_1879034

http://dev.mysql.com/doc/refman/5.6/en/explain-output.html


Storage engine specifics

They care about physical data, so all data information are on their level
●
Corruption
●
Index statistics
CHECK TABLE to check for errors


MyISAM
Stores data in files
●
*frm – table definition, common for all storage engines
●
*MYD – data file
●
*MYI – index file
myisamchk
myisam_recovery_options to check each time you open a table
●
First access – table is opened
●
Second, third, .. access – existent descriptor used
●
FLUSH TABLES – closes opened descriptors


InnoDB
Transactional storage engine
Physical layout
●
*frm file – table definition
●
Shared tablespace
●
*ibd file – tablespace for individual table
●
Optional: innodb_file_per_table
●

Recommended

Redo log files
OPTIMIZE TABLE = ALTER + ANALYZE
Automatic startup check
●


Concurrency issues
Insert Picture Here


Lock types

MDL locks
Table locks
Row locks
Read locks
●
Block writes
Write locks
●
Block both read and writes


Transactions and Their Relationship With Locks

Server-level
●
MDL locks
Engine level
●
Table locks
●
Row locks
AUTOCOMMIT
●
Supported


Diagnostic Tools

SHOW [FULL] PROCESSLIST
●
Universal tool which lists all currently running connections
SHOW ENGINE INNODB STATUS
INFORMATION SCHEMA
●
PROCESSLIST
●
InnoDB tables
PERFORMANCE SCHEMA
●
MDL locks
Unlocking MySQL: Dealing with Locking in MySQL [CON4038, passed]


INFORMATION_SCHEMA.PROCESSLIST
structure
mysql> DESCRIBE INFORMATION_SCHEMA.PROCESSLIST;
+++++++
| Field          | Type            | Null   | Key | Default  | Extra  |
+++++++
| ID             | bigint(4)       | NO    |      | 0        |        |
| USER           | varchar(16)     | NO    |      |          |        |
| HOST           | varchar(64)     | NO    |      |          |        |
| DB             | varchar(64)     | YES   |      | NULL     |        |
| COMMAND        | varchar(16)     | NO    |      |          |        |
| TIME           | int(7)          | NO    |      | 0        |        |
| STATE          | varchar(64)     | YES   |      | NULL     |        |
| INFO           | longtext        | YES   |      | NULL     |        |
+++++++



Transactions and engine-level locks are done at the engine level
Storage engines have own diagnostic tools
Use them when hit an issue with such a lock


InnoDB
InnoDB Monitors
●
●
Lock Monitor
INFORMATION_SCHEMA tables
●
INNODB_TRX
●
INNODB_LOCKS
●
INNODB_LOCK_WAITS


Diagnostics: P_S

Monitors internal operations
●
Events
●
Waits
●
Mutexes
●
Statements
●
Stages
Similar to Oracle wait interface
Improving Performance with MySQL Performance Schema [HOL9733, Sunday,
1:00 PM]
Making the Performance Schema Easier to Use [CON5282, Sunday, 10:00 AM]
Performance Schema and ps_helper [CON4077, passed]

MDL Locks

Waiting thread only
SHOW PROCESSLIST, INFORMATION_SCHEMA.PROCESSLIST
●
Waiting for table metadata lock
Performance Schema
●
MUTEX_INSTANCES
●
EVENTS_WAITS_CURRENT
●
THREADS


Performance Schema MDL queries
mysql> SELECT * FROM mutex_instances WHERE
LOCKED_BY_THREAD_ID
IS NOT NULLG
*********************** 1. row ***********************
NAME:
wait/synch/mutex/sql/MDL_wait::LOCK_wait_status
OBJECT_INSTANCE_BEGIN: 37104744
LOCKED_BY_THREAD_ID: 18


Performance Schema MDL queries
mysql> SELECT THREAD_ID, EVENT_ID, EVENT_NAME, SOURCE, TIMER_START,
OBJECT_INSTANCE_BEGIN, OPERATION FROM events_waits_current WHERE
THREAD_ID IN(SELECT LOCKED_BY_THREAD_ID FROM mutex_instances WHERE
LOCKED_BY_THREAD_ID IS NOT NULL)G
*************************** 1. row ***************************
            THREAD_ID: 18
             EVENT_ID: 461
           EVENT_NAME: wait/synch/cond/sql/MDL_context::COND_wait_status
               SOURCE: mdl.cc:1210
          TIMER_START: 97623253523306
OBJECT_INSTANCE_BEGIN: 0
            OPERATION: timed_wait


High availability solutions
Insert Picture Here


MySQL Cluster

Special storage engine: NDB
Stores data on two or more physical machines
Two or more copies
General troubleshooting techniques
●
Applicable
Specific NDB storage engine techniques


Replication
Overview
Always available, but you must setup it before use
Asynchronous master-slave
Master
●
Keeps all updates in separate file: binary log
Slave
●
IO thread read updates from master and stores in relay log file
●
SQL thread executes updates


Replication
Troubleshooting tools
Error log file
Slave
●
SHOW SLAVE STATUS
Master
●
SHOW MASTER STATUS
mysqlbinlog


Tools
Insert Picture Here


For developers

MySQL Workbench
●
SQL editor
●
Database designer
●
Introduction to MySQL Database Development with MySQL Workbench
[CON3967, Sunday, 5:30 PM]
MySQL Command line client
●
Command-line interface for SQL
●
SELECT, INSERT, UPDATE, DELETE, etc.


For DBA

MySQL Enterprise Monitor
●
Real-time MySQL performance and availability monitoring
●
Error information
●
Advisors
●
Forcasts


For DBA

MySQL Workbench
●
Server administration
●
Backups
●
Using the New MySQL Workbench Tools for Performance Tuning
[HOL9786, passed]
MySQL command line client
●
Command-line interface for SQL
●
GRANT, CREATE, OPTIMIZE, etc.
mysqladmin


Command line
They are not
Set of tools to do various administrative job
Same!
●
MySQL Workbench Utilities
●
http://dev.mysql.com/downloads/tools/utilities/
●

●

Development of Fault-Tolerant Failover Tools with MySQL Utilities
[CON4276, Sunday, 2:30 PM]

Percona Toolkit
●
http://www.percona.com/software/percona-toolkit


Command line

MySQL Sandbox
●
Sandbox for your tests
●
https://launchpad.net/mysql-sandbox
PS_HELPER view
●
www.markleith.co.uk/ps_helper/
●
Performance Schema and ps_helper [CON4077, passed]


References

MySQL User Reference Manual
●
http://dev.mysql.com/doc/refman/5.6/en/index.html
Knowledge Management Database
Forums
●
http://forums.mysql.com
Bug trackers
●
http://bugs.mysql.com
●
Oracle Internal Bugs database
My Oracle Support
●
https://support.oracle.com


References

MySQL Troubleshooting book
●
http://shop.oreilly.com/product/0636920021964.do
Marc Alff's Performance Schema blog
●
http://marcalff.blogspot.ru/
Planet MySQL
●
http://planet.mysql.com/
●
http://dev.mysql.com/support/blogs/
●
https://blogs.oracle.com/mysqlinnodb/


?
Insert Picture Here


THANK YOU!
https://twitter.com/#!/svetsmirnova
https://blogs.oracle.com/svetasmirnova/


Insert Picture Here

Graphic Section Divider


The preceding is intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.


MySQL Troubleshooting for
Oracle DBAs
Part II: details
Sveta Smirnova
Principal Technical Support Engineer


Insert Picture Here

Replication
Insert Picture Here


Replication IO thread:
communication issues
Access error
●
Check slave error log
●
SHOW SLAVE STATUS
●
Try to connect using normal MySQL client using slave credentials
●
SHOW GRANTS
●
●

Fix privileges on master
Restart slave


Replication SQL thread:
typical issues
Simple master-slave
●
Data is different on master and slave
●
Replication event can not be applied
Different errors on master and slave
●
Slave lags far behind the master
Circular replication or other writes in addition to slave SQL thread
●
●


Was the table modified besides the SQL thread?
●
How?
●
Can it affect content of the table in the wrong way?
Are the table definitions same on master and slave?
Is it possible that master events was applied in wrong order?
●
Use mysqlbinlog to find queries which caused the issue
●
Check master's application to find what is wrong


Events from master were applied in wrong order
Lock issues
●
InnoDB
Triggers
●
SET GLOBAL slave_skip_counter
●
Synchronize tables!
Different options
●
Start slave with master's options, then check


Slave lags far behind the master
Threads
●
Master runs in multiple update threads
●
Slave uses single
Seconds_behind_master is growing
Tune slave performance
●
Buffers
●
Indexes (for statement)


Optimizer
Insert Picture Here


Optimizer
Handler_% status variables
mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW STATUS LIKE 'Handler_%';
+++
| Variable_name              | Value |
+++
| Handler_commit             | 0     |
| Handler_delete             | 0     |
| Handler_discover           | 0     |
| Handler_prepare            | 0     |
| Handler_read_first         | 0     |
| Handler_read_key           | 0     |
| Handler_read_last          | 0     |
| Handler_read_next          | 0     |
| Handler_read_prev          | 0     |
| Handler_read_rnd           | 0     |


Optimizer
| Handler_read_rnd_next      | 0     |
| Handler_rollback           | 0     |
| Handler_savepoint          | 0     |
| Handler_savepoint_rollback | 0     |
| Handler_update             | 0     |
| Handler_write              | 0     |
+++
mysql> select count(*) from employees
join titles using(emp_no) where title='Senior Engineer';
++
| count(*) |
++
|    97750 |
++


Optimizer
mysql> SHOW STATUS LIKE 'Handler_%';
+++
+++
| Handler_commit             | 1      |
| Handler_delete             | 0      |
| Handler_discover           | 0      |
| Handler_prepare            | 0      |
| Handler_read_first         | 1      |
| Handler_read_key           | 300027 |
| Handler_read_last          | 0      |
| Handler_read_next          | 397774 |
| Handler_read_prev          | 0      |
| Handler_read_rnd           | 0      |
...


Optimizer
Variables:
●
optimizer_trace=”enabled=on|off,one_line=off|on”
●
optimizer_trace_features=”greedy_search=on,range_optimi
zer=on,dynamic_range=on,repeated_subselect=on”
●
optimizer_trace_limit=1
●
optimizer_trace_max_mem_size=16384
●
optimizer_trace_offset=1
●
end_markers_in_json=0|1


Optimizer
Turn tracing on (it's off by default):
SET optimizer_trace="enabled=on";
SELECT <your query here>;
SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
More queries...
When done with tracing, disable it:
SET optimizer_trace="enabled=off";
http://dev.mysql.com/doc/internals/en/optimizer-tracing.html


Optimizer
mysql> SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACEG
*************************** 1. row ***************************
                        QUERY: select emp_no, min(from_date) from titles
group by emp_no limit 10
                        TRACE: {
  "steps": [
    {
      "join_preparation": {
        "select#": 1,
        "steps": [
          {
            "expanded_query": "/* select#1 */ select `titles`.èmp_no` AS
èmp_no`,min(`titles`.`from_date`) AS `min(from_date)` from `titles` group
by `titles`.èmp_no` limit 10"


Optimizer
INFORMATION_SCHEMA.TRACE
join_preparation
join_optimization
●
table_dependencies
●
rows_estimation
●
considered_execution_plans
●
attaching_conditions_to_tables
●
clause_processing
●
refine_plan
●
reconsidering_access_paths_for_index_ordering
join_execution


Options
Insert Picture Here


Options
Scope
Global
●
Control parameters, necessary for all server processes
●
Location of server files: datadir etc.
●

Shared buffers

●

More

Session
●
Control connection-specific parameters
http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html


Options
How to set
SET [GLOBAL] var_name = NEW_VAL
Command line option
Configuration file
●
In default location
●
Specified by option defaultsfile


Options
Who can change
Global options and few session options
●
A user with privilege SUPER
Session options
●
Anybody
There is no limits!


Options
When allocated
Those which control behavior of whole server
●
Once at server startup
●
Can start with low values, then grow to specified
Connection options
●
For every connection when connection opens
Operation-specific
●
For every operation when needed


Options
How to control
SHOW [GLOBAL] STATUS
GLOBAL
●
Since server start
SESSION
●
For operations in current session
●
Can be reset
●
FLUSH STATUS


Options
Global example
mysql> show global status like 'Handler_read_rnd_next';
+++
+++
+++


Options
Session example
mysql> show status like 'Handler_read_rnd_next';
+++
+++
+++


Options
Troubleshooting best practices
Record currently used variables
●
Make change dynamically if possible
●
SET [GLOBAL] var_name=NEW_VAL
Test in one session first
Then change global variable
Change configuration file after you are happy with results


Options
When affecting option is not known
Record currently used variables
●
Start mysqld with option –nodefaults
●
This option must be first one!
Check if problem is solved
Change variable values one-by-one until you find one which leads to the
problem


InnoDB Monitors and
Information Schema tables


Insert Picture Here


TRANSACTIONS

Trx id counter 1B1C
Purge done for trx's n:o < 1B19 undo n:o < 0
History list length 189
LIST OF TRANSACTIONS FOR EACH SESSION:
TRANSACTION 0, not started
MySQL thread id 28, OS thread handle 0x7fa9301a0700, query
id 8653 localhost root
show engine innodb status


TRANSACTION 1B1B, ACTIVE 6 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 31, OS thread handle 0x7fa9301e1700, query id 8652 localhost sveta Updating
update tbllog set `Employee ID`=2345 where `Log Number`=1
TRX HAS BEEN WAITING 6 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 81967 n bits 72 index `PRIMARY` of table `test`.`tlog`
trx id 1B1B lock_mode X locks rec but not gap waiting
Record lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000001; asc     ;;
1: len 6; hex 000000001b19; asc       ;;
2: len 7; hex 14000140340110; asc    @4  ;;
3: len 4; hex 80003039; asc   09;;
4: SQL NULL;



TRANSACTION 1B19, ACTIVE 787 sec
2 lock struct(s), heap size 376, 1 row lock(s), undo log
entries 1
MySQL thread id 32, OS thread handle 0x7fa930222700, query
id 8647 localhost sveta
No lock information
For the transaction
Which holds the lock!


InnoDB Monitors

Pseudo-tables
Turn periodical logging into error log
●
CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;
Lock monitor
●
Changes output format of InnoDB Monitor
●
CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;


InnoDB Lock Monitor
TRANSACTION 1B19, ACTIVE 1935 sec
2 lock struct(s), heap size 376, 1 row lock(s), undo log entries 1
MySQL thread id 32, OS thread handle 0x7fa930222700, query id 8647 localhost sveta
TABLE LOCK table `test`.`tlog` trx id 1B19 lock mode IX
RECORD LOCKS space id 0 page no 81967 n bits 72 index `PRIMARY` of table
`test`.`tlog` trx id 1B19 lock_mode X locks rec but not gap
Record lock, heap no 6 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 80000001; asc     ;;
1: len 6; hex 000000001b19; asc       ;;
2: len 7; hex 14000140340110; asc    @4  ;;
3: len 4; hex 80003039; asc   09;;
4: SQL NULL;


InnoDB Information Schema tables

INNODB_TRX
INNODB_LOCKS
INNODB_LOCK_WAITS



mysql> SELECT * FROM INNODB_LOCK_WAITS;
+++++
|requesting_trx_id|requested_lock_id|blocking_trx_id|blocking_lock_id|
+++++
| 1B1F | 1B1F:0:81967:6 | 1B19 | 1B19:0:81967:6 |
+++++



mysql> select trx_mysql_thread_id, trx_id from innodb_trx;
+++
| trx_mysql_thread_id | trx_id   |
+++
|                  31 | 1B1F     |
|                  32 | 1B19     |
+++


MySQL Access Privilege
System


Insert Picture Here

Overview
No roles by default, limited user limits
All records are in the mysql database (schema)
Pluggable authentication since version 5.5
Connections
●

TCP/IP with login-password

●

Socket (Unix)

●

Named pipe (Windows)


How to handle connection issues
SELECT user, host FROM mysql.user
Most descriptive host first, then wildcard
Socket connection by default on Unix


Wrong access
SHOW GRANTS [FOR user@host]
Grant tables
●

mysql.db

●

mysql.tables_priv

●

mysql.columns_priv

●

mysql.procs_priv

●

mysql.proxies_priv

SELECT USER(), CURRENT_USER()

THANK YOU!
https://twitter.com/#!/svetsmirnova
https://blogs.oracle.com/svetasmirnova/


Insert Picture Here

Graphic Section Divider


The preceding is intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.


Basic MySQL Troubleshooting for Oracle DBAs

More Related Content

What's hot

Similar to Basic MySQL Troubleshooting for Oracle DBAs

More from Sveta Smirnova

Recently uploaded

Basic MySQL Troubleshooting for Oracle DBAs