MySQL Monitoring Mechanisms

MySQL Monitoring Mechanisms: How the different pieces fit together Mark Leith (@MarkLeith)

Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Program Agenda
Overview of Past MySQL Monitoring
The New MySQL Monitoring Vision
Performance Schema
New InnoDB Instrumentation
Optimizer Trace
Enhanced EXPLAIN
Other Goodies
Summary

A long time ago in a galaxy far, far away.... MySQL Episode IV.I The SHOW statement continues to rule supreme... USE THE FORCE LUKE!!

Overview of Past MySQL Monitoring Prior to 5.5 instrumentation within MySQL was limited
Various SHOW statements allowed some global visibility
INFORMATION_SCHEMA for (mostly) object metadata
Slow / General query logs for statement history
EXPLAIN / SHOW PROFILE for statement profiling

SHOW Statements
Great for simple commands to get metadata, i.e:
SHOW CREATE TABLE ...;
SHOW TABLES;
Not so great for scripting or tooling, i.e:
SHOW ENGINE INNODB STATUS parsing hurts
Can not filter, or re-order columns
Can not use dynamically (prepared statements etc.)

INFORMATION_SCHEMA
Added within 5.0+
It's just SQL and (virtual) tables..
Can be used dynamically (prepared statements etc.)
Order output independently (columns, rows, etc.)
Plugin interface available
Optimization path is unique, and disk accesses are high with some tables (I_S.TABLES etc.)

Slow / General Query Logging
Available in all versions of MySQL
Allow analyzing problems after the fact
Not nearly enough information within them for serious diagnosis (even with patches)
Overhead can be excessive
General query log bottleneck at start of statements
CSV based log tables in 5.1+ are not ideal

EXPLAIN / SHOW PROFILE
Good high-level overviews of how a query runs
Allows some response time break downs
Table based outputs belie true execution flows
Not enough statistics are reported
EXPLAIN doesn't show all optimizer decisions
EXPLAIN doesn't support all statement types
SHOW PROFILE often isn't fine grained enough

MySQL Episode V.V A NEW HOPE

5.5+ Instrumentation
5.5 saw a huge investment in instrumentation
The Performance Schema framework was introduced
Many “finer detail” instrumentation points added
Focused on IO and internal synchronization points
New INFORMATION_SCHEMA tables for InnoDB
Current transaction and locking statistics
Compression statistics

5.6 And Beyond Accelerating investment in instrumentation and tooling
Fill more gaps in Performance Schema statistics
Expose even more detailed InnoDB statistics
Take the guess work out of optimizer workings
Increase flexibility of instrumentation configuration
Provide more structured data for richer tooling

5.6 And Beyond – Instrumentation Vision
SHOW - Object Metdata, limited extension otherwise
INFORMATION_SCHEMA – Standard Object Metadata
INFORMATION_SCHEMA – Plugin Specific Extensions
PERFORMANCE_SCHEMA – Consolidated Statistics

Performance Schema

Performance Schema in 5.5
Foundation for a powerful instrumentation framework
Similar in goals to the “ Oracle Wait Interface ”
Tracks Events by time/latency (to picosecond precision)
Entirely non-locking, and in fixed memory
Implemented as a real storage engine
No unique optimization paths
Not persistent

Performance Schema in 5.5 What is an “ Event ”?
At it's most basic, a measurement of a unit of work
In all cases this includes time taken to complete
In some we record other things, like bytes requested
Naming roughly follows “ Linnean Taxinomy ”
/Class/Order/Family/Genus/Species

First instrumentation is more MySQL developer focused
Started at some of the smallest internal units of work
Mutexes - wait/synch/mutex/
Read/Write locks - wait/synch/rwlock/
Conditions - wait/synch/cond/
File IO instrumented as well, across all file types
wait/io/file/
Tracks both time waited and bytes per request

Performance Schema in 5.5 THREAD_ID: 17 EVENT_ID: 57265474 EVENT_NAME: wait/synch/mutex/innodb/kernel_mutex SOURCE: lock0lock.c:5489 TIMER_START: 1716417836794180 TIMER_END: 1716417836828110 TIMER_WAIT: 33930 SPINS: NULL OBJECT_SCHEMA: NULL OBJECT_NAME: NULL OBJECT_TYPE: NULL OBJECT_INSTANCE_BEGIN: 4307564056 NESTING_EVENT_ID: NULL OPERATION: lock NUMBER_OF_BYTES: NULL FLAGS: 0 Mutex Event ~33 nanosecs To acquire lock

Performance Schema in 5.5 SELECT event_name, ROUND(SUM(sum_timer_wait) / 1000000000000, 2) total_sec, ROUND(MIN(min_timer_wait) / 1000000, 2) min_usec, ROUND((SUM(sum_timer_wait) / SUM(COUNT_STAR))/1000000, 2) avg_usec, ROUND(MAX(max_timer_wait) / 1000000, 2) max_usec FROM performance_schema.events_waits_summary_global_by_event_name WHERE sum_timer_wait > 0 GROUP BY event_name ORDER BY total_sec DESC limit 10; Top waits by latency

Performance Schema in 5.5 THREAD_ID: 6 EVENT_ID: 1036 EVENT_NAME: wait/io/file/innodb/innodb_data_file SOURCE: os0file.c:4891 TIMER_START: 338825682207740 TIMER_END: 338825703463000 TIMER_WAIT: 21255260 SPINS: NULL OBJECT_SCHEMA: NULL OBJECT_NAME: /Users/mark/sandboxes/msb_5_5_16/data/ibdata1 OBJECT_TYPE: FILE OBJECT_INSTANCE_BEGIN: 4683519488 NESTING_EVENT_ID: NULL OPERATION: write NUMBER_OF_BYTES: 32768 FLAGS: 0 File IO Event ~21 microsecs To write 32K On this file

Performance Schema in 5.5 SELECT substring_index(file_name, '/', -1) AS file, count_read, count_write, ROUND(sum_number_of_bytes_read / (1024*1024)) read_mb, ROUND(sum_number_of_bytes_write / (1024*1024)) write_mb, ROUND((sum_number_of_bytes_read + sum_number_of_bytes_write) / (1024*1024*1024)) total_gb FROM performance_schema.file_summary_by_instance ORDER BY total_gb DESC LIMIT 10; Top files by IO consumption

Proved the model, can handle the “flood”
Already aided finding bottlenecks
But..
Needed more DBA/Developer focused statistics
Needed more control on who/what is traced
Needed a little tuning for high load systems

Adding DBA / Developer focused instrumentation
With the ability to drill down between each level
Statements – Latency and key statistics
Stages – Statement execution states (i.e. “ Sending data ”)
Network IO / Idle time - at the TCP / socket layers
Table IO - at the storage engine handler layer
Table Locks - at the storage engine handler layer
.. as well as file IO , mutexes , conditions , rwlocks

Performance Schema Statement Statistics

Performance Schema Stages The “stage” (state) Within statement execution When it started, ended, and How long it took The statement ID it was related to

Performance Schema in 5.6 – Event Nesting
Events are related in a hierarchy
Currently uses the “Adjacency List” model
EVENT_ID (child) -> NESTING_EVENT_ID (parent)
Investigating the “Nested Set” model

Performance Schema in 5.6 – Event Nesting

Performance Schema in 5.6 – Graph Events http://www.markleith.co.uk/?p=471 The Statement The Stages The individual wait events by type

Performance Schema in 5.6 – Graph Events

Performance Schema in 5.6 – Summaries Summarize (GROUP BY) each class of event by user, host, account ( [email_address] ), thread, and globally New Object, Network IO Table IO and Table Lock Wait summaries

Performance Schema Table IO Summary The table and it's high level aggregated stats A breakdown By table IO type

“ Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein Yet 60 nanoseconds of time could mean SO MUCH!

Performance Schema Configuration
There is always some overhead with instrumentation
Whether, and how, we time events can be important
Set expectations up front on both:
What you want to monitor
How selectively you want to monitor it

Performance Schema Overhead http://marcalff.blogspot.com/2011/06/performance-schema-overhead-tuning.html June 2011 Lots of tuning done since this. New benchmarks should be out soon

Performance Schema Setup

Performance Schema – What's Missing?
Storage engine level lock latency – i.e. InnoDB row locks
User lock latency – i.e. SELECT GET_LOCK();
A normalized statement level digest
Transactions level statistics (tied to global transaction ID)
Memory allocations
Distribution statistics (histograms)
Background thread “stages”, server cache statistics
...and...and...and... - We're listening, tell us your needs

MySQL 5.6: 11 New INFORMATION_SCHEMA tables
7 Data Dictionary related
3 Buffer Pool related
1 General Statistics gold mine
MySQL 5.5: 7 New INFORMATION_SCHEMA tables
3 Transaction / Locking related
4 Compression related

InnoDB System Tables http://dev.mysql.com/doc/refman/5.6/en/innodb-sys-tablestats-table.html View InnoDB's internal Data Dictionary and table stats

INNODB_SYS_TABLES[TATS] SELECT innodb_sys_tables.name, CASE flag >> 5 WHEN 0 THEN 'Antelope' WHEN 1 THEN 'Barracuda' WHEN 2 THEN 'Cheetah' END AS file_format, @page_flags := (flag >> 1) & 15 AS page_flags, @page_size := IF(@page_flags = 0,16384, 512 << @page_flags) AS page_size, ROUND(( @page_size * clust_index_size ) /(1024*1024)) AS pk_mb, ROUND(( @page_size * other_index_size ) /(1024*1024)) AS secidx_mb FROM innodb_sys_tables JOIN innodb_sys_tablestats USING (table_id) ORDER BY pk_mb DESC;

INNODB_BUFFER_PAGE Tables

INNODB_BUFFER_PAGE - Summarized http://dev.mysql.com/doc/refman/5.6/en/innodb-buffer-page-table.html SELECT page_type, page_state, table_name, index_name, SUM(data_size) / (1024*1024) AS used_mb, SUM( IF(compressed_size = 0, 16384, compressed_size) ) / (1024*1024) AS total_mb FROM innodb_buffer_page GROUP BY page_type, page_state, table_name, index_name HAVING total_mb > 0 ORDER BY total_mb DESC;

INNODB_METRICS
Consolidated InnoDB statistics
199 metrics currently collected ( suggest just one more! )
More flexible statistics collection than SHOW STATUS
Can enable/disable individually
Track when / how long each has been enabled
Provide extra statistics inline per metric
Provide extra metadata inline per metric

INNODB_METRICS – Metric Details http://dev.mysql.com/doc/refman/5.6/en/innodb-metrics-table.html The variable Stats since start Stats since “reset” How long enabled Stat Metadata

INNODB_METRICS – Configuration Enable named metric Disable named metric Enable metric “module” Reset metric “module”

INNODB_METRICS – Looking at Counters http://dev.mysql.com/doc/refman/5.6/en/innodb-metrics-table.html SELECT concat(subsystem,'/', name) AS event, count, ROUND(count / time_elapsed) AS avg_per_second FROM information_schema.innodb_metrics WHERE status = 'enabled' AND type LIKE '%counter%' OR type = 'set_owner' ORDER BY avg_per_second DESC;

Optimizer Trace

Optimizer Trace
EXPLAIN tells you how a query will execute
Optimizer Trace tells you why it will execute that way
New INFORMATION_SCHEMA.OPTIMIZER_TRACE
Contains trace of optimizer decisions, in JSON format
Enabled per thread, on demand
Targeted for advanced users, or in use with bug reporting

Optimizer Trace Enable Tracing Run Statement Select the trace

Optimizer Trace Expose query transformations

Optimizer Trace Considered plans With costs Estimations Of rows And costs Whether Indexes are Usable or not

Enhanced EXPLAIN

Enhanced EXPLAIN Plans
In the past EXPLAIN was only supported with SELECT
To EXPLAIN an UPDATE - “Re-write equivalent SELECT”
However optimization paths may be different
Now you can EXPLAIN:
UPDATE | DELETE | REPLACE | INSERT

Enhanced EXPLAIN Plans
“ Structured EXPLAIN” is in development
A properly structured EXPLAIN plan (again, JSON)
Sorry – no examples yet, it's very early in implementation

Other Goodies

Instance UUIDs / SHOW SLAVE STATUS
Automatically generated unique instance identifiers
Persisted in $datadir/auto.cnf
Can be used for reliable replication topology discovery
Hard in past given IP/Host matching (virtual IPs?)
More configuration and thread statistics

http://www.markleith.co.uk	507
http://blogs.oracle.com	161
https://blogs.oracle.com	136
http://s386513550.websitehome.co.uk	6
http://www.hanrss.com	3
http://localhost	3
http://sqldump.cool-tools.co.uk	2
http://feeds.iti.bz	1
http://rscat.com	1
http://www.netvibes.com	1
https://blogs.oraclegha.com	1

MySQL Monitoring Mechanisms

by Mark Leith on Oct 12, 2011

Accessibility

Categories

Upload Details

Usage Rights

11 Embeds 822

Statistics

MySQL Monitoring Mechanisms Presentation Transcript