MySQL Monitoring Mechanisms
Upcoming SlideShare
Loading in...5

MySQL Monitoring Mechanisms



The 5.5 and 5.6 releases of MySQL introduce several new mechanisms that provide improved monitoring and performance tuning functionality. Examples are performance schemas, InnoDB metrics tables, ...

The 5.5 and 5.6 releases of MySQL introduce several new mechanisms that provide improved monitoring and performance tuning functionality. Examples are performance schemas, InnoDB metrics tables, optimizer trace, and extended explain functionality.

This session outlines the vision for monitoring-related functionality in MySQL and presents an overview of the new mechanisms. It shows how these are integrated with MySQL management tools. Furthermore, it discusses how these mechanisms can be utilized by application developers, DBAs, and production engineers for tracking down performance issues and monitoring production systems.



Total Views
Views on SlideShare
Embed Views



13 Embeds 1,494 1072 239 161 6 3 3
http://localhost 3 2 1 1 1 1 1



Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

MySQL Monitoring Mechanisms MySQL Monitoring Mechanisms Presentation Transcript

  • MySQL Monitoring Mechanisms: How the different pieces fit together Mark Leith (@MarkLeith)
  • Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. View slide
  • Program Agenda
    • Overview of Past MySQL Monitoring View slide
    • The New MySQL Monitoring Vision
    • Performance Schema
    • New InnoDB Instrumentation
    • Optimizer Trace
    • Enhanced EXPLAIN
    • Other Goodies
    • Summary
  • A long time ago in a galaxy far, far away.... MySQL Episode IV.I The SHOW statement continues to rule supreme... USE THE FORCE LUKE!!
  • Overview of Past MySQL Monitoring Prior to 5.5 instrumentation within MySQL was limited
    • Various SHOW statements allowed some global visibility
    • INFORMATION_SCHEMA for (mostly) object metadata
    • Slow / General query logs for statement history
    • EXPLAIN / SHOW PROFILE for statement profiling
  • SHOW Statements
    • Great for simple commands to get metadata, i.e:
      • SHOW CREATE TABLE ...;
      • SHOW TABLES;
    • Not so great for scripting or tooling, i.e:
      • SHOW ENGINE INNODB STATUS parsing hurts
      • Can not filter, or re-order columns
      • Can not use dynamically (prepared statements etc.)
    • Added within 5.0+
    • It's just SQL and (virtual) tables..
      • Can be used dynamically (prepared statements etc.)
      • Order output independently (columns, rows, etc.)
    • Plugin interface available
    • Optimization path is unique, and disk accesses are high with some tables (I_S.TABLES etc.)
  • Slow / General Query Logging
    • Available in all versions of MySQL
    • Allow analyzing problems after the fact
    • Not nearly enough information within them for serious diagnosis (even with patches)
    • Overhead can be excessive
      • General query log bottleneck at start of statements
      • CSV based log tables in 5.1+ are not ideal
    • Good high-level overviews of how a query runs
    • Allows some response time break downs
    • Table based outputs belie true execution flows
    • Not enough statistics are reported
      • EXPLAIN doesn't show all optimizer decisions
      • EXPLAIN doesn't support all statement types
      • SHOW PROFILE often isn't fine grained enough
  • MySQL Episode V.V A NEW HOPE
  • 5.5+ Instrumentation
    • 5.5 saw a huge investment in instrumentation
    • The Performance Schema framework was introduced
      • Many “finer detail” instrumentation points added
      • Focused on IO and internal synchronization points
    • New INFORMATION_SCHEMA tables for InnoDB
      • Current transaction and locking statistics
      • Compression statistics
  • 5.6 And Beyond Accelerating investment in instrumentation and tooling
    • Fill more gaps in Performance Schema statistics
    • Expose even more detailed InnoDB statistics
    • Take the guess work out of optimizer workings
    • Increase flexibility of instrumentation configuration
    • Provide more structured data for richer tooling
  • 5.6 And Beyond – Instrumentation Vision
    • SHOW - Object Metdata, limited extension otherwise
    • INFORMATION_SCHEMA – Standard Object Metadata
    • INFORMATION_SCHEMA – Plugin Specific Extensions
    • PERFORMANCE_SCHEMA – Consolidated Statistics
  • Performance Schema
  • Performance Schema in 5.5
    • Foundation for a powerful instrumentation framework
    • Similar in goals to the “ Oracle Wait Interface ”
    • Tracks Events by time/latency (to picosecond precision)
    • Entirely non-locking, and in fixed memory
    • Implemented as a real storage engine
      • No unique optimization paths
      • Not persistent
  • Performance Schema in 5.5 What is an “ Event ”?
    • At it's most basic, a measurement of a unit of work
      • In all cases this includes time taken to complete
      • In some we record other things, like bytes requested
    • Naming roughly follows “ Linnean Taxinomy ”
      • /Class/Order/Family/Genus/Species
  • Performance Schema in 5.5
    • First instrumentation is more MySQL developer focused
    • Started at some of the smallest internal units of work
      • Mutexes - wait/synch/mutex/
      • Read/Write locks - wait/synch/rwlock/
      • Conditions - wait/synch/cond/
    • File IO instrumented as well, across all file types
      • wait/io/file/
      • Tracks both time waited and bytes per request
  • Performance Schema in 5.5 THREAD_ID: 17 EVENT_ID: 57265474 EVENT_NAME: wait/synch/mutex/innodb/kernel_mutex SOURCE: lock0lock.c:5489 TIMER_START: 1716417836794180 TIMER_END: 1716417836828110 TIMER_WAIT: 33930 SPINS: NULL OBJECT_SCHEMA: NULL OBJECT_NAME: NULL OBJECT_TYPE: NULL OBJECT_INSTANCE_BEGIN: 4307564056 NESTING_EVENT_ID: NULL OPERATION: lock NUMBER_OF_BYTES: NULL FLAGS: 0 Mutex Event ~33 nanosecs To acquire lock
  • Performance Schema in 5.5 SELECT event_name, ROUND(SUM(sum_timer_wait) / 1000000000000, 2) total_sec, ROUND(MIN(min_timer_wait) / 1000000, 2) min_usec, ROUND((SUM(sum_timer_wait) / SUM(COUNT_STAR))/1000000, 2) avg_usec, ROUND(MAX(max_timer_wait) / 1000000, 2) max_usec FROM performance_schema.events_waits_summary_global_by_event_name WHERE sum_timer_wait > 0 GROUP BY event_name ORDER BY total_sec DESC limit 10; Top waits by latency
  • Performance Schema in 5.5 THREAD_ID: 6 EVENT_ID: 1036 EVENT_NAME: wait/io/file/innodb/innodb_data_file SOURCE: os0file.c:4891 TIMER_START: 338825682207740 TIMER_END: 338825703463000 TIMER_WAIT: 21255260 SPINS: NULL OBJECT_SCHEMA: NULL OBJECT_NAME: /Users/mark/sandboxes/msb_5_5_16/data/ibdata1 OBJECT_TYPE: FILE OBJECT_INSTANCE_BEGIN: 4683519488 NESTING_EVENT_ID: NULL OPERATION: write NUMBER_OF_BYTES: 32768 FLAGS: 0 File IO Event ~21 microsecs To write 32K On this file
  • Performance Schema in 5.5 SELECT substring_index(file_name, '/', -1) AS file, count_read, count_write, ROUND(sum_number_of_bytes_read / (1024*1024)) read_mb, ROUND(sum_number_of_bytes_write / (1024*1024)) write_mb, ROUND((sum_number_of_bytes_read + sum_number_of_bytes_write) / (1024*1024*1024)) total_gb FROM performance_schema.file_summary_by_instance ORDER BY total_gb DESC LIMIT 10; Top files by IO consumption
  • Performance Schema in 5.5
    • Proved the model, can handle the “flood”
    • Already aided finding bottlenecks
    • But..
      • Needed more DBA/Developer focused statistics
      • Needed more control on who/what is traced
      • Needed a little tuning for high load systems
  • Performance Schema in 5.6
    • Adding DBA / Developer focused instrumentation
    • With the ability to drill down between each level
    • Statements – Latency and key statistics
      • Stages – Statement execution states (i.e. “ Sending data ”)
        • Network IO / Idle time - at the TCP / socket layers
        • Table IO - at the storage engine handler layer
        • Table Locks - at the storage engine handler layer
        • .. as well as file IO , mutexes , conditions , rwlocks
  • Performance Schema Statement Statistics
  • Performance Schema Stages The “stage” (state) Within statement execution When it started, ended, and How long it took The statement ID it was related to
  • Performance Schema in 5.6 – Event Nesting
    • Events are related in a hierarchy
    • Currently uses the “Adjacency List” model
    • EVENT_ID (child) -> NESTING_EVENT_ID (parent)
    • Investigating the “Nested Set” model
  • Performance Schema in 5.6 – Event Nesting
  • Performance Schema in 5.6 – Graph Events The Statement The Stages The individual wait events by type
  • Performance Schema in 5.6 – Graph Events
  • Performance Schema in 5.6 – Summaries Summarize (GROUP BY) each class of event by user, host, account ( [email_address] ), thread, and globally New Object, Network IO Table IO and Table Lock Wait summaries
  • Performance Schema Table IO Summary The table and it's high level aggregated stats A breakdown By table IO type
  • “ Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein Yet 60 nanoseconds of time could mean SO MUCH!
  • Performance Schema Configuration
    • There is always some overhead with instrumentation
    • Whether, and how, we time events can be important
    • Set expectations up front on both:
      • What you want to monitor
      • How selectively you want to monitor it
  • Performance Schema Overhead June 2011 Lots of tuning done since this. New benchmarks should be out soon
  • Performance Schema Setup
  • Performance Schema – What's Missing?
    • Storage engine level lock latency – i.e. InnoDB row locks
    • User lock latency – i.e. SELECT GET_LOCK();
    • A normalized statement level digest
    • Transactions level statistics (tied to global transaction ID)
    • Memory allocations
    • Distribution statistics (histograms)
    • Background thread “stages”, server cache statistics
    • ...and...and...and... - We're listening, tell us your needs
  • New InnoDB Instrumentation
  • New InnoDB Instrumentation
    • MySQL 5.6: 11 New INFORMATION_SCHEMA tables
      • 7 Data Dictionary related
      • 3 Buffer Pool related
      • 1 General Statistics gold mine
    • MySQL 5.5: 7 New INFORMATION_SCHEMA tables
      • 3 Transaction / Locking related
      • 4 Compression related
  • InnoDB System Tables View InnoDB's internal Data Dictionary and table stats
  • INNODB_SYS_TABLES[TATS] SELECT, CASE flag >> 5 WHEN 0 THEN 'Antelope' WHEN 1 THEN 'Barracuda' WHEN 2 THEN 'Cheetah' END AS file_format, @page_flags := (flag >> 1) & 15 AS page_flags, @page_size := IF(@page_flags = 0,16384, 512 << @page_flags) AS page_size, ROUND(( @page_size * clust_index_size ) /(1024*1024)) AS pk_mb, ROUND(( @page_size * other_index_size ) /(1024*1024)) AS secidx_mb FROM innodb_sys_tables JOIN innodb_sys_tablestats USING (table_id) ORDER BY pk_mb DESC;
  • INNODB_BUFFER_PAGE - Summarized SELECT page_type, page_state, table_name, index_name, SUM(data_size) / (1024*1024) AS used_mb, SUM( IF(compressed_size = 0, 16384, compressed_size) ) / (1024*1024) AS total_mb FROM innodb_buffer_page GROUP BY page_type, page_state, table_name, index_name HAVING total_mb > 0 ORDER BY total_mb DESC;
    • Consolidated InnoDB statistics
    • 199 metrics currently collected ( suggest just one more! )
    • More flexible statistics collection than SHOW STATUS
      • Can enable/disable individually
      • Track when / how long each has been enabled
      • Provide extra statistics inline per metric
      • Provide extra metadata inline per metric
  • INNODB_METRICS – Metric Details The variable Stats since start Stats since “reset” How long enabled Stat Metadata
  • INNODB_METRICS – Configuration Enable named metric Disable named metric Enable metric “module” Reset metric “module”
  • INNODB_METRICS – Looking at Counters SELECT concat(subsystem,'/', name) AS event, count, ROUND(count / time_elapsed) AS avg_per_second FROM information_schema.innodb_metrics WHERE status = 'enabled' AND type LIKE '%counter%' OR type = 'set_owner' ORDER BY avg_per_second DESC;
  • Optimizer Trace
  • Optimizer Trace
    • EXPLAIN tells you how a query will execute
    • Optimizer Trace tells you why it will execute that way
    • Contains trace of optimizer decisions, in JSON format
    • Enabled per thread, on demand
    • Targeted for advanced users, or in use with bug reporting
  • Optimizer Trace Enable Tracing Run Statement Select the trace
  • Optimizer Trace Expose query transformations
  • Optimizer Trace Considered plans With costs Estimations Of rows And costs Whether Indexes are Usable or not
  • Enhanced EXPLAIN
  • Enhanced EXPLAIN Plans
    • In the past EXPLAIN was only supported with SELECT
    • To EXPLAIN an UPDATE - “Re-write equivalent SELECT”
      • However optimization paths may be different
    • Now you can EXPLAIN:
  • Enhanced EXPLAIN Plans
    • “ Structured EXPLAIN” is in development
    • A properly structured EXPLAIN plan (again, JSON)
    • Sorry – no examples yet, it's very early in implementation
  • Other Goodies
    • Automatically generated unique instance identifiers
    • Persisted in $datadir/auto.cnf
    • Can be used for reliable replication topology discovery
      • Hard in past given IP/Host matching (virtual IPs?)
    More configuration and thread statistics
  • Q&A