Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer
Overview <ul><li>Lots of information </li></ul><ul><ul><li>Hard to understand
Easy to get overwhelmed </li></ul></ul><ul><li>Hard to figure out what is useful
Two methods for troubleshooting </li></ul><ul><ul><li>Symptom based </li><ul><li>What we will be doing today </li></ul><li...
Information Sources
Sources <ul><li>MySQL info </li></ul><ul><ul><li>SHOW ENGINE INNODB STATUS
InnoDB monitors
SHOW GLOBAL STATUS
INFORMATION_SCHEMA  tables </li></ul></ul><ul><ul><li>SHOW PROCESSLIST
SHOW MUTEX STATUS </li></ul><li>OS info </li></ul><ul><ul><li>vmstat / iostat  or Performance Monitor
Gdb ( Poor mans profiler (PMP))
OS specific tools  </li></ul></ul>
SHOW INNODB STATUS <ul><li>Be careful of truncation </li><ul><li>innodb_status_file
Table monitors </li></ul><li>Ensure enough time has passed
Exact format varies </li><ul><li>InnoDB built-in
InnoDB plugin
XtraDB </li></ul></ul>
Data Dictionary
Problems <ul>InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted ...
Can't drop and/or create tables or databases
Example errors: </li></ul>
Diagnostics <ul><li>InnoDB table monitor </li></ul><ul><ul><li>Dumps InnoDB data dictionary
Output to error log
CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb; </li></ul></ul>
Interpreting <ul><li>Ignore system tables </li><ul><li>Named  SYS_ </li></ul><li>Ignore hidden fields from InnoDB </li></u...
DB_TRX_ID
DB_ROLL_PTR </li></ul></ul><ul><li>Ignore hidden indexes </li><ul><li>GEN_CLUST_INDEX </li></ul></ul>
Solutions <ul><li>Ensure  .frm  matches InnoDB </li></ul><ul><ul><li>Create  .frm  as needed </li><ul><li>Create another t...
Copy  .frm  to the name of the table </li></ul><li>Drop orphaned  .frm  files </li><ul><li>If not in InnoDB
Remove them </li></ul></ul></ul>
Crashing
Problem <ul><li>mysqld Crash </li></ul><ul><ul><li>Signal 11 </li><ul><li>Segfault </li></ul><li>Signal 6 </li><ul><li>Abort
InnoDB loves this one </li></ul></ul></ul><ul><li>May not actually notice immediately </li></ul><ul><ul><li>mysqld_safe re...
Diagnostics <ul><li>Error log </li></ul><ul><ul><li>Hopefully it has info in it </li></ul></ul><ul><li>Core file </li></ul...
OS often restricts setuid cores </li><ul><li>/proc/sys/fs/suid_dumpable
coreadm   </li></ul></ul></ul><ul><li>General log </li></ul><ul><ul><li>When all else fails </li></ul></ul>
Interpreting Error log <ul><li>Error log </li></ul><ul><ul><li>Look for signal type </li><ul><li>Signal 11
Signal 6 </li></ul><li>Backtrace </li><ul><li>May or may not be present </li><ul><li>OS
Upcoming SlideShare
Loading in …5
×

Harrison fisk masteringinnodb-diagnostics

2,519
-1

Published on

2 Comments
13 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,519
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
118
Comments
2
Likes
13
Embeds 0
No embeds

No notes for slide

Harrison fisk masteringinnodb-diagnostics

  1. 1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer
  2. 2. Overview <ul><li>Lots of information </li></ul><ul><ul><li>Hard to understand
  3. 3. Easy to get overwhelmed </li></ul></ul><ul><li>Hard to figure out what is useful
  4. 4. Two methods for troubleshooting </li></ul><ul><ul><li>Symptom based </li><ul><li>What we will be doing today </li></ul><li>Data based </li></ul></ul>
  5. 5. Information Sources
  6. 6. Sources <ul><li>MySQL info </li></ul><ul><ul><li>SHOW ENGINE INNODB STATUS
  7. 7. InnoDB monitors
  8. 8. SHOW GLOBAL STATUS
  9. 9. INFORMATION_SCHEMA tables </li></ul></ul><ul><ul><li>SHOW PROCESSLIST
  10. 10. SHOW MUTEX STATUS </li></ul><li>OS info </li></ul><ul><ul><li>vmstat / iostat or Performance Monitor
  11. 11. Gdb ( Poor mans profiler (PMP))
  12. 12. OS specific tools </li></ul></ul>
  13. 13. SHOW INNODB STATUS <ul><li>Be careful of truncation </li><ul><li>innodb_status_file
  14. 14. Table monitors </li></ul><li>Ensure enough time has passed
  15. 15. Exact format varies </li><ul><li>InnoDB built-in
  16. 16. InnoDB plugin
  17. 17. XtraDB </li></ul></ul>
  18. 18. Data Dictionary
  19. 19. Problems <ul>InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted the .frm file InnoDB: and not used DROP TABLE? InnoDB: Cannot find table test/child2 from the internal data dictionary InnoDB: of InnoDB though the .frm file for the table exists. Maybe you InnoDB: have deleted and recreated InnoDB data files but have forgotten InnoDB: to delete the corresponding .frm files of InnoDB tables? </ul><ul><li>Various errors about not being in sync
  20. 20. Can't drop and/or create tables or databases
  21. 21. Example errors: </li></ul>
  22. 22. Diagnostics <ul><li>InnoDB table monitor </li></ul><ul><ul><li>Dumps InnoDB data dictionary
  23. 23. Output to error log
  24. 24. CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb; </li></ul></ul>
  25. 25. Interpreting <ul><li>Ignore system tables </li><ul><li>Named SYS_ </li></ul><li>Ignore hidden fields from InnoDB </li></ul><ul><ul><li>DB_ROW_ID
  26. 26. DB_TRX_ID
  27. 27. DB_ROLL_PTR </li></ul></ul><ul><li>Ignore hidden indexes </li><ul><li>GEN_CLUST_INDEX </li></ul></ul>
  28. 28. Solutions <ul><li>Ensure .frm matches InnoDB </li></ul><ul><ul><li>Create .frm as needed </li><ul><li>Create another table with same structure
  29. 29. Copy .frm to the name of the table </li></ul><li>Drop orphaned .frm files </li><ul><li>If not in InnoDB
  30. 30. Remove them </li></ul></ul></ul>
  31. 31. Crashing
  32. 32. Problem <ul><li>mysqld Crash </li></ul><ul><ul><li>Signal 11 </li><ul><li>Segfault </li></ul><li>Signal 6 </li><ul><li>Abort
  33. 33. InnoDB loves this one </li></ul></ul></ul><ul><li>May not actually notice immediately </li></ul><ul><ul><li>mysqld_safe restarts </li></ul></ul>
  34. 34. Diagnostics <ul><li>Error log </li></ul><ul><ul><li>Hopefully it has info in it </li></ul></ul><ul><li>Core file </li></ul><ul><ul><li>Not enabled by default
  35. 35. OS often restricts setuid cores </li><ul><li>/proc/sys/fs/suid_dumpable
  36. 36. coreadm </li></ul></ul></ul><ul><li>General log </li></ul><ul><ul><li>When all else fails </li></ul></ul>
  37. 37. Interpreting Error log <ul><li>Error log </li></ul><ul><ul><li>Look for signal type </li><ul><li>Signal 11
  38. 38. Signal 6 </li></ul><li>Backtrace </li><ul><li>May or may not be present </li><ul><li>OS
  39. 39. Version </li></ul><li>In 5.0 and before you need to resolve it </li><ul><li>resolve_stack_dump </li></ul><li>Search bug system for function names </li></ul></ul></ul>
  40. 40. Interpreting Cores <ul><li>Must be enabled
  41. 41. Memory image produced
  42. 42. Requires exactly the same binary and libraries
  43. 43. With rpm ensure debuginfo is installed
  44. 44. Non-stripped binaries
  45. 45. Load with gdb </li></ul><ul><ul><li>thread apply all bt
  46. 46. Use PMP to summarize if needed </li></ul></ul><ul><li>Solaris: pstack
  47. 47. Windows: Minidump/DrWatson/Windbg </li></ul>
  48. 48. Solutions <ul><li>Find cause of the crash </li></ul><ul><ul><li>InnoDB doing on purpose
  49. 49. Real crash </li></ul></ul><ul><li>Find query causing problem </li></ul><ul><ul><li>Stop doing it
  50. 50. Report bug to us </li></ul></ul><ul><li>Upgrade </li></ul><ul><ul><li>Lots of bugs fixed each release </li></ul></ul>
  51. 51. Locking
  52. 52. Problems <ul><li>Deadlocks </li></ul><ul><ul><li>Real deadlocks </li></ul></ul><ul><ul><li>ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction </li></ul></ul><ul><li>Lock wait timeouts </li><ul><li>ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction </li></ul><li>General locking slowness </li></ul>
  53. 53. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>TRANSACTIONS section </li></ul></ul><ul><li>innodb_lock_monitor table </li></ul><ul><ul><li>Changes output of SHOW INNODB STATUS </li></ul></ul><ul><li>INFORMATION_SCHEMA </li></ul><ul><ul><li>INNODB_LOCKS
  54. 54. INNODB_LOCK_WAITS </li></ul></ul><ul><li>SHOW GLOBAL STATUS LIKE 'innodb_row_lock%' </li></ul>
  55. 55. Interpreting <ul><li>Find what lock is needed </li></ul><ul><ul><li>Who is holding conflicting lock </li></ul></ul><ul><li>Look for type of lock </li></ul><ul><ul><li>X (exclusive)
  56. 56. S (shared) </li><ul><li>Relatively rare </li></ul><li>Others </li></ul></ul><ul><li>Look for what the lock is on </li></ul><ul><ul><li>Table (primary key)
  57. 57. Index </li></ul></ul>
  58. 58. Solutions <ul><li>Fix queries to reduce locks
  59. 59. Shared locks </li></ul><ul><ul><li>Find source
  60. 60. Eliminate </li><ul><li>Upgrade to 5.1 run in read committed
  61. 61. Remove source of lock </li></ul></ul></ul><ul><li>AUTO-INC </li></ul><ul><ul><li>Upgrade to 5.1 </li><ul><li>innodb_autoinc_lock = 2? </li></ul></ul></ul><ul><li>Long running transactions </li></ul>
  62. 62. Performance
  63. 63. Overview <ul><li>Lots of potential performance issues </li></ul><ul><ul><li>Disk I/O </li><ul><li>Log files
  64. 64. Table space </li></ul><li>CPU </li><ul><li>Too many rows being read
  65. 65. Thread thrashing </li></ul></ul></ul><ul><li>Gather lots of information up front </li></ul><ul><ul><li>Big picture, not details first </li></ul></ul><ul><li>Ensure it is during “problem” time </li></ul>
  66. 66. Disk: Log files <ul><li>Many small serialized synchronous writes
  67. 67. InnoDB redo log writes bottleneck </li></ul><ul><ul><li>fdatasync () is expensive in many setups </li></ul></ul><ul><li>innodb_flush_log_at_trx_commit </li></ul><ul><ul><li>1 Safest, sync every commit
  68. 68. 2 Write every commit
  69. 69. 0 Write only every second </li></ul></ul>
  70. 70. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>State of transactions </li><ul><li>COMMITTED IN MEMORY
  71. 71. PREPARED </li></ul><li>FILE I/O </li><ul><li>Pending flushes (fsync) log
  72. 72. fsyncs/s </li></ul></ul></ul><ul><li>iostat </li></ul><ul><ul><li>Number of write operations ( w/s ) </li></ul></ul><ul><ul><li>await and/or svctm </li></ul></ul>
  73. 73. Solutions <ul><li>InnoDB plugin group commit
  74. 74. Change innodb_flush_log_at_trx_commit
  75. 75. Better disks </li></ul><ul><ul><li>BBU write cache
  76. 76. Faster writes </li></ul></ul>
  77. 77. Disk: Tablespace <ul><li>Reading and writing your actual tables and indexes
  78. 78. More common with very large tables </li></ul><ul><ul><li>Compare to buffer pool size </li></ul></ul><ul><li>Sometimes caused by read ahead in InnoDB </li></ul>
  79. 79. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>Buffer pool hit rate </li><ul><li>Watch absolute values too </li></ul><li>FILE I/O </li><ul><li>Reads/s </li><ul><li>Avg bytes/read shows reada head </li></ul><li>Pending reads </li></ul><li>Read ahead </li><ul><li>Pages read ahead/evicted without access </li></ul></ul></ul><ul><li>iostat </li><ul><li>avgqu-sz
  80. 80. await </li></ul></ul>
  81. 81. Solutions <ul><li>Optimize queries
  82. 82. Increase innodb_buffer_pool_size
  83. 83. New LRU algorithm in InnoDB plugin
  84. 84. Tune/disable readahead </li><ul><li>In plugin, can be tuned
  85. 85. In non-plugin </li><ul><li>Use gdb hack
  86. 86. set srv_startup_is_before_trx_rollback_phase=1 </li></ul></ul><li>Buy better disks </li></ul>
  87. 87. CPU: Lots of Rows <ul><li>Bad queries can read lots of rows </li></ul><ul><ul><li>Or lots and lots of smaller queries </li></ul></ul><ul><li>Data fully cached </li></ul><ul><ul><li>Lots of buffer pool reading
  88. 88. Often smaller dataset </li></ul></ul><ul><li>Eventually will bottleneck InnoDB on CPU </li></ul>
  89. 89. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>ROW OPERATIONS </li><ul><li>Reads/s </li></ul></ul></ul><ul><li>SHOW GLOBAL STATUS LIKE 'Handler%' </li></ul><ul><ul><li>Type of row operations going on </li></ul></ul><ul><li>Slow query log </li></ul><ul><ul><li>log-queries-not-using-indexes </li></ul></ul>
  90. 90. Solutions <ul><li>Fix your queries </li></ul><ul><ul><li>Ensure proper index usage </li></ul></ul><ul><li>Do less queries </li></ul><ul><ul><li>External caching </li></ul></ul><ul><li>Scale reads </li></ul><ul><ul><li>Replication </li></ul></ul>
  91. 91. CPU: Thread Thrashing <ul><li>InnoDB can have issues at high concurrency
  92. 92. Later releases help reduce this dramatically </li></ul><ul><ul><li>InnoDB plugin
  93. 93. Further versions </li><ul><li>XtraDB
  94. 94. MySQL 5.5 </li></ul></ul></ul>
  95. 95. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>Lots of queued threads </li><ul><li>TRANSACTIONS
  96. 96. Queries in queue </li></ul></ul></ul><ul><ul><li>ROW OPERATIONS </li></ul></ul><ul><ul><ul><li>reads/s is “low” </li></ul></ul></ul><ul><li>vmstat </li></ul><ul><ul><li>Very high CPU usage
  97. 97. Lots of “cs” context switches </li></ul></ul><ul><li>SHOW MUTEX STATUS </li></ul>
  98. 98. Solutions <ul><li>innodb_thread_concurrency </li></ul><ul><ul><li>Very high or 0 works best until it doesn't </li></ul></ul><ul><ul><li>Sometimes very low works best
  99. 99. A bit of a black art in tuning </li></ul></ul><ul><li>innodb_concurrency_tickets </li></ul><ul><ul><li>Size based on number of rows being touched </li></ul></ul><ul><li>Use newer version </li><ul><li>Lots of work to help this in InnoDB Plugin </li></ul></ul>
  100. 100. Questions?
  101. 101. The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×