Your SlideShare is downloading. ×
0
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Mastering InnoDB Diagnostics
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Mastering InnoDB Diagnostics

6,552

Published on

Presentation from MySQL Conference 2010

Presentation from MySQL Conference 2010

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,552
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
256
Comments
0
Likes
10
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer
  • 2. Overview <ul><li>Lots of information </li></ul><ul><ul><li>Hard to understand
  • 3. Easy to get overwhelmed </li></ul></ul><ul><li>Hard to figure out what is useful
  • 4. Two methods for troubleshooting </li></ul><ul><ul><li>Symptom based </li><ul><li>What we will be doing today </li></ul><li>Data based </li></ul></ul>
  • 5. Information Sources
  • 6. Sources <ul><li>MySQL info </li></ul><ul><ul><li>SHOW ENGINE INNODB STATUS
  • 7. InnoDB monitors
  • 8. SHOW GLOBAL STATUS
  • 9. INFORMATION_SCHEMA tables </li></ul></ul><ul><ul><li>SHOW PROCESSLIST
  • 10. SHOW MUTEX STATUS </li></ul><li>OS info </li></ul><ul><ul><li>vmstat / iostat or Performance Monitor
  • 11. Gdb ( Poor mans profiler (PMP))
  • 12. OS specific tools </li></ul></ul>
  • 13. SHOW INNODB STATUS <ul><li>Be careful of truncation </li><ul><li>innodb_status_file
  • 14. Table monitors </li></ul><li>Ensure enough time has passed
  • 15. Exact format varies </li><ul><li>InnoDB built-in
  • 16. InnoDB plugin
  • 17. XtraDB </li></ul></ul>
  • 18. Data Dictionary
  • 19. Problems <ul>InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted the .frm file InnoDB: and not used DROP TABLE? InnoDB: Cannot find table test/child2 from the internal data dictionary InnoDB: of InnoDB though the .frm file for the table exists. Maybe you InnoDB: have deleted and recreated InnoDB data files but have forgotten InnoDB: to delete the corresponding .frm files of InnoDB tables? </ul><ul><li>Various errors about not being in sync
  • 20. Can't drop and/or create tables or databases
  • 21. Example errors: </li></ul>
  • 22. Diagnostics <ul><li>InnoDB table monitor </li></ul><ul><ul><li>Dumps InnoDB data dictionary
  • 23. Output to error log
  • 24. CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb; </li></ul></ul>
  • 25. Interpreting <ul><li>Ignore system tables </li><ul><li>Named SYS_ </li></ul><li>Ignore hidden fields from InnoDB </li></ul><ul><ul><li>DB_ROW_ID
  • 26. DB_TRX_ID
  • 27. DB_ROLL_PTR </li></ul></ul><ul><li>Ignore hidden indexes </li><ul><li>GEN_CLUST_INDEX </li></ul></ul>
  • 28. Solutions <ul><li>Ensure .frm matches InnoDB </li></ul><ul><ul><li>Create .frm as needed </li><ul><li>Create another table with same structure
  • 29. Copy .frm to the name of the table </li></ul><li>Drop orphaned .frm files </li><ul><li>If not in InnoDB
  • 30. Remove them </li></ul></ul></ul>
  • 31. Crashing
  • 32. Problem <ul><li>mysqld Crash </li></ul><ul><ul><li>Signal 11 </li><ul><li>Segfault </li></ul><li>Signal 6 </li><ul><li>Abort
  • 33. InnoDB loves this one </li></ul></ul></ul><ul><li>May not actually notice immediately </li></ul><ul><ul><li>mysqld_safe restarts </li></ul></ul>
  • 34. Diagnostics <ul><li>Error log </li></ul><ul><ul><li>Hopefully it has info in it </li></ul></ul><ul><li>Core file </li></ul><ul><ul><li>Not enabled by default
  • 35. OS often restricts setuid cores </li><ul><li>/proc/sys/fs/suid_dumpable
  • 36. coreadm </li></ul></ul></ul><ul><li>General log </li></ul><ul><ul><li>When all else fails </li></ul></ul>
  • 37. Interpreting Error log <ul><li>Error log </li></ul><ul><ul><li>Look for signal type </li><ul><li>Signal 11
  • 38. Signal 6 </li></ul><li>Backtrace </li><ul><li>May or may not be present </li><ul><li>OS
  • 39. Version </li></ul><li>In 5.0 and before you need to resolve it </li><ul><li>resolve_stack_dump </li></ul><li>Search bug system for function names </li></ul></ul></ul>
  • 40. Interpreting Cores <ul><li>Must be enabled
  • 41. Memory image produced
  • 42. Requires exactly the same binary and libraries
  • 43. With rpm ensure debuginfo is installed
  • 44. Non-stripped binaries
  • 45. Load with gdb </li></ul><ul><ul><li>thread apply all bt
  • 46. Use PMP to summarize if needed </li></ul></ul><ul><li>Solaris: pstack
  • 47. Windows: Minidump/DrWatson/Windbg </li></ul>
  • 48. Solutions <ul><li>Find cause of the crash </li></ul><ul><ul><li>InnoDB doing on purpose
  • 49. Real crash </li></ul></ul><ul><li>Find query causing problem </li></ul><ul><ul><li>Stop doing it
  • 50. Report bug to us </li></ul></ul><ul><li>Upgrade </li></ul><ul><ul><li>Lots of bugs fixed each release </li></ul></ul>
  • 51. Locking
  • 52. Problems <ul><li>Deadlocks </li></ul><ul><ul><li>Real deadlocks </li></ul></ul><ul><ul><li>ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction </li></ul></ul><ul><li>Lock wait timeouts </li><ul><li>ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction </li></ul><li>General locking slowness </li></ul>
  • 53. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>TRANSACTIONS section </li></ul></ul><ul><li>innodb_lock_monitor table </li></ul><ul><ul><li>Changes output of SHOW INNODB STATUS </li></ul></ul><ul><li>INFORMATION_SCHEMA </li></ul><ul><ul><li>INNODB_LOCKS
  • 54. INNODB_LOCK_WAITS </li></ul></ul><ul><li>SHOW GLOBAL STATUS LIKE 'innodb_row_lock%' </li></ul>
  • 55. Interpreting <ul><li>Find what lock is needed </li></ul><ul><ul><li>Who is holding conflicting lock </li></ul></ul><ul><li>Look for type of lock </li></ul><ul><ul><li>X (exclusive)
  • 56. S (shared) </li><ul><li>Relatively rare </li></ul><li>Others </li></ul></ul><ul><li>Look for what the lock is on </li></ul><ul><ul><li>Table (primary key)
  • 57. Index </li></ul></ul>
  • 58. Solutions <ul><li>Fix queries to reduce locks
  • 59. Shared locks </li></ul><ul><ul><li>Find source
  • 60. Eliminate </li><ul><li>Upgrade to 5.1 run in read committed
  • 61. Remove source of lock </li></ul></ul></ul><ul><li>AUTO-INC </li></ul><ul><ul><li>Upgrade to 5.1 </li><ul><li>innodb_autoinc_lock = 2? </li></ul></ul></ul><ul><li>Long running transactions </li></ul>
  • 62. Performance
  • 63. Overview <ul><li>Lots of potential performance issues </li></ul><ul><ul><li>Disk I/O </li><ul><li>Log files
  • 64. Table space </li></ul><li>CPU </li><ul><li>Too many rows being read
  • 65. Thread thrashing </li></ul></ul></ul><ul><li>Gather lots of information up front </li></ul><ul><ul><li>Big picture, not details first </li></ul></ul><ul><li>Ensure it is during “problem” time </li></ul>
  • 66. Disk: Log files <ul><li>Many small serialized synchronous writes
  • 67. InnoDB redo log writes bottleneck </li></ul><ul><ul><li>fdatasync () is expensive in many setups </li></ul></ul><ul><li>innodb_flush_log_at_trx_commit </li></ul><ul><ul><li>1 Safest, sync every commit
  • 68. 2 Write every commit
  • 69. 0 Write only every second </li></ul></ul>
  • 70. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>State of transactions </li><ul><li>COMMITTED IN MEMORY
  • 71. PREPARED </li></ul><li>FILE I/O </li><ul><li>Pending flushes (fsync) log
  • 72. fsyncs/s </li></ul></ul></ul><ul><li>iostat </li></ul><ul><ul><li>Number of write operations ( w/s ) </li></ul></ul><ul><ul><li>await and/or svctm </li></ul></ul>
  • 73. Solutions <ul><li>InnoDB plugin group commit
  • 74. Change innodb_flush_log_at_trx_commit
  • 75. Better disks </li></ul><ul><ul><li>BBU write cache
  • 76. Faster writes </li></ul></ul>
  • 77. Disk: Tablespace <ul><li>Reading and writing your actual tables and indexes
  • 78. More common with very large tables </li></ul><ul><ul><li>Compare to buffer pool size </li></ul></ul><ul><li>Sometimes caused by read ahead in InnoDB </li></ul>
  • 79. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>Buffer pool hit rate </li><ul><li>Watch absolute values too </li></ul><li>FILE I/O </li><ul><li>Reads/s </li><ul><li>Avg bytes/read shows reada head </li></ul><li>Pending reads </li></ul><li>Read ahead </li><ul><li>Pages read ahead/evicted without access </li></ul></ul></ul><ul><li>iostat </li><ul><li>avgqu-sz
  • 80. await </li></ul></ul>
  • 81. Solutions <ul><li>Optimize queries
  • 82. Increase innodb_buffer_pool_size
  • 83. New LRU algorithm in InnoDB plugin
  • 84. Tune/disable readahead </li><ul><li>In plugin, can be tuned
  • 85. In non-plugin </li><ul><li>Use gdb hack
  • 86. set srv_startup_is_before_trx_rollback_phase=1 </li></ul></ul><li>Buy better disks </li></ul>
  • 87. CPU: Lots of Rows <ul><li>Bad queries can read lots of rows </li></ul><ul><ul><li>Or lots and lots of smaller queries </li></ul></ul><ul><li>Data fully cached </li></ul><ul><ul><li>Lots of buffer pool reading
  • 88. Often smaller dataset </li></ul></ul><ul><li>Eventually will bottleneck InnoDB on CPU </li></ul>
  • 89. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>ROW OPERATIONS </li><ul><li>Reads/s </li></ul></ul></ul><ul><li>SHOW GLOBAL STATUS LIKE 'Handler%' </li></ul><ul><ul><li>Type of row operations going on </li></ul></ul><ul><li>Slow query log </li></ul><ul><ul><li>log-queries-not-using-indexes </li></ul></ul>
  • 90. Solutions <ul><li>Fix your queries </li></ul><ul><ul><li>Ensure proper index usage </li></ul></ul><ul><li>Do less queries </li></ul><ul><ul><li>External caching </li></ul></ul><ul><li>Scale reads </li></ul><ul><ul><li>Replication </li></ul></ul>
  • 91. CPU: Thread Thrashing <ul><li>InnoDB can have issues at high concurrency
  • 92. Later releases help reduce this dramatically </li></ul><ul><ul><li>InnoDB plugin
  • 93. Further versions </li><ul><li>XtraDB
  • 94. MySQL 5.5 </li></ul></ul></ul>
  • 95. Diagnostics <ul><li>SHOW INNODB STATUS </li></ul><ul><ul><li>Lots of queued threads </li><ul><li>TRANSACTIONS
  • 96. Queries in queue </li></ul></ul></ul><ul><ul><li>ROW OPERATIONS </li></ul></ul><ul><ul><ul><li>reads/s is “low” </li></ul></ul></ul><ul><li>vmstat </li></ul><ul><ul><li>Very high CPU usage
  • 97. Lots of “cs” context switches </li></ul></ul><ul><li>SHOW MUTEX STATUS </li></ul>
  • 98. Solutions <ul><li>innodb_thread_concurrency </li></ul><ul><ul><li>Very high or 0 works best until it doesn't </li></ul></ul><ul><ul><li>Sometimes very low works best
  • 99. A bit of a black art in tuning </li></ul></ul><ul><li>innodb_concurrency_tickets </li></ul><ul><ul><li>Size based on number of rows being touched </li></ul></ul><ul><li>Use newer version </li><ul><li>Lots of work to help this in InnoDB Plugin </li></ul></ul>
  • 100. Questions?
  • 101. The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  • 102.  

×