3. #cpgethightech #cpanelconf
A Quick Refresher
• InnoDB is the default engine as of MySQL 5.5.5+
• Non-portable file structure
• Crash recovery functionality using redo logs
• Uses the data dictionary to maintain table
definitions, structures, and indexes.
• Physical file structure made up of tablespaces,
pages, extents, segments, and inodes.
• Principles of ACID compliance
4. #cpgethightech #cpanelconf
DataDictionary
Table
id
flags
columns
indexes
appr. rows
Columns
names
lengths
data types
Indexes
id
fields
unique?
location
Constraints
What does the data dictionary contain?
Tables
• IDs
• Structural
Metadata
Columns
• Names
• Attributes
Indexes
• IDs
• Types
Where is the data dictionary stored?
ibdata
1
Stores the system tablespace, which
contains the data dictionary
Composed internally of "system tables"
that store metadata on table structures.
Stored by default in the data directory,
which defaults to /var/lib/mysql
5. #cpgethightech #cpanelconf
Removing this is essentially the same as
removing access to all of your InnoDB tables.
That's the only reason that this can appear to
"fix" a crashed MySQL environment.
ibdata1
Can be removed for re-creation fairly safely,
and can be necessary in some
recovery/repair situations.
ib_logfileX
In the Data Directory
9. #cpgethightech #cpanelconf
Deciphering the Error Logs
Error
Type
s
Data Dictionary
Checksum Mismatch
Unable to
open/read/write/access file
Assertion failure
Signal 6/Signal 11
Page checksum 863794368, prior-to-4.0.14-form checksum
stored checksum 1169627998, prior-to-4.0.14-form stored
checksum 2838245694
Examples
Can't open table from data dictionary though frm
file exists
Error: trying to open a table, but could not open the
tablespace file './horde/horde_cache.ibd'!
Assertion failure in thread 3067665264 in file
buf0buf.cc line 2641
20:13:06 UTC - mysqld got signal 6 ;
10. #cpgethightech #cpanelconf
Data Dictionary Errors
Why do they occur? What are your options?
Deleted or moved ibdata1 file
Migrated from another environment
Copies are made, but extensions
are left intact
Files are backed up directly, then
restored
Restore from a backup
Recover an ibdata1 file
Try to establish a copy of the data
from a dump or table copy
Recover the CREATE TABLE
statement. re-create the table.
11. #cpgethightech #cpanelconf
Alchemy Solutions
WARNING
Before attempting any alternative recovery solutions, always make sure to back up
your MySQL data in whatever way possible:
Perform MySQL dumps (when possible):
mysqldump –AER > /root/dump.sql
Make a copy of the MySQL data directory (or at least the ibdata* and ib_log* files):
cp –Rp /var/lib/mysql /var/lib/mysql.bak
12. #cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
Name Byte Length Offset Description
FIL_PAGE_SPACE_OR_CHKSUM 4 0 Checksum (Space ID in older MySQL versions)
FIL_PAGE_OFFSET 4 4 Page Number
FIL_PAGE_PREV 4 8 Previous Page (in key order)
FIL_PAGE_NEXT 4 12 Next Page (in key order)
FIL_PAGE_LSN 8 16 LSN of page’s latest log record
FIL_PAGE_TYPE 2 24 Page Type
FIL_PAGE_FILE_FLUSH_LSN 8 26 Flushed-up-to LSN (only in space ID 0, page 0)
FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID 4 34 Space ID
InnoDB data is frequently stored in predictable, usable patterns at the byte level, in a page-by-page
format, each page representing a 16384-byte block of data. Using this, you can directly modify a
number of values in files stored by InnoDB to affect their behavior.
storage/innobase/include/fil0fil.h
13. #cpgethightech #cpanelconf
Alchemy Solutions
Data dictionary example
scenario:Space ID conflict shows in the error log ( /var/lib/mysql/`hostname`.err ):
Error: tablespace id in file ‘dictionary.ibd' is 12, but in the InnoDB data dictionary it is 6.
-bash-4.1# xxd -ps -s 34 -l 4 dictionary.ibd
0000000c
-bash-4.1# xxd -ps -s 38 -l 4 dictionary.ibd
0000000c
-bash-4.1# echo $((0x0c))
12
Hex value stored in two primary locations in the .ibd file:
14. #cpgethightech #cpanelconf
Alchemy Solutions
Data dictionary example
scenario:Writing the new space ID to the appropriate byte locations in the tablespace file (.ibd file):
-bash-4.1# cd /var/lib/mysql/roundcube
-bash-4.1# printf ‘x00x00x00x06’ | dd of=dictionary.ibd bs=1 seek=34 count=4 conv=notrunc
-bash-4.1# printf ‘x00x00x00x06’ | dd of=dictionary.ibd bs=1 seek=38 count=4 conv=notrunc
The stored space ID within this tablespace file is now “6”, in the primary two locations.
Once the table is successfully loaded back into MySQL, be sure to make a dump, drop the table,
and re-import that dump to make sure everything has properly and “organically” been added back
into the data dictionary and MySQL metadata.
15. #cpgethightech #cpanelconf
Page Corruption Errors
Why do they occur? What are your options?
Power failure / Physical failure
Hardware failure
Driver / Kernel bug
File system write errors
No built-in mechanism to “repair” a
corrupted page
MySQL may be offline
Logged information can be
ambiguous or unspecific
Backups may not be available
16. #cpgethightech #cpanelconf
mysql> SELECT COUNT(1) FROM customer_corrupt;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 1
Current database: testdb
+----------+
| COUNT(1) |
+----------+
| 599 |
+----------+
1 row in set (0.00 sec)
You might see something
like..
17. #cpgethightech #cpanelconf
Alchemy Solutions
Page corruption example
scenario:
140911 13:21:36 InnoDB: Page checksum 4116551042, prior-to-4.0.14-form checksum 351785139
InnoDB: stored checksum 4003879582, prior-to-4.0.14-form stored checksum 2163901611
InnoDB: Page lsn 3886914220 2890537894, low 4 bytes of lsn at page end 12867924
InnoDB: Page number (if stored to page already) 2948225628,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 3390548810
…
140911 13:21:36 InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 3390548810:2948225628, should be 236:8!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 8.40
4116551042
351785139
4003879582
2163901611
2948225628
3390548810
3390548810:2948225628 236:8
Space ID (and potentially page ID) hex edit method can potentially be applied here
as well, however the checksum will still need to be adjusted.
18. #cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
Name Byte Length Offset Description
FIL_PAGE_SPACE_OR_CHKSUM 4 0 Checksum (Space ID in older MySQL versions)
FIL_PAGE_OFFSET 4 4 Page Number
FIL_PAGE_PREV 4 8 Previous Page (in key order)
… … … …
So we know how to correct that space ID, and we can probably translate that over to changing the page
number (referred to as “page offset” in the source) as well. How about changing the checksum?
The thing about doing any kind of manual changes to the physical InnoDB files is that this method will
ALWAYS produce a checksum mismatch, because any byte changes in the file will of course affect the
calculated checksum. So not only can updating the checksum alone be one way to try getting back some
quick access to your table in a serious recovery scenario, but it is in fact a required final step to doing any
byte manipulation with InnoDB.
storage/innobase/include/fil0fil.h
19. #cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
storage/innobase/include/fil0fil.h
FIL Trailer (8)
Name Byte Length Offset Description
FIL_PAGE_END_LSN 8 16376
Low 4 bytes: Checksum, Last 4 bytes:
FIL_PAGE_LSN
In order to do this properly, we’ll need to be aware of one other location in which the checksum for each
page is stored – the trailer. This is an 8-byte set of values that are stored at the end of each 16384-byte
page, meaning that It begins at byte offset 16376 of a page, and ends with the byte at offset 16383.
The checksum stored here is a little different, as it is considered the old-style (pre-4.0) checksum. This is
stored here simply as a secondary means of verification, along with the primary, “new-style” checksum at
the beginning of the page.
21. #cpgethightech #cpanelconf
The Future
• MariaDB defaulting with XtraDB
• Aria being developed to eventually be a combination of MyISAM and
InnoDB’s key features
• InnoDB features being developed to allow more flexibility and fewer
limitations on the way that the file system can be utilized, or that
tablespace files can be imported.
• Data dictionary being revamped.
22. #cpgethightech #cpanelconf
Online Courses
A 3-unit advanced course on InnoDB is currently being developed for our cPanel University learning
environment. The first one, which is based on last year’s InnoDB Anatomy presentation, is available
online now for preview access by using the group code: cPanelConf2015
Welcome / thanks for coming
My name is.. got started with InnoDB/MySQL last year when a surge of InnoDB corruption/error incidents were flooding in.
InnoDB is a huge topic in and of itself, and because of this, there's never enough time to really cover the full scope in one presentation, or even a week of presentations. So last year, I kicked it off with InnoDB Anatomy, which went over a lot of the structural ins and outs. InnoDB Alchemy is kind of like a sequel to that, and I'm hoping to go a little more in depth into some of the fun stuff that I breezed past last year.
1 – cPanel still sets MyISAM as default engine in my.cnf
2 – redo logging
physiological
idempotent
space id, page no, op #, data
ACID
atomicity
consistency
isolation
durability
Emphasis on ibdata1’s tie to the data dictionary
Data Dictionary System Tables:
SYS_TABLES
SYS_COLUMNS
SYS_INDEXES
SYS_FIELDS
Removing ibdata1 file single most common “fix” that we see problems arise from.
* Max value of redo log files as of 5.6.2+ = 512GB – previous max value was 4GB
Large log files can cause crash recovery to be slow, though this is primarily an issue on MySQL 5.5 and prior.
Log file size, as a rule, should be greater than 10 times the largest BLOB data size found in the table rows, plus length of other variable length fields (varchar, varbinary, and text types)
There are two real core components that need to be intact for InnoDB to recognize and properly load a table and it's data - the data dictionary, and the table's files system.
There are two real core components that need to be intact for InnoDB to recognize and properly load a table and it's data - the data dictionary, and the table's files system.
There are two real core components that need to be intact for InnoDB to recognize and properly load a table and it's data - the data dictionary, and the table's files system.