Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Undelete (and more) rows from the binary log

744 views

Published on

This is a short hacking session on how to revert some operations in MySQL using the binary log.

Published in: Engineering
  • Be the first to comment

Undelete (and more) rows from the binary log

  1. 1. Hacking Session: undelete (and more) rows from the binary log Percona Live Europe Amsterdam 2015 Frédéric -lefred- Descamps
  2. 2. Who am I ? ● Frédéric Descamps ● @lefred - follow me on twitter if you want ;-) ● http://about.me/lefred ● Working for Percona since 2011 ● Managing MySQL since 3.23 ● devops believer
  3. 3. Why ? Because of Scott Noyes' blog post : http://thenoyes.com/littlenoise/?p=307 ● Because it's faster than point-in-time recovery ● Because I didn't remember the awk script in Scott's blog post
  4. 4. Requirements ● Have the binary logs in ROW format ● Have binlog_row_image set to FULL ● Have access to the binary logs ● Have access to MySQL ● Have access to the MySQL source code [not mandatory] ● Have a brain and being able to use it ;-) Percona Live Europe Amsterdam 2015
  5. 5. Point-in-Time Recovery Usually when we need to «undo» even one single statement, we need to perform a point-in-time recovery ● restore the last backup (full or several incrementals) ● then replay some binary logs (and it can be a bunch of them!) ● this is a very slow operation and while the binary logs are replayed, usually everything needs to be stopped. Percona Live Europe Amsterdam 2015
  6. 6. Let's hack together a quicker process Ready ?
  7. 7. Let's delete a row mysql> select @@version, @@version_comment; +------------+------------------------------+ | @@version | @@version_comment | +------------+------------------------------+ | 5.6.22-log | MySQL Community Server (GPL) | +------------+------------------------------+ mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 2 | Liz | 2 | 24 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 4 rows in set (0.00 sec) Percona Live Europe Amsterdam 2015
  8. 8. Let's delete a row (2) mysql> delete from community_dinner where id=2; Query OK, 1 row affected (0.05 sec) mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 3 rows in set (0.00 sec) Percona Live Europe Amsterdam 2015
  9. 9. Let's find the event in the binary log mysql> show master statusG *************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) mysql> show binlog events in 'mysql-bin.000018'; +------------------+-----+-------------+-----------+-------------+---------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+-----+-------------+-----------+------------+----------------------------+ | mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... | | mysql-bin.000018 | 120 | Query | 1 | 220 | create ... | | mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; | | mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN | | mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ | | mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ | +------------------+-----+-------------+-----------+------------+====------------------------+ Percona Live Europe Amsterdam 2015
  10. 10. Let's find the event in the binary log mysql> show master statusG *************************** 1. row *************************** File: mysql-bin.000018 Position: 907 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) mysql> show binlog events in 'mysql-bin.000018'; +------------------+-----+-------------+-----------+-------------+---------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+-----+-------------+-----------+------------+----------------------------+ | mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... | | mysql-bin.000018 | 120 | Query | 1 | 220 | create ... | | mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; | | mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN | | mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ | | mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ | +------------------+-----+-------------+-----------+------------+====------------------------+ The event we are looking for Percona Live Europe Amsterdam 2015
  11. 11. binary log's event content # mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v 5.6.22-log created 141221 21:20:55 at startup # Warning: this binlog is either in use or was not closed properly. ROLLBACK/*!*/; BINLOG ' pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMT if8= '/*!*/; # at 824 #150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags: STMT_END_F BINLOG ' BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== '/*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; Percona Live Europe Amsterdam 2015
  12. 12. binary log's event content # mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v 5.6.22-log created 141221 21:20:55 at startup # Warning: this binlog is either in use or was not closed properly. ROLLBACK/*!*/; BINLOG ' pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMT if8= '/*!*/; # at 824 #150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags: STMT_END_F BINLOG ' BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== '/*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; Percona Live Europe Amsterdam 2015
  13. 13. binary log's event content decoded ... BINLOG ' BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== '/*!*/; ... ● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h WRITE_ROWS_EVENT = 30, UPDATE_ROWS_EVENT = 31, DELETE_ROWS_EVENT = 32, Percona Live Europe Amsterdam 2015
  14. 14. binary log's event content decoded ... BINLOG ' BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== '/*!*/; ... ● Let's check from the source the type code: https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h WRITE_ROWS_EVENT = 30, UPDATE_ROWS_EVENT = 31, DELETE_ROWS_EVENT = 32, MySQL 5.5 and MariaDB use V1: WRITE_ROWS_EVENT_V1 = 23, UPDATE_ROWS_EVENT_V1 = 24, DELETE_ROWS_EVENT_V1 = 25, Percona Live Europe Amsterdam 2015
  15. 15. binary log's event content decoded # python -c "print hex(32)" 0x20 ● Again from the source we can find the event type's offset #define EVENT_TYPE_OFFSET 4 # python -c "import base64; print ord(base64.b64decode(b'BRGoVCABA AAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAA AA7inDqg==')[4])" 32 ● This is exactly the event we were looking for !! Percona Live Europe Amsterdam 2015
  16. 16. rebuild a new binlog event ● Now we can rebuild a new event by replacing x32 by x30 # python >>> import base64 >>> data=base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYA AAA7inDqg==') >>> data 'x05x11xa8T x01x00x00x004x00x00x00lx03x00x00x00x00L x00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03 Lizx02x00x00x00x18x00x00x00xee)xc3xaa' >>> hex(30) '0x1e' >>> data2=base64.b64encode("x05x11xa8Tx1ex01x00x00x004x00x00x00lx03x00 x00x00x00Lx00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03 Lizx02x00x00x00x18x00x00x00xee)xc3xaa") >>> data2 'BRGoVDABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==' ● Let's compare the two lines: BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== BRGoVB4BAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg== Percona Live Europe Amsterdam 2015
  17. 17. rebuild a new binlog event & replay it ● ready to replay ? ●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | > sed s/BRGoVCAB/BRGoVB4B/ | mysql mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 3 rows in set (0.00 sec) Percona Live Europe Amsterdam 2015
  18. 18. rebuild a new binlog event & replay it ● ready to replay ? ●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 | sed s/BRGoVCAB/BRGoVB4B/ | mysql mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 3 rows in set (0.00 sec) uuh ? did it fail ? Percona Live Europe Amsterdam 2015
  19. 19. find what to replay mysql> show binlog events in 'mysql-bin.000018'; +------------------+-----+-------------+-----------+-------------+---------------------------+ | Log_name | Pos | Event_type | Server_id | End_log_pos | Info | +------------------+-----+-------------+-----------+------------+----------------------------+ | mysql-bin.000018 | 4 | Format_desc | 1 | 120 | Server ... | | mysql-bin.000018 | 120 | Query | 1 | 220 | create ... | | mysql-bin.000018 | 220 | Query | 1 | 404 | use `fosdem`; | | mysql-bin.000018 | 404 | Query | 1 | 478 | BEGIN | | mysql-bin.000018 | 478 | Table_map | 1 | 544 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 544 | Write_rows | 1 | 653 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 653 | Xid | 1 | 684 | COMMIT /* xid=32 */ | | mysql-bin.000018 | 684 | Query | 1 | 758 | BEGIN | | mysql-bin.000018 | 758 | Table_map | 1 | 824 | table_id: 76 (fosdem.commu| | mysql-bin.000018 | 824 | Delete_rows | 1 | 876 | table_id: 76 flags: STMT_E| | mysql-bin.000018 | 876 | Xid | 1 | 907 | COMMIT /* xid=34 */ | +------------------+-----+-------------+-----------+------------+====------------------------+ ● We need to replay more, from the BEGIN to the COMMIT Percona Live Europe Amsterdam 2015
  20. 20. replay it, 2nd try ● ready to replay, again ? ●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 | > sed s/BRGoVCAB/BRGoVB4B/ | mysql mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 2 | Liz | 2 | 24 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 4 rows in set (0.00 sec) Percona Live Europe Amsterdam 2015
  21. 21. replay it, 2nd try ● ready to replay, again ? ●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 | > sed s/BRGoVCAB/BRGoVB4B/ | mysql mysql> select * from fosdem.community_dinner; +----+--------+--------+-------+ | id | name | pizzas | beers | +----+--------+--------+-------+ | 1 | dim0 | 99 | 99 | | 2 | Liz | 2 | 24 | | 3 | Kenny | 12 | 32 | | 4 | lefred | 10 | 10 | +----+--------+--------+-------+ 4 rows in set (0.00 sec) Percona Live Europe Amsterdam 2015
  22. 22. More ? ● With the same kind of technique, we can also: – delete rows that were inserted (sql injection?) – un-update modified rows (more complicated) ● I've created a script that automates all that: MyUndelete Percona Live Europe Amsterdam 2015
  23. 23. MyUndelete : «un-insert» ● We can delete an INSERT Percona Live Europe Amsterdam 2015 # ./MyUndelete.py -s 41989 -e 42207 -i -b mysql-bin.000004 *** WARNING *** USE WITH CARE **** Binlog file is /var/lib/mysql/mysqld-bin.000004 Start Position file is 41989 End Postision file is 42207 We also look to undo INSERTs Event type ('x1e') is an insert v2 Ready to revert the statement ? [y/n] y Done... I hope it worked ;)
  24. 24. MyUndelete : «un-update» ● This is a bit more difficult as we need to find the lenght of the record to be able to split both records and rebuild a working binary event Percona Live Europe Amsterdam 2015 mysql> select * from community_dinner; +----+-------+--------+-------+ | id | name | pizzas | beers | +----+-------+--------+-------+ | 1 | fred | 3 | 3 | | 2 | lizz | 2 | 10 | | 3 | dimi | 99 | 99 | | 4 | kenny | 20 | 20 | +----+-------+--------+-------+
  25. 25. MyUndelete : «un-update» (2) Percona Live Europe Amsterdam 2015 mysql> update community_dinner set beers=0 where id=3; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> select * from community_dinner; +----+-------+--------+-------+ | id | name | pizzas | beers | +----+-------+--------+-------+ | 1 | fred | 3 | 3 | | 2 | lizz | 2 | 10 | | 3 | dimi | 99 | 0 | | 4 | kenny | 20 | 20 | +----+-------+--------+-------+ 4 rows in set (0.00 sec)
  26. 26. MyUndelete : «un-update» (3) Percona Live Europe Amsterdam 2015 mysql> show binlog events in 'mysqld-bin.000018'; ... | mysqld-bin.000018 | 1145 | Query | 1 | 1219 | BEGIN | | mysqld-bin.000018 | 1219 | Table_map | 1 | 1285 | table_id: 70 (fosdem.community_dinner) | | mysqld-bin.000018 | 1285 | Update_rows | 1 | 1357 | table_id: 70 flags: STMT_END_F | mysqld-bin.000018 | 1357 | Xid | 1 | 1388 | COMMIT /* xid=44 */ ... # ./MyUndelete.py -s 1145 -e 1388 -u -b /var/lib/mysql/mysqld-bin.000018 *** WARNING *** USE WITH CARE **** Binlog file is /var/lib/mysql/mysqld-bin.000018 Start Position file is 1145 End Postision file is 1357 Event type ('x1f') is an update v2 We got an update!! Ready to revert the statement ? [y/n] y Sending to mysql... Done... I hope it worked ;)
  27. 27. MyUndelete : «un-update» (4) Percona Live Europe Amsterdam 2015 mysql> select * from community_dinner; +----+-------+--------+-------+ | id | name | pizzas | beers | +----+-------+--------+-------+ | 1 | fred | 3 | 3 | | 2 | lizz | 2 | 10 | | 3 | dimi | 99 | 99 | | 4 | kenny | 20 | 20 | +----+-------+--------+-------+
  28. 28. Resources ● Oracle MySQL Source code — https://github.com/mysql/mysql-server ● MariaDB Source Code — https://github.com/MariaDB/server ● Scott Noyes Blog Post - http://thenoyes.com/littlenoise/?p=307 ● MyUndelete on github — https://github.com/lefred/MyUndelete Percona Live Europe Amsterdam 2015
  29. 29. Related topic ● MySQL released «Binary Log API» that could be use to decode in a better way the binary logs – http://mysqlhighavailability.com/mysql-binlog-events-reading-and-handling- information-from-your-binary-log/ – http://mysqlhighavailability.com/mysql-binlog-events-use-case-and-examples/ ● Jeremy Cole wrote a Ruby Library to also parse binary logs – https://github.com/jeremycole/mysql_binlog Percona Live Europe Amsterdam 2015
  30. 30. Thank you Questions ? Percona Live Europe Amsterdam 2015

×