1. Hacking Session: undelete (and more) rows
from the binary log
Percona Live Europe Amsterdam 2015 Frédéric -lefred- Descamps
2. Who am I ?
●
Frédéric Descamps
●
@lefred - follow me on twitter if you want ;-)
●
http://about.me/lefred
●
Working for Percona since 2011
●
Managing MySQL since 3.23
●
devops believer
3. Why ?
Because of Scott Noyes' blog post :
http://thenoyes.com/littlenoise/?p=307
●
Because it's faster than point-in-time
recovery
●
Because I didn't remember the awk script
in Scott's blog post
4. Requirements
●
Have the binary logs in ROW format
●
Have binlog_row_image set to FULL
●
Have access to the binary logs
●
Have access to MySQL
●
Have access to the MySQL source code [not mandatory]
●
Have a brain and being able to use it ;-)
Percona Live Europe Amsterdam 2015
5. Point-in-Time Recovery
Usually when we need to «undo» even one single statement, we
need to perform a point-in-time recovery
●
restore the last backup (full or several incrementals)
●
then replay some binary logs (and it can be a bunch of them!)
●
this is a very slow operation and while the binary logs are
replayed, usually everything needs to be stopped.
Percona Live Europe Amsterdam 2015
11. binary log's event content
# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v
5.6.22-log created 141221 21:20:55 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
BINLOG '
pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMT
if8=
'/*!*/;
# at 824
#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags:
STMT_END_F
BINLOG '
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
'/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
Percona Live Europe Amsterdam 2015
12. binary log's event content
# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#141221 21:20:55 server id 1 end_log_pos 120 CRC32 0xff8913a3 Start: binlog v 4, server v
5.6.22-log created 141221 21:20:55 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
BINLOG '
pyuXVA8BAAAAdAAAAHgAAAABAAQANS42LjIyLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAACnK5dUEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAaMT
if8=
'/*!*/;
# at 824
#150103 16:55:49 server id 1 end_log_pos 876 CRC32 0xaac329ee Delete_rows: table id 76 flags:
STMT_END_F
BINLOG '
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
'/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
Percona Live Europe Amsterdam 2015
13. binary log's event content decoded
...
BINLOG '
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
'/*!*/;
...
●
Let's check from the source the type code:
https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,
Percona Live Europe Amsterdam 2015
14. binary log's event content decoded
...
BINLOG '
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
'/*!*/;
...
●
Let's check from the source the type code:
https://github.com/mysql/mysql-server/blob/5.6/sql/log_event.h
WRITE_ROWS_EVENT = 30,
UPDATE_ROWS_EVENT = 31,
DELETE_ROWS_EVENT = 32,
MySQL 5.5 and MariaDB use V1:
WRITE_ROWS_EVENT_V1 = 23,
UPDATE_ROWS_EVENT_V1 = 24,
DELETE_ROWS_EVENT_V1 = 25,
Percona Live Europe Amsterdam 2015
15. binary log's event content decoded
# python -c "print hex(32)"
0x20
●
Again from the source we can find the event type's offset
#define EVENT_TYPE_OFFSET 4
# python -c "import base64; print ord(base64.b64decode(b'BRGoVCABA
AAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAA
AA7inDqg==')[4])"
32
●
This is exactly the event we were looking for !!
Percona Live Europe Amsterdam 2015
16. rebuild a new binlog event
●
Now we can rebuild a new event by replacing x32 by x30
# python
>>> import base64
>>>
data=base64.b64decode(b'BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYA
AAA7inDqg==')
>>> data
'x05x11xa8T x01x00x00x004x00x00x00lx03x00x00x00x00L
x00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03
Lizx02x00x00x00x18x00x00x00xee)xc3xaa'
>>> hex(30)
'0x1e'
>>> data2=base64.b64encode("x05x11xa8Tx1ex01x00x00x004x00x00x00lx03x00
x00x00x00Lx00x00x00x00x00x01x00x02x00x04xffxf0x02x00x00x00x03
Lizx02x00x00x00x18x00x00x00xee)xc3xaa")
>>> data2
'BRGoVDABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg=='
●
Let's compare the two lines:
BRGoVCABAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
BRGoVB4BAAAANAAAAGwDAAAAAEwAAAAAAAEAAgAE//ACAAAAA0xpegIAAAAYAAAA7inDqg==
Percona Live Europe Amsterdam 2015
17. rebuild a new binlog event & replay it
●
ready to replay ?
●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 |
> sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;
+----+--------+--------+-------+
| id | name | pizzas | beers |
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
3 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
18. rebuild a new binlog event & replay it
●
ready to replay ?
●# mysqlbinlog mysql-bin.000018 -j 824 --stop-position=876 |
sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;
+----+--------+--------+-------+
| id | name | pizzas | beers |
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
3 rows in set (0.00 sec)
uuh ? did it fail ?
Percona Live Europe Amsterdam 2015
20. replay it, 2nd try
●
ready to replay, again ?
●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 |
> sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;
+----+--------+--------+-------+
| id | name | pizzas | beers |
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 2 | Liz | 2 | 24 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
4 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
21. replay it, 2nd try
●
ready to replay, again ?
●# mysqlbinlog mysql-bin.000018 -j 684 --stop-position=907 |
> sed s/BRGoVCAB/BRGoVB4B/ | mysql
mysql> select * from fosdem.community_dinner;
+----+--------+--------+-------+
| id | name | pizzas | beers |
+----+--------+--------+-------+
| 1 | dim0 | 99 | 99 |
| 2 | Liz | 2 | 24 |
| 3 | Kenny | 12 | 32 |
| 4 | lefred | 10 | 10 |
+----+--------+--------+-------+
4 rows in set (0.00 sec)
Percona Live Europe Amsterdam 2015
22. More ?
●
With the same kind of technique, we can also:
– delete rows that were inserted (sql injection?)
– un-update modified rows (more complicated)
●
I've created a script that automates all that: MyUndelete
Percona Live Europe Amsterdam 2015
23. MyUndelete : «un-insert»
●
We can delete an INSERT
Percona Live Europe Amsterdam 2015
# ./MyUndelete.py -s 41989 -e 42207 -i -b mysql-bin.000004
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000004
Start Position file is 41989
End Postision file is 42207
We also look to undo INSERTs
Event type ('x1e') is an insert v2
Ready to revert the statement ? [y/n]
y
Done... I hope it worked ;)
24. MyUndelete : «un-update»
●
This is a bit more difficult as we need to find the lenght of the
record to be able to split both records and rebuild a working
binary event
Percona Live Europe Amsterdam 2015
mysql> select * from community_dinner;
+----+-------+--------+-------+
| id | name | pizzas | beers |
+----+-------+--------+-------+
| 1 | fred | 3 | 3 |
| 2 | lizz | 2 | 10 |
| 3 | dimi | 99 | 99 |
| 4 | kenny | 20 | 20 |
+----+-------+--------+-------+
25. MyUndelete : «un-update» (2)
Percona Live Europe Amsterdam 2015
mysql> update community_dinner set beers=0 where id=3;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from community_dinner;
+----+-------+--------+-------+
| id | name | pizzas | beers |
+----+-------+--------+-------+
| 1 | fred | 3 | 3 |
| 2 | lizz | 2 | 10 |
| 3 | dimi | 99 | 0 |
| 4 | kenny | 20 | 20 |
+----+-------+--------+-------+
4 rows in set (0.00 sec)
26. MyUndelete : «un-update» (3)
Percona Live Europe Amsterdam 2015
mysql> show binlog events in 'mysqld-bin.000018';
...
| mysqld-bin.000018 | 1145 | Query | 1 | 1219 | BEGIN |
| mysqld-bin.000018 | 1219 | Table_map | 1 | 1285 | table_id: 70
(fosdem.community_dinner) |
| mysqld-bin.000018 | 1285 | Update_rows | 1 | 1357 | table_id: 70 flags: STMT_END_F
| mysqld-bin.000018 | 1357 | Xid | 1 | 1388 | COMMIT /* xid=44 */
...
# ./MyUndelete.py -s 1145 -e 1388 -u -b /var/lib/mysql/mysqld-bin.000018
*** WARNING *** USE WITH CARE ****
Binlog file is /var/lib/mysql/mysqld-bin.000018
Start Position file is 1145
End Postision file is 1357
Event type ('x1f') is an update v2
We got an update!!
Ready to revert the statement ? [y/n]
y
Sending to mysql...
Done... I hope it worked ;)
28. Resources
●
Oracle MySQL Source code — https://github.com/mysql/mysql-server
●
MariaDB Source Code — https://github.com/MariaDB/server
●
Scott Noyes Blog Post - http://thenoyes.com/littlenoise/?p=307
●
MyUndelete on github — https://github.com/lefred/MyUndelete
Percona Live Europe Amsterdam 2015
29. Related topic
●
MySQL released «Binary Log API» that could be use to decode
in a better way the binary logs
– http://mysqlhighavailability.com/mysql-binlog-events-reading-and-handling-
information-from-your-binary-log/
– http://mysqlhighavailability.com/mysql-binlog-events-use-case-and-examples/
●
Jeremy Cole wrote a Ruby Library to also parse binary logs
– https://github.com/jeremycole/mysql_binlog
Percona Live Europe Amsterdam 2015