Percona Live 2014 presentation https://www.percona.com/live/mysql-conference-2014/sessions/fast-incremental-backups-percona-server-and-percona-xtrabackup

1. Fast Incremental Backups
with Percona Server and Percona XtraBackup
Laurynas
Biveinis

2. Agenda
• Incremental XtraBackup: performance
• Incremental XtraBackup with bitmaps:
performance
• The cost of the feature
• INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Implementation
– Bitmap file format
– New server thread
2

3. Incremental XtraBackup:
Performance
3
0.10% 1.00% 10.00% 100.00%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Delta Size
BackupTime
• Does time to backup depend on the % of changed data?

4. Incremental XtraBackup: How Data
Page Copying Works
4
LSN = 950
LSN = 960LSN = 960
LSN = 1002
LSN = 1003
LSN = 940
LSN = 1010
table.ibd
LSN
>
1000
?
Base Backup
LSN = 1000
read
read
read
read
read
read
write
write
write
Table.ibd.delta
Can we avoid reading
the old pages?
MySQL

5. Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• http://bit.ly/FBIncBackup
5

6. Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• How do we know which pages to read
then?
• Two ways to get the modification LSN of a
page:
– It is written on the page, - or -
– We can figure it out from the redo log
• The log is cyclical, we must, in the server,
save the info before it is overwritten
6

7. Changed Page Tracking
• Server:
– --innodb-track-changed-pages=TRUE
– Documentation at
http://bit.ly/psbmpdoc
– 5.1/5.5/5.6
• XtraBackup:
– Zero configuration!
7

8. Incremental XtraBackup with
Changed Page Tracking
8
LSN = 950
LSN = 960LSN = 960
LSN = 1002
LSN = 1003
LSN = 940
LSN = 1010
table.ibd
LSN
>
1000
?
Base Backup
LSN = 1000
read
read
read
write
write
write
Table.ibd.delta
Percona
Server
…
Changed pages between
LSNs 980 and 1020:
1002, 1003, 1010
...

9. Incremental XtraBackup with
Changed Page Tracking: Performance
9
0.00% 0.01% 1.00% 100.00%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Full Scan
Bitmap
Delta Size
BackupTime

10. Percona Server with Changed Page
Tracking: Server Overhead
• Nothing is ever free!
– But the price might be very well
acceptable
• Potential overhead #1: extra disk
space requirements
• Potential overhead #2: extra code
running in the server
1
0

11. Percona Server with Changed Page
Tracking: Server Overhead
1
1
1 2 3 4 5 6 7 8
0
100
200
300
400
500
600
700
800
Log and bitmap file size comparison
Bitmap file #
Logbytes/bitmapbyte
• A good case: > 100 log bytes for 1 bmp byte

12. Percona Server with Changed Page
Tracking: Server Overhead
1
2
• A bad case: 3-15 log bytes per 1 bmp byte
• https://bugs.launchpad.net/bugs/1269547
– We are considering fix options

13. Percona Server with Changed Page
Tracking: Server Overhead
• Impact on TPS and response time:
– Couldn't find it
– If you ever do find it, report it to us and
try
--innodb_log_checksum_algorithm=crc32
●
http://bit.ly/pslogcrc32
1
3

14. Bitmap File Naming & Sizing
• ib_modified_log_<seq>_<LSN>.xdb
– <Seq>: 1, 2, 3, ...
– <LSN>: the server LSN at the file create
time
• Rotated on
–Server start
–innodb_max_bitmap_file_size
1
4

15. Bitmap File Management
• PURGE CHANGED_PAGE_BITMAPS BEFORE <lsn>
– ib_1_8192.xdb
– ib_2_10000.xdb
– ib_3_20000.xdb
– Full backup taken, LSN = 22000
– PURGE C_P_B BEFORE 22000;
– ib_4_30000.xdb
– Incremental backup taken, LSN = 33000
– PURGE C_P_B BEFORE 33000;
1
5

16. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Percona Server can read the bitmaps too
1
6
SHOW CREATE TABLE INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
CREATE TABLE `INNODB_CHANGED_PAGES` (
`space_id` int(11) unsigned NOT NULL DEFAULT '0',
`page_id` int(11) unsigned NOT NULL DEFAULT '0',
`start_lsn` bigint(21) unsigned NOT NULL DEFAULT '0',
`end_lsn` bigint(21) unsigned NOT NULL DEFAULT '0'
)
• start_lsn and end_lsn are always at the checkpoint boundary
• Does not show the exact LSN of a change
• Does not show the number of changes for one page
• Does show the number of flushes for a page over the workload

17. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
1
7
SELECT * FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
space_id page_id start_lsn end_lsn
0 0 8204 38470
0 1 8204 38470
5 0 8204 38470
5 3 8204 38470
0 1 38471 50000
5 3 38471 50000
5 3 50001 60000
• Don't query like that in production!
– It will read all the bitmaps you have. Gigabytes, terabytes, ...
– Add WHERE start_lsn > X AND end_lsn < Y (index condition pushdown
implemented for this case)

18. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Which tables are written to?
1
8
SELECT DISTINCT space_id FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE ...;
space_id
0
10
SELECT DISTINCT t1.space_id AS space_id, t2.schema AS db,
t2.name AS tname
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES AS t1,
INFORMATION_SCHEMA.INNODB_SYS_TABLES AS t2
WHERE t1.space_id = t2.space AND t1.start_lsn >...
space_id db tname
0 SYS_FOREIGN
0 SYS_FOREIGN_COLS
10 test foo

19. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest tables?
1
9
SELECT space_id,
COUNT(space_id) AS number_of_flushes
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
GROUP BY space_id
ORDER BY number_of_flushes DESC;
space_id number_of_flushes
0 65
10 5
11 4

20. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest pages?
2
0
SELECT space_id, page_id,
COUNT(page_id) AS number_of_flushes
FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
GROUP BY space_id, page_id
HAVING number_of_flushes > 2
ORDER BY number_of_flushes DESC
LIMIT 8;
space_id page_id number_of_flushes
0 5 3
0 7 3
0 0 2
0 11 2
10 3 2
0 1 2
0 12 2
0 2 2

21. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
2
1
CREATE TEMPORARY TABLE icp (
space_id INT(11) NOT NULL,
page_id INT(11) NOT NULL,
start_lsn BIGINT(21) NOT NULL,
end_lsn BIGINT(21) NOT NULL,
INDEX page_id(space_id, page_id),
INDEX start_lsn(start_lsn),
INDEX end_lsn(end_lsn)) ENGINE=InnoDB;
INSERT INTO icp SELECT * FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE
start_lsn > 8000;

22. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
2
2
EXPLAIN SELECT DISTINCT space_id FROM
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
id select_type table type possible_keys key key_len
ref rows Extra
1 SIMPLE INNODB_CHANGED_PAGES ALL NULL NULL NULL
NULL NULL Using temporary
EXPLAIN SELECT DISTINCT space_id FROM icp;
id select_type table type possible_keys key key_len
ref rows Extra
1 SIMPLE icp index NULL page_id 8 NULL 74 Using
index

23. Implementation: File Format
2
3
Data for checkpoint at LSN 9000
LSN 10000
LSN 10500
A sequence of per-checkpoint varying number of data pages:
For each checkpoint:
space, start page space, start page space, start page
4KB
Each page contains a bitmap for the next 32480 pages in space starting
from start page

24. Implementation: Server Side
• A new XtraDB thread
– 1. Wait for log checkpoint completed event
– 2. Read the log up to the checkpoint, write the bitmap
– 3. goto 1
• Little data sharing with the rest of XtraDB
– log_sys->mutex for:
●
setting and getting LSNs;
●
calculating log read offset from LSN.
• Little extra code for the query threads
– Unread log overwrite check
– Firing of the log checkpoint completed event
2
4

25. Implementation: Things We Had to
Account For
• Maximum checkpoint age violation
– Destroys untracked log data
– Make effort to avoid, but in the end we
allow to overwrite it
– Responding server > fast backups
• Crash recovery
– Re-read the log if available
2
5

26. Conclusions
• Percona Server together with Percona XtraBackup:
• Enable faster incremental backups
• Enable more frequent incremental backups
• Does not hurt server operation, but have to
manage the bitmaps now
• New INFORMATION_SCHEMA table for gaining
insight into data change patterns
• Is actually being used, http://bit.ly/psbmpbugs
• Thank you! Questions?
2
6