Fast Incremental Backups
with Percona Server and Percona XtraBackup
Laurynas
Biveinis
Agenda
• Incremental XtraBackup: performance
• Incremental XtraBackup with bitmaps:
performance
• The cost of the feature
...
Incremental XtraBackup:
Performance
3
0.10% 1.00% 10.00% 100.00%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Delta Size
Ba...
Incremental XtraBackup: How Data
Page Copying Works
4
LSN = 950
LSN = 960LSN = 960
LSN = 1002
LSN = 1003
LSN = 940
LSN = 1...
Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• http://bit.ly/FBIncBackup
5
Incremental XtraBackup: Can We
Avoid Reading the Old Pages?
• How do we know which pages to read
then?
• Two ways to get t...
Changed Page Tracking
• Server:
– --innodb-track-changed-pages=TRUE
– Documentation at
http://bit.ly/psbmpdoc
– 5.1/5.5/5....
Incremental XtraBackup with
Changed Page Tracking
8
LSN = 950
LSN = 960LSN = 960
LSN = 1002
LSN = 1003
LSN = 940
LSN = 101...
Incremental XtraBackup with
Changed Page Tracking: Performance
9
0.00% 0.01% 1.00% 100.00%
0%
10%
20%
30%
40%
50%
60%
70%
...
Percona Server with Changed Page
Tracking: Server Overhead
• Nothing is ever free!
– But the price might be very well
acce...
Percona Server with Changed Page
Tracking: Server Overhead
1
1
1 2 3 4 5 6 7 8
0
100
200
300
400
500
600
700
800
Log and b...
Percona Server with Changed Page
Tracking: Server Overhead
1
2
• A bad case: 3-15 log bytes per 1 bmp byte
• https://bugs....
Percona Server with Changed Page
Tracking: Server Overhead
• Impact on TPS and response time:
– Couldn't find it
– If you ...
Bitmap File Naming & Sizing
• ib_modified_log_<seq>_<LSN>.xdb
– <Seq>: 1, 2, 3, ...
– <LSN>: the server LSN at the file cr...
Bitmap File Management
• PURGE CHANGED_PAGE_BITMAPS BEFORE <lsn>
– ib_1_8192.xdb
– ib_2_10000.xdb
– ib_3_20000.xdb
– Full ...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Percona Server can read the bitmaps too
1
6
SHOW CREATE TABLE INFORMATION_SCHEMA...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
1
7
SELECT * FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES;
space_id page_id start_...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• Which tables are written to?
1
8
SELECT DISTINCT space_id FROM
INFORMATION_SCHEM...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest tables?
1
9
SELECT space_id,
COUNT(space_id) AS number_of_f...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• What are the hottest pages?
2
0
SELECT space_id, page_id,
COUNT(page_id) AS numb...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
2
1
CREATE TEMPORARY TABLE icp (
space_id I...
INFORMATION_SCHEMA.INNODB_CHANGED_PAGES
• For complex queries, copy data first
2
2
EXPLAIN SELECT DISTINCT space_id FROM
I...
Implementation: File Format
2
3
Data for checkpoint at LSN 9000
LSN 10000
LSN 10500
A sequence of per-checkpoint varying n...
Implementation: Server Side
• A new XtraDB thread
– 1. Wait for log checkpoint completed event
– 2. Read the log up to the...
Implementation: Things We Had to
Account For
• Maximum checkpoint age violation
– Destroys untracked log data
– Make effor...
Conclusions
• Percona Server together with Percona XtraBackup:
• Enable faster incremental backups
• Enable more frequent ...
Upcoming SlideShare
Loading in …5
×

Fast Incremental Backups with Percona Server and Percona XtraBackup / PLMCE 2014

1,206 views

Published on

Percona Live 2014 presentation

https://www.percona.com/live/mysql-conference-2014/sessions/fast-incremental-backups-percona-server-and-percona-xtrabackup

Published in: Software
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,206
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Fast Incremental Backups with Percona Server and Percona XtraBackup / PLMCE 2014

  1. 1. Fast Incremental Backups with Percona Server and Percona XtraBackup Laurynas Biveinis
  2. 2. Agenda • Incremental XtraBackup: performance • Incremental XtraBackup with bitmaps: performance • The cost of the feature • INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • Implementation – Bitmap file format – New server thread 2
  3. 3. Incremental XtraBackup: Performance 3 0.10% 1.00% 10.00% 100.00% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Delta Size BackupTime • Does time to backup depend on the % of changed data?
  4. 4. Incremental XtraBackup: How Data Page Copying Works 4 LSN = 950 LSN = 960LSN = 960 LSN = 1002 LSN = 1003 LSN = 940 LSN = 1010 table.ibd LSN > 1000 ? Base Backup LSN = 1000 read read read read read read write write write Table.ibd.delta Can we avoid reading the old pages? MySQL
  5. 5. Incremental XtraBackup: Can We Avoid Reading the Old Pages? • http://bit.ly/FBIncBackup 5
  6. 6. Incremental XtraBackup: Can We Avoid Reading the Old Pages? • How do we know which pages to read then? • Two ways to get the modification LSN of a page: – It is written on the page, - or - – We can figure it out from the redo log • The log is cyclical, we must, in the server, save the info before it is overwritten 6
  7. 7. Changed Page Tracking • Server: – --innodb-track-changed-pages=TRUE – Documentation at http://bit.ly/psbmpdoc – 5.1/5.5/5.6 • XtraBackup: – Zero configuration! 7
  8. 8. Incremental XtraBackup with Changed Page Tracking 8 LSN = 950 LSN = 960LSN = 960 LSN = 1002 LSN = 1003 LSN = 940 LSN = 1010 table.ibd LSN > 1000 ? Base Backup LSN = 1000 read read read write write write Table.ibd.delta Percona Server … Changed pages between LSNs 980 and 1020: 1002, 1003, 1010 ...
  9. 9. Incremental XtraBackup with Changed Page Tracking: Performance 9 0.00% 0.01% 1.00% 100.00% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Full Scan Bitmap Delta Size BackupTime
  10. 10. Percona Server with Changed Page Tracking: Server Overhead • Nothing is ever free! – But the price might be very well acceptable • Potential overhead #1: extra disk space requirements • Potential overhead #2: extra code running in the server 1 0
  11. 11. Percona Server with Changed Page Tracking: Server Overhead 1 1 1 2 3 4 5 6 7 8 0 100 200 300 400 500 600 700 800 Log and bitmap file size comparison Bitmap file # Logbytes/bitmapbyte • A good case: > 100 log bytes for 1 bmp byte
  12. 12. Percona Server with Changed Page Tracking: Server Overhead 1 2 • A bad case: 3-15 log bytes per 1 bmp byte • https://bugs.launchpad.net/bugs/1269547 – We are considering fix options
  13. 13. Percona Server with Changed Page Tracking: Server Overhead • Impact on TPS and response time: – Couldn't find it – If you ever do find it, report it to us and try --innodb_log_checksum_algorithm=crc32 ● http://bit.ly/pslogcrc32 1 3
  14. 14. Bitmap File Naming & Sizing • ib_modified_log_<seq>_<LSN>.xdb – <Seq>: 1, 2, 3, ... – <LSN>: the server LSN at the file create time • Rotated on –Server start –innodb_max_bitmap_file_size 1 4
  15. 15. Bitmap File Management • PURGE CHANGED_PAGE_BITMAPS BEFORE <lsn> – ib_1_8192.xdb – ib_2_10000.xdb – ib_3_20000.xdb – Full backup taken, LSN = 22000 – PURGE C_P_B BEFORE 22000; – ib_4_30000.xdb – Incremental backup taken, LSN = 33000 – PURGE C_P_B BEFORE 33000; 1 5
  16. 16. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • Percona Server can read the bitmaps too 1 6 SHOW CREATE TABLE INFORMATION_SCHEMA.INNODB_CHANGED_PAGES; CREATE TABLE `INNODB_CHANGED_PAGES` ( `space_id` int(11) unsigned NOT NULL DEFAULT '0', `page_id` int(11) unsigned NOT NULL DEFAULT '0', `start_lsn` bigint(21) unsigned NOT NULL DEFAULT '0', `end_lsn` bigint(21) unsigned NOT NULL DEFAULT '0' ) • start_lsn and end_lsn are always at the checkpoint boundary • Does not show the exact LSN of a change • Does not show the number of changes for one page • Does show the number of flushes for a page over the workload
  17. 17. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES 1 7 SELECT * FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES; space_id page_id start_lsn end_lsn 0 0 8204 38470 0 1 8204 38470 5 0 8204 38470 5 3 8204 38470 0 1 38471 50000 5 3 38471 50000 5 3 50001 60000 • Don't query like that in production! – It will read all the bitmaps you have. Gigabytes, terabytes, ... – Add WHERE start_lsn > X AND end_lsn < Y (index condition pushdown implemented for this case)
  18. 18. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • Which tables are written to? 1 8 SELECT DISTINCT space_id FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE ...; space_id 0 10 SELECT DISTINCT t1.space_id AS space_id, t2.schema AS db, t2.name AS tname FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES AS t1, INFORMATION_SCHEMA.INNODB_SYS_TABLES AS t2 WHERE t1.space_id = t2.space AND t1.start_lsn >... space_id db tname 0 SYS_FOREIGN 0 SYS_FOREIGN_COLS 10 test foo
  19. 19. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • What are the hottest tables? 1 9 SELECT space_id, COUNT(space_id) AS number_of_flushes FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES GROUP BY space_id ORDER BY number_of_flushes DESC; space_id number_of_flushes 0 65 10 5 11 4
  20. 20. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • What are the hottest pages? 2 0 SELECT space_id, page_id, COUNT(page_id) AS number_of_flushes FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES GROUP BY space_id, page_id HAVING number_of_flushes > 2 ORDER BY number_of_flushes DESC LIMIT 8; space_id page_id number_of_flushes 0 5 3 0 7 3 0 0 2 0 11 2 10 3 2 0 1 2 0 12 2 0 2 2
  21. 21. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • For complex queries, copy data first 2 1 CREATE TEMPORARY TABLE icp ( space_id INT(11) NOT NULL, page_id INT(11) NOT NULL, start_lsn BIGINT(21) NOT NULL, end_lsn BIGINT(21) NOT NULL, INDEX page_id(space_id, page_id), INDEX start_lsn(start_lsn), INDEX end_lsn(end_lsn)) ENGINE=InnoDB; INSERT INTO icp SELECT * FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES WHERE start_lsn > 8000;
  22. 22. INFORMATION_SCHEMA.INNODB_CHANGED_PAGES • For complex queries, copy data first 2 2 EXPLAIN SELECT DISTINCT space_id FROM INFORMATION_SCHEMA.INNODB_CHANGED_PAGES; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE INNODB_CHANGED_PAGES ALL NULL NULL NULL NULL NULL Using temporary EXPLAIN SELECT DISTINCT space_id FROM icp; id select_type table type possible_keys key key_len ref rows Extra 1 SIMPLE icp index NULL page_id 8 NULL 74 Using index
  23. 23. Implementation: File Format 2 3 Data for checkpoint at LSN 9000 LSN 10000 LSN 10500 A sequence of per-checkpoint varying number of data pages: For each checkpoint: space, start page space, start page space, start page 4KB Each page contains a bitmap for the next 32480 pages in space starting from start page
  24. 24. Implementation: Server Side • A new XtraDB thread – 1. Wait for log checkpoint completed event – 2. Read the log up to the checkpoint, write the bitmap – 3. goto 1 • Little data sharing with the rest of XtraDB – log_sys->mutex for: ● setting and getting LSNs; ● calculating log read offset from LSN. • Little extra code for the query threads – Unread log overwrite check – Firing of the log checkpoint completed event 2 4
  25. 25. Implementation: Things We Had to Account For • Maximum checkpoint age violation – Destroys untracked log data – Make effort to avoid, but in the end we allow to overwrite it – Responding server > fast backups • Crash recovery – Re-read the log if available 2 5
  26. 26. Conclusions • Percona Server together with Percona XtraBackup: • Enable faster incremental backups • Enable more frequent incremental backups • Does not hurt server operation, but have to manage the bitmaps now • New INFORMATION_SCHEMA table for gaining insight into data change patterns • Is actually being used, http://bit.ly/psbmpbugs • Thank you! Questions? 2 6

×