Making Backups in Extreme Situations
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Making Backups in Extreme Situations

on

  • 421 views

Slides which I presented at MySQL Connect 2013 together with Matt Lord

Slides which I presented at MySQL Connect 2013 together with Matt Lord

Statistics

Views

Total Views
421
Views on SlideShare
420
Embed Views
1

Actions

Likes
1
Downloads
8
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Making Backups in Extreme Situations Presentation Transcript

  • 1. 1Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 2. Safe Harbour Statement THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES O FUNCTIONALITY DESCRIBED FOR ORACLE’S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE. 2Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 3. Making Backups in Extreme Situations Sveta Smirnova Principal Technical Support Engineer Matt Lord MySQL Product Manager 3Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Picture Here
  • 4. Program Agenda         Introduction Agenda Overview Dealing With Space Constraints Dealing With Time Constraints Dealing with Resource Usage Constraints Minimizing Locking Impacts Handling Data Corruption Conclusion 4Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 5. Introduction Insert Picture Here 5Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 6. Who We Are Matt Lord  Former MySQL Support engineer  Worked with hundreds of backup issues  Now Product Manager Sveta Smirnova  MySQL Support engineer  Support representative in MySQL Enterprise Backup (MEB) team 6Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 7. Why Backups? 1. They are a critical insurance against:  Hardware failure  User error  Program error 2. You can use them to deploy new instances 3. Setup replication clones 4. Data is your most valuable IP. You have to protect it from loss. 7Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 8. The Backup Conundrum 1. Backup tools guarantee consistency by:  Locking tables  Running long transactions  Even using global locks 2. Backup jobs generate a lot of system overhead 3. Backups take up a lot of storage space How do you backup your critical production systems then?     Servers are performing critical business functions Downtime is not acceptable Negative performance impacts are not acceptable Storage space is critical 8Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 9. So How Do I Perform Backups...  without using local storage?  with minimal local resource usage?  with minimal impact on the MySQL instance I'm backing up?  when I need the backup to complete in a small time window?  without locking tables and entire MySQL instances?  when the data or the backup itself has corrupt data? 9Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 10. What We Won't Cover  Full backups  Partial backups  Incremental backups  Backup automation  General backup tuning Instead, see:  MySQL Enterprise Backup BOF [BOF3558, Saturday, 5:30 PM]  MySQL Backup-and-Recovery [CON7659, Sunday, 11:30 AM]  Backing Up MySQL Databases [CON3278, Sunday, 1:00 PM]  MySQL Enterprise Edition – Hands-On Lab, Sunday, 10:00 AM 10Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 11. Dealing With Storage Constraints 11Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Picture Here
  • 12. When There Is Little Storage Space  Very large data directory size  Small amount of local storage space left  But MEB process should be run on the same machine! 12Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 13. Backup Options  Compressed backups: --compress, --compress-level=LEVEL  Stream backups over SSH  Backup directly to tape, using the SBT interface  Create partial backups: --include, --with-tts  Create incremental backups  Don’t backup empty and unused pages: --skip-unused-pages 13Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 14. Backup Options: Compression  Compression ratios of 80% or more are possible! shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed > backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed > backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed 334M /opt/mysql/backups/uncompressed 334M /opt/mysql/backups/uncompressed 334M total 334M total shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed >> --compress --compress-level=9 backup 2>/dev/null && du –csh --compress --compress-level=9 backup 2>/dev/null && du –csh >> /opt/mysql/backups/compressed /opt/mysql/backups/compressed 81M /opt/mysql/backups/compressed 81M /opt/mysql/backups/compressed 81M total 81M total 14Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 15. Backup Options: Stream Backups  We can avoid local storage altogether Simply specify the tmpdir as the backup location (for temporary data)  Should be big enough to keep all changes made during backup  Still no space requirements for data files shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup > --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo > --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo > "cat > /tmp/backup.img“ > "cat > /tmp/backup.img“ shell# ssh matt@solo "ls -lh /tmp/backup.img" shell# ssh matt@solo "ls -lh /tmp/backup.img" -rw-r--r-staff 333M Sep 10 15:54 /tmp/backup.img -rw-r--r-- 11 matt matt staff 333M Sep 10 15:54 /tmp/backup.img 15Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 16. Backup Options: Using the SBT Interface  We can avoid local storage altogether  Backup directly to your company’s MMS – Oracle Secure Backup (fully supported) – IBM Tivoli Storage Manager – Symantec NetBackup shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08 shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08 >> --backup-dir=/tmp/backup backup-to-image --backup-dir=/tmp/backup backup-to-image shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup > --sbt-lib-path=/usr/lib/libobk.so > --sbt-lib-path=/usr/lib/libobk.so > --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt” > --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt” > --backup-dir=/tmp/backup backup-to-image > --backup-dir=/tmp/backup backup-to-image 16Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 17. Backup Options: Partial Backups  Only backup your important schemas and tables  MEB now supports InnoDB transportable tablespaces shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related >> --backup-dir=/opt/mysql/backups backup --backup-dir=/opt/mysql/backups backup shell# mysqlbackup --use-tts=with-minimum-locking shell# mysqlbackup --use-tts=with-minimum-locking > --include=employees.* --backup-dir=/opt/mysql/backups backup > --include=employees.* --backup-dir=/opt/mysql/backups backup 17Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 18. Backup Options: Incremental Backups  Only backup what’s changed since your last full backup shell# mysqlbackup --incremental shell# mysqlbackup --incremental >> --incremental-base=dir:/opt/mysql/backup/monday --incremental-base=dir:/opt/mysql/backup/monday >> --incremental-backup-dir=/opt/mysql/backup/tuesday backup --incremental-backup-dir=/opt/mysql/backup/tuesday backup shell# mysqlbackup --incremental shell# mysqlbackup --incremental > --incremental-base=history:last_backup --with-timestamp > --incremental-base=history:last_backup --with-timestamp > --incremental-backup-dir=/opt/mysql/backup backup > --incremental-backup-dir=/opt/mysql/backup backup 18Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 19. Backup Options: Skip Blank & Unused Pages  Tell InnoDB not to backup blank or unused pages  Use this in combination with compression for optimal storage efficiency shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed >> --compress --compress-level=9 --skip-unused-pages backup --compress --compress-level=9 --skip-unused-pages backup 19Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 20. Restore Options  MEB 3.9 introduces single-step restores – Even for compressed and single image file backups  This allows us to avoid unnecessary duplication of data shell# ssh matt@solo "cat /opt/mysql/backups/backup.img" || mysqlbackup shell# ssh matt@solo "cat /opt/mysql/backups/backup.img" mysqlbackup >> --backup-dir=/tmp/backup --uncompress --backup-image=- --backup-dir=/tmp/backup --uncompress --backup-image=> --datadir=/var/lib/mysql --innodb_log_group_home_dir=. > --datadir=/var/lib/mysql --innodb_log_group_home_dir=. > --innodb_log_files_in_group=4 --innodb_log_file_size=2G > --innodb_log_files_in_group=4 --innodb_log_file_size=2G > --innodb_data_file_path="ibdata1:1G:autoextend" > --innodb_data_file_path="ibdata1:1G:autoextend" > copy-back-and-apply-log > copy-back-and-apply-log 20Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 21. Dealing With Time Constraints Insert Picture Here 21Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 22. I Need the Backup Now!  Scheduled maintenance windows may be 30 mins or less  The backup may be needed immediately for a new deployment  One or more schemas may need to be moved to another host ASAP 22Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 23. Backups Methods  Direct FS backup – Requires mysqld to be shutdown, or at least globally locked  Logical backups (mysqldump) – VERY slow, consumes a lot of disk space, requires too many locks  MySQL Enterprise Backup (MEB) – Provides online hot backups for transactional storage engine InnoDB – Includes many performance and resource usage controls – Supports partial backups and transportable tablespaces – Supports incremental backups 23Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 24. MySQL Enterprise Backup 49x faster than mysqldump 24Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 25. MySQL Enterprise Backup 80x faster than mysqldump 25Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 26. MEB Performance Options  Increase buffers for CPU intensive multi-threaded ops: --number-of-buffers  Increase read I/O threads: --read-threads  Increase write I/O threads: --write-threads  Increase CPU processing threads: --process-threads  Increase the maximum memory used for caching: --limit-memory  Don't backup unused pages in InnoDB tables: --skip-unused-pages 26Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 27. MEB Example: Time Comparison  2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp backup 2>/tmp/log.txt backup 2>/tmp/log.txt real 0m34.403s real 0m34.403s user 0m3.069s user 0m3.069s sys 0m2.043s sys 0m2.043s shell# time mysqlbackup --backup-dir=/opt/mysql/backups shell# time mysqlbackup --backup-dir=/opt/mysql/backups > --process-threads=4 --limit-memory=1000 --skip-unused-pages > --process-threads=4 --limit-memory=1000 --skip-unused-pages > --with-timestamp backup 2>/tmp/log.txt > --with-timestamp backup 2>/tmp/log.txt real 0m29.883s real 0m29.883s user 0m3.121s user 0m3.121s sys 0m1.889s sys 0m1.889s  Cut backup times by 50%+ on enterprise class machines with large datasets 27Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 28. Dealing With Resource Usage Constraints 28Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Picture Here
  • 29. Why is the Application Slow Right Now?  Backups can be CPU intensive  Backups can be I/O intensive, also causing CPU cycles in IO_Wait  Backups may use various locks within MySQL  So how to take backups without having a noticeable affect… 29Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 30. Backup Options  Limit memory usage: --limit-memory  Limit CPU usage: --process-threads  Sleep X ms after each time we copy 1 block of data: --sleep  Avoid internal locks entirely: --no-locking, --innodb-only  Limit disk I/O: --read-threads, --write-threads  Don’t copy empty and unused pages: --skip-unused-pages  Process limits at OS level: cgroups, taskset, cpulimit, etc. 30Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 31. MEB Example: Resource Usage  2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' >> mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp >> backup 2>&1 || tail -1 backup 2>&1 tail -1 Exec Time: 0:23.11, CPU: 21%, MEM: 951648KB, IO: 4229360 Exec Time: 0:23.11, CPU: 21%, MEM: 951648KB, IO: 4229360 shell# taskset -c 1 /usr/bin/time -f shell# taskset -c 1 /usr/bin/time -f > 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup > 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup > --backup-dir=/opt/mysql/backups --limit-memory=50 > --backup-dir=/opt/mysql/backups --limit-memory=50 > --process-threads=1 --skip-unused-pages --no-locking --sleep=2000 > --process-threads=1 --skip-unused-pages --no-locking --sleep=2000 >> --with-timestamp backup 2>&1 || tail -1 --with-timestamp backup 2>&1 tail -1 Exec Time: 4:22.06, CPU: 1%, MEM: 222096KB, IO: 3384064 Exec Time: 4:22.06, CPU: 1%, MEM: 222096KB, IO: 3384064 31Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 32. Minimizing Locking Impacts Insert Picture Here 32Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 33. MEB Locking Related Options  Skip ALL locking: --no-locking (not 100% safe for non-transactional data)  Only backup transactional data: --innodb-only  Incremental backups using the redo log: --incremental-with-redo-log-only  Partial backups with minimal locking: --use-tts=with-minimum-locking 33Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 34. I Thought MEB Backups Were “Hot”?  FLUSH TABLES WITH READ LOCK is used at the end – Necessary to get a snapshot of all non-transactional data  Can be avoided with various options: – Don’t take any locks at all: --no-locking – Only backup InnoDB tables: --only-innodb[-with-frm] 34Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 35. What About the Non-Transactional Data?  mysqlhotcopy – lock and copy each non-transactional table  mysqldump – lock and logically export each table  FS snapshots – copy individual table files out of the snapshot  Manage the backups on an existing MySQL Replication Slave  mysqlbinlog –read-from-remote-server --raww  The amount of non-transactional data will continue to shrink in MySQL 5.7 35Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 36. Recap: Backups With Minimal Impact Limit shared resource usage:  MEB: --limit-memory, --sleep, --[read|write|process]-threads  Stream the backup to a remote FS Backup only as much data as required:  MEB: Partial backups with --use-tts=with-minimum-locking  MEB: Skip unused pages with --skip-unused-pages (InnoDB only)  MEB: Redo log only incremental backups (InnoDB only) Use 100% non-locking backups:  MEB: --only-innodb, --only-innodb-with-frm, --no-locking – Not 100% safe for .frm files, and for non-transactional data 36Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 37. Handling Data Corruption Insert Picture Here 37Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 38. Backups with Corruption Sometimes tables get corrupted! If production is corrupted, but the backup is fine:  Do a full or partial (--with-tts) restore from the backup  Apply changes from binary logs If both the production data, and the backup itself are corrupted:  InnoDB – --innodb_force_recovery – SELECT INTO OUTFILE  MyISAM – myisamchk 38Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 39. Backups with Corruption What to do if:  You have only full MEB backup, but need to restore single table  You can not setup spare instance  You can not use TTS MyISAM  Simply copy .frm, .MYI, and .MYD files InnoDB  Things get harder  Each table needs to have the same table ID in the shared tablespace, in both the backup and production server 39Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 40. Backups with Corruption: Full Backup and InnoDB If you can setup separate instance and restore full backup there  Do it!  If you can use TTS – Backup needed table using MEB with option --use-tts – Restore it 40Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 41. Backups are Corrupted: Full Backup and InnoDB If you can not!  The problem  Table ID in the InnoDB shared tablespace must be same in the backup and production server – They are if: – this is a backup of the same production server – you originally cloned this server from this backup and – You use option –innodb-file-per-table – Simply copy the .ibd file 41Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 42. Backups are Corrupted: Full Backup and InnoDB If table ID is not the same and you use option –innodb-file_per_table  Recover manually – Simulate the internal InnoDB table counter – Start spare instance – Create, then drop table with the same structure as the original one until counter increases to appropriate value – Import backed up tablespace into spare instance – Copy .ibd file to production server 42Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 43. Backups are Corrupted: Full Backup and InnoDB But how will you find what to type in CREATE TABLE statement?  mysqlfrm, the FRM reader will help you $$ mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333 mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333 ## Starting the spawned server on port 3333 ... done. Starting the spawned server on port 3333 ... done. ## Reading .frm files Reading .frm files ## ## Reading the city.frm file. Reading the city.frm file. # # # CREATE statement for city.frm: # CREATE statement for city.frm: # # CREATE TABLE `test1`.`city` ( CREATE TABLE `test1`.`city` ( … … 43Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 44. Conclusion Insert Picture Here 44Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 45. Conclusion  Backups are a must  Production constraints are a fact of life  Difficult problems can be solved with MySQL Enterprise Backup  MySQL Support is there to help with your specific problems 45Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 46. References Documentation:  http://dev.mysql.com/doc/mysql-enterprise-backup/3.9/en/index.html  http://dev.mysql.com/doc/workbench/en/mysqlfrm.html  https://blogs.oracle.com/mysqlenterprisebackup/ MySQL Enterprise Backup Trial Downloads:  https://edelivery.oracle.com/ 46Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 47. Graphic Section Divider 47Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  • 48. 48Copyright © 2013, Oracle and/or its affiliates. All rights reserved.