A Backup Today Saves Tomorrow
Upcoming SlideShare
Loading in...5

A Backup Today Saves Tomorrow



A Backup Today Saves Tomorrow is a presentation from Percona Live 2013 that provides insight into planning and the tools used today to capture MySQL backups.

A Backup Today Saves Tomorrow is a presentation from Percona Live 2013 that provides insight into planning and the tools used today to capture MySQL backups.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

A Backup Today Saves Tomorrow A Backup Today Saves Tomorrow Presentation Transcript

  • A backup today saves youtomorrowBecause bad things do happenBen Mildren, MySQL Team Technical LeadAndrew Moore, MySQL DBA
  • About Pythian•  Recognized Leader:–  Global industry-leader in remote database administration services and consulting for MySQL,Oracle, Oracle Applications and Microsoft SQL Server–  Work with over 250 multinational companies such as Forbes.com, Fox Sports, Nordion andWestern Union to help manage their complex IT deployments•  Expertise:–  Pythian’s data experts are the elite in their field. We have the highest concentration of OracleACEs on staff—10 including 2 ACE Directors and 2 Microsoft MVPs.–  Pythian holds 7 Specializations under Oracle Platinum Partner program, including OracleExadata, Oracle GoldenGate & Oracle RAC•  Global Reach & Scalability:–  Around the clock global remote support for DBA and consulting, systems administration,special projects or emergency response
  • So then…Backups
  • Agenda
  • Disaster Recovery Plan“a documented process or set of proceduresto recover and protect a business ITinfrastructure in the event of a disaster.”http://en.wikipedia.org/wiki/Disaster_recovery_plan
  • Disaster Recovery PlanBackup & RecoveryprocessDisaster RecoveryPlan
  • Disaster Recovery PlanDesigning a Disaster Recover Plan•  Define your boundaries. What can you affordto lose? Time or data?•  Backup, what, when, where•  Organize (find what you need at 4am)•  Protect against disaster, removing SPOF•  Document and train•  Test restore, automation, review
  • Disaster Recovery PlanTime or data?As defined in ‘business continuity planning’ global standardsRTO & RPO
  • Disaster Recovery PlanRecovery Time Objective“the duration of time and service level withinwhich a business process must be restoredafter a disaster or disruption”http://en.wikipedia.org/wiki/Recovery_Time_Objective
  • Disaster Recovery PlanRecovery Time ObjectiveIncludes•  Time allowed to troubleshoot (without recovery/fix)•  The recovery time itself•  Time for communication to stakeholders
  • Disaster Recovery PlanRecovery Point Objective“the maximum tolerable period in which datamight be lost from an IT service due to a majorincident”http://en.wikipedia.org/wiki/Recovery_Point_Objective
  • Disaster Recovery PlanCan you afford…•  Downtime?•  Data loss?Generally it costs too much $ tosay no to both
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Disaster Recovery Plan
  • Why Recover?Data Loss•  Hardware failures e.g. HDD failure•  Corruption•  Natural disasters / Man-made disasters
  • Why Recover?Time Travel•  Data accidentally changed•  Runaway bug in the software
  • Why Restore•  Audit–  Your company may be subject to auditprocesses•  Legal–  You may be required to supply data aspart of legal proceedings•  Testing–  “Other” environments–  Testing backup process–  Verification of backup files•  Scale out–  Building replicasRestoring a backup for reasons other then disastercan include but not limited to;
  • How to RestoreA valid BackupBefore you can restore you need an importantingredient;
  • !MySQL BackupsSome alternatives are NOT backups•  Standby Replicas/ Time delay Replicas•  Passive Cluster Nodes•  RAID•  Storage Snapshots•  Untested backups
  • MySQL BackupsChallenges that face MySQL Backups•  No one tool to rule them all•  Mixed engine environments•  MySQL Surface area•  Production impact of backup
  • MySQL Backup TypesHot vs. Warm vs. ColdLogical vs. PhysicalLocal vs. RemoteFull vs. Point in Time
  • Backup Repository ModelFULL: Complete system imagesDIFFERENTIAL: Changes since last full backupINCREMENTAL: Changes between two points in time.Backups need to be stored and organized
  • MySQL Backup TypesHot vs. Warm vs. ColdCan you take your MySQL server offline to make a backup?Do you have a replica where your backup will not impact the master?Logical vs. PhysicalBackup the files or dump out the data so that you can recreate your serverLocal vs. RemoteCan you afford the latency of a network round-tripFull vs. Point in TimeLinked to your RPO, how granular should you go?
  • MySQL Backup ToolsLogical•  mysqldump•  mydumperPhysical•  Cold Backup•  MySQL Enterprise Backup•  XtrabackupSnapshot•  SAN•  LVM•  ZFSFrameworks•  Zmanda•  Holland•  Xtrabackup Manager
  • mysqldumpThe command line utility to create logical dumps of your schema, databaseobjects and data Good solution for small to medium datasets (0G>20G)Pros•  Packaged with MySQL•  Broad compatibility (engines)•  Flexible use with pipelines(gzip, sed, awk, pv, ssh, nc)•  No locking in --single-transaction with innodb onlytablesCons•  Single threaded•  Locking by default•  Can be hard to troubleshooterrors (syntax error on line14917212938)•  Slow to reload data•  Be wary of foreign keys andtriggers when restoring.Type: Logical Heat: Hot[innodb only]:Warm[myisam] Impact: Medium Speed: Slow
  • mysqldumpBackup ExamplesBackup all tablesmysqldump –u user –p pass --all-databases > backup.sqlBackup all tables compressedmysqldump –u user –p pass --all-databases | gzip -5 > backup.sql.gzBackup with database objectsmysqldump –u user –p pass --routines --triggers --events > backup.sqlBackup with no datamysqldump –u user –p pass --no-data --triggers –events > backup.sql
  • Restore ExamplesRestore mysqldumpmysql –u user –p < backup.sqlRestore from withinsource backup.sql;Backup & restore one linermysqldump db_one | ssh moore@myslave mysql –u user db_onemysqldumpRestore a compressed dump binlog off(echo "set session sql_log_bin=0;" ; zcat dump.sql.gz) | mysql –u userRestore table from dumpfile using sedcat dump.sql | sed -n /^-- Table structure for table `t1`/,/^UNLOCK TABLES;/p’ | mysql –u user
  • TipsRTFMThere’s much more to see and test then what I’ve shown--where = “id between 10000 and 20000”--single-transactionCheck the timeTime your backup and recovery durations this will allow you to set expectations if you need to use thebackups. The time and pv unix programs will help youtime mysqldump > backup.sqlreal 0m0.108suser 0m0.003ssys 0m0.006spv backup.sql | mysql -u user -p101MB 0:00:39 [5.6MB/s][===> ] 13% ETA 0:03:12mysqldump
  • mydumperA parallel logical dumper for MySQL developed and maintained by ex-MySQLemployees.Type: Logical Heat: Warm Impact: Medium Speed: FastPros•  It’s fast! Multithreded•  Human readable output•  Good solution for larger datasets(multi-threaded)•  Native compression•  Dump remote host•  Compatible with drizzle•  Nearly hot if using only innodb tablesCons•  No official binaries shipped•  Slower restore then physical backupsbut faster then mysqldump•  Caveats with restoration•  Relies on mysqldump for databaseobjects (routines, events, etc)https://launchpad.net/mydumper
  • Build from source;[moore@bkphost ~]# wget https://launchpad.net/mydumper/0.5/0.5.2/+download/mydumper-0.5.2.tar.gz[moore@bkphost ~]# cmake .-- The CXX compiler identification is GNU…-- Build files have been written to: ~/mydumper-0.5.2[moore@bkphost ~]# makeScanning dependencies of target mydumper[ 20%] Building C object CMakeFiles/mydumper.dir/…[ 80%] Built target mydumperScanning dependencies of target myloader[100%] Built target myloader[moore@bkphost ~]# mydumper --help…Dependencies(MySQL, GLib, ZLib, PCRE)mydumper
  • mydumperExample of mydumper & myloaderDump data[moore@bkphost ~]# mydumper -h localhost –u bkpuser –p secret --database world --outputdir /backup_dir --verbose 3** Message: Connected to a MySQL server** Message: Started dump at: 2013-04-15 12:22:48** Message: Thread 1 connected using MySQL connection ID 4** Message: Thread 1 dumping data for `world`.`City`** Message: Thread 1 shutting down** Message: Non-InnoDB dump complete, unlocking tables** Message: Finished dump at: 2013-04-15 12:22:54Restore data[moore@bkphost ~]# myloader -h localhost –u bkpuser –p secret --database world –d /backup_dir --verbose 3** Message: n threads created** Message: Creating table `world`.`City`** Message: Thread 1 restoring `world`.`City` part 0** Message: Thread 1 shutting down
  • Percona Xtrabackup 2.xOne of the strongest and widely used solutions for consistently backing up MySQL files focused onXtradb/Innodb but also support for non-transactional tables too. Xtrabackup makes use of the XtraDB/InnoDB crash recovery cycle to apply the redo logs to the data when preparing for a restore. Thisprepare phase can happen at the end of the backup or as part of the recovery phase.Type: Physical Heat: Hot Impact: Low Speed: FastPros•  Free, GPL Licensed•  Hot & Physical•  Throttles to keep load low•  Native compression (qpress)•  Export tables•  Parallel backup•  Wide OS compatibility•  Consistent backup of MyISAM•  Great documentation and recipes•  Compatible with XtraDB ClusterCons•  Windows version in Alpha•  Multiple stage restore•  Cannot prepare compressed (on the fly)backups•  Qpress compression < gzip/pigzpercona.com/software/percona-xtrabackup
  • XtrabackupC program that takes care of copying XtraDB/InnoDB data files and logs. Tails the iblog files whilst activity on the servercontinues whilst the XtraDB/InnoDB files are copied into the backup locationInnobackupexThe perl script that oversees the backup process. This allows the backup to handle the non-transactional tables. Itconnects to the server once the Xtrabackup binary suspends after copying the XtraDB/InnoDB files. Innobackupex issues a“FLUSH TABLES WITH READ LOCK” so that the supported non-transactional tables are not written on. If –safe-slave-backup was issued then Innobackupex stops and starts the slave before and after the file copy.Tar4ibdA special version of tar that understands how to handle innodb / xtradb data files. Archives built using the stream to taroptions need to be extracted using the –i option or the extraction will not succeed.XbstreamCustom streaming format which allows the dynamic compression and parallel copy of files to improve the performance ofthe overall backup.Percona Xtrabackup 2.x
  • Percona Xtrabackup 2.x
  • ExamplesFull Backup Restoreinnobackupex --apply-log /path/to/fulldestRestore Streamed Backup (tar & gzip)gunzip backup.gztar –xivf backup.tarinnobackupex --apply-log backup/Backup Incrementalinnobackupex --incremental /path/to/incdest --incremental-basedir=/path/to/fullbackupRecipes: http://www.percona.com/doc/percona-xtrabackup/how-tos.htmlPercona Xtrabackup 2.x
  • More ExamplesOptimizing the restore phaseinnobackupex --use-memory=2G --apply-log /path/to/backupBackups with pigzinnobackupex --stream=[xbstream|tar] . | [pigz|gzip] > backup.gzBackup Incrementalinnobackupex --incremental /path/to/incdest --incremental-basedir=/path/to/fullbackupRecipes: http://www.percona.com/doc/percona-xtrabackup/how-tos.htmlMany, many more options…Percona Xtrabackup 2.x
  • Other hintsStream to another hostssh or netcatNative parallel & compress--compress & --parallelNon-transactional tables?Use --rsync for much shorter lock time for large non-trx surface areaSingle table?--apply-log --exportPercona Xtrabackup 2.x
  • The official MySQL hot backup product. Proprietary license as a large expense per server.Lacks some of the features of Xtrabackup’s latest version. Solid solution for Enterprisecustomers.Type: Physical Heat: Hot Impact: Medium Speed: FastPros•  Hot Backups•  Physical Backups•  Compressed backups•  Throttling to avoid overloadingserverCons•  It costs real money L•  Throttling is a sleep callwhereas PXB uses IOPsMySQL Enterprise Backup (MEB)
  • Taking a cold backup of your system is one sure way to a consistent backup. By stopping mysqld you cancopy the files you need to the location you want, to gain a fully consistent backup of your data and config.You have to be able to afford the time to gracefully stop the server and then complete the copy. Buffersare lost from shutdown so this will impact the performance of the instance when started again.Type: Physical Heat: Cold Impact: Downtime Speed: FastPros•  Fast•  Consistent Backup across allengines•  Easily scripted•  No new software to learnCons•  It’s cold•  Buffers require warming onrestart•  Unsuitable for 24/7 operationsCold Backup
  • A new MySQL feature as of MySQL version 5.6. The ability to stream all binary logs to another host toenable redundancy for your binary logs the method used for incremental/point in time recovery. A goodaddition to the backup and recovery arsenal and worthy of a mention. Can stream binlogs from olderversions. --read-from-remote-server --raw --stop-neverType: Physical Heat: Hot Impact: Low Speed: FastPros•  Hot streaming•  Physical Backup of binarylogs•  Low processing costCons•  Dependent on connectivitybetween servers•  Not a complete backup•  Restore could becomecomplex if many binlogs areneededMySQL Binary Log Streaming (5.6)
  • Techniques as seen in mylvmbackup show that snapshots can be used to make the backup online usingcopy on write technologies. Using a snapshot capable filesystem such as LVM, ZFS, VSS or SAN storagewith the same ability can afford you a storage checkpoint where changes can be simply rolled back ifissues ariseType: Physical Heat: Warm Impact: Varies Speed: FastPros•  Fast•  Warm backups•  Familiar commands as withstorage toolsCons•  Crash recovery needed forInnoDB to apply redo log•  The ‘copy on write’ overheadcould impact performancefrom the point the snapshot iscreated until it’s destroyed.Snapshot Backups
  • Contrary to popular belief the vast majority of MySQL backups arewarm not hot.The FLUSH TABLES WITH READ LOCK statement is still requiredto guarantee consistency for several aspects of a MySQL backup.These include;•  MyISAM and other non transactional tables•  .frm files•  Binary log co-ordinates.The dreaded global READ LOCK!
  • In the FrameFrameworks for MySQL backupsTraits inherent of the framework concept–  Simplify complex technologies–  Implement specific design ‘rules’–  UI consistencyMySQL Backup Frameworks aim to solve;–  Wrap best practices into common interface–  Large environment administration pains–  Centralize groups configuration
  • ZRM (zmanda recovery manager)Offering both commercial and GPL versions, ZRM is a great option for organizing andscheduling your backups. Helps to simplify schedules, restores and verification.Type: Framework Heat: Varied Impact: Varied Speed: VariedPros•  Handles many backup methods,mysqldump, xtrabackup, snapshotbased full backups inc. EBS & SAN•  Commercial version has a centraldashboard GUI•  Automated alerts & reports•  Integrated into enterprise solutionssuch as Symantec NetBackup andTivoli TSMCons•  Commercial nature hides much of thenice features that make ZRM adesirable solution.•  Xtrabackup, VSS & SAN snapshotsonly available in commercial version•  Last release to community edition was2010.https://www.zmanda.com
  • HollandThe relatively unsung framework is originally from Rackspace and written in python under a New-BSDlicense. It is a pluggable framework with MySQL, Xtrabackup, SQLite and Postgres support. Due topluggable nature there is scope to backup more than just databases.Type: Framework Heat: Varied Impact: Varied Speed: VariedPros•  Open Source•  Pluggable structure•  In production at RackSpace•  Backup more then MySQL•  Manages retentionCons•  Small user base (get involvedfor the good of mankind!)•  No central control towerhttp://hollandbackup.org/
  • Xtrabackup ManagerWinner of 2012 community award, Xtrabackup Manager (XBM) was created by Lauchlan Muchay. XBMgives you a way to manage your Xtrabackup tasks for multiple hosts via a command line interface. Writtenin php XBM uses cron, netcat, pv and ssh to get things done.Type: Framework Heat: Warm/Hot Impact: low Speed: FastPros•  GPL License•  Low overhead•  Handles incrementalxtrabackups•  Manage schedule, retentionpolicyCons•  Still in beta and developmenthas slowed•  Tested on 1.x so far•  Small user base (get involvedfor the good of mankind!)https://code.google.com/p/xtrabackup-manager
  • Backup TestingRegular development / Staging restores•  Can be sufficient when infrastructure is limitedRestore server•  Dedicated•  Shared – Sandbox / mysqld_multiVerification•  Tests using information schema•  Best to include data taken at time of backup
  • Backup PhilosophyAdd redundancy•  Use Logical and Physical types•  Don’t forget your binary logs (5.6 can stream to non-mysql server)•  Local copies (HDD, DAS) avoid network copying•  Remote copies (NFS, SAN)•  Offsite copies (S3, secure tape storage i.e. Iron Mountain)Monitor those backups•  Check pass/fail and document a process for how to react if a backup fails•  A framework or custom wrapper will help
  • When backups go wrongmysqldumpsyntax error at line 1000101020020charsets and collation issues (check config)XtrabackupGhost backups.Shared tempdir for xtrabackup_checkpoint
  • Q&A
  • Thank youTo contact ussales@pythian.com1-877-PYTHIANTo follow ushttp://www.pythian.com/bloghttp://www.facebook.com/pages/The-Pythian-Group/163902527671@pythianhttp://www.linkedin.com/company/pythian
  • Your SpeakersAndrew Moore–  MySQL Production DBA–  Based in Bristol, UK–  @mysqlboy on twitter
  • Fin.