Your SlideShare is downloading. ×
How we setup Rsync-powered Incremental Backups
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

How we setup Rsync-powered Incremental Backups

5,296

Published on

Published in: Technology
1 Comment
5 Likes
Statistics
Notes
  • Very interesting solution. Thanks for sharing!
    Out of curiosity.. any reason why you have disabled saving the powerpoint/pdf ?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
5,296
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
1
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • TTP Conference 2011 – Celebrating 25 years of supporting Novell Solutions
  • According to the man page, --link-dest hard links to files in the link destination when the backup is the same as the source. (ie. identical files).
  • We set our link dest to current, a symbolic link always pointing to the current backup, then preform the backup. Next we set our current link to the latest backup and delete backups older than 30 days.
  • This is how we backed up a Netware server using NCP.
  • ..and a Windows one, using CIFS.
  • This is a screenshot of alpine-backup's code. As you can see, it is quite similar to our bash scripts, but adds error checking and is written in PHP. If you want to have a look at alpine-backup, the URLs are on the bottom.
  • This what our backup directory looks like. As you can see, only the first backup is large, the rest are quite small, only about 200MB large.
  • This an example of the configuration file format alpine-backup uses. It is an INI file. (I used this format because PHP had a built-in parser for it.)
  • This what our mount drivers look like. This particular one actually stops and restarts services for backup. (We use it for iPrint backups.)
  • Here is a screenshot of the mail we get from cron. As you can see from the summary, the backups are all successful!
  • This is a diagram of Selkirk College's organizational structure. All our servers are backed up in at least two locations. As you can see, there are two islands: the Nelson-centric one and the Castlegar-centric one. This is due to the fact that there are no fibre links between Castlegar and Nelson.
  • ...about half. (we sometimes use different parameters for our main and offsite backups.)
  • At this point I'd like to show you our first demo: How no-change syncing works. QNAP doing a Live Backup
  • QNAP in Slave Mode
  • Well, this does not explain much, let's have a look at snapup.sh
  • The first two commands SSH into our GroupWise server and use the perl scripts to create and activate a snapshot. Then we mount the snapshot and perform the backup via SSH.
  • Now, I'm going to move on to our next demo. Live Post Office
  • We start up the POA using an a backup.
  • ..and now: some old mail.
  • alp-dbrestore does not like dashes in DB Names.
  • Next, I'm going to show you how we restore our MySQL backups. First, we create test article.
  • Then, we create a new DB with an old version of the site. (Avoid dashes in database name)
  • We switch Drupal to the restore DB.
  • The article has now been rolled back.
  • This is a screenshot of our Oracle backups in action. Since we're running early, I'm going to show you what alpine-backup's code is like.
  • Any Questions? TTP Conference 2011 – Celebrating 25 years of supporting Novell Solutions
  • Transcript

    • 1.
        TTP Conference 2011
        Celebrating 25 years of supporting Novell Solutions
    • 2. Rsync Backups
        How we set up rsync powered incremental backups for Linux.
      Nicholas and Nick Paun
    • 3. The Plan
      • Because our obsolete tape backups were very slow, unreliable and hard to manage, we decided to go hard disk.
      • 4. We needed to be able to keep the backups for at least 30 days.
      • 5. The only way we could do this is by using incremental backups.
      • 6. After ruling out large, expensive proprietary systems, we chose rsync .
    • 7. About rsync
      • Using -- link-dest , rsync can make incremental backups look like full ones without wasting any disk space by using hard links. This makes for easy restoring!
      • 8. Because of rsync 's algorithm we do not transmit the full data over the network (except for the first time.)
      • 9. Rsync is very reliable and rarely breaks due to file changes.
    • 10. Version 1: The Scripts
      • Our scripts were based on some heavily tweaked examples on the rsync site.
      • 11. These simple scripts had no error checking, but they worked very well!
      • 12. We needed to backup Netware servers too, so we used NCP mounting , which was very slow.
    • 13. The Scripts echo "Here we rsync DATA volume on CASADMLX server to /bkup/casadmlx/DATA/ folder on CASARKLX server" rsync -av --link-dest= /bkup/casadmlx/DATA/current root@casadmlx:/media/nss/DATA/ /bkup/casadmlx/DATA/ $date rm /bkup/casadmlx/DATA/current ln -s /bkup/casadmlx/DATA/ $date /bkup/casadmlx/DATA/current rm -r /bkup/casadmlx/DATA/ `date --date='30 days ago' +%F`
    • 14. The Scripts echo "Here we rsync DATA volume on CASADM server to /bkup/casadm/DATA/ folder on CASARKLX server" ncpmount -S casadm -A casadm -U .bkup.cs.users.selkirk -P provo /mnt/casadm/ rsync -av --link-dest= /bkup/casadm/DATA/current /mnt/casadm/DATA/ /bkup/casadm/DATA/ $date rm /bkup/casadm/DATA/current ln -s /bkup/casadm/DATA/ $date /bkup/casadm/DATA/current rm -r /bkup/casadm/DATA/ `date --date='30 days ago' +%F` umount /mnt/casadm/
    • 15. The Scripts echo "Here we rsync INSTRUCTORS volume on FS1 server to /bkup/fs1/INSTRUCTORS/ folder on CASARKLX server" mount.cifs //fs1/INSTRUCTORS /mnt/fs1/ -o user=Administrator,pass=provo rsync -av -e 'ssh -p7774' --link-dest= /bkup/fs1/INSTRUCTORS/current /mnt/fs1/INSTRUCTORS/ /bkup/fs1/INSTRUCTORS/ $date rm /bkup/fs1/INSTRUCTORS/current ln -s /bkup/fs1/INSTRUCTORS/ $date /bkup/fs1/INSTRUCTORA/current rm -r /bkup/fs1/INSTRUCTORS/ `date --date='30 days ago' +%F` umount /mnt/fs1/
    • 16. Version 2: alpine-backup
      • Eventually we decided that we needed a proper backup system.
      • 17. We wanted to have good error checking, replication, reporting and central control.
      • 18. We also wanted to actually understand what was going on and we wanted a system that would fit our exact needs. So we wrote our own.
    • 19. Version 2: alpine-backup
      • After learning PHP programming and experimenting for awhile, I wrote alpine-backup , our current backup solution.
      • 20. We decided to implement only the bare-minimum that we needed.
      • 21. The result was a 100-line script that supports custom mount drivers and performs an incremental backup using rsync .
    • 22. alpine-backup function  backup ( $opt , $src , $dest , $date , $name )  {    $_opt  =  OPTIONS ;    passthru ( "rsync  $_opt   $opt  --link-dest  $dest /current  $src   $dest / $date " , $return );   if ( $return  !=  0 )    {     echo( "* Backup Failure ( $name ): rsync did not finish sucessfully.n" );     die( 1 );    }      unlink ( " $dest /current" );    symlink ( " $dest / $date " , " $dest /current" );   echo( "* Backup Successful:  $name n" );  }
        • Website: http://www.icebergsystems.ca/
        • 23. Google Code: http://alpine-backup.googlecode.com
    • 24. Our backup directory
    • 25. Configuration files ;; silvoes2lx backup config ;; September 2010 -- Nicholas Paun [APPS] source = "root@silvoes2lx:/media/nss/APPS/" dest = "/bkup/silvoes2lx/APPS" opt = "-e 'ssh -p7774'" delete_after = "+30" [HOME] source = "root@silvoes2lx:/media/nss/HOME/" dest = "/bkup/silvoes2lx/HOME" opt = "-e 'ssh -p7774'" delete_after = "+30"
    • 26. Mount Drivers function  mountfs ( $user , $server , $command )   {     $return  =  passthru ( "ssh  $user @ $server   ' $command   stop'" );     return(! $return );   } function  umountfs ( $user , $server , $command )   {     $return  =  passthru ( "ssh  $user @ $server   ' $command   start'" );     return(! $return );   }
    • 27. Logging
    • 28. Experiments
      • We setup offsite mirroring
      • 29. Experimented with rsync daemons
      • 30. We used batch files for replication
      • 31. We ran rsync on cygwin (too slow.)
      • 32. Backed up Windows servers via CIFS mounting
      • 33. We setup Groupwise backups
      • 34. We also started iPrint backups
    • 35. Syncing using Batch Files
      • Batch files allow you to send the same file list to multiple backup servers at once.
        • ie. mirroring, replication, etc.
      • We used them to sync our backups between 2 campuses.
      • 36. This was our first attempt at offsite backups.
      • 37. It had a drawback of losing sync if files had changed in between batches.
    • 38. Syncing using Batch Files
      • The first command uses --write-batch to create a batch file.
      • 39. The second one uses --read-batch to perform the backup.
      • 40. rsync -av --write-batch=SCRATCH.batch --link-dest=/bkup/caslablx/SCRATCH/current /mnt/caslablx/SCRATCH/ /bkup/caslablx/SCRATCH/$date
      • 41. ssh n10arklx rsync -av --read-batch=- --link-dest=/bkup/caslablx/SCRATCH/current /bkup/caslablx/SCRATCH/$date <SCRATCH.batch
    • 42. Offsite Backups
      • The college bought 5 QNAPs, for offsite backups.
      • 43. The QNAPs are NASs running some mystery meat Linux, that was the source of many headaches.
      • 44. Initially, we added more configuration files that would copy the backups from the main backup server to the offsite one, but this grew quite complicated.
    • 45. Selkirk College Backup System
    • 46. No-change Syncing
      • So, we came up with a better idea: the configuration files would not be modified, and still point at the real backup server.
      • 47. But, when a QNAP reads the configuration files, it would run in slave mode and pull the backups from the backup server, instead of the live server.
      • 48. This change removed the need for half of our configuration files.
    • 49. Demo: No-change Syncing
    • 50. Demo: No-change Syncing
    • 51. GroupWise Backups
      • Because dbcopy was slow and inefficient for backups, we decided to snapshot our GroupWise volumes.
      • 52. This means that we can apply rsync 's deduplication ( --link-dest ) to GroupWise too.
      • 53. We save 90-95% disk space and it is 10 times faster compared to a full backup.
    • 54. GroupWise Backups
      • As an added benefit, GroupWise and FS backups can now be handled by the same tool.
      • 55. To snapshot NSS volumes we used Dean Giles' perl scripts (found on Cool Solutions).
      • 56. If we ran GroupWise on an LVM volume, we could use LVM snapshots instead.
    • 57. GroupWise Mount Driver function  mountfs ( $user , $server , $command )   {     $return  =  passthru ( &quot;snapup.sh&quot; );     return(! $return );   } function  umountfs ( $user , $server , $command )   {     $return  =  passthru ( &quot;snapdn.sh&quot; );     return(! $return );   }
    • 58. Snapup.sh
        ssh casgrplx perl /usr/local/bin/AddPoolSnapshot.pl /dev/evms/CASGRPLXPOOL sdb 200000000 CASGRPLXPOOL_S1 ssh casgrplx perl /usr/local/bin/ActivatePoolSnapshot.pl CASGRPLXPOOL_S1 ssh casgrplx mount GRPWISE_SV /media/nss/GRPWISE_SV -t nssvol -o rw,name=GRPWISE_SV
    • 59. Demo: GroupWise PO Restore
    • 60. Demo: GroupWise PO Restore
    • 61. Demo: GroupWise PO Restore
    • 62. MySQL Backups
      • We wanted to incrementally backup a database, but still use native dumping tools.
      • 63. So we added a MySQL agent to our tools.
      • 64. To gain maximum space savings, we split up the SQL files into one file per table.
      • 65. Now we can use rsync 's --link-dest feature on our DBs too.
    • 66. MySQL Backups
      • This is the the command we use to backup databases
      mysql -p password db -e 'show tables' | grep -v 'Tables_in_' #show all tables in database mysqldump -p password --skip-dump-date db $table > $dumpdir/$table #create a dump of each individual table (with no timestamp, for linking) find $dumpdir -exec touch -d 2010-04-01 {} ; #touch dumps to a consistent date (for linking)
    • 67. Demo: Drupal DB Restore
    • 68. Demo: Drupal DB Restore
    • 69. Demo: Drupal DB Restore
    • 70. Demo: Drupal DB Restore
    • 71. Oracle Backups
      • One day, we got a user request to set up Oracle backups on a Windows server.
      • 72. We had never seen an Oracle database before, so we did some searching and figured out how to use expdp.
      • 73. First we logged in to the database and set up a directory handle:
      • 74. CREATE DIRECTORY sgrcdev AS '/g/sgrcdev' ;
    • 75. Oracle Backups
      • Next, we use cygwin to run expdp on the server:
      • 76. expdp system/$passwd@$db directory=$db full=y dumpfile=$db.dmp logfile=$db.log
      • 77. … And then, we just copy the backups over and reset dump directory.
    • 78. Oracle Backups
    • 79. Conclusion
      • In 2007, we didn't have a real backup, even if we bought an enterprise solution.
      • 80. Now, we can control all of our backups (FS, GroupWise and Database) from one place, using one lightweight tool.
      • 81. We save a lot of disk space by using rsync .
      • 82. Most importantly, we achieved peace of mind.
    • 83.
        TTP Conference 2011
        Celebrating 25 years of supporting Novell Solutions

    ×