Linux Disaster Recovery Made Easy

Linux Disaster Recovery Made Easy Schlomo Schapiro Jeff Lindholm Systems Architect / Open Source Evangelist Technology Sales Specialist – Linux Immobilien Scout GmbH Novell, Inc. schlomo.schapiro@immobilienscout24.de jlindholm@novell.com

Agenda Linux and disaster recovery Solutions: relax and recover Secure disaster recovery for enterprise environments SUSE Linux Enterprise Server 11 SP1 HA – ® ReaR availability and support Live demo and use cases Questions and answers 2 © Novell, Inc. All rights reserved.

Linux and Disaster Recovery

Disaster Recovery Strategies Copy Files and Store Disk Disk Imaging Layout and Boot Info + Online -> no problem + Offline -> no open + Backup independent of disk file issues layout and sizes + Simple to restore – More effort required to – Online -> very restore (can be scripted) problematic – Maybe consistency – No “perfect” open problems, but should be source tool for solved by backup solution Linux available 4 © Novell, Inc. All rights reserved.

Advantages of a Linux System All information is stored in files; all files always readable Operation system and applications can be “slim” - 600MB enough for complete standard server Open system – open methods and procedures All steps of an installation can be scripts: partitioning, file systems, boot loader etc. 5 © Novell, Inc. All rights reserved.

Disaster Recovery – Media • Most important: external storage! • Bootable media: CD/DVD, USB key, LAN, tape ... • Media usually combination boot and backup media: – Bootable CD/DVD, USB key with backup data on it – LAN boot (PXE) with backup data via CIFS, NFS ... – Bootable tapes - HP OBDR (CD emulation) • Separation between boot media and backup data – Boot the system from a (small) USB key, CD/DVD or LAN – Recover the system with backup software, tar, rsync ... 6 © Novell, Inc. All rights reserved.

Disaster Recovery – How It Works • Store the disk layout – Partitioning, LVM and RAID configuration – File systems, file system labels ... – Boot loader (GRUB, LILO, ELILO) • Store the files (tgz, rsync, through backup software ...) • Create bootable rescue media with system configuration (and backup data) • Can be done online – No business interruption – 100% compatible with original systems hard- and software 7 © Novell, Inc. All rights reserved.

Disaster Recovery – Rescue Media • Create “rescue linux” from running system • Optimally compatible “tool box” • Clone the system environment – Linux kernel and modules – Device driver configuration – Network configuration – Basic system software and tools • Operate entirely in RAM (initrd) 8 © Novell, Inc. All rights reserved.

Disaster Recovery – In Action • Boot system from rescue media • Restore disk layout – Create partitions, RAID configuration and LVM – Create file systems (mkfs, mkswap) – Configure file systems (labels, mount points) • Restore the backup data • Restore the boot loader • Reboot • Done! 9 © Novell, Inc. All rights reserved.

Solutions: Relax and Recover

Commercial Solutions • Usually part of backup software • Extra licenses and can be costly (up to 1000€ / server) • Some are free, but are the any good? • Level of integration into backup software varies • Not all backup software actually supports disaster recovery • Cover only simple setups • Often static boot media, inflexible with regard to drivers • Certification on server hardware missing or lagging behind procurement cycles 11 © Novell, Inc. All rights reserved.

Open Source Solutions • Several software projects on the internet: – Relax and recover – mkCDrec – Mondo Rescue – CRU – BACULA – Your-Favorite-Selfmade-Script – <something you know but I never heard about> • In 2006, all of them where lacking something... 12 © Novell, Inc. All rights reserved.

Relax and Recover – ReaR http://rear.sourceforge.net • GPL Software – Developers in Germany and Belgium • 100% Bash script – no GUI and no dependencies • Utilize kernel, modules, binaries of host (kernel ≥ 2.6) • Support any combination of SW/HW RAID, LVM • Internal backup on CIFS, NFS ... • Boot media on CD/DVD, USB key and LAN (PXE) • Successor of mkCDrec 13 © Novell, Inc. All rights reserved.

ReaR – Features • Focus on disaster recovery and not backup • Tight integration with common backup software – delegate file backup to backup infrastructure • Simple full backup integrated • Complements backup software: – Backup software: Data storage and retrieval – ReaR: Recover system layout and make it work again – ReaR utilizes the backup software to restore the backup data • Use the best tool for the job 14 © Novell, Inc. All rights reserved.

ReaR – Backup Software • Supported solutions include: – CommVault Galaxy – IBM Tivoli Storage Manager – Veritas NetBackup – HP Data Protector – Rsync and other “external” methods – tar.gz archive on NAS share – CIFS, NFS, NCP ... • Very transparent integration • Quick implementation: 2-3 PT programming • Can be easily extended to support other vendors 15 © Novell, Inc. All rights reserved.

ReaR – Network Integration • Disaster recovery as part of network infrastructure – Backup software – file-level backup storage – ReaR – system environment – Boot rescue media via PXE – no physical media required – Very scalable – automated installation of entire disaster recovery data center ReaR Boot Files Software Backup PXE Boot Backup Network Storage infrastructure 16 © Novell, Inc. All rights reserved.

ReaR – Status • Stable software – i386 and x86_64 well tested – ia64 and ppc experimental • Regular releases (RPM, DEB, TGZ) • Major Linux distributions ship ReaR: – SUSE Linux Enterprise High Availability Extension 11 SP1 ® – openSUSE 11.2 and Fedora 11 ® • Community and commercial support available • Regular patch submissions from ReaR users 17 © Novell, Inc. All rights reserved.

ReaR - Development • Open Source development model: – Submit patches and feedback – “field testing” – Sponsoring • Modular design: – ReaR is a framework to plug together many small bash scripts – Maximize code reusability – Simple development model (vi works fine) – Little to no “interferences” between different areas of code • Documentation on project homepage – Hierarchical structure – References to source code 18 © Novell, Inc. All rights reserved.

ReaR – Configuration • Shell-script style configuration file • Usually: Set backup software and boot media System definition: ARCH = Linux-x86_64 OS = GNU/Linux OS_VENDOR = SUSE_LINUX OS_VERSION = 11 OS_VENDOR_ARCH = SUSE_LINUX/x86_64 OS_VENDOR_VERSION = SUSE_LINUX/11 OS_VENDOR_VERSION_ARCH = SUSE_LINUX/11/x86_64 Configuration tree: Linux-x86_64.conf : OK GNU/Linux.conf : OK SUSE_LINUX.conf : missing/empty SUSE_LINUX/x86_64.conf : missing/empty SUSE_LINUX/11.conf : missing/empty SUSE_LINUX/11/x86_64.conf : missing/empty site.conf : OK local.conf : OK 19 © Novell, Inc. All rights reserved.

ReaR – Operation rear [Options] <command> [command options ...] Relax & Recover Version 1.7.23.2 / 2010-03-19 Build: b80d32a97e41a6ec8a5b19cd6399e444 Copyright (C) 2006-2009 Schlomo Schapiro Gratien D'haese, IT3 Consultants Relax & Recover comes with ABSOLUTELY NO WARRANTY; for details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html Available Options: -V version information -d debug mode -D debugscript mode -S Step-by-step mode -s Simulation mode (shows the scripts included) -q Quiet mode -r a.b.c-xx-yy kernel version to use (current: 2.6.32.8-0.3-default) 20 © Novell, Inc. All rights reserved.

ReaR – Operation (continued) List of commands: dump Dump configuration and system information help print out usage mkbackup Create rescue media and backup system. mkbackuponly Backup system without creating a (new) rescue media. mkdeb Create DEB packages with this rear version mkdist Create distribution tar archive with this rear version mkrescue Create rescue media only mkrpm Create RPM packages with this rear version mktar Create tar archive with this rear installation mkvendorrpm Create vendor RPM with this rear version recover Recover the system validate Submit validation information 21 © Novell, Inc. All rights reserved.

ReaR – Example Execution # rear mkbackup Relax & Recover Version 1.7.23.2 / 2010-03-19 The preparation phase OK Physical devices that will be recovered: /dev/sda /dev/sdb Creating root FS layout OK Copy files and directories OK Copy program files & libraries OK Copy kernel modules OK Create initramfs OK Creating archive 'nfs://server.domain/backup-path/rear/backup.tar.gz' Transferred 297 MB in 254 seconds [1197 KB/sec] Making ISO image OK Wrote ISO Image /tmp/ReaR.iso (19M) Copying resulting files to network location OK The cleanup phase OK Finished in 258 seconds. 23 © Novell, Inc. All rights reserved.

ReaR – Disk Layout # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 2,5G 927M 1,4G 40% / devtmpfs 244M 128K 244M 1% /dev tmpfs 247M 0 247M 0% /dev/shm /dev/sda1 84M 19M 61M 24% /boot /dev/mapper/system-home 524M 17M 481M 4% /home /dev/md0 680M 55M 591M 9% /var 24 © Novell, Inc. All rights reserved.

ReaR – Disk Layout # pvs PV VG Fmt Attr PSize PFree /dev/sda3 system lvm2 a- 3,21G 0 /dev/sdb2 system lvm2 a- 312,00M 0 # lvs LV VG Attr LSize Origin Snap% home system -wi-ao 532,00M root system -wi-ao 2,44G swap system -wi-ao 572,00M # cat /proc/mdstat Personalities : [linear] [raid1] md0 : active raid1 sda2[0] sdb1[1] 706816 blocks super 1.0 [2/2] [UU] 25 © Novell, Inc. All rights reserved.

ReaR – Recovery Example # rear recover Relax & Recover Version 1.7.23.2 / 2010-03-19 Backup archive size is 298M Initializing physical devices OK Creating Software RAID devices OK Creating the LVM2 devices OK Creating file systems OK Recreated this filesystem layout: Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 2.5G 68M 2.3G 3% /mnt/local /dev/sda1 84M 5.6M 74M 7% /mnt/local/boot /dev/mapper/system-home 524M 17M 481M 4% /mnt/local/home /dev/md0 680M 17M 629M 3% /mnt/local/var Restoring from 'nfs://laptop/dev/shm/nfs/rear/backup.tar.gz' OK 26 © Novell, Inc. All rights reserved.

ReaR – Recovery Example (continued) Restoring the bootloader (SuSE style) Creating swap files and partitions OK Finished recovering your system You can explore it under /mnt/local Thank you for using Relax & Recover Version 1.7.23.2 Finished in 150 seconds. 27 © Novell, Inc. All rights reserved.

ReaR – Vision • Do more than just disaster recovery – System cloning and provisioning – System migration (P2V, V2P, V2V, P2P) – Generic system tool-kit for offline tasks and rescue media • Centralized information and management: – rear-server – 1.0: Gather and aggregate information about ReaR > System protection status > Rescue media boot files – 2.0: Centrally manage configuration and control ReaR > Site-wide and system-local configuration store > Trigger and control disaster recovery tasks – 3.0: Integrate into 3rd party solutions 28 © Novell, Inc. All rights reserved.

Secure Disaster Recovery Enterprise Environments

Enterprise DR – Requirements Standardization Scalability Automation Integration in existing system environment Security Coverage over all systems in use Support and maintenance Education and certification Affordable licensing – site license 30 © Novell, Inc. All rights reserved.

Enterprise DR – Relax and Recover • Standardization through global configuration: – /etc/rear/site.conf • Scalability and automation: – rear mkrescue can be run as a cron job – Automated processing of rescue boot media fairly simple – Rescue boot media also stored with backup software – LAN boot – extreme scalability and automated DR possible • Integration in existing system environment – Support internal and external backup solutions – E-mail reports 31 © Novell, Inc. All rights reserved.

Enterprise DR – Relax and Recover • Security considerations – No “secret” data in rescue media – Secure export of rescue media through backup software and E-mail – Remote access to rescue system only with SSH keys • Coverage – i386 and x86_64 stable – Enterprise Linux (Novell , Red Hat, Ubuntu) stable ® – Community distributions also work, less tested • Maintenance and support, education – Please contact Novell and the authors • Site license: Included with GPL 32 © Novell, Inc. All rights reserved.

Security Aspects – Disaster Recovery Backup server DR boot server DR client Transfer of Backup Data Transfer of Boot Riles • How secure is the • Insecure (anonymous): backup software? – NFS • Insecure: NFS2, NFS3 – CIFS • Secure (with – SSH (keys) authentication): – RCP – NFS4, CIFS • Manipulation proof: – SSH (rsync) – SMTP 33 © Novell, Inc. All rights reserved.

Enterprise Linux Disaster Recovery • Linux – Simple Disaster Recovery • Integration: Disaster recovery – backup software • Automated creation of disaster recovery information • Network boot into rescue system • Disaster recovery becomes part of infrastructure • Scales well even for very large data centers • Disaster recovery becomes a valid option for clients, not only for servers 34 © Novell, Inc. All rights reserved.

SUSE Linux Enterprise High Availability ® Extension 11 Service Pack 11 – ReaR Availability and Support

SUSE Linux Enterprise ® Modular Portfolio SUSE Linux Enterprise Common Code Level 3 Support Runs on x86, Base foundation x86_64, Itanium, Power, and s390x SUSE Linux SLES for SAP, Enterprise SLED JeOS SLE SDK SLE POS Expanded Server Support Server Desktop Custom Developer Bundles Support OS OS OS Tools SKUs High Availability Real Time Extension Mono Extension Optional extensions for high availability clustering, Pay only for deterministic computing, and running .NET applications what you need 36 © Novell, Inc. All rights reserved.

SUSE Linux Enterprise High Availability Extension 11 ® Key Features • Flexible, policy driven • Continuous data clustering solution replication – OpenAIS messaging and – Distributed Replicated Block membership layer Device (DRBD8) – Pacemaker cluster resource manager • GUI and CLI tools – Graphical user interface • Cluster-aware file system – Unified command line interface and volume manager – Oracle Cluster File • IP load balancing System (OCFS2) – Clustered Logical Volume • Resource agents Manager (cLVM2) • Virtualization aware • Bare Metal Recovery (ReaR) 37 © Novell, Inc. All rights reserved.

SUSE Linux Enterprise High Availability Extension 11 ® Differentiators • Affordable Costs less than other Linux clustering solutions, with tools and resource agents included at no extra charge • Virtualization-aware Supports hybrid virtual and physical clusters • Open Source No proprietary modules or user space applications means your Linux clustering solution is secure and easily kept up to date • Integrated Streamlined support and an optimized solution from a single vendor that delivers both the OS and clustering software 38 © Novell, Inc. All rights reserved.

SUSE Linux Enterprise High Availability Extension 11 ® New Features/Enhancements SP1 ReaR – Relax and Recover • ReaR will be included with SUSE Linux Entreprise Server 11 SP1 HA Add-on • ReaR is considered to be an extensible framework, allowing for custom scripting and extension of the recovery toolset • ReaR RPM packages will be fully supported (L1, L2, L3) • SLE11 High Availability Add-on inherits service level agreement from the SLES Subscription – Priority (24x7) – Standard (12x5) • Customization is not covered by Novell Technical Support ® – Integration Partners are available (contact Novell Sales) 39 © Novell, Inc. All rights reserved. •

SUSE Linux Enterprise High Availability Extension 11 ® More Information - SP1 For more information on what is new with SLE11 HA SP1: ELS307 Building High Availability Clusters with SUSE Linux Enterprise High Availability Extension ELS302 SUSE Linux Enterprise High Availability Extension 11: Support and Troubleshooting ATT307 Quickly Configure a High Availability Cluster with SUSE Linux Enterprise 11 High Availability Extension 40 © Novell, Inc. All rights reserved.

Live Demo/Use Cases

Relax and Recover – Live Demo Demo (~10 min): • SUSE Linux ® Enterprise High Availability Extension 11 SP1 • ReaR creates backup and boot CD • “PEBKAC” at work • Boot system from CD • Recover system • Happy End 42 © Novell, Inc. All rights reserved.

Questions and Answers

Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

Linux Disaster Recovery Made Easy

by Novell

Accessibility

Upload Details

Usage Rights

1 Embed 31

Statistics

Linux Disaster Recovery Made Easy Presentation Transcript