EMC IT‘S ORACLE BACKUP ANDRECOVERY—4X CHEAPER, 8XFASTER, AND 10X BETTER—SYMMETRIX VMAX, DATA DOMAIN,AND NETWORKERDATA DOMA...
Copyright © 2012 EMC Corporation. All Rights Reserved.    EMC believes the information in this publication is accurate as ...
TABLE OF CONTENTS    EXECUTIVE SUMMARY ......................................................................................
EXECUTIVE SUMMARY    EMC IT has seen explosive growth over the last five years accelerating the need to move from its lega...
EMC IT’S BACKUP AND RECOVERY JOURNEY    This section covers the drivers and the size of the journey, with respect to EMC‘s...
Oracle Recovery Manager (RMAN)    Oracle RMAN is a built-in tool that allows the database administrator (DBA) to easily ba...
Proxy Host    A Proxy Host is a server used to mount a cloned copy of a database for backup purposes. It is primarily used...
Legacy Backup and Recovery Pain Points    The legacy Backup and Recovery infrastructure, described above, created some ―pa...
EMC Data Domain Deployment Models    EMC Data Domain systems can be deployed as a NAS device or as a VTL:    NAS deploymen...
GDW/CRM BACKUP AND RECOVERY COMPONENTS     The following illustrate the new EMC IT Backup and Recovery infrastructure comp...
EMC IT’S ORACLE BACKUP AND RECOVERY METHODS     EMC IT’s Offloaded Backup and Recovery Process for Oracle Databases     Th...
   The deduplication benefits of Data Domain allow many more images of full backups to be stored with minimal storage    ...
EMC IT’S OFFLOADED BACKUP EXAMPLES     Global Data Warehouse (GDW) Backup Size     EMC‘s Global Data Warehouse (GDW) is a ...
EMC TimeFinder Snapshot     Another option is to use EMC TimeFinder snapshots.     A snapshot is a point-in time "virtual"...
ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN     DEPLOYMENT     The following are the advantages the EMC IT gained...
EMC IT LESSON LEARNED     As with all functions in IT, following is a set of best practices that will yield better perform...
CONCLUSION     EMC IT implemented a Backup and Recovery solution to deal with the growing size of its Oracle environments:...
References     EMC Symmetrix VMAX     http://www.emc.com/storage/symmetrix-vmax/symmetrix-vmax.htm     EMC TimeFinder     ...
Upcoming SlideShare
Loading in …5
×

White Paper - EMC IT's Oracle Backup and Recovery-4X Cheaper, 8X Faster, and 10X Better-Symmetrix VMAX, Data Domain, and NetWorker

921 views
792 views

Published on

Migrating from a legacy availability infrastructure for Backup and Recovery
creates challenges in terms of what are the best practices for a new Backup and
Recovery deployment with EMC‘s Oracle databases for Global Data Warehouse
and mission-critical Oracle applications. This white paper will illustrate the
transformation of EMC IT Oracle Backup and Recovery Infrastructure and
highlight how the Data Domain appliance transforms EMC IT Oracle Backup
infrastructure

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
921
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

White Paper - EMC IT's Oracle Backup and Recovery-4X Cheaper, 8X Faster, and 10X Better-Symmetrix VMAX, Data Domain, and NetWorker

  1. 1. EMC IT‘S ORACLE BACKUP ANDRECOVERY—4X CHEAPER, 8XFASTER, AND 10X BETTER—SYMMETRIX VMAX, DATA DOMAIN,AND NETWORKERDATA DOMAIN TRANSFORMS EMC IT‘S ORACLEBACKUP AND RECOVERY INFRASTRUCTURE ABSTRACT Migrating from a legacy availability infrastructure for Backup and Recovery creates challenges in terms of what are the best practices for a new Backup and Recovery deployment with EMC‘s Oracle databases for Global Data Warehouse and mission-critical Oracle applications. This white paper will illustrate the transformation of EMC IT Oracle Backup and Recovery Infrastructure and highlight how the Data Domain appliance transforms EMC IT Oracle Backup infrastructure. August 2012WHITE PAPER1
  2. 2. Copyright © 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided ―as is.‖ EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. EMC2, EMC, the EMC logo, and the RSA logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. VMware is a registered trademark of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. © Copyright 2012 EMC Corporation. All rights reserved. Published in the USA. 08/12 White Paper H109862
  3. 3. TABLE OF CONTENTS EXECUTIVE SUMMARY ........................................................................................................................... 4 Audience ............................................................................................................................................ 4 EMC IT’S BACKUP AND RECOVERY JOURNEY ......................................................................................... 5 EMC IT Overview ................................................................................................................................ 5 EMC IT‘s Backup and Recovery business drivers ...................................................................................... 5 EMC IT‘s Backup and Recovery Legacy Architecture................................................................................. 5 EMC Backup Profile – Legacy EMC IT VTL infrastructure...................................................................... 5 Legacy Backup and Recovery Components........................................................................................ 5 Legacy Backup and Recovery Pain Points ............................................................................................... 8 EMC IT‘s New Backup and Recovery—Data Domain Infrastructure ............................................................. 8 Components ................................................................................................................................. 8 EMC Data Domain Deployment Models ............................................................................................. 9 GDW/CRM BACKUP AND RECOVERY COMPONENTS.............................................................................. 10 Backup and Recovery enablers ............................................................................................................ 10 EMC IT’S ORACLE BACKUP AND RECOVERY METHODS ......................................................................... 11 EMC IT‘s Offloaded Backup and Recovery Process for Oracle Databases .................................................... 11 Step 1- TimeFinder Clone .............................................................................................................. 11 Step 2- Clone Mounted on Proxy/Backup Host .................................................................................. 11 Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain .................................. 11 EMC IT‘s Regular Backup Process ......................................................................................................... 12 EMC IT’S OFFLOADED BACKUP EXAMPLES ........................................................................................... 13 Global Data Warehouse (GDW) Backup Size .......................................................................................... 13 GDW Pain Points ................................................................................................................................ 13 Oracle CRM Backup and Recovery Problem Statement ............................................................................ 13 Advantages of Proxy Host Solution ....................................................................................................... 13 EMC TimeFinder Clone .................................................................................................................. 13 EMC TimeFinder Snapshot ............................................................................................................. 14 ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN DEPLOYMENT ............................... 15 Reuse of the legacy EMC IT Backup and Recovery process ...................................................................... 15 EMC Data Domain – Deduplication technology ....................................................................................... 15 Deduplication Benefits................................................................................................................... 15 Full Backups versus Incremental .................................................................................................... 15 ROI/TCO ........................................................................................................................................... 15 4X Cheaper.................................................................................................................................. 15 10X Better ................................................................................................................................... 15 EMC IT LESSON LEARNED .................................................................................................................... 16 Bottlenecks ....................................................................................................................................... 16 Disks .......................................................................................................................................... 16 Proxy Server: 8X Faster ............................................................................................................... 16 EMC NetWorker Server .................................................................................................................. 16 Utilization versus Vulnerability ....................................................................................................... 16 CONCLUSION ....................................................................................................................................... 17 References ........................................................................................................................................ 18 Acknowledgments .............................................................................................................................. 183
  4. 4. EXECUTIVE SUMMARY EMC IT has seen explosive growth over the last five years accelerating the need to move from its legacy, Virtual Tape Library (VTL) Backup infrastructure to a new EMC Data Domain Backup infrastructure. This creates challenges in terms of what are the best practices for a new Backup and Recovery deployment with EMC‘s Oracle databases for the Global Data Warehouse and mission-critical Oracle applications. EMC IT implemented a phased approach because of the size of the Oracle environments:  Over 670 Oracle databases in all environments: o Production o Test/Development o QA o Performance o Patch  Greater than 30 TB of Oracle Redo Logs generated per day  Mission-Critical environments (databases) backed up daily Moving to the new EMC IT Data Domain infrastructure, for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and Oracle CRM production environments has delivered the following advantages:  4X Cheaper - The Data Domain appliances are a quarter the cost of the legacy EDL/VTL  8X Faster – Move from an incumbent EDL/VTL speed of 500 MB/hour to Data Domain speed of 4 TB/hour  10X Better – Based on the following: o Reliability - More reliable than tapes – guaranteed that backups can be restored. o Density - Stores more backups for a longer period of time, even old backups can be quickly restored. o Protection – Production Data Domain appliances are replicated to a remote Data Domain appliance off-site. o Speed – Both backups and restores are significantly faster. Backups are now completed within the Service Level Agreement (SLA) with fewer support resources. o Complexity – Standard full backups are much easier to execute and restore than incremental backups. o Cost – Data Domain units are one-fourth the cost with up 10 times the capacity, after deduplication. This white paper will illustrate the journey and benefits of EMC IT‘s Oracle Backup and Recovery Infrastructure and highlight how the Data Domain appliance transformed EMC IT Oracle Backup architecture. It will also include lessons learned from EMC IT on this migration from a legacy VTL infrastructure to a Data Domain infrastructure. Audience This white paper is intended for CIOs, Oracle architects, Backup and Recovery architects, storage architects, Oracle Database Administrators (DBAs), and server and network administrators.4
  5. 5. EMC IT’S BACKUP AND RECOVERY JOURNEY This section covers the drivers and the size of the journey, with respect to EMC‘s Backup and Recovery profile. It details the legacy Backup and Recovery infrastructure, as well as the new EMC IT Backup and Recovery Data Domain infrastructure deployment models. EMC IT Overview EMC is a company with over 53,000 users of IT services. It supports over 400,000 customers and partners in 5 Data Centers with over 16 PB of storage. EMC IT has a portfolio with over 500 business applications and tools and over 8000 OS images with more than 90 percent of all servers virtualized in 80 countries and 20 languages. EMC IT’s Backup and Recovery business drivers The following are the business drivers for EMC IT‘s Backup and Recovery:  Inability to meet EMC‘s backup or restore SLAs with the legacy EMC VTL infrastructure o Erratic backup windows, some spanning over 24 hours o Daily ―hand-holding‖/‖fire fighting‖ of the backup process by EMC IT Backup and Oracle SMEs o Limitations on the number of backup images stored on the legacy EMC VTL infrastructure EMC IT’s Backup and Recovery Legacy Architecture This section describes EMC‘s Backup and Recovery profile as well as the legacy Backup and Recovery components. EMC Backup Profile – Legacy EMC IT VTL infrastructure The following is the current EMC IT Oracle Backup and Recovery profile:  Over 670 Oracle databases  Over 100 Terabytes of Oracle Databases backed up per day  Mission-Critical production environments backed up daily FULL  Mission-Critical Dev and Test environments backed up every two days FULL  Non Mission Critical and dev/test environments backed up FULL 2X per week  Archive log backups are run daily and when triggered by space alerts Legacy Backup and Recovery Components The following are the baseline Backup and Recovery components used in the legacy (VTL) infrastructure: EMC NetWorker NetWorker delivers complete Oracle protection through the NetWorker Module for Oracle. NetWorker integrates with Oracle Recovery Manager (RMAN)—Oracle‘s recommended Backup and Recovery interface for its databases. Through this integration, NetWorker simplifies Backup and Recovery for Oracle and is able to integrate Oracle server protection into several core features, including wizard-driven configuration and push installation. The EMC NetWorker Module for Oracle has value-added features available to reduce backup impact and required storage. This includes integrated deduplication and snapshot management functionality. Additionally, an exclusive feature called event-based backup helps deliver more effective backup operations based on probing for conditions rather than starting merely by time. To gain the benefits of consolidation and flexible availability, the world is moving increasingly to operation in virtual server environments. NetWorker, and its Oracle support, integrate with VMware, delivering in guest, application-consistent protection and support for environments that leverage VMware technologies such as Distributed Resource Scheduler, vMotion, and High Availability that move and migrate virtual machines in a VMware environment. Another benefit of NetWorker is that it is tested, certified and supported in a broad range of operating systems and environments.5
  6. 6. Oracle Recovery Manager (RMAN) Oracle RMAN is a built-in tool that allows the database administrator (DBA) to easily backup and recovery data in an Oracle database. RMAN handles the coordination required to ensure that transaction integrity is preserved, and sufficient information is maintained to recover the database to any appropriate point. RMAN can create backup sets that comprise as much or as little recovery information as the DBA requires but usually include information from the database datafiles, control files, redo files and archived log files. EMC NetWorker/RMAN Integration NetWorker delivers full integration and is fully compliant with Oracle RMAN. Through this integration, NetWorker delivers complete Backup and Recovery for Oracle databases, including the ability to capture and protect databases, individual datafiles, tablespaces, and logs—all of the things that comprise an Oracle installation. NetWorker, through the RMAN integration, is able to provide backups for Oracle whether the database is online or offline. Other highlights include the ability to deliver block-level incremental backups, which helps to ensure backups are fast and capturing the unique changes to the data. Other RMAN features include block corruption detection and surgical recovery of any detected corrupted blocks to help ensure data is accurate and recoverable. The solution also compresses any unused blocks to save time, space, and bandwidth. In addition, RMAN delivers restartable backups from the point of failure for interrupted backups—very helpful, especially for very large databases. Through Oracle‘s Data Recovery Advisor, any issues encountered and diagnosed are automatically repaired. All of these capabilities help ensure recovery, which is the primary goal. By providing automatic log management, NetWorker is able to bring the database back to a point-in-time, thereby restoring all data and replaying any needed logs to facilitate a complete restore. Finally, not only can you restore back to where the data came from, but also in cases where there is other or new hardware in place, NetWorker will restore the data to an alternate location. EMC Disk Library (EDL)/Virtual Tape Library (VTL) A virtual tape library is a disk based storage array front-ended by servers that emulate tape drive systems. Disk based VTLs have significant advantages over their tape counterparts as they dramatically improve throughput speeds. This is even more significant when it comes time for a restore. Since the backups already exist on disk, restores do not suffer from the mechanical limitations of either the tape drive streaming speeds or the robotic arms or even the fast forward and rewind actions associated with finding the backup set. The following is a high-level illustration of the legacy Backup and Recovery infrastructure: NetWorker Virtual Tape Library Database Server Figure 1. Legacy Backup and Recovery infrastructure with VTL6
  7. 7. Proxy Host A Proxy Host is a server used to mount a cloned copy of a database for backup purposes. It is primarily used to offload the backup overhead from the production database. It can be used by many databases and has the advantage that the VTL can be directly mounted to a single Proxy Host, giving significantly more throughput with SAN protocols, vs. backups over NFS. This is much more important when performing partial restores, as the database can be rapidly restored to the Proxy Host and a partial, or surgical restore can be performed very rapidly. Of course, if the data required is from the most recent backup, the database does not even need to be restored from the VTL, as it still exists on the Proxy Host. NetWorker Proxy Host Virtual Tape Library Database Server Figure 2. Legacy Backup and Recovery infrastructure with Proxy Host The configuration of the proxy server does not require significant CPU resources. The focus of the server is strictly for IO throughput. For this reason, EMC IT selects a server with a large number of expansion slots. As an example, for EMC IT‘s CRM environment the proxy server configuration is shown in Table 1. Quantity Hardware 1 Cisco UCS C460 M2 - 128GB RAM - 2 x 73GB HDD 8GB DDR3-1333MHz 2 10 GB Network Card - Bonded 4 lpe1200E dual-port 8GB HBA – 4 ports zoned for VMAX, 4 ports zoned to VTL device SOFTWARE Oracle RDBMS – single instance RedHat 5.6 ASM Table 1. CRM Proxy Host configuration TimeFinder Clone EMC TimeFinder is an EMC replication technology that is used by EMC Symmetrix VMAX storage arrays to instantly create an exact copy of a set of LUNS. For a database, these LUNS would represent the database files for building a standby or backup copy and optionally include the redo files, for creating a reporting copy. This capability allows the backup or clone of a database without putting the database in a ―HOT‖ backup mode.7
  8. 8. Legacy Backup and Recovery Pain Points The legacy Backup and Recovery infrastructure, described above, created some ―pain points‖ for EMC IT. The following are the two that most impacted EMC IT: Limited retention for old backups Like all organizations, EMC IT has a need for retaining backups for extended periods. This generally consists of writing backups off to tape and sending them to an offsite storage facility. Given the fragility of tape backup media, EMC IT could never be 100 percent sure that it was possible to restore these tapes and the cost of limited use media was expensive to maintain. By placing these EDLs at a data center nearby, EMC IT was solving the offsite requirement, but at a cost of backup throughput. Storage needed (large overhead) With the introduction of EDL devices, EMC‘s version of a VTL, EMC IT quickly adopted them and began deploying the backups to these, disk-backed, virtual tape libraries. It did not take long to outgrow these EDLs, as the amount of disk required to retain EMC IT‘s long-term backups was massive. This required EMC IT to continue to rely on tapes for long-term backup retention. Fortunately, the introduction of deduplication technology and devices like Data Domain‘s VTL, Virtual Tape Library helped address this challenge. EMC IT’s New Backup and Recovery—Data Domain Infrastructure This section describes the legacy Backup and Recovery tools that are re-used in the new Backup and Recovery Data Domain infrastructure. This includes: the Data Domain appliance Backup and Recovery; the deployment model needed for the EMC IT‘s shared GDW/CRM Backup and Recovery process; and components for their Oracle Database Proxy Host Backup and Recovery process. Components The new Backup and Recovery infrastructure consists of EMC IT‘s current Backup and Recovery tools:  EMC NetWorker - 7.6.3  Oracle RMAN  EMC NetWorker Module for Oracle (NMO) – 5.0  Proxy Hosts  EMC TimeFinder Clones The major difference, between the new infrastructure and the legacy infrastructure, is the destination of the backups. In the new infrastructure, EMC IT writes to the Data Domain appliances. The Data Domain appliances, though the deduplication feature, allows EMC IT to store many more backups in a fraction of the space previously required. This capability has eliminated the need for the EDL/VTL and more importantly the need for any fragile tapes. For the offsite requirements, EMC IT uses Data Domain replication to replicate the backups to a secondary datacenter 600 miles away, completely eliminating the need for any tapes. Data Domain EMC Data Domain deduplication storage systems are designed and optimized, specifically for backup and archive of data.  Support for any conventional backup or archive application through generalized support for network-attached storage (NAS) interfaces over Ethernet; a virtual tape library (VTL) interface option over Fibre Channel; and product-specific interfaces such as NetBackup OpenStorage and EMC Data Domain Boost  High-speed, inline deduplication using small, variable-sized sequences to identify and eliminate redundant data segments before storing to disk  Integrated data protection technologies such as RAID 6, post-backup data verification, and periodic validation checks of existing data sets  Automated replication of backup data for disaster recovery (DR) using cost-effective, low-bandwidth WAN links, which enables faster ―time-to-DR‖ readiness8
  9. 9. EMC Data Domain Deployment Models EMC Data Domain systems can be deployed as a NAS device or as a VTL: NAS deployment When the Data Domain system is deployed as NAS, RMAN can write backups directly to it as a disk backup. This can be extremely useful for environments that have only a few databases to backup. As the number of databases grows, the management all of these disparate mounts can be overwhelming. For larger organizations, integration with an enterprise backup solution, such as EMC NetWorker, is recommended. This integration provides many benefits, such as a centralized location for managing and cataloging backups. EMC NetWorker, along with many of the other solutions, also includes technology that can significantly improve backup throughputs. Data Domain Database Server Figure 3. Data Domain deployed as NFS mount to Host VTL deployment When the Data Domain system is deployed as a VTL, RMAN must be integrated with an enterprise backup application like EMC NetWorker. This is because, like all VTLs, Data Domain systems operate with VTL or Open Storage protocols and RMAN is not capable of interfacing with them directly. With a VTL deployment, Data Domain systems can be connected though a 1 GB or 10 GB network typically dedicated to backup traffic. They can also be connected directly to database server HBA ports achieving much higher rates of throughput. NetWorker Data Domain Database Server Figure 4. Data Domain deployed as a VTL device9
  10. 10. GDW/CRM BACKUP AND RECOVERY COMPONENTS The following illustrate the new EMC IT Backup and Recovery infrastructure components, when offloading backups to a Proxy Host: Figure 5. Oracle Database Proxy Host Backup and Recovery Process As the diagram illustrates, EMC IT deploys a shared Backup and Recovery infrastructure using the following components and best practices:  VMAX TimeFinder Clones  Deployment of a Proxy Host or EMC NetWorker Storage node o Oracle database – single instance (backup license) o EMC NetWorker Module for Oracle o RedHat AS 5.6  Data Domain appliance deployed as a VTL device – DD890 Backup and Recovery enablers The following technologies enable EMC IT‘s Oracle Backup and Recovery today:  8 Gigabit Storage Area Network  10 Gigabit dedicated Local Area Network  EMC Data Domain 890 Appliances  EMC NetWorker – Version 7.6.3  EMC NetWorker Module for Oracle (NMO) – Version 5.0  Oracle RMAN – 11.2  EMC VMAX TimeFinder Clone  Proxy Clone Hosts10
  11. 11. EMC IT’S ORACLE BACKUP AND RECOVERY METHODS EMC IT’s Offloaded Backup and Recovery Process for Oracle Databases The following are the high-level steps for the offloaded Backup and Recovery Proxy Host backup process: 1. Create a TimeFinder Clone 2. Include the redo logs so that the database can be verified prior to the backup. This is not necessary, but adds an extra level of confidence that the database backup is good. It also allows the database to be opened, read-only for investigation, reporting or extract purposes. 3. Mount the Clone on the Proxy Host server 4. Invoke the NetWorker/RMAN scrip to backup the TimeFinder Clone to Data Domain infrastructure Step 1- TimeFinder Clone EMC IT uses EMC TimeFinder/Clone Technology as the first step in the backup process. The Clone is a full size point-in-time replica of the source database. Once the Clone is originally created, subsequent ―recreate‖ operations only apply the incremental changes made to the source since the last ―recreate‖ of the Clone. This reduces the first step of the backup process. EMC IT‘s daily clone typically takes 40 minutes for a 20TB database. After 40 minutes, the database has a viable Backup and Recovery solution. This backup, is now available for surgical or complete restores. Step 2- Clone Mounted on Proxy/Backup Host EMC IT uses a Proxy Host, or backup host, to offload the RMAN backup to a virtual tape that is the Data Domain appliance. With a Proxy Host, a separate host is used, configured with Oracle Binaries and ASM installed. The ASM diskgroups on the Proxy Host are made up of the source database‘s cloned devices. Deploying a backup host has many benefits:  Allows full backup to run on the Proxy Host producing no performance impact to the production database and production user community  Enables surgical restores from the Proxy Host from the most recent backup or from a restored backup from any point in time. Having a point-in-time restore on a separate host allows individual tables or tablespaces to be restored using export/import or transportable tablespace or even database links for very specific requirements.  Enables ad-hoc querying for end-users. Queries can run from the clone for ad-hoc reporting – allowing full autonomy for reports running from the clone. This can stop less efficient or resource intensive reports from interfering with the source database (Production).  Reverse Clone – For databases that require a quick and efficient RTO (Recovery Time Objective) the Reverse Clone allows super quick restores back to the source.  Offloaded Stats Generation – before the next backup run, the database can be opened and stats generated on the cloned version of the database. This allows for complete level of generation, with no impact to the production system. Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain Once the Clone operation is complete, the Clone is opened as a read only database, thus verifying a valid copy, and backed up to Data Domain using RMAN/NetWorker. EMC IT re-used the same legacy scripts that were already in place when EMC IT moved from the EDL to the Data Domain appliance deployed as a VTL. The only additional consideration was to change the ―filesperset‖ setting, which was changed to 1. This contributes to the most favorable compression ratio and efficiently stores subsequent full backs to the Data Domain with a very small storage footprint. Using the Data Domain appliance to store backups has enabled many advantages including:  Virtually no change to existing backup scripts, RMAN catalogs or pre-determined scheduling – the only consideration or potential change being a change to the filesperset parameter.  Integrates easily with EMC NetWorker infrastructure.11
  12. 12.  The deduplication benefits of Data Domain allow many more images of full backups to be stored with minimal storage overhead. This is especially significant for databases that carry a lot of unchanged history as the bulk of the data, which is the case in all data warehouses and most OLTP databases.  Data Domain enables EMC IT to eliminate the need for a traditional Oracle incremental backup strategy. This simplifies the restore steps and shortens the recovery time window. RMAN Script Example Here is an RMAN script excerpt from EMC IT‘s backup script used to backup EMC IT‘s offloaded databases. The script uses standard Oracle RMAN commands: connect rcvcat DatabaseA/xxxxxx@rmancatolog connect target xxxxxxx/xxxxxx@DatabaseA run { set command id to rman database; allocate channel t1 type SBT_TAPE format %d_full_%U; allocate channel t2 type SBT_TAPE format %d_full_%U; allocate channel t3 type SBT_TAPE format %d_full_%U; …. (EMC IT uses 24 channels) ….. send NSR_ENV=(NSR_DATA_VOLUME_POOL=DBORAP); backup tag = ‗DatabaseAbkp_hot_0209_214224 filesperset 1 database; backup current controlfile channel t1 format %d_cf_%U; release channel t1; release channel t1; release channel t1; …… } Table 2. EMC IT legacy RMAN script example EMC IT’s Regular Backup Process For the majority of EMC‘s databases offloading to a proxy server is not required. For those databases, where speed is not an absolute necessity and the overhead of an RMAN backup directly on the database server does cause performance issues, EMC IT will not offload the backup to a proxy server. In this case, EMC IT executes the backup directly from the database server. Backups are scheduled every other day as full database backups to the same Data Domain appliances as the offloaded backups. Since the deduplication benefits are the same, as offloaded backups, there is no need to do incremental backups, which are impacted by the extra complexity, and restore times associated with incremental backups. For EMC IT‘s larger databases, using this regular backup process EMC IT installs a dedicated network card to the database server. This additional network card in the database server reduces the overhead to the public network that a large backup would add, and assists with overall throughput. In the future, EMC IT is looking at Data Domain‘s Boost technology to both alleviate that overhead as well as reduce the backup window from deduplication at the source.12
  13. 13. EMC IT’S OFFLOADED BACKUP EXAMPLES Global Data Warehouse (GDW) Backup Size EMC‘s Global Data Warehouse (GDW) is a central source of relational data that provides business intelligence to many key business groups within EMC including Sales, Marketing, Finance, Manufacturing, Customer Service, Professional Services, Human Resources and Legal departments. It is a critical component of intellectual property and it‘s supported and maintained accordingly. One key and highly important support requirement is the ability to backup and provide realistic and flexible recovery options to the business groups that rely on this data. GDW runs on a six-node Linux-based Oracle Real Application (RAC) Cluster that went live in November of 2009, as a 10TB database. Since 2009, GDW has significantly grown and as of May of 2012 GDW has doubled to 20 TB with archive log generation totaling from 500 to over 1500 GB daily. GDW Pain Points In the summer of 2011, the time to complete an online RMAN/NetWorker backup of the GDW to an EMC Disk Library (EDL) degraded. The throughput was erratic and at times would exceed 24 hours, ―wrapping‖ over the scheduled start time of the daily TimeFinder Clone used as part of the backup solution. This undermined EMC IT‘s ability to successfully provide a backup of the database. There were periods where it took a week to successfully backup the database to EDL, as the scheduled clone would cause the online backup to fail. This resulted in daily handholding of backups and undermined the ability of IT to protect this valuable investment. The backup/recovery solution was in need of remediation and some redesign as: backups ran too long and would sometimes fail leaving data protection capability limited and vulnerable. Additional staff-hours were needed to ―fire-fight‖ daily challenges. A strict limitation to the number of full backup images that could be stored on the Disk Library, due to the large storage footprint each backup imposed. Archived log backups became increasingly important and the number of logs needed to satisfy recovery points became suspect. Oracle CRM Backup and Recovery Problem Statement EMC IT‘s Oracle 11i eBusiness Suite Database grew from 8 TB to 10 TB over a period of two years. During this two-year period, the backup time increased from 6 hours to over 22 hours, causing significant challenges in getting a full, successful, daily backup. This also meant that restores, to this environment, would take longer than 22 hours. These times were well beyond EMC IT‘s SLA for the databases Recovery Time Objective (RTO). This, over 3X increase in the backup window, was due to more than just growth in database size. It also was due to the increase in the number of databases that needed to backup to this shared infrastructure, and was guaranteed not to be solved on its own. As this was EMC IT‘s mission-critical database, with a zero data loss policy, 11i Database had to be backed up daily. The increase in backup time, violated EMC IT‘s restore SLAs as well. To resolve this challenge, EMC IT embarked on a new solution with EMC VMAX Clone and Data Domain. Advantages of Proxy Host Solution EMC IT uses a Proxy Host to run backups for its Oracle 11i eBusiness Suite database. The copy that is mounted to the Proxy Host is built using EMC TimeFinder clone technology. EMC TimeFinder Clone TimeFinder Clone provides single or multiple point-in-time copies of the full database within the same storage array. The cloned database is mounted on the Proxy Host for backup. A database clone requires the same disk space as the source database. Clones are extremely fast, as the data never leaves the storage array. The initial clone for a 20 TB database takes about 4 hours, with a low quality of service, QOS. EMC IT uses the low QOS to eliminate the impact to production. After the initial clone, all subsequent clones use a differential clone, only copying the changed tracks to the cloned devices. For example, EMC‘s GDW 20 TB database clone takes 40 minutes to complete. From the Proxy Host, backup of this 20 TB database to Data Domain appliance takes 4 hours.13
  14. 14. EMC TimeFinder Snapshot Another option is to use EMC TimeFinder snapshots. A snapshot is a point-in time "virtual" copy that occupies very little disk space. A snap requires only 10 to 20 percent of source database space for storing only changed tracks. In this case, backup will actually read most of the data from production disks. This is a very efficient method of offloading the backups, but does incur a small overhead to the production database, since the database and backup are competing for the same physical disks. A good way to minimize this is to use virtual pools in the storage array. This provides many more disks to spread out the I/O load. The full clone solution offloads the entire backup load from the production database server and doesn‘t impact performance for EMC IT‘s OLTP online users. This is especially useful when backup runs for a long time. Proxy Host copy also provides much better Recovery Time Objectives (RTO). In the event you need to restore a production database, restore from backup media will take hours. Since proxy copy is already on disk, it takes significantly less time to restore. You can do Reverse Clone to restore a production database and then apply missing archive logs. This method also provides some protection against accidental data corruption/loss due to human errors. For example, a few critical tables in the production database were dropped one hour ago due to human error. Application will work fine if you can restore only these tables. You cannot afford to restore complete production database and suffer 1-hour data loss. Instead, you can restore proxy database point in time just before table drop, and then copy tables back to production manually. Proxy copy is extremely useful in such surgical repair scenarios.14
  15. 15. ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN DEPLOYMENT The following are the advantages the EMC IT gained when deploying the new Backup and Recovery Data Domain infrastructure Reuse of the legacy EMC IT Backup and Recovery process The first gain was virtually no change to existing EMC IT Backup and Recovery processes such as backup scripts and RMAN catalogs. There was one consideration potential change to ―filesperset‖ as defined in the EMC white paper, Oracle RMAN Design Best Practices with EMC Data Domain. Another advantage was that the Data Domain infrastructure integrated seamless with the legacy EMC IT backup software, EMC NetWorker. EMC Data Domain – Deduplication technology Deduplication technology allowed EMC IT to move its long-term backups to the Data Domain infrastructure with complete confidence that EMC IT would be able recover those backups, if the need ever arose. It also solved the issue with off-site storage as these Data Domain devices have the ability to replicate all backups to a Data Domain unit at a remote site. For EMC IT, the replicated Data Domain unit is located at EMC IT‘s DR site, giving us additional protection. Deduplication Benefits Data Domains deduplication technology has enabled EMC IT is to retain many more backup images with minimal storage overhead. EMC IT can now restore backup images from several months past—without having to go through the difficult process of locating off-site tapes and hoping that they are still valid. Full Backups versus Incremental Due to Data Domain‘s ability to deduplicate the backups, EMC IT was able to use daily full backups versus having to do incremental backups. This allows a much quicker recovery process since incremental backups require restoring the full backup then restoring the incrementals on top of that. This not only reduces the complexity of the restore process, but it also reduces EMC IT‘s recovery window. ROI/TCO 4X Cheaper Typically, it is difficult to calculate actual dollars saved when utilizing shared infrastructure and the costs to administer an environment that is not meeting SLA‘s. In EMC IT‘s case, the actual cost of the Data Domain units EMC IT replacing its EDL with are one fourth the cost and between double and quadruple the capacity—depending on the actual de-dupe rates. In addition, to these savings, Data Domain technology is also increasing EMC IT‘s ability to retain backups longer, and completely eliminate the need for tapes. When factoring in all of these additional cost reductions, it is clear that overall costs greatly exceed the 4x cheaper claim. 10X Better The following is a mix of quantitative/qualitative measurements of ―Better‖ that include people, process, and technology: 1) Faster—8X faster with Proxy server deployment with 8GB FC deployment 2) Reliability—More reliable than tapes – guaranteed backup can be restored 3) Cost—Costs less for hardware and environments (power and cooling) 4) Density—Stores more backups creating a greater retention period 5) Protection—Replicated remote Data Domain appliance can be deployed off-site15
  16. 16. EMC IT LESSON LEARNED As with all functions in IT, following is a set of best practices that will yield better performance, scalability and stability. Bottlenecks Disks When configuring for high speed backups, there are many potential locations for bottlenecks. The most obvious of these is the disk layout, in the storage array. When it comes to I/O, the capability of the disks is most often the bottleneck. To give some idea, a 15K fibre channel drive is capable of driving between 47 and 53 GB/hour. This means that to support an 8TB/hour backup, you would need 155 of these disk drives. For the Data Domain Backup and Recovery environments, EMC IT does not need 8 TB/hour. This means EMC IT configured its disk pool with only 76 disks and is achieving 4 TB/hour. Proxy Server: 8X Faster Next up, in the stack, is the connectivity between the storage array and the proxy server. This was initially overlooked when EMC IT migrated from the EDL to the Data Domain. The Backup and Recovery infrastructure had 8 GB SAN switches, HBAs and ports on the SAN; however, the Backup and Recovery infrastructure initially connected it to the 2 GB SAN switch. Even after re-cabling to the 8 GB SAN, EMC IT still found the ports on the storage array had been configured for 4 GB. After fixing these issues, the new Backup and Recovery infrastructure effectively doubled the initial throughput of 2 TB/hour to 4 TB/hour. When compared to EMC IT‘s legacy EDL backup speeds, EMC IT improved its backup speeds by a factor of 8 from about 500 MB/hour on the legacy EMC DL4206 EDLs to about 4 TB/hour on the new Data Domain VTL EMC NetWorker Server After relieving that bottleneck, the new Backup and Recovery infrastructure began to have a problem with the NetWorker server. The NetWorker server that was deployed was already overloaded with backup requests and was not able to keep up the large number of backup sets that were now being requested to be processed. Keep in mind that EMC IT set files per set to 1, for best results in deduplication. Since this node was part of the original shared infrastructure and was clearly overloaded, configuring and deploying another NetWorker server relieved the bottleneck and improved overall backup performance throughout the data center. Utilization versus Vulnerability One of the constant challenges EMC IT faces with their backup infrastructure is how to maximize the utilization without creating vulnerability. In order to maximize the investment in the new backup infrastructure EMC IT moved from a dedicated model to shared model. This has allowed EMC IT‘s Backup and Recovery team to become much more efficient. However, there was no mechanism to measure the effect of consolidation on individual backup and restore times. This is what had lead EMC IT into the problem of backups not completing. It is extremely important to monitor backup times and investigate individual backups that start taking longer than was originally configured. An obvious example of this would be to use the proxy backup server for more than just one environment—but without coordinated backup windows, both backups could suddenly fall out of the configured times.16
  17. 17. CONCLUSION EMC IT implemented a Backup and Recovery solution to deal with the growing size of its Oracle environments:  Over 670 Oracle databases in all environments (Test/Development/QA/Performance/Patch/Production)  Over 30 TB of Oracle Redo Logs per day  Mission-critical environments (databases) backed up daily Migrating from a legacy EMC EDL/VTL solution to Data Domain brought the following advantages to EMC IT‘s Backup and Recovery deployment:  Moving away from limited Backup and Recovery backup retention to enabling of simple full backup to reduce the complexity of incremental restore process  Accommodates months of full Backup and Recovery backups enabled by the Data Domain appliance  Deployment of a Proxy host architect in the Backup and Recovery architect delivers the following: o Off load backup process off of production enabling EMC business users experience no disruption or performance issues on the CRM production server o Ability to reuse legacy Backup and Recovery processes/scripts  Implementation of a shared infrastructure for EMC IT‘s mission critical CRM and GDW environments  EMC TimeFinder technology, which was able to create database clones in minutes (18 TB in 40 minutes) off the production server and mount on the proxy server and backup in less than four hours EMC IT also learned valuable lessons from the EMC EDL/VTL to Data Domain Backup and Recovery journey:  Disks—To meet the throughput in a Data Domain the source database Array needs the correct number of drives. To drive to 8 TB/hour the array would need 155 drives  Proxy Server—Confirm the infrastructure cabled correctly and enough HBAs  NetWorker Server—Identify a near capacity NetWorker server and plan for enough nodes to meet your total Backup and Recovery community needs  Utilization versus Vulnerability—Monitor the backup and restore times in a shared deployment to increase the Backup and Recovery infrastructure utilization while diminishing vulnerability of long running backups , failed backed or long running restores or failed restores Moving to the new EMC IT Data Domain infrastructure for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and Oracle CRM production environments has enabled the following:  4X Cheaper—The Data Domain appliances are a quarter the cost  8X Faster—Move from a legacy EDL speed of 500 MB/hour to an EMC Data Domain speed of 4 TB/hour  10X Better—Based on the following: o Reliability—More reliable than tapes; guaranteed backup restoration o Density—Stores more backups for a longer period of time; even restores for old backups can restored quickly o Protection—Production Data Domain appliances are replicated to a remote Data Domain appliance offsite o Speed—Both backups and restores are significantly faster; backups are completed within the SLA with fewer support resources o Complexity—Standard full backups are much easier to execute and restore than incrementals o Cost—EMC Data Domain appliances are one-fourth of the cost with up 10 times the capacity, after deduplication17
  18. 18. References EMC Symmetrix VMAX http://www.emc.com/storage/symmetrix-vmax/symmetrix-vmax.htm EMC TimeFinder http://www.emc.com/storage/symmetrix-vmax/timefinder.htm EMC Data Domain http://www.emc.com/backup-and-recovery/data-domain/data-domain.htm EMC NetWorker http://www.emc.com/backup-and-recovery/networker/networker.htm IDC Study – Worldwide Purpose Built Backup Appliances http://www.emc.com/collateral/analyst-reports/11530-idc-ww-pbba-2011-2015-forecast.pdf EMC IT http://www.emc.com/microsites/emc-it-proven/index.htm Acknowledgments The author would like to thank the EMC IT teams for assistance in the creation of this white paper. EMC2, EMC, the EMC logo are registered trademarks or trademarks of EMC Corporation in the CONTACT US United States and other countries. VMware [add additional per above, if required] are registered trademarks or trademarks of VMware, Inc., in the United States and other jurisdictions. To learn more about how © Copyright 2012 EMC Corporation. All rights reserved. Published in the USA. 08/12 EMC products, services, and EMC White Paper H10986 solutions can help solve your EMC believes the information in this document is accurate as of its publication date. business and IT challenges, The information is subject to change without notice. contact your local representative or authorized reseller—or visit us at www.EMC.com.EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 In North America 1-866-464-7381www.EMC.com18

×