Category: Backup and Recovery Product Line: Oracle Database 1.45v 2013.04.10 YOU MOST PROBABLY DON T NEED RMAN CATALOG DATABASEYury Velikanov, The Pythian GroupABSTRACTThis paper discusses in details the cost and value of RMAN catalog database.TARGET AUDIENCEThis paper is targeted for experienced Oracle DBAs who supported and implemented RMAN based backup andrecovery procedures.EXECUTIVE SUMMARYLearner will be able to: Justify RMAN catalog database to system’s business owner Leverage additional RMAN catalog benefits Improve RMAN based backup proceduresBACKGROUNDThe title of this paper is purposely thought provoking. My intention is not to discourage you from using catalogdatabase, but rather encourage you to think about the cost and value it adds. In this paper I will discuss legitimatereasons why one should use an RMAN catalog database, if a catalog database is in use, then what additional value it canprovide to improve operations.Many DBAs continue to use RMAN catalog database simply because of historical reasons. Oracle’s general advice inearlier database versions was to use Catalog database. In fact most of us know that in first versions of a recoverymanager (RMAN) “catalog” was the default option. With newer versions, Oracle changed the default behavior to“nocatalog” and I think there have been some legitimate reasons for it.This paper will review: What costs are involved with running catalog database; What problems does it address and whatrisks does it help mitigate; What additional value (e.g. reporting, metadata backup and long storage etc) you can leveragefrom it. In addition it reviews special cases such as RMAN using MML (Media Management Layer or simply a tapesystem, some DBAs know it as SBT_TAPE) integration; RMAN in Disaster Recovery configuration; RMAN backupsbased on file system; Retention policy management and some others. Finally you will find practical and general hints onhow to use a catalog database at the end of this paper.I must warn you that this paper assumes that you have previous RMAN knowledge. In the ideal case you should be anOracle DBA for some time and have experience in backup and recovery. This paper doesn’t provide ready to usesolutions. The main purpose is to make a complete overview of why you may need or may not need an RMAN catalogdatabase. Solutions to be provided as additional work related to this paper.
Acknowledgments:I would like to say a huge thanks to all my social media friends, workmates at Pythian and other DBAs who contributedto this paper with their ideas and suggestions. Thanks a lot for the following people for investing their time and effortsto review and improve the paper: Steve Karam one of the first reviewers / editors Brian Pardy one of the first reviewers / editors Fuad Arshad John Piwowar Martin Bach Kamran Agayev Andrey Goryunov Sergey Kosourikhin Christine Kivi a significant proofreading contributorWhy RMAN catalog is an overhead? / COSTLet me start with not so obvious statement for many DBAs: RMAN catalog database is an overhead that needsjustification to exist. Unlike any general business related database it does not store application data. Therefore a businessowner of an IT environment may have a valid question: How much does this component cost me and what value itadds? We as IT representatives should have a very good answers, right? Most of DBAs that I have spoken to about itadmitted that they use a catalog database because it was been created before them or they followed Oracle’srecommendations without analyzing if it is necessary. Let us start with understanding how big an overhead the catalogdatabase is and how much does it cost?So, what costs are involved running an RMAN catalog database? By saying “cost” I do not mean just server hardwareresources1. In many cases hardware costs are not the main contributing factor (hardware is cheap today, right?). Time isa cost and “inexpensive” DBA time needs to be invested to maintain, troubleshoot, patch and update catalog database. Letus go through some typical activities associated with RMAN catalog database implementation and maintenance: IMPLEMENTATION COST: Catalog database and associated RMAN catalog schema need to be created o Partially we can mitigate this cost by using OEM database (if exists then we do not need to create additional database). It is common to see DBAs using OEM database for the catalog schema as it has the similar availability requirements and licensing implications. However this introduces dependency between OEM and RMAN catalog. This decision limits our maintenance windows and upgrade options to some extent. o If we do not have a database where to locate RMAN schema then we need to create a new one. Any new database creation involves planning and communication with friendly neighbour departments (system administrators, network team, backup team etc). This consumes time and resources. o In business critical environments we should think and ensure that catalog database is highly available. This adds to the implementation costs. o In some larger enterprise networks, DBAs sometimes decides to introduce a separate RMAN catalogs based on geographical locations to reduce risk of network issues impacting database backups. This increases the implementation and maintenance costs in proportion to the number of RMAN catalog databases. MAINTENANCE COST: A catalog database needs to be maintained. A DBA should apply patches and keep the database on the supported versions level. You may say it does not need too much attention. I do agree. However there is still some time a DBA spends planning, preparing and executing maintenance activities. As1 It is worth mentioning that that you do not need an Oracle licence to run RMAN database unless youcustomise it or use for other purpose that requires a licence. You may check with your Oracle sales representativesthough if you need a licence for RMAN database in DR configuration.
RMAN catalog database is just another Oracle database it requires some DBA’s attention regularly. Depending on the databases’ versions used in your network you may require to keep RMAN catalog on relatively fresh level2. If Oracle databases maintenance processes are well designed and automated in your organisation it helps reduce catalog database maintenance costs. Delaying RMAN catalog database upgrade to the latest version can save on the maintenance costs, as quite often there is no necessity to upgrade RMAN database as often as other business critical databases. DEPENDENCIES: RMAN Catalog database is typically implemented in organisations with several or even hundreds of databases. The more databases that use Catalog database, the less downtime that is available to maintain it. It can become very difficult to close RMAN database for maintenance resulting in additional planning time. DOWNTIME: Quite often, depending on the implementation, RMAN backups may fail if a catalog database is not available3. Any unplanned downtime of catalog database may trigger backup failures and therefore puts possible recovery at risk. AVAILABILITY (licensing): As catalog database becomes a critical resource we should ensure that it is highly available. We do not want to lose critical backup data, do we? The process could involve Data Guard implementation and other measures that adds to the total cost of catalog database. As the downtime should be minimal the implementation may need a bit more efforts. By the way, you may call your Oracle Sales representative to discuss if you need any additional licences in this case 4. TUNING: In the case of managing hundreds or thousands of databases the catalog database may require additional tuning efforts. However with the right approach (statistics, additional indexes etc) it can manage a lot of databases’ metadata. At the end of the day RMAN catalog is nothing but a schema with approximately 40 tables. Most of traditional optimisation methods apply to catalog database.Some may say that the RMAN catalog database maintenance costs are relatively low as it is nothing but an Oracledatabase. I do agree. In fact with each additional database that we manage within the catalog database, we reduce theoverall cost per database. Plus RMAN database unlike many business application databases does not generate too muchheadache5.However we do have a real alternative, right? Instead of having an additional database and associated costs, we may usean individual databases controlfiles. Question is: : when does the benefits of having a catalog database justify theadditional costs associated with it. If we have just one business database can we justify a catalog database or not? Howmany databases should an organisation have in order to justify the cost? How critical should a business database be tomitigate associated recoverability risks?Now that we have some ideas on what the costs consist of, let us have a look at the benefits. I would like to start withscenarios when there are technical dependences from catalog database and we must have it.Cases when catalog database is a MUSTThere are just a few technical reasons when we must have a catalog database and where a control file based RMANcatalog is not good enough. There are solutions for some and limited options for others. Let us discuss each of those: Disaster Recovery: When an organisation uses a Data Guard (standby database) for a business database backed up by RMAN and there are backups that takes place on both DR sides 6 then we must have a catalog database2 For example can’t use a 10.2 catalog database with a 18.104.22.168 production environment3 See “Use resync catalog” section for a discussion on how to reduce this risk4 If you need a license for a catalog database standby you may workaround it by synchronizing you RMANbackups with a RMAN catalogs on both DR sides. I mention it later in the paper.5 We do not change it as often as other databases/applications6 It would be my preferable option to reduce backup load on primary database and offsite backups
to synchronise backup information for both primary and disaster recovery sites. See the “Catalog & DR” section bellow for further discussion. I think there is no work around here. KEEP: There are requirements to use a RMAN KEEP command to keep a backup for longer than current retention policy and a controlfile (control_file_record_keep_time) allows. This way information about backups that we would like to keep will be stored in a catalog database until it expires. However long term backups need a special care and should be treated as archiving rather than as regular backups. We may also want to backup oracle database software that is associated with the backup and possibly operational system. In case of file system backups many of us feel more comfortable if one off backup’s files are moved (copied to other directory or volumes) rather than keeping those on the original location. As alternative we could create a backup on an alternative location and UNCATALOG those to exclude from the purge process. At the time of recovery you either use a control file created along with the backup or use CATALOG or CATALOG START WITH RMAN commands to register it in the current control file 7. In both cases KEEP command is not necessary or in other words there are alternative solutions and the KEEP command is a nice to have. In case of tapes (MML) backups, most often a retention policy is managed by MML layer (with no good visibility for DBA teams) and therefore RMAN KEEP command would not be very useful8. The most important point here would be to ensure that the backup is sent to the right MML retention pool. RMAN backup’s log file contains important recovery information (e.g. handle and pool names). Those are to be kept a safe place until possible recovery. However if you have a need for the KEEP command then you should use a catalog database. If you decide to use RMAN catalog for this reason please confirm that the MML solution used in your environments supports RMAN retention policy and it has a priority over MML side retention policies. CONTROL FILE SIZE: If the size of database control files (control_file_record_keep_time) does not allow storing as long a history as necessary for current retention policy, then you need a catalog database to ensure obsolete backups get deleted and you can restore from backups older than what is stored in the control file. In this scenario ability to purge obsolete backups is more important than a restore operation. If for example we use MML and it controls the backup data purging process, then it is not critical that a current controlfile “knows” about all backups available. We can always restore a controlfile we need using the right MML “handle”. The handle could be obtained from either a backup’s log file or recovering an older controlfile copy that the current controlfile is aware of9. However, based on my experience, if an organisation uses a file system (e.g. NFS) to store backups then it is rare when a retention policy is greater than a reasonable sized controlfile could handle (e.g. 30 days). If an organisation uses a tape solution, then most often the retention policy is managed by MML layer and therefore catalog does not play any role in a backup’s purging process. The controlfile size limitation could be a legitimate reason to introduce a catalog database, however as you can see there are still some methods to address some of the challenges without requiring a catalog database. DUPLICATE does not need a catalog database. Some of you may say that RMAN DUPLICATE from backups without target connection 10 requires a catalog database. As a matter of fact there is the following statement in the oracle 11GR2 documentation - “RMAN can perform backup-based duplication with or without either of the following connections: Target, Recovery catalog”. DUPLICATE does not require catalog database. You may want to review the “Creation Of Rman Duplicate Without Target And Recovery Catalog Connection. [ID 1113713.1]” note for additional details.7 There is a risk however that the current controlfile may be on the different version. This is why the suggestion is toinclude operation system and oracle home backups along with the database backup that you are planning to keep longerthan normal retention policy.8 With exception of recovery process. In this case we will need to restore “old” controlfile from tapes and then processwith regular recovery. However recovery from “old” backup may treated as exceptional case and therefore may not besignificant enough reason for a catalog database introduction. As an additional argument I would like to mention thatrestore from an “old” backup often involves more complex activities (e.g. OS and Oracle Home restore) to be treated asexception.9 This method could be time consuming as we may need to recover several controlfiles before getting to theright one. In this scenario storing backups’ log files may be more reasonable solution.10 This is a new 11G R2 feature
This leaves us with three legitimate technical reasons when you may need catalog database 11.If you found that one or a combination of the reasons for RMAN catalog database applies to your configuration andyou cannot use possible solutions suggested then you may read the next section to learn about possible additionalbenefits you can leverage from the catalog database. Others may find that the benefits are more valuable than associatedcosts and therefore leverage catalog database benefits.Benefits of using RMAN catalog / VALUEWe have discussed COSTs involved in creating and maintaining catalog database as well as reviewed cases where catalogdatabase is a MUST. It is time to focus on the VALUE added by catalog database. Let us start with a simple and easy toimplement optional feature. ADDITIONAL CONTROL: An additional, centralized control level for day to day backups. I like this idea very much. If we have a catalog database with hundreds if not thousands of databases registered and backed up on regular basis then a report could be introduced that would notify a DBA team’s manager (or backup DBA, oncall, etc) about problems with last night backups or just provide a summary for review. It would not replace proper error handling functionality with oncall DBA notification 12 built within the backup process. However, if a DBA manages hundreds of databases then he/she may miss some of the issues (I was there personally). This option introduces a “safety fuse”. It protects an organisation from possible human and technical errors. If I was a DBA manager, I would strongly consider this option so I may sleep soundly at night. MONITORING & REPORTING: Backups’ volumes and throughput day to day monitoring. One of the main Oracle DBA responsibilities is to ensure that database backups run well. Catalog database provides an easy and centralized day to day monitoring repository. o You may introduce several reports that would help you identify any changes in the backups patterns (e.g. backup network slowdown or backup volumes increase because of redo volumes increase etc) without connecting to each of possibly many database servers. o It also could be used as a troubleshooting tool if backup run time increases (comparing the backup volumes, throughput etc with previous executions). For example if there is an archived logs volume increases you may check TOP objects with most block changes or if the volumes are the same it could indicate a problem with backup network or MML layer. o As catalog database contains information about backups’ timing, volumes and compression rates you can use this information for tuning backups, changing parallel parameters and control the impact. o Some service based organisation may charge their clients (e.g. internal clients, projects etc) per backup GB. The catalog database could be used to measure and report the backup volume per database or set of databases, simplifying cost estimates. CAPACITY PLANNING: Depending on the retention policy used and backup cycles it could be challenging to estimate the total capacity necessary for all databases’ backups. As an example, if NFS is used as storage for backups and there are hourly archive logs, daily incremental and weekly full backups then total storage used by the database backup may change significantly 13 depending on the day of the week. Capacity planning may become even more challenging if backups are executed from multiple RAC nodes and there are several databases that use the same NFS mount for storing backups. A centralized catalog makes the space utilisation reporting and capacity planning easier. Please bear in mind that records about backups are removed from the catalog database at the time that backups are obsoleted and deleted based on the retention policy used. There is no difference from that perspective between a controlfile or database based catalog. Therefore for capacity11 If you disagree and know other reasons please get in touch with me (google my name). Be sure that I am quitehappy to be wrong and will include your reason and mention your name (if you ok with it)12 I strongly believe that oncall DBA must be paged about any backup related problem. As backup is one of themost critical tasks that a DBA manages and responsible for. Therefore he/she should be notified about any problemimmediately. Ok, ok there are some exceptions could be made. But those should be very well controlled.13 The maximum difference is equal to size of all weekly incremental and all archive log backup volumes.
planning you may want to introduce an additional (custom) set of tables to store day to day measurements for long term capacity planning. MML: MML backups tend to be stored longer than file system backups, therefore there is a tendency to use an RMAN catalog in such configurations. However, quite often MML layer has its own retention policy and RMAN is rarely used to manage the backup retention policy. Therefore, if we ensure that we can restore a controlfile from MML (e.g., using MML repository or backup log files) we may not need an RMAN catalog database. See the “Catalog & MML integration” section for additional details on this topic. RMAN SCRIPTS: A catalog database allows us to store RMAN scripts so they can be reused to backup databases in a standard way across an enterprise. This allows the use of unified backup scripts across many (if not all) databases. The scripts are pulled out from the catalog database each time a backup is performed. Along with RMAN default configuration settings stored on a catalog database, this makes it easy to control and adjust backup scripts without connecting to each server and adjusting every single backup script. o Cons: Backups become highly dependent on the catalog database. If it is not available then ALL backups fail. Alternatively we can make local backups less dependent on the catalog by using local shell scripts, and even if the catalog database is not available backups still run (synchronizing backup metadata later on at the time the catalog becomes available). EASY CONTROLFILE RECOVERY: Independent of the configuration you use, catalog database makes controlfile recovery very easy and a standard procedure across all Oracle environments. In most cases, you just run a single command to recover a controlfile. However you only will need this in the scenario where there is complete loss of the current controlfile.This concludes my list of additional value added options. Now it is up to you to put things together and present it toyour system business owner. If you have used an RMAN catalog database before reading this paper you may considerimplementing some of the ideas I listed above in order to streamline your operations and get more value from theconfiguration.Catalog usageIn this section I would like to discuss some of the important RMAN catalog database usage scenarios and options youshould be careful with. Let me start with a controlfile recovery from an auto backup. While this is not directly related tothe catalog database it may explain an additional use case compared to a configuration without a catalog database.CONTROL FILE AUTO RECOVERYFrom a complete database recovery point of view (e.g., complete server or storage loss 14) the main task for which wepossibly need a catalog database is a controlfile restore. As soon as the controlfile is restored, in most cases we do notneed a catalog database as the controlfile contains all information necessary for further recovery.It is not as big a problem for file system based backups (e.g., NFS filers) as for tape based backups. Quite often we needto know a “handle” name15 to restore a file or group of files from tapes (MML). Some of you may say that if weconfigure a controlfile auto backup (RMAN command “CONFIGURE CONTROLFILE AUTOBACKUP ON”) thenthe controlfile recovery problem is solved. Do not say that too quickly. The problem with controlfile auto recovery fromMML is that a restore of a single control file could take several hours. A controlfiles auto backup uses a standard handlename (%F format). The handle name consists of the following components: “c-“ + “database ID” + “date” + “XX”,where “XX” is a hexadecimal sequence that starts with 00 and has a maximum of FF. The sequence increases each timea new controlfile auto backup is created in a given day. It is important to mention that the index resets daily. At recoverytime, RMAN, in order to recover the freshest auto backup copy, tries to use a handle name with FF at the end, and thenFE and so on until MML sends back a stream of data instead of returning a file/stream not found error. Sometimes itmay take up to 240 roundtrips to get to the right index/handle name. It is not rare to see one roundtrip to MML service14 Please note that while full recovery scenarios happen, most often DBAs deal with cases where either a currentcontrolfile copy is available or it is known from where to recover a controlfile (e.g., cloning from production). Thereforefull recovery happens rarely. Nevertheless a DBA should be ready for a worst case scenario.15 A name assigned to a backup stream at the time of backup.
taking 2-5 minutes. After a simple calculation it brings us to 8-20 hours16 just for a controlfile recovery. Because of theway a controlfile auto backup works, controlfile file auto recovery may not be an acceptable option to use for acontrolfile recovery. An alternative would be to provide a handle name within the recovery command. The handle namecould be found in a backup log file or from the MML repository.CATALOG & MML INTEGRATIONThe next catalog database usage scenario I would like to cover is catalog and MML (tape) configuration. This is aconfiguration where I personally find a catalog database most reasonable to use, as it reduces significant risks andstreamlines the recovery process.CONTROL FILE RECOVERYOne important step in the recovery process is a controlfile restore. Depending on your recovery parameters you maywant to restore a controlfile from older backups rather than from latest backup (e.g., auto backup). The good news isany controlfile that contains information about a backup you want to restore from will be sufficient for a recovery. Thismeans you have flexibility to restore a controlfile from up to several days after a target backup has been taken. Thelength of the interval will be dependent on retention policy and controlfile size (control_file_record_keep_time). Let us listseveral common ways to restore a controlfile from MML repository WITHOUT an RMAN catalog database to make acase for use of a catalog database: - MML REPOSITORY: First of all it is always possible to find a stream of data using MML repository that most likely will be a controlfile. Most MMLs repositories have information about the client that sent each stream of data and the time when it happened. It is just matter of luck and time (trial and error method). However, often in a restore scenario we do not have much time. The fact that there could be several databases running on the same MML client (DB server) could introduce an additional challenge. - RMAN LOG: Recovery Manager records handle names each time it sends data to MML in log files. This is one additional argument for why it is important to store RMAN log files in an easily accessible place (e.g., most commonly the DBA team’s mailbox). This can reduce the trial and error time significantly - HANDLE: For most RMAN & MML configurations it is possible to assign a custom handle name at the time you execute a backup. The method to do so is dependent on the MML software. Most commonly you either set it via RMAN “ENV=” clause or “format” channel keyword. If you assign a meaningful name to a controlfile backup piece (e.g., DB name, date & time) then it will be an easy task to identify a handle you are interested in from the MML repository as it lists all the handles names.As you can see there are options that allow you to restore a controlfile from MML without using a catalog database. Thethird option even allows you to pass metadata information to MML that allows you to identify a controlfile “handle” youneed to recover from the MML side without any issues. However it requires very close cooperation between the OracleDBA and the tape system’s management team. In many organizations those two teams do not talk to each other veryoften. Sometimes MML management is outsourced to third organization. Communications between Oracle DBA andMML teams may take significant time, if possible at all.However RMAN catalog database allows avoiding sometimes “difficult” communications. For many DBA teams it iseasier to introduce a catalog database rather than streamline communications that may be a crucial component in apotential restore process. For tape administration teams it has its own benefit: they do not need to provide access toMML repository data to Oracle DBAs.Finally, a catalog database introduces a standard way for restoring controlfiles and databases. This means that even anexternal Oracle DBA who is not familiar with an organizations specific setup (e.g., log file locations, MML integrationdetails, etc) would not have significant problems restoring controlfiles or databases.If an organization has hundreds of databases it often much easier (cheaper) to introduce a catalog database than to workon other backup and restore options. In addition, the organization could use the other benefits of an RMAN catalog tojustify maintenance costs.16 You can reduce controlfile recovery time from autobackup dramatically with maxseq and maxdays options. SeerestoreSpecOperand, FROM AUTOBACKUP from Oracle® Database Backup and Recovery Reference 11g Release 2 (11.2)
RETENTION POLICY MANAGEMENTTypically an organization uses centralized MML services where Oracle databases are just one client of many (otherscould be: filesystems, MS Exchange, other databases, etc). In such a configuration there are often common backup dataretention pools defined (e.g., 1-2 weeks storage, 1-2 month storage & long term or custom) to ensure effective tape lifecycles and recycling. RMAN is not used to ensure the backup retention policy and there is a bit of a challenge tosynchronize RMAN and MML layer data. RMAN may still contain data about backups that are not available in MML.One of the common approaches used in such a case is to change the status of old backups that have crossed the taperetention policy to UNAVAILABLE to keep records for capacity planning purposes and to UNCATALOG at the timewe no longer need the records. Please note that the DELETE command will delete records from the catalog databaseand they would no longer be available for further capacity planning activities. As the RMAN catalog and MML data arenot synchronized in an automatic way there could be situations when RMAN would try to request a non-existentbackup. This configuration should be carefully planned and tested to avoid such situations.CATALOG & DROne of the configurations where an RMAN catalog is a MUST is if an organization takes backups from both sites orfrom the Disaster Recovery (DR) site only 17. If you are running backups from both primary and standby databases youshould be aware about backups associations to different sites (SITE_KEY column in RMAN catalog view). Any diskbased backups considered as local to the site/database that they have been taken from. Tape based backups areconsidered to be accessible from both sites. You may want to review “RMAN Backups in a Data Guard Environment” from“Oracle Database Backup and Recovery Reference 11g Release 2 (11.2)” for more information. Special care should be takenrunning backups in Disaster Recovery configuration and an RMAN catalog is required to complete the task.CATALOG & FSSome may think that this is a very simple configuration, and in fact it is, unless there is a separate file system backup thatbacks up your RMAN backup directories. In such a case I would strongly suggest you consider implementing MMLintegration as otherwise you are at risk of either missing copying some RMAN backup files to tape (puttingrecoverability at risk) or making too many tape backups for some of the RMAN backup files. You may find the reasonsbehind this advice in the “10 Problems with your RMAN backup script” paper.Despite the warning, many organizations run such a configuration. Therefore in the context of an RMAN catalog andrecoverability I would like to suggest you to keep RMAN metadata records in the catalog repository as long as backupfiles are available on tapes. Do not use DELETE OBSOLETE commands. This ensures that the RMAN catalogrepository is “aware” of backups available on the tapes and you could use the “RESTORE … PREVIEW” command toget the full list of files necessary for an intended recovery.Practical hints for catalog usageI am providing additional hints for catalog database usage for your consideration under this section. Most of theinformation here, for one reason or another, did not find a good place in the previous section.DO NOT SEPARATE DEVELOPMENT & PRODUCTION CATALOGSHaving a centralized RMAN repository across development and production environments will make your environmentcloning (DUPLICATION) efforts easier, avoiding unnecessary additional operations. You still may want to have anRMAN development environment for upgrade testing. You may want either to keep a database synchronized with bothcatalogs or to switch some development databases to the second catalog only for the duration of your upgrade testing.DBID MUST BE DIFFERENT FOR ALL DATABASESThis is rather small comment. Before Oracle introduced the DB New ID utility (nid) it was a common practice to havemultiple databases with the same ID. Nowadays the DUPLICATE command changes the DB’s ID for you and if you17 I would strongly suggest this option in DR configuration as it allows you to take backup load away from theproduction system and to create offsite backups at the same time.
need to you can run the “nid” utility to change the ID yourself. You cannot register multiple databases with the same IDunder the same RMAN catalog repository, with the exception of databases in a Data Guard configuration.TO USE OR NOT TO USE RMAN CATALOG STORED SCRIPTS?The RMAN stored scripts are a very powerful feature that simplifies backup script change management across the wholeenterprise network. Before each backup the catalog stored script is retrieved from the repository. It is enough to changea script in one place to adjust all or a subset of databases backup procedures. It pays off in environments with manydatabases (e.g., private clouds). Shell backup script management may introduce significant overhead in suchenvironments.However, if we introduce catalog stored RMAN scripts we should make sure that the RMAN catalog database is highlyavailable. Otherwise backups will fail when unable to access the backup scripts.An alternative solution is to sacrifice manageability and make sure that backups may happen even if a catalog is notavailable.RMAN SETUP FOR CATALOG DATABASE FAILURESIt is relatively simple to ensure that backups keep running even if a catalog database for one reason or another is notavailable. First of all you must use shell backup scripts as opposed to RMAN catalog stored scripts. Ensure that thecatalog connection string is executed within the RMAN script’s body instead of connecting to a catalog when calling therman executable. This way even if the catalog database is not available, the backup will continue, reporting a warning oncatalog database unavailability.Do not use the following syntax in your backup scripts: rman target / catalog user/password@catalog. This command will failif the catalog database is not available because of network or other issues.USE RESYNC CATALOGInstead of keeping a connection to a catalog database during the whole backup, we could connect to the catalog databaseand issue “resync catalog” command at the end of each backup. This provides several benefits, including the following: - Reduces RMAN catalog dependencies for long duration backups. The backup will need connectivity to a catalog database only for a very short time at the end of each run. - If you use traditional connectivity a backup will fail if there is a connection failure to a catalog database during the backup execution. However this is avoided, reducing failures, if we use “resync catalog” at the end of each backup. - Backups will run with no problems even if an RMAN catalog database is not available (e.g., if there is ongoing RMAN catalog maintenance). The backup metadata will get synchronized during the next backup execution.INTRODUCE TWO CATALOG DATABASES TO ENSURE HIGH AVAILABILITYThere is nothing that would prevent you from registering a database in two catalog databases if you need it. Onepotential use case is a DR solution where there are two catalog databases on both sites. Each database would connect toboth catalogs at the end of each backup run and issue the “resync catalog” command to keep both up to date. Thiseffectively ensures the catalog databases high availability in the case of a disaster recovery scenario and there is at leastone catalog database to synchronize with in case the other RMAN catalog database is unavailable for a maintenanceprocess. Note that a database can resynchronise with the catalog database any time, even after a few weeks out ofsynchronisation.This solution could be used as an alternative to Data Guard that potentially may trigger additional licensing 18.18You must check the licensing question with you Oracle Sales representatives. This paper has a technical nature andcannot be used as a reference in licensing questions. The only purpose here is to provide a reader with different possibletechnical solutions to ensure RMAN catalog database high availability.
KEEP RMAN CATALOG ON THE SAME HARDWARE AS MML SOFTWAREIf possible, you may consider keeping both the RMAN catalog database and MML software on the same piece ofequipment. These two components have the same availability requirements. If MML is not available, the fact that thecatalog database is unavailable would not make any difference. However, while this configuration saves some resources,it makes MML server dependent on Oracle catalog requirement and vice versa. A proper capacity planning should beput in place before making such decision as MML side processes quite often as resources intensive.ConclusionsDatabases backup is an Oracle DBA’s primary responsibility area. It is our responsibility to ensure that databases have avalid backup and could be restored based on business requirements at any time. We should streamline and bullet proofthe recovery process. The best time to prepare for a comfortable recovery is weeks, months or even years before youneed to execute it. An RMAN catalog database helps simplify potential recovery and provides a suite of other benefits.However any additional database in an organization’s network increases operational costs. Today when organizationshave limited budgets and resources any cost increase needs to be justified. Many Oracle DBAs do not have a strongjustification for RMAN catalog database. Typically if an RMAN catalog database exists it was created for historicalreasons or because of Oracle recommendation without requirements analysis. This paper provided you with list ofscenarios where an RMAN catalog database must be used because of technical dependences and additional value addedand risk reducing use-cases.In large environments with hundreds or even thousands of databases RMAN database is easier to justify as itsmanagement costs spread across many databases and value increases as it helps to manage large number of databasesreducing related maintenance costs.In configurations where RMAN is integrated with MML (Tapes based backup solution) catalog database simplifiesrecovery operations. It does not apply to technical procedures only. It also reduces the number of people andcommunication steps involved in most recovery scenarios and therefore simplify and reduce the time necessary for adatabase recovery. On the other hand it is simpler to restore a database from file system based backups. In suchconfiguration Oracle DBAs should assess other benefits that RMAN catalog database provides.If a catalog database exists in the environment you are managing already consider implementing other use-cases you andyour team can leverage from, simplifying and bulletproofing backup operations and reducing recoverability risks.