Your SlideShare is downloading. ×
  • Like
10 Problems with your RMAN backup script
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

10 Problems with your RMAN backup script

  • 20,962 views
Published

Yuri is called to audit RMAN backup scripts on regular basis for several years now as part of his Day to Day duties. He see the same errors in scripts that Oracle DBAs using to backup critical …

Yuri is called to audit RMAN backup scripts on regular basis for several years now as part of his Day to Day duties. He see the same errors in scripts that Oracle DBAs using to backup critical databases over and over again. Those errors may play a significant role in a recovery process when you working under stress. During that presentation you will be introduced to typical issues and hints how to address those.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • @linamos It could be a bit of challenge to came with a universal script that would suite most of the needs wihtout introducing potential issues. To be honest I don't have a universal script that I could share right away. Will keep this request in mind and see if I can come with one.
    Are you sure you want to
    Your message goes here
  • @muschinskiy In this particular example we make a control file backup and then deleting obsolete backups. The copy of the controlfile we just made doesn't 'know' about about deleted backups. Instead of that I am suggesting to make a controlfile backup right after the delete obsolete command. Hope it clarifies the statement.
    Are you sure you want to
    Your message goes here
  • Yuri, could you please explain why we are making controlfile backup inconsistent ? (#3)

    Thanks much in advance!
    Are you sure you want to
    Your message goes here
  • Hi Yury,
    Thank you very much for your excellent presentation. Do you mind sharing with us a sample rman backup script you use on daily basis?
    Thanks,
    Amos
    Are you sure you want to
    Your message goes here
  • Excellent suggestions!
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
20,962
On SlideShare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
642
Comments
6
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Even if you use catalog try to keep your control file as consistent as possibleIt may happen that catalog database is not available at the time of recovery
  • Assumption: backups are taken on primary sideThe same principals+ apply for standby backupsIf you want to keep ARCHs on a disk then it makes sense to keep ARCHs on disk since the last data files backup
  • Make an explicit controlfile copy at the end of each backup...BACKUP CURRENT CONTROLFILE;EXIT
  • RMAN> RUN{SET UNTIL SCN 898570;RESTORE DATABASE PREVIEW;}2> 3> 4> 5>executing command: SET until clauseStarting restore at 2011/11/01 13:11:30…BS Key Type LV Size Device Type Elapsed Time Completion Time------- ---- -- ---------- ----------- ------------ -------------------515 Full 183.52M DISK 00:00:56 2011/10/20 22:34:31 BP Key: 524 Status: AVAILABLE Compressed: YES Tag: TAG20111020T223336 Piece Name: /u02/fast_recovery_area/PROD1/backupset/2011_10_20/o1_mf_nnndf_TAG20111020T223336_7b01rkqd_.bkp List of Datafiles in backup set 515 File LV Type Ckp SCN Ckp Time Name ---- -- ---- ---------- ------------------- ---- 1 Full 898442 2011/10/20 22:33:37 /u02/oradata/PROD1/system01.dbf…BS Key Size Device Type Elapsed Time Completion Time------- ---------- ----------- ------------ -------------------564 1.18M DISK 00:00:00 2011/10/20 22:35:37 BP Key: 567 Status: AVAILABLE Compressed: YES Tag: TAG20111020T223536 Piece Name: /u02/fast_recovery_area/PROD1/backupset/2011_10_20/o1_mf_annnn_TAG20111020T223536_7b01w936_.bkp List of Archived Logs in backup set 564ThrdSeq Low SCN Low Time Next SCN Next Time ---- ------- ---------- ------------------- ---------- --------- 1 32 897131 2011/10/20 22:16:14 898570 2011/10/20 22:35:34BS Key Size Device Type Elapsed Time Completion Time------- ---------- ----------- ------------ -------------------648 435.50K DISK 00:00:00 2011/10/20 22:37:47 BP Key: 651 Status: AVAILABLE Compressed: YES Tag: TAG20111020T223746 Piece Name: /u02/fast_recovery_area/PROD1/backupset/2011_10_20/o1_mf_annnn_TAG20111020T223746_7b020csz_.bkp List of Archived Logs in backup set 648ThrdSeq Low SCN Low Time Next SCN Next Time ---- ------- ---------- ------------------- ---------- --------- 1 33 898570 2011/10/20 22:35:34 898848 2011/10/20 22:37:46List of Archived Log Copies for database with db_unique_name PROD1=====================================================================Key ThrdSeq S Low Time------- ---- ------- - -------------------759 1 34 A 2011/10/20 22:37:46 Name: /u02/fast_recovery_area/PROD1/archivelog/2011_10_22/o1_mf_1_34_7b40yobq_.arc760 1 35 A 2011/10/22 10:44:12 Name: /u02/fast_recovery_area/PROD1/archivelog/2011_10_22/o1_mf_1_35_7b4mywqq_.arc761 1 36 A 2011/10/22 16:08:42 Name: /u02/fast_recovery_area/PROD1/archivelog/2011_10_24/o1_mf_1_36_7bbonx10_.arc887 1 37 A 2011/10/24 23:14:20 Name: /u02/fast_recovery_area/PROD1/archivelog/2011_10_25/o1_mf_1_37_7bf977j3_.arc888 1 38 A 2011/10/25 23:00:39 Name: /u02/fast_recovery_area/PROD1/archivelog/2011_11_01/o1_mf_1_38_7byo324j_.arcMedia recovery start SCN is 898442Recovery must be done beyond SCN 898498 to clear datafile fuzzinessFinished restore at 2011/11/01 13:11:30RMAN>

Transcript

  • 1. Michael S. Abbey 110 Problems with your RMAN backup script Presented by: Yury Velikanov Senior Oracle DBA The Pythian Group April 2012
  • 2. WHY PYTHIAN● Recognized Leader: – Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server – Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments● Expertise: – One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 7 Oracle ACEs/ACE Directors – Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC● Global Reach & Scalability: – 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response2
  • 3. FEW WORDS ABOUT YURY • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …] - Email me to get the presentation • Sr. Oracle DBA at Pythian, Oracle ACE and OCM • Started as Oracle DBA - with 7.2 (in 1997, 14+) • First international appearance - 2005 - Hotsos Symposium 2005, … • Education (Masters Degree in Computer science) - OCP 7/8/8i/9i/10g/11g + OCM 9i/10g/11g • Oracle DBA consultant experience (14+ years) • Pythian Oracle Clients support (2+ years) - 140+ Clients around the world - RMAN scripts audit, troubleshooting, recovery3
  • 4. MISSION Give you practical 10 hints on RMAN script improvements. Encourage you to think on what can possibly go wrong before it happens. Give away some prizes 4
  • 5. RIGHT APPROACH ... ! BE SKEPTICAL ! ● If backups and trial recovery works it doesn’t mean you don’t have issues (must test/document/practice recovery) ● Challenge your backup procedures! - Think about what can possibly go wrong - Think now as in the middle of an emergency recovery it may be way too late or too challenging ● Prepare all you may need for smooth recovery while working on backup procedures5
  • 6. FEW GENERAL THOUGHTS … NEVER rely on backups stored on the same physical media as the database! Mark Brinsmead, Sr. Oracle DBA, Pythian Even if your storage is the fanciest disk array (misnamed "SAN" by many) in the world, there exist failure modes in which ALL data in the disk array can be lost simultaneously. (Aside from fire or other disaster, failed firmware upgrades are the most common.) You dont really have a "backup" until the backup is written to separate physical media!6
  • 7. FEW GENERAL THOUGHTS … Avoid situations where the loss of a single piece of physical media can destroy more than one backup. When backing up to tape, for example, if the tape capacity is much larger than your backups, consider alternating backups between multiple tape pools. ("Self-redundant" backups are of little value if you are able to lose several consecutive backups simply by damaging one tape cartridge). If your backup and recovery procedures violate some of the base concepts - state risks clearly and sign/discuss those with business on regular basis.7
  • 8. #1 RMAN LOG FILES part of a log file ... RMAN> Do you see any issues? Starting backup at 18-OCT-11 current log archived allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=63 device type=DISK channel ORA_DISK_1: starting compressed archived log backup set channel ORA_DISK_1: specifying archived log(s) in backup set input archived log thread=1 sequence=4 RECID=2 STAMP=764855059 input archived log thread=1 sequence=5 RECID=3 STAMP=764855937 ... Finished backup at 18-OCT-11Prepare all you may need for smooth recovery while working on backup procedures 8
  • 9. #1 RMAN LOG FILESpart of a log file ...RMAN> backup as compressed backupset database How about now?2> include current controlfile3> plus archivelog delete input;Starting backup at 2011/10/18 12:30:46current log archivedallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=56 device type=DISKchannel ORA_DISK_1: starting compressed archived log backup setchannel ORA_DISK_1: specifying archived log(s) in backup setinput archived log thread=1 sequence=8 RECID=6 STAMP=764856204input archived log thread=1 sequence=9 RECID=7 STAMP=764857848...Finished backup at 2011/10/18 12:33:549
  • 10. #1 RMAN LOG FILES Before calling RMAN • export NLS_DATE_FORMAT="YYYY/MM/DD HH24:MI:SS" • export NLS_LANG="XX.XXX_XXX"(for non standard char sets) before running commands • set echo on Nice to have: total execution time at the end of log file c_begin_time_sec=`date +%s` ... c_end_time_sec=`date +%s` v_total_execution_time_sec=`expr ${c_end_time_sec} - ${c_begin_time_sec}` echo "Script execution time is $v_total_execution_time_sec seconds"10
  • 11. #1 RMAN LOG FILES ● do not overwrite log file from previous backup full_backup_${ORACLE_SID}.`date +%Y%m%d_%H%M%S`.log Use case: a backup failed • should I run the backup now? • would it interfere with business activities?11
  • 12. KISS = KEEP IT STUPID SIMPLE crosscheck archivelog all; delete noprompt expired archivelog all; backup database include current controlfile plus archivelog delete input; delete noprompt obsolete;14
  • 13. #2 DO NOT USE CROSSCHECK (ANTI KISS)● Do not use CROSSCHECK in your day to day backup scripts!● If you do, RMAN silently ignores missing files, possibly making your recovery impossible● CROSSCHECK should be a manual activity executed by a DBA to resolve an issue15
  • 14. #3 BACKUP CONTROL FILE AS THE LAST STEP backup as compressed backupset database plus archivelog delete input include current controlfile; Is it right? delete noprompt obsolete; exit We are making the controlfile backup inconsistent immediately16
  • 15. #3 BACKUP CONTROL FILE AS THE LAST STEP backup as compressed backupset database plus archivelog delete input; Is it right? delete noprompt obsolete; backup spfile; backup current controlfile; exit17
  • 16. #4 DO NOT RELY ON ONE BACKUP ONLY● Do not rely on ONE backup only! • You should always have a second option● Especially true talking about ARCHIVE LOGS • If you miss a single ARCHIVE LOG your recoverability is compromised-- ONE COPY ONLYBACKUP DATABASE ... PLUS ARCHIVELOG DELETE INPUT;-- SEVERAL COPIESBACKUP ARCHIVELOG ALL NOT BACKED UP $v_del_arch_copies TIMES;18
  • 17. #5 DO NOT DELETE ARCHIVE LOGS BASED ON TIME ONLY -- TIMESTAMP DELETE NOPROMPT BACKUP OF ARCHIVELOG ALL COMPLETED BEFORE SYSDATE-6/24 DEVICE TYPE DISK; -- SEVERAL COPIES + TIME DELETE NOPROMPT ARCHIVELOG ALL BACKED UP $v_del_arch_copies TIMES TO DISK COMPLETED BEFORE $p_start_of_last_db_backup; -- SEVERAL COPIES + TIME + STANDBY CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;19
  • 18. #6 USE CONTROLFILE IF CATALOG DB ISNT AVAILABLE [oracle@host01 ~]$ rman target / catalog rdata/xxx Recovery Manager: Release 11.2.0.2.0 - Production on Tue Oct 18 15:15:25 2011 ... connected to target database: PROD1 (DBID=1973883562) RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-00554: initialization of internal recovery manager package failed RMAN-04004: error from recovery catalog database: ORA-28000: the account is locked [oracle@host01 ~]$ ● Check if catalog DB is available in your script • If it is, connect to catalog DB • If it isn’t, use controlfile only (flagging it as warning)20
  • 19. #6 USE CONTROLFILE IF CATALOG DB ISNT AVAILABLE rman target / RMAN> echo set on RMAN> connect target * connected to target database: PROD1 (DBID=1973883562) RMAN> connect catalog * RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-04004: error from recovery catalog database: ORA-28000: the account is locked RMAN> backup as compressed backupset database 2> include current controlfile 3> plus archivelog delete input; Starting backup at 2011/10/18 15:22:30 current log archived using target database control file instead of recovery catalog special THX 2 @pfierens 4 discussion in tweeter21
  • 20. #6 USE CONTROLFILE IF CATALOG DB ISNT AVAILABLE -- Backup part rman target / <<! backup as compressed backupset database ... ! -- Catalog synchronization part rman target / <<! connect catalog rdata/xxx resync catalog; ! special THX 2 @martinberx 4 discussion in tweeter22
  • 21. #7 DO NOT RELY ON RMAN STORED CONFIGURATION ● Do not rely on controlfile autobackupCONFIGURE CONTROLFILE AUTOBACKUP ON;CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO /b01/rman/prod/%F’ ● Oracle creates a controlfile backup copy ● each time you make any db files related changes ● at the end of each RMAN backup ● What would happen if someone switched autobackup off?23
  • 22. #7 DO NOT RELY ON RMAN STORED CONFIGURATION ● Document configuration in a log file (show all;) ● If you change configuration restore it at the end of your script show all; v_init_rman_setup=`$ORACLE_HOME/bin/rman target / <<_EOF_ 2>&1| grep "CONFIGURE " | sed s/"# default"/""/g show all; _EOF_` ... < script > ... echo $v_init_rman_setup | $ORACLE_HOME/bin/rman target /24
  • 23. #8 BACKUPS’ CONSISTENCY CONTROLFailure Verification and Notification ● How do you report backup failures and errors? ● We don’t report at all ● DBA checks logs sometimes ● Backup logs are sent to a shared email address (good!) ● DBA on duty checks emails (what if no one available/no email received?) ● We check RMAN command errors code $? and sending email25
  • 24. #8 BACKUPS’ CONSISTENCY CONTROLFailure Verification and Notification ● I would suggest • Run log files check within backup script and page immediately • Script all checks and use "OR" in between • echo $? • egrep "ORA-|RMAN-" < log file > • Improve your scripts and test previous adjustments on regular basis • PAGE about any failure to oncall DBA immediately • DBA makes a judgment and takes a conscious decision • PAGE about LONG running backups26
  • 25. #8 BACKUPS’ CONSISTENCY CONTROLNotifications are not enough! ● How do you check if your database is safely backed up based on your business requirements? ● Make a separate check that would page you if your backups don’t satisfy recoverability requirements REPORT NEED BACKUP ... -- datafiles that weren’t backed up last 24 hours! – a bit excessive (2) REPORT NEED BACKUP RECOVERY WINDOW OF 1 DAYS; REPORT NEED BACKUP REDUNDANCY 10; REPORT NEED BACKUP DAYS 2; REPORT UNRECOVERABLE;27 May not available in all Versions!
  • 26. IF YOU DON’T HAVE RMAN AND MML INTEGRATION● You should consider using it! • Otherwise it is extremely difficult to ensure backup consistency● If you don’t use it then your backups are exposed to many issues • At best, your backups will take much more space on tapes than should • In worst case you may miss to backup some of the backup pieces , putting the database recoverability at risk● The next few slides discuss some issues28
  • 27. #9 ENSURE 3 TIMES FULL BACKUPS SPACE + ARCH● IF you don’t have ● A smart backup software (incremental/opened files) ● Sophisticated backup procedures● THEN you need space on a file system for at least 3 FULL backups and ARCHIVE LOGS generated in between 3 backups ● If REDUNDANCY 1 then previous backup and ARCHIVE LOGS got removed after next backup is completed. There is no continued REDO stream on tapes. ● If REDUNDANCY 2 then you need space for third full backup of backup time only (as soon as third backup completed you remove the first one)29
  • 28. #9 DON’T USE “DELETE OBSOLETE” (DISK + NO MML TAPE) ● This way you wipe out RMAN memory. There is no way RMAN knows about backups available on tapes. ● Think about recovery (if you use “delete noprompt obsolete”) 1. You need to recover a control file (possibly from offsite backups) 2. Find and bring onsite all tapes involved (possibly several iterations) 3. Restore and recover (possibly restoring more ARCH backups)backup as compressed backupset databaseplus archivelog delete inputinclude current controlfile;delete noprompt obsolete;exit30
  • 29. #9 DON’T USE “DELETE OBSOLETE” (DISK + NO MML TAPE)-A- LIST BASED ON DISK RETENTIONreport obsolete recovery window of ${DISK_RETENTION_OS} days device type disk;-B- REMOVE FILES BASED ON DISK RETENTION!checking if each of reported files have been backed up to tapes & rm it from FS!-C- WIPEOUT FROM REPOSITORYdelete force noprompt obsolete recovery window of ${TAPE_DAY_RETENTION} days device type disk;-!- AT THE RECOVERY TIMERUN{SET UNTIL SCN 898570;RESTORE DATABASE PREVIEW;}31
  • 30. #9 NEVER KEEP DEFAULT RETENTION POLICY● NEVER allow the RMAN RETENTION POLICY to remain at the default or lower level than TAPE retention • other Oracle DBA can run DELETE OBSOLETE command and wipe all catalog records outCONFIGURE RETENTION POLICY TO REDUNDANCY 1000;CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1000 DAYS;33
  • 31. #10 HALF WAY BACKED UP FILE SYSTEM FILES ● Make sure that your File System backup doesn’t backup half finished backupset-A- BACKUP AS TMPBACKUP DATABASE FORMAT ${file}.tmp_rman;-B- MOVE TO PERMANENTmv ${file}.tmp_rman ${file}.rman-C- MAKE CATALOG AWARECHANGE BACKUPPIECE ${file}.tmp_rman UNCATALOG;CATALOG BACKUPPIECE ${file}.rman;34
  • 32. DO WE HAVE A WINNER? #1 RMAN Log files #2 Do not use CROSSCHECK #3 Backup control file as the last step #4 Do not rely on ONE backup only #5 Do not delete ARCHIVE LOGS based on time only #6 Use controlfile if catalog DB isnt available #7 Do not rely on RMAN stored configuration #8 Backups’ consistency control #9 Don’t use “delete obsolete” (disk + no mml tape) #10 Half way backed up File System files35
  • 33. CONTACT ME Yury Oracle velikanovs@pythian.com @yvelikanov www.pythian.com/news/velikanov36
  • 34. THANK YOU AND Q&A To contact us… sales@pythian.com 1-866-PYTHIAN To follow us… http://www.pythian.com/news/ http://www.facebook.com/pages/The-Pythian-Group/ http://twitter.com/pythian http://www.linkedin.com/company/pythian37 3
  • 35. ADDITIONAL TOPICS #A DELETE OBSOLETE to be executed at the begging #B CATALOG or not to CATALOG #C Number of backup & recovery processes (# of backup pieces) #D CATALOG Keep as less information as reasonable #E “rman target / catalog rdata/xxx” – isn’t secure38