Never a DULL Moment
 How to Avoid Costly Data Recovery


                              RMOUG QEW
                         ...
Who am I?
Daniel Fink
  Oracle DBA since 1996
  Diagnosis, Optimization, Data Recovery and
  Training
  Member of Oak Tabl...
Agenda
  DULs
  Recoveries
  Case Studies
      Worst Practices
      Best Practices

Some case studies provided by Kurt V...
Never a DULL Moment
DUL – Data UnLoader
  Extract data from a down database
Option of last resort
  Downtime
  Expensive
 ...
Why do you need a DUL?
Seed
 Incorrect Configurations
 Poor Policies/Procedures
 Inflexible Processes
 Lack of Security
Tr...
Bullet Proof Backups
Simply don’t exist
  There will always be a point of failure
Keep it simple, but thorough
  Added com...
An Unrecovered Backup
Is No Backup At ALL!
Recovery is Job One
 “Contrary to common opinion, a DBA does not have a
 respon...
Audience Participation
Today
 Did you check your backup log?
This Week/Month
 Did you check the backup process?
 Did you r...
Best Core Practices
Find Recovery Opportunities
  Environment Refreshes
  Upgrade/Patch Testing
  Disaster Recovery Traini...
Best Core Practices
You have known, good processes
 This does not mean every backup is good
 Always test after any changes...
Best Core Practices
Prevention
  Audits
  Implementation Checklists
Find Opportunities to Recover
  Refreshes
  DBA sandbo...
Case Studies
Situation
  Summary of the issue
Seed
  A condtion that is present
Trigger
  An event that causes the failure...
“Hot” Backups
Files were not being properly backed up
Seed
  DBA did not understand how files were managed
  Backup the fi...
Best Practice
Basic Oracle, Backup and Recovery
knowledge
  Oracle Documentation
  DBA Training
Good backup process
No Backup
No backup for production
Seed
 Backups not set up
Trigger
 Media failure
Red Flag
 Production use of a database ...
Best Practice
Backups are part of the implementation
check off/hand over
Test Recovery before implementation
A backup that may work
Backup set does not encapsulate full recovery
set
Seed
  Custom script does not include all command...
Best Practice
Custom scripts require complete
knowledge
  Full backup set
  Command sequence
Every backup set should be se...
Known Bad Backup
Archived redo logs were known to be corrupt
Seed
  Bug in Oracle caused corrupt archived redo logs
  Appl...
Best Practice
Be careful of complicated application
architectures
Have the political will to do the right
thing
Find an in...
User not in the Specs
User level export as only backup
Seed
  User added to database, but not script
Trigger
  Media failu...
Best Practice
If you are responsible for the database,
for recovery of the database…you are
responsible for the backup!
  ...
You are…the weakest link
Improper tape management
Seed
 Unskilled, unmotivated operations personnel
Trigger
 Anything…
Red...
Best Practice
If you are responsible for the database,
for recovery of the database…you are
responsible for the backup!
Yo...
We can just Reload
Data warehouse recovery strategy was
to reload
Seed
 Database grew, but backup strategy did not
Trigger...
Best Practice
Periodically revisit non-standard backup
strategies
Better yet…avoid non-standard backup
strategies
We don’t need no stinkin’
     SYSTEM tablespace
Default installation on local drive with
additional datafiles on external...
Best Practice
Properly plan and install databases
Verify that all needed parts of the
database are being backed up
  Witho...
Security
Table is dropped in production
Seed
  Improper security
  Invalid Backups
Trigger
  Wrong environment
  Wrong act...
Best Practice
Appropriate Architecture and Policies
  Schema owner logins
  Non-database tier authentication
Security Audi...
It’s Hammer Time!
Disks failed and user level export was
incomplete
Seed
  Known bad hardware
  Exports not dynamic
Trigge...
Best Practice
DON’T USE A HAMMER!!!!!
Use dynamic scripting techniques
  Backups
  Exports
Validate scripting
SOX and Recoveries
7 years of data
Could you recover a 7 year old backup?
  2001 – Oracle 9i introduced
    Most systems 7...
How to avoid calling me…
Backups are part of any installation
  Test recovery before turning over to
  user/developer
  Do...
The only good recovery is a successful
recovery
  Determine likely, unlikely and worst-case scenarios
  Look for opportuni...
Audit security
  Know who can access production and how
  Establish policies and procedures to minimize risk
Annual Reviews
Go Forth
  and
Recover!
Upcoming SlideShare
Loading in...5
×

Nevera Dul Moment

1,878

Published on

One of the worst situations is when you have a database in need of recovery...and find that you don't have a usable backup. One option is to use a Data Unloader (DUL), a costly tool/service that is able to extract data from a down and unusable database. However, the best option is to avoid this situation completely through recovery testing, database refreshes, and proper security. This presentation focuses on common reasons for data recovery (worst practices) and how to avoid them (best practices).

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,878
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
27
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Nevera Dul Moment

  1. 1. Never a DULL Moment How to Avoid Costly Data Recovery RMOUG QEW November 2008
  2. 2. Who am I? Daniel Fink Oracle DBA since 1996 Diagnosis, Optimization, Data Recovery and Training Member of Oak Table, BAARF and BAAG www.optimaldba.com daniel.fink@optimaldba.com
  3. 3. Agenda DULs Recoveries Case Studies Worst Practices Best Practices Some case studies provided by Kurt Van Meerbeeck (www.ora600.be)
  4. 4. Never a DULL Moment DUL – Data UnLoader Extract data from a down database Option of last resort Downtime Expensive May or May not work
  5. 5. Why do you need a DUL? Seed Incorrect Configurations Poor Policies/Procedures Inflexible Processes Lack of Security Trigger Human Error Technology Failure
  6. 6. Bullet Proof Backups Simply don’t exist There will always be a point of failure Keep it simple, but thorough Added complexity = Added risk Change management Protect redo Once redo is lost, recovery stops
  7. 7. An Unrecovered Backup Is No Backup At ALL! Recovery is Job One “Contrary to common opinion, a DBA does not have a responsibility to back up a database. The DBA’s real responsibility is to be able to recover the database.” Essential Oracle8i Data Warehousing (Dodge/Gorman) “The actual responsibility is to restore or recover the database to the point in time and within the downtime window determined by the business needs.” Real Life Recovery (RMOUG Training Days 1999)
  8. 8. Audience Participation Today Did you check your backup log? This Week/Month Did you check the backup process? Did you recover a backup? Ever Did you check your backup log? Did you recover a backup?
  9. 9. Best Core Practices Find Recovery Opportunities Environment Refreshes Upgrade/Patch Testing Disaster Recovery Training Every single case study presented would have been avoided if they had tested recovery
  10. 10. Best Core Practices You have known, good processes This does not mean every backup is good Always test after any changes You have documented the processes Help when thinking is not clear
  11. 11. Best Core Practices Prevention Audits Implementation Checklists Find Opportunities to Recover Refreshes DBA sandboxes
  12. 12. Case Studies Situation Summary of the issue Seed A condtion that is present Trigger An event that causes the failure Red Flag A “recognized” indication of a future problem
  13. 13. “Hot” Backups Files were not being properly backed up Seed DBA did not understand how files were managed Backup the files without putting them into backup mode Trigger Media failure Red Flag Lack of desire to learn
  14. 14. Best Practice Basic Oracle, Backup and Recovery knowledge Oracle Documentation DBA Training Good backup process
  15. 15. No Backup No backup for production Seed Backups not set up Trigger Media failure Red Flag Production use of a database without backup
  16. 16. Best Practice Backups are part of the implementation check off/hand over Test Recovery before implementation
  17. 17. A backup that may work Backup set does not encapsulate full recovery set Seed Custom script does not include all commands within backup set Trigger Fraud investigation Red Flag Custom hot backup script command sequence incorrect
  18. 18. Best Practice Custom scripts require complete knowledge Full backup set Command sequence Every backup set should be self- contained Can you backup your worst-case recovery scenario?
  19. 19. Known Bad Backup Archived redo logs were known to be corrupt Seed Bug in Oracle caused corrupt archived redo logs Application owner “could not afford downtime to fix” Trigger Rollback segment tablespace went offline Monitoring software failed Red Flag Backups known to be unrecoverable
  20. 20. Best Practice Be careful of complicated application architectures Have the political will to do the right thing Find an interim solution
  21. 21. User not in the Specs User level export as only backup Seed User added to database, but not script Trigger Media failure Red Flag Static scripts Development responsible for backups
  22. 22. Best Practice If you are responsible for the database, for recovery of the database…you are responsible for the backup! Export can only restore a database, not perform full recovery Audit schema owners v. users being backed up
  23. 23. You are…the weakest link Improper tape management Seed Unskilled, unmotivated operations personnel Trigger Anything… Red Flag Non-technical personnel in charge of tape management
  24. 24. Best Practice If you are responsible for the database, for recovery of the database…you are responsible for the backup! You have to trust those responsible for operations
  25. 25. We can just Reload Data warehouse recovery strategy was to reload Seed Database grew, but backup strategy did not Trigger Current redo log corruption Red Flag Backup strategy not revisited as database grew
  26. 26. Best Practice Periodically revisit non-standard backup strategies Better yet…avoid non-standard backup strategies
  27. 27. We don’t need no stinkin’ SYSTEM tablespace Default installation on local drive with additional datafiles on external drives Seed Single database has files on separate storage systems Trigger Media Failure Red Flag Never checking backup process
  28. 28. Best Practice Properly plan and install databases Verify that all needed parts of the database are being backed up Without SYSTEM tablespace, you lose the ‘map’ to tables…and data Know what is and is not needed
  29. 29. Security Table is dropped in production Seed Improper security Invalid Backups Trigger Wrong environment Wrong action Red Flag Access to production
  30. 30. Best Practice Appropriate Architecture and Policies Schema owner logins Non-database tier authentication Security Audits Know who has what and why Balance safety v. security
  31. 31. It’s Hammer Time! Disks failed and user level export was incomplete Seed Known bad hardware Exports not dynamic Trigger Disk crash…finally Red Flag A hammer attached to a storage device is rarely a good sign
  32. 32. Best Practice DON’T USE A HAMMER!!!!! Use dynamic scripting techniques Backups Exports Validate scripting
  33. 33. SOX and Recoveries 7 years of data Could you recover a 7 year old backup? 2001 – Oracle 9i introduced Most systems 7.3 and 8.x Do you have a 7.3 install? Do you have 7 year old Hardware? O/S and drivers?
  34. 34. How to avoid calling me… Backups are part of any installation Test recovery before turning over to user/developer Document the process Understand the implications of changes Adapt the strategy to the system Monitor backups on a daily basis Exception reporting is good, but not perfect Know what to do if a backup fails
  35. 35. The only good recovery is a successful recovery Determine likely, unlikely and worst-case scenarios Look for opportunities to perform recoveries Understand the implications of changes Don’t uncover issues on production systems
  36. 36. Audit security Know who can access production and how Establish policies and procedures to minimize risk Annual Reviews
  37. 37. Go Forth and Recover!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×