www.BravePoint.comProgressProgressDatabaseDatabaseRepair &Repair &RecoveryRecoveryDan ForemanUnited Systems, Inc.Email: da...
www.BravePoint.comIntroduction- Dan Foreman• Progress user since 1984 (V2.1)• Guest speaker at every USA Progress UsersCon...
www.BravePoint.comIntroduction - Who Are We?• BravePoint Inc. (www.BravePoint.com)• Formerly United Systems, Inc.• 125+ Em...
www.BravePoint.comIntroduction - Who Are You• Progress Version: V6, V7, V8.2, V8.3, V9• Unix? NT? Is there anything else?•...
www.BravePoint.comSpecial Request• Phasers on Stun• Isnt it interesting that the Microsoftspell checker can’t find the wor...
www.BravePoint.comGoals• Can I teach you database brain surgery inonly one hour?
www.BravePoint.comPreventative Maintenance• Backups (yes, I know you’ve heard of them buthave you tested them recently?)• ...
www.BravePoint.comPreventative Maintenance• Unix: Don’t use kill -9 to terminate a Progresssession; You might bring the da...
www.BravePoint.comPreventative Maintenance• Monitor the BI file High Water Mark to avoid the2gb size limit bug which will ...
www.BravePoint.comPreventative Maintenance• Monitor the database High Water Mark to avoidgrowing into the Variable Length ...
www.BravePoint.comQuiz Question• Who are the Smartest People in theRoom?
www.BravePoint.comAnswer• Those running After Imaging on theirMission Critical Databases• Or stated in another way, if you...
www.BravePoint.comAfter Imaging• Who is currently using After Imaging?• If no, why not?• The PSC documentation says it off...
www.BravePoint.comAfter Imaging - Why Use It?• But you say…”I have disk mirroring (alsoknown as RAID 1) so I’m protected a...
www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #1• A DBA FTP’d a test database into thedirectory where t...
www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #2• A user ran an archiving program on livedata that wasn...
www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #3• BI file hit the 2GB bug (true even with MV BI files)@...
www.BravePoint.comAfter Imaging - Why Use It?• Avoid the Problems related to probkup online• Transaction Activity is Froze...
www.BravePoint.comAfter Imaging - Why Use It?• Easy generation of a Warm Spare DB• A Warm Spare DB is:• A standby database...
www.BravePoint.comOnline Backup → Warm SpareProductionSystemWarm SpareProductionDBWarm SpareDBAI Files
www.BravePoint.comAfter Imaging - Why Use It?• Easy generation of a Report Server DB• A Report Server DB is:• A database o...
www.BravePoint.comPreventative Maintenance• Corruption Checks• proutil dbanalys Blocks, Records, & Indexes• probkup/procop...
www.BravePoint.comLog File Checking• Check the Database log (.lg) file for errors DAILY.Look for words such as:• kill* dra...
www.BravePoint.com1124 Errors• SYSTEM ERROR: wrong dbkey in block…..• Indicates (with 99.9999% probability) ahardware prob...
www.BravePoint.comCommon Problems• Corrupt Blocks• Running out of disk space• Deleted/Damaged Portion of the Database• Dat...
www.BravePoint.comCorrupt Blocks• What Kind of Block? Index or Data• Block Type 2 (Index - IX) or 3 (Record Mgr - RM)• If ...
www.BravePoint.comEmergency Dump• ‘Front and Back’ 4GL Dump• for each customer by cust-num (until you hitthe bad spot)• fo...
www.BravePoint.comMiscellaneous Hints• If you can’t get into the database with -F orany other way, try the Read Only (-RO)...
www.BravePoint.comDeleted Extents• First, backup the remaining pieces of thedatabase• This may seem like a useless step bu...
www.BravePoint.comDeleted AI Extents• If an AI Extent is Deleted, simply disable AIand...• What? You’re not running AI? BA...
www.BravePoint.comDeleted AI Extents• Disable AI with rfutil aimage end. You may get anerror message regarding the missing...
www.BravePoint.comDeleted BI Extents• Force Access with -F• V8.2 and later -F only forces access on proutiltruncate bi• If...
www.BravePoint.comDeleted DB Extents• Restore the the DB and BI from Backup• Apply the AI files• Re-enable AI• BI Grow• Do...
www.BravePoint.comDeleted DB Extents• Use prostrct unlock if the deleted Extentwas Empty (above the High Water Mark)• pros...
www.BravePoint.comDeleted DB Extents• Backup Extent Substitute• This technique is for Extents that contain Dataexcept the ...
www.BravePoint.comDeleted DB Extents• If the database Broker is still running:• DON’T Shutdown the Database• That ‘closes’...
www.BravePoint.comDeleted DB Extents - Unix• Warm Boot the System Immediately• Don’t Shut Down the DB First• When Unix fsc...
www.BravePoint.comOverlay Database• .db File is lost• Create a Void Multi-volume Structure thatmatches the problem Databas...
www.BravePoint.comOverlay Database• On V9 it is much easier to restore the .db file• prostrct builddb• Requires an up-to-d...
www.BravePoint.comDisk Full• Use prostrct repair to relocate Extents to alocation with more space• Copy the Extent• Create...
www.BravePoint.comBI Disk Full• Do Not Run out of Space on the BI Disk if:• You can’t make more space for the BI file• You...
www.BravePoint.comProgress HA Challenges• Online Backups• probkup online• AI Backup (already covered)• Split Mirror• Multi...
www.BravePoint.comOnline Backup Options• probkup online• Transaction Freeze during BI Backup• Usually High I/O Volume• Onl...
www.BravePoint.comOnline Backup Options• Split Mirror Backup• Quiet the DB with proquiet enable• Break the Disk Mirror• Un...
www.BravePoint.comOnline Backup Options• Split Mirror Backup• Database is vulnerable during the backupunless you have trip...
www.BravePoint.comProgress HA Challenges• Schema Changes. It is not possible to makeSchema changes online, even in V9.1• A...
www.BravePoint.comProgress HA Challenges• Database Reorg. In V8 is possible to build indexesonline. In V9 it is possible m...
www.BravePoint.comSources of Help• Progress Documentation• Progress Database Administration Guide• dba@peg.com• Progress K...
www.BravePoint.com
Upcoming SlideShare
Loading in...5
×

Technical track-afterimaging Progress Database

507

Published on

Technical track-afterimaging Progress Database


Contact us anytime: Think Next - QAD Partner in Vietnam

- Website: http://www.thinknext.vn
- Email: contact [at] thinknext.vn

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
507
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Technical track-afterimaging Progress Database

  1. 1. www.BravePoint.comProgressProgressDatabaseDatabaseRepair &Repair &RecoveryRecoveryDan ForemanUnited Systems, Inc.Email: danf@prodb.com
  2. 2. www.BravePoint.comIntroduction- Dan Foreman• Progress user since 1984 (V2.1)• Guest speaker at every USA Progress UsersConference from 1990 to 1998• Quest speaker at QAD Conference: 1997, 1999-2001• Author of:• Progress Performance Tuning Guide• Progress Database Administration Guide (Nov 2001)• Progress Virtual System Tables• Progress V9 Database Administration Jumpstart• A paper purchase now allows free online access• ProMonitor - Performance Monitoring Tool• ProD&L - Accelerated Dump/Load Utility
  3. 3. www.BravePoint.comIntroduction - Who Are We?• BravePoint Inc. (www.BravePoint.com)• Formerly United Systems, Inc.• 125+ Employees• Three of us have used Progress since 1984• Progress Service Provider• Performance Tuning• Database Administration and Recovery• Progress Training• Internet, Webspeed, Java• Custom Programming Services
  4. 4. www.BravePoint.comIntroduction - Who Are You• Progress Version: V6, V7, V8.2, V8.3, V9• Unix? NT? Is there anything else?• Largest Single Database?• Highest User Count?
  5. 5. www.BravePoint.comSpecial Request• Phasers on Stun• Isnt it interesting that the Microsoftspell checker can’t find the word“Phaser”• Cell Phones on Mute Please!
  6. 6. www.BravePoint.comGoals• Can I teach you database brain surgery inonly one hour?
  7. 7. www.BravePoint.comPreventative Maintenance• Backups (yes, I know you’ve heard of them buthave you tested them recently?)• Test your Entire Recovery Plan• Warm Spare Database - A database on anothermachine with a recent copy of the production DBThis is easy to do in Progress...covered soon• Unix: don’t logon as root unless you really need to• Use O/S security to protect the DB, BI, and AI filesfrom accidental/casual/intentional deletion
  8. 8. www.BravePoint.comPreventative Maintenance• Unix: Don’t use kill -9 to terminate a Progresssession; You might bring the database DOWN!if you kill a session that is holding a Latch• Always have an up-to-date Structure (.st) fileavailable• Make sure your Sequences are not about tooverflow (at approx. 2 billion)
  9. 9. www.BravePoint.comPreventative Maintenance• Monitor the BI file High Water Mark to avoid the2gb size limit bug which will corrupt your DB(_dbstatus-bisize Virtual System Table); notusing VSTs? Turn Them On!• Monitor Delinquent Transactions (transactionslonger than 30-60 minutes)• Monitor Large Transactions (-L Overflow)• longtrx4.p Progress program on the BravePointwebsite to Automatically disconnect DelinquentTransactions• Use the V8.3 -bithold parameter as an extrasafeguard; Set to 999 or less; Set even in V9• V9 supports Terabyte sized BI Files
  10. 10. www.BravePoint.comPreventative Maintenance• Monitor the database High Water Mark to avoidgrowing into the Variable Length Extent• Could hit the 2gb size limit• Unbuffered I/O to the Variable Extent• Sync call issued each time the Extent grows• Size is requested from the OS in increasinglylarger chunks (16-128 DB Blocks)• Could run out of disk space
  11. 11. www.BravePoint.comQuiz Question• Who are the Smartest People in theRoom?
  12. 12. www.BravePoint.comAnswer• Those running After Imaging on theirMission Critical Databases• Or stated in another way, if you’re notusing AI, you shouldn’t be responsible foryour company’s databases• If your company can’t afford to lose aday’s worth of data you Absolutely needAfter Imaging
  13. 13. www.BravePoint.comAfter Imaging• Who is currently using After Imaging?• If no, why not?• The PSC documentation says it offersprotection against media failure• Disk fails 5 minutes before the backup startson the final day of your year end close• No paper trail• Ouch! Time to work on your resume (C.V.)• After Image File(s) + Last (Good) Backup =State of DB at time of crash
  14. 14. www.BravePoint.comAfter Imaging - Why Use It?• But you say…”I have disk mirroring (alsoknown as RAID 1) so I’m protected against adisk failure”• BUT Mirroring does NOT protect against alldatabase evils
  15. 15. www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #1• A DBA FTP’d a test database into thedirectory where the production databaseresided... unfortunately they had the samename• Disk Mirroring worked just fine…..• After Imaging would have probably savedthe day
  16. 16. www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #2• A user ran an archiving program on livedata that wasn’t ready to be archived• Once again the mirroring performedperfectly• After Imaging might have improved thesituation as it is possible to Roll Forward to aspecific point in time
  17. 17. www.BravePoint.comAfter Imaging - Why Use It?• True Horror Story #3• BI file hit the 2GB bug (true even with MV BI files)@ 4PM on a busy day (300+ users)• 10GB Database was corrupted• Progress Tech Support Recommendation: dump& load or restore from backup which meantsubstantial down time or data loss• Fortunately the customer called me and I wasable to temporarily patch the database• I had recommended AI to this customer over oneyear prior to this event
  18. 18. www.BravePoint.comAfter Imaging - Why Use It?• Avoid the Problems related to probkup online• Transaction Activity is Frozen while the BI Fileis Backed Up• The Buffer Cache is "polluted" with DatabaseBlocks that may not be Commonly Accessed• The Overhead of Running a Streaming TapeDrive on the Same Machine as the DB• Possible Solution• Backup the AI Files• Restore/Copy to Another System (or disk)• Apply the AI Files (Roll Forward)• Backup the Replicated Database
  19. 19. www.BravePoint.comAfter Imaging - Why Use It?• Easy generation of a Warm Spare DB• A Warm Spare DB is:• A standby database• On another machine• That can be brought online quickly in caseof failure to the production system• It’s ‘warm’ because it is not 100% current• A HOT spare is not possible using AI but ispossible using Replication Triggers
  20. 20. www.BravePoint.comOnline Backup → Warm SpareProductionSystemWarm SpareProductionDBWarm SpareDBAI Files
  21. 21. www.BravePoint.comAfter Imaging - Why Use It?• Easy generation of a Report Server DB• A Report Server DB is:• A database on another system• Used for reporting only• To relieve the production system of theload imposed by reporting• Doesn’t require same level of hardware orProgress license
  22. 22. www.BravePoint.comPreventative Maintenance• Corruption Checks• proutil dbanalys Blocks, Records, & Indexes• probkup/procopy Blocks only• proutil dbrpr Blocks and Records• proutil dbscan Same as dbrpr (cmd line)• proutil idxfix Indexes <-> Records• Run periodically to make sure you don’t havehidden or unreported corruption
  23. 23. www.BravePoint.comLog File Checking• Check the Database log (.lg) file for errors DAILY.Look for words such as:• kill* drastic warn* errorsystem abnormal exceed* fail*wrong unexpected*invalid dieddamage* dead overflow* violationfatal insufficient missing disappear*corrupt* allow* attempt* cannotenough illegal impossible increaseunknown unable stop* beyond• There are programs on the our Web Site to assistwith log file checking or products such asProMonitor
  24. 24. www.BravePoint.com1124 Errors• SYSTEM ERROR: wrong dbkey in block…..• Indicates (with 99.9999% probability) ahardware problem• Don’t limit your search to disks; also consider:Disk Controllers, SCSI termination, RAM (parityerrors), Firmware, etc.• Don’t Let the Hardware Technician BlameProgress or the Application• Don’t let the Hardware Technician escapewithout solving the problem - every componentmight need to be swapped out
  25. 25. www.BravePoint.comCommon Problems• Corrupt Blocks• Running out of disk space• Deleted/Damaged Portion of the Database• Database brought down by the Watchdog orthe Broker itself• Upgrade Progress (V7.3E with Patches,V8.2C with Patches)• Use Loopback Client connections (-H & -S)
  26. 26. www.BravePoint.comCorrupt Blocks• What Kind of Block? Index or Data• Block Type 2 (Index - IX) or 3 (Record Mgr - RM)• If IX block, try rebuilding the indexes• If RM block, consider the following• Reformat a block as a Free block (proutil dbrpr)• Replace the block with the same block fromanother DB (probably restored from a backup)• proutil dbrpr• 5. Dump Block (from the good database)• 4. Load Block (into the bad database)
  27. 27. www.BravePoint.comEmergency Dump• ‘Front and Back’ 4GL Dump• for each customer by cust-num (until you hitthe bad spot)• for each customer by cust-num descending• If the Primary Index is Damaged, try Dumpingusing a non-Primary Index• RECID Dump• Doesn’t require an Index• Very Slow on a Large Database• Last Resort
  28. 28. www.BravePoint.comMiscellaneous Hints• If you can’t get into the database with -F orany other way, try the Read Only (-RO) option
  29. 29. www.BravePoint.comDeleted Extents• First, backup the remaining pieces of thedatabase• This may seem like a useless step but if yourbackup is defective you may need to repair thebroken DB and that’s difficult if it’s deleted• The Backup gives you time to:• Prepare a plan of action• Call outside resources (like me) for help• Calm down• Lock your door• Prepare a new Resume (C.V.)
  30. 30. www.BravePoint.comDeleted AI Extents• If an AI Extent is Deleted, simply disable AIand...• What? You’re not running AI? BAD IDEA!• Disable AI (rfutil aimage end)• Fix the problem that caused the lost Extent• Recreate the Extent with prostrct add• Restart AI (rfutil aimage begin)• If this doesn’t work, go to the next slide
  31. 31. www.BravePoint.comDeleted AI Extents• Disable AI with rfutil aimage end. You may get anerror message regarding the missing AI Extentbut typically AI is still disabled• Truncate the BI file with proutil truncate bi. Youmay get an error message regarding the missingAI Extent but typically the BI file is still truncated• Remove all AI Extents with prostrct remove• Recreate the original AI Extents with prostrct add• Restart After Imaging with rfutil aimage begin• Reformat the truncated BI file with proutil bigrow
  32. 32. www.BravePoint.comDeleted BI Extents• Force Access with -F• V8.2 and later -F only forces access on proutiltruncate bi• If you Force Access, consider the DBdamaged!• Forcing Access THROWS AWAY the BI file• Forcing Access sets the ‘Tainted Flag’• Even if you fix the Tainted Flag, consider theDB damaged!• Dump & Load (this is if AI is not enabled)
  33. 33. www.BravePoint.comDeleted DB Extents• Restore the the DB and BI from Backup• Apply the AI files• Re-enable AI• BI Grow• Done!• Still not running AI?, next slide please
  34. 34. www.BravePoint.comDeleted DB Extents• Use prostrct unlock if the deleted Extentwas Empty (above the High Water Mark)• prostrct unlock will recreate missingExtents• However unlock also changes the timestamps on the AI files and they can’t beused any longer
  35. 35. www.BravePoint.comDeleted DB Extents• Backup Extent Substitute• This technique is for Extents that contain Dataexcept the .d1 Extent (.d1 contains Master Block)• Restore a copy of the deleted Extent from aBackup• The Extent’s ‘Last Opened’ time stamps won’tmatch• Use prostrct unlock to sync the time stamps(broken in V8.2 and V9)• The data in the Extent might not match but…• Use the -miracle option to re-create the Data 
  36. 36. www.BravePoint.comDeleted DB Extents• If the database Broker is still running:• DON’T Shutdown the Database• That ‘closes’ the database extents and youwon’t be able to re-open them• If a Client is still attached to the DB andthat Client can access the Progress Editor,simply Dump the Database from theDictionary• Even if they can’t get into the Editor, putdict.p (renamed as a menu item) into theirPROPATH
  37. 37. www.BravePoint.comDeleted DB Extents - Unix• Warm Boot the System Immediately• Don’t Shut Down the DB First• When Unix fsck runs, it will *probably*recover the deleted Extent• Why?• A file is not finally deleted until every processthat has it open is gone (the Broker still has itopen)
  38. 38. www.BravePoint.comOverlay Database• .db File is lost• Create a Void Multi-volume Structure thatmatches the problem Database (hopefullyyou have a current .st file• Take the .db from the Void Structure anduse it for the problem DB• Use prostrct unlock to Sync the ExtentTimestamps• The above steps apply if you are usingprobkup otherwise restore the .db from 3dparty backup
  39. 39. www.BravePoint.comOverlay Database• On V9 it is much easier to restore the .db file• prostrct builddb• Requires an up-to-date Structure File(remember that from the PreventativeMaintenance list?)
  40. 40. www.BravePoint.comDisk Full• Use prostrct repair to relocate Extents to alocation with more space• Copy the Extent• Create a new Structure File (.st) that reflectthe current location of the Database Extents(one good reason to have a current one)• Run prostrct repair new.st• Done!• There is a bug in prostrct repair. Don’tmisspell the name of the Structure File or your.db file will vanish forever (fixed in V8.3C)
  41. 41. www.BravePoint.comBI Disk Full• Do Not Run out of Space on the BI Disk if:• You can’t make more space for the BI file• You can’t relocate the BI file• To Perform Crash Recovery, the BI file mustgrow• If there is no space for the BI file to grow,there is no Crash Recovery• Force Access (-F) is the only option (if youdon’t have AI enabled)
  42. 42. www.BravePoint.comProgress HA Challenges• Online Backups• probkup online• AI Backup (already covered)• Split Mirror• Multi-DB backups cannot be restored toexactly the same point in time unless you areusing Two Phase Commit
  43. 43. www.BravePoint.comOnline Backup Options• probkup online• Transaction Freeze during BI Backup• Usually High I/O Volume• Only DB and BI are Backed Up• Performs an AI Extent Switch• If backup to disk, backup file can’t be greater2gb (except on NT) until V9.1B• Use the Volume Size (-vs) Parameter if backupfile is greater than 2gb
  44. 44. www.BravePoint.comOnline Backup Options• Split Mirror Backup• Quiet the DB with proquiet enable• Break the Disk Mirror• Unquiet the DB with proquiet disable• Backup the Offline Mirror• Resync the Mirrors
  45. 45. www.BravePoint.comOnline Backup Options• Split Mirror Backup• Database is vulnerable during the backupunless you have triple mirroring or similarfeature• Resynchronization of the Disks can be abig Performance Problem• Still Can’t Backup (and restore) MultipleDBs Simultaneously
  46. 46. www.BravePoint.comProgress HA Challenges• Schema Changes. It is not possible to makeSchema changes online, even in V9.1• Adding New Extents. It is not possible to addExtents online, even in V9.1• The BI Notes counter is limited to 2 billion untilV8.3D and V9.1C• Promon/VST Statistics. The numbers recorded bypromon are limited to 4 billion• Updating the Progress Version. It is necessary toshutdown the DB to change Progress versions• The DB log (.lg) file is limited to 2gb. A systemthat runs for months could approach this limit.There is no way to ‘truncate’ the log file online.
  47. 47. www.BravePoint.comProgress HA Challenges• Database Reorg. In V8 is possible to build indexesonline. In V9 it is possible move tables (andindexes) from one Area to another but thetablemove option has the following disadvantages:• The table being moved is EXCLUSIVEly lockedfor the duration of the move.• Because the RECIDs will be different in the newArea, all indexes on the table must be rebuilt.• The table is moved in one transaction. Thismeans that the BI file can grow quite large. If AI isenabled, the notes generated by the table move iswritten to the AI file the same way that othertransaction related activities are
  48. 48. www.BravePoint.comSources of Help• Progress Documentation• Progress Database Administration Guide• dba@peg.com• Progress Knowledgebase(techweb.progress.com)• My Home Number: 541-754-2116• My Mobile Number is: 541-829-7741• For those weekend emergencies when youneed expert assistance• This is not a free call
  49. 49. www.BravePoint.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×