This document discusses how to prevent corruption in Firebird databases. It outlines key things to monitor like server parameters, database files, backups, and indices to recognize potential problems. Regular maintenance like validating metadata, checking backups, and analyzing logs and errors is important. The document also introduces FBDataGuard, a tool that automates monitoring of 26 database and server parameters, alerts of issues, and helps maintain databases to prevent corruption from hardware failures or other causes.

1. Anti-corruptionHow to prevent Firebird database corruptionAlexey Kovyazin,IBSurgeon,ak@ib-aid.com

2. IBSurgeon – 8 years!ProductsIBFirstAID/FBFirstAID, etcFBScannerFBDataGuardClientsCarl Zeiss Meditec, USAVneshtorgbank, RussiaWells FargoBank, USAWatermark Software, UKBas-X, AustraliaVictoria University, New ZealandKingsway Management, UKTeamDmitry KuzmenkoAlexey KovyazinSergey NikitinOleg Mateveev& teamConsultantsDmitry Yemanov, VladKhorsun, Alex PeshkoffPartnersIBPhoenix

3. Alexey KovyazinHosting & cloud partnersYes, it’s me! http://ru.linkedin.com/in/kovyazin2006200820092010In 2007 we sold 3mln of Delphi to all Russian schools

4. AgendaWhy bother?Why corruption happens?ReasonsSymptomsWhat things are to monitor to recognize problem?Problems with server Problems with environmentProblems with databaseMaintenance improvements to prevent corruptions.BackupsWhy we created FBDataGuad?

5. Why bother?Firebird databases become bigger and bigger every yearInformation inside Firebird can costs $XXXXXXOutage (corruption, backup/restore breaks) can costs $XXXXX tooReal-world examplesBas-X, Australia – Firebird 2.x, 250Gb, no BLOBs, 250 usersWatermark Software, UK – Firebird 2.x, up to 400Gb, with BLOBs Profitmed, Russia, medical distribution, Firebird 1.5, 65Gb, 250 users1 Terabyte Firebird 2.1 databasehttp://www.ib-aid.com/articles/item1043.8Billions of records in the biggest table

6. Corruption reasons & symptonsReasonsMisadministrationHardware failuresBugsThe actual reason often remains undiscoveredSymptomsRepeatableTrackableComplimentaryWe can prevent corruption if see its symptoms.

7. Sample Firebird environmentServerDatabaseFirebirdCopy of backupsBackups

8. What to monitor at Firebird instance levelIs Server online?General parametersHow much RAM?Mb#Temp files?Records to analyze6 levelsLogsSize of logsIs it recommended? Bugs, issuesServer version

9. Firebird instance key parametersServer availabilityConsumed RAM Temp files size Temp files quantityRecords in logsLogs’ sizeServer version related issuesNeed to watch for 7 key parameters which can indicate possible or actual problems

10. What to monitor-1General database checksDatabase availability -> Outages, firewalls, stabilityLog records related with database in firebird.log -> early symptomsCheck metadata – validate all metadata -> early showingsTransactionsTransaction markers monitoring (garbage problems)Limit (2 billions between backup/restore)UsersMin/max/avg users –> peaks problems, design of application

11. What to monitor-2Database filesSingle volume and multi-volume -> Volumes in binPaths – where to stored (not at the same drive with temp files and backups!)Sizes and growth limits -> Warnings about growthDelta-files (nbackup)Life-time and sizes -> Huge/aged delta problemsBackup filesExistence, sizes and growth limits -> Backup could kill database

12. What to monitor-3Number of formats per tableNo more than 255 -> corruptionLess formats in production -> performance problemNon-activated and deactivated indicesDeactivated – explicitly deactivated (why deactivated?)Non-activated – indicates problems during restore

13. What to monitor-4Periodical statistics (gstat) -> deep look into databaseFirebird server versionExamples - problems with nbackupLatest patches are recommendedFirebird fbclient.dll versionIf fbclient.dll <> fbserver - > Problems (disconnects, 10054, errors)Firebird installation sizeDefault database place is %Firebird%\BinFirebird logs size and pathsBig logs quickly exhaust space -> corruption

14. Maintenance-1BackupsRevolver (days, week, month copies) backupsBackup depthChecking restore (need to check results)Growth prognosis (if not enough space, backup should be canceled)Control backup time (too long backup indicates problems)Today5..7YesterdayWeekly

15. Big database requires individual maintenance plan Maintenance plan depends on size of database and work mode (8x5, 24x7)Backups scheme is not simplePerform test restores separatelyTo be checkedErrors – in firebird.log and run error checking quries on live databaseMetadata – check integrity Metadata limitsData & BLOBs – walk through data, check segmentationIndices – check indices healthTransactions – any gaps, garbage growth, other problems

16. Everyday minimal (!) maintenance plan for big database

17. Example of backup plan for big Firebird databaseMaintenance serverMain serverFirebird databaseNbackup copyChecking restoreGbak-bAnd each step should be confirmed and reported.

18. Maintenance-2IndicesRecalculate indices statistics -> PerformanceSelected or excludedCheck index status – active/in-active/non-activated -> Problems, corruptionsCheck physical index healthEarly showings of corruptions

19. Maintenance-3Validate database with gfixDon’t forget to shutdown databaseAnalysis (includingfirebird.log)Metadata validationCheck important system tablesFirebird.log maintenanceWhen log becomes very big, copy it to backup log filesAnd some more things….

20. And this is not enough!Business wants to have warranty - even if hardware fails data should be recovered!

21. A big jobImplement scriptsCheck them in the test environmentExplore errors messages and codes of FirebirdWe spent 6 years getting the necessary information…

22. That’s why we created Fbdataguard

23. FBDataGuarddoes all above things…Watches database files, volumes, deltas, performs and checks backups in the right wayVerifies metadata, data and indicesWatches for errors, limits and wrong versionsSends alerts and recommendations

24. Example with TEMPFBDataGuard found the temp files size = NNot enough spaceM – N<XNot enough space – administrator will have alert and recommendation to increase TEMPFree space at TEMP- locations= M

25. Example alertToo big temporary files Total size of all temporary files 3 Gb is more than recommended: 500 Mb Firebird creates temporary files for some SQL queries (PLAN SORT). Too big size of temporary files can indicate performance problems. This is not a strictly defined number, so this threshold depends on particular database and application.

26. Index problem examplenon-activatedindices usually indicates corruption (missed Foreign Keys)FBDataGuard found non-activated index after restoreAdministrator will get alert and recommendation to check indicesPossible perfomance problem prevented!

27. Example of backup problem resolutionFBDataGuard found the backup size =MNot enough spaceM>=NBackup cancelled, database status is set toCritical, administrator gotalertFBDataGuard found free space at backups’ disk = NCorruption of backup was prevented!

28. Example of backup problem alertJob backup@[ server-0000 / db-0000 ] malfunctionUnexpected job backup@[ server-0000 / db-0000 ] error: There is not enough space on the disk

29. Example of good backup notification

30. Hardware and UNDELETE failuresHDD corruptionFlash-drive corruptionUNDELETE problem

31. And even more – protects from hardware failuresMetadata repositoryFBDataGuard Extractor extracts data from corrupted database and inserts to the newNew DBTables dataBLOBs

32. Firebird DataGuardWatch for 26 important database and server parametersAlerts for potential and real problem by emailProper automation of database maintenanceWindows, Linux, MacOS, Firebird 1.5-2.1 (not 2.5 yet)Special licensing for ISV (Independent Software Vendors) Firebird developers

33. Get FBDataGuard 1 yearFree 1 year license for all attendeesSend request to dataguard@ib-aid.com

34. Thank youQuestions? dataguard@ib-aid.com