Hotsos 2017 - Protect or Perform by Paul G. Matuszyk

1. Paul G. Matuszyk

2.  Introduction
 Corruptions –What are they?
 Recover options
 How to prevent corruptions?
 Impact on database
 Conclusions
 Questions

3.  Oracle (mainly) contractor
 Current contract in Poland
 8th time here in Dallas

5. Corruption is a form of dishonest conduct by a
person entrusted with a position of authority,
often to acquire personal benefit.
(Wikipedia)

6. Block ≠ Oracle expects

8.  Logical
 Physical

10.  Checksum OK
 Data inside block is bad
 Examples:
 Stale lock
 Wrong free space indicators
(avsp or tosp)
 Wrong block type
 ORA-600 reported

11.  Limited reporting
 Extended reporting by DBV

12.  ORA-1578

13.  Causes
 Oracle bug
 Failure of the Oracle software infrastructure
 Memory error
 Direct-load inserts with no-logging

15.  a.k.a media corruption
 Most common type of corruptions
 Any major damage to the block
 'zeroed' block
 Block checksums inconsistent
 Wrong internal block structures
 Oracle doesn’t recognize block as a
legitimate

16.  Problem with software
 Firmware
 Kernel
 File system corruption
 Controller error
 (...)
 Problem with hardware
 CPU or memory error
 Cache on storage array or disk
 (...)

17.  Intrablock – within single block
- physical
- logical
 Interblock – corruption between > 1 block
- only logical

19.  alert.log
 trace files
 stdout
 views
 v$database_block_corruption
 v$backup_corruption
 v$copy_corruption
 SQLs: ANALYZE table/index
 RMAN commands like:
validate / backup validate
list failure

21.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

22.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

23.  Check for logical / physical corruptions
 NO logical checks (by default)
 VALIDATE / BACKUPVALIDATE
 VALIDATE DATABASE;
 VALIDATE DATAFILE 1;
 VALIDATE DATAFILE 1 BLOCK 10;
 BACKUPVALIDATE DATABASEARCHIVELOGALL;
 BACKUPVALIDATE CHECK LOGICAL DATABASE
ARCHIVELOGALL;

25.  BLOCKRECOVER command
 v$database_block_corruption
and
RECOVER command

26.  RECOVER ... BLOCK ...

27.  Parallel operations
 Progress in:
 v$session_longops
and
 v$database_block_corruption

28.  NOARCHIVELOG  Block media recovery
 Only complete block media recovery
 Blocks unavailable until recovered
 Full backups only for BMR
 BMR only on corrupted blocks ;-)
 First block is unrecoverable

29.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

31.  Command line ($ORACLE_HOME/bin/dbv)
 Physical data structure check
 ONLINE / OFFLINE tool
 Modes: file level / segment level

32.  (-) Can be slow
 (+) easy to use
 (+) works offline

33.  dbv userid="'/ as sysdba‚'" file=TB_COMPRESS.dbf
 ASM / OMF / normal files

34.  dbv userid='/ as sysdba' file=TB_COMPRESS.dbf

35.  dbv …. segment_id=4.4.86289

36.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

37.  Introduced Oracle8i
 Online repair tool
 Two tables are created:
 Repair table
 Orphan key table
 Helps with:
 Detection
 Repair
 Index problems (pointing to corrupt data, index
status fix)

38.  DBMS_REPAIR.CHECK_OBJECT()
 DBMS_REPAIR.FIX_CORRUPT_BLOCKS()
Does NOT repair data
Marks invalid blocks
 DBMS_REPAIR.SKIP_CORRUPT_BLOCKS()

39.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

40.  To break or fix
 Metalink DocID 62015.1
 Shipped until 10g ... works in 12c
 Be careful while using it
 Be extremely careful while in EDIT MODE
 Allows to calculate block checksums

41.  Mark a block as valid (block sequence reset)
 Noarchivelog failures  problems with db
opening
 Datafiles from older backups

42.  RMAN
 DBV
 DBMS_REPAIR
 (BBED)
 Active Data Guard

43.  Physical (only) corruption  repair
 Automatic (mostly)
 Transparent to apps & users (mostly)
 Exceptions:
 Block header corruption
 > 100 block corruptions
 Timeout after 60 s.

44.  If exception  ORA-1578
 Manual block media recovery

46. Set init.ora parameters

47.  DB_BLOCK_CHECKING
 DB_BLOCK_CHECKSUM
 DB_LOST_WRITE_PROTECT
 DB_ULTRA_SAFE

48.  DB_BLOCK_CHECKING
 DB_BLOCK_CHECKSUM
 DB_LOST_WRITE_PROTECT
 DB_ULTRA_SAFE

49.  Introduced in Oracle8i
 Validate data in database blocks
 Memory corruption  data corruption
 Logical corruption
 Dynamic parameter (alter system)
 PDB modifiable
 Default: FALSE

50. Semantic block checking
Data in SYSTEM tbs is always checked

51.  Replaces events:
 10210 (tables)
 10211 (indexes)
 DB_BLOCK_CHECKING higher priority than
events

52.  Valid values
 FALSE
 TRUE
 OFF
 LOW
 MEDIUM
 FULL
For backward
compatibility
10gR2+

53.  Valid values for compatibility
 FALSE  OFF
 TRUE  FULL

54. OFF (FALSE)
 Default value: FALSE
 Oracle's internal data  checks on
 User data  checks off

55. LOW
Oracle's internal data  checks on
User data  basic checks

56. MEDIUM
Oracle's internal data  checks on
User data  LOW and
full semantic checks
but NOT on indexes

57. FULL (TRUE)
Oracle's internal data  checks on
User data  LOW+MEDIUM
full semantic checks
also on indexes

58.  DB_BLOCK_CHECKING
 DB_BLOCK_CHECKSUM
 DB_LOST_WRITE_PROTECT
 DB_ULTRA_SAFE

59.  From (at least) Oracle7
 Enables checksum calculation & checks
 Physical corruption
 Dynamic parameter
 Alter system / alter session
 PDB
 Defaults to TYPICAL

60.  XOR on DATA in the block
 READS affected more
 WRITES affected less
 Oracle says:
 TYPICAL  1-2%
 FULL  4-5%

61.  Checksum calculation for :
 Data block
 Log blocks
 Valid values
 FALSE
 TRUE
 OFF
 TYPICAL
 FULL
For backward
compatibility
10gR2+

62. OFF (FALSE)
Oracle's internal data  checks on
User data  checks off

63. TYPICAL (TRUE)
Oracle's internal data  checks on
User data  checks on
(when applicable)

64. FULL
Oracle's internal data  checks on
User data  checks on
(when applicable)
logs checksumed
in-memory errors

65.  DB_BLOCK_CHECKING
 DB_BLOCK_CHECKSUM
 DB_LOST_WRITE_PROTECT
 DB_ULTRA_SAFE

66.  New in 11gR1
(10gR2 „_db_lost_write_checking”)
 Record SCN of read block into REDO stream
 Dynamic (alter system)
 Meant for Data Guard environments
 Must be the same for all instances (RAC)
 Other techniques may not detect lost writes
 ~50 bytes / read block

67.  NONE  functionality disabled (default)
 TYPICAL  READS logged for
R/W tablespaces
 FULL  READS logged for
R/W and R/O tablespaces

68.  12cR1 new property
 PrimaryLostWriteAction =
CONTINUE  Primary: BAU (continue)
SHUTDOWN  Primary: shutdown abort
 Diag info written:
 Primary site: alert.log + DG Broker log
 Standby site: alert.log + DG Broker log

70.  Faster lost writes detection
 Solution:
- failover to standby (if LW in PROD)
- recreate standby (if LW in STBY)
- FIX STORAGE

71.  Impact on performance:
 Primary db – increased REDO stream
 Standby db - validate + additional reads

72.  DB_BLOCK_CHECKING
 DB_BLOCK_CHECKSUM
 DB_LOST_WRITE_PROTECT
 DB_ULTRA_SAFE

73.  New in Oracle11g
 Static parameter
 PDB unmodifiable
 Defaults to OFF
DB_ULTRA_SAFE = { OFF | DATA_ONLY | DATA_AND_INDEX }
DB_ULTRA_SAFE = { OFF | DATA_ONLY | DATA_AND_INDEX }

74.  OFF
 DATA_ONLY
 DB_BLOCK_CHECKING = MEDIUM
 DB_BLOCK_CHECKSUM = FULL
 DB_LOST_WRITE_PROTECT =TYPICAL
 DATA_AND_INDEX
 DB_BLOCK_CHECKING = FULL
 DB_BLOCK_CHECKSUM = FULL
 DB_LOST_WRITE_PROTECT =TYPICAL
DB_ULTRA_SAFE = { OFF | DATA_ONLY | DATA_AND_INDEX }

75. alter system set db_ultra_safe=DATA_ONLY scope=spfile;
-- instance restart

76. alter system set db_block_checking=FALSE scope=both;
-- instance restart

77. alter system set db_ultra_safe=DATA_ONLY scope=spfile;
-- instance restart

78.  Not very useful
 Static
 Can introduce confusion
 Why to use it?

80.  Intel Xeon CPU
 4TB SSD for the DB
 8 vCPU
 16GB RAM
 VMware

81. Users off low medium full %
30 265 344 261 255 104
70 411 630 409 391 105
100 653 402 370 367 178
db_block_checksum=OFF
db_block_checking = …

82. Users off low medium full %
30 23 17 24 26 110
70 30 21 32 35 117
100 38 47 56 57 151
db_block_checksum=OFF
db_block_checking = …

83. Users off low medium full %
30 235 191 200 183 128
70 272 248 273 211 129
100 427 225 189 191 223
db_block_checksum=FULL
db_block_checking = …

84. Users off low medium full %
30 39 47 54 54 139
70 75 89 107 107 142
100 86 131 161 161 187
db_block_checksum=FULL
db_block_checking = …

85. db_block_checksum= …
db_block_checking = …
Users: db_block_checking
30 off low medium full
db_block_checksum
off 265 344 261 255
full 235 191 200 183

86. db_block_checksum= …
db_block_checking = …
Users: db_block_checking
30 off low medium full
db_block_checksum
off 23 17 24 26
full 39 48 55 55

88.  Corruptions are rare
 Corruptions  serious consequencies
 „A stitch in time saves nine”
 Impact can be huge:

89.  Db_block_checking  medium for new DBs
 If I/O or CPU usage high:
PRI: db_block_checksum OFF / LOW
PRI: db_block_checking OFF
 SBY: enable
 Consider DB_LOST_WRITE_PROTECT
(dynamic / revertible)

90.  Oracle-base.com (Tim Hall)
 Oracle DOC & Metalink
 Blogs @ Pythian.com
 Disassembling the Oracle Data Block by
GrahamThornton (2005)
 (many others)

91. Paul G. Matuszyk
matuszyk@matuszyk.com
@pmatuszy