The control file was lost and had to be recreated manually. When attempting to open the database, ORA-600 errors occurred due to block SCN being ahead of current SCN. This was caused by an improperly set hidden parameter. The level was adjusted, but ORA-607 errors still occurred on open due to corrupted rollback segments. A hidden parameter was set to list corrupted segments and new undo tablespace was created to resolve the issue.

1. A Recovery Case:control file lost Sidney Chen 2011-Apr-05

2. Control file is missing, alter database mount failed

3. SQL> startup ORACLE instance started. Total System Global Area 3559873248 bytes Fixed Size 731872 bytes Variable Size 1493172224 bytes Database Buffers 2063597568 bytes Redo Buffers 2371584 bytes ORA-00205: error in identifying controlfile, check alert log for more info

4. confirm the control file are lost from alert log

5. Sun Apr 3 10:13:48 2011 Errors in file /export/home/icssprd/admin/bdump/icssprd_lgwr_326.trc: ORA-00210: cannot open the specified controlfile ORA-00202: controlfile: '/export/home/icssprd/data/data02/icssprd_ctrl01.ctl' ORA-27041: unable to open file HPUX-ia64 Error: 2: No such file or directory Additional information: 3 Sun Apr 3 10:13:48 2011 LGWR: terminating instance due to error 210 Instance terminated by LGWR, pid = 326 Mon Apr 4 10:15:50 2011

6. create control file Manually

7. CREATE CONTROLFILE REUSE DATABASE "ICSSPRD" RESETLOGS ARCHIVELOG MAXLOGFILES 48 MAXLOGMEMBERS 4 MAXDATAFILES 1024 MAXINSTANCES 1 MAXLOGHISTORY 4084 LOGFILE GROUP 1 ( '/export/home/icssprd/data/data01/icssprd_redo_01a.rdo', '/export/home/icssprd/data/data12/icssprd_redo_01b.rdo' ) SIZE 1000M, . . . GROUP 8 ( '/export/home/icssprd/data/data10/icssprd_redo_08a.rdo', '/export/home/icssprd/data/data16/icssprd_redo_08b.rdo' ) SIZE 1000M DATAFILE '/export/home/icssprd/data/data01/icssprd_system_01.dbf', '/export/home/icssprd/data/data14/icssprd_rbs_01.dbf', . . . '/export/home/icssprd/data/data34/icssprd_css_ref_data_04.dbf', '/export/home/icssprd/data/data34/icssprd_css_ref_indx_05.dbf' CHARACTER SET US7ASCII

8. "Alter database open resetlogs" failed due to ora-00600 2662

9. Mon Apr 4 12:27:07 2011 Errors in file /export/home/icssprd/admin/udump/icssprd_ora_15737.trc: ORA-00600: internal error code, arguments: [2662], [8], [1439299167], [8], [1441267038], [25165833], [], [] Mon Apr 4 12:27:08 2011 Errors in file /export/home/icssprd/admin/udump/icssprd_ora_15737.trc: ORA-00600: internal error code, arguments: [2662], [8], [1439299167], [8], [1441267038], [25165833], [], [] Mon Apr 4 12:27:08 2011 Error 600 happened during db open, shutting down database USER: terminating instance due to error 600 Instance terminated by USER, pid = 15737 ORA-1092 signalled during: Alter database open... Mon Apr 4 12:32:09 2011 USER: terminating instance due to error 1092 Instance terminated by USER, pid = 15737

10. ORA-600 [2662] "Block SCN is ahead of Current SCN" ORA-600 [2662] [a] [b] [c] [d] [e] ARGUMENTS: Arg [a] Current SCN WRAP Arg [b] Current SCN BASE Arg [c] dependent SCN WRAP Arg [d] dependent SCN BASE Arg [e] Where present this is the DBA where the dependent SCN came from.

11. ORA-00600: internal error code, arguments: [2662], [8], [1439299167], [8], [1441267038], [25165833], [], [] Current SCN WRAP=8 Current SCN Base = 1439299167 dependent SCN WRAP=8 dependent SCN BASE= 1441267038 Current SCN Base < dependent SCN BASE

12. When scn base up to 4 * 1024 * 1024 * 1024 SCN Wrap = SCN Wrap + 1 SCN Base reset to 1

13. ALTER SESSION SET EVENTS '10015 TRACE NAME ADJUST_SCN LEVEL N'; N = 1024*1024*1024

14. N=? dependent SCN WRAP=8 dependent SCN BASE= 1441267038 Set Current SCN WRAP =9 N = (8+1) * 4 = 36

15. Wrongly Set level=1 startup mount; ALTER SESSION SET EVENTS '10015 TRACE NAME ADJUST_SCN LEVEL 1'; ALTER DATABASE OPEN;

16. SQL> startup mount; ORACLE instance started. Total System Global Area 3559873248 bytes Fixed Size 731872 bytes Variable Size 1493172224 bytes Database Buffers 2063597568 bytes Redo Buffers 2371584 bytes Database mounted. SQL> ALTER SESSION SET EVENTS '10015 TRACE NAME ADJUST_SCN LEVEL 1'; Session altered. SQL> ALTER DATABASE OPEN; ALTER DATABASE OPEN * ERROR at line 1: ORA-01113: file 1 needs media recovery ORA-01110: data file 1: '/export/home/icssprd/data/data01/icssprd_system_01.dbf'

17. Mon Apr 4 14:12:18 2011 Errors in file /export/home/icssprd/admin/udump/icssprd_ora_25729.trc: ORA-00600: internal error code, arguments: [2256], [0], [1073741824], [8], [1439319171], [], [], [] Mon Apr 4 14:12:19 2011 Errors in file /export/home/icssprd/admin/udump/icssprd_ora_25729.trc: ORA-00600: internal error code, arguments: [2256], [0], [1073741824], [8], [1439319171], [], [], [] Mon Apr 4 14:12:19 2011 Error 600 happened during db open, shutting down database USER: terminating instance due to error 600 Instance terminated by USER, pid = 25729 ORA-1092 signalled during: ALTER DATABASE OPEN...

18. 1073741824 = 1024 * 1024 * 1024

19. Adjust LEVEL to 36 startup mount; ALTER SESSION SET EVENTS '10015 TRACE NAME ADJUST_SCN LEVEL 36'; ALTER DATABASE OPEN;

20. Errors in file /export/home/icssprd/admin/udump/icssprd_ora_6821.trc: ORA-00607: Internal error occurred while making a change to a data block ORA-00600: internal error code, arguments: [4194], [52], [33], [], [], [], [], [] Error 607 happened during db open, shutting down database USER: terminating instance due to error 607 Instance terminated by USER, pid = 6821 ORA-1092 signalled during: ALTER DATABASE OPEN...

21. Hidden Parameter _corrupted_rollback_segments ._corrupted_rollback_segments='rbs01','rbs02','rbs03','rbs04','rbs05','rbs06','rbs07','rbs08','rbs09','rbs10','rbs11','rbs12','rbs_batch1','rbs_batch2','rbs_batch3'

22. $ORACLE_HOME/dbs/initicssprd.ora rollback_segments = (rbs01,rbs02,rbs03,rbs04,rbs05,rbs06,rbs07,rbs08,rbs09,rbs10,rbs11,rbs12,rbs_batch1,rbs_batch2,rbs_batch3)

23. If the undo management is auto undo_management = auto

24. [Itanium] oracle@qadb01:/export/home/icssprd/data/data01> strings icssprd_system_01.dbf | grep _SYSSMU | cut -d $ -f 1 | sort -u _SYSSMU16 _SYSSMU17 _SYSSMU18 _SYSSMU19 _SYSSMU20 _SYSSMU21 _SYSSMU22 _SYSSMU23 _SYSSMU24 _SYSSMU25

25. create new undo tablespace, swtch default undo tablespace, drop the obsolete undo tablespace

26. Reference Oracle Diag:如何处理ORA-600 2662错误 http://www.eygle.com/archives/2005/12/oracle_diagnostics_howto_deal_2662_error.html http://goo.gl/UobE7 https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=HOWTO&id=281429.1