環境概述 操作系統:WIN2019 ORACLE版本:19.17.0.0.0 故障現象 客戶反饋資料庫經常用著用著就連不上,通過查看alert日誌發現是資料庫實例崩潰。具體信息如下 2023-02-09T03:04:36.619773+08:00 Errors in file D:\APP\ADMI ...
環境概述
操作系統:WIN2019 ORACLE版本:19.17.0.0.0
故障現象
客戶反饋資料庫經常用著用著就連不上,通過查看alert日誌發現是資料庫實例崩潰。具體信息如下
2023-02-09T03:04:36.619773+08:00 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_smon_9116.trc (incident=114022): ORA-00600: internal error code, arguments: [13013], [5001], [423], [12633689], [5], [12633689], [17], [], [], [], [], [] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_114022\his_smon_9116_i114022.trc 2023-02-09T03:04:37.994869+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2023-02-09T03:04:38.010412+08:00 Non-fatal internal error happened while SMON was doing logging scn->time mapping. SMON encountered 10 out of maximum 100 non-fatal internal errors. 2023-02-09T03:04:38.775992+08:00 Dumping diagnostic data in directory=[cdmp_20230209030438], requested by (instance=1, osid=9116 (SMON)), summary=[incident=114022]. 2023-02-09T03:09:39.269309+08:00 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_smon_9116.trc (incident=114023): ORA-00600: internal error code, arguments: [13013], [5001], [423], [12633689], [5], [12633689], [17], [], [], [], [], [] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_114023\his_smon_9116_i114023.trc 2023-02-09T03:09:40.675580+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2023-02-09T03:09:40.691188+08:00 Non-fatal internal error happened while SMON was doing logging scn->time mapping. SMON encountered 11 out of maximum 100 non-fatal internal errors. 2023-02-09T03:09:41.347488+08:00 Dumping diagnostic data in directory=[cdmp_20230209030941], requested by (instance=1, osid=9116 (SMON)), summary=[incident=114023]. 2023-02-09T03:14:22.329332+08:00 PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old 2023-02-09T03:14:41.925792+08:00 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_smon_9116.trc (incident=114024): ORA-00600: internal error code, arguments: [13013], [5001], [423], [12633689], [5], [12633689], [17], [], [], [], [], [] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_114024\his_smon_9116_i114024.trc 2023-02-09T03:14:43.316412+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2023-02-09T03:14:43.332041+08:00 Non-fatal internal error happened while SMON was doing logging scn->time mapping. SMON encountered 12 out of maximum 100 non-fatal internal errors. 2023-02-09T03:14:43.910201+08:00 Dumping diagnostic data in directory=[cdmp_20230209031443], requested by (instance=1, osid=9116 (SMON)), summary=[incident=114024]. 2023-02-09T03:19:44.527920+08:00 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_smon_9116.trc (incident=114025): ORA-00600: internal error code, arguments: [13013], [5001], [423], [12633689], [5], [12633689], [17], [], [], [], [], [] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_114025\his_smon_9116_i114025.trc 2023-02-09T03:20:05.624807+08:00 Warning: VKTM detected a forward time drift. Time drifts can result in unexpected behavior such as time-outs. Please see the VKTM trace file for more details: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_vktm_3748.trc 2023-02-09T03:20:08.665424+08:00 PMON (ospid: 1716): terminating the instance due to ORA error 474 2023-02-09T03:20:08.712297+08:00 Cause - 'Instance is being terminated due to fatal process death (pid: 20, ospid: 9116, SMON)' 2023-02-09T03:20:08.743541+08:00 System state dump requested by (instance=1, osid=1716 (PMON)), summary=[abnormal instance termination]. System State dumped to trace file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_diag_808.trc 2023-02-09T03:20:25.485187+08:00 Instance terminated by PMON, pid = 1716
通過上面日誌可以看出來423這個對象存在問題,導致smon意外終止
參數解釋
ORA-00600: internal error code, arguments: [13013], [5001], [423], [12633689], [5], [12633689], [17], [], [], [], [], []
通過ORA-600報錯參數收集對象信息, ORA-600 [13013]參數說明如下:
-----------------------------------------------------------------------------------------
ORA-600 [13013] [a] [b] [c] [d] [e] [f]
Arg [a] Passcount
Arg [b] Data Object number
Arg [c] Tablespace Relative DBA of block containing the row to be updated
Arg [d] Row Slot number
Arg [e] Relative DBA of block being updated (should be same as [c])
Arg [f] Code
------------------------------------------------------------------------------------------
Arg[b]對應的是資料庫對象ID
解決方法
#定位問題 SQL> select object_name,object_type,owner from dba_objects where data_object_id="423"; OBJECT_NAME OBJECT_TYPE OWNER --------------- -------------------------------------- --------------- SMON_SCN_TO_TIME_AUX CLUSTER SYS #使用validate 檢驗表關聯對象的可用性 SQL> Analyze cluster sys.smon_scn_to_time_aux validate structure; Table analyzed. SQL> Analyze table sys.smon_scn_time validate structure online; Table analyzed. SQL> Analyze table sys.smon_scn_time validate structure cascade online; Analyze table sys.smon_scn_time validate structure cascade online * ERROR at line 1: ORA-01499: table/index cross reference failure - see trace file #解決辦法 SQL> drop index SYS.SMON_SCN_TIME_TIM_IDX; SQL> drop index SYS.SMON_SCN_TIME_SCN_IDX; SQL> CREATE UNIQUE INDEX SYS.SMON_SCN_TIME_TIM_IDX ON SYS.SMON_SCN_TIME(TIME_MP) tablespace sysaux; SQL> CREATE UNIQUE INDEX SYS.SMON_SCN_TIME_SCN_IDX ON SYS.SMON_SCN_TIME(SCN) tablespace sysaux; #再次驗證,對象已恢復正常。 SQL> Analyze table SYS.SMON_SCN_TIME validate structure cascade online ; Table analyzed.
通過查看報錯之前的日誌可以發現,資料庫3號文件存在壞塊
2023-02-09T03:00:15.435555+08:00 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_m003_6096.trc (incident=114030): ORA-01578: ORACLE data block corrupted (file # ORA-01578: ORACLE data block corrupted (file # 3, block # 183896) ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\DATAFILE\O1_MF_SYSAUX_KPMDGX61_.DBF' , block # ) Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_m003_6096.trc (incident=114031): ORA-01578: ORACLE data block corrupted (file # 3, block # 82310) ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\DATAFILE\O1_MF_SYSAUX_KPMDGX61_.DBF'
檢測數據文件是否存在壞塊
dbv
dbv file=D:\APP\ADMINISTRATOR\ORADATA\HIS\DATAFILE\O1_MF_SYSAUX_KPMDGX61_.DBF DBVERIFY: Release 19.0.0.0.0 - Production on 星期四 2月 9 10:52:03 2023 Copyright (c) 1982, 2022, Oracle and/or its affiliates. All rights reserved. DBVERIFY - 開始驗證: FILE = D:\APP\ADMINISTRATOR\ORADATA\HIS\DATAFILE\O1_MF_SYSAUX_KPMDGX61_.DBF 頁 65861 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c10145 (file 3, block 65861) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c10145 last change scn: 0x0000.0000.00dfdef6 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0xb9c00601 check value in block header: 0x95af computed block checksum: 0x6736 頁 73760 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c12020 (file 3, block 73760) Fractured block found during dbv: Data in bad block: type: 32 format: 2 rdba: 0x00c12020 last change scn: 0x0000.0000.00e8f1c5 seq: 0x1 flg: 0x04 spare3: 0x0 consistency value in tail: 0xda9e2015 check value in block header: 0xc548 computed block checksum: 0x2b4f 頁 76384 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c12a60 (file 3, block 76384) Fractured block found during dbv: Data in bad block: type: 32 format: 2 rdba: 0x00c12a60 last change scn: 0x0000.0000.00dfdb00 seq: 0x7 flg: 0x04 spare3: 0x0 consistency value in tail: 0x8bb82001 check value in block header: 0x376b computed block checksum: 0x50be 頁 82310 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c14186 (file 3, block 82310) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c14186 last change scn: 0x0000.0000.00e8183b seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0xbf200601 check value in block header: 0x1f15 computed block checksum: 0xa71b 頁 82318 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c1418e (file 3, block 82318) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c1418e last change scn: 0x0000.0000.00de0ca8 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0xa4f40601 check value in block header: 0xcdc7 computed block checksum: 0xa85c 頁 82342 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c141a6 (file 3, block 82342) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c141a6 last change scn: 0x0000.0000.00e866f1 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0x30f60601 check value in block header: 0x37b9 computed block checksum: 0x304 頁 82350 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c141ae (file 3, block 82350) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c141ae last change scn: 0x0000.0000.00dd30f6 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0x776f0601 check value in block header: 0xfbca computed block checksum: 0x4799 頁 82406 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c141e6 (file 3, block 82406) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c141e6 last change scn: 0x0000.0000.00e6bb63 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0x183b0601 check value in block header: 0x42c computed block checksum: 0xcf34 頁 82414 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c141ee (file 3, block 82414) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c141ee last change scn: 0x0000.0000.00e8183b seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0xbb630601 check value in block header: 0x3dbf computed block checksum: 0xa034 頁 94625 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c171a1 (file 3, block 94625) Fractured block found during dbv: Data in bad block: type: 6 format: 2 rdba: 0x00c171a1 last change scn: 0x0000.0000.00e78ae0 seq: 0x1 flg: 0x06 spare3: 0x0 consistency value in tail: 0x17860601 check value in block header: 0x981f computed block checksum: 0x9d66 頁 105985 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c19e01 (file 3, block 105985) Fractured block found during dbv: Data in bad block: type: 32 format: 2 rdba: 0x00c19e01 last change scn: 0x0000.0000.00e85de4 seq: 0x1 flg: 0x04 spare3: 0x0 consistency value in tail: 0xe79d2001 check value in block header: 0x18be computed block checksum: 0xba79 頁 107649 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c1a481 (file 3, block 107649) Fractured block found during dbv: Data in bad block: type: 32 format: 2 rdba: 0x00c1a481 last change scn: 0x0000.0000.00dfdf39 seq: 0x2 flg: 0x04 spare3: 0x0 consistency value in tail: 0x7b882001 check value in block header: 0x407 computed block checksum: 0xa4b2 頁 107729 流入 - 很可能是介質損壞 Corrupt block relative dba: 0x00c1a4d1 (file 3, block 107729) Fractured block found during dbv: Data in bad block: type: